Do Vision Models Truly Forget? New Findings from Representation-Level Certification of Visual Unlearning in Vertical Federated Learning

Code

• github.com/yuzhenyulindy/mirage
  OfficialIn paper★ 0

Abstract

Machine unlearning in Vertical Federated Learning (VFL) has attracted growing interest, yet existing methods certify forgetting solely using output-level metrics. We challenge these works by introducing Mirage, a representation-level auditing framework that comprises four complementary diagnostics: Linear probe recovery (LPR), centered kernel alignment (CKA), feature separability scoring, and layer-wise recovery analysis. Extensive experiments across seven datasets and seven baseline methods following recent VFL unlearning protocols reveal three key findings: (1) Forgetting gap: methods that pass output-level certification still retain substantial class structure in their representations, with LPR exceeding the retrained baseline by up to 15.4 points; CKA shows that these models remain structurally closer to the original than to the retrained reference, while separability scores indicate persistent geometric discrimination. (2) Unlearning trilemma: no existing method simultaneously achieves high utility, output-level forgetting, and representation-level forgetting. (3) Class-sample asymmetry: class-level forgetting leaves strong representational traces (LPR exceeding 96 percent on several datasets), whereas sample-level forgetting is indistinguishable from chance (LPR is approximately 50 percent); layer-wise analysis further shows that residual class information persists across network depths. These findings call for representation-aware evaluation standards in federated unlearning research. Code is publicly available at https://github.com/YuZhenyuLindy/Mirage.

Reproductions