New Framework for Auditing Machine Unlearning

Google introduces a new framework called Regularized f-Divergence Kernel Tests, designed for auditing machine unlearning. Machine unlearning allows AI systems to "forget" specific parts of training data without full retraining, which is crucial for regulatory compliance (e.g., the "right to be forgotten" under the General Data Protection Regulation, GDPR), AI safety, and model quality. The traditional verification method is two-sample testing, which compares the output distributions of a model that never saw a record and one that has "forgotten" it. However, as models grow, this method loses statistical power and requires a huge number of samples, making it computationally expensive. The new framework, presented at the 26th International Conference on Artificial Intelligence and Statistics in 2026 (AISTATS 2026), uses regularized tests based on f-divergences and automatically selects the best divergence and hyperparameters, eliminating the need for manual tuning. The authors theoretically prove that the tests control false positives for any sample size, and the risk of false negatives tends to zero with increasing data. The framework also offers a relative three-sample test that compares the forgetting model with a safely retrained one and the original compromised model, addressing the impossibility of achieving perfect retrain equivalence. In experiments on synthetic data, high-energy physics data (search for rare particles), auditing of differential privacy, and evaluation of forgetting methods, the framework outperformed or matched baseline methods while requiring fewer samples and less tuning. For example, in privacy auditing, the hockey-stick divergence test detected violations in the SVT3 mechanism using several thousand samples, whereas DP-Auditorium required millions. In forgetting evaluation, only the random label method passed the test, while finetuning, pruning, and Selective Synaptic Dampening proved ineffective. The authors emphasize that they used simplified implementations; more rigorous settings are needed for production environments. This work was done jointly with Antonín Schrab and Arthur Gretton.