Efficient Machine Unlearning via Influence Approximation
Due to growing privacy concerns, machine unlearning, which aims at enabling
machine learning models to ``forget" specific training data, has received
increasing attention. Among existing methods, influence-based unlearning has
emerged as a prominent approach due to its ability to estimate the impact of
individual training samples on model parameters without retraining. However,
this approach suffers from prohibitive computational overhead arising from the
necessity to compute the Hessian matrix and its inverse across all training
samples and parameters, rendering it impractical for large-scale models and
scenarios involving frequent data deletion requests. This highlights the
difficulty of forgetting. Inspired by cognitive science, which suggests that
memorizing is easier than forgetting, this paper establishes a theoretical link
between memorizing (incremental learning) and forgetting (unlearning). This
connection allows machine unlearning to be addressed from the perspective of
incremental learning. Unlike the time-consuming Hessian computations in
unlearning (forgetting), incremental learning (memorizing) typically relies on
more efficient gradient optimization, which supports the aforementioned
cognitive theory. Based on this connection, we introduce the Influence
Approximation Unlearning (IAU) algorithm for efficient machine unlearning from
the incremental perspective. Extensive empirical evaluations demonstrate that
IAU achieves a superior balance among removal guarantee, unlearning efficiency,
and comparable model utility, while outperforming state-of-the-art methods
across diverse datasets and model architectures. Our code is available at
https://github.com/Lolo1222/IAU.
688c929b8ed271fb327f8f71
Jiawei Liu
688c929b8ed271fb327f8f72
Chenwang Wu
688c929b8ed271fb327f8f73
Defu Lian
688c929b8ed271fb327f8f74
Enhong Chen
Efficient Machine Unlearning via Influence Approximation
Due to growing privacy concerns, machine unlearning, which aims at enabling
machine learning models to ``forget" specific training data, has received
increasing attention. Among existing methods, influence-based unlearning has
emerged as a prominent approach due to its ability to estimate the impact of
individual training samples on model parameters without retraining. However,
this approach suffers from prohibitive computational overhead arising from the
necessity to compute the Hessian matrix and its inverse across all training
samples and parameters, rendering it impractical for large-scale models and
scenarios involving frequent data deletion requests. This highlights the
difficulty of forgetting. Inspired by cognitive science, which suggests that
memorizing is easier than forgetting, this paper establishes a theoretical link
between memorizing (incremental learning) and forgetting (unlearning). This
connection allows machine unlearning to be addressed from the perspective of
incremental learning. Unlike the time-consuming Hessian computations in
unlearning (forgetting), incremental learning (memorizing) typically relies on
more efficient gradient optimization, which supports the aforementioned
cognitive theory. Based on this connection, we introduce the Influence
Approximation Unlearning (IAU) algorithm for efficient machine unlearning from
the incremental perspective. Extensive empirical evaluations demonstrate that
IAU achieves a superior balance among removal guarantee, unlearning efficiency,
and comparable model utility, while outperforming state-of-the-art methods
across diverse datasets and model architectures. Our code is available at
https://github.com/Lolo1222/IAU.
The paper introduces the Influence Approximation Unlearning (IAU) algorithm, which leverages incremental learning principles to efficiently address the computational challenges of influence-based unlearning in machine learning models.
machine unlearning influence-based unlearning Hessian matrix incremental learning gradient optimization Influence Approximation Unlearning (IAU)