ReLearn: Unlearning via Learning for Large Language Models
Current unlearning methods for large language models usually rely on reverse
optimization to reduce target token probabilities. However, this paradigm
disrupts the subsequent tokens prediction, degrading model performance and
linguistic coherence. Moreover, existing evaluation metrics overemphasize
contextual forgetting while inadequately assessing response fluency and
relevance. To address these challenges, we propose ReLearn, a data augmentation
and fine-tuning pipeline for effective unlearning, along with a comprehensive
evaluation framework. This framework introduces Knowledge Forgetting Rate (KFR)
and Knowledge Retention Rate (KRR) to measure knowledge-level preservation, and
Linguistic Score (LS) to evaluate generation quality. Our experiments show that
ReLearn successfully achieves targeted forgetting while preserving high-quality
output. Through mechanistic analysis, we further demonstrate how reverse
optimization disrupts coherent text generation, while ReLearn preserves this
essential capability. Code is available at https://github.com/zjunlp/unlearn.
67b420dfb2528c023491f455
Haoming Xu
67b420dfb2528c023491f456
Ningyuan Zhao
admin_assigned
2025-02-19T15:20:38.762Z
6641f74c3bd2f40eb41fb0d8
/avatars/b0167a521b6e4aea27e245dd9e026ef3.svg
zhaoningyuan
user
zhaoningyuan
67b420dfb2528c023491f457
Liming Yang
67b420dfb2528c023491f458
Sendong Zhao
67b420dfb2528c023491f459
Shumin Deng
admin_assigned
2025-02-19T15:20:59.377Z
6441f1d2603214724ec0c1c2
/avatars/d3c4b759e6a5635e37ff715fae52e5ba.svg
Shumin Deng
user
231sm
67b420dfb2528c023491f45a
Mengru Wang
admin_assigned
2025-02-19T15:21:08.761Z
64bf898d979949d2e2585c9a
/avatars/da77c856ec997e2b812c06272a01c8b2.svg
mengruwang
user
mengru
67b420dfb2528c023491f45b
Bryan Hooi
admin_assigned
2025-02-19T15:21:14.704Z
651d8032c50012d33e914f2f
/avatars/0a44c9f51fc50ce86582e328c361ea00.svg
Bryan Hooi
user
bhooi
67b420dfb2528c023491f45c
Nay Oo
67b420dfb2528c023491f45d
Huajun Chen
admin_assigned
2025-02-19T15:21:32.744Z
64931296137833d7ec7689cd
https://cdn-avatars.huggingface.co/v1/production/uploads/64931296137833d7ec7689cd/TBihNdp1ZwIWjhfAWjRr6.jpeg
Huajun Chen
user
huajunsir
67b420dfb2528c023491f45e
Ningyu Zhang
claimed_verified
2025-02-18T09:31:11.243Z
620b3bbb0668e435407c8d0a
/avatars/e0fccbb2577d76088e09f054c35cffbc.svg
Ningyu Zhang
user
Ningyu
ReLearn: Unlearning via Learning for Large Language Models
Current unlearning methods for large language models usually rely on reverse
optimization to reduce target token probabilities. However, this paradigm
disrupts the subsequent tokens prediction, degrading model performance and
linguistic coherence. Moreover, existing evaluation metrics overemphasize
contextual forgetting while inadequately assessing response fluency and
relevance. To address these challenges, we propose ReLearn, a data augmentation
and fine-tuning pipeline for effective unlearning, along with a comprehensive
evaluation framework. This framework introduces Knowledge Forgetting Rate (KFR)
and Knowledge Retention Rate (KRR) to measure knowledge-level preservation, and
Linguistic Score (LS) to evaluate generation quality. Our experiments show that
ReLearn successfully achieves targeted forgetting while preserving high-quality
output. Through mechanistic analysis, we further demonstrate how reverse
optimization disrupts coherent text generation, while ReLearn preserves this
essential capability. Code is available at https://github.com/zjunlp/unlearn.
ReLearn mitigates issues in current unlearning methods for large language models by proposing a data augmentation and fine-tuning pipeline that effectively targets forgetting while preserving quality and coherence, using novel metrics for evaluation.
reverse optimization unlearning data augmentation fine-tuning Knowledge Forgetting Rate Knowledge Retention Rate Linguistic Score coherent text generation