title: ReLearn: Unlearning via Learning for Large Language Models
summary: Current unlearning methods for large language models usually rely on reverse
  optimization to reduce target token probabilities. However, this paradigm
  disrupts the subsequent tokens prediction, degrading model performance and
  linguistic coherence. Moreover, existing evaluation metrics overemphasize
  contextual forgetting while inadequately assessing response fluency and
  relevance. To address these challenges, we propose ReLearn, a data augmentation
  and fine-tuning pipeline for effective unlearning, along with a comprehensive
  evaluation framework. This framework introduces Knowledge Forgetting Rate (KFR)
  and Knowledge Retention Rate (KRR) to measure knowledge-level preservation, and
  Linguistic Score (LS) to evaluate generation quality. Our experiments show that
  ReLearn successfully achieves targeted forgetting while preserving high-quality
  output. Through mechanistic analysis, we further demonstrate how reverse
  optimization disrupts coherent text generation, while ReLearn preserves this
  essential capability. Code is available at https://github.com/zjunlp/unlearn.
paper_authors._id: 67b420dfb2528c023491f455
paper_authors.name: Haoming Xu
paper_authors._id: 67b420dfb2528c023491f456
paper_authors.name: Ningyuan Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T15:20:38.762Z
paper_authors._id: 6641f74c3bd2f40eb41fb0d8
paper_authors.avatarUrl: /avatars/b0167a521b6e4aea27e245dd9e026ef3.svg
paper_authors.fullname: zhaoningyuan
paper_authors.type: user
paper_authors.user: zhaoningyuan
paper_authors._id: 67b420dfb2528c023491f457
paper_authors.name: Liming Yang
paper_authors._id: 67b420dfb2528c023491f458
paper_authors.name: Sendong Zhao
paper_authors._id: 67b420dfb2528c023491f459
paper_authors.name: Shumin Deng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T15:20:59.377Z
paper_authors._id: 6441f1d2603214724ec0c1c2
paper_authors.avatarUrl: /avatars/d3c4b759e6a5635e37ff715fae52e5ba.svg
paper_authors.fullname: Shumin Deng
paper_authors.type: user
paper_authors.user: 231sm
paper_authors._id: 67b420dfb2528c023491f45a
paper_authors.name: Mengru Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T15:21:08.761Z
paper_authors._id: 64bf898d979949d2e2585c9a
paper_authors.avatarUrl: /avatars/da77c856ec997e2b812c06272a01c8b2.svg
paper_authors.fullname: mengruwang
paper_authors.type: user
paper_authors.user: mengru
paper_authors._id: 67b420dfb2528c023491f45b
paper_authors.name: Bryan Hooi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T15:21:14.704Z
paper_authors._id: 651d8032c50012d33e914f2f
paper_authors.avatarUrl: /avatars/0a44c9f51fc50ce86582e328c361ea00.svg
paper_authors.fullname: Bryan Hooi
paper_authors.type: user
paper_authors.user: bhooi
paper_authors._id: 67b420dfb2528c023491f45c
paper_authors.name: Nay Oo
paper_authors._id: 67b420dfb2528c023491f45d
paper_authors.name: Huajun Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T15:21:32.744Z
paper_authors._id: 64931296137833d7ec7689cd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64931296137833d7ec7689cd/TBihNdp1ZwIWjhfAWjRr6.jpeg
paper_authors.fullname: Huajun Chen
paper_authors.type: user
paper_authors.user: huajunsir
paper_authors._id: 67b420dfb2528c023491f45e
paper_authors.name: Ningyu Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-18T09:31:11.243Z
paper_authors._id: 620b3bbb0668e435407c8d0a
paper_authors.avatarUrl: /avatars/e0fccbb2577d76088e09f054c35cffbc.svg
paper_authors.fullname: Ningyu Zhang
paper_authors.type: user
paper_authors.user: Ningyu
paper_title: ReLearn: Unlearning via Learning for Large Language Models
paper_summary: Current unlearning methods for large language models usually rely on reverse
  optimization to reduce target token probabilities. However, this paradigm
  disrupts the subsequent tokens prediction, degrading model performance and
  linguistic coherence. Moreover, existing evaluation metrics overemphasize
  contextual forgetting while inadequately assessing response fluency and
  relevance. To address these challenges, we propose ReLearn, a data augmentation
  and fine-tuning pipeline for effective unlearning, along with a comprehensive
  evaluation framework. This framework introduces Knowledge Forgetting Rate (KFR)
  and Knowledge Retention Rate (KRR) to measure knowledge-level preservation, and
  Linguistic Score (LS) to evaluate generation quality. Our experiments show that
  ReLearn successfully achieves targeted forgetting while preserving high-quality
  output. Through mechanistic analysis, we further demonstrate how reverse
  optimization disrupts coherent text generation, while ReLearn preserves this
  essential capability. Code is available at https://github.com/zjunlp/unlearn.
paper_ai_keywords: reverse optimization target token probabilities tokens prediction model performance linguistic coherence evaluation metrics contextual forgetting response fluency relevance data augmentation fine-tuning Knowledge Forgetting Rate (KFR) Knowledge Retention Rate (KRR) Knowledge-level preservation Linguistic Score (LS) generation quality coherent text generation