title: To Forget or Not? Towards Practical Knowledge Unlearning for Large
    Language Models
summary: Large Language Models (LLMs) trained on extensive corpora inevitably retain
  sensitive data, such as personal privacy information and copyrighted material.
  Recent advancements in knowledge unlearning involve updating LLM parameters to
  erase specific knowledge. However, current unlearning paradigms are mired in
  vague forgetting boundaries, often erasing knowledge indiscriminately. In this
  work, we introduce KnowUnDo, a benchmark containing copyrighted content and
  user privacy domains to evaluate if the unlearning process inadvertently erases
  essential knowledge. Our findings indicate that existing unlearning methods
  often suffer from excessive unlearning. To address this, we propose a simple
  yet effective method, MemFlex, which utilizes gradient information to precisely
  target and unlearn sensitive parameters. Experimental results show that MemFlex
  is superior to existing methods in both precise knowledge unlearning and
  general knowledge retaining of LLMs. Code and dataset will be released at
  https://github.com/zjunlp/KnowUnDo.
paper_authors._id: 6685059fdee2fc81210a28f6
paper_authors.name: Bozhong Tian
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-03T14:59:34.439Z
paper_authors._id: 626e9144afb8cb8f7a5cb503
paper_authors.avatarUrl: /avatars/7e9df47dd4e3cb6df193878e561cc2bc.svg
paper_authors.fullname: Bozhong Tian
paper_authors.type: user
paper_authors.user: bozhong
paper_authors._id: 6685059fdee2fc81210a28f7
paper_authors.name: Xiaozhuan Liang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-03T14:59:42.079Z
paper_authors._id: 6216eefab64eee25bf8f2302
paper_authors.avatarUrl: /avatars/1b81081c568271fe20d45130e271d0c4.svg
paper_authors.fullname: Xiaozhuan Liang
paper_authors.type: user
paper_authors.user: liangxz
paper_authors._id: 6685059fdee2fc81210a28f8
paper_authors.name: Siyuan Cheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-03T14:59:48.395Z
paper_authors._id: 6216f334b64eee25bf8f2309
paper_authors.avatarUrl: /avatars/30657bc50c660e256743c495f776d1c5.svg
paper_authors.fullname: Siyuan Cheng
paper_authors.type: user
paper_authors.user: ChancesYuan
paper_authors._id: 6685059fdee2fc81210a28f9
paper_authors.name: Qingbin Liu
paper_authors._id: 6685059fdee2fc81210a28fa
paper_authors.name: Mengru Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-03T15:00:06.315Z
paper_authors._id: 64bf898d979949d2e2585c9a
paper_authors.avatarUrl: /avatars/da77c856ec997e2b812c06272a01c8b2.svg
paper_authors.fullname: mengruwang
paper_authors.type: user
paper_authors.user: mengru
paper_authors._id: 6685059fdee2fc81210a28fb
paper_authors.name: Dianbo Sui
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-03T15:00:13.444Z
paper_authors._id: 643e8b572263cdc630f80fee
paper_authors.avatarUrl: /avatars/93b11c28a58c5db71afd44e782493140.svg
paper_authors.fullname: Dianbo Sui
paper_authors.type: user
paper_authors.user: suidianbo
paper_authors._id: 6685059fdee2fc81210a28fc
paper_authors.name: Xi Chen
paper_authors._id: 6685059fdee2fc81210a28fd
paper_authors.name: Huajun Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-03T15:00:28.191Z
paper_authors._id: 64931296137833d7ec7689cd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64931296137833d7ec7689cd/TBihNdp1ZwIWjhfAWjRr6.jpeg
paper_authors.fullname: Huajun Chen
paper_authors.type: user
paper_authors.user: huajunsir
paper_authors._id: 6685059fdee2fc81210a28fe
paper_authors.name: Ningyu Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-03T13:44:50.301Z
paper_authors._id: 620b3bbb0668e435407c8d0a
paper_authors.avatarUrl: /avatars/e0fccbb2577d76088e09f054c35cffbc.svg
paper_authors.fullname: Ningyu Zhang
paper_authors.type: user
paper_authors.user: Ningyu
paper_title: To Forget or Not? Towards Practical Knowledge Unlearning for Large
    Language Models
paper_summary: Large Language Models (LLMs) trained on extensive corpora inevitably retain
  sensitive data, such as personal privacy information and copyrighted material.
  Recent advancements in knowledge unlearning involve updating LLM parameters to
  erase specific knowledge. However, current unlearning paradigms are mired in
  vague forgetting boundaries, often erasing knowledge indiscriminately. In this
  work, we introduce KnowUnDo, a benchmark containing copyrighted content and
  user privacy domains to evaluate if the unlearning process inadvertently erases
  essential knowledge. Our findings indicate that existing unlearning methods
  often suffer from excessive unlearning. To address this, we propose a simple
  yet effective method, MemFlex, which utilizes gradient information to precisely
  target and unlearn sensitive parameters. Experimental results show that MemFlex
  is superior to existing methods in both precise knowledge unlearning and
  general knowledge retaining of LLMs. Code and dataset will be released at
  https://github.com/zjunlp/KnowUnDo.
paper_ai_summary: MemFlex, a method leveraging gradient information, improves precise knowledge unlearning and retains essential knowledge in Large Language Models without excessive data erasure.
paper_ai_keywords: Large Language Models LLMs knowledge unlearning unlearning paradigms forgetting boundaries copyrighted content user privacy MemFlex gradient information