To Forget or Not? Towards Practical Knowledge Unlearning for Large
  Language Models
Large Language Models (LLMs) trained on extensive corpora inevitably retain
sensitive data, such as personal privacy information and copyrighted material.
Recent advancements in knowledge unlearning involve updating LLM parameters to
erase specific knowledge. However, current unlearning paradigms are mired in
vague forgetting boundaries, often erasing knowledge indiscriminately. In this
work, we introduce KnowUnDo, a benchmark containing copyrighted content and
user privacy domains to evaluate if the unlearning process inadvertently erases
essential knowledge. Our findings indicate that existing unlearning methods
often suffer from excessive unlearning. To address this, we propose a simple
yet effective method, MemFlex, which utilizes gradient information to precisely
target and unlearn sensitive parameters. Experimental results show that MemFlex
is superior to existing methods in both precise knowledge unlearning and
general knowledge retaining of LLMs. Code and dataset will be released at
https://github.com/zjunlp/KnowUnDo.
6685059fdee2fc81210a28f6
Bozhong Tian
admin_assigned
2024-07-03T14:59:34.439Z
626e9144afb8cb8f7a5cb503
/avatars/7e9df47dd4e3cb6df193878e561cc2bc.svg
Bozhong Tian
user
bozhong
6685059fdee2fc81210a28f7
Xiaozhuan Liang
admin_assigned
2024-07-03T14:59:42.079Z
6216eefab64eee25bf8f2302
/avatars/1b81081c568271fe20d45130e271d0c4.svg
Xiaozhuan Liang
user
liangxz
6685059fdee2fc81210a28f8
Siyuan Cheng
admin_assigned
2024-07-03T14:59:48.395Z
6216f334b64eee25bf8f2309
/avatars/30657bc50c660e256743c495f776d1c5.svg
Siyuan Cheng
user
ChancesYuan
6685059fdee2fc81210a28f9
Qingbin Liu
6685059fdee2fc81210a28fa
Mengru Wang
admin_assigned
2024-07-03T15:00:06.315Z
64bf898d979949d2e2585c9a
/avatars/da77c856ec997e2b812c06272a01c8b2.svg
mengruwang
user
mengru
6685059fdee2fc81210a28fb
Dianbo Sui
admin_assigned
2024-07-03T15:00:13.444Z
643e8b572263cdc630f80fee
/avatars/93b11c28a58c5db71afd44e782493140.svg
Dianbo Sui
user
suidianbo
6685059fdee2fc81210a28fc
Xi Chen
6685059fdee2fc81210a28fd
Huajun Chen
admin_assigned
2024-07-03T15:00:28.191Z
64931296137833d7ec7689cd
https://cdn-avatars.huggingface.co/v1/production/uploads/64931296137833d7ec7689cd/TBihNdp1ZwIWjhfAWjRr6.jpeg
Huajun Chen
user
huajunsir
6685059fdee2fc81210a28fe
Ningyu Zhang
claimed_verified
2024-07-03T13:44:50.301Z
620b3bbb0668e435407c8d0a
/avatars/e0fccbb2577d76088e09f054c35cffbc.svg
Ningyu Zhang
user
Ningyu
To Forget or Not? Towards Practical Knowledge Unlearning for Large
  Language Models
Large Language Models (LLMs) trained on extensive corpora inevitably retain
sensitive data, such as personal privacy information and copyrighted material.
Recent advancements in knowledge unlearning involve updating LLM parameters to
erase specific knowledge. However, current unlearning paradigms are mired in
vague forgetting boundaries, often erasing knowledge indiscriminately. In this
work, we introduce KnowUnDo, a benchmark containing copyrighted content and
user privacy domains to evaluate if the unlearning process inadvertently erases
essential knowledge. Our findings indicate that existing unlearning methods
often suffer from excessive unlearning. To address this, we propose a simple
yet effective method, MemFlex, which utilizes gradient information to precisely
target and unlearn sensitive parameters. Experimental results show that MemFlex
is superior to existing methods in both precise knowledge unlearning and
general knowledge retaining of LLMs. Code and dataset will be released at
https://github.com/zjunlp/KnowUnDo.
MemFlex, a method leveraging gradient information, improves precise knowledge unlearning and retains essential knowledge in Large Language Models without excessive data erasure.
Large Language Models LLMs knowledge unlearning unlearning paradigms forgetting boundaries copyrighted content user privacy MemFlex gradient information