WaterDrum: Watermarking for Data-centric Unlearning Metric
Large language model (LLM) unlearning is critical in real-world applications
where it is necessary to efficiently remove the influence of private,
copyrighted, or harmful data from some users. However, existing utility-centric
unlearning metrics (based on model utility) may fail to accurately evaluate the
extent of unlearning in realistic settings such as when (a) the forget and
retain set have semantically similar content, (b) retraining the model from
scratch on the retain set is impractical, and/or (c) the model owner can
improve the unlearning metric without directly performing unlearning on the
LLM. This paper presents the first data-centric unlearning metric for LLMs
called WaterDrum that exploits robust text watermarking for overcoming these
limitations. We also introduce new benchmark datasets for LLM unlearning that
contain varying levels of similar data points and can be used to rigorously
evaluate unlearning algorithms using WaterDrum. Our code is available at
https://github.com/lululu008/WaterDrum and our new benchmark datasets are
released at https://huggingface.co/datasets/Glow-AI/WaterDrum-Ax.
681d93e929119d6660775937
Xinyang Lu
claimed_verified
2025-05-09T13:54:48.001Z
64424c46afba527ed365bf7a
/avatars/29ed5c6cbc397932ba5434dfd52f705c.svg
Xinyang Lu
user
xylu
681d93e929119d6660775938
Xinyuan Niu
681d93e929119d6660775939
Gregory Kang Ruey Lau
681d93e929119d666077593a
Bui Thi Cam Nhung
681d93e929119d666077593b
Rachael Hwee Ling Sim
681d93e929119d666077593c
Fanyu Wen
681d93e929119d666077593d
Chuan-Sheng Foo
681d93e929119d666077593e
See-Kiong Ng
681d93e929119d666077593f
Bryan Kian Hsiang Low
WaterDrum: Watermarking for Data-centric Unlearning Metric
Large language model (LLM) unlearning is critical in real-world applications
where it is necessary to efficiently remove the influence of private,
copyrighted, or harmful data from some users. However, existing utility-centric
unlearning metrics (based on model utility) may fail to accurately evaluate the
extent of unlearning in realistic settings such as when (a) the forget and
retain set have semantically similar content, (b) retraining the model from
scratch on the retain set is impractical, and/or (c) the model owner can
improve the unlearning metric without directly performing unlearning on the
LLM. This paper presents the first data-centric unlearning metric for LLMs
called WaterDrum that exploits robust text watermarking for overcoming these
limitations. We also introduce new benchmark datasets for LLM unlearning that
contain varying levels of similar data points and can be used to rigorously
evaluate unlearning algorithms using WaterDrum. Our code is available at
https://github.com/lululu008/WaterDrum and our new benchmark datasets are
released at https://huggingface.co/datasets/Glow-AI/WaterDrum-Ax.
WaterDrum introduces a data-centric unlearning metric for LLMs using robust text watermarking to address limitations in existing utility-centric metrics.
Large language model (LLM) unlearning data-centric unlearning WaterDrum text watermarking benchmark datasets