title: WaterDrum: Watermarking for Data-centric Unlearning Metric
summary: Large language model (LLM) unlearning is critical in real-world applications
  where it is necessary to efficiently remove the influence of private,
  copyrighted, or harmful data from some users. However, existing utility-centric
  unlearning metrics (based on model utility) may fail to accurately evaluate the
  extent of unlearning in realistic settings such as when (a) the forget and
  retain set have semantically similar content, (b) retraining the model from
  scratch on the retain set is impractical, and/or (c) the model owner can
  improve the unlearning metric without directly performing unlearning on the
  LLM. This paper presents the first data-centric unlearning metric for LLMs
  called WaterDrum that exploits robust text watermarking for overcoming these
  limitations. We also introduce new benchmark datasets for LLM unlearning that
  contain varying levels of similar data points and can be used to rigorously
  evaluate unlearning algorithms using WaterDrum. Our code is available at
  https://github.com/lululu008/WaterDrum and our new benchmark datasets are
  released at https://huggingface.co/datasets/Glow-AI/WaterDrum-Ax.
paper_authors._id: 681d93e929119d6660775937
paper_authors.name: Xinyang Lu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-09T13:54:48.001Z
paper_authors._id: 64424c46afba527ed365bf7a
paper_authors.avatarUrl: /avatars/29ed5c6cbc397932ba5434dfd52f705c.svg
paper_authors.fullname: Xinyang Lu
paper_authors.type: user
paper_authors.user: xylu
paper_authors._id: 681d93e929119d6660775938
paper_authors.name: Xinyuan Niu
paper_authors._id: 681d93e929119d6660775939
paper_authors.name: Gregory Kang Ruey Lau
paper_authors._id: 681d93e929119d666077593a
paper_authors.name: Bui Thi Cam Nhung
paper_authors._id: 681d93e929119d666077593b
paper_authors.name: Rachael Hwee Ling Sim
paper_authors._id: 681d93e929119d666077593c
paper_authors.name: Fanyu Wen
paper_authors._id: 681d93e929119d666077593d
paper_authors.name: Chuan-Sheng Foo
paper_authors._id: 681d93e929119d666077593e
paper_authors.name: See-Kiong Ng
paper_authors._id: 681d93e929119d666077593f
paper_authors.name: Bryan Kian Hsiang Low
paper_title: WaterDrum: Watermarking for Data-centric Unlearning Metric
paper_summary: Large language model (LLM) unlearning is critical in real-world applications
  where it is necessary to efficiently remove the influence of private,
  copyrighted, or harmful data from some users. However, existing utility-centric
  unlearning metrics (based on model utility) may fail to accurately evaluate the
  extent of unlearning in realistic settings such as when (a) the forget and
  retain set have semantically similar content, (b) retraining the model from
  scratch on the retain set is impractical, and/or (c) the model owner can
  improve the unlearning metric without directly performing unlearning on the
  LLM. This paper presents the first data-centric unlearning metric for LLMs
  called WaterDrum that exploits robust text watermarking for overcoming these
  limitations. We also introduce new benchmark datasets for LLM unlearning that
  contain varying levels of similar data points and can be used to rigorously
  evaluate unlearning algorithms using WaterDrum. Our code is available at
  https://github.com/lululu008/WaterDrum and our new benchmark datasets are
  released at https://huggingface.co/datasets/Glow-AI/WaterDrum-Ax.
paper_ai_summary: WaterDrum introduces a data-centric unlearning metric for LLMs using robust text watermarking to address limitations in existing utility-centric metrics.
paper_ai_keywords: Large language model (LLM) unlearning data-centric unlearning WaterDrum text watermarking benchmark datasets