title: Intrinsic Evaluation of Unlearning Using Parametric Knowledge Traces
summary: The task of "unlearning" certain concepts in large language models (LLMs) has
  attracted immense attention recently, due to its importance for mitigating
  undesirable model behaviours, such as the generation of harmful, private, or
  incorrect information. Current protocols to evaluate unlearning methods largely
  rely on behavioral tests, without monitoring the presence of unlearned
  knowledge within the model's parameters. This residual knowledge can be
  adversarially exploited to recover the erased information post-unlearning. We
  argue that unlearning should also be evaluated internally, by considering
  changes in the parametric knowledge traces of the unlearned concepts. To this
  end, we propose a general methodology for eliciting directions in the parameter
  space (termed "concept vectors") that encode concrete concepts, and construct
  ConceptVectors, a benchmark dataset containing hundreds of common concepts and
  their parametric knowledge traces within two open-source LLMs. Evaluation on
  ConceptVectors shows that existing unlearning methods minimally impact concept
  vectors, while directly ablating these vectors demonstrably removes the
  associated knowledge from the LLMs and significantly reduces their
  susceptibility to adversarial manipulation. Our results highlight limitations
  in behavioral-based unlearning evaluations and call for future work to include
  parametric-based evaluations. To support this, we release our code and
  benchmark at https://github.com/yihuaihong/ConceptVectors.
paper_authors._id: 66746362d812e2028ffaaef0
paper_authors.name: Yihuai Hong
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-06-20T17:14:10.838Z
paper_authors._id: 64fe727b2a6d4e23be4db45a
paper_authors.avatarUrl: /avatars/57e8f9801661c53187890c904adc2d81.svg
paper_authors.fullname: YihuaiHong
paper_authors.type: user
paper_authors.user: YihuaiHong
paper_authors._id: 66746362d812e2028ffaaef1
paper_authors.name: Lei Yu
paper_authors._id: 66746362d812e2028ffaaef2
paper_authors.name: Shauli Ravfogel
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-25T13:34:20.761Z
paper_authors._id: 635686ec5aeb69011c7d1abd
paper_authors.avatarUrl: /avatars/c59034ad2c9c2daf4b4a8d3c56449f5e.svg
paper_authors.fullname: Shauli Ravfogel
paper_authors.type: user
paper_authors.user: ravfogs
paper_authors._id: 66746362d812e2028ffaaef3
paper_authors.name: Haiqin Yang
paper_authors._id: 66746362d812e2028ffaaef4
paper_authors.name: Mor Geva
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-20T21:03:24.886Z
paper_authors._id: 610b729f9da682cd54ad9adf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1628140189042-noauth.jpeg
paper_authors.fullname: Mor Geva
paper_authors.type: user
paper_authors.user: mega
paper_title: Intrinsic Evaluation of Unlearning Using Parametric Knowledge Traces
paper_summary: The task of "unlearning" certain concepts in large language models (LLMs) has
  attracted immense attention recently, due to its importance for mitigating
  undesirable model behaviours, such as the generation of harmful, private, or
  incorrect information. Current protocols to evaluate unlearning methods largely
  rely on behavioral tests, without monitoring the presence of unlearned
  knowledge within the model's parameters. This residual knowledge can be
  adversarially exploited to recover the erased information post-unlearning. We
  argue that unlearning should also be evaluated internally, by considering
  changes in the parametric knowledge traces of the unlearned concepts. To this
  end, we propose a general methodology for eliciting directions in the parameter
  space (termed "concept vectors") that encode concrete concepts, and construct
  ConceptVectors, a benchmark dataset containing hundreds of common concepts and
  their parametric knowledge traces within two open-source LLMs. Evaluation on
  ConceptVectors shows that existing unlearning methods minimally impact concept
  vectors, while directly ablating these vectors demonstrably removes the
  associated knowledge from the LLMs and significantly reduces their
  susceptibility to adversarial manipulation. Our results highlight limitations
  in behavioral-based unlearning evaluations and call for future work to include
  parametric-based evaluations. To support this, we release our code and
  benchmark at https://github.com/yihuaihong/ConceptVectors.
paper_ai_summary: Evaluating concept unlearning in large language models by analyzing parametric knowledge traces reveals limitations in behavioral evaluations and suggests removing specific concept vectors to improve unlearning effectiveness.
paper_ai_keywords: large language models unlearning concept vectors parametric knowledge traces behavioral tests adversarial manipulation