LLM Unlearning Should Be Form-Independent
Large Language Model (LLM) unlearning aims to erase or suppress undesirable
knowledge within the model, offering promise for controlling harmful or private
information to prevent misuse. However, recent studies highlight its limited
efficacy in real-world scenarios, hindering practical adoption. In this study,
we identify a pervasive issue underlying many downstream failures: the
effectiveness of existing unlearning methods heavily depends on the form of
training samples and frequently fails to generalize to alternate expressions of
the same knowledge. We formally characterize this problem as Form-Dependent
Bias and systematically investigate its specific manifestation patterns across
various downstream tasks. To quantify its prevalence and support future
research, we introduce ORT, a novel benchmark designed to evaluate the
robustness of unlearning methods against variations in knowledge expression.
Results reveal that Form-Dependent Bias is both widespread and severe among
current techniques.
  We argue that LLM unlearning should be form-independent to address the
endless forms of downstream tasks encountered in real-world security-critical
scenarios. Towards this goal, we introduce Rank-one Concept Redirection (ROCR),
a novel training-free method, as a promising solution path. ROCR performs
unlearning by targeting the invariants in downstream tasks, specifically the
activated dangerous concepts. It is capable of modifying model parameters
within seconds to redirect the model's perception of a specific unlearning
target concept to another harmless concept. Extensive experiments demonstrate
that ROCR significantly improves unlearning effectiveness compared to
traditional methods while generating highly natural outputs.
6848dca942e4f9106973f25c
Xiaotian Ye
claimed_verified
2025-06-11T08:35:10.086Z
6659b410a69183808d04b22f
/avatars/a16d1fed9ef87163fe458b10c477140b.svg
Xiaotian Ye
user
Acruxos
6848dca942e4f9106973f25d
Mengqi Zhang
6848dca942e4f9106973f25e
Shu Wu
LLM Unlearning Should Be Form-Independent
Large Language Model (LLM) unlearning aims to erase or suppress undesirable
knowledge within the model, offering promise for controlling harmful or private
information to prevent misuse. However, recent studies highlight its limited
efficacy in real-world scenarios, hindering practical adoption. In this study,
we identify a pervasive issue underlying many downstream failures: the
effectiveness of existing unlearning methods heavily depends on the form of
training samples and frequently fails to generalize to alternate expressions of
the same knowledge. We formally characterize this problem as Form-Dependent
Bias and systematically investigate its specific manifestation patterns across
various downstream tasks. To quantify its prevalence and support future
research, we introduce ORT, a novel benchmark designed to evaluate the
robustness of unlearning methods against variations in knowledge expression.
Results reveal that Form-Dependent Bias is both widespread and severe among
current techniques.
  We argue that LLM unlearning should be form-independent to address the
endless forms of downstream tasks encountered in real-world security-critical
scenarios. Towards this goal, we introduce Rank-one Concept Redirection (ROCR),
a novel training-free method, as a promising solution path. ROCR performs
unlearning by targeting the invariants in downstream tasks, specifically the
activated dangerous concepts. It is capable of modifying model parameters
within seconds to redirect the model's perception of a specific unlearning
target concept to another harmless concept. Extensive experiments demonstrate
that ROCR significantly improves unlearning effectiveness compared to
traditional methods while generating highly natural outputs.
Form-Dependent Bias limits the effectiveness of LLM unlearning across different knowledge expressions, and Rank-one Concept Redirection (ROCR) is proposed as a form-independent solution that enhances unlearning efficacy.
Large Language Model (LLM) unlearning Form-Dependent Bias ORT Rank-one Concept Redirection (ROCR) downstream tasks unlearning methods concept redirection model parameters activated dangerous concepts