Should We Really Edit Language Models? On the Evaluation of Edited
  Language Models
Model editing has become an increasingly popular alternative for efficiently
updating knowledge within language models. Current methods mainly focus on
reliability, generalization, and locality, with many methods excelling across
these criteria. Some recent works disclose the pitfalls of these editing
methods such as knowledge distortion or conflict. However, the general
abilities of post-edited language models remain unexplored. In this paper, we
perform a comprehensive evaluation on various editing methods and different
language models, and have following findings. (1) Existing editing methods lead
to inevitable performance deterioration on general benchmarks, indicating that
existing editing methods maintain the general abilities of the model within
only a few dozen edits. When the number of edits is slightly large, the
intrinsic knowledge structure of the model is disrupted or even completely
damaged. (2) Instruction-tuned models are more robust to editing, showing less
performance drop on general knowledge after editing. (3) Language model with
large scale is more resistant to editing compared to small model. (4) The
safety of the edited model, is significantly weakened, even for those
safety-aligned models. Our findings indicate that current editing methods are
only suitable for small-scale knowledge updates within language models, which
motivates further research on more practical and reliable editing methods. The
details of code and reproduction can be found in
https://github.com/lqinfdim/EditingEvaluation.
671af7d30dd7c441d171e307
Qi Li
671af7d30dd7c441d171e308
Xiang Liu
claimed_verified
2024-10-25T09:30:17.003Z
63024676056ec3a2a8714b24
https://cdn-avatars.huggingface.co/v1/production/uploads/1661093436322-noauth.jpeg
Xiang Liu
user
Dominic789654
671af7d30dd7c441d171e309
Zhenheng Tang
671af7d30dd7c441d171e30a
Peijie Dong
claimed_verified
2025-06-02T07:56:34.008Z
6395f845aec00abff778ad31
https://cdn-avatars.huggingface.co/v1/production/uploads/6395f845aec00abff778ad31/bZkAlchSvqER1HgBKmcHI.jpeg
PeijieDong
user
pprp
671af7d30dd7c441d171e30b
Zeyu Li
671af7d30dd7c441d171e30c
Xinglin Pan
671af7d30dd7c441d171e30d
Xiaowen Chu
Should We Really Edit Language Models? On the Evaluation of Edited
  Language Models
Model editing has become an increasingly popular alternative for efficiently
updating knowledge within language models. Current methods mainly focus on
reliability, generalization, and locality, with many methods excelling across
these criteria. Some recent works disclose the pitfalls of these editing
methods such as knowledge distortion or conflict. However, the general
abilities of post-edited language models remain unexplored. In this paper, we
perform a comprehensive evaluation on various editing methods and different
language models, and have following findings. (1) Existing editing methods lead
to inevitable performance deterioration on general benchmarks, indicating that
existing editing methods maintain the general abilities of the model within
only a few dozen edits. When the number of edits is slightly large, the
intrinsic knowledge structure of the model is disrupted or even completely
damaged. (2) Instruction-tuned models are more robust to editing, showing less
performance drop on general knowledge after editing. (3) Language model with
large scale is more resistant to editing compared to small model. (4) The
safety of the edited model, is significantly weakened, even for those
safety-aligned models. Our findings indicate that current editing methods are
only suitable for small-scale knowledge updates within language models, which
motivates further research on more practical and reliable editing methods. The
details of code and reproduction can be found in
https://github.com/lqinfdim/EditingEvaluation.
A comprehensive evaluation of language model editing methods reveals performance deterioration with large-scale edits, enhanced robustness in instruction-tuned models, and reduced safety, advocating for further research on practical and reliable editing techniques.
model editing reliability generalization locality knowledge distortion conflict post-edited language models instruction-tuned models safety safety-aligned models