title: Should We Really Edit Language Models? On the Evaluation of Edited
    Language Models
summary: Model editing has become an increasingly popular alternative for efficiently
  updating knowledge within language models. Current methods mainly focus on
  reliability, generalization, and locality, with many methods excelling across
  these criteria. Some recent works disclose the pitfalls of these editing
  methods such as knowledge distortion or conflict. However, the general
  abilities of post-edited language models remain unexplored. In this paper, we
  perform a comprehensive evaluation on various editing methods and different
  language models, and have following findings. (1) Existing editing methods lead
  to inevitable performance deterioration on general benchmarks, indicating that
  existing editing methods maintain the general abilities of the model within
  only a few dozen edits. When the number of edits is slightly large, the
  intrinsic knowledge structure of the model is disrupted or even completely
  damaged. (2) Instruction-tuned models are more robust to editing, showing less
  performance drop on general knowledge after editing. (3) Language model with
  large scale is more resistant to editing compared to small model. (4) The
  safety of the edited model, is significantly weakened, even for those
  safety-aligned models. Our findings indicate that current editing methods are
  only suitable for small-scale knowledge updates within language models, which
  motivates further research on more practical and reliable editing methods. The
  details of code and reproduction can be found in
  https://github.com/lqinfdim/EditingEvaluation.
paper_authors._id: 671af7d30dd7c441d171e307
paper_authors.name: Qi Li
paper_authors._id: 671af7d30dd7c441d171e308
paper_authors.name: Xiang Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-25T09:30:17.003Z
paper_authors._id: 63024676056ec3a2a8714b24
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1661093436322-noauth.jpeg
paper_authors.fullname: Xiang Liu
paper_authors.type: user
paper_authors.user: Dominic789654
paper_authors._id: 671af7d30dd7c441d171e309
paper_authors.name: Zhenheng Tang
paper_authors._id: 671af7d30dd7c441d171e30a
paper_authors.name: Peijie Dong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-02T07:56:34.008Z
paper_authors._id: 6395f845aec00abff778ad31
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6395f845aec00abff778ad31/bZkAlchSvqER1HgBKmcHI.jpeg
paper_authors.fullname: PeijieDong
paper_authors.type: user
paper_authors.user: pprp
paper_authors._id: 671af7d30dd7c441d171e30b
paper_authors.name: Zeyu Li
paper_authors._id: 671af7d30dd7c441d171e30c
paper_authors.name: Xinglin Pan
paper_authors._id: 671af7d30dd7c441d171e30d
paper_authors.name: Xiaowen Chu
paper_title: Should We Really Edit Language Models? On the Evaluation of Edited
    Language Models
paper_summary: Model editing has become an increasingly popular alternative for efficiently
  updating knowledge within language models. Current methods mainly focus on
  reliability, generalization, and locality, with many methods excelling across
  these criteria. Some recent works disclose the pitfalls of these editing
  methods such as knowledge distortion or conflict. However, the general
  abilities of post-edited language models remain unexplored. In this paper, we
  perform a comprehensive evaluation on various editing methods and different
  language models, and have following findings. (1) Existing editing methods lead
  to inevitable performance deterioration on general benchmarks, indicating that
  existing editing methods maintain the general abilities of the model within
  only a few dozen edits. When the number of edits is slightly large, the
  intrinsic knowledge structure of the model is disrupted or even completely
  damaged. (2) Instruction-tuned models are more robust to editing, showing less
  performance drop on general knowledge after editing. (3) Language model with
  large scale is more resistant to editing compared to small model. (4) The
  safety of the edited model, is significantly weakened, even for those
  safety-aligned models. Our findings indicate that current editing methods are
  only suitable for small-scale knowledge updates within language models, which
  motivates further research on more practical and reliable editing methods. The
  details of code and reproduction can be found in
  https://github.com/lqinfdim/EditingEvaluation.
paper_ai_summary: A comprehensive evaluation of language model editing methods reveals performance deterioration with large-scale edits, enhanced robustness in instruction-tuned models, and reduced safety, advocating for further research on practical and reliable editing techniques.
paper_ai_keywords: model editing reliability generalization locality knowledge distortion conflict post-edited language models instruction-tuned models safety safety-aligned models