title: In-Context Editing: Learning Knowledge from Self-Induced Distributions
summary: The existing fine-tuning paradigm for language models is brittle in knowledge
  editing scenarios, where the model must incorporate new information without
  extensive retraining. This brittleness often results in overfitting, reduced
  performance, and unnatural language generation. To address this, we propose
  Consistent In-Context Editing (ICE), a novel approach that leverages the
  model's in-context learning capability to tune toward a contextual distribution
  rather than a one-hot target. ICE introduces a straightforward optimization
  framework that includes both a target and a procedure, enhancing the robustness
  and effectiveness of gradient-based tuning methods. We provide analytical
  insights into ICE across four critical aspects of knowledge editing: accuracy,
  locality, generalization, and linguistic quality, showing its advantages.
  Experimental results across four datasets confirm the effectiveness of ICE and
  demonstrate its potential for continual editing, ensuring that updated
  information is incorporated while preserving the integrity of the model.
paper_authors._id: 66719193202196a178f8d501
paper_authors.name: Siyuan Qi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-18T14:13:55.628Z
paper_authors._id: 6483ec5fe2f6f33be175c5c3
paper_authors.avatarUrl: /avatars/2dc35b31e8906c38f0db6d742936a862.svg
paper_authors.fullname: Siyuan Qi
paper_authors.type: user
paper_authors.user: syqi
paper_authors._id: 66719193202196a178f8d502
paper_authors.name: Bangcheng Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-02T07:39:42.252Z
paper_authors._id: 63ece843c8827dd0f0f7b8c4
paper_authors.avatarUrl: /avatars/e6c5840510ab88b007183cf79ed1bdf9.svg
paper_authors.fullname: Bangcheng Yang
paper_authors.type: user
paper_authors.user: dumbmouse
paper_authors._id: 66719193202196a178f8d503
paper_authors.name: Kailin Jiang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-19T20:21:01.784Z
paper_authors._id: 65745569839aa08899ea5d27
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/4X8waDwiphbfKZySrYlFy.jpeg
paper_authors.fullname: kailinjiang
paper_authors.type: user
paper_authors.user: kailinjiang
paper_authors._id: 66719193202196a178f8d504
paper_authors.name: Xiaobo Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-19T20:20:59.856Z
paper_authors._id: 65e933fcacb29a596cd6eb6e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65e933fcacb29a596cd6eb6e/dlH14rO2I4EUMVyjEjdRK.jpeg
paper_authors.fullname: Xiaobo Wang
paper_authors.type: user
paper_authors.user: Yofuria
paper_authors._id: 66719193202196a178f8d505
paper_authors.name: Jiaqi Li
paper_authors._id: 66719193202196a178f8d506
paper_authors.name: Yifan Zhong
paper_authors._id: 66719193202196a178f8d507
paper_authors.name: Yaodong Yang
paper_authors._id: 66719193202196a178f8d508
paper_authors.name: Zilong Zheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-19T07:39:17.582Z
paper_authors._id: 63a95a6a7930fa8c7dd63d4e
paper_authors.avatarUrl: /avatars/d9d0420f7ddfe2f3a7e029fb05f1c89f.svg
paper_authors.fullname: Zilong Zheng
paper_authors.type: user
paper_authors.user: zlzheng
paper_title: In-Context Editing: Learning Knowledge from Self-Induced Distributions
paper_summary: The existing fine-tuning paradigm for language models is brittle in knowledge
  editing scenarios, where the model must incorporate new information without
  extensive retraining. This brittleness often results in overfitting, reduced
  performance, and unnatural language generation. To address this, we propose
  Consistent In-Context Editing (ICE), a novel approach that leverages the
  model's in-context learning capability to tune toward a contextual distribution
  rather than a one-hot target. ICE introduces a straightforward optimization
  framework that includes both a target and a procedure, enhancing the robustness
  and effectiveness of gradient-based tuning methods. We provide analytical
  insights into ICE across four critical aspects of knowledge editing: accuracy,
  locality, generalization, and linguistic quality, showing its advantages.
  Experimental results across four datasets confirm the effectiveness of ICE and
  demonstrate its potential for continual editing, ensuring that updated
  information is incorporated while preserving the integrity of the model.
paper_ai_keywords: Consistent In-Context Editing (ICE) in-context learning capability contextual distribution gradient-based tuning methods