title: Lifelong Sequential Knowledge Editing without Model Degradation
summary: Prior work in parameter-modifying knowledge editing has shown that
  large-scale sequential editing leads to significant model degradation. In this
  paper, we study the reasons behind this and scale sequential knowledge editing
  to 10,000 sequential edits, while maintaining the downstream performance of the
  original model. We first show that locate-then-edit knowledge editing methods
  lead to overfitting on the edited facts. We also show that continuous knowledge
  editing using these methods leads to disproportionate growth in the norm of the
  edited matrix. We then provide a crucial insight into the inner workings of
  locate-then-edit methods. We show that norm-growth is a hidden trick employed
  by these methods that gives larger importance to the output activations
  produced from the edited layers. With this "importance hacking", the edited
  layers provide a much larger contributions to the model's output. To mitigate
  these issues, we present ENCORE - Early stopping and Norm-Constrained Robust
  knowledge Editing. ENCORE controls for overfitting and the disproportionate
  norm-growth to enable long-term sequential editing, where we are able to
  perform up to 10,000 sequential edits without loss of downstream performance.
  ENCORE is also 61% faster than MEMIT and 64% faster than AlphaEdit on
  Llama3-8B.
paper_authors._id: 67a1aa5dc7fa0ccf0a32ceb1
paper_authors.name: Akshat Gupta
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-02-04T05:53:11.213Z
paper_authors._id: 64e8f4a24f3f7b0b84834315
paper_authors.avatarUrl: /avatars/242bb68c7ccffe5061c2d1c229ea3b0b.svg
paper_authors.fullname: Akshat Gupta
paper_authors.type: user
paper_authors.user: akshat57
paper_authors._id: 67a1aa5dc7fa0ccf0a32ceb2
paper_authors.name: Phudish Prateepamornkul
paper_authors._id: 67a1aa5dc7fa0ccf0a32ceb3
paper_authors.name: Maochuan Lu
paper_authors._id: 67a1aa5dc7fa0ccf0a32ceb4
paper_authors.name: Ahmed Alaa
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-07T09:58:25.997Z
paper_authors._id: 6353668fc0b9d81cd2668a2c
paper_authors.avatarUrl: /avatars/4f6e54e702945d7d58b933ba4115a3e0.svg
paper_authors.fullname: Ahmed Alaa
paper_authors.type: user
paper_authors.user: amalaa
paper_authors._id: 67a1aa5dc7fa0ccf0a32ceb5
paper_authors.name: Thomas Hartvigsen
paper_authors._id: 67a1aa5dc7fa0ccf0a32ceb6
paper_authors.name: Gopala Anumanchipalli
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-04T21:11:09.416Z
paper_authors._id: 60523e4aa7226b25aaeea2b8
paper_authors.avatarUrl: /avatars/316ca348da91ebced86991f36150c959.svg
paper_authors.fullname: Gopala Anumanchipalli
paper_authors.type: user
paper_authors.user: gopalakr
paper_title: Lifelong Sequential Knowledge Editing without Model Degradation
paper_summary: Prior work in parameter-modifying knowledge editing has shown that
  large-scale sequential editing leads to significant model degradation. In this
  paper, we study the reasons behind this and scale sequential knowledge editing
  to 10,000 sequential edits, while maintaining the downstream performance of the
  original model. We first show that locate-then-edit knowledge editing methods
  lead to overfitting on the edited facts. We also show that continuous knowledge
  editing using these methods leads to disproportionate growth in the norm of the
  edited matrix. We then provide a crucial insight into the inner workings of
  locate-then-edit methods. We show that norm-growth is a hidden trick employed
  by these methods that gives larger importance to the output activations
  produced from the edited layers. With this "importance hacking", the edited
  layers provide a much larger contributions to the model's output. To mitigate
  these issues, we present ENCORE - Early stopping and Norm-Constrained Robust
  knowledge Editing. ENCORE controls for overfitting and the disproportionate
  norm-growth to enable long-term sequential editing, where we are able to
  perform up to 10,000 sequential edits without loss of downstream performance.
  ENCORE is also 61% faster than MEMIT and 64% faster than AlphaEdit on
  Llama3-8B.
paper_ai_summary: ENCORE, an early stopping and norm-constrained robust knowledge editing method, allows for up to 10,000 sequential edits without degrading downstream performance and improves speed compared to existing techniques.
paper_ai_keywords: locate-then-edit knowledge editing overfitting norm growth importance hacking ENCORE MEMIT AlphaEdit Llama3-8B