title: Is Bigger Edit Batch Size Always Better? -- An Empirical Study on Model
    Editing with Llama-3
summary: This study presents a targeted model editing analysis focused on the latest
  large language model, Llama-3. We explore the efficacy of popular model editing
  techniques - ROME, MEMIT, and EMMET, which are designed for precise layer
  interventions. We identify the most effective layers for targeted edits through
  an evaluation that encompasses up to 4096 edits across three distinct
  strategies: sequential editing, batch editing, and a hybrid approach we call as
  sequential-batch editing. Our findings indicate that increasing edit
  batch-sizes may degrade model performance more significantly than using smaller
  edit batches sequentially for equal number of edits. With this, we argue that
  sequential model editing is an important component for scaling model editing
  methods and future research should focus on methods that combine both batched
  and sequential editing. This observation suggests a potential limitation in
  current model editing methods which push towards bigger edit batch sizes, and
  we hope it paves way for future investigations into optimizing batch sizes and
  model editing performance.
paper_authors._id: 663303d143f364c13e3d5acc
paper_authors.name: Junsang Yoon
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-02T07:22:12.956Z
paper_authors._id: 65ab438ee0ee7990a68500ab
paper_authors.avatarUrl: /avatars/507aa8d0eb595feb7ad9a9e8c39721da.svg
paper_authors.fullname: Junsang Yoon
paper_authors.type: user
paper_authors.user: junimda
paper_authors._id: 663303d143f364c13e3d5acd
paper_authors.name: Akshat Gupta
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-02T07:09:25.109Z
paper_authors._id: 64e8f4a24f3f7b0b84834315
paper_authors.avatarUrl: /avatars/242bb68c7ccffe5061c2d1c229ea3b0b.svg
paper_authors.fullname: Akshat Gupta
paper_authors.type: user
paper_authors.user: akshat57
paper_authors._id: 663303d143f364c13e3d5ace
paper_authors.name: Gopala Anumanchipalli
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-02T07:22:20.771Z
paper_authors._id: 60523e4aa7226b25aaeea2b8
paper_authors.avatarUrl: /avatars/316ca348da91ebced86991f36150c959.svg
paper_authors.fullname: Gopala Anumanchipalli
paper_authors.type: user
paper_authors.user: gopalakr
paper_title: Is Bigger Edit Batch Size Always Better? -- An Empirical Study on Model
    Editing with Llama-3
paper_summary: This study presents a targeted model editing analysis focused on the latest
  large language model, Llama-3. We explore the efficacy of popular model editing
  techniques - ROME, MEMIT, and EMMET, which are designed for precise layer
  interventions. We identify the most effective layers for targeted edits through
  an evaluation that encompasses up to 4096 edits across three distinct
  strategies: sequential editing, batch editing, and a hybrid approach we call as
  sequential-batch editing. Our findings indicate that increasing edit
  batch-sizes may degrade model performance more significantly than using smaller
  edit batches sequentially for equal number of edits. With this, we argue that
  sequential model editing is an important component for scaling model editing
  methods and future research should focus on methods that combine both batched
  and sequential editing. This observation suggests a potential limitation in
  current model editing methods which push towards bigger edit batch sizes, and
  we hope it paves way for future investigations into optimizing batch sizes and
  model editing performance.
paper_ai_summary: The analysis of ROME, MEMIT, and EMMET editing techniques on Llama-3 highlights that sequential editing preserves model performance better than large batch editing across various strategies.
paper_ai_keywords: large language model ROME MEMIT EMMET targeted model editing layer interventions sequential editing batch editing sequential-batch editing model performance