Is Bigger Edit Batch Size Always Better? -- An Empirical Study on Model
  Editing with Llama-3
This study presents a targeted model editing analysis focused on the latest
large language model, Llama-3. We explore the efficacy of popular model editing
techniques - ROME, MEMIT, and EMMET, which are designed for precise layer
interventions. We identify the most effective layers for targeted edits through
an evaluation that encompasses up to 4096 edits across three distinct
strategies: sequential editing, batch editing, and a hybrid approach we call as
sequential-batch editing. Our findings indicate that increasing edit
batch-sizes may degrade model performance more significantly than using smaller
edit batches sequentially for equal number of edits. With this, we argue that
sequential model editing is an important component for scaling model editing
methods and future research should focus on methods that combine both batched
and sequential editing. This observation suggests a potential limitation in
current model editing methods which push towards bigger edit batch sizes, and
we hope it paves way for future investigations into optimizing batch sizes and
model editing performance.
663303d143f364c13e3d5acc
Junsang Yoon
admin_assigned
2024-05-02T07:22:12.956Z
65ab438ee0ee7990a68500ab
/avatars/507aa8d0eb595feb7ad9a9e8c39721da.svg
Junsang Yoon
user
junimda
663303d143f364c13e3d5acd
Akshat Gupta
claimed_verified
2024-05-02T07:09:25.109Z
64e8f4a24f3f7b0b84834315
/avatars/242bb68c7ccffe5061c2d1c229ea3b0b.svg
Akshat Gupta
user
akshat57
663303d143f364c13e3d5ace
Gopala Anumanchipalli
admin_assigned
2024-05-02T07:22:20.771Z
60523e4aa7226b25aaeea2b8
/avatars/316ca348da91ebced86991f36150c959.svg
Gopala Anumanchipalli
user
gopalakr
Is Bigger Edit Batch Size Always Better? -- An Empirical Study on Model
  Editing with Llama-3
This study presents a targeted model editing analysis focused on the latest
large language model, Llama-3. We explore the efficacy of popular model editing
techniques - ROME, MEMIT, and EMMET, which are designed for precise layer
interventions. We identify the most effective layers for targeted edits through
an evaluation that encompasses up to 4096 edits across three distinct
strategies: sequential editing, batch editing, and a hybrid approach we call as
sequential-batch editing. Our findings indicate that increasing edit
batch-sizes may degrade model performance more significantly than using smaller
edit batches sequentially for equal number of edits. With this, we argue that
sequential model editing is an important component for scaling model editing
methods and future research should focus on methods that combine both batched
and sequential editing. This observation suggests a potential limitation in
current model editing methods which push towards bigger edit batch sizes, and
we hope it paves way for future investigations into optimizing batch sizes and
model editing performance.
The analysis of ROME, MEMIT, and EMMET editing techniques on Llama-3 highlights that sequential editing preserves model performance better than large batch editing across various strategies.
large language model ROME MEMIT EMMET targeted model editing layer interventions sequential editing batch editing sequential-batch editing model performance