title: A Unified View of Delta Parameter Editing in Post-Trained Large-Scale
    Models
summary: Post-training has emerged as a crucial paradigm for adapting large-scale
  pre-trained models to various tasks, whose effects are fully reflected by delta
  parameters (i.e., the disparity between post-trained and pre-trained
  parameters). While numerous studies have explored delta parameter properties
  via operations like pruning, quantization, low-rank approximation, and
  extrapolation, a unified framework for systematically examining these
  characteristics has been lacking. In this paper, we propose a novel perspective
  based on Riemann sum approximation of the loss function to elucidate delta
  parameter editing operations. Our analysis categorizes existing methods into
  three classes based on their post-editing performance: competitive, decreased,
  and improved, explaining how they are expressed by the Riemann sum
  approximation term and how they alter the model performance. Extensive
  experiments on both visual and language models, including ViT, LLaMA 3, Qwen 2,
  and Mistral, corroborate our theoretical findings. Furthermore, we introduce
  extensions to existing techniques like DARE and BitDelta, highlighting their
  limitations in leveraging the properties of delta parameters and reorganizing
  them into general expressions to enhance the applicability and effectiveness of
  delta parameter editing in post-trained models.
paper_authors._id: 6711c61c1e98ec0ba6bc418c
paper_authors.name: Qiaoyu Tang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-18T08:12:33.542Z
paper_authors._id: 64ed96f3067fbb625f83766a
paper_authors.avatarUrl: /avatars/e8794ba19bdd965615bd9ead7df77d7d.svg
paper_authors.fullname: Qiaoyu Tang
paper_authors.type: user
paper_authors.user: TangQiaoYu
paper_authors._id: 6711c61c1e98ec0ba6bc418d
paper_authors.name: Le Yu
paper_authors._id: 6711c61c1e98ec0ba6bc418e
paper_authors.name: Bowen Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-18T12:36:39.144Z
paper_authors._id: 6438b43ab2ea24b52ebac2b9
paper_authors.avatarUrl: /avatars/84133cd719a4b1e2f5c1a74178425f86.svg
paper_authors.fullname: Bowen Yu
paper_authors.type: user
paper_authors.user: bwy
paper_authors._id: 6711c61c1e98ec0ba6bc418f
paper_authors.name: Hongyu Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-18T12:36:25.734Z
paper_authors._id: 6711c702f858a456b4b9f3a4
paper_authors.avatarUrl: /avatars/178e9567c3111ab22717c3c0dd003a6a.svg
paper_authors.fullname: Hongyu  Lin
paper_authors.type: user
paper_authors.user: sanmusunrise
paper_authors._id: 6711c61c1e98ec0ba6bc4190
paper_authors.name: Keming Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-18T12:36:53.561Z
paper_authors._id: 6453fa96ed6d7fede94408e0
paper_authors.avatarUrl: /avatars/e8c9025ef24cec958c87a1008bb54fd7.svg
paper_authors.fullname: Keming Lu
paper_authors.type: user
paper_authors.user: keminglu
paper_authors._id: 6711c61c1e98ec0ba6bc4191
paper_authors.name: Yaojie Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-18T12:36:59.079Z
paper_authors._id: 6216496a9b34d2fb49144599
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6216496a9b34d2fb49144599/41CKA_h1Ffj3RzVabSAkm.jpeg
paper_authors.fullname: Yaojie Lu
paper_authors.type: user
paper_authors.user: luyaojie
paper_authors._id: 6711c61c1e98ec0ba6bc4192
paper_authors.name: Xianpei Han
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-18T12:36:19.025Z
paper_authors._id: 65e99a77e71555ed193609cf
paper_authors.avatarUrl: /avatars/38ceb127883944677665da967d17dd18.svg
paper_authors.fullname: Xianpei Han
paper_authors.type: user
paper_authors.user: xphan
paper_authors._id: 6711c61c1e98ec0ba6bc4193
paper_authors.name: Le Sun
paper_title: A Unified View of Delta Parameter Editing in Post-Trained Large-Scale
    Models
paper_summary: Post-training has emerged as a crucial paradigm for adapting large-scale
  pre-trained models to various tasks, whose effects are fully reflected by delta
  parameters (i.e., the disparity between post-trained and pre-trained
  parameters). While numerous studies have explored delta parameter properties
  via operations like pruning, quantization, low-rank approximation, and
  extrapolation, a unified framework for systematically examining these
  characteristics has been lacking. In this paper, we propose a novel perspective
  based on Riemann sum approximation of the loss function to elucidate delta
  parameter editing operations. Our analysis categorizes existing methods into
  three classes based on their post-editing performance: competitive, decreased,
  and improved, explaining how they are expressed by the Riemann sum
  approximation term and how they alter the model performance. Extensive
  experiments on both visual and language models, including ViT, LLaMA 3, Qwen 2,
  and Mistral, corroborate our theoretical findings. Furthermore, we introduce
  extensions to existing techniques like DARE and BitDelta, highlighting their
  limitations in leveraging the properties of delta parameters and reorganizing
  them into general expressions to enhance the applicability and effectiveness of
  delta parameter editing in post-trained models.
paper_ai_keywords: Riemann sum approximation delta parameter editing pruning quantization low-rank approximation extrapolation post-editing performance competitive methods seeking methods improved methods post-training ViT LLaMA 3 Qwen 2 Mistral DARE BitDelta