title: Accelerating Direct Preference Optimization with Prefix Sharing
summary: Offline paired preference optimization algorithms have become a popular
  approach for fine-tuning on preference data, outperforming traditional
  supervised fine-tuning in various tasks. However, traditional implementations
  often involve redundant computations, especially for tasks with long shared
  prompts. We introduce prefix sharing for preference tuning, a novel technique
  that processes chosen and rejected responses as one sequence with a shared
  prefix. To prevent cross-response contamination, we use a custom block-sparse
  attention mask. Our method achieves 1.1-1.5times improvement in training
  throughput on popular DPO datasets, without any effect on convergence. When
  combined with sequence packing, we observe consistent 1.3-1.6times
  speedups, benefiting even datasets with smaller sequence lengths. While we
  focus on Direct Preference Optimization (DPO), our approach is applicable to
  other paired preference tuning methods. By enhancing computational efficiency,
  our work contributes to making preference-based fine-tuning more accessible for
  a wider range of applications and model sizes. We open-source our code at
  https://github.com/frankxwang/dpo-prefix-sharing.
paper_authors._id: 67226f3f8fedb32c0f117ccc
paper_authors.name: Franklin Wang
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-10-30T17:44:39.930Z
paper_authors._id: 6537ec2f33b10aa38f3f6469
paper_authors.avatarUrl: /avatars/f25b2dfb2dc4f03b245761dccf3df2d7.svg
paper_authors.fullname: Franklin Wang
paper_authors.type: user
paper_authors.user: fxwang
paper_authors._id: 67226f3f8fedb32c0f117ccd
paper_authors.name: Sumanth Hegde
paper_title: Accelerating Direct Preference Optimization with Prefix Sharing
paper_summary: Offline paired preference optimization algorithms have become a popular
  approach for fine-tuning on preference data, outperforming traditional
  supervised fine-tuning in various tasks. However, traditional implementations
  often involve redundant computations, especially for tasks with long shared
  prompts. We introduce prefix sharing for preference tuning, a novel technique
  that processes chosen and rejected responses as one sequence with a shared
  prefix. To prevent cross-response contamination, we use a custom block-sparse
  attention mask. Our method achieves 1.1-1.5times improvement in training
  throughput on popular DPO datasets, without any effect on convergence. When
  combined with sequence packing, we observe consistent 1.3-1.6times
  speedups, benefiting even datasets with smaller sequence lengths. While we
  focus on Direct Preference Optimization (DPO), our approach is applicable to
  other paired preference tuning methods. By enhancing computational efficiency,
  our work contributes to making preference-based fine-tuning more accessible for
  a wider range of applications and model sizes. We open-source our code at
  https://github.com/frankxwang/dpo-prefix-sharing.
paper_ai_summary: Prefix sharing with a block-sparse attention mask improves training throughput in preference-based fine-tuning without affecting convergence, making it accessible for various model sizes and applications.
paper_ai_keywords: offline paired preference optimization fine-tuning Direct Preference Optimization prefix sharing block-sparse attention mask sequence packing