title: LOGO -- Long cOntext aliGnment via efficient preference Optimization
summary: Long-context models(LCMs) have shown great potential in processing long input
  sequences(even more than 100M tokens) conveniently and effectively. With
  significant progress, recent research has pointed out that LCMs can accurately
  locate token-level salient information within the context. Yet, the generation
  performance of these LCMs is far from satisfactory and might result in
  misaligned responses, such as hallucinations. To enhance the generation
  capability of LCMs, existing works have investigated the effects of data size
  and quality for both pre-training and instruction tuning. Though achieving
  meaningful improvement, previous methods fall short in either effectiveness or
  efficiency. In this paper, we introduce LOGO(Long cOntext aliGnment via
  efficient preference Optimization), a training strategy that first introduces
  preference optimization for long-context alignment. To overcome the GPU
  memory-bound issue caused by the long sequence, LOGO employs a reference-free
  preference optimization strategy and adopts a position synthesis method to
  construct the training data. By training with only 0.3B data on a single
  8timesA800 GPU machine for 16 hours, LOGO allows the Llama-3-8B-Instruct-80K
  model to achieve comparable performance with GPT-4 in real-world long-context
  tasks while preserving the model's original capabilities on other tasks, e.g.,
  language modeling and MMLU. Moreover, LOGO can extend the model's context
  window size while enhancing its generation performance.
paper_authors._id: 671b0a2da9a89705ec15a277
paper_authors.name: Zecheng Tang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-25T11:24:53.935Z
paper_authors._id: 64096ef79e9f790c905b846d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64096ef79e9f790c905b846d/hVzw656lXdzbCxTtnheud.jpeg
paper_authors.fullname: Zecheng Tang
paper_authors.type: user
paper_authors.user: ZetangForward
paper_authors._id: 671b0a2da9a89705ec15a278
paper_authors.name: Zechen Sun
paper_authors._id: 671b0a2da9a89705ec15a279
paper_authors.name: Juntao Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-25T11:25:12.073Z
paper_authors._id: 6670e285b0c03c4e9d6e0985
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/uCZHm4gKSHZ2b0hpHWgZv.jpeg
paper_authors.fullname: Juntao Li
paper_authors.type: user
paper_authors.user: douvleplus
paper_authors._id: 671b0a2da9a89705ec15a27a
paper_authors.name: Qiaoming Zhu
paper_authors._id: 671b0a2da9a89705ec15a27b
paper_authors.name: Min Zhang
paper_title: LOGO -- Long cOntext aliGnment via efficient preference Optimization
paper_summary: Long-context models(LCMs) have shown great potential in processing long input
  sequences(even more than 100M tokens) conveniently and effectively. With
  significant progress, recent research has pointed out that LCMs can accurately
  locate token-level salient information within the context. Yet, the generation
  performance of these LCMs is far from satisfactory and might result in
  misaligned responses, such as hallucinations. To enhance the generation
  capability of LCMs, existing works have investigated the effects of data size
  and quality for both pre-training and instruction tuning. Though achieving
  meaningful improvement, previous methods fall short in either effectiveness or
  efficiency. In this paper, we introduce LOGO(Long cOntext aliGnment via
  efficient preference Optimization), a training strategy that first introduces
  preference optimization for long-context alignment. To overcome the GPU
  memory-bound issue caused by the long sequence, LOGO employs a reference-free
  preference optimization strategy and adopts a position synthesis method to
  construct the training data. By training with only 0.3B data on a single
  8timesA800 GPU machine for 16 hours, LOGO allows the Llama-3-8B-Instruct-80K
  model to achieve comparable performance with GPT-4 in real-world long-context
  tasks while preserving the model's original capabilities on other tasks, e.g.,
  language modeling and MMLU. Moreover, LOGO can extend the model's context
  window size while enhancing its generation performance.
paper_ai_keywords: Long-context models (LCMs) Long input sequences Token-level salient information Generation performance Misaligned responses Hallucinations Data size and quality Pre-training Instruction tuning Preference optimization Long-context alignment GPU memory-bound issue Reference-free preference optimization Position synthesis method Context window size Language modeling MMLU