title: LongPO: Long Context Self-Evolution of Large Language Models through
    Short-to-Long Preference Optimization
summary: Large Language Models (LLMs) have demonstrated remarkable capabilities
  through pretraining and alignment. However, superior short-context LLMs may
  underperform in long-context scenarios due to insufficient long-context
  alignment. This alignment process remains challenging due to the impracticality
  of human annotation for extended contexts and the difficulty in balancing
  short- and long-context performance. To address these challenges, we introduce
  LongPO, that enables short-context LLMs to self-evolve to excel on long-context
  tasks by internally transferring short-context capabilities. LongPO harnesses
  LLMs to learn from self-generated short-to-long preference data, comprising
  paired responses generated for identical instructions with long-context inputs
  and their compressed short-context counterparts, respectively. This preference
  reveals capabilities and potentials of LLMs cultivated during short-context
  alignment that may be diminished in under-aligned long-context scenarios.
  Additionally, LongPO incorporates a short-to-long KL constraint to mitigate
  short-context performance decline during long-context alignment. When applied
  to Mistral-7B-Instruct-v0.2 from 128K to 512K context lengths, LongPO fully
  retains short-context performance and largely outperforms naive SFT and DPO in
  both long- and short-context tasks. Specifically, \ourMethod-trained models can
  achieve results on long-context benchmarks comparable to, or even surpassing,
  those of superior LLMs (e.g., GPT-4-128K) that involve extensive long-context
  annotation and larger parameter scales.
paper_authors._id: 67b6948dbef24bad725b5d4b
paper_authors.name: Guanzheng Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-20T16:08:30.816Z
paper_authors._id: 645475e2548f22be59847604
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/645475e2548f22be59847604/EhSurrZ25u31qQ2TVXQXt.jpeg
paper_authors.fullname: Chen
paper_authors.type: user
paper_authors.user: Guanzheng
paper_authors._id: 67b6948dbef24bad725b5d4c
paper_authors.name: Xin Li
paper_authors._id: 67b6948dbef24bad725b5d4d
paper_authors.name: Michael Qizhe Shieh
paper_authors._id: 67b6948dbef24bad725b5d4e
paper_authors.name: Lidong Bing
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-20T16:08:48.204Z
paper_authors._id: 6454685a548f22be598414c4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/eMjMWKJ-AouF7eY1-RzGF.jpeg
paper_authors.fullname: Lidong Bing
paper_authors.type: user
paper_authors.user: LidongBing
paper_title: LongPO: Long Context Self-Evolution of Large Language Models through
    Short-to-Long Preference Optimization
paper_summary: Large Language Models (LLMs) have demonstrated remarkable capabilities
  through pretraining and alignment. However, superior short-context LLMs may
  underperform in long-context scenarios due to insufficient long-context
  alignment. This alignment process remains challenging due to the impracticality
  of human annotation for extended contexts and the difficulty in balancing
  short- and long-context performance. To address these challenges, we introduce
  LongPO, that enables short-context LLMs to self-evolve to excel on long-context
  tasks by internally transferring short-context capabilities. LongPO harnesses
  LLMs to learn from self-generated short-to-long preference data, comprising
  paired responses generated for identical instructions with long-context inputs
  and their compressed short-context counterparts, respectively. This preference
  reveals capabilities and potentials of LLMs cultivated during short-context
  alignment that may be diminished in under-aligned long-context scenarios.
  Additionally, LongPO incorporates a short-to-long KL constraint to mitigate
  short-context performance decline during long-context alignment. When applied
  to Mistral-7B-Instruct-v0.2 from 128K to 512K context lengths, LongPO fully
  retains short-context performance and largely outperforms naive SFT and DPO in
  both long- and short-context tasks. Specifically, \ourMethod-trained models can
  achieve results on long-context benchmarks comparable to, or even surpassing,
  those of superior LLMs (e.g., GPT-4-128K) that involve extensive long-context
  annotation and larger parameter scales.
paper_ai_keywords: Transformers LLMs long-context scenarios short- and long-context performance LongPO short-context LLMs self-evolve long-context grounds short-to-long preference data self-generated short-to-long preference data paired responses identical instructions long-context inputs short-context counterparts short-to-long preference short-context alignment under-aligned long-context scenarios short-to-long FL constraint SFT DPO long-context performance short-context performance long-context benchmarks long-context alignment long-context annotation parameter scales