title: Language Models can Self-Lengthen to Generate Long Texts
summary: Recent advancements in Large Language Models (LLMs) have significantly
  enhanced their ability to process long contexts, yet a notable gap remains in
  generating long, aligned outputs. This limitation stems from a training gap
  where pre-training lacks effective instructions for long-text generation, and
  post-training data primarily consists of short query-response pairs. Current
  approaches, such as instruction backtranslation and behavior imitation, face
  challenges including data quality, copyright issues, and constraints on
  proprietary model usage. In this paper, we introduce an innovative iterative
  training framework called Self-Lengthen that leverages only the intrinsic
  knowledge and skills of LLMs without the need for auxiliary data or proprietary
  models. The framework consists of two roles: the Generator and the Extender.
  The Generator produces the initial response, which is then split and expanded
  by the Extender. This process results in a new, longer response, which is used
  to train both the Generator and the Extender iteratively. Through this process,
  the models are progressively trained to handle increasingly longer responses.
  Experiments on benchmarks and human evaluations show that Self-Lengthen
  outperforms existing methods in long-text generation, when applied to top
  open-source LLMs such as Qwen2 and LLaMA3. Our code is publicly available at
  https://github.com/QwenLM/Self-Lengthen.
paper_authors._id: 67242c98d26ffc2cbd545d21
paper_authors.name: Shanghaoran Quan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-03T14:22:34.065Z
paper_authors._id: 64b9954845ce8d7ad607c14d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64b9954845ce8d7ad607c14d/LZ9yeTOz4J_YKrnGkcmnL.jpeg
paper_authors.fullname: Shanghaoran Quan
paper_authors.type: user
paper_authors.user: quanshr
paper_authors._id: 67242c98d26ffc2cbd545d22
paper_authors.name: Tianyi Tang
paper_authors._id: 67242c98d26ffc2cbd545d23
paper_authors.name: Bowen Yu
paper_authors._id: 67242c98d26ffc2cbd545d24
paper_authors.name: An Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-19T06:43:14.449Z
paper_authors._id: 62088594a5943c8a8fc94560
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1644733028938-62088594a5943c8a8fc94560.png
paper_authors.fullname: An Yang
paper_authors.type: user
paper_authors.user: yangapku
paper_authors._id: 67242c98d26ffc2cbd545d25
paper_authors.name: Dayiheng Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-20T08:34:13.388Z
paper_authors._id: 6434d4989bd5a84b5dd0b0f5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6434d4989bd5a84b5dd0b0f5/0Elf9qbfG9Hkgypm9pTGm.jpeg
paper_authors.fullname: Dayiheng Liu
paper_authors.type: user
paper_authors.user: Losin94
paper_authors._id: 67242c98d26ffc2cbd545d26
paper_authors.name: Bofei Gao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:38:53.850Z
paper_authors._id: 65ae21adabf6d1ccb795e9a4
paper_authors.avatarUrl: /avatars/b5dced62c6a3564095a8fa0959bc06cb.svg
paper_authors.fullname: Bofei Gao
paper_authors.type: user
paper_authors.user: KbsdJames
paper_authors._id: 67242c98d26ffc2cbd545d27
paper_authors.name: Jianhong Tu
paper_authors._id: 67242c98d26ffc2cbd545d28
paper_authors.name: Yichang Zhang
paper_authors._id: 67242c98d26ffc2cbd545d29
paper_authors.name: Jingren Zhou
paper_authors._id: 67242c98d26ffc2cbd545d2a
paper_authors.name: Junyang Lin
paper_title: Language Models can Self-Lengthen to Generate Long Texts
paper_summary: Recent advancements in Large Language Models (LLMs) have significantly
  enhanced their ability to process long contexts, yet a notable gap remains in
  generating long, aligned outputs. This limitation stems from a training gap
  where pre-training lacks effective instructions for long-text generation, and
  post-training data primarily consists of short query-response pairs. Current
  approaches, such as instruction backtranslation and behavior imitation, face
  challenges including data quality, copyright issues, and constraints on
  proprietary model usage. In this paper, we introduce an innovative iterative
  training framework called Self-Lengthen that leverages only the intrinsic
  knowledge and skills of LLMs without the need for auxiliary data or proprietary
  models. The framework consists of two roles: the Generator and the Extender.
  The Generator produces the initial response, which is then split and expanded
  by the Extender. This process results in a new, longer response, which is used
  to train both the Generator and the Extender iteratively. Through this process,
  the models are progressively trained to handle increasingly longer responses.
  Experiments on benchmarks and human evaluations show that Self-Lengthen
  outperforms existing methods in long-text generation, when applied to top
  open-source LLMs such as Qwen2 and LLaMA3. Our code is publicly available at
  https://github.com/QwenLM/Self-Lengthen.
paper_ai_summary: The Self-Lengthen framework iteratively trains LLMs to generate longer, coherent outputs using only intrinsic model capabilities without requiring additional data or proprietary models.
paper_ai_keywords: Large Language Models LLMs long contexts long-text generation pre-training instruction backtranslation behavior imitation Self-Lengthen iterative training Generator Extender