title: LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs
summary: Current long context large language models (LLMs) can process inputs up to
  100,000 tokens, yet struggle to generate outputs exceeding even a modest length
  of 2,000 words. Through controlled experiments, we find that the model's
  effective generation length is inherently bounded by the sample it has seen
  during supervised fine-tuning (SFT). In other words, their output limitation is
  due to the scarcity of long-output examples in existing SFT datasets. To
  address this, we introduce AgentWrite, an agent-based pipeline that decomposes
  ultra-long generation tasks into subtasks, enabling off-the-shelf LLMs to
  generate coherent outputs exceeding 20,000 words. Leveraging AgentWrite, we
  construct LongWriter-6k, a dataset containing 6,000 SFT data with output
  lengths ranging from 2k to 32k words. By incorporating this dataset into model
  training, we successfully scale the output length of existing models to over
  10,000 words while maintaining output quality. We also develop LongBench-Write,
  a comprehensive benchmark for evaluating ultra-long generation capabilities.
  Our 9B parameter model, further improved through DPO, achieves state-of-the-art
  performance on this benchmark, surpassing even much larger proprietary models.
  In general, our work demonstrates that existing long context LLM already
  possesses the potential for a larger output window--all you need is data with
  extended output during model alignment to unlock this capability. Our code &
  models are at: https://github.com/THUDM/LongWriter.
paper_authors._id: 66bc1166d56c232066b9e717
paper_authors.name: Yushi Bai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-14T08:11:32.209Z
paper_authors._id: 64ed568ccf6118a9379a61b8
paper_authors.avatarUrl: /avatars/6d040cbcb4a9b624cbe64c9d01cd5c88.svg
paper_authors.fullname: Yushi Bai
paper_authors.type: user
paper_authors.user: bys0318
paper_authors._id: 66bc1166d56c232066b9e718
paper_authors.name: Jiajie Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-04T07:25:35.184Z
paper_authors._id: 66cdd285c51a915bd5f2d017
paper_authors.avatarUrl: /avatars/14e5794307e4672b1b51d26b31227e0f.svg
paper_authors.fullname: Jiajie Zhang
paper_authors.type: user
paper_authors.user: NeoZ123
paper_authors._id: 66bc1166d56c232066b9e719
paper_authors.name: Xin Lv
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-14T08:11:29.883Z
paper_authors._id: 648c64829c935db2b527a764
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/648c64829c935db2b527a764/AnTu1dvpac3hUxLQo8A8K.jpeg
paper_authors.fullname: Xin Lv
paper_authors.type: user
paper_authors.user: davidlvxin
paper_authors._id: 66bc1166d56c232066b9e71a
paper_authors.name: Linzhi Zheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-14T08:59:55.458Z
paper_authors._id: 66a99e585d7a1151f2cc41fe
paper_authors.avatarUrl: /avatars/c79feaa920e66864090358716eb0133c.svg
paper_authors.fullname: Linzhi Zheng
paper_authors.type: user
paper_authors.user: linzhizheng
paper_authors._id: 66bc1166d56c232066b9e71b
paper_authors.name: Siqi Zhu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-29T15:22:14.840Z
paper_authors._id: 65621fd68631d43d2baf33b2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/ZrNK7489lqq_vLeQgzhcR.png
paper_authors.fullname: siqi zhu
paper_authors.type: user
paper_authors.user: zsqzz
paper_authors._id: 66bc1166d56c232066b9e71c
paper_authors.name: Lei Hou
paper_authors._id: 66bc1166d56c232066b9e71d
paper_authors.name: Yuxiao Dong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-14T10:34:59.607Z
paper_authors._id: 640e73bdfdeaae1390857b62
paper_authors.avatarUrl: /avatars/cd6779e30f716002a7838ed93d5c0754.svg
paper_authors.fullname: Yuxiao Dong
paper_authors.type: user
paper_authors.user: yuxiaod
paper_authors._id: 66bc1166d56c232066b9e71e
paper_authors.name: Jie Tang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-14T10:34:53.801Z
paper_authors._id: 640dff05474aa6f89556677e
paper_authors.avatarUrl: /avatars/1b4591c7322d649c797b3125148f1915.svg
paper_authors.fullname: Jie Tang
paper_authors.type: user
paper_authors.user: jerytang
paper_authors._id: 66bc1166d56c232066b9e71f
paper_authors.name: Juanzi Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-14T10:34:46.893Z
paper_authors._id: 65df8cbc2705d9672f55d1aa
paper_authors.avatarUrl: /avatars/63e46f15bb76bd9d4508fd0f54f39829.svg
paper_authors.fullname: Juanzi Li
paper_authors.type: user
paper_authors.user: juanli
paper_title: LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs
paper_summary: Current long context large language models (LLMs) can process inputs up to
  100,000 tokens, yet struggle to generate outputs exceeding even a modest length
  of 2,000 words. Through controlled experiments, we find that the model's
  effective generation length is inherently bounded by the sample it has seen
  during supervised fine-tuning (SFT). In other words, their output limitation is
  due to the scarcity of long-output examples in existing SFT datasets. To
  address this, we introduce AgentWrite, an agent-based pipeline that decomposes
  ultra-long generation tasks into subtasks, enabling off-the-shelf LLMs to
  generate coherent outputs exceeding 20,000 words. Leveraging AgentWrite, we
  construct LongWriter-6k, a dataset containing 6,000 SFT data with output
  lengths ranging from 2k to 32k words. By incorporating this dataset into model
  training, we successfully scale the output length of existing models to over
  10,000 words while maintaining output quality. We also develop LongBench-Write,
  a comprehensive benchmark for evaluating ultra-long generation capabilities.
  Our 9B parameter model, further improved through DPO, achieves state-of-the-art
  performance on this benchmark, surpassing even much larger proprietary models.
  In general, our work demonstrates that existing long context LLM already
  possesses the potential for a larger output window--all you need is data with
  extended output during model alignment to unlock this capability. Our code &
  models are at: https://github.com/THUDM/LongWriter.
paper_ai_summary: AgentWrite enables long context LLMs to generate outputs exceeding 20,000 words by decomposing tasks into subtasks, using a new dataset to train models with extended output capabilities.
paper_ai_keywords: LLMs long context tokens AgentWrite subtasks LongWriter-6k long-output examples SFT datasets output quality LongBench-Write DPO