title: LongWriter-V: Enabling Ultra-Long and High-Fidelity Generation in
    Vision-Language Models
summary: Existing Large Vision-Language Models (LVLMs) can process inputs with context
  lengths up to 128k visual and text tokens, yet they struggle to generate
  coherent outputs beyond 1,000 words. We find that the primary limitation is the
  absence of long output examples during supervised fine-tuning (SFT). To tackle
  this issue, we introduce LongWriter-V-22k, a SFT dataset comprising 22,158
  examples, each with multiple input images, an instruction, and corresponding
  outputs ranging from 0 to 10,000 words. Moreover, to achieve long outputs that
  maintain high-fidelity to the input images, we employ Direct Preference
  Optimization (DPO) to the SFT model. Given the high cost of collecting human
  feedback for lengthy outputs (e.g., 3,000 words), we propose IterDPO, which
  breaks long outputs into segments and uses iterative corrections to form
  preference pairs with the original outputs. Additionally, we develop
  MMLongBench-Write, a benchmark featuring six tasks to evaluate the
  long-generation capabilities of VLMs. Our 7B parameter model, trained with
  LongWriter-V-22k and IterDPO, achieves impressive performance on this
  benchmark, outperforming larger proprietary models like GPT-4o. Code and data:
  https://github.com/THU-KEG/LongWriter-V
paper_authors._id: 67b7f3c4d00e69f10cff219e
paper_authors.name: Shangqing Tu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-21T14:47:11.953Z
paper_authors._id: 648c48d8c0ddeee6df5b6d22
paper_authors.avatarUrl: /avatars/8706b0b16dfc332b96c91d3ced31bd0b.svg
paper_authors.fullname: Shangqing Tu
paper_authors.type: user
paper_authors.user: tsq2000
paper_authors._id: 67b7f3c4d00e69f10cff219f
paper_authors.name: Yucheng Wang
paper_authors._id: 67b7f3c4d00e69f10cff21a0
paper_authors.name: Daniel Zhang-Li
paper_authors._id: 67b7f3c4d00e69f10cff21a1
paper_authors.name: Yushi Bai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-21T14:47:42.383Z
paper_authors._id: 64ed568ccf6118a9379a61b8
paper_authors.avatarUrl: /avatars/6d040cbcb4a9b624cbe64c9d01cd5c88.svg
paper_authors.fullname: Yushi Bai
paper_authors.type: user
paper_authors.user: bys0318
paper_authors._id: 67b7f3c4d00e69f10cff21a2
paper_authors.name: Jifan Yu
paper_authors._id: 67b7f3c4d00e69f10cff21a3
paper_authors.name: Yuhao Wu
paper_authors._id: 67b7f3c4d00e69f10cff21a4
paper_authors.name: Lei Hou
paper_authors._id: 67b7f3c4d00e69f10cff21a5
paper_authors.name: Huiqin Liu
paper_authors._id: 67b7f3c4d00e69f10cff21a6
paper_authors.name: Zhiyuan Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-21T14:48:24.334Z
paper_authors._id: 6310a3cd531cc21f9e06de6a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6310a3cd531cc21f9e06de6a/aTGMx3O41lUARK9s3dAik.jpeg
paper_authors.fullname: Zhiyuan Liu
paper_authors.type: user
paper_authors.user: acharkq
paper_authors._id: 67b7f3c4d00e69f10cff21a7
paper_authors.name: Bin Xu
paper_authors._id: 67b7f3c4d00e69f10cff21a8
paper_authors.name: Juanzi Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-21T14:48:30.590Z
paper_authors._id: 65df8cbc2705d9672f55d1aa
paper_authors.avatarUrl: /avatars/63e46f15bb76bd9d4508fd0f54f39829.svg
paper_authors.fullname: Juanzi Li
paper_authors.type: user
paper_authors.user: juanli
paper_title: LongWriter-V: Enabling Ultra-Long and High-Fidelity Generation in
    Vision-Language Models
paper_summary: Existing Large Vision-Language Models (LVLMs) can process inputs with context
  lengths up to 128k visual and text tokens, yet they struggle to generate
  coherent outputs beyond 1,000 words. We find that the primary limitation is the
  absence of long output examples during supervised fine-tuning (SFT). To tackle
  this issue, we introduce LongWriter-V-22k, a SFT dataset comprising 22,158
  examples, each with multiple input images, an instruction, and corresponding
  outputs ranging from 0 to 10,000 words. Moreover, to achieve long outputs that
  maintain high-fidelity to the input images, we employ Direct Preference
  Optimization (DPO) to the SFT model. Given the high cost of collecting human
  feedback for lengthy outputs (e.g., 3,000 words), we propose IterDPO, which
  breaks long outputs into segments and uses iterative corrections to form
  preference pairs with the original outputs. Additionally, we develop
  MMLongBench-Write, a benchmark featuring six tasks to evaluate the
  long-generation capabilities of VLMs. Our 7B parameter model, trained with
  LongWriter-V-22k and IterDPO, achieves impressive performance on this
  benchmark, outperforming larger proprietary models like GPT-4o. Code and data:
  https://github.com/THU-KEG/LongWriter-V
paper_ai_keywords: Large Vision-Language Models (LVLMs) context lengths visual and text tokens supervised fine-tuning (SFT) LongWriter-V-22k Direct Preference Optimization (DPO) IterDPO MMLongBench-Write parameter model GPT-4o