LongWriter-V: Enabling Ultra-Long and High-Fidelity Generation in
  Vision-Language Models
Existing Large Vision-Language Models (LVLMs) can process inputs with context
lengths up to 128k visual and text tokens, yet they struggle to generate
coherent outputs beyond 1,000 words. We find that the primary limitation is the
absence of long output examples during supervised fine-tuning (SFT). To tackle
this issue, we introduce LongWriter-V-22k, a SFT dataset comprising 22,158
examples, each with multiple input images, an instruction, and corresponding
outputs ranging from 0 to 10,000 words. Moreover, to achieve long outputs that
maintain high-fidelity to the input images, we employ Direct Preference
Optimization (DPO) to the SFT model. Given the high cost of collecting human
feedback for lengthy outputs (e.g., 3,000 words), we propose IterDPO, which
breaks long outputs into segments and uses iterative corrections to form
preference pairs with the original outputs. Additionally, we develop
MMLongBench-Write, a benchmark featuring six tasks to evaluate the
long-generation capabilities of VLMs. Our 7B parameter model, trained with
LongWriter-V-22k and IterDPO, achieves impressive performance on this
benchmark, outperforming larger proprietary models like GPT-4o. Code and data:
https://github.com/THU-KEG/LongWriter-V
67b7f3c4d00e69f10cff219e
Shangqing Tu
admin_assigned
2025-02-21T14:47:11.953Z
648c48d8c0ddeee6df5b6d22
/avatars/8706b0b16dfc332b96c91d3ced31bd0b.svg
Shangqing Tu
user
tsq2000
67b7f3c4d00e69f10cff219f
Yucheng Wang
67b7f3c4d00e69f10cff21a0
Daniel Zhang-Li
67b7f3c4d00e69f10cff21a1
Yushi Bai
admin_assigned
2025-02-21T14:47:42.383Z
64ed568ccf6118a9379a61b8
/avatars/6d040cbcb4a9b624cbe64c9d01cd5c88.svg
Yushi Bai
user
bys0318
67b7f3c4d00e69f10cff21a2
Jifan Yu
67b7f3c4d00e69f10cff21a3
Yuhao Wu
67b7f3c4d00e69f10cff21a4
Lei Hou
67b7f3c4d00e69f10cff21a5
Huiqin Liu
67b7f3c4d00e69f10cff21a6
Zhiyuan Liu
admin_assigned
2025-02-21T14:48:24.334Z
6310a3cd531cc21f9e06de6a
https://cdn-avatars.huggingface.co/v1/production/uploads/6310a3cd531cc21f9e06de6a/aTGMx3O41lUARK9s3dAik.jpeg
Zhiyuan Liu
user
acharkq
67b7f3c4d00e69f10cff21a7
Bin Xu
67b7f3c4d00e69f10cff21a8
Juanzi Li
admin_assigned
2025-02-21T14:48:30.590Z
65df8cbc2705d9672f55d1aa
/avatars/63e46f15bb76bd9d4508fd0f54f39829.svg
Juanzi Li
user
juanli
LongWriter-V: Enabling Ultra-Long and High-Fidelity Generation in
  Vision-Language Models
Existing Large Vision-Language Models (LVLMs) can process inputs with context
lengths up to 128k visual and text tokens, yet they struggle to generate
coherent outputs beyond 1,000 words. We find that the primary limitation is the
absence of long output examples during supervised fine-tuning (SFT). To tackle
this issue, we introduce LongWriter-V-22k, a SFT dataset comprising 22,158
examples, each with multiple input images, an instruction, and corresponding
outputs ranging from 0 to 10,000 words. Moreover, to achieve long outputs that
maintain high-fidelity to the input images, we employ Direct Preference
Optimization (DPO) to the SFT model. Given the high cost of collecting human
feedback for lengthy outputs (e.g., 3,000 words), we propose IterDPO, which
breaks long outputs into segments and uses iterative corrections to form
preference pairs with the original outputs. Additionally, we develop
MMLongBench-Write, a benchmark featuring six tasks to evaluate the
long-generation capabilities of VLMs. Our 7B parameter model, trained with
LongWriter-V-22k and IterDPO, achieves impressive performance on this
benchmark, outperforming larger proprietary models like GPT-4o. Code and data:
https://github.com/THU-KEG/LongWriter-V
A novel dataset and optimization method improve long-output generation in vision-language models while maintaining high-fidelity to input images.
Large Vision-Language Models LongWriter-V-22k Direct Preference Optimization IterDPO MMLongBench-Write