LongVie: Multimodal-Guided Controllable Ultra-Long Video Generation
Controllable ultra-long video generation is a fundamental yet challenging
task. Although existing methods are effective for short clips, they struggle to
scale due to issues such as temporal inconsistency and visual degradation. In
this paper, we initially investigate and identify three key factors: separate
noise initialization, independent control signal normalization, and the
limitations of single-modality guidance. To address these issues, we propose
LongVie, an end-to-end autoregressive framework for controllable long video
generation. LongVie introduces two core designs to ensure temporal consistency:
1) a unified noise initialization strategy that maintains consistent generation
across clips, and 2) global control signal normalization that enforces
alignment in the control space throughout the entire video. To mitigate visual
degradation, LongVie employs 3) a multi-modal control framework that integrates
both dense (e.g., depth maps) and sparse (e.g., keypoints) control signals,
complemented by 4) a degradation-aware training strategy that adaptively
balances modality contributions over time to preserve visual quality. We also
introduce LongVGenBench, a comprehensive benchmark consisting of 100
high-resolution videos spanning diverse real-world and synthetic environments,
each lasting over one minute. Extensive experiments show that LongVie achieves
state-of-the-art performance in long-range controllability, consistency, and
quality.
6892b64d8da45ffb0a2b23d4
Jianxiong Gao
6892b64d8da45ffb0a2b23d5
Zhaoxi Chen
6892b64d8da45ffb0a2b23d6
Xian Liu
6892b64d8da45ffb0a2b23d7
Jianfeng Feng
6892b64d8da45ffb0a2b23d8
Chenyang Si
6892b64d8da45ffb0a2b23d9
Yanwei Fu
6892b64d8da45ffb0a2b23da
Yu Qiao
6892b64d8da45ffb0a2b23db
Ziwei Liu
LongVie: Multimodal-Guided Controllable Ultra-Long Video Generation
Controllable ultra-long video generation is a fundamental yet challenging
task. Although existing methods are effective for short clips, they struggle to
scale due to issues such as temporal inconsistency and visual degradation. In
this paper, we initially investigate and identify three key factors: separate
noise initialization, independent control signal normalization, and the
limitations of single-modality guidance. To address these issues, we propose
LongVie, an end-to-end autoregressive framework for controllable long video
generation. LongVie introduces two core designs to ensure temporal consistency:
1) a unified noise initialization strategy that maintains consistent generation
across clips, and 2) global control signal normalization that enforces
alignment in the control space throughout the entire video. To mitigate visual
degradation, LongVie employs 3) a multi-modal control framework that integrates
both dense (e.g., depth maps) and sparse (e.g., keypoints) control signals,
complemented by 4) a degradation-aware training strategy that adaptively
balances modality contributions over time to preserve visual quality. We also
introduce LongVGenBench, a comprehensive benchmark consisting of 100
high-resolution videos spanning diverse real-world and synthetic environments,
each lasting over one minute. Extensive experiments show that LongVie achieves
state-of-the-art performance in long-range controllability, consistency, and
quality.
LongVie, an end-to-end autoregressive framework, addresses temporal consistency and visual degradation in ultra-long video generation through unified noise initialization, global control signal normalization, multi-modal control, and degradation-aware training.
autoregressive framework temporal consistency visual degradation unified noise initialization global control signal normalization multi-modal control degradation-aware training LongVGenBench