Captain Cinema: Towards Short Movie Generation
We present Captain Cinema, a generation framework for short movie generation.
Given a detailed textual description of a movie storyline, our approach firstly
generates a sequence of keyframes that outline the entire narrative, which
ensures long-range coherence in both the storyline and visual appearance (e.g.,
scenes and characters). We refer to this step as top-down keyframe planning.
These keyframes then serve as conditioning signals for a video synthesis model,
which supports long context learning, to produce the spatio-temporal dynamics
between them. This step is referred to as bottom-up video synthesis. To support
stable and efficient generation of multi-scene long narrative cinematic works,
we introduce an interleaved training strategy for Multimodal Diffusion
Transformers (MM-DiT), specifically adapted for long-context video data. Our
model is trained on a specially curated cinematic dataset consisting of
interleaved data pairs. Our experiments demonstrate that Captain Cinema
performs favorably in the automated creation of visually coherent and narrative
consistent short movies in high quality and efficiency. Project page:
https://thecinema.ai
6882f472846d2e78b7548a58
Junfei Xiao
6882f472846d2e78b7548a59
Ceyuan Yang
6882f472846d2e78b7548a5a
Lvmin Zhang
6882f472846d2e78b7548a5b
Shengqu Cai
6882f472846d2e78b7548a5c
Yang Zhao
6882f472846d2e78b7548a5d
Yuwei Guo
6882f472846d2e78b7548a5e
Gordon Wetzstein
6882f472846d2e78b7548a5f
Maneesh Agrawala
6882f472846d2e78b7548a60
Alan Yuille
6882f472846d2e78b7548a61
Lu Jiang
Captain Cinema: Towards Short Movie Generation
We present Captain Cinema, a generation framework for short movie generation.
Given a detailed textual description of a movie storyline, our approach firstly
generates a sequence of keyframes that outline the entire narrative, which
ensures long-range coherence in both the storyline and visual appearance (e.g.,
scenes and characters). We refer to this step as top-down keyframe planning.
These keyframes then serve as conditioning signals for a video synthesis model,
which supports long context learning, to produce the spatio-temporal dynamics
between them. This step is referred to as bottom-up video synthesis. To support
stable and efficient generation of multi-scene long narrative cinematic works,
we introduce an interleaved training strategy for Multimodal Diffusion
Transformers (MM-DiT), specifically adapted for long-context video data. Our
model is trained on a specially curated cinematic dataset consisting of
interleaved data pairs. Our experiments demonstrate that Captain Cinema
performs favorably in the automated creation of visually coherent and narrative
consistent short movies in high quality and efficiency. Project page:
https://thecinema.ai
Captain Cinema generates coherent short movies from textual descriptions using top-down keyframe planning and bottom-up video synthesis with Multimodal Diffusion Transformers.
top-down keyframe planning bottom-up video synthesis Multimodal Diffusion Transformers MM-DiT long-context learning cinematic dataset interleaved training strategy