title: MovieDreamer: Hierarchical Generation for Coherent Long Visual Sequence
summary: Recent advancements in video generation have primarily leveraged diffusion
  models for short-duration content. However, these approaches often fall short
  in modeling complex narratives and maintaining character consistency over
  extended periods, which is essential for long-form video production like
  movies. We propose MovieDreamer, a novel hierarchical framework that integrates
  the strengths of autoregressive models with diffusion-based rendering to
  pioneer long-duration video generation with intricate plot progressions and
  high visual fidelity. Our approach utilizes autoregressive models for global
  narrative coherence, predicting sequences of visual tokens that are
  subsequently transformed into high-quality video frames through diffusion
  rendering. This method is akin to traditional movie production processes, where
  complex stories are factorized down into manageable scene capturing. Further,
  we employ a multimodal script that enriches scene descriptions with detailed
  character information and visual style, enhancing continuity and character
  identity across scenes. We present extensive experiments across various movie
  genres, demonstrating that our approach not only achieves superior visual and
  narrative quality but also effectively extends the duration of generated
  content significantly beyond current capabilities. Homepage:
  https://aim-uofa.github.io/MovieDreamer/.
paper_authors._id: 66a05dafcf52b98e1d2873e8
paper_authors.name: Canyu Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-24T08:06:36.821Z
paper_authors._id: 646efd223dd912a539e0bd46
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/EOFAv5xvOgJOzuDgh4nSb.png
paper_authors.fullname: Canyu Zhao
paper_authors.type: user
paper_authors.user: Canyu
paper_authors._id: 66a05dafcf52b98e1d2873e9
paper_authors.name: Mingyu Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-24T08:06:54.688Z
paper_authors._id: 652e25d2e647b0ee0a024f26
paper_authors.avatarUrl: /avatars/b5c65cf6c8d0ddc9b8ef0226e0295d56.svg
paper_authors.fullname: Mingyu Liu
paper_authors.type: user
paper_authors.user: MingyuLiu
paper_authors._id: 66a05dafcf52b98e1d2873ea
paper_authors.name: Wen Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-28T11:28:49.379Z
paper_authors._id: 63f089456309c84d5f47f951
paper_authors.avatarUrl: /avatars/04b926a7f2ad091ee00fef0c59903492.svg
paper_authors.fullname: Wen Wang
paper_authors.type: user
paper_authors.user: wwen1997
paper_authors._id: 66a05dafcf52b98e1d2873eb
paper_authors.name: Jianlong Yuan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-24T08:07:29.462Z
paper_authors._id: 66684c82473b56ef2d7c340b
paper_authors.avatarUrl: /avatars/9158f3f2c0e287c26d6910ea14edaa3a.svg
paper_authors.fullname: Yuan Jian-Long
paper_authors.type: user
paper_authors.user: OxyOksirotl
paper_authors._id: 66a05dafcf52b98e1d2873ec
paper_authors.name: Hao Chen
paper_authors._id: 66a05dafcf52b98e1d2873ed
paper_authors.name: Bo Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-21T07:53:52.959Z
paper_authors._id: 6237f3eb4cf571bb1ed4036b
paper_authors.avatarUrl: /avatars/cbfb8ddda18e65d18a818db9f176e16e.svg
paper_authors.fullname: Bo Zhang
paper_authors.type: user
paper_authors.user: zhangmozhe
paper_authors._id: 66a05dafcf52b98e1d2873ee
paper_authors.name: Chunhua Shen
paper_title: MovieDreamer: Hierarchical Generation for Coherent Long Visual Sequence
paper_summary: Recent advancements in video generation have primarily leveraged diffusion
  models for short-duration content. However, these approaches often fall short
  in modeling complex narratives and maintaining character consistency over
  extended periods, which is essential for long-form video production like
  movies. We propose MovieDreamer, a novel hierarchical framework that integrates
  the strengths of autoregressive models with diffusion-based rendering to
  pioneer long-duration video generation with intricate plot progressions and
  high visual fidelity. Our approach utilizes autoregressive models for global
  narrative coherence, predicting sequences of visual tokens that are
  subsequently transformed into high-quality video frames through diffusion
  rendering. This method is akin to traditional movie production processes, where
  complex stories are factorized down into manageable scene capturing. Further,
  we employ a multimodal script that enriches scene descriptions with detailed
  character information and visual style, enhancing continuity and character
  identity across scenes. We present extensive experiments across various movie
  genres, demonstrating that our approach not only achieves superior visual and
  narrative quality but also effectively extends the duration of generated
  content significantly beyond current capabilities. Homepage:
  https://aim-uofa.github.io/MovieDreamer/.
paper_ai_keywords: diffusion models hierarchical framework autoregressive models diffusion-based rendering global narrative coherence visual tokens high-quality video frames multimodal script character information visual style scene capturing movie production processes