title: WorldDreamer: Towards General World Models for Video Generation via
    Predicting Masked Tokens
summary: World models play a crucial role in understanding and predicting the dynamics
  of the world, which is essential for video generation. However, existing world
  models are confined to specific scenarios such as gaming or driving, limiting
  their ability to capture the complexity of general world dynamic environments.
  Therefore, we introduce WorldDreamer, a pioneering world model to foster a
  comprehensive comprehension of general world physics and motions, which
  significantly enhances the capabilities of video generation. Drawing
  inspiration from the success of large language models, WorldDreamer frames
  world modeling as an unsupervised visual sequence modeling challenge. This is
  achieved by mapping visual inputs to discrete tokens and predicting the masked
  ones. During this process, we incorporate multi-modal prompts to facilitate
  interaction within the world model. Our experiments show that WorldDreamer
  excels in generating videos across different scenarios, including natural
  scenes and driving environments. WorldDreamer showcases versatility in
  executing tasks such as text-to-video conversion, image-tovideo synthesis, and
  video editing. These results underscore WorldDreamer's effectiveness in
  capturing dynamic elements within diverse general world environments.
paper_authors._id: 65a9d8ec848a78d4da076152
paper_authors.name: Xiaofeng Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-22T10:31:36.202Z
paper_authors._id: 6426616ea5ec4a5cbc535634
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6426616ea5ec4a5cbc535634/hH6JsxnXeakH3mBTeNNmO.jpeg
paper_authors.fullname: JeffWang
paper_authors.type: user
paper_authors.user: Jeff-Wang
paper_authors._id: 65a9d8ec848a78d4da076153
paper_authors.name: Zheng Zhu
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-01-19T02:05:33.694Z
paper_authors._id: 656e9b562cd7a3e348011d26
paper_authors.avatarUrl: /avatars/bcca51bdc27c664f8f132420e6ed99fa.svg
paper_authors.fullname: Zheng Zhu
paper_authors.type: user
paper_authors.user: ZhengZhu
paper_authors._id: 65a9d8ec848a78d4da076154
paper_authors.name: Guan Huang
paper_authors._id: 65a9d8ec848a78d4da076155
paper_authors.name: Boyuan Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-19T13:27:06.403Z
paper_authors._id: 632c74564a4991e711531495
paper_authors.avatarUrl: /avatars/f2c122c534127cf4c788d18156af2d98.svg
paper_authors.fullname: wang
paper_authors.type: user
paper_authors.user: boyuan
paper_authors._id: 65a9d8ec848a78d4da076156
paper_authors.name: Xinze Chen
paper_authors._id: 65a9d8ec848a78d4da076157
paper_authors.name: Jiwen Lu
paper_title: WorldDreamer: Towards General World Models for Video Generation via
    Predicting Masked Tokens
paper_summary: World models play a crucial role in understanding and predicting the dynamics
  of the world, which is essential for video generation. However, existing world
  models are confined to specific scenarios such as gaming or driving, limiting
  their ability to capture the complexity of general world dynamic environments.
  Therefore, we introduce WorldDreamer, a pioneering world model to foster a
  comprehensive comprehension of general world physics and motions, which
  significantly enhances the capabilities of video generation. Drawing
  inspiration from the success of large language models, WorldDreamer frames
  world modeling as an unsupervised visual sequence modeling challenge. This is
  achieved by mapping visual inputs to discrete tokens and predicting the masked
  ones. During this process, we incorporate multi-modal prompts to facilitate
  interaction within the world model. Our experiments show that WorldDreamer
  excels in generating videos across different scenarios, including natural
  scenes and driving environments. WorldDreamer showcases versatility in
  executing tasks such as text-to-video conversion, image-tovideo synthesis, and
  video editing. These results underscore WorldDreamer's effectiveness in
  capturing dynamic elements within diverse general world environments.
paper_ai_keywords: World models diffusion dynamic environments visual sequence modeling discrete tokens masked predictions multi-modal objects text-to-video conversion image-to-video synthesis video editing