title: MagicTime: Time-lapse Video Generation Models as Metamorphic Simulators
summary: Recent advances in Text-to-Video generation (T2V) have achieved remarkable
  success in synthesizing high-quality general videos from textual descriptions.
  A largely overlooked problem in T2V is that existing models have not adequately
  encoded physical knowledge of the real world, thus generated videos tend to
  have limited motion and poor variations. In this paper, we propose
  MagicTime, a metamorphic time-lapse video generation model, which
  learns real-world physics knowledge from time-lapse videos and implements
  metamorphic generation. First, we design a MagicAdapter scheme to decouple
  spatial and temporal training, encode more physical knowledge from metamorphic
  videos, and transform pre-trained T2V models to generate metamorphic videos.
  Second, we introduce a Dynamic Frames Extraction strategy to adapt to
  metamorphic time-lapse videos, which have a wider variation range and cover
  dramatic object metamorphic processes, thus embodying more physical knowledge
  than general videos. Finally, we introduce a Magic Text-Encoder to improve the
  understanding of metamorphic video prompts. Furthermore, we create a time-lapse
  video-text dataset called ChronoMagic, specifically curated to unlock
  the metamorphic video generation ability. Extensive experiments demonstrate the
  superiority and effectiveness of MagicTime for generating high-quality and
  dynamic metamorphic videos, suggesting time-lapse video generation is a
  promising path toward building metamorphic simulators of the physical world.
paper_authors._id: 6614b4f2c13e5d702f55d7f5
paper_authors.name: Shenghai Yuan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-28T14:46:06.995Z
paper_authors._id: 63468720dd6d90d82ccf3450
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63468720dd6d90d82ccf3450/tVBFlmZNz8FRMkOrDaDID.jpeg
paper_authors.fullname: YSH
paper_authors.type: user
paper_authors.user: BestWishYsh
paper_authors._id: 6614b4f2c13e5d702f55d7f6
paper_authors.name: Jinfa Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-27T18:48:06.189Z
paper_authors._id: 63f37af60be81bdc5d92eebb
paper_authors.avatarUrl: /avatars/b8dfdff4ab36988ec9a8643e82a3d2db.svg
paper_authors.fullname: Huang
paper_authors.type: user
paper_authors.user: Jinfa
paper_authors._id: 6614b4f2c13e5d702f55d7f7
paper_authors.name: Yujun Shi
paper_authors._id: 6614b4f2c13e5d702f55d7f8
paper_authors.name: Yongqi Xu
paper_authors._id: 6614b4f2c13e5d702f55d7f9
paper_authors.name: Ruijie Zhu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-14T19:12:37.238Z
paper_authors._id: 63ff09f24852102d4871c19c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63ff09f24852102d4871c19c/lyE3xemtZss3qebK5sEXw.png
paper_authors.fullname: Rui-Jie Zhu
paper_authors.type: user
paper_authors.user: ridger
paper_authors._id: 6614b4f2c13e5d702f55d7fa
paper_authors.name: Bin Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-09T19:47:53.689Z
paper_authors._id: 6367a8175bb06007ea099b8f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6367a8175bb06007ea099b8f/IjG7HyWyWRlVt_XwRbxRW.jpeg
paper_authors.fullname: linbin
paper_authors.type: user
paper_authors.user: LanguageBind
paper_authors._id: 6614b4f2c13e5d702f55d7fb
paper_authors.name: Xinhua Cheng
paper_authors._id: 6614b4f2c13e5d702f55d7fc
paper_authors.name: Li Yuan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-04T07:12:11.281Z
paper_authors._id: 66135a5e50350afe76beebce
paper_authors.avatarUrl: /avatars/370a4b83949355feb050c2cb0425c264.svg
paper_authors.fullname: yl2488
paper_authors.type: user
paper_authors.user: yl2488
paper_authors._id: 6614b4f2c13e5d702f55d7fd
paper_authors.name: Jiebo Luo
paper_title: MagicTime: Time-lapse Video Generation Models as Metamorphic Simulators
paper_summary: Recent advances in Text-to-Video generation (T2V) have achieved remarkable
  success in synthesizing high-quality general videos from textual descriptions.
  A largely overlooked problem in T2V is that existing models have not adequately
  encoded physical knowledge of the real world, thus generated videos tend to
  have limited motion and poor variations. In this paper, we propose
  MagicTime, a metamorphic time-lapse video generation model, which
  learns real-world physics knowledge from time-lapse videos and implements
  metamorphic generation. First, we design a MagicAdapter scheme to decouple
  spatial and temporal training, encode more physical knowledge from metamorphic
  videos, and transform pre-trained T2V models to generate metamorphic videos.
  Second, we introduce a Dynamic Frames Extraction strategy to adapt to
  metamorphic time-lapse videos, which have a wider variation range and cover
  dramatic object metamorphic processes, thus embodying more physical knowledge
  than general videos. Finally, we introduce a Magic Text-Encoder to improve the
  understanding of metamorphic video prompts. Furthermore, we create a time-lapse
  video-text dataset called ChronoMagic, specifically curated to unlock
  the metamorphic video generation ability. Extensive experiments demonstrate the
  superiority and effectiveness of MagicTime for generating high-quality and
  dynamic metamorphic videos, suggesting time-lapse video generation is a
  promising path toward building metamorphic simulators of the physical world.
paper_ai_summary: MagicTime enhances Text-to-Video synthesis by integrating real-world physics knowledge through metamorphic time-lapse videos, using the MagicAdapter, Dynamic Frames Extraction, and Magic Text-Encoder techniques.
paper_ai_keywords: Text-to-Video generation MagicTime metamorphic time-lapse video generation MagicAdapter Dynamic Frames Extraction Magic Text-Encoder ChronoMagic metamorphic simulators