title: 4Real: Towards Photorealistic 4D Scene Generation via Video Diffusion
    Models
summary: Existing dynamic scene generation methods mostly rely on distilling knowledge
  from pre-trained 3D generative models, which are typically fine-tuned on
  synthetic object datasets. As a result, the generated scenes are often
  object-centric and lack photorealism. To address these limitations, we
  introduce a novel pipeline designed for photorealistic text-to-4D scene
  generation, discarding the dependency on multi-view generative models and
  instead fully utilizing video generative models trained on diverse real-world
  datasets. Our method begins by generating a reference video using the video
  generation model. We then learn the canonical 3D representation of the video
  using a freeze-time video, delicately generated from the reference video. To
  handle inconsistencies in the freeze-time video, we jointly learn a per-frame
  deformation to model these imperfections. We then learn the temporal
  deformation based on the canonical representation to capture dynamic
  interactions in the reference video. The pipeline facilitates the generation of
  dynamic scenes with enhanced photorealism and structural integrity, viewable
  from multiple perspectives, thereby setting a new standard in 4D scene
  generation.
paper_authors._id: 6669056f61588d66710d13ba
paper_authors.name: Heng Yu
paper_authors._id: 6669056f61588d66710d13bb
paper_authors.name: Chaoyang Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-12T08:17:01.607Z
paper_authors._id: 64c166de11f3f1d23a99c6fd
paper_authors.avatarUrl: /avatars/d4730f70a6f1cb51266862c7a8d54f77.svg
paper_authors.fullname: Chaoyang Wang
paper_authors.type: user
paper_authors.user: cwang9
paper_authors._id: 6669056f61588d66710d13bc
paper_authors.name: Peiye Zhuang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-12T08:17:07.735Z
paper_authors._id: 63e473cf9db5da2dc1f4e9ff
paper_authors.avatarUrl: /avatars/6973b4c959ef8bb72c40c611d384a78a.svg
paper_authors.fullname: peiye zhuang
paper_authors.type: user
paper_authors.user: Kelest
paper_authors._id: 6669056f61588d66710d13bd
paper_authors.name: Willi Menapace
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-12T08:17:14.379Z
paper_authors._id: 6315358a362e3e95ea538081
paper_authors.avatarUrl: /avatars/3b089a25a87c2e83c6b23ccb5d2dc73e.svg
paper_authors.fullname: Willi Menapace
paper_authors.type: user
paper_authors.user: willi-menapace
paper_authors._id: 6669056f61588d66710d13be
paper_authors.name: Aliaksandr Siarohin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-12T08:17:23.361Z
paper_authors._id: 64276311eb9a0ed86180715b
paper_authors.avatarUrl: /avatars/76f933cd549f10e5e2db379de235d304.svg
paper_authors.fullname: Aliaksandr Siarohin
paper_authors.type: user
paper_authors.user: aliaksandr-siarohin
paper_authors._id: 6669056f61588d66710d13bf
paper_authors.name: Junli Cao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-12T08:17:29.415Z
paper_authors._id: 63f54aa73aa49d8cb97b84bc
paper_authors.avatarUrl: /avatars/c73c5870039611ab9162daad46a1ba20.svg
paper_authors.fullname: junli cao
paper_authors.type: user
paper_authors.user: jlcao2
paper_authors._id: 6669056f61588d66710d13c0
paper_authors.name: Laszlo A Jeni
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-12T08:17:39.866Z
paper_authors._id: 64bbd2f865b648b2df859bc6
paper_authors.avatarUrl: /avatars/698d98d8b42c478a3d7584b93c6dbcc2.svg
paper_authors.fullname: Laszlo Jeni
paper_authors.type: user
paper_authors.user: laszlo-ai
paper_authors._id: 6669056f61588d66710d13c1
paper_authors.name: Sergey Tulyakov
paper_authors._id: 6669056f61588d66710d13c2
paper_authors.name: Hsin-Ying Lee
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-12T08:17:54.621Z
paper_authors._id: 631bbd918de8e645af71eed9
paper_authors.avatarUrl: /avatars/1cb94deadc288d07c571c9289085548c.svg
paper_authors.fullname: Hsin-Ying Lee
paper_authors.type: user
paper_authors.user: james371507
paper_title: 4Real: Towards Photorealistic 4D Scene Generation via Video Diffusion
    Models
paper_summary: Existing dynamic scene generation methods mostly rely on distilling knowledge
  from pre-trained 3D generative models, which are typically fine-tuned on
  synthetic object datasets. As a result, the generated scenes are often
  object-centric and lack photorealism. To address these limitations, we
  introduce a novel pipeline designed for photorealistic text-to-4D scene
  generation, discarding the dependency on multi-view generative models and
  instead fully utilizing video generative models trained on diverse real-world
  datasets. Our method begins by generating a reference video using the video
  generation model. We then learn the canonical 3D representation of the video
  using a freeze-time video, delicately generated from the reference video. To
  handle inconsistencies in the freeze-time video, we jointly learn a per-frame
  deformation to model these imperfections. We then learn the temporal
  deformation based on the canonical representation to capture dynamic
  interactions in the reference video. The pipeline facilitates the generation of
  dynamic scenes with enhanced photorealism and structural integrity, viewable
  from multiple perspectives, thereby setting a new standard in 4D scene
  generation.
paper_ai_summary: A novel pipeline for photorealistic text-to-4D scene generation uses video generative models and learns canonical 3D and temporal deformations to enhance photorealism and structural integrity.
paper_ai_keywords: 3D generative models video generative models freeze-time video per-frame deformation temporal deformation 4D scene generation