title: 4Real-Video: Learning Generalizable Photo-Realistic 4D Video Diffusion
summary: We propose 4Real-Video, a novel framework for generating 4D videos, organized
  as a grid of video frames with both time and viewpoint axes. In this grid, each
  row contains frames sharing the same timestep, while each column contains
  frames from the same viewpoint. We propose a novel two-stream architecture. One
  stream performs viewpoint updates on columns, and the other stream performs
  temporal updates on rows. After each diffusion transformer layer, a
  synchronization layer exchanges information between the two token streams. We
  propose two implementations of the synchronization layer, using either hard or
  soft synchronization. This feedforward architecture improves upon previous work
  in three ways: higher inference speed, enhanced visual quality (measured by
  FVD, CLIP, and VideoScore), and improved temporal and viewpoint consistency
  (measured by VideoScore and Dust3R-Confidence).
paper_authors._id: 67534514839eb7c6d8875893
paper_authors.name: Chaoyang Wang
paper_authors._id: 67534514839eb7c6d8875894
paper_authors.name: Peiye Zhuang
paper_authors._id: 67534514839eb7c6d8875895
paper_authors.name: Tuan Duc Ngo
paper_authors._id: 67534514839eb7c6d8875896
paper_authors.name: Willi Menapace
paper_authors._id: 67534514839eb7c6d8875897
paper_authors.name: Aliaksandr Siarohin
paper_authors._id: 67534514839eb7c6d8875898
paper_authors.name: Michael Vasilkovsky
paper_authors._id: 67534514839eb7c6d8875899
paper_authors.name: Ivan Skorokhodov
paper_authors._id: 67534514839eb7c6d887589a
paper_authors.name: Sergey Tulyakov
paper_authors._id: 67534514839eb7c6d887589b
paper_authors.name: Peter Wonka
paper_authors._id: 67534514839eb7c6d887589c
paper_authors.name: Hsin-Ying Lee
paper_title: 4Real-Video: Learning Generalizable Photo-Realistic 4D Video Diffusion
paper_summary: We propose 4Real-Video, a novel framework for generating 4D videos, organized
  as a grid of video frames with both time and viewpoint axes. In this grid, each
  row contains frames sharing the same timestep, while each column contains
  frames from the same viewpoint. We propose a novel two-stream architecture. One
  stream performs viewpoint updates on columns, and the other stream performs
  temporal updates on rows. After each diffusion transformer layer, a
  synchronization layer exchanges information between the two token streams. We
  propose two implementations of the synchronization layer, using either hard or
  soft synchronization. This feedforward architecture improves upon previous work
  in three ways: higher inference speed, enhanced visual quality (measured by
  FVD, CLIP, and VideoScore), and improved temporal and viewpoint consistency
  (measured by VideoScore and Dust3R-Confidence).
paper_ai_summary: A two-stream framework for generating 4D videos improves inference speed and visual quality with synchronization layers in diffusion transformer architecture.
paper_ai_keywords: two-stream architecture viewpoint updates temporal updates diffusion transformer synchronization layer hard synchronization soft synchronization FVD CLIP VideoScore Dust3R-Confidence