title: GS-DiT: Advancing Video Generation with Pseudo 4D Gaussian Fields
    through Efficient Dense 3D Point Tracking
summary: 4D video control is essential in video generation as it enables the use of
  sophisticated lens techniques, such as multi-camera shooting and dolly zoom,
  which are currently unsupported by existing methods. Training a video Diffusion
  Transformer (DiT) directly to control 4D content requires expensive multi-view
  videos. Inspired by Monocular Dynamic novel View Synthesis (MDVS) that
  optimizes a 4D representation and renders videos according to different 4D
  elements, such as camera pose and object motion editing, we bring pseudo 4D
  Gaussian fields to video generation. Specifically, we propose a novel framework
  that constructs a pseudo 4D Gaussian field with dense 3D point tracking and
  renders the Gaussian field for all video frames. Then we finetune a pretrained
  DiT to generate videos following the guidance of the rendered video, dubbed as
  GS-DiT. To boost the training of the GS-DiT, we also propose an efficient Dense
  3D Point Tracking (D3D-PT) method for the pseudo 4D Gaussian field
  construction. Our D3D-PT outperforms SpatialTracker, the state-of-the-art
  sparse 3D point tracking method, in accuracy and accelerates the inference
  speed by two orders of magnitude. During the inference stage, GS-DiT can
  generate videos with the same dynamic content while adhering to different
  camera parameters, addressing a significant limitation of current video
  generation models. GS-DiT demonstrates strong generalization capabilities and
  extends the 4D controllability of Gaussian splatting to video generation beyond
  just camera poses. It supports advanced cinematic effects through the
  manipulation of the Gaussian field and camera intrinsics, making it a powerful
  tool for creative video production. Demos are available at
  https://wkbian.github.io/Projects/GS-DiT/.
paper_authors._id: 677cbd8ef7134ae168acc1fc
paper_authors.name: Weikang Bian
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-07T13:01:59.093Z
paper_authors._id: 65b8b840a81930c593073370
paper_authors.avatarUrl: /avatars/31b399d3da33e8c02f9e06a16dd85779.svg
paper_authors.fullname: Weikang BIAN
paper_authors.type: user
paper_authors.user: wkbian
paper_authors._id: 677cbd8ef7134ae168acc1fd
paper_authors.name: Zhaoyang Huang
paper_authors._id: 677cbd8ef7134ae168acc1fe
paper_authors.name: Xiaoyu Shi
paper_authors._id: 677cbd8ef7134ae168acc1ff
paper_authors.name: Yijin Li
paper_authors._id: 677cbd8ef7134ae168acc200
paper_authors.name: Fu-Yun Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-07T08:41:34.222Z
paper_authors._id: 63e9e92f20c109718713f5eb
paper_authors.avatarUrl: /avatars/9ff312e854d803e1a2e9e685a21d12f8.svg
paper_authors.fullname: Fu-Yun Wang
paper_authors.type: user
paper_authors.user: wangfuyun
paper_authors._id: 677cbd8ef7134ae168acc201
paper_authors.name: Hongsheng Li
paper_title: GS-DiT: Advancing Video Generation with Pseudo 4D Gaussian Fields
    through Efficient Dense 3D Point Tracking
paper_summary: 4D video control is essential in video generation as it enables the use of
  sophisticated lens techniques, such as multi-camera shooting and dolly zoom,
  which are currently unsupported by existing methods. Training a video Diffusion
  Transformer (DiT) directly to control 4D content requires expensive multi-view
  videos. Inspired by Monocular Dynamic novel View Synthesis (MDVS) that
  optimizes a 4D representation and renders videos according to different 4D
  elements, such as camera pose and object motion editing, we bring pseudo 4D
  Gaussian fields to video generation. Specifically, we propose a novel framework
  that constructs a pseudo 4D Gaussian field with dense 3D point tracking and
  renders the Gaussian field for all video frames. Then we finetune a pretrained
  DiT to generate videos following the guidance of the rendered video, dubbed as
  GS-DiT. To boost the training of the GS-DiT, we also propose an efficient Dense
  3D Point Tracking (D3D-PT) method for the pseudo 4D Gaussian field
  construction. Our D3D-PT outperforms SpatialTracker, the state-of-the-art
  sparse 3D point tracking method, in accuracy and accelerates the inference
  speed by two orders of magnitude. During the inference stage, GS-DiT can
  generate videos with the same dynamic content while adhering to different
  camera parameters, addressing a significant limitation of current video
  generation models. GS-DiT demonstrates strong generalization capabilities and
  extends the 4D controllability of Gaussian splatting to video generation beyond
  just camera poses. It supports advanced cinematic effects through the
  manipulation of the Gaussian field and camera intrinsics, making it a powerful
  tool for creative video production. Demos are available at
  https://wkbian.github.io/Projects/GS-DiT/.
paper_ai_summary: A novel framework combining pseudo 4D Gaussian fields with a video Diffusion Transformer (DiT) allows for more efficient and flexible 4D video generation, supporting advanced cinematic effects and different camera parameters.
paper_ai_keywords: Diffusion Transformer (DiT) pseudo 4D Gaussian fields dense 3D point tracking efficient Dense 3D Point Tracking (D3D-PT) Gaussian splatting 4D video control camera poses camera intrinsics video generation cinematic effects dynamic content