title: GFlow: Recovering 4D World from Monocular Video
summary: Reconstructing 4D scenes from video inputs is a crucial yet challenging task.
  Conventional methods usually rely on the assumptions of multi-view video
  inputs, known camera parameters, or static scenes, all of which are typically
  absent under in-the-wild scenarios. In this paper, we relax all these
  constraints and tackle a highly ambitious but practical task, which we termed
  as AnyV4D: we assume only one monocular video is available without any camera
  parameters as input, and we aim to recover the dynamic 4D world alongside the
  camera poses. To this end, we introduce GFlow, a new framework that utilizes
  only 2D priors (depth and optical flow) to lift a video (3D) to a 4D explicit
  representation, entailing a flow of Gaussian splatting through space and time.
  GFlow first clusters the scene into still and moving parts, then applies a
  sequential optimization process that optimizes camera poses and the dynamics of
  3D Gaussian points based on 2D priors and scene clustering, ensuring fidelity
  among neighboring points and smooth movement across frames. Since dynamic
  scenes always introduce new content, we also propose a new pixel-wise
  densification strategy for Gaussian points to integrate new visual content.
  Moreover, GFlow transcends the boundaries of mere 4D reconstruction; it also
  enables tracking of any points across frames without the need for prior
  training and segments moving objects from the scene in an unsupervised way.
  Additionally, the camera poses of each frame can be derived from GFlow,
  allowing for rendering novel views of a video scene through changing camera
  pose. By employing the explicit representation, we may readily conduct
  scene-level or object-level editing as desired, underscoring its versatility
  and power. Visit our project website at: https://littlepure2333.github.io/GFlow
paper_authors._id: 665688e07c0be58143d1fcb2
paper_authors.name: Shizun Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-29T08:19:48.514Z
paper_authors._id: 63492af03caab0136df08d25
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63492af03caab0136df08d25/-aZzajKl38jT29A-kziCI.jpeg
paper_authors.fullname: Shizun Wang
paper_authors.type: user
paper_authors.user: littlepure2333
paper_authors._id: 665688e07c0be58143d1fcb3
paper_authors.name: Xingyi Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-29T08:19:54.609Z
paper_authors._id: 634cfebc350bcee9bed20a4d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/634cfebc350bcee9bed20a4d/fN47nN5rhw-HJaFLBZWQy.png
paper_authors.fullname: Xingyi Yang
paper_authors.type: user
paper_authors.user: adamdad
paper_authors._id: 665688e07c0be58143d1fcb4
paper_authors.name: Qiuhong Shen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-29T08:20:01.015Z
paper_authors._id: 643a6e89a856622f9788bf67
paper_authors.avatarUrl: /avatars/419c0379f072295b27d4bfe2f8fb946d.svg
paper_authors.fullname: qiuhong shen
paper_authors.type: user
paper_authors.user: florinshum
paper_authors._id: 665688e07c0be58143d1fcb5
paper_authors.name: Zhenxiang Jiang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-29T08:22:55.969Z
paper_authors._id: 6622415672e99868e9212e45
paper_authors.avatarUrl: /avatars/f22dd522a9e04840533fe735d9b811f3.svg
paper_authors.fullname: Zhenxiang Jiang
paper_authors.type: user
paper_authors.user: royjiang
paper_authors._id: 665688e07c0be58143d1fcb6
paper_authors.name: Xinchao Wang
paper_title: GFlow: Recovering 4D World from Monocular Video
paper_summary: Reconstructing 4D scenes from video inputs is a crucial yet challenging task.
  Conventional methods usually rely on the assumptions of multi-view video
  inputs, known camera parameters, or static scenes, all of which are typically
  absent under in-the-wild scenarios. In this paper, we relax all these
  constraints and tackle a highly ambitious but practical task, which we termed
  as AnyV4D: we assume only one monocular video is available without any camera
  parameters as input, and we aim to recover the dynamic 4D world alongside the
  camera poses. To this end, we introduce GFlow, a new framework that utilizes
  only 2D priors (depth and optical flow) to lift a video (3D) to a 4D explicit
  representation, entailing a flow of Gaussian splatting through space and time.
  GFlow first clusters the scene into still and moving parts, then applies a
  sequential optimization process that optimizes camera poses and the dynamics of
  3D Gaussian points based on 2D priors and scene clustering, ensuring fidelity
  among neighboring points and smooth movement across frames. Since dynamic
  scenes always introduce new content, we also propose a new pixel-wise
  densification strategy for Gaussian points to integrate new visual content.
  Moreover, GFlow transcends the boundaries of mere 4D reconstruction; it also
  enables tracking of any points across frames without the need for prior
  training and segments moving objects from the scene in an unsupervised way.
  Additionally, the camera poses of each frame can be derived from GFlow,
  allowing for rendering novel views of a video scene through changing camera
  pose. By employing the explicit representation, we may readily conduct
  scene-level or object-level editing as desired, underscoring its versatility
  and power. Visit our project website at: https://littlepure2333.github.io/GFlow
paper_ai_summary: GFlow reconstructs 4D dynamic scenes and相机 poses from a single monocular video using Gaussian splatting and 2D priors, enabling novel view rendering and object tracking.
paper_ai_keywords: monocular video 4D reconstruction camera poses Gaussian splatting 2D priors depth optical flow scene clustering sequential optimization pixel-wise densification dynamic scenes object tracking novel view rendering scene editing object-level editing