title: 3DGStream: On-the-Fly Training of 3D Gaussians for Efficient Streaming
    of Photo-Realistic Free-Viewpoint Videos
summary: Constructing photo-realistic Free-Viewpoint Videos (FVVs) of dynamic scenes
  from multi-view videos remains a challenging endeavor. Despite the remarkable
  advancements achieved by current neural rendering techniques, these methods
  generally require complete video sequences for offline training and are not
  capable of real-time rendering. To address these constraints, we introduce
  3DGStream, a method designed for efficient FVV streaming of real-world dynamic
  scenes. Our method achieves fast on-the-fly per-frame reconstruction within 12
  seconds and real-time rendering at 200 FPS. Specifically, we utilize 3D
  Gaussians (3DGs) to represent the scene. Instead of the na\"ive approach of
  directly optimizing 3DGs per-frame, we employ a compact Neural Transformation
  Cache (NTC) to model the translations and rotations of 3DGs, markedly reducing
  the training time and storage required for each FVV frame. Furthermore, we
  propose an adaptive 3DG addition strategy to handle emerging objects in dynamic
  scenes. Experiments demonstrate that 3DGStream achieves competitive performance
  in terms of rendering speed, image quality, training time, and model storage
  when compared with state-of-the-art methods.
paper_authors._id: 65e74b52bb169c92fb66e8d4
paper_authors.name: Jiakai Sun
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-05T20:40:22.729Z
paper_authors._id: 64744432e0b188d3cb1f6790
paper_authors.avatarUrl: /avatars/01c4d7d4724170daba3df60467713345.svg
paper_authors.fullname: Jac Sun
paper_authors.type: user
paper_authors.user: SJoJoK
paper_authors._id: 65e74b52bb169c92fb66e8d5
paper_authors.name: Han Jiao
paper_authors._id: 65e74b52bb169c92fb66e8d6
paper_authors.name: Guangyuan Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-05T20:58:07.919Z
paper_authors._id: 655198a01e13b38aecdf2db4
paper_authors.avatarUrl: /avatars/f1bced0e2f4cac2c56ca0dff786c762a.svg
paper_authors.fullname: liguangyuan
paper_authors.type: user
paper_authors.user: liguangyuan
paper_authors._id: 65e74b52bb169c92fb66e8d7
paper_authors.name: Zhanjie Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-05T20:58:15.475Z
paper_authors._id: 640496240ab5e22719f31719
paper_authors.avatarUrl: /avatars/be957a1b0e1c037f03f1439d6142e9ce.svg
paper_authors.fullname: Zhanjie Zhang
paper_authors.type: user
paper_authors.user: zhangzhanjay
paper_authors._id: 65e74b52bb169c92fb66e8d8
paper_authors.name: Lei Zhao
paper_authors._id: 65e74b52bb169c92fb66e8d9
paper_authors.name: Wei Xing
paper_title: 3DGStream: On-the-Fly Training of 3D Gaussians for Efficient Streaming
    of Photo-Realistic Free-Viewpoint Videos
paper_summary: Constructing photo-realistic Free-Viewpoint Videos (FVVs) of dynamic scenes
  from multi-view videos remains a challenging endeavor. Despite the remarkable
  advancements achieved by current neural rendering techniques, these methods
  generally require complete video sequences for offline training and are not
  capable of real-time rendering. To address these constraints, we introduce
  3DGStream, a method designed for efficient FVV streaming of real-world dynamic
  scenes. Our method achieves fast on-the-fly per-frame reconstruction within 12
  seconds and real-time rendering at 200 FPS. Specifically, we utilize 3D
  Gaussians (3DGs) to represent the scene. Instead of the na\"ive approach of
  directly optimizing 3DGs per-frame, we employ a compact Neural Transformation
  Cache (NTC) to model the translations and rotations of 3DGs, markedly reducing
  the training time and storage required for each FVV frame. Furthermore, we
  propose an adaptive 3DG addition strategy to handle emerging objects in dynamic
  scenes. Experiments demonstrate that 3DGStream achieves competitive performance
  in terms of rendering speed, image quality, training time, and model storage
  when compared with state-of-the-art methods.
paper_ai_keywords: 3DGStream Free-Viewpoint Videos (FVVs) neural rendering techniques 3D Gaussians (3DGs) Neural Transformation Cache (NTC) adaptive 3DG addition strategy