title: VideoRFSplat: Direct Scene-Level Text-to-3D Gaussian Splatting
    Generation with Flexible Pose and Multi-View Joint Modeling
summary: We propose VideoRFSplat, a direct text-to-3D model leveraging a video
  generation model to generate realistic 3D Gaussian Splatting (3DGS) for
  unbounded real-world scenes. To generate diverse camera poses and unbounded
  spatial extent of real-world scenes, while ensuring generalization to arbitrary
  text prompts, previous methods fine-tune 2D generative models to jointly model
  camera poses and multi-view images. However, these methods suffer from
  instability when extending 2D generative models to joint modeling due to the
  modality gap, which necessitates additional models to stabilize training and
  inference. In this work, we propose an architecture and a sampling strategy to
  jointly model multi-view images and camera poses when fine-tuning a video
  generation model. Our core idea is a dual-stream architecture that attaches a
  dedicated pose generation model alongside a pre-trained video generation model
  via communication blocks, generating multi-view images and camera poses through
  separate streams. This design reduces interference between the pose and image
  modalities. Additionally, we propose an asynchronous sampling strategy that
  denoises camera poses faster than multi-view images, allowing rapidly denoised
  poses to condition multi-view generation, reducing mutual ambiguity and
  enhancing cross-modal consistency. Trained on multiple large-scale real-world
  datasets (RealEstate10K, MVImgNet, DL3DV-10K, ACID), VideoRFSplat outperforms
  existing text-to-3D direct generation methods that heavily depend on post-hoc
  refinement via score distillation sampling, achieving superior results without
  such refinement.
paper_authors._id: 67dd072f1c182b6168eaa004
paper_authors.name: Hyojun Go
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-21T13:10:51.488Z
paper_authors._id: 649f65a4ca03a1a35e3dac14
paper_authors.avatarUrl: /avatars/b0dcd8ad795b1e666ee247b2ac024d53.svg
paper_authors.fullname: Hyojun GO
paper_authors.type: user
paper_authors.user: HJGO
paper_authors._id: 67dd072f1c182b6168eaa005
paper_authors.name: Byeongjun Park
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-21T13:10:58.350Z
paper_authors._id: 653929a66da48e0d21e65e17
paper_authors.avatarUrl: /avatars/e34ae3411d689b4280ff34c1b680f283.svg
paper_authors.fullname: Byeongjun Park
paper_authors.type: user
paper_authors.user: byeongjun-park
paper_authors._id: 67dd072f1c182b6168eaa006
paper_authors.name: Hyelin Nam
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-21T13:11:04.355Z
paper_authors._id: 668e2d6bf7c08c3f9e837b1e
paper_authors.avatarUrl: /avatars/ad769e4b5af78e6da8a1090b379932c9.svg
paper_authors.fullname: hyelin nam
paper_authors.type: user
paper_authors.user: hyelinnam
paper_authors._id: 67dd072f1c182b6168eaa007
paper_authors.name: Byung-Hoon Kim
paper_authors._id: 67dd072f1c182b6168eaa008
paper_authors.name: Hyungjin Chung
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-21T13:11:13.503Z
paper_authors._id: 62b5e7543797f21de0bd00db
paper_authors.avatarUrl: /avatars/8cd3153fcbe340abbd3b795b58f70957.svg
paper_authors.fullname: Hyungjin Chung
paper_authors.type: user
paper_authors.user: 95harry
paper_authors._id: 67dd072f1c182b6168eaa009
paper_authors.name: Changick Kim
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-21T13:11:19.337Z
paper_authors._id: 676e5f07aa6895d518748e52
paper_authors.avatarUrl: /avatars/11e30698017dd98eedff3274c9cc5f19.svg
paper_authors.fullname: Changick Kim
paper_authors.type: user
paper_authors.user: CK8308
paper_title: VideoRFSplat: Direct Scene-Level Text-to-3D Gaussian Splatting
    Generation with Flexible Pose and Multi-View Joint Modeling
paper_summary: We propose VideoRFSplat, a direct text-to-3D model leveraging a video
  generation model to generate realistic 3D Gaussian Splatting (3DGS) for
  unbounded real-world scenes. To generate diverse camera poses and unbounded
  spatial extent of real-world scenes, while ensuring generalization to arbitrary
  text prompts, previous methods fine-tune 2D generative models to jointly model
  camera poses and multi-view images. However, these methods suffer from
  instability when extending 2D generative models to joint modeling due to the
  modality gap, which necessitates additional models to stabilize training and
  inference. In this work, we propose an architecture and a sampling strategy to
  jointly model multi-view images and camera poses when fine-tuning a video
  generation model. Our core idea is a dual-stream architecture that attaches a
  dedicated pose generation model alongside a pre-trained video generation model
  via communication blocks, generating multi-view images and camera poses through
  separate streams. This design reduces interference between the pose and image
  modalities. Additionally, we propose an asynchronous sampling strategy that
  denoises camera poses faster than multi-view images, allowing rapidly denoised
  poses to condition multi-view generation, reducing mutual ambiguity and
  enhancing cross-modal consistency. Trained on multiple large-scale real-world
  datasets (RealEstate10K, MVImgNet, DL3DV-10K, ACID), VideoRFSplat outperforms
  existing text-to-3D direct generation methods that heavily depend on post-hoc
  refinement via score distillation sampling, achieving superior results without
  such refinement.
paper_ai_summary: VideoRFSplat, a text-to-3D model, uses a video generation model to produce realistic 3D Gaussian Splatting, employing a dual-stream architecture for joint modeling of camera poses and multi-view images, and an asynchronous sampling strategy to enhance cross-modal consistency.
paper_ai_keywords: 3D Gaussian Splatting dual-stream architecture camera poses multi-view images asynchronous sampling strategy score distillation sampling