VideoRFSplat: Direct Scene-Level Text-to-3D Gaussian Splatting
  Generation with Flexible Pose and Multi-View Joint Modeling
We propose VideoRFSplat, a direct text-to-3D model leveraging a video
generation model to generate realistic 3D Gaussian Splatting (3DGS) for
unbounded real-world scenes. To generate diverse camera poses and unbounded
spatial extent of real-world scenes, while ensuring generalization to arbitrary
text prompts, previous methods fine-tune 2D generative models to jointly model
camera poses and multi-view images. However, these methods suffer from
instability when extending 2D generative models to joint modeling due to the
modality gap, which necessitates additional models to stabilize training and
inference. In this work, we propose an architecture and a sampling strategy to
jointly model multi-view images and camera poses when fine-tuning a video
generation model. Our core idea is a dual-stream architecture that attaches a
dedicated pose generation model alongside a pre-trained video generation model
via communication blocks, generating multi-view images and camera poses through
separate streams. This design reduces interference between the pose and image
modalities. Additionally, we propose an asynchronous sampling strategy that
denoises camera poses faster than multi-view images, allowing rapidly denoised
poses to condition multi-view generation, reducing mutual ambiguity and
enhancing cross-modal consistency. Trained on multiple large-scale real-world
datasets (RealEstate10K, MVImgNet, DL3DV-10K, ACID), VideoRFSplat outperforms
existing text-to-3D direct generation methods that heavily depend on post-hoc
refinement via score distillation sampling, achieving superior results without
such refinement.
67dd072f1c182b6168eaa004
Hyojun Go
admin_assigned
2025-03-21T13:10:51.488Z
649f65a4ca03a1a35e3dac14
/avatars/b0dcd8ad795b1e666ee247b2ac024d53.svg
Hyojun GO
user
HJGO
67dd072f1c182b6168eaa005
Byeongjun Park
admin_assigned
2025-03-21T13:10:58.350Z
653929a66da48e0d21e65e17
/avatars/e34ae3411d689b4280ff34c1b680f283.svg
Byeongjun Park
user
byeongjun-park
67dd072f1c182b6168eaa006
Hyelin Nam
admin_assigned
2025-03-21T13:11:04.355Z
668e2d6bf7c08c3f9e837b1e
/avatars/ad769e4b5af78e6da8a1090b379932c9.svg
hyelin nam
user
hyelinnam
67dd072f1c182b6168eaa007
Byung-Hoon Kim
67dd072f1c182b6168eaa008
Hyungjin Chung
admin_assigned
2025-03-21T13:11:13.503Z
62b5e7543797f21de0bd00db
/avatars/8cd3153fcbe340abbd3b795b58f70957.svg
Hyungjin Chung
user
95harry
67dd072f1c182b6168eaa009
Changick Kim
admin_assigned
2025-03-21T13:11:19.337Z
676e5f07aa6895d518748e52
/avatars/11e30698017dd98eedff3274c9cc5f19.svg
Changick Kim
user
CK8308
VideoRFSplat: Direct Scene-Level Text-to-3D Gaussian Splatting
  Generation with Flexible Pose and Multi-View Joint Modeling
We propose VideoRFSplat, a direct text-to-3D model leveraging a video
generation model to generate realistic 3D Gaussian Splatting (3DGS) for
unbounded real-world scenes. To generate diverse camera poses and unbounded
spatial extent of real-world scenes, while ensuring generalization to arbitrary
text prompts, previous methods fine-tune 2D generative models to jointly model
camera poses and multi-view images. However, these methods suffer from
instability when extending 2D generative models to joint modeling due to the
modality gap, which necessitates additional models to stabilize training and
inference. In this work, we propose an architecture and a sampling strategy to
jointly model multi-view images and camera poses when fine-tuning a video
generation model. Our core idea is a dual-stream architecture that attaches a
dedicated pose generation model alongside a pre-trained video generation model
via communication blocks, generating multi-view images and camera poses through
separate streams. This design reduces interference between the pose and image
modalities. Additionally, we propose an asynchronous sampling strategy that
denoises camera poses faster than multi-view images, allowing rapidly denoised
poses to condition multi-view generation, reducing mutual ambiguity and
enhancing cross-modal consistency. Trained on multiple large-scale real-world
datasets (RealEstate10K, MVImgNet, DL3DV-10K, ACID), VideoRFSplat outperforms
existing text-to-3D direct generation methods that heavily depend on post-hoc
refinement via score distillation sampling, achieving superior results without
such refinement.
VideoRFSplat, a text-to-3D model, uses a video generation model to produce realistic 3D Gaussian Splatting, employing a dual-stream architecture for joint modeling of camera poses and multi-view images, and an asynchronous sampling strategy to enhance cross-modal consistency.
3D Gaussian Splatting dual-stream architecture camera poses multi-view images asynchronous sampling strategy score distillation sampling