arxiv:2507.23785

Gaussian Variation Field Diffusion for High-fidelity Video-to-4D Synthesis

Published on Jul 31

· Submitted by

BwZhang on Aug 7

Upvote

Authors:

Bowen Zhang ,

Abstract

A novel framework uses a Direct 4DMesh-to-GS Variation Field VAE and Gaussian Variation Field diffusion model to generate high-quality dynamic 3D content from single video inputs, demonstrating superior quality and generalization.

AI-generated summary

In this paper, we present a novel framework for video-to-4D generation that creates high-quality dynamic 3D content from single video inputs. Direct 4D diffusion modeling is extremely challenging due to costly data construction and the high-dimensional nature of jointly representing 3D shape, appearance, and motion. We address these challenges by introducing a Direct 4DMesh-to-GS Variation Field VAE that directly encodes canonical Gaussian Splats (GS) and their temporal variations from 3D animation data without per-instance fitting, and compresses high-dimensional animations into a compact latent space. Building upon this efficient representation, we train a Gaussian Variation Field diffusion model with temporal-aware Diffusion Transformer conditioned on input videos and canonical GS. Trained on carefully-curated animatable 3D objects from the Objaverse dataset, our model demonstrates superior generation quality compared to existing methods. It also exhibits remarkable generalization to in-the-wild video inputs despite being trained exclusively on synthetic data, paving the way for generating high-quality animated 3D content. Project page: https://gvfdiffusion.github.io/.

View arXiv page View PDF Project page GitHub 62 Add to collection

Community

BwZhang

Paper author Paper submitter 2 days ago

In this paper, we introduce a novel framework to address the challenging task of 4D generative modeling. To efficiently construct the large-scale training dataset and reduce the modeling difficulty for diffusion, we first introduce a Direct 4DMesh-to-GS Variation Field VAE, which is able to efficiently compress complex motion information into a compact latent space without requiring costly per-instance fitting. Then, a Gaussian Variation Field diffusion model that generates high-quality dynamic variation fields conditioned on input videos and canonical 3DGS. By decomposing 4D generation into canonical 3DGS generation and Gaussian Variation Field modeling, our method significantly reduces computational complexity while maintaining high fidelity. Quantitative and qualitative evaluations demonstrate that our approach consistently outperforms existing methods. Furthermore, our model exhibits remarkable generalization capability with in-the-wild video inputs, advancing the state of high-quality animated 3D content generation.