title: Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed
    Diffusion Models
summary: Text-guided diffusion models have revolutionized image and video generation
  and have also been successfully used for optimization-based 3D object
  synthesis. Here, we instead focus on the underexplored text-to-4D setting and
  synthesize dynamic, animated 3D objects using score distillation methods with
  an additional temporal dimension. Compared to previous work, we pursue a novel
  compositional generation-based approach, and combine text-to-image,
  text-to-video, and 3D-aware multiview diffusion models to provide feedback
  during 4D object optimization, thereby simultaneously enforcing temporal
  consistency, high-quality visual appearance and realistic geometry. Our method,
  called Align Your Gaussians (AYG), leverages dynamic 3D Gaussian Splatting with
  deformation fields as 4D representation. Crucial to AYG is a novel method to
  regularize the distribution of the moving 3D Gaussians and thereby stabilize
  the optimization and induce motion. We also propose a motion amplification
  mechanism as well as a new autoregressive synthesis scheme to generate and
  combine multiple 4D sequences for longer generation. These techniques allow us
  to synthesize vivid dynamic scenes, outperform previous work qualitatively and
  quantitatively and achieve state-of-the-art text-to-4D performance. Due to the
  Gaussian 4D representation, different 4D animations can be seamlessly combined,
  as we demonstrate. AYG opens up promising avenues for animation, simulation and
  digital content creation as well as synthetic data generation.
paper_authors._id: 6584e8d7ebd17631e9921c6e
paper_authors.name: Huan Ling
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-22T12:02:21.125Z
paper_authors._id: 6344bf36762379fc6302abed
paper_authors.avatarUrl: /avatars/f4442fef316ac96652d4f99238822209.svg
paper_authors.fullname: huanling
paper_authors.type: user
paper_authors.user: huanling
paper_authors._id: 6584e8d7ebd17631e9921c6f
paper_authors.name: Seung Wook Kim
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-22T12:02:28.075Z
paper_authors._id: 64fb5827d82fc6977d70582d
paper_authors.avatarUrl: /avatars/54180874d2f162572c414970b8750759.svg
paper_authors.fullname: Seung Wook Kim
paper_authors.type: user
paper_authors.user: swkimbd
paper_authors._id: 6584e8d7ebd17631e9921c70
paper_authors.name: Antonio Torralba
paper_authors._id: 6584e8d7ebd17631e9921c71
paper_authors.name: Sanja Fidler
paper_authors._id: 6584e8d7ebd17631e9921c72
paper_authors.name: Karsten Kreis
paper_title: Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed
    Diffusion Models
paper_summary: Text-guided diffusion models have revolutionized image and video generation
  and have also been successfully used for optimization-based 3D object
  synthesis. Here, we instead focus on the underexplored text-to-4D setting and
  synthesize dynamic, animated 3D objects using score distillation methods with
  an additional temporal dimension. Compared to previous work, we pursue a novel
  compositional generation-based approach, and combine text-to-image,
  text-to-video, and 3D-aware multiview diffusion models to provide feedback
  during 4D object optimization, thereby simultaneously enforcing temporal
  consistency, high-quality visual appearance and realistic geometry. Our method,
  called Align Your Gaussians (AYG), leverages dynamic 3D Gaussian Splatting with
  deformation fields as 4D representation. Crucial to AYG is a novel method to
  regularize the distribution of the moving 3D Gaussians and thereby stabilize
  the optimization and induce motion. We also propose a motion amplification
  mechanism as well as a new autoregressive synthesis scheme to generate and
  combine multiple 4D sequences for longer generation. These techniques allow us
  to synthesize vivid dynamic scenes, outperform previous work qualitatively and
  quantitatively and achieve state-of-the-art text-to-4D performance. Due to the
  Gaussian 4D representation, different 4D animations can be seamlessly combined,
  as we demonstrate. AYG opens up promising avenues for animation, simulation and
  digital content creation as well as synthetic data generation.
paper_ai_keywords: diffusion models text-to-4D score distillation compositional generation text-to-image text-to-video 3D-aware multiview diffusion models temporal consistency high-quality visual appearance realistic geometry dynamic 3D Gaussian Splatting deformation fields 4D representation regularize the distribution moving 3D Gaussians motion amplification mechanism autoregressive synthesis Scheme state-of-the-art text-to-4D animations seamless combination synthetic data generation