title: HoloTime: Taming Video Diffusion Models for Panoramic 4D Scene
    Generation
summary: The rapid advancement of diffusion models holds the promise of
  revolutionizing the application of VR and AR technologies, which typically
  require scene-level 4D assets for user experience. Nonetheless, existing
  diffusion models predominantly concentrate on modeling static 3D scenes or
  object-level dynamics, constraining their capacity to provide truly immersive
  experiences. To address this issue, we propose HoloTime, a framework that
  integrates video diffusion models to generate panoramic videos from a single
  prompt or reference image, along with a 360-degree 4D scene reconstruction
  method that seamlessly transforms the generated panoramic video into 4D assets,
  enabling a fully immersive 4D experience for users. Specifically, to tame video
  diffusion models for generating high-fidelity panoramic videos, we introduce
  the 360World dataset, the first comprehensive collection of panoramic videos
  suitable for downstream 4D scene reconstruction tasks. With this curated
  dataset, we propose Panoramic Animator, a two-stage image-to-video diffusion
  model that can convert panoramic images into high-quality panoramic videos.
  Following this, we present Panoramic Space-Time Reconstruction, which leverages
  a space-time depth estimation method to transform the generated panoramic
  videos into 4D point clouds, enabling the optimization of a holistic 4D
  Gaussian Splatting representation to reconstruct spatially and temporally
  consistent 4D scenes. To validate the efficacy of our method, we conducted a
  comparative analysis with existing approaches, revealing its superiority in
  both panoramic video generation and 4D scene reconstruction. This demonstrates
  our method's capability to create more engaging and realistic immersive
  environments, thereby enhancing user experiences in VR and AR applications.
paper_authors._id: 681b258f23dbf6b59c5fc735
paper_authors.name: Haiyang Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-08T08:59:30.816Z
paper_authors._id: 66eeda3676a8038cb448f11d
paper_authors.avatarUrl: /avatars/8d6e61f4c9354c6720ccaa7be0fe1d9f.svg
paper_authors.fullname: Haiyang Zhou
paper_authors.type: user
paper_authors.user: Marblueocean
paper_authors._id: 681b258f23dbf6b59c5fc736
paper_authors.name: Wangbo Yu
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-05-07T09:19:13.250Z
paper_authors._id: 63f095be6309c84d5f48848a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63f095be6309c84d5f48848a/pL2CKi-r-0mMfIGhYSAsm.jpeg
paper_authors.fullname: Wangbo Yu
paper_authors.type: user
paper_authors.user: Drexubery
paper_authors._id: 681b258f23dbf6b59c5fc737
paper_authors.name: Jiawen Guan
paper_authors._id: 681b258f23dbf6b59c5fc738
paper_authors.name: Xinhua Cheng
paper_authors._id: 681b258f23dbf6b59c5fc739
paper_authors.name: Yonghong Tian
paper_authors._id: 681b258f23dbf6b59c5fc73a
paper_authors.name: Li Yuan
paper_title: HoloTime: Taming Video Diffusion Models for Panoramic 4D Scene
    Generation
paper_summary: The rapid advancement of diffusion models holds the promise of
  revolutionizing the application of VR and AR technologies, which typically
  require scene-level 4D assets for user experience. Nonetheless, existing
  diffusion models predominantly concentrate on modeling static 3D scenes or
  object-level dynamics, constraining their capacity to provide truly immersive
  experiences. To address this issue, we propose HoloTime, a framework that
  integrates video diffusion models to generate panoramic videos from a single
  prompt or reference image, along with a 360-degree 4D scene reconstruction
  method that seamlessly transforms the generated panoramic video into 4D assets,
  enabling a fully immersive 4D experience for users. Specifically, to tame video
  diffusion models for generating high-fidelity panoramic videos, we introduce
  the 360World dataset, the first comprehensive collection of panoramic videos
  suitable for downstream 4D scene reconstruction tasks. With this curated
  dataset, we propose Panoramic Animator, a two-stage image-to-video diffusion
  model that can convert panoramic images into high-quality panoramic videos.
  Following this, we present Panoramic Space-Time Reconstruction, which leverages
  a space-time depth estimation method to transform the generated panoramic
  videos into 4D point clouds, enabling the optimization of a holistic 4D
  Gaussian Splatting representation to reconstruct spatially and temporally
  consistent 4D scenes. To validate the efficacy of our method, we conducted a
  comparative analysis with existing approaches, revealing its superiority in
  both panoramic video generation and 4D scene reconstruction. This demonstrates
  our method's capability to create more engaging and realistic immersive
  environments, thereby enhancing user experiences in VR and AR applications.
paper_ai_summary: HoloTime uses a two-stage panoramic diffusion model and a space-time depth estimation method to generate high-fidelity 4D assets for immersive VR and AR experiences.
paper_ai_keywords: diffusion models panoramic videos 4D scene reconstruction 360World dataset Panoramic Animator two-stage image-to-video diffusion model Panoramic Space-Time Reconstruction space-time depth estimation 4D Gaussian Splatting spatially and temporally consistent 4D scenes