title: 4K4DGen: Panoramic 4D Generation at 4K Resolution
summary: The blooming of virtual reality and augmented reality (VR/AR) technologies
  has driven an increasing demand for the creation of high-quality, immersive,
  and dynamic environments. However, existing generative techniques either focus
  solely on dynamic objects or perform outpainting from a single perspective
  image, failing to meet the needs of VR/AR applications. In this work, we tackle
  the challenging task of elevating a single panorama to an immersive 4D
  experience. For the first time, we demonstrate the capability to generate
  omnidirectional dynamic scenes with 360-degree views at 4K resolution, thereby
  providing an immersive user experience. Our method introduces a pipeline that
  facilitates natural scene animations and optimizes a set of 4D Gaussians using
  efficient splatting techniques for real-time exploration. To overcome the lack
  of scene-scale annotated 4D data and models, especially in panoramic formats,
  we propose a novel Panoramic Denoiser that adapts generic 2D diffusion priors
  to animate consistently in 360-degree images, transforming them into panoramic
  videos with dynamic scenes at targeted regions. Subsequently, we elevate the
  panoramic video into a 4D immersive environment while preserving spatial and
  temporal consistency. By transferring prior knowledge from 2D models in the
  perspective domain to the panoramic domain and the 4D lifting with spatial
  appearance and geometry regularization, we achieve high-quality Panorama-to-4D
  generation at a resolution of (4096 times 2048) for the first time. See the
  project website at https://4k4dgen.github.io.
paper_authors._id: 6676dd829329dbc7eaf2a41b
paper_authors.name: Renjie Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-24T07:29:57.080Z
paper_authors._id: 66269d9849b603e2487c244d
paper_authors.avatarUrl: /avatars/cd782c35417cd71b94c37b3216dfc3ea.svg
paper_authors.fullname: rj.lee
paper_authors.type: user
paper_authors.user: Rj-Lee
paper_authors._id: 6676dd829329dbc7eaf2a41c
paper_authors.name: Panwang Pan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-07T09:59:07.484Z
paper_authors._id: 654866e8cd0a5621395f8287
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/654866e8cd0a5621395f8287/4Bccwd1ehn-Ee4T1rId5S.jpeg
paper_authors.fullname: Panwang Pan
paper_authors.type: user
paper_authors.user: paulpanwang
paper_authors._id: 6676dd829329dbc7eaf2a41d
paper_authors.name: Bangbang Yang
paper_authors._id: 6676dd829329dbc7eaf2a41e
paper_authors.name: Dejia Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-15T09:11:49.057Z
paper_authors._id: 62f687a0c58915315c4ff75d
paper_authors.avatarUrl: /avatars/b657180c7666735062782edd4f6a69c9.svg
paper_authors.fullname: Dejia Xu
paper_authors.type: user
paper_authors.user: ir1d
paper_authors._id: 6676dd829329dbc7eaf2a41f
paper_authors.name: Shijie Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-04T09:40:09.825Z
paper_authors._id: 642a276516d4d8293c9a47e8
paper_authors.avatarUrl: /avatars/80e6db8bc2544f3486b11b57858a8692.svg
paper_authors.fullname: Shijie Zhou
paper_authors.type: user
paper_authors.user: shijiezhou
paper_authors._id: 6676dd829329dbc7eaf2a420
paper_authors.name: Xuanyang Zhang
paper_authors._id: 6676dd829329dbc7eaf2a421
paper_authors.name: Zeming Li
paper_authors._id: 6676dd829329dbc7eaf2a422
paper_authors.name: Achuta Kadambi
paper_authors._id: 6676dd829329dbc7eaf2a423
paper_authors.name: Zhangyang Wang
paper_authors._id: 6676dd829329dbc7eaf2a424
paper_authors.name: Zhiwen Fan
paper_title: 4K4DGen: Panoramic 4D Generation at 4K Resolution
paper_summary: The blooming of virtual reality and augmented reality (VR/AR) technologies
  has driven an increasing demand for the creation of high-quality, immersive,
  and dynamic environments. However, existing generative techniques either focus
  solely on dynamic objects or perform outpainting from a single perspective
  image, failing to meet the needs of VR/AR applications. In this work, we tackle
  the challenging task of elevating a single panorama to an immersive 4D
  experience. For the first time, we demonstrate the capability to generate
  omnidirectional dynamic scenes with 360-degree views at 4K resolution, thereby
  providing an immersive user experience. Our method introduces a pipeline that
  facilitates natural scene animations and optimizes a set of 4D Gaussians using
  efficient splatting techniques for real-time exploration. To overcome the lack
  of scene-scale annotated 4D data and models, especially in panoramic formats,
  we propose a novel Panoramic Denoiser that adapts generic 2D diffusion priors
  to animate consistently in 360-degree images, transforming them into panoramic
  videos with dynamic scenes at targeted regions. Subsequently, we elevate the
  panoramic video into a 4D immersive environment while preserving spatial and
  temporal consistency. By transferring prior knowledge from 2D models in the
  perspective domain to the panoramic domain and the 4D lifting with spatial
  appearance and geometry regularization, we achieve high-quality Panorama-to-4D
  generation at a resolution of (4096 times 2048) for the first time. See the
  project website at https://4k4dgen.github.io.
paper_ai_keywords: diffusion priors splatting techniques panoramic denoiser 4D Gaussians 4D generation spatial consistency temporal consistency panoramic videos 4D lifting spatial appearance regularization geometry regularization