title: DimensionX: Create Any 3D and 4D Scenes from a Single Image with
    Controllable Video Diffusion
summary: In this paper, we introduce DimensionX, a framework designed to
  generate photorealistic 3D and 4D scenes from just a single image with video
  diffusion. Our approach begins with the insight that both the spatial structure
  of a 3D scene and the temporal evolution of a 4D scene can be effectively
  represented through sequences of video frames. While recent video diffusion
  models have shown remarkable success in producing vivid visuals, they face
  limitations in directly recovering 3D/4D scenes due to limited spatial and
  temporal controllability during generation. To overcome this, we propose
  ST-Director, which decouples spatial and temporal factors in video diffusion by
  learning dimension-aware LoRAs from dimension-variant data. This controllable
  video diffusion approach enables precise manipulation of spatial structure and
  temporal dynamics, allowing us to reconstruct both 3D and 4D representations
  from sequential frames with the combination of spatial and temporal dimensions.
  Additionally, to bridge the gap between generated videos and real-world scenes,
  we introduce a trajectory-aware mechanism for 3D generation and an
  identity-preserving denoising strategy for 4D generation. Extensive experiments
  on various real-world and synthetic datasets demonstrate that DimensionX
  achieves superior results in controllable video generation, as well as in 3D
  and 4D scene generation, compared with previous methods.
paper_authors._id: 672d8febb3f7da8ed807dc6b
paper_authors.name: Wenqiang Sun
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:29:57.027Z
paper_authors._id: 64897b1f0ec897cfe579a399
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64897b1f0ec897cfe579a399/ICR_75b877BaSE94gjBuj.jpeg
paper_authors.fullname: wenq
paper_authors.type: user
paper_authors.user: wenqsun
paper_authors._id: 672d8febb3f7da8ed807dc6c
paper_authors.name: Shuo Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:29:58.991Z
paper_authors._id: 6556031ab7bad186e86e96e7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/_SjKb8h1N8URnQchqu327.jpeg
paper_authors.fullname: ShuoChen
paper_authors.type: user
paper_authors.user: ShuoChen20
paper_authors._id: 672d8febb3f7da8ed807dc6d
paper_authors.name: Fangfu Liu
paper_authors._id: 672d8febb3f7da8ed807dc6e
paper_authors.name: Zilong Chen
paper_authors._id: 672d8febb3f7da8ed807dc6f
paper_authors.name: Yueqi Duan
paper_authors._id: 672d8febb3f7da8ed807dc70
paper_authors.name: Jun Zhang
paper_authors._id: 672d8febb3f7da8ed807dc71
paper_authors.name: Yikai Wang
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-11-15T17:23:45.274Z
paper_authors._id: 64206b35e40f66bcd1e3c985
paper_authors.avatarUrl: /avatars/a8f830d1c47702f642045399e59c9fe8.svg
paper_authors.fullname: Yikai Wang
paper_authors.type: user
paper_authors.user: yikaiw
paper_title: DimensionX: Create Any 3D and 4D Scenes from a Single Image with
    Controllable Video Diffusion
paper_summary: In this paper, we introduce DimensionX, a framework designed to
  generate photorealistic 3D and 4D scenes from just a single image with video
  diffusion. Our approach begins with the insight that both the spatial structure
  of a 3D scene and the temporal evolution of a 4D scene can be effectively
  represented through sequences of video frames. While recent video diffusion
  models have shown remarkable success in producing vivid visuals, they face
  limitations in directly recovering 3D/4D scenes due to limited spatial and
  temporal controllability during generation. To overcome this, we propose
  ST-Director, which decouples spatial and temporal factors in video diffusion by
  learning dimension-aware LoRAs from dimension-variant data. This controllable
  video diffusion approach enables precise manipulation of spatial structure and
  temporal dynamics, allowing us to reconstruct both 3D and 4D representations
  from sequential frames with the combination of spatial and temporal dimensions.
  Additionally, to bridge the gap between generated videos and real-world scenes,
  we introduce a trajectory-aware mechanism for 3D generation and an
  identity-preserving denoising strategy for 4D generation. Extensive experiments
  on various real-world and synthetic datasets demonstrate that DimensionX
  achieves superior results in controllable video generation, as well as in 3D
  and 4D scene generation, compared with previous methods.
paper_ai_keywords: DimensionX video diffusion photorealistic 3D and 4D scenes spatial structure temporal evolution sequences of video frames dimension-aware LoRAs ST-Director spatial and temporal controllability dimension-variant data trajectory-aware mechanism identity-preserving denoising strategy controllable video generation