title: GenXD: Generating Any 3D and 4D Scenes
summary: Recent developments in 2D visual generation have been remarkably successful.
  However, 3D and 4D generation remain challenging in real-world applications due
  to the lack of large-scale 4D data and effective model design. In this paper,
  we propose to jointly investigate general 3D and 4D generation by leveraging
  camera and object movements commonly observed in daily life. Due to the lack of
  real-world 4D data in the community, we first propose a data curation pipeline
  to obtain camera poses and object motion strength from videos. Based on this
  pipeline, we introduce a large-scale real-world 4D scene dataset: CamVid-30K.
  By leveraging all the 3D and 4D data, we develop our framework, GenXD, which
  allows us to produce any 3D or 4D scene. We propose multiview-temporal modules,
  which disentangle camera and object movements, to seamlessly learn from both 3D
  and 4D data. Additionally, GenXD employs masked latent conditions to support a
  variety of conditioning views. GenXD can generate videos that follow the camera
  trajectory as well as consistent 3D views that can be lifted into 3D
  representations. We perform extensive evaluations across various real-world and
  synthetic datasets, demonstrating GenXD's effectiveness and versatility
  compared to previous methods in 3D and 4D generation.
paper_authors._id: 6729a976ac899234c0609d00
paper_authors.name: Yuyang Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:38:06.544Z
paper_authors._id: 64638bd36c27a7e33b26654b
paper_authors.avatarUrl: /avatars/2ef5aeb94ef7016082975b4cc201873e.svg
paper_authors.fullname: Yuyang
paper_authors.type: user
paper_authors.user: Yuyang-z
paper_authors._id: 6729a976ac899234c0609d01
paper_authors.name: Chung-Ching Lin
paper_authors._id: 6729a976ac899234c0609d02
paper_authors.name: Kevin Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-05T14:42:13.832Z
paper_authors._id: 6298fd95b58e71e2ac9f3ad8
paper_authors.avatarUrl: /avatars/7d34644d537bc5c17cf1e4ce4095355c.svg
paper_authors.fullname: Kevin Lin
paper_authors.type: user
paper_authors.user: kevinlin311tw
paper_authors._id: 6729a976ac899234c0609d03
paper_authors.name: Zhiwen Yan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-05T14:41:36.966Z
paper_authors._id: 636eae4d9b61600b629e3815
paper_authors.avatarUrl: /avatars/bf11616d5cefe6a0679d2902e3416257.svg
paper_authors.fullname: Zhiwen Yan
paper_authors.type: user
paper_authors.user: Zhiweny
paper_authors._id: 6729a976ac899234c0609d04
paper_authors.name: Linjie Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-05T14:42:00.444Z
paper_authors._id: 63db16fff03c3d71ef397206
paper_authors.avatarUrl: /avatars/bfb7e0d730b7d03302799d5d2828d97d.svg
paper_authors.fullname: Linjie Li
paper_authors.type: user
paper_authors.user: linjieli222
paper_authors._id: 6729a976ac899234c0609d05
paper_authors.name: Zhengyuan Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-05T14:41:44.006Z
paper_authors._id: 630713411801ecc7d2592a7c
paper_authors.avatarUrl: /avatars/fb36f69f03421c3a2a7f72ba0858fa60.svg
paper_authors.fullname: Zhengyuan Yang
paper_authors.type: user
paper_authors.user: zyang39
paper_authors._id: 6729a976ac899234c0609d06
paper_authors.name: Jianfeng Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-05T14:41:12.628Z
paper_authors._id: 641404e74e5305c14f269b3c
paper_authors.avatarUrl: /avatars/0dbe862cde7e4bef25bedf860584d8c1.svg
paper_authors.fullname: Jianfeng Wang
paper_authors.type: user
paper_authors.user: amsword8
paper_authors._id: 6729a976ac899234c0609d07
paper_authors.name: Gim Hee Lee
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-05T14:40:55.992Z
paper_authors._id: 64bc7568b7375f6b84503ce2
paper_authors.avatarUrl: /avatars/d970c9990d9d84ac51ff225c1b26e304.svg
paper_authors.fullname: Gim Hee Lee
paper_authors.type: user
paper_authors.user: gimhee-lee
paper_authors._id: 6729a976ac899234c0609d08
paper_authors.name: Lijuan Wang
paper_title: GenXD: Generating Any 3D and 4D Scenes
paper_summary: Recent developments in 2D visual generation have been remarkably successful.
  However, 3D and 4D generation remain challenging in real-world applications due
  to the lack of large-scale 4D data and effective model design. In this paper,
  we propose to jointly investigate general 3D and 4D generation by leveraging
  camera and object movements commonly observed in daily life. Due to the lack of
  real-world 4D data in the community, we first propose a data curation pipeline
  to obtain camera poses and object motion strength from videos. Based on this
  pipeline, we introduce a large-scale real-world 4D scene dataset: CamVid-30K.
  By leveraging all the 3D and 4D data, we develop our framework, GenXD, which
  allows us to produce any 3D or 4D scene. We propose multiview-temporal modules,
  which disentangle camera and object movements, to seamlessly learn from both 3D
  and 4D data. Additionally, GenXD employs masked latent conditions to support a
  variety of conditioning views. GenXD can generate videos that follow the camera
  trajectory as well as consistent 3D views that can be lifted into 3D
  representations. We perform extensive evaluations across various real-world and
  synthetic datasets, demonstrating GenXD's effectiveness and versatility
  compared to previous methods in 3D and 4D generation.
paper_ai_keywords: multiview-temporal modules masked latent conditions camera trajectory 3D representations