GenXD: Generating Any 3D and 4D Scenes
Recent developments in 2D visual generation have been remarkably successful.
However, 3D and 4D generation remain challenging in real-world applications due
to the lack of large-scale 4D data and effective model design. In this paper,
we propose to jointly investigate general 3D and 4D generation by leveraging
camera and object movements commonly observed in daily life. Due to the lack of
real-world 4D data in the community, we first propose a data curation pipeline
to obtain camera poses and object motion strength from videos. Based on this
pipeline, we introduce a large-scale real-world 4D scene dataset: CamVid-30K.
By leveraging all the 3D and 4D data, we develop our framework, GenXD, which
allows us to produce any 3D or 4D scene. We propose multiview-temporal modules,
which disentangle camera and object movements, to seamlessly learn from both 3D
and 4D data. Additionally, GenXD employs masked latent conditions to support a
variety of conditioning views. GenXD can generate videos that follow the camera
trajectory as well as consistent 3D views that can be lifted into 3D
representations. We perform extensive evaluations across various real-world and
synthetic datasets, demonstrating GenXD's effectiveness and versatility
compared to previous methods in 3D and 4D generation.
6729a976ac899234c0609d00
Yuyang Zhao
claimed_verified
2024-11-15T09:38:06.544Z
64638bd36c27a7e33b26654b
/avatars/2ef5aeb94ef7016082975b4cc201873e.svg
Yuyang
user
Yuyang-z
6729a976ac899234c0609d01
Chung-Ching Lin
claimed_verified
2025-07-23T08:38:46.983Z
6669f406d21cf7d39b1d98ba
/avatars/5ec270cb8bcb68786379d8a8bc411aaa.svg
CC Lin
user
cclin10
6729a976ac899234c0609d02
Kevin Lin
admin_assigned
2024-11-05T14:42:13.832Z
6298fd95b58e71e2ac9f3ad8
/avatars/7d34644d537bc5c17cf1e4ce4095355c.svg
Kevin Lin
user
kevinlin311tw
6729a976ac899234c0609d03
Zhiwen Yan
admin_assigned
2024-11-05T14:41:36.966Z
636eae4d9b61600b629e3815
/avatars/bf11616d5cefe6a0679d2902e3416257.svg
Zhiwen Yan
user
Zhiweny
6729a976ac899234c0609d04
Linjie Li
admin_assigned
2024-11-05T14:42:00.444Z
63db16fff03c3d71ef397206
/avatars/bfb7e0d730b7d03302799d5d2828d97d.svg
Linjie Li
user
linjieli222
6729a976ac899234c0609d05
Zhengyuan Yang
admin_assigned
2024-11-05T14:41:44.006Z
630713411801ecc7d2592a7c
/avatars/fb36f69f03421c3a2a7f72ba0858fa60.svg
Zhengyuan Yang
user
zyang39
6729a976ac899234c0609d06
Jianfeng Wang
admin_assigned
2024-11-05T14:41:12.628Z
641404e74e5305c14f269b3c
/avatars/0dbe862cde7e4bef25bedf860584d8c1.svg
Jianfeng Wang
user
amsword8
6729a976ac899234c0609d07
Gim Hee Lee
admin_assigned
2024-11-05T14:40:55.992Z
64bc7568b7375f6b84503ce2
/avatars/d970c9990d9d84ac51ff225c1b26e304.svg
Gim Hee Lee
user
gimhee-lee
6729a976ac899234c0609d08
Lijuan Wang
GenXD: Generating Any 3D and 4D Scenes
Recent developments in 2D visual generation have been remarkably successful.
However, 3D and 4D generation remain challenging in real-world applications due
to the lack of large-scale 4D data and effective model design. In this paper,
we propose to jointly investigate general 3D and 4D generation by leveraging
camera and object movements commonly observed in daily life. Due to the lack of
real-world 4D data in the community, we first propose a data curation pipeline
to obtain camera poses and object motion strength from videos. Based on this
pipeline, we introduce a large-scale real-world 4D scene dataset: CamVid-30K.
By leveraging all the 3D and 4D data, we develop our framework, GenXD, which
allows us to produce any 3D or 4D scene. We propose multiview-temporal modules,
which disentangle camera and object movements, to seamlessly learn from both 3D
and 4D data. Additionally, GenXD employs masked latent conditions to support a
variety of conditioning views. GenXD can generate videos that follow the camera
trajectory as well as consistent 3D views that can be lifted into 3D
representations. We perform extensive evaluations across various real-world and
synthetic datasets, demonstrating GenXD's effectiveness and versatility
compared to previous methods in 3D and 4D generation.
A framework called GenXD generates 3D and 4D scenes from multiview-temporal data, leveraging camera and object movements, and demonstrates superior performance compared to existing methods.
multiview-temporal modules masked latent conditions 3D generation 4D generation camera trajectory 3D representations GenXD CamVid-30K