Towards Physical Understanding in Video Generation: A 3D Point
  Regularization Approach
We present a novel video generation framework that integrates 3-dimensional
geometry and dynamic awareness. To achieve this, we augment 2D videos with 3D
point trajectories and align them in pixel space. The resulting 3D-aware video
dataset, PointVid, is then used to fine-tune a latent diffusion model, enabling
it to track 2D objects with 3D Cartesian coordinates. Building on this, we
regularize the shape and motion of objects in the video to eliminate undesired
artifacts, \eg, nonphysical deformation. Consequently, we enhance the quality
of generated RGB videos and alleviate common issues like object morphing, which
are prevalent in current video models due to a lack of shape awareness. With
our 3D augmentation and regularization, our model is capable of handling
contact-rich scenarios such as task-oriented videos. These videos involve
complex interactions of solids, where 3D information is essential for
perceiving deformation and contact. Furthermore, our model improves the overall
quality of video generation by promoting the 3D consistency of moving objects
and reducing abrupt changes in shape and motion.
67a59193f86e1b9d7ae7cd55
Yunuo Chen
admin_assigned
2025-02-07T10:50:19.710Z
65a47b4d60cc6b04c9ebb0ff
/avatars/b35ae99eab95e95a327c30b6d3ad6c83.svg
Yunuo Chen
user
yunuoch
67a59193f86e1b9d7ae7cd56
Junli Cao
admin_assigned
2025-02-07T10:50:25.916Z
63f54aa73aa49d8cb97b84bc
/avatars/c73c5870039611ab9162daad46a1ba20.svg
junli cao
user
jlcao2
67a59193f86e1b9d7ae7cd57
Anil Kag
admin_assigned
2025-02-07T10:50:32.227Z
66b01ee8e53bbad918362856
/avatars/293529589a91dd7a95909d66727db224.svg
Anil Kag
user
anilkagak2
67a59193f86e1b9d7ae7cd58
Vidit Goel
admin_assigned
2025-02-07T10:50:37.829Z
636c0c1a15cd58e915bb8139
/avatars/7c675ac6a7d303d3425e498c4e939eb0.svg
Vidit Goel
user
vidit98
67a59193f86e1b9d7ae7cd59
Sergei Korolev
67a59193f86e1b9d7ae7cd5a
Chenfanfu Jiang
admin_assigned
2025-02-07T10:50:50.619Z
655683727be68c0961673f45
/avatars/cddca36c041fa04860a4d42c0feaa07f.svg
Chenfanfu Jiang
user
cffjiang
67a59193f86e1b9d7ae7cd5b
Sergey Tulyakov
67a59193f86e1b9d7ae7cd5c
Jian Ren
claimed_verified
2025-02-10T09:50:19.428Z
61f19829233c91cbd2f79e70
/avatars/a0735a94542b4f7cda5aed8bc4be0538.svg
Jian Ren
user
alanspike
Towards Physical Understanding in Video Generation: A 3D Point
  Regularization Approach
We present a novel video generation framework that integrates 3-dimensional
geometry and dynamic awareness. To achieve this, we augment 2D videos with 3D
point trajectories and align them in pixel space. The resulting 3D-aware video
dataset, PointVid, is then used to fine-tune a latent diffusion model, enabling
it to track 2D objects with 3D Cartesian coordinates. Building on this, we
regularize the shape and motion of objects in the video to eliminate undesired
artifacts, \eg, nonphysical deformation. Consequently, we enhance the quality
of generated RGB videos and alleviate common issues like object morphing, which
are prevalent in current video models due to a lack of shape awareness. With
our 3D augmentation and regularization, our model is capable of handling
contact-rich scenarios such as task-oriented videos. These videos involve
complex interactions of solids, where 3D information is essential for
perceiving deformation and contact. Furthermore, our model improves the overall
quality of video generation by promoting the 3D consistency of moving objects
and reducing abrupt changes in shape and motion.
A 3D-aware video generation framework using latent diffusion modeling with point trajectory augmentation enhances video quality and addresses artifacts through motion and shape regularization.
3D geometry dynamic awareness 3D point trajectories pixel space 3D-aware video dataset latent diffusion model 3D Cartesian coordinates shape regularization motion regularization object morphing shape awareness task-oriented videos 3D consistency abrupt changes in shape and motion