title: Towards Physical Understanding in Video Generation: A 3D Point
    Regularization Approach
summary: We present a novel video generation framework that integrates 3-dimensional
  geometry and dynamic awareness. To achieve this, we augment 2D videos with 3D
  point trajectories and align them in pixel space. The resulting 3D-aware video
  dataset, PointVid, is then used to fine-tune a latent diffusion model, enabling
  it to track 2D objects with 3D Cartesian coordinates. Building on this, we
  regularize the shape and motion of objects in the video to eliminate undesired
  artifacts, \eg, nonphysical deformation. Consequently, we enhance the quality
  of generated RGB videos and alleviate common issues like object morphing, which
  are prevalent in current video models due to a lack of shape awareness. With
  our 3D augmentation and regularization, our model is capable of handling
  contact-rich scenarios such as task-oriented videos. These videos involve
  complex interactions of solids, where 3D information is essential for
  perceiving deformation and contact. Furthermore, our model improves the overall
  quality of video generation by promoting the 3D consistency of moving objects
  and reducing abrupt changes in shape and motion.
paper_authors._id: 67a59193f86e1b9d7ae7cd55
paper_authors.name: Yunuo Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-07T10:50:19.710Z
paper_authors._id: 65a47b4d60cc6b04c9ebb0ff
paper_authors.avatarUrl: /avatars/b35ae99eab95e95a327c30b6d3ad6c83.svg
paper_authors.fullname: Yunuo Chen
paper_authors.type: user
paper_authors.user: yunuoch
paper_authors._id: 67a59193f86e1b9d7ae7cd56
paper_authors.name: Junli Cao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-07T10:50:25.916Z
paper_authors._id: 63f54aa73aa49d8cb97b84bc
paper_authors.avatarUrl: /avatars/c73c5870039611ab9162daad46a1ba20.svg
paper_authors.fullname: junli cao
paper_authors.type: user
paper_authors.user: jlcao2
paper_authors._id: 67a59193f86e1b9d7ae7cd57
paper_authors.name: Anil Kag
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-07T10:50:32.227Z
paper_authors._id: 66b01ee8e53bbad918362856
paper_authors.avatarUrl: /avatars/293529589a91dd7a95909d66727db224.svg
paper_authors.fullname: Anil Kag
paper_authors.type: user
paper_authors.user: anilkagak2
paper_authors._id: 67a59193f86e1b9d7ae7cd58
paper_authors.name: Vidit Goel
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-07T10:50:37.829Z
paper_authors._id: 636c0c1a15cd58e915bb8139
paper_authors.avatarUrl: /avatars/7c675ac6a7d303d3425e498c4e939eb0.svg
paper_authors.fullname: Vidit Goel
paper_authors.type: user
paper_authors.user: vidit98
paper_authors._id: 67a59193f86e1b9d7ae7cd59
paper_authors.name: Sergei Korolev
paper_authors._id: 67a59193f86e1b9d7ae7cd5a
paper_authors.name: Chenfanfu Jiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-07T10:50:50.619Z
paper_authors._id: 655683727be68c0961673f45
paper_authors.avatarUrl: /avatars/cddca36c041fa04860a4d42c0feaa07f.svg
paper_authors.fullname: Chenfanfu Jiang
paper_authors.type: user
paper_authors.user: cffjiang
paper_authors._id: 67a59193f86e1b9d7ae7cd5b
paper_authors.name: Sergey Tulyakov
paper_authors._id: 67a59193f86e1b9d7ae7cd5c
paper_authors.name: Jian Ren
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-10T09:50:19.428Z
paper_authors._id: 61f19829233c91cbd2f79e70
paper_authors.avatarUrl: /avatars/a0735a94542b4f7cda5aed8bc4be0538.svg
paper_authors.fullname: Jian Ren
paper_authors.type: user
paper_authors.user: alanspike
paper_title: Towards Physical Understanding in Video Generation: A 3D Point
    Regularization Approach
paper_summary: We present a novel video generation framework that integrates 3-dimensional
  geometry and dynamic awareness. To achieve this, we augment 2D videos with 3D
  point trajectories and align them in pixel space. The resulting 3D-aware video
  dataset, PointVid, is then used to fine-tune a latent diffusion model, enabling
  it to track 2D objects with 3D Cartesian coordinates. Building on this, we
  regularize the shape and motion of objects in the video to eliminate undesired
  artifacts, \eg, nonphysical deformation. Consequently, we enhance the quality
  of generated RGB videos and alleviate common issues like object morphing, which
  are prevalent in current video models due to a lack of shape awareness. With
  our 3D augmentation and regularization, our model is capable of handling
  contact-rich scenarios such as task-oriented videos. These videos involve
  complex interactions of solids, where 3D information is essential for
  perceiving deformation and contact. Furthermore, our model improves the overall
  quality of video generation by promoting the 3D consistency of moving objects
  and reducing abrupt changes in shape and motion.
paper_ai_summary: A 3D-aware video generation framework using latent diffusion modeling with point trajectory augmentation enhances video quality and addresses artifacts through motion and shape regularization.
paper_ai_keywords: 3D geometry dynamic awareness 3D point trajectories pixel space 3D-aware video dataset latent diffusion model 3D Cartesian coordinates shape regularization motion regularization object morphing shape awareness task-oriented videos 3D consistency abrupt changes in shape and motion