title: Towards Physically Plausible Video Generation via VLM Planning
summary: Video diffusion models (VDMs) have advanced significantly in recent years,
  enabling the generation of highly realistic videos and drawing the attention of
  the community in their potential as world simulators. However, despite their
  capabilities, VDMs often fail to produce physically plausible videos due to an
  inherent lack of understanding of physics, resulting in incorrect dynamics and
  event sequences. To address this limitation, we propose a novel two-stage
  image-to-video generation framework that explicitly incorporates physics. In
  the first stage, we employ a Vision Language Model (VLM) as a coarse-grained
  motion planner, integrating chain-of-thought and physics-aware reasoning to
  predict a rough motion trajectories/changes that approximate real-world
  physical dynamics while ensuring the inter-frame consistency. In the second
  stage, we use the predicted motion trajectories/changes to guide the video
  generation of a VDM. As the predicted motion trajectories/changes are rough,
  noise is added during inference to provide freedom to the VDM in generating
  motion with more fine details. Extensive experimental results demonstrate that
  our framework can produce physically plausible motion, and comparative
  evaluations highlight the notable superiority of our approach over existing
  methods. More video results are available on our Project Page:
  https://madaoer.github.io/projects/physically_plausible_video_generation.
paper_authors._id: 67ee6cee153d50e65e63d024
paper_authors.name: Xindi Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-03T13:55:35.772Z
paper_authors._id: 6760e5fcf3591a93356e9a00
paper_authors.avatarUrl: /avatars/e4ab60ee6fa498170c901b3c4599ef4d.svg
paper_authors.fullname: Xindi Yang
paper_authors.type: user
paper_authors.user: Madaoer-Otaku
paper_authors._id: 67ee6cee153d50e65e63d025
paper_authors.name: Baolu Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-03T13:32:52.747Z
paper_authors._id: 652cb3e1c6857682d30d4c2b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/JEQW6WZBcmjAWwODiDkYA.jpeg
paper_authors.fullname: Libaolu
paper_authors.type: user
paper_authors.user: 8ruceLi
paper_authors._id: 67ee6cee153d50e65e63d026
paper_authors.name: Yiming Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-08T06:56:52.511Z
paper_authors._id: 66f2748bfa0f7d682326c834
paper_authors.avatarUrl: /avatars/cbd0e257ec752d803249d5422f7f7998.svg
paper_authors.fullname: Yiming  Zhang
paper_authors.type: user
paper_authors.user: zh1ym
paper_authors._id: 67ee6cee153d50e65e63d027
paper_authors.name: Zhenfei Yin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-03T13:56:12.459Z
paper_authors._id: 64e314ad24809d7fa0f20fbc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/bHE0w_hjDFvU-Aul0_E7g.jpeg
paper_authors.fullname: Zhenfei Yin
paper_authors.type: user
paper_authors.user: JeremyYin
paper_authors._id: 67ee6cee153d50e65e63d028
paper_authors.name: Lei Bai
paper_authors._id: 67ee6cee153d50e65e63d029
paper_authors.name: Liqian Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-03T13:56:36.706Z
paper_authors._id: 645b3f63f9d4ec91fdd5129f
paper_authors.avatarUrl: /avatars/e31b57294cd6622b42a96d5f83e95495.svg
paper_authors.fullname: Ma Liqian
paper_authors.type: user
paper_authors.user: Joyhunter9
paper_authors._id: 67ee6cee153d50e65e63d02a
paper_authors.name: Zhiyong Wang
paper_authors._id: 67ee6cee153d50e65e63d02b
paper_authors.name: Jianfei Cai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-03T13:56:25.898Z
paper_authors._id: 656fda082f058b368c27d1b9
paper_authors.avatarUrl: /avatars/cba33f6b20fc93ac9d3d579cbd7f839a.svg
paper_authors.fullname: cai jianfei
paper_authors.type: user
paper_authors.user: genify
paper_authors._id: 67ee6cee153d50e65e63d02c
paper_authors.name: Tien-Tsin Wong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-03T13:56:52.247Z
paper_authors._id: 65574f0fc4865c852d5eec15
paper_authors.avatarUrl: /avatars/1e03db4f2de4959dee620c577fbbb063.svg
paper_authors.fullname: Tien-Tsin Wong
paper_authors.type: user
paper_authors.user: ttwong
paper_authors._id: 67ee6cee153d50e65e63d02d
paper_authors.name: Huchuan Lu
paper_authors._id: 67ee6cee153d50e65e63d02e
paper_authors.name: Xu Jia
paper_title: Towards Physically Plausible Video Generation via VLM Planning
paper_summary: Video diffusion models (VDMs) have advanced significantly in recent years,
  enabling the generation of highly realistic videos and drawing the attention of
  the community in their potential as world simulators. However, despite their
  capabilities, VDMs often fail to produce physically plausible videos due to an
  inherent lack of understanding of physics, resulting in incorrect dynamics and
  event sequences. To address this limitation, we propose a novel two-stage
  image-to-video generation framework that explicitly incorporates physics. In
  the first stage, we employ a Vision Language Model (VLM) as a coarse-grained
  motion planner, integrating chain-of-thought and physics-aware reasoning to
  predict a rough motion trajectories/changes that approximate real-world
  physical dynamics while ensuring the inter-frame consistency. In the second
  stage, we use the predicted motion trajectories/changes to guide the video
  generation of a VDM. As the predicted motion trajectories/changes are rough,
  noise is added during inference to provide freedom to the VDM in generating
  motion with more fine details. Extensive experimental results demonstrate that
  our framework can produce physically plausible motion, and comparative
  evaluations highlight the notable superiority of our approach over existing
  methods. More video results are available on our Project Page:
  https://madaoer.github.io/projects/physically_plausible_video_generation.
paper_ai_summary: A two-stage framework combines a Vision Language Model for coarse motion planning with a Video Diffusion Model to generate physically plausible videos.
paper_ai_keywords: Video diffusion models VDMs Vision Language Model VLM chain-of-thought reasoning physics-aware reasoning motion trajectories inter-frame consistency noise addition video generation