Towards Physically Plausible Video Generation via VLM Planning
Video diffusion models (VDMs) have advanced significantly in recent years,
enabling the generation of highly realistic videos and drawing the attention of
the community in their potential as world simulators. However, despite their
capabilities, VDMs often fail to produce physically plausible videos due to an
inherent lack of understanding of physics, resulting in incorrect dynamics and
event sequences. To address this limitation, we propose a novel two-stage
image-to-video generation framework that explicitly incorporates physics. In
the first stage, we employ a Vision Language Model (VLM) as a coarse-grained
motion planner, integrating chain-of-thought and physics-aware reasoning to
predict a rough motion trajectories/changes that approximate real-world
physical dynamics while ensuring the inter-frame consistency. In the second
stage, we use the predicted motion trajectories/changes to guide the video
generation of a VDM. As the predicted motion trajectories/changes are rough,
noise is added during inference to provide freedom to the VDM in generating
motion with more fine details. Extensive experimental results demonstrate that
our framework can produce physically plausible motion, and comparative
evaluations highlight the notable superiority of our approach over existing
methods. More video results are available on our Project Page:
https://madaoer.github.io/projects/physically_plausible_video_generation.
67ee6cee153d50e65e63d024
Xindi Yang
admin_assigned
2025-04-03T13:55:35.772Z
6760e5fcf3591a93356e9a00
/avatars/e4ab60ee6fa498170c901b3c4599ef4d.svg
Xindi Yang
user
Madaoer-Otaku
67ee6cee153d50e65e63d025
Baolu Li
claimed_verified
2025-04-03T13:32:52.747Z
652cb3e1c6857682d30d4c2b
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/JEQW6WZBcmjAWwODiDkYA.jpeg
Libaolu
user
8ruceLi
67ee6cee153d50e65e63d026
Yiming Zhang
claimed_verified
2025-04-08T06:56:52.511Z
66f2748bfa0f7d682326c834
/avatars/cbd0e257ec752d803249d5422f7f7998.svg
Yiming  Zhang
user
zh1ym
67ee6cee153d50e65e63d027
Zhenfei Yin
admin_assigned
2025-04-03T13:56:12.459Z
64e314ad24809d7fa0f20fbc
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/bHE0w_hjDFvU-Aul0_E7g.jpeg
Zhenfei Yin
user
JeremyYin
67ee6cee153d50e65e63d028
Lei Bai
67ee6cee153d50e65e63d029
Liqian Ma
admin_assigned
2025-04-03T13:56:36.706Z
645b3f63f9d4ec91fdd5129f
/avatars/e31b57294cd6622b42a96d5f83e95495.svg
Ma Liqian
user
Joyhunter9
67ee6cee153d50e65e63d02a
Zhiyong Wang
67ee6cee153d50e65e63d02b
Jianfei Cai
admin_assigned
2025-04-03T13:56:25.898Z
656fda082f058b368c27d1b9
/avatars/cba33f6b20fc93ac9d3d579cbd7f839a.svg
cai jianfei
user
genify
67ee6cee153d50e65e63d02c
Tien-Tsin Wong
admin_assigned
2025-04-03T13:56:52.247Z
65574f0fc4865c852d5eec15
/avatars/1e03db4f2de4959dee620c577fbbb063.svg
Tien-Tsin Wong
user
ttwong
67ee6cee153d50e65e63d02d
Huchuan Lu
67ee6cee153d50e65e63d02e
Xu Jia
Towards Physically Plausible Video Generation via VLM Planning
Video diffusion models (VDMs) have advanced significantly in recent years,
enabling the generation of highly realistic videos and drawing the attention of
the community in their potential as world simulators. However, despite their
capabilities, VDMs often fail to produce physically plausible videos due to an
inherent lack of understanding of physics, resulting in incorrect dynamics and
event sequences. To address this limitation, we propose a novel two-stage
image-to-video generation framework that explicitly incorporates physics. In
the first stage, we employ a Vision Language Model (VLM) as a coarse-grained
motion planner, integrating chain-of-thought and physics-aware reasoning to
predict a rough motion trajectories/changes that approximate real-world
physical dynamics while ensuring the inter-frame consistency. In the second
stage, we use the predicted motion trajectories/changes to guide the video
generation of a VDM. As the predicted motion trajectories/changes are rough,
noise is added during inference to provide freedom to the VDM in generating
motion with more fine details. Extensive experimental results demonstrate that
our framework can produce physically plausible motion, and comparative
evaluations highlight the notable superiority of our approach over existing
methods. More video results are available on our Project Page:
https://madaoer.github.io/projects/physically_plausible_video_generation.
A two-stage framework combines a Vision Language Model for coarse motion planning with a Video Diffusion Model to generate physically plausible videos.
Video diffusion models VDMs Vision Language Model VLM chain-of-thought reasoning physics-aware reasoning motion trajectories inter-frame consistency noise addition video generation