WISA: World Simulator Assistant for Physics-Aware Text-to-Video
  Generation
Recent rapid advancements in text-to-video (T2V) generation, such as SoRA and
Kling, have shown great potential for building world simulators. However,
current T2V models struggle to grasp abstract physical principles and generate
videos that adhere to physical laws. This challenge arises primarily from a
lack of clear guidance on physical information due to a significant gap between
abstract physical principles and generation models. To this end, we introduce
the World Simulator Assistant (WISA), an effective framework for decomposing
and incorporating physical principles into T2V models. Specifically, WISA
decomposes physical principles into textual physical descriptions, qualitative
physical categories, and quantitative physical properties. To effectively embed
these physical attributes into the generation process, WISA incorporates
several key designs, including Mixture-of-Physical-Experts Attention (MoPA) and
a Physical Classifier, enhancing the model's physics awareness. Furthermore,
most existing datasets feature videos where physical phenomena are either
weakly represented or entangled with multiple co-occurring processes, limiting
their suitability as dedicated resources for learning explicit physical
principles. We propose a novel video dataset, WISA-32K, collected based on
qualitative physical categories. It consists of 32,000 videos, representing 17
physical laws across three domains of physics: dynamics, thermodynamics, and
optics. Experimental results demonstrate that WISA can effectively enhance the
compatibility of T2V models with real-world physical laws, achieving a
considerable improvement on the VideoPhy benchmark. The visual exhibitions of
WISA and WISA-32K are available in the https://360cvgroup.github.io/WISA/.
67d98394f333b86d4b17fc17
Jing Wang
67d98394f333b86d4b17fc18
Ao Ma
67d98394f333b86d4b17fc19
Ke Cao
67d98394f333b86d4b17fc1a
Jun Zheng
claimed_verified
2025-03-18T14:57:42.218Z
6381847a471a4550ff298c63
https://cdn-avatars.huggingface.co/v1/production/uploads/6381847a471a4550ff298c63/RTKepvX67R6pLiiUidpUO.png
Jun
user
zxbsmk
67d98394f333b86d4b17fc1b
Zhanjie Zhang
67d98394f333b86d4b17fc1c
Jiasong Feng
claimed_verified
2025-07-25T08:39:56.203Z
65542ae4520d9bb9d9d21659
/avatars/c89f2e76f2f1f4e162d15f4da8280f19.svg
js
user
feng8
67d98394f333b86d4b17fc1d
Shanyuan Liu
67d98394f333b86d4b17fc1e
Yuhang Ma
67d98394f333b86d4b17fc1f
Bo Cheng
67d98394f333b86d4b17fc20
Dawei Leng
extracted_confirmed
2025-03-19T01:32:34.905Z
649935abbe8fd92c27ab1ed8
https://cdn-avatars.huggingface.co/v1/production/uploads/649935abbe8fd92c27ab1ed8/ueWnaZtJa-oWpzupP6FV8.png
David Leon
user
DavidLeon
67d98394f333b86d4b17fc21
Yuhui Yin
67d98394f333b86d4b17fc22
Xiaodan Liang
WISA: World Simulator Assistant for Physics-Aware Text-to-Video
  Generation
Recent rapid advancements in text-to-video (T2V) generation, such as SoRA and
Kling, have shown great potential for building world simulators. However,
current T2V models struggle to grasp abstract physical principles and generate
videos that adhere to physical laws. This challenge arises primarily from a
lack of clear guidance on physical information due to a significant gap between
abstract physical principles and generation models. To this end, we introduce
the World Simulator Assistant (WISA), an effective framework for decomposing
and incorporating physical principles into T2V models. Specifically, WISA
decomposes physical principles into textual physical descriptions, qualitative
physical categories, and quantitative physical properties. To effectively embed
these physical attributes into the generation process, WISA incorporates
several key designs, including Mixture-of-Physical-Experts Attention (MoPA) and
a Physical Classifier, enhancing the model's physics awareness. Furthermore,
most existing datasets feature videos where physical phenomena are either
weakly represented or entangled with multiple co-occurring processes, limiting
their suitability as dedicated resources for learning explicit physical
principles. We propose a novel video dataset, WISA-32K, collected based on
qualitative physical categories. It consists of 32,000 videos, representing 17
physical laws across three domains of physics: dynamics, thermodynamics, and
optics. Experimental results demonstrate that WISA can effectively enhance the
compatibility of T2V models with real-world physical laws, achieving a
considerable improvement on the VideoPhy benchmark. The visual exhibitions of
WISA and WISA-32K are available in the https://360cvgroup.github.io/WISA/.
WISA enhances text-to-video models' physical accuracy by decomposing and embedding physical principles using MoPA and a Physical Classifier, validated on a new dataset of 32,000 videos representing 17 physical laws.
text-to-video T2V World Simulator Assistant WISA Mixture-of-Physical-Experts Attention MoPA Physical Classifier VideoPhy benchmark physical laws dynamics thermodynamics optics