title: WISA: World Simulator Assistant for Physics-Aware Text-to-Video
    Generation
summary: Recent rapid advancements in text-to-video (T2V) generation, such as SoRA and
  Kling, have shown great potential for building world simulators. However,
  current T2V models struggle to grasp abstract physical principles and generate
  videos that adhere to physical laws. This challenge arises primarily from a
  lack of clear guidance on physical information due to a significant gap between
  abstract physical principles and generation models. To this end, we introduce
  the World Simulator Assistant (WISA), an effective framework for decomposing
  and incorporating physical principles into T2V models. Specifically, WISA
  decomposes physical principles into textual physical descriptions, qualitative
  physical categories, and quantitative physical properties. To effectively embed
  these physical attributes into the generation process, WISA incorporates
  several key designs, including Mixture-of-Physical-Experts Attention (MoPA) and
  a Physical Classifier, enhancing the model's physics awareness. Furthermore,
  most existing datasets feature videos where physical phenomena are either
  weakly represented or entangled with multiple co-occurring processes, limiting
  their suitability as dedicated resources for learning explicit physical
  principles. We propose a novel video dataset, WISA-32K, collected based on
  qualitative physical categories. It consists of 32,000 videos, representing 17
  physical laws across three domains of physics: dynamics, thermodynamics, and
  optics. Experimental results demonstrate that WISA can effectively enhance the
  compatibility of T2V models with real-world physical laws, achieving a
  considerable improvement on the VideoPhy benchmark. The visual exhibitions of
  WISA and WISA-32K are available in the https://360cvgroup.github.io/WISA/.
paper_authors._id: 67d98394f333b86d4b17fc17
paper_authors.name: Jing Wang
paper_authors._id: 67d98394f333b86d4b17fc18
paper_authors.name: Ao Ma
paper_authors._id: 67d98394f333b86d4b17fc19
paper_authors.name: Ke Cao
paper_authors._id: 67d98394f333b86d4b17fc1a
paper_authors.name: Jun Zheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-18T14:57:42.218Z
paper_authors._id: 6381847a471a4550ff298c63
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6381847a471a4550ff298c63/RTKepvX67R6pLiiUidpUO.png
paper_authors.fullname: Jun
paper_authors.type: user
paper_authors.user: zxbsmk
paper_authors._id: 67d98394f333b86d4b17fc1b
paper_authors.name: Zhanjie Zhang
paper_authors._id: 67d98394f333b86d4b17fc1c
paper_authors.name: Jiasong Feng
paper_authors._id: 67d98394f333b86d4b17fc1d
paper_authors.name: Shanyuan Liu
paper_authors._id: 67d98394f333b86d4b17fc1e
paper_authors.name: Yuhang Ma
paper_authors._id: 67d98394f333b86d4b17fc1f
paper_authors.name: Bo Cheng
paper_authors._id: 67d98394f333b86d4b17fc20
paper_authors.name: Dawei Leng
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-03-19T01:32:34.905Z
paper_authors._id: 649935abbe8fd92c27ab1ed8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/649935abbe8fd92c27ab1ed8/ueWnaZtJa-oWpzupP6FV8.png
paper_authors.fullname: David Leon
paper_authors.type: user
paper_authors.user: DavidLeon
paper_authors._id: 67d98394f333b86d4b17fc21
paper_authors.name: Yuhui Yin
paper_authors._id: 67d98394f333b86d4b17fc22
paper_authors.name: Xiaodan Liang
paper_title: WISA: World Simulator Assistant for Physics-Aware Text-to-Video
    Generation
paper_summary: Recent rapid advancements in text-to-video (T2V) generation, such as SoRA and
  Kling, have shown great potential for building world simulators. However,
  current T2V models struggle to grasp abstract physical principles and generate
  videos that adhere to physical laws. This challenge arises primarily from a
  lack of clear guidance on physical information due to a significant gap between
  abstract physical principles and generation models. To this end, we introduce
  the World Simulator Assistant (WISA), an effective framework for decomposing
  and incorporating physical principles into T2V models. Specifically, WISA
  decomposes physical principles into textual physical descriptions, qualitative
  physical categories, and quantitative physical properties. To effectively embed
  these physical attributes into the generation process, WISA incorporates
  several key designs, including Mixture-of-Physical-Experts Attention (MoPA) and
  a Physical Classifier, enhancing the model's physics awareness. Furthermore,
  most existing datasets feature videos where physical phenomena are either
  weakly represented or entangled with multiple co-occurring processes, limiting
  their suitability as dedicated resources for learning explicit physical
  principles. We propose a novel video dataset, WISA-32K, collected based on
  qualitative physical categories. It consists of 32,000 videos, representing 17
  physical laws across three domains of physics: dynamics, thermodynamics, and
  optics. Experimental results demonstrate that WISA can effectively enhance the
  compatibility of T2V models with real-world physical laws, achieving a
  considerable improvement on the VideoPhy benchmark. The visual exhibitions of
  WISA and WISA-32K are available in the https://360cvgroup.github.io/WISA/.
paper_ai_summary: WISA enhances text-to-video models' physical accuracy by decomposing and embedding physical principles using MoPA and a Physical Classifier, validated on a new dataset of 32,000 videos representing 17 physical laws.
paper_ai_keywords: text-to-video T2V World Simulator Assistant WISA Mixture-of-Physical-Experts Attention MoPA Physical Classifier VideoPhy benchmark physical laws dynamics thermodynamics optics