PhysGame: Uncovering Physical Commonsense Violations in Gameplay Videos
Recent advancements in video-based large language models (Video LLMs) have
witnessed the emergence of diverse capabilities to reason and interpret dynamic
visual content. Among them, gameplay videos stand out as a distinctive data
source, often containing glitches that defy physics commonsense. This
characteristic renders them an effective benchmark for assessing the
under-explored capability of physical commonsense understanding in video LLMs.
In this paper, we propose PhysGame as a pioneering benchmark to evaluate
physical commonsense violations in gameplay videos. PhysGame comprises 880
videos associated with glitches spanning four fundamental domains (i.e.,
mechanics, kinematics, optics, and material properties) and across 12 distinct
physical commonsense. Through extensively evaluating various state-ofthe-art
video LLMs, our findings reveal that the performance of current open-source
video LLMs significantly lags behind that of proprietary counterparts. To
bridge this gap, we curate an instruction tuning dataset PhysInstruct with
140,057 question-answering pairs to facilitate physical commonsense learning.
In addition, we also propose a preference optimization dataset PhysDPO with
34,358 training pairs, where the dis-preferred responses are generated
conditioned on misleading titles (i.e., meta information hacking), fewer frames
(i.e., temporal hacking) and lower spatial resolutions (i.e., spatial hacking).
Based on the suite of datasets, we propose PhysVLM as a physical
knowledge-enhanced video LLM. Extensive experiments on both physical-oriented
benchmark PhysGame and general video understanding benchmarks demonstrate the
state-ofthe-art performance of PhysVLM.
674edab4e8df509dc90ab774
Meng Cao
674edab4e8df509dc90ab775
Haoran Tang
674edab4e8df509dc90ab776
Haoze Zhao
admin_assigned
2024-12-03T16:12:38.116Z
6713de8eb8c4b8d06fc44418
/avatars/7753bd37363632e03180d7c3048b59f7.svg
Haoze ZHAO
user
ZHZ2002
674edab4e8df509dc90ab777
Hangyu Guo
admin_assigned
2024-12-03T16:12:50.388Z
646b43deb1202bc77c1024a4
/avatars/cf791574ab986bac274e7fbcf04e2a59.svg
hangyu guo
user
Rosiness
674edab4e8df509dc90ab778
Jiaheng Liu
admin_assigned
2024-12-03T16:12:56.177Z
65377c30e48353201e6fdda0
/avatars/a8f803b6f2e598eaee9c52c0d2ddfc16.svg
Jiaheng Liu
user
CheeryLJH
674edab4e8df509dc90ab779
Ge Zhang
claimed_verified
2024-12-03T11:14:40.537Z
638efcf4c67af472d316d424
/avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
Ge Zhang
user
zhangysk
674edab4e8df509dc90ab77a
Ruyang Liu
admin_assigned
2024-12-03T16:13:02.101Z
650f0e3988cdfe73a864b9c5
/avatars/ce74b96743a99819a762c23b4f2204fd.svg
Ruyang Liu
user
farewellthree
674edab4e8df509dc90ab77b
Qiang Sun
admin_assigned
2024-12-03T16:13:10.028Z
66a850167747a5de8ec191c0
/avatars/c47b1e4ba6299f185887f281c3e24dc6.svg
Qiang Sun
user
sunisrisingnow
674edab4e8df509dc90ab77c
Ian Reid
674edab4e8df509dc90ab77d
Xiaodan Liang
PhysGame: Uncovering Physical Commonsense Violations in Gameplay Videos
Recent advancements in video-based large language models (Video LLMs) have
witnessed the emergence of diverse capabilities to reason and interpret dynamic
visual content. Among them, gameplay videos stand out as a distinctive data
source, often containing glitches that defy physics commonsense. This
characteristic renders them an effective benchmark for assessing the
under-explored capability of physical commonsense understanding in video LLMs.
In this paper, we propose PhysGame as a pioneering benchmark to evaluate
physical commonsense violations in gameplay videos. PhysGame comprises 880
videos associated with glitches spanning four fundamental domains (i.e.,
mechanics, kinematics, optics, and material properties) and across 12 distinct
physical commonsense. Through extensively evaluating various state-ofthe-art
video LLMs, our findings reveal that the performance of current open-source
video LLMs significantly lags behind that of proprietary counterparts. To
bridge this gap, we curate an instruction tuning dataset PhysInstruct with
140,057 question-answering pairs to facilitate physical commonsense learning.
In addition, we also propose a preference optimization dataset PhysDPO with
34,358 training pairs, where the dis-preferred responses are generated
conditioned on misleading titles (i.e., meta information hacking), fewer frames
(i.e., temporal hacking) and lower spatial resolutions (i.e., spatial hacking).
Based on the suite of datasets, we propose PhysVLM as a physical
knowledge-enhanced video LLM. Extensive experiments on both physical-oriented
benchmark PhysGame and general video understanding benchmarks demonstrate the
state-ofthe-art performance of PhysVLM.
PhysGame and PhysInstruct datasets, along with preference optimization PhysDPO, enhance video LLMs' physical commonsense understanding, demonstrating superior performance over existing models.
video LLMs PhysGame game glitches physical commonsense PhysInstruct PhysDPO preference optimization PhysVLM mechanics kinematics optics material properties