title: PhysGame: Uncovering Physical Commonsense Violations in Gameplay Videos
summary: Recent advancements in video-based large language models (Video LLMs) have
  witnessed the emergence of diverse capabilities to reason and interpret dynamic
  visual content. Among them, gameplay videos stand out as a distinctive data
  source, often containing glitches that defy physics commonsense. This
  characteristic renders them an effective benchmark for assessing the
  under-explored capability of physical commonsense understanding in video LLMs.
  In this paper, we propose PhysGame as a pioneering benchmark to evaluate
  physical commonsense violations in gameplay videos. PhysGame comprises 880
  videos associated with glitches spanning four fundamental domains (i.e.,
  mechanics, kinematics, optics, and material properties) and across 12 distinct
  physical commonsense. Through extensively evaluating various state-ofthe-art
  video LLMs, our findings reveal that the performance of current open-source
  video LLMs significantly lags behind that of proprietary counterparts. To
  bridge this gap, we curate an instruction tuning dataset PhysInstruct with
  140,057 question-answering pairs to facilitate physical commonsense learning.
  In addition, we also propose a preference optimization dataset PhysDPO with
  34,358 training pairs, where the dis-preferred responses are generated
  conditioned on misleading titles (i.e., meta information hacking), fewer frames
  (i.e., temporal hacking) and lower spatial resolutions (i.e., spatial hacking).
  Based on the suite of datasets, we propose PhysVLM as a physical
  knowledge-enhanced video LLM. Extensive experiments on both physical-oriented
  benchmark PhysGame and general video understanding benchmarks demonstrate the
  state-ofthe-art performance of PhysVLM.
paper_authors._id: 674edab4e8df509dc90ab774
paper_authors.name: Meng Cao
paper_authors._id: 674edab4e8df509dc90ab775
paper_authors.name: Haoran Tang
paper_authors._id: 674edab4e8df509dc90ab776
paper_authors.name: Haoze Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-03T16:12:38.116Z
paper_authors._id: 6713de8eb8c4b8d06fc44418
paper_authors.avatarUrl: /avatars/7753bd37363632e03180d7c3048b59f7.svg
paper_authors.fullname: Haoze ZHAO
paper_authors.type: user
paper_authors.user: ZHZ2002
paper_authors._id: 674edab4e8df509dc90ab777
paper_authors.name: Hangyu Guo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-03T16:12:50.388Z
paper_authors._id: 646b43deb1202bc77c1024a4
paper_authors.avatarUrl: /avatars/cf791574ab986bac274e7fbcf04e2a59.svg
paper_authors.fullname: hangyu guo
paper_authors.type: user
paper_authors.user: Rosiness
paper_authors._id: 674edab4e8df509dc90ab778
paper_authors.name: Jiaheng Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-03T16:12:56.177Z
paper_authors._id: 65377c30e48353201e6fdda0
paper_authors.avatarUrl: /avatars/a8f803b6f2e598eaee9c52c0d2ddfc16.svg
paper_authors.fullname: Jiaheng Liu
paper_authors.type: user
paper_authors.user: CheeryLJH
paper_authors._id: 674edab4e8df509dc90ab779
paper_authors.name: Ge Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-03T11:14:40.537Z
paper_authors._id: 638efcf4c67af472d316d424
paper_authors.avatarUrl: /avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
paper_authors.fullname: Ge Zhang
paper_authors.type: user
paper_authors.user: zhangysk
paper_authors._id: 674edab4e8df509dc90ab77a
paper_authors.name: Ruyang Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-03T16:13:02.101Z
paper_authors._id: 650f0e3988cdfe73a864b9c5
paper_authors.avatarUrl: /avatars/ce74b96743a99819a762c23b4f2204fd.svg
paper_authors.fullname: Ruyang Liu
paper_authors.type: user
paper_authors.user: farewellthree
paper_authors._id: 674edab4e8df509dc90ab77b
paper_authors.name: Qiang Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-03T16:13:10.028Z
paper_authors._id: 66a850167747a5de8ec191c0
paper_authors.avatarUrl: /avatars/c47b1e4ba6299f185887f281c3e24dc6.svg
paper_authors.fullname: Qiang Sun
paper_authors.type: user
paper_authors.user: sunisrisingnow
paper_authors._id: 674edab4e8df509dc90ab77c
paper_authors.name: Ian Reid
paper_authors._id: 674edab4e8df509dc90ab77d
paper_authors.name: Xiaodan Liang
paper_title: PhysGame: Uncovering Physical Commonsense Violations in Gameplay Videos
paper_summary: Recent advancements in video-based large language models (Video LLMs) have
  witnessed the emergence of diverse capabilities to reason and interpret dynamic
  visual content. Among them, gameplay videos stand out as a distinctive data
  source, often containing glitches that defy physics commonsense. This
  characteristic renders them an effective benchmark for assessing the
  under-explored capability of physical commonsense understanding in video LLMs.
  In this paper, we propose PhysGame as a pioneering benchmark to evaluate
  physical commonsense violations in gameplay videos. PhysGame comprises 880
  videos associated with glitches spanning four fundamental domains (i.e.,
  mechanics, kinematics, optics, and material properties) and across 12 distinct
  physical commonsense. Through extensively evaluating various state-ofthe-art
  video LLMs, our findings reveal that the performance of current open-source
  video LLMs significantly lags behind that of proprietary counterparts. To
  bridge this gap, we curate an instruction tuning dataset PhysInstruct with
  140,057 question-answering pairs to facilitate physical commonsense learning.
  In addition, we also propose a preference optimization dataset PhysDPO with
  34,358 training pairs, where the dis-preferred responses are generated
  conditioned on misleading titles (i.e., meta information hacking), fewer frames
  (i.e., temporal hacking) and lower spatial resolutions (i.e., spatial hacking).
  Based on the suite of datasets, we propose PhysVLM as a physical
  knowledge-enhanced video LLM. Extensive experiments on both physical-oriented
  benchmark PhysGame and general video understanding benchmarks demonstrate the
  state-ofthe-art performance of PhysVLM.
paper_ai_summary: PhysGame and PhysInstruct datasets, along with preference optimization PhysDPO, enhance video LLMs' physical commonsense understanding, demonstrating superior performance over existing models.
paper_ai_keywords: video LLMs PhysGame game glitches physical commonsense PhysInstruct PhysDPO preference optimization PhysVLM mechanics kinematics optics material properties