title: MMWorld: Towards Multi-discipline Multi-faceted World Model Evaluation
    in Videos
summary: Multimodal Language Language Models (MLLMs) demonstrate the emerging
  abilities of "world models" -- interpreting and reasoning about complex
  real-world dynamics. To assess these abilities, we posit videos are the ideal
  medium, as they encapsulate rich representations of real-world dynamics and
  causalities. To this end, we introduce MMWorld, a new benchmark for
  multi-discipline, multi-faceted multimodal video understanding. MMWorld
  distinguishes itself from previous video understanding benchmarks with two
  unique advantages: (1) multi-discipline, covering various disciplines that
  often require domain expertise for comprehensive understanding; (2)
  multi-faceted reasoning, including explanation, counterfactual thinking, future
  prediction, etc. MMWorld consists of a human-annotated dataset to evaluate
  MLLMs with questions about the whole videos and a synthetic dataset to analyze
  MLLMs within a single modality of perception. Together, MMWorld encompasses
  1,910 videos across seven broad disciplines and 69 subdisciplines, complete
  with 6,627 question-answer pairs and associated captions. The evaluation
  includes 2 proprietary and 10 open-source MLLMs, which struggle on MMWorld
  (e.g., GPT-4V performs the best with only 52.3\% accuracy), showing large room
  for improvement. Further ablation studies reveal other interesting findings
  such as models' different skill sets from humans. We hope MMWorld can serve as
  an essential step towards world model evaluation in videos.
paper_authors._id: 666a574b5c3bb7e4a6c7ff51
paper_authors.name: Xuehai He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-13T20:09:58.196Z
paper_authors._id: 6310493158d83e8f64dc8c55
paper_authors.avatarUrl: /avatars/5f91ac4dfec0d6a5bf7bad6094f0fd0f.svg
paper_authors.fullname: Xuehai He
paper_authors.type: user
paper_authors.user: Xuehai
paper_authors._id: 666a574b5c3bb7e4a6c7ff52
paper_authors.name: Weixi Feng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-13T20:10:04.923Z
paper_authors._id: 6259e9489326c60347dd2b6c
paper_authors.avatarUrl: /avatars/6aae6282ddee96b899b7384154e9b0a0.svg
paper_authors.fullname: Weixi Feng
paper_authors.type: user
paper_authors.user: weixifeng
paper_authors._id: 666a574b5c3bb7e4a6c7ff53
paper_authors.name: Kaizhi Zheng
paper_authors._id: 666a574b5c3bb7e4a6c7ff54
paper_authors.name: Yujie Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-13T20:10:19.029Z
paper_authors._id: 633cfb6ad5935998f7582b7b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/633cfb6ad5935998f7582b7b/mzU4lI_GWPsYOFZHTBsYA.jpeg
paper_authors.fullname: Yujie Lu
paper_authors.type: user
paper_authors.user: FunCube
paper_authors._id: 666a574b5c3bb7e4a6c7ff55
paper_authors.name: Wanrong Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-13T20:10:25.249Z
paper_authors._id: 633a7e91c8eb7b089034fb3c
paper_authors.avatarUrl: /avatars/0e03d80aa469de131cc17bb0c5c016b9.svg
paper_authors.fullname: Wanrong Zhu
paper_authors.type: user
paper_authors.user: VegB
paper_authors._id: 666a574b5c3bb7e4a6c7ff56
paper_authors.name: Jiachen Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-13T09:31:10.205Z
paper_authors._id: 640d7ec5fdeaae13907fc488
paper_authors.avatarUrl: /avatars/acf43ea155105a51c8612dacc4725091.svg
paper_authors.fullname: Jiachen Li
paper_authors.type: user
paper_authors.user: jiachenli-ucsb
paper_authors._id: 666a574b5c3bb7e4a6c7ff57
paper_authors.name: Yue Fan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-30T20:58:09.562Z
paper_authors._id: 656b8a62e8bf55919a6a2d61
paper_authors.avatarUrl: /avatars/26b75683dff73c51f966e9c15f04fd9f.svg
paper_authors.fullname: Yue Fan
paper_authors.type: user
paper_authors.user: yfan1997
paper_authors._id: 666a574b5c3bb7e4a6c7ff58
paper_authors.name: Jianfeng Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-13T20:10:55.977Z
paper_authors._id: 641404e74e5305c14f269b3c
paper_authors.avatarUrl: /avatars/0dbe862cde7e4bef25bedf860584d8c1.svg
paper_authors.fullname: Jianfeng Wang
paper_authors.type: user
paper_authors.user: amsword8
paper_authors._id: 666a574b5c3bb7e4a6c7ff59
paper_authors.name: Linjie Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-13T20:11:02.085Z
paper_authors._id: 63db16fff03c3d71ef397206
paper_authors.avatarUrl: /avatars/bfb7e0d730b7d03302799d5d2828d97d.svg
paper_authors.fullname: Linjie Li
paper_authors.type: user
paper_authors.user: linjieli222
paper_authors._id: 666a574b5c3bb7e4a6c7ff5a
paper_authors.name: Zhengyuan Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-13T20:11:08.514Z
paper_authors._id: 630713411801ecc7d2592a7c
paper_authors.avatarUrl: /avatars/fb36f69f03421c3a2a7f72ba0858fa60.svg
paper_authors.fullname: Zhengyuan Yang
paper_authors.type: user
paper_authors.user: zyang39
paper_authors._id: 666a574b5c3bb7e4a6c7ff5b
paper_authors.name: Kevin Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-13T20:11:21.085Z
paper_authors._id: 6298fd95b58e71e2ac9f3ad8
paper_authors.avatarUrl: /avatars/7d34644d537bc5c17cf1e4ce4095355c.svg
paper_authors.fullname: Kevin Lin
paper_authors.type: user
paper_authors.user: kevinlin311tw
paper_authors._id: 666a574b5c3bb7e4a6c7ff5c
paper_authors.name: William Yang Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-13T20:11:27.471Z
paper_authors._id: 6476607efb22e3b77f3f1452
paper_authors.avatarUrl: /avatars/22f40c81e6b6e09bdb66a5afd978f98b.svg
paper_authors.fullname: William Yang Wang
paper_authors.type: user
paper_authors.user: wangwilliamyang
paper_authors._id: 666a574b5c3bb7e4a6c7ff5d
paper_authors.name: Lijuan Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-13T20:11:32.768Z
paper_authors._id: 6413521d4e5305c14f22e110
paper_authors.avatarUrl: /avatars/a6f8d0573e678f79bc3c0b7897b818ce.svg
paper_authors.fullname: Lijuan Wang
paper_authors.type: user
paper_authors.user: Lijuan
paper_authors._id: 666a574b5c3bb7e4a6c7ff5e
paper_authors.name: Xin Eric Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-13T09:31:12.418Z
paper_authors._id: 64679a226192d39142245e5e
paper_authors.avatarUrl: /avatars/05abee0b6317f100923936ca2099e9eb.svg
paper_authors.fullname: Xin Eric Wang
paper_authors.type: user
paper_authors.user: xw-eric
paper_title: MMWorld: Towards Multi-discipline Multi-faceted World Model Evaluation
    in Videos
paper_summary: Multimodal Language Language Models (MLLMs) demonstrate the emerging
  abilities of "world models" -- interpreting and reasoning about complex
  real-world dynamics. To assess these abilities, we posit videos are the ideal
  medium, as they encapsulate rich representations of real-world dynamics and
  causalities. To this end, we introduce MMWorld, a new benchmark for
  multi-discipline, multi-faceted multimodal video understanding. MMWorld
  distinguishes itself from previous video understanding benchmarks with two
  unique advantages: (1) multi-discipline, covering various disciplines that
  often require domain expertise for comprehensive understanding; (2)
  multi-faceted reasoning, including explanation, counterfactual thinking, future
  prediction, etc. MMWorld consists of a human-annotated dataset to evaluate
  MLLMs with questions about the whole videos and a synthetic dataset to analyze
  MLLMs within a single modality of perception. Together, MMWorld encompasses
  1,910 videos across seven broad disciplines and 69 subdisciplines, complete
  with 6,627 question-answer pairs and associated captions. The evaluation
  includes 2 proprietary and 10 open-source MLLMs, which struggle on MMWorld
  (e.g., GPT-4V performs the best with only 52.3\% accuracy), showing large room
  for improvement. Further ablation studies reveal other interesting findings
  such as models' different skill sets from humans. We hope MMWorld can serve as
  an essential step towards world model evaluation in videos.
paper_ai_keywords: Multimodal Language Language Models (MLLMs) World models Real-world dynamics Videos Multimodal video understanding MMWorld Multi-discipline Multi-faceted reasoning Human-annotated dataset Synthetic dataset Question-answer pairs GPT-4V Room for improvement Skill sets