title: Two Giraffes in a Dirt Field: Using Game Play to Investigate Situation
    Modelling in Large Multimodal Models
summary: While the situation has improved for text-only models, it again seems to be
  the case currently that multimodal (text and image) models develop faster than
  ways to evaluate them. In this paper, we bring a recently developed evaluation
  paradigm from text models to multimodal models, namely evaluation through the
  goal-oriented game (self) play, complementing reference-based and
  preference-based evaluation. Specifically, we define games that challenge a
  model's capability to represent a situation from visual information and align
  such representations through dialogue. We find that the largest closed models
  perform rather well on the games that we define, while even the best
  open-weight models struggle with them. On further analysis, we find that the
  exceptional deep captioning capabilities of the largest models drive some of
  the performance. There is still room to grow for both kinds of models, ensuring
  the continued relevance of the benchmark.
paper_authors._id: 66790802f804642cd62b145b
paper_authors.name: Sherzod Hakimov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-24T13:10:33.090Z
paper_authors._id: 647061e39ad4008a29061769
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/647061e39ad4008a29061769/nHm9LquGI0Nh-MutiBvah.jpeg
paper_authors.fullname: Sherzod Hakimov
paper_authors.type: user
paper_authors.user: sherzod-hakimov
paper_authors._id: 66790802f804642cd62b145c
paper_authors.name: Yerkezhan Abdullayeva
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-24T14:00:02.020Z
paper_authors._id: 654cc79d34188e42e12ef229
paper_authors.avatarUrl: /avatars/226d37b867f8a71a542e5e168bb78b42.svg
paper_authors.fullname: Yerkezhan Abdullayeva
paper_authors.type: user
paper_authors.user: yerkezhan
paper_authors._id: 66790802f804642cd62b145d
paper_authors.name: Kushal Koshti
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-15T13:39:48.966Z
paper_authors._id: 64fb4864c7f04f7cee779241
paper_authors.avatarUrl: /avatars/ac4ace5f8483511f177efb67782878d6.svg
paper_authors.fullname: Koshti
paper_authors.type: user
paper_authors.user: Koshti10
paper_authors._id: 66790802f804642cd62b145e
paper_authors.name: Antonia Schmidt
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-24T14:00:18.260Z
paper_authors._id: 652401544f3337c6b1107b44
paper_authors.avatarUrl: /avatars/d15e7702479fb9e7d8110905543ce44f.svg
paper_authors.fullname: Antonia Schmidt
paper_authors.type: user
paper_authors.user: AntoniaSch
paper_authors._id: 66790802f804642cd62b145f
paper_authors.name: Yan Weiser
paper_authors._id: 66790802f804642cd62b1460
paper_authors.name: Anne Beyer
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-24T14:00:32.483Z
paper_authors._id: 62f529fd2f63f904a0c90788
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62f529fd2f63f904a0c90788/rBpfRUwXVPe3_MJg5EhHO.jpeg
paper_authors.fullname: Anne Beyer
paper_authors.type: user
paper_authors.user: annebeyer
paper_authors._id: 66790802f804642cd62b1461
paper_authors.name: David Schlangen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-24T14:00:38.271Z
paper_authors._id: 60c117aaea206f28132e3457
paper_authors.avatarUrl: /avatars/94d1dab7d7c26f69b2f865bebd13f6ee.svg
paper_authors.fullname: David Schlangen
paper_authors.type: user
paper_authors.user: davids
paper_title: Two Giraffes in a Dirt Field: Using Game Play to Investigate Situation
    Modelling in Large Multimodal Models
paper_summary: While the situation has improved for text-only models, it again seems to be
  the case currently that multimodal (text and image) models develop faster than
  ways to evaluate them. In this paper, we bring a recently developed evaluation
  paradigm from text models to multimodal models, namely evaluation through the
  goal-oriented game (self) play, complementing reference-based and
  preference-based evaluation. Specifically, we define games that challenge a
  model's capability to represent a situation from visual information and align
  such representations through dialogue. We find that the largest closed models
  perform rather well on the games that we define, while even the best
  open-weight models struggle with them. On further analysis, we find that the
  exceptional deep captioning capabilities of the largest models drive some of
  the performance. There is still room to grow for both kinds of models, ensuring
  the continued relevance of the benchmark.
paper_ai_keywords: goal-oriented game self play reference-based evaluation preference-based evaluation dialogue deep captioning