title: VideoGameBunny: Towards vision assistants for video games
summary: Large multimodal models (LMMs) hold substantial promise across various
  domains, from personal assistance in daily tasks to sophisticated applications
  like medical diagnostics. However, their capabilities have limitations in the
  video game domain, such as challenges with scene understanding, hallucinations,
  and inaccurate descriptions of video game content, especially in open-source
  models. This paper describes the development of VideoGameBunny, a LLaVA-style
  model based on Bunny, specifically tailored for understanding images from video
  games. We release intermediate checkpoints, training logs, and an extensive
  dataset comprising 185,259 video game images from 413 titles, along with
  389,565 image-instruction pairs that include image captions, question-answer
  pairs, and a JSON representation of 16 elements of 136,974 images. Our
  experiments show that our high quality game-related data has the potential to
  make a relatively small model outperform the much larger state-of-the-art model
  LLaVa-1.6-34b (which has more than 4x the number of parameters). Our study
  paves the way for future research in video game understanding on tasks such as
  playing, commentary, and debugging. Code and data are available at
  https://videogamebunny.github.io/
paper_authors._id: 669f38e477380045e45c03a3
paper_authors.name: Mohammad Reza Taesiri
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-07-23T05:07:43.285Z
paper_authors._id: 6039478ab3ecf716b1a5fd4d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6039478ab3ecf716b1a5fd4d/_Thy4E7taiSYBLKxEKJbT.jpeg
paper_authors.fullname: taesiri
paper_authors.type: user
paper_authors.user: taesiri
paper_authors._id: 669f38e477380045e45c03a4
paper_authors.name: Cor-Paul Bezemer
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-07-23T05:08:09.145Z
paper_authors._id: 644feede17b6189cda58575d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/WVo1Ah7xmHEeBOUQpkYgS.png
paper_authors.fullname: Cor-Paul
paper_authors.type: user
paper_authors.user: corpaul
paper_title: VideoGameBunny: Towards vision assistants for video games
paper_summary: Large multimodal models (LMMs) hold substantial promise across various
  domains, from personal assistance in daily tasks to sophisticated applications
  like medical diagnostics. However, their capabilities have limitations in the
  video game domain, such as challenges with scene understanding, hallucinations,
  and inaccurate descriptions of video game content, especially in open-source
  models. This paper describes the development of VideoGameBunny, a LLaVA-style
  model based on Bunny, specifically tailored for understanding images from video
  games. We release intermediate checkpoints, training logs, and an extensive
  dataset comprising 185,259 video game images from 413 titles, along with
  389,565 image-instruction pairs that include image captions, question-answer
  pairs, and a JSON representation of 16 elements of 136,974 images. Our
  experiments show that our high quality game-related data has the potential to
  make a relatively small model outperform the much larger state-of-the-art model
  LLaVa-1.6-34b (which has more than 4x the number of parameters). Our study
  paves the way for future research in video game understanding on tasks such as
  playing, commentary, and debugging. Code and data are available at
  https://videogamebunny.github.io/
paper_ai_keywords: multimodal models (LMMs) scene understanding hallucinations LLaVA-style model controller Bunny video game images image-instruction pairs image captions question-answer pairs JSON representation high quality game-related data relative performance state-of-the-art model LLaVa-1.6-34b video game understanding playing tasks commentary debugging