VideoGameBunny: Towards vision assistants for video games
Large multimodal models (LMMs) hold substantial promise across various
domains, from personal assistance in daily tasks to sophisticated applications
like medical diagnostics. However, their capabilities have limitations in the
video game domain, such as challenges with scene understanding, hallucinations,
and inaccurate descriptions of video game content, especially in open-source
models. This paper describes the development of VideoGameBunny, a LLaVA-style
model based on Bunny, specifically tailored for understanding images from video
games. We release intermediate checkpoints, training logs, and an extensive
dataset comprising 185,259 video game images from 413 titles, along with
389,565 image-instruction pairs that include image captions, question-answer
pairs, and a JSON representation of 16 elements of 136,974 images. Our
experiments show that our high quality game-related data has the potential to
make a relatively small model outperform the much larger state-of-the-art model
LLaVa-1.6-34b (which has more than 4x the number of parameters). Our study
paves the way for future research in video game understanding on tasks such as
playing, commentary, and debugging. Code and data are available at
https://videogamebunny.github.io/
669f38e477380045e45c03a3
Mohammad Reza Taesiri
extracted_confirmed
2024-07-23T05:07:43.285Z
6039478ab3ecf716b1a5fd4d
https://cdn-avatars.huggingface.co/v1/production/uploads/6039478ab3ecf716b1a5fd4d/_Thy4E7taiSYBLKxEKJbT.jpeg
taesiri
user
taesiri
669f38e477380045e45c03a4
Cor-Paul Bezemer
extracted_confirmed
2024-07-23T05:08:09.145Z
644feede17b6189cda58575d
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/WVo1Ah7xmHEeBOUQpkYgS.png
Cor-Paul
user
corpaul
VideoGameBunny: Towards vision assistants for video games
Large multimodal models (LMMs) hold substantial promise across various
domains, from personal assistance in daily tasks to sophisticated applications
like medical diagnostics. However, their capabilities have limitations in the
video game domain, such as challenges with scene understanding, hallucinations,
and inaccurate descriptions of video game content, especially in open-source
models. This paper describes the development of VideoGameBunny, a LLaVA-style
model based on Bunny, specifically tailored for understanding images from video
games. We release intermediate checkpoints, training logs, and an extensive
dataset comprising 185,259 video game images from 413 titles, along with
389,565 image-instruction pairs that include image captions, question-answer
pairs, and a JSON representation of 16 elements of 136,974 images. Our
experiments show that our high quality game-related data has the potential to
make a relatively small model outperform the much larger state-of-the-art model
LLaVa-1.6-34b (which has more than 4x the number of parameters). Our study
paves the way for future research in video game understanding on tasks such as
playing, commentary, and debugging. Code and data are available at
https://videogamebunny.github.io/
VideoGameBunny, a LLaVA-style model, demonstrates superior performance on video game image understanding tasks using high-quality game-related data.
VideoGameBunny LLaVA-style model scene understanding hallucinations Video game images image-instruction pairs image captions question-answer pairs JSON representation model parameters state-of-the-art model LLaVa-1.6-34b