title: KOFFVQA: An Objectively Evaluated Free-form VQA Benchmark for Large
    Vision-Language Models in the Korean Language
summary: The recent emergence of Large Vision-Language Models(VLMs) has resulted in a
  variety of different benchmarks for evaluating such models. Despite this, we
  observe that most existing evaluation methods suffer from the fact that they
  either require the model to choose from pre-determined responses, sacrificing
  open-endedness, or evaluate responses using a judge model, resulting in
  subjective and unreliable evaluation. In addition, we observe a lack of
  benchmarks for VLMs in the Korean language, which are necessary as a separate
  metric from more common English language benchmarks, as the performance of
  generative language models can differ significantly based on the language being
  used. Therefore, we present KOFFVQA, a general-purpose free-form visual
  question answering benchmark in the Korean language for the evaluation of VLMs.
  Our benchmark consists of 275 carefully crafted questions each paired with an
  image and grading criteria covering 10 different aspects of VLM performance.
  The grading criteria eliminate the problem of unreliability by allowing the
  judge model to grade each response based on a pre-determined set of rules. By
  defining the evaluation criteria in an objective manner, even a small
  open-source model can be used to evaluate models on our benchmark reliably. In
  addition to evaluating a large number of existing VLMs on our benchmark, we
  also experimentally verify that our method of using pre-existing grading
  criteria for evaluation is much more reliable than existing methods. Our
  evaluation code is available at https://github.com/maum-ai/KOFFVQA
paper_authors._id: 67eb567141abf40cd86e0e15
paper_authors.name: Yoonshik Kim
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-01T07:46:53.729Z
paper_authors._id: 67038a66eb760972bcb62c70
paper_authors.avatarUrl: /avatars/8cc82af8f11cae994bc83f4bd99b51bc.svg
paper_authors.fullname: 김윤식
paper_authors.type: user
paper_authors.user: yoonshik1205
paper_authors._id: 67eb567141abf40cd86e0e16
paper_authors.name: Jaeyoon Jung
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-01T07:46:56.151Z
paper_authors._id: 646484cfb90150b2706df03b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/646484cfb90150b2706df03b/8ocSbXBSbrruhlhcxwzEt.png
paper_authors.fullname: Jaeyoon Jung
paper_authors.type: user
paper_authors.user: lastdefiance20
paper_title: KOFFVQA: An Objectively Evaluated Free-form VQA Benchmark for Large
    Vision-Language Models in the Korean Language
paper_summary: The recent emergence of Large Vision-Language Models(VLMs) has resulted in a
  variety of different benchmarks for evaluating such models. Despite this, we
  observe that most existing evaluation methods suffer from the fact that they
  either require the model to choose from pre-determined responses, sacrificing
  open-endedness, or evaluate responses using a judge model, resulting in
  subjective and unreliable evaluation. In addition, we observe a lack of
  benchmarks for VLMs in the Korean language, which are necessary as a separate
  metric from more common English language benchmarks, as the performance of
  generative language models can differ significantly based on the language being
  used. Therefore, we present KOFFVQA, a general-purpose free-form visual
  question answering benchmark in the Korean language for the evaluation of VLMs.
  Our benchmark consists of 275 carefully crafted questions each paired with an
  image and grading criteria covering 10 different aspects of VLM performance.
  The grading criteria eliminate the problem of unreliability by allowing the
  judge model to grade each response based on a pre-determined set of rules. By
  defining the evaluation criteria in an objective manner, even a small
  open-source model can be used to evaluate models on our benchmark reliably. In
  addition to evaluating a large number of existing VLMs on our benchmark, we
  also experimentally verify that our method of using pre-existing grading
  criteria for evaluation is much more reliable than existing methods. Our
  evaluation code is available at https://github.com/maum-ai/KOFFVQA
paper_ai_summary: KOFFVQA is a Korean language-based visual question answering benchmark for evaluating vision-language models, featuring 275 questions and objective grading criteria to enhance reliability.
paper_ai_keywords: Large Vision-Language Models VLMs visual question answering grading criteria Korean language reliability evaluation benchmark