title: On Large Multimodal Models as Open-World Image Classifiers
summary: Traditional image classification requires a predefined list of semantic
  categories. In contrast, Large Multimodal Models (LMMs) can sidestep this
  requirement by classifying images directly using natural language (e.g.,
  answering the prompt "What is the main object in the image?"). Despite this
  remarkable capability, most existing studies on LMM classification performance
  are surprisingly limited in scope, often assuming a closed-world setting with a
  predefined set of categories. In this work, we address this gap by thoroughly
  evaluating LMM classification performance in a truly open-world setting. We
  first formalize the task and introduce an evaluation protocol, defining various
  metrics to assess the alignment between predicted and ground truth classes. We
  then evaluate 13 models across 10 benchmarks, encompassing prototypical,
  non-prototypical, fine-grained, and very fine-grained classes, demonstrating
  the challenges LMMs face in this task. Further analyses based on the proposed
  metrics reveal the types of errors LMMs make, highlighting challenges related
  to granularity and fine-grained capabilities, showing how tailored prompting
  and reasoning can alleviate them.
paper_authors._id: 67ea45e0cdd38e64b1134ec3
paper_authors.name: Alessandro Conti
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-31T08:11:29.043Z
paper_authors._id: 633f243c13e836a0fc507388
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/633f243c13e836a0fc507388/Td8TWn1q2L78sZN9098AO.jpeg
paper_authors.fullname: Alessandro Conti
paper_authors.type: user
paper_authors.user: altndrr
paper_authors._id: 67ea45e0cdd38e64b1134ec4
paper_authors.name: Massimiliano Mancini
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-31T10:01:57.391Z
paper_authors._id: 62cf293d3200bfd438e81f1f
paper_authors.avatarUrl: /avatars/608c19ee375ef091ca77d7cfbc40e76e.svg
paper_authors.fullname: Massimiliano Mancini
paper_authors.type: user
paper_authors.user: massimilianom
paper_authors._id: 67ea45e0cdd38e64b1134ec5
paper_authors.name: Enrico Fini
paper_authors._id: 67ea45e0cdd38e64b1134ec6
paper_authors.name: Yiming Wang
paper_authors._id: 67ea45e0cdd38e64b1134ec7
paper_authors.name: Paolo Rota
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-31T10:02:22.544Z
paper_authors._id: 62f3b1ea81861bd9bc5c5538
paper_authors.avatarUrl: /avatars/0aef9ac5bfa91b9894166fe3c29925da.svg
paper_authors.fullname: Paolo Rota
paper_authors.type: user
paper_authors.user: paolorota
paper_authors._id: 67ea45e0cdd38e64b1134ec8
paper_authors.name: Elisa Ricci
paper_title: On Large Multimodal Models as Open-World Image Classifiers
paper_summary: Traditional image classification requires a predefined list of semantic
  categories. In contrast, Large Multimodal Models (LMMs) can sidestep this
  requirement by classifying images directly using natural language (e.g.,
  answering the prompt "What is the main object in the image?"). Despite this
  remarkable capability, most existing studies on LMM classification performance
  are surprisingly limited in scope, often assuming a closed-world setting with a
  predefined set of categories. In this work, we address this gap by thoroughly
  evaluating LMM classification performance in a truly open-world setting. We
  first formalize the task and introduce an evaluation protocol, defining various
  metrics to assess the alignment between predicted and ground truth classes. We
  then evaluate 13 models across 10 benchmarks, encompassing prototypical,
  non-prototypical, fine-grained, and very fine-grained classes, demonstrating
  the challenges LMMs face in this task. Further analyses based on the proposed
  metrics reveal the types of errors LMMs make, highlighting challenges related
  to granularity and fine-grained capabilities, showing how tailored prompting
  and reasoning can alleviate them.
paper_ai_summary: LMMs are evaluated for image classification in open-world settings, revealing challenges in handling granularity and fine-grained categories, and suggesting improvements through tailored prompting.
paper_ai_keywords: Large Multimodal Models LMMs open-world setting evaluation protocol metrics prototypical non-prototypical fine-grained very fine-grained tailored prompting reasoning