title: Task Me Anything
summary: Benchmarks for large multimodal language models (MLMs) now serve to
  simultaneously assess the general capabilities of models instead of evaluating
  for a specific capability. As a result, when a developer wants to identify
  which models to use for their application, they are overwhelmed by the number
  of benchmarks and remain uncertain about which benchmark's results are most
  reflective of their specific use case. This paper introduces Task-Me-Anything,
  a benchmark generation engine which produces a benchmark tailored to a user's
  needs. Task-Me-Anything maintains an extendable taxonomy of visual assets and
  can programmatically generate a vast number of task instances. Additionally, it
  algorithmically addresses user queries regarding MLM performance efficiently
  within a computational budget. It contains 113K images, 10K videos, 2K 3D
  object assets, over 365 object categories, 655 attributes, and 335
  relationships. It can generate 750M image/video question-answering pairs, which
  focus on evaluating MLM perceptual capabilities. Task-Me-Anything reveals
  critical insights: open-source MLMs excel in object and attribute recognition
  but lack spatial and temporal understanding; each model exhibits unique
  strengths and weaknesses; larger models generally perform better, though
  exceptions exist; and GPT4o demonstrates challenges in recognizing
  rotating/moving objects and distinguishing colors.
paper_authors._id: 66712d27713bc068bbbeabde
paper_authors.name: Jieyu Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-18T08:18:13.201Z
paper_authors._id: 640131b08ba76abe4b71b5d0
paper_authors.avatarUrl: /avatars/2288b96a9a0ae8f584768f54e098def1.svg
paper_authors.fullname: Jieyu Zhang
paper_authors.type: user
paper_authors.user: jieyuz2
paper_authors._id: 66712d27713bc068bbbeabdf
paper_authors.name: Weikai Huang
paper_authors._id: 66712d27713bc068bbbeabe0
paper_authors.name: Zixian Ma
paper_authors._id: 66712d27713bc068bbbeabe1
paper_authors.name: Oscar Michel
paper_authors._id: 66712d27713bc068bbbeabe2
paper_authors.name: Dong He
paper_authors._id: 66712d27713bc068bbbeabe3
paper_authors.name: Tanmay Gupta
paper_authors._id: 66712d27713bc068bbbeabe4
paper_authors.name: Wei-Chiu Ma
paper_authors._id: 66712d27713bc068bbbeabe5
paper_authors.name: Ali Farhadi
paper_authors._id: 66712d27713bc068bbbeabe6
paper_authors.name: Aniruddha Kembhavi
paper_authors._id: 66712d27713bc068bbbeabe7
paper_authors.name: Ranjay Krishna
paper_title: Task Me Anything
paper_summary: Benchmarks for large multimodal language models (MLMs) now serve to
  simultaneously assess the general capabilities of models instead of evaluating
  for a specific capability. As a result, when a developer wants to identify
  which models to use for their application, they are overwhelmed by the number
  of benchmarks and remain uncertain about which benchmark's results are most
  reflective of their specific use case. This paper introduces Task-Me-Anything,
  a benchmark generation engine which produces a benchmark tailored to a user's
  needs. Task-Me-Anything maintains an extendable taxonomy of visual assets and
  can programmatically generate a vast number of task instances. Additionally, it
  algorithmically addresses user queries regarding MLM performance efficiently
  within a computational budget. It contains 113K images, 10K videos, 2K 3D
  object assets, over 365 object categories, 655 attributes, and 335
  relationships. It can generate 750M image/video question-answering pairs, which
  focus on evaluating MLM perceptual capabilities. Task-Me-Anything reveals
  critical insights: open-source MLMs excel in object and attribute recognition
  but lack spatial and temporal understanding; each model exhibits unique
  strengths and weaknesses; larger models generally perform better, though
  exceptions exist; and GPT4o demonstrates challenges in recognizing
  rotating/moving objects and distinguishing colors.
paper_ai_keywords: Task-Me-Anything benchmark generation engine visual assets task instances computational budget image/video question-answering pairs perceptual capabilities object recognition attribute recognition spatial understanding temporal understanding GPT4o