Task Me Anything
Benchmarks for large multimodal language models (MLMs) now serve to
simultaneously assess the general capabilities of models instead of evaluating
for a specific capability. As a result, when a developer wants to identify
which models to use for their application, they are overwhelmed by the number
of benchmarks and remain uncertain about which benchmark's results are most
reflective of their specific use case. This paper introduces Task-Me-Anything,
a benchmark generation engine which produces a benchmark tailored to a user's
needs. Task-Me-Anything maintains an extendable taxonomy of visual assets and
can programmatically generate a vast number of task instances. Additionally, it
algorithmically addresses user queries regarding MLM performance efficiently
within a computational budget. It contains 113K images, 10K videos, 2K 3D
object assets, over 365 object categories, 655 attributes, and 335
relationships. It can generate 750M image/video question-answering pairs, which
focus on evaluating MLM perceptual capabilities. Task-Me-Anything reveals
critical insights: open-source MLMs excel in object and attribute recognition
but lack spatial and temporal understanding; each model exhibits unique
strengths and weaknesses; larger models generally perform better, though
exceptions exist; and GPT4o demonstrates challenges in recognizing
rotating/moving objects and distinguishing colors.
66712d27713bc068bbbeabde
Jieyu Zhang
admin_assigned
2024-06-18T08:18:13.201Z
640131b08ba76abe4b71b5d0
/avatars/2288b96a9a0ae8f584768f54e098def1.svg
Jieyu Zhang
user
jieyuz2
66712d27713bc068bbbeabdf
Weikai Huang
66712d27713bc068bbbeabe0
Zixian Ma
66712d27713bc068bbbeabe1
Oscar Michel
66712d27713bc068bbbeabe2
Dong He
66712d27713bc068bbbeabe3
Tanmay Gupta
claimed_verified
2025-06-10T09:32:29.545Z
62a108242e30aaf94ec283bb
/avatars/4da19ae99df6a3fa2d81336eb59cdaa7.svg
Tanmay Gupta
user
tanmayg
66712d27713bc068bbbeabe4
Wei-Chiu Ma
66712d27713bc068bbbeabe5
Ali Farhadi
66712d27713bc068bbbeabe6
Aniruddha Kembhavi
66712d27713bc068bbbeabe7
Ranjay Krishna
Task Me Anything
Benchmarks for large multimodal language models (MLMs) now serve to
simultaneously assess the general capabilities of models instead of evaluating
for a specific capability. As a result, when a developer wants to identify
which models to use for their application, they are overwhelmed by the number
of benchmarks and remain uncertain about which benchmark's results are most
reflective of their specific use case. This paper introduces Task-Me-Anything,
a benchmark generation engine which produces a benchmark tailored to a user's
needs. Task-Me-Anything maintains an extendable taxonomy of visual assets and
can programmatically generate a vast number of task instances. Additionally, it
algorithmically addresses user queries regarding MLM performance efficiently
within a computational budget. It contains 113K images, 10K videos, 2K 3D
object assets, over 365 object categories, 655 attributes, and 335
relationships. It can generate 750M image/video question-answering pairs, which
focus on evaluating MLM perceptual capabilities. Task-Me-Anything reveals
critical insights: open-source MLMs excel in object and attribute recognition
but lack spatial and temporal understanding; each model exhibits unique
strengths and weaknesses; larger models generally perform better, though
exceptions exist; and GPT4o demonstrates challenges in recognizing
rotating/moving objects and distinguishing colors.
Task-Me-Anything is a benchmark generation engine tailored to evaluate large multimodal language models' perceptual capabilities, revealing their strengths and weaknesses in various tasks.
multimodal language models benchmark generation engine visual assets task instances algorithmic queries perceptual capabilities object recognition attribute recognition spatial understanding temporal understanding GPT4o