Modeling Collaborator: Enabling Subjective Vision Classification With
  Minimal Human Effort via LLM Tool-Use
From content moderation to wildlife conservation, the number of applications
that require models to recognize nuanced or subjective visual concepts is
growing. Traditionally, developing classifiers for such concepts requires
substantial manual effort measured in hours, days, or even months to identify
and annotate data needed for training. Even with recently proposed Agile
Modeling techniques, which enable rapid bootstrapping of image classifiers,
users are still required to spend 30 minutes or more of monotonous, repetitive
data labeling just to train a single classifier. Drawing on Fiske's Cognitive
Miser theory, we propose a new framework that alleviates manual effort by
replacing human labeling with natural language interactions, reducing the total
effort required to define a concept by an order of magnitude: from labeling
2,000 images to only 100 plus some natural language interactions. Our framework
leverages recent advances in foundation models, both large language models and
vision-language models, to carve out the concept space through conversation and
by automatically labeling training data points. Most importantly, our framework
eliminates the need for crowd-sourced annotations. Moreover, our framework
ultimately produces lightweight classification models that are deployable in
cost-sensitive scenarios. Across 15 subjective concepts and across 2 public
image classification datasets, our trained models outperform traditional Agile
Modeling as well as state-of-the-art zero-shot classification models like
ALIGN, CLIP, CuPL, and large visual question-answering models like PaLI-X.
65e7ff6cbcb5b0e1e9a8001f
Imad Eddine Toubal
admin_assigned
2024-03-06T10:38:19.467Z
6410a12ec29887604fb5240f
/avatars/63d8c6b4ddfd90f59e9b6e2773c49f92.svg
Imad Eddine Toubal
user
imadtoubal
65e7ff6cbcb5b0e1e9a80020
Aditya Avinash
claimed_verified
2024-03-07T09:15:14.842Z
65e907c4acb29a596cc8ee84
/avatars/4f1d9cfb8a3967ec7c29d58d9342475b.svg
Aditya Avinash
user
adityaavin
65e7ff6cbcb5b0e1e9a80021
Neil Gordon Alldrin
claimed_verified
2024-03-07T09:15:17.457Z
65e8aebb3109b25c7641d4fb
/avatars/bfa0f258d548f0d472bf10c30551ee48.svg
Neil G. Alldrin
user
nalldrin
65e7ff6cbcb5b0e1e9a80022
Jan Dlabal
claimed_verified
2024-03-07T09:15:21.492Z
65e8a3653109b25c763e0ca0
/avatars/2a7ad565395c775a26105f6eb0881686.svg
Jan Dlabal
user
jdlabal
65e7ff6cbcb5b0e1e9a80023
Wenlei Zhou
65e7ff6cbcb5b0e1e9a80024
Enming Luo
65e7ff6cbcb5b0e1e9a80025
Otilia Stretcu
claimed_verified
2024-03-07T09:15:19.535Z
65e8a55bf4335428792a9c16
/avatars/f6c2cf3076fa8146d6c1892f65d86c05.svg
Otilia Stretcu
user
otiliastr
65e7ff6cbcb5b0e1e9a80026
Hao Xiong
65e7ff6cbcb5b0e1e9a80027
Chun-Ta Lu
65e7ff6cbcb5b0e1e9a80028
Howard Zhou
admin_assigned
2024-03-06T10:39:38.538Z
642fb509ff56d6c20043ccfd
/avatars/c22283720bb440e9b521cd141bfcfa96.svg
Howard Zhou
user
howardzzh
65e7ff6cbcb5b0e1e9a80029
Ranjay Krishna
65e7ff6cbcb5b0e1e9a8002a
Ariel Fuxman
65e7ff6cbcb5b0e1e9a8002b
Tom Duerig
admin_assigned
2024-03-06T10:39:51.452Z
631a4acfc9f8cd19a735a0ab
/avatars/61cebf77358634876d978d87248a62f3.svg
Tom Duerig
user
tduerig
Modeling Collaborator: Enabling Subjective Vision Classification With
  Minimal Human Effort via LLM Tool-Use
From content moderation to wildlife conservation, the number of applications
that require models to recognize nuanced or subjective visual concepts is
growing. Traditionally, developing classifiers for such concepts requires
substantial manual effort measured in hours, days, or even months to identify
and annotate data needed for training. Even with recently proposed Agile
Modeling techniques, which enable rapid bootstrapping of image classifiers,
users are still required to spend 30 minutes or more of monotonous, repetitive
data labeling just to train a single classifier. Drawing on Fiske's Cognitive
Miser theory, we propose a new framework that alleviates manual effort by
replacing human labeling with natural language interactions, reducing the total
effort required to define a concept by an order of magnitude: from labeling
2,000 images to only 100 plus some natural language interactions. Our framework
leverages recent advances in foundation models, both large language models and
vision-language models, to carve out the concept space through conversation and
by automatically labeling training data points. Most importantly, our framework
eliminates the need for crowd-sourced annotations. Moreover, our framework
ultimately produces lightweight classification models that are deployable in
cost-sensitive scenarios. Across 15 subjective concepts and across 2 public
image classification datasets, our trained models outperform traditional Agile
Modeling as well as state-of-the-art zero-shot classification models like
ALIGN, CLIP, CuPL, and large visual question-answering models like PaLI-X.
A framework using language-vision models and conversations automates data labeling and produces efficient classifiers for subjective visual concepts, outperforming traditional methods and zero-shot models.
foundation models large language models vision-language models zero-shot classification ALIGN CLIP CuPL PaLI-X