title: Modeling Collaborator: Enabling Subjective Vision Classification With
    Minimal Human Effort via LLM Tool-Use
summary: From content moderation to wildlife conservation, the number of applications
  that require models to recognize nuanced or subjective visual concepts is
  growing. Traditionally, developing classifiers for such concepts requires
  substantial manual effort measured in hours, days, or even months to identify
  and annotate data needed for training. Even with recently proposed Agile
  Modeling techniques, which enable rapid bootstrapping of image classifiers,
  users are still required to spend 30 minutes or more of monotonous, repetitive
  data labeling just to train a single classifier. Drawing on Fiske's Cognitive
  Miser theory, we propose a new framework that alleviates manual effort by
  replacing human labeling with natural language interactions, reducing the total
  effort required to define a concept by an order of magnitude: from labeling
  2,000 images to only 100 plus some natural language interactions. Our framework
  leverages recent advances in foundation models, both large language models and
  vision-language models, to carve out the concept space through conversation and
  by automatically labeling training data points. Most importantly, our framework
  eliminates the need for crowd-sourced annotations. Moreover, our framework
  ultimately produces lightweight classification models that are deployable in
  cost-sensitive scenarios. Across 15 subjective concepts and across 2 public
  image classification datasets, our trained models outperform traditional Agile
  Modeling as well as state-of-the-art zero-shot classification models like
  ALIGN, CLIP, CuPL, and large visual question-answering models like PaLI-X.
paper_authors._id: 65e7ff6cbcb5b0e1e9a8001f
paper_authors.name: Imad Eddine Toubal
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-06T10:38:19.467Z
paper_authors._id: 6410a12ec29887604fb5240f
paper_authors.avatarUrl: /avatars/63d8c6b4ddfd90f59e9b6e2773c49f92.svg
paper_authors.fullname: Imad Eddine Toubal
paper_authors.type: user
paper_authors.user: imadtoubal
paper_authors._id: 65e7ff6cbcb5b0e1e9a80020
paper_authors.name: Aditya Avinash
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-07T09:15:14.842Z
paper_authors._id: 65e907c4acb29a596cc8ee84
paper_authors.avatarUrl: /avatars/4f1d9cfb8a3967ec7c29d58d9342475b.svg
paper_authors.fullname: Aditya Avinash
paper_authors.type: user
paper_authors.user: adityaavin
paper_authors._id: 65e7ff6cbcb5b0e1e9a80021
paper_authors.name: Neil Gordon Alldrin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-07T09:15:17.457Z
paper_authors._id: 65e8aebb3109b25c7641d4fb
paper_authors.avatarUrl: /avatars/bfa0f258d548f0d472bf10c30551ee48.svg
paper_authors.fullname: Neil G. Alldrin
paper_authors.type: user
paper_authors.user: nalldrin
paper_authors._id: 65e7ff6cbcb5b0e1e9a80022
paper_authors.name: Jan Dlabal
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-07T09:15:21.492Z
paper_authors._id: 65e8a3653109b25c763e0ca0
paper_authors.avatarUrl: /avatars/2a7ad565395c775a26105f6eb0881686.svg
paper_authors.fullname: Jan Dlabal
paper_authors.type: user
paper_authors.user: jdlabal
paper_authors._id: 65e7ff6cbcb5b0e1e9a80023
paper_authors.name: Wenlei Zhou
paper_authors._id: 65e7ff6cbcb5b0e1e9a80024
paper_authors.name: Enming Luo
paper_authors._id: 65e7ff6cbcb5b0e1e9a80025
paper_authors.name: Otilia Stretcu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-07T09:15:19.535Z
paper_authors._id: 65e8a55bf4335428792a9c16
paper_authors.avatarUrl: /avatars/f6c2cf3076fa8146d6c1892f65d86c05.svg
paper_authors.fullname: Otilia Stretcu
paper_authors.type: user
paper_authors.user: otiliastr
paper_authors._id: 65e7ff6cbcb5b0e1e9a80026
paper_authors.name: Hao Xiong
paper_authors._id: 65e7ff6cbcb5b0e1e9a80027
paper_authors.name: Chun-Ta Lu
paper_authors._id: 65e7ff6cbcb5b0e1e9a80028
paper_authors.name: Howard Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-06T10:39:38.538Z
paper_authors._id: 642fb509ff56d6c20043ccfd
paper_authors.avatarUrl: /avatars/c22283720bb440e9b521cd141bfcfa96.svg
paper_authors.fullname: Howard Zhou
paper_authors.type: user
paper_authors.user: howardzzh
paper_authors._id: 65e7ff6cbcb5b0e1e9a80029
paper_authors.name: Ranjay Krishna
paper_authors._id: 65e7ff6cbcb5b0e1e9a8002a
paper_authors.name: Ariel Fuxman
paper_authors._id: 65e7ff6cbcb5b0e1e9a8002b
paper_authors.name: Tom Duerig
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-06T10:39:51.452Z
paper_authors._id: 631a4acfc9f8cd19a735a0ab
paper_authors.avatarUrl: /avatars/61cebf77358634876d978d87248a62f3.svg
paper_authors.fullname: Tom Duerig
paper_authors.type: user
paper_authors.user: tduerig
paper_title: Modeling Collaborator: Enabling Subjective Vision Classification With
    Minimal Human Effort via LLM Tool-Use
paper_summary: From content moderation to wildlife conservation, the number of applications
  that require models to recognize nuanced or subjective visual concepts is
  growing. Traditionally, developing classifiers for such concepts requires
  substantial manual effort measured in hours, days, or even months to identify
  and annotate data needed for training. Even with recently proposed Agile
  Modeling techniques, which enable rapid bootstrapping of image classifiers,
  users are still required to spend 30 minutes or more of monotonous, repetitive
  data labeling just to train a single classifier. Drawing on Fiske's Cognitive
  Miser theory, we propose a new framework that alleviates manual effort by
  replacing human labeling with natural language interactions, reducing the total
  effort required to define a concept by an order of magnitude: from labeling
  2,000 images to only 100 plus some natural language interactions. Our framework
  leverages recent advances in foundation models, both large language models and
  vision-language models, to carve out the concept space through conversation and
  by automatically labeling training data points. Most importantly, our framework
  eliminates the need for crowd-sourced annotations. Moreover, our framework
  ultimately produces lightweight classification models that are deployable in
  cost-sensitive scenarios. Across 15 subjective concepts and across 2 public
  image classification datasets, our trained models outperform traditional Agile
  Modeling as well as state-of-the-art zero-shot classification models like
  ALIGN, CLIP, CuPL, and large visual question-answering models like PaLI-X.
paper_ai_keywords: vision-language models cognitive miser theory adaptive computation natural language interactions zero-shot classification large language models state-of-the-art zero-shot classification models ALIGN CLIP CuPL PaLI-X