title: Thinking Like an Annotator: Generation of Dataset Labeling Instructions
summary: Large-scale datasets are essential to modern day deep learning. Advocates
  argue that understanding these methods requires dataset transparency (e.g.
  "dataset curation, motivation, composition, collection process, etc...").
  However, almost no one has suggested the release of the detailed definitions
  and visual category examples provided to annotators - information critical to
  understanding the structure of the annotations present in each dataset. These
  labels are at the heart of public datasets, yet few datasets include the
  instructions that were used to generate them. We introduce a new task, Labeling
  Instruction Generation, to address missing publicly available labeling
  instructions. In Labeling Instruction Generation, we take a reasonably
  annotated dataset and: 1) generate a set of examples that are visually
  representative of each category in the dataset; 2) provide a text label that
  corresponds to each of the examples. We introduce a framework that requires no
  model training to solve this task and includes a newly created rapid retrieval
  system that leverages a large, pre-trained vision and language model. This
  framework acts as a proxy to human annotators that can help to both generate a
  final labeling instruction set and evaluate its quality. Our framework
  generates multiple diverse visual and text representations of dataset
  categories. The optimized instruction set outperforms our strongest baseline
  across 5 folds by 7.06 mAP for NuImages and 12.9 mAP for COCO.
paper_authors._id: 649a445806690b9593124d36
paper_authors.name: Nadine Chang
paper_authors._id: 649a445806690b9593124d37
paper_authors.name: Francesco Ferroni
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-06-27T02:09:13.659Z
paper_authors._id: 6353d3e95eac2d2efa7501f9
paper_authors.avatarUrl: /avatars/4b063f54000bed4bfb1bfcc3cde1a09e.svg
paper_authors.fullname: Francesco Ferroni
paper_authors.type: user
paper_authors.user: fferroni
paper_authors._id: 649a445806690b9593124d38
paper_authors.name: Michael J. Tarr
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-04-01T02:28:44.148Z
paper_authors._id: 649c2cdc92525b3ac9fe4f98
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/E4QRz0o81ROM4GyYo-xZN.jpeg
paper_authors.fullname: Michael J Tarr
paper_authors.type: user
paper_authors.user: Academic-Mercenary
paper_authors._id: 649a445806690b9593124d39
paper_authors.name: Martial Hebert
paper_authors._id: 649a445806690b9593124d3a
paper_authors.name: Deva Ramanan
paper_title: Thinking Like an Annotator: Generation of Dataset Labeling Instructions
paper_summary: Large-scale datasets are essential to modern day deep learning. Advocates
  argue that understanding these methods requires dataset transparency (e.g.
  "dataset curation, motivation, composition, collection process, etc...").
  However, almost no one has suggested the release of the detailed definitions
  and visual category examples provided to annotators - information critical to
  understanding the structure of the annotations present in each dataset. These
  labels are at the heart of public datasets, yet few datasets include the
  instructions that were used to generate them. We introduce a new task, Labeling
  Instruction Generation, to address missing publicly available labeling
  instructions. In Labeling Instruction Generation, we take a reasonably
  annotated dataset and: 1) generate a set of examples that are visually
  representative of each category in the dataset; 2) provide a text label that
  corresponds to each of the examples. We introduce a framework that requires no
  model training to solve this task and includes a newly created rapid retrieval
  system that leverages a large, pre-trained vision and language model. This
  framework acts as a proxy to human annotators that can help to both generate a
  final labeling instruction set and evaluate its quality. Our framework
  generates multiple diverse visual and text representations of dataset
  categories. The optimized instruction set outperforms our strongest baseline
  across 5 folds by 7.06 mAP for NuImages and 12.9 mAP for COCO.
paper_ai_keywords: dataset curation dataset composition dataset collection process labeling instructions Labeling Instruction Generation pre-trained vision and language model rapid retrieval system mAP (mean Average Precision)