Thinking Like an Annotator: Generation of Dataset Labeling Instructions
Large-scale datasets are essential to modern day deep learning. Advocates
argue that understanding these methods requires dataset transparency (e.g.
"dataset curation, motivation, composition, collection process, etc...").
However, almost no one has suggested the release of the detailed definitions
and visual category examples provided to annotators - information critical to
understanding the structure of the annotations present in each dataset. These
labels are at the heart of public datasets, yet few datasets include the
instructions that were used to generate them. We introduce a new task, Labeling
Instruction Generation, to address missing publicly available labeling
instructions. In Labeling Instruction Generation, we take a reasonably
annotated dataset and: 1) generate a set of examples that are visually
representative of each category in the dataset; 2) provide a text label that
corresponds to each of the examples. We introduce a framework that requires no
model training to solve this task and includes a newly created rapid retrieval
system that leverages a large, pre-trained vision and language model. This
framework acts as a proxy to human annotators that can help to both generate a
final labeling instruction set and evaluate its quality. Our framework
generates multiple diverse visual and text representations of dataset
categories. The optimized instruction set outperforms our strongest baseline
across 5 folds by 7.06 mAP for NuImages and 12.9 mAP for COCO.
649a445806690b9593124d36
Nadine Chang
649a445806690b9593124d37
Francesco Ferroni
extracted_confirmed
2023-06-27T02:09:13.659Z
6353d3e95eac2d2efa7501f9
/avatars/4b063f54000bed4bfb1bfcc3cde1a09e.svg
Francesco Ferroni
user
fferroni
649a445806690b9593124d38
Michael J. Tarr
extracted_confirmed
2024-04-01T02:28:44.148Z
649c2cdc92525b3ac9fe4f98
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/E4QRz0o81ROM4GyYo-xZN.jpeg
Michael J Tarr
user
Academic-Mercenary
649a445806690b9593124d39
Martial Hebert
649a445806690b9593124d3a
Deva Ramanan
Thinking Like an Annotator: Generation of Dataset Labeling Instructions
Large-scale datasets are essential to modern day deep learning. Advocates
argue that understanding these methods requires dataset transparency (e.g.
"dataset curation, motivation, composition, collection process, etc...").
However, almost no one has suggested the release of the detailed definitions
and visual category examples provided to annotators - information critical to
understanding the structure of the annotations present in each dataset. These
labels are at the heart of public datasets, yet few datasets include the
instructions that were used to generate them. We introduce a new task, Labeling
Instruction Generation, to address missing publicly available labeling
instructions. In Labeling Instruction Generation, we take a reasonably
annotated dataset and: 1) generate a set of examples that are visually
representative of each category in the dataset; 2) provide a text label that
corresponds to each of the examples. We introduce a framework that requires no
model training to solve this task and includes a newly created rapid retrieval
system that leverages a large, pre-trained vision and language model. This
framework acts as a proxy to human annotators that can help to both generate a
final labeling instruction set and evaluate its quality. Our framework
generates multiple diverse visual and text representations of dataset
categories. The optimized instruction set outperforms our strongest baseline
across 5 folds by 7.06 mAP for NuImages and 12.9 mAP for COCO.
A novel framework generates labeling instructions for datasets using a pre-trained vision and language model, improving the quality and accessibility of dataset annotations.
Labeling Instruction Generation pre-trained vision and language model rapid retrieval system mAP NuImages COCO