Prompt Candidates, then Distill: A Teacher-Student Framework for
  LLM-driven Data Annotation
Recently, Large Language Models (LLMs) have demonstrated significant
potential for data annotation, markedly reducing the labor costs associated
with downstream applications. However, existing methods mostly adopt an
aggressive strategy by prompting LLM to determine a single gold label for each
unlabeled sample. Due to the inherent uncertainty within LLMs, they often
produce incorrect labels for difficult samples, severely compromising the data
quality for downstream applications. Motivated by ambiguity aversion in human
behaviors, we propose a novel candidate annotation paradigm wherein large
language models are encouraged to output all possible labels when incurring
uncertainty. To ensure unique labels are provided for downstream tasks, we
develop a teacher-student framework CanDist that distills candidate annotations
with a Small Language Model (SLM). We further provide a rigorous justification
demonstrating that distilling candidate annotations from the teacher LLM offers
superior theoretical guarantees compared to directly using single annotations.
Extensive experiments across six text classification tasks validate the
effectiveness of our proposed method. The source code is available at
https://github.com/MingxuanXia/CanDist.
684ff7e31d9b438aa3957a9b
Mingxuan Xia
claimed_verified
2025-06-16T12:56:05.905Z
65dc040952eca001fd0bb142
/avatars/cd8e54ceef7c9e4a3bb4b0900c47a8b6.svg
Mingxuan Xia
user
MingxuanXia
684ff7e31d9b438aa3957a9c
Haobo Wang
684ff7e31d9b438aa3957a9d
Yixuan Li
684ff7e31d9b438aa3957a9e
Zewei Yu
684ff7e31d9b438aa3957a9f
Jindong Wang
684ff7e31d9b438aa3957aa0
Junbo Zhao
684ff7e31d9b438aa3957aa1
Runze Wu
Prompt Candidates, then Distill: A Teacher-Student Framework for
  LLM-driven Data Annotation
Recently, Large Language Models (LLMs) have demonstrated significant
potential for data annotation, markedly reducing the labor costs associated
with downstream applications. However, existing methods mostly adopt an
aggressive strategy by prompting LLM to determine a single gold label for each
unlabeled sample. Due to the inherent uncertainty within LLMs, they often
produce incorrect labels for difficult samples, severely compromising the data
quality for downstream applications. Motivated by ambiguity aversion in human
behaviors, we propose a novel candidate annotation paradigm wherein large
language models are encouraged to output all possible labels when incurring
uncertainty. To ensure unique labels are provided for downstream tasks, we
develop a teacher-student framework CanDist that distills candidate annotations
with a Small Language Model (SLM). We further provide a rigorous justification
demonstrating that distilling candidate annotations from the teacher LLM offers
superior theoretical guarantees compared to directly using single annotations.
Extensive experiments across six text classification tasks validate the
effectiveness of our proposed method. The source code is available at
https://github.com/MingxuanXia/CanDist.
A novel candidate annotation paradigm using a teacher-student framework improves data quality for下游 applications by encouraging large language models to output multiple labels when uncertain.
Large Language Models candidate annotation paradigm teacher-student framework small language model text classification