title: PCoreSet: Effective Active Learning through Knowledge Distillation from
    Vision-Language Models
summary: Knowledge distillation (KD) is a widely used framework for training compact,
  task-specific models by leveraging the knowledge of teacher models. However,
  its application to active learning (AL), which aims to minimize annotation
  costs through iterative sample selection, remains underexplored. This gap stems
  from the fact that KD typically assumes access to sufficient labeled data,
  whereas AL operates in data-scarce scenarios where task-specific teacher models
  are often unavailable. In this paper, we introduce ActiveKD, a framework that
  integrates AL with KD by leveraging the zero- and few-shot capabilities of
  large vision-language models (VLMs). A key aspect of ActiveKD is the structured
  prediction bias of VLMs -- i.e., their predictions form clusters in the
  probability space. We regard this structure as an inductive bias of the teacher
  model, capturing generalizable output patterns beneficial to student learning.
  To exploit this bias, we propose Probabilistic CoreSet (PCoreSet), a selection
  strategy that maximizes coverage in the probability space rather than the
  feature space. PCoreSet strategically selects categorically diverse unlabeled
  samples, facilitating more efficient transfer of teacher knowledge under
  limited annotation budgets. Evaluations on 11 datasets show that PCoreSet
  consistently outperforms existing selection methods within the ActiveKD
  framework, advancing research at the intersection of AL and KD.
paper_authors._id: 683fcd9956d0f1cfb34f1d51
paper_authors.name: Seongjae Kang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-06-06T15:42:16.819Z
paper_authors._id: 6357a08f8ed056fa1ccd3b38
paper_authors.avatarUrl: /avatars/07d4ca8f3197a6945ad71e6150801135.svg
paper_authors.fullname: Seongjae Kang
paper_authors.type: user
paper_authors.user: erjui
paper_authors._id: 683fcd9956d0f1cfb34f1d52
paper_authors.name: Dong Bok Lee
paper_authors._id: 683fcd9956d0f1cfb34f1d53
paper_authors.name: Hyungjoon Jang
paper_authors._id: 683fcd9956d0f1cfb34f1d54
paper_authors.name: Dongseop Kim
paper_authors._id: 683fcd9956d0f1cfb34f1d55
paper_authors.name: Sung Ju Hwang
paper_title: PCoreSet: Effective Active Learning through Knowledge Distillation from
    Vision-Language Models
paper_summary: Knowledge distillation (KD) is a widely used framework for training compact,
  task-specific models by leveraging the knowledge of teacher models. However,
  its application to active learning (AL), which aims to minimize annotation
  costs through iterative sample selection, remains underexplored. This gap stems
  from the fact that KD typically assumes access to sufficient labeled data,
  whereas AL operates in data-scarce scenarios where task-specific teacher models
  are often unavailable. In this paper, we introduce ActiveKD, a framework that
  integrates AL with KD by leveraging the zero- and few-shot capabilities of
  large vision-language models (VLMs). A key aspect of ActiveKD is the structured
  prediction bias of VLMs -- i.e., their predictions form clusters in the
  probability space. We regard this structure as an inductive bias of the teacher
  model, capturing generalizable output patterns beneficial to student learning.
  To exploit this bias, we propose Probabilistic CoreSet (PCoreSet), a selection
  strategy that maximizes coverage in the probability space rather than the
  feature space. PCoreSet strategically selects categorically diverse unlabeled
  samples, facilitating more efficient transfer of teacher knowledge under
  limited annotation budgets. Evaluations on 11 datasets show that PCoreSet
  consistently outperforms existing selection methods within the ActiveKD
  framework, advancing research at the intersection of AL and KD.
paper_ai_summary: ActiveKD integrates active learning with knowledge distillation using large vision-language models to efficiently select diverse, unlabeled samples for annotation.
paper_ai_keywords: knowledge distillation active learning task-specific models teacher models zero-shot few-shot large vision-language models structured prediction bias inductive bias Probabilistic CoreSet PCoreSet probability space categorically diverse samples