title: Simple Semi-supervised Knowledge Distillation from Vision-Language
    Models via texttt{D}ual-texttt{H}ead
    texttt{O}ptimization
summary: Vision-language models (VLMs) have achieved remarkable success across diverse
  tasks by leveraging rich textual information with minimal labeled data.
  However, deploying such large models remains challenging, particularly in
  resource-constrained environments. Knowledge distillation (KD) offers a
  well-established solution to this problem; however, recent KD approaches from
  VLMs often involve multi-stage training or additional tuning, increasing
  computational overhead and optimization complexity. In this paper, we propose
  texttt{D}ual-texttt{H}ead
  texttt{O}ptimization (texttt{DHO}) -- a simple yet
  effective KD framework that transfers knowledge from VLMs to compact,
  task-specific models in semi-supervised settings. Specifically, we introduce
  dual prediction heads that independently learn from labeled data and teacher
  predictions, and propose to linearly combine their outputs during inference. We
  observe that DHO mitigates gradient conflicts between supervised and
  distillation signals, enabling more effective feature learning than single-head
  KD baselines. As a result, extensive experiments show that DHO
  consistently outperforms baselines across multiple domains and fine-grained
  datasets. Notably, on ImageNet, it achieves state-of-the-art performance,
  improving accuracy by 3% and 0.1% with 1% and 10% labeled data, respectively,
  while using fewer parameters.
paper_authors._id: 6829dcab0daa5ccc817e6ec8
paper_authors.name: Seongjae Kang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-19T17:47:06.348Z
paper_authors._id: 6357a08f8ed056fa1ccd3b38
paper_authors.avatarUrl: /avatars/07d4ca8f3197a6945ad71e6150801135.svg
paper_authors.fullname: Seongjae Kang
paper_authors.type: user
paper_authors.user: erjui
paper_authors._id: 6829dcab0daa5ccc817e6ec9
paper_authors.name: Dong Bok Lee
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-19T06:42:58.152Z
paper_authors._id: 64f000769e7770db74d44bba
paper_authors.avatarUrl: /avatars/d015820380ffb823b1b35df64dcd3457.svg
paper_authors.fullname: Dong-Bok Lee
paper_authors.type: user
paper_authors.user: dongboklee
paper_authors._id: 6829dcab0daa5ccc817e6eca
paper_authors.name: Hyungjoon Jang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-19T14:20:23.560Z
paper_authors._id: 633ad9e4976a7d6910e84d15
paper_authors.avatarUrl: /avatars/bb464e7503d6e865d1c351430982f7dd.svg
paper_authors.fullname: Hyungjoon Jang
paper_authors.type: user
paper_authors.user: hjoon
paper_authors._id: 6829dcab0daa5ccc817e6ecb
paper_authors.name: Sung Ju Hwang
paper_title: Simple Semi-supervised Knowledge Distillation from Vision-Language
    Models via texttt{D}ual-texttt{H}ead
    texttt{O}ptimization
paper_summary: Vision-language models (VLMs) have achieved remarkable success across diverse
  tasks by leveraging rich textual information with minimal labeled data.
  However, deploying such large models remains challenging, particularly in
  resource-constrained environments. Knowledge distillation (KD) offers a
  well-established solution to this problem; however, recent KD approaches from
  VLMs often involve multi-stage training or additional tuning, increasing
  computational overhead and optimization complexity. In this paper, we propose
  texttt{D}ual-texttt{H}ead
  texttt{O}ptimization (texttt{DHO}) -- a simple yet
  effective KD framework that transfers knowledge from VLMs to compact,
  task-specific models in semi-supervised settings. Specifically, we introduce
  dual prediction heads that independently learn from labeled data and teacher
  predictions, and propose to linearly combine their outputs during inference. We
  observe that DHO mitigates gradient conflicts between supervised and
  distillation signals, enabling more effective feature learning than single-head
  KD baselines. As a result, extensive experiments show that DHO
  consistently outperforms baselines across multiple domains and fine-grained
  datasets. Notably, on ImageNet, it achieves state-of-the-art performance,
  improving accuracy by 3% and 0.1% with 1% and 10% labeled data, respectively,
  while using fewer parameters.
paper_ai_summary: Dual-Head Optimization (DHO) framework improves knowledge distillation from vision-language models to compact task-specific models by using dual prediction heads, achieving state-of-the-art performance with fewer parameters.
paper_ai_keywords: vision-language models knowledge distillation dual prediction heads gradient conflicts ImageNet