title: PLaD: Preference-based Large Language Model Distillation with
    Pseudo-Preference Pairs
summary: Large Language Models (LLMs) have exhibited impressive capabilities in
  various tasks, yet their vast parameter sizes restrict their applicability in
  resource-constrained settings. Knowledge distillation (KD) offers a viable
  solution by transferring expertise from large teacher models to compact student
  models. However, traditional KD techniques face specific challenges when
  applied to LLMs, including restricted access to LLM outputs, significant
  teacher-student capacity gaps, and the inherited mis-calibration issue. In this
  work, we present PLaD, a novel preference-based LLM distillation framework.
  PLaD exploits the teacher-student capacity discrepancy to generate
  pseudo-preference pairs where teacher outputs are preferred over student
  outputs. Then, PLaD leverages a ranking loss to re-calibrate student's
  estimation of sequence likelihood, which steers the student's focus towards
  understanding the relative quality of outputs instead of simply imitating the
  teacher. PLaD bypasses the need for access to teacher LLM's internal states,
  tackles the student's expressivity limitations, and mitigates the student
  mis-calibration issue. Through extensive experiments on two sequence generation
  tasks and with various LLMs, we demonstrate the effectiveness of our proposed
  PLaD framework.
paper_authors._id: 66612600fb3dfd49f05b2215
paper_authors.name: Rongzhi Zhang
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-06-06T02:59:13.439Z
paper_authors._id: 644a2c4e9a1c5faef7a5dbd8
paper_authors.avatarUrl: /avatars/fbbbc1347f8e423b2477e2506fdb43d9.svg
paper_authors.fullname: Rongzhi Zhang
paper_authors.type: user
paper_authors.user: Solute
paper_authors._id: 66612600fb3dfd49f05b2216
paper_authors.name: Jiaming Shen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-06T10:08:16.564Z
paper_authors._id: 5f7d1ecdb1a525442ff96ee6
paper_authors.avatarUrl: /avatars/cbc7840a6fb318dd37ce91347a7f96e4.svg
paper_authors.fullname: Jiaming Shen
paper_authors.type: user
paper_authors.user: jmshen
paper_authors._id: 66612600fb3dfd49f05b2217
paper_authors.name: Tianqi Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-06T10:08:45.814Z
paper_authors._id: 646784af696e7355f5d5001f
paper_authors.avatarUrl: /avatars/39ce8999c296584ee0bdeb7848eee6d9.svg
paper_authors.fullname: Tianqi Liu
paper_authors.type: user
paper_authors.user: TianqiLiuAI
paper_authors._id: 66612600fb3dfd49f05b2218
paper_authors.name: Haorui Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-06T10:08:58.413Z
paper_authors._id: 63476afcdbe3f8b009e2f758
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1665624822880-noauth.jpeg
paper_authors.fullname: Wang Haorui
paper_authors.type: user
paper_authors.user: JerrySkywalker
paper_authors._id: 66612600fb3dfd49f05b2219
paper_authors.name: Zhen Qin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-18T08:29:42.402Z
paper_authors._id: 64a2bd9245dde1d6ac7b4d81
paper_authors.avatarUrl: /avatars/8a238a832bfcf14e7704a2152b254a0e.svg
paper_authors.fullname: Zhen Qin
paper_authors.type: user
paper_authors.user: paulqin
paper_authors._id: 66612600fb3dfd49f05b221a
paper_authors.name: Feng Han
paper_authors._id: 66612600fb3dfd49f05b221b
paper_authors.name: Jialu Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-06T10:10:23.991Z
paper_authors._id: 64acb90a5d48838462da09ae
paper_authors.avatarUrl: /avatars/dc87416dcb2806a6e0d60bd636024978.svg
paper_authors.fullname: liujialu
paper_authors.type: user
paper_authors.user: Jialuliu
paper_authors._id: 66612600fb3dfd49f05b221c
paper_authors.name: Simon Baumgartner
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-06T10:10:32.982Z
paper_authors._id: 6560f697e0a7720b6ae377bc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6560f697e0a7720b6ae377bc/8b-p-lQ7KV_VygBJ6Vf3_.jpeg
paper_authors.fullname: Simon Baumgartner
paper_authors.type: user
paper_authors.user: sens3
paper_authors._id: 66612600fb3dfd49f05b221d
paper_authors.name: Michael Bendersky
paper_authors._id: 66612600fb3dfd49f05b221e
paper_authors.name: Chao Zhang
paper_title: PLaD: Preference-based Large Language Model Distillation with
    Pseudo-Preference Pairs
paper_summary: Large Language Models (LLMs) have exhibited impressive capabilities in
  various tasks, yet their vast parameter sizes restrict their applicability in
  resource-constrained settings. Knowledge distillation (KD) offers a viable
  solution by transferring expertise from large teacher models to compact student
  models. However, traditional KD techniques face specific challenges when
  applied to LLMs, including restricted access to LLM outputs, significant
  teacher-student capacity gaps, and the inherited mis-calibration issue. In this
  work, we present PLaD, a novel preference-based LLM distillation framework.
  PLaD exploits the teacher-student capacity discrepancy to generate
  pseudo-preference pairs where teacher outputs are preferred over student
  outputs. Then, PLaD leverages a ranking loss to re-calibrate student's
  estimation of sequence likelihood, which steers the student's focus towards
  understanding the relative quality of outputs instead of simply imitating the
  teacher. PLaD bypasses the need for access to teacher LLM's internal states,
  tackles the student's expressivity limitations, and mitigates the student
  mis-calibration issue. Through extensive experiments on two sequence generation
  tasks and with various LLMs, we demonstrate the effectiveness of our proposed
  PLaD framework.
paper_ai_keywords: Knowledge distillation (KD) Large Language Models (LLMs) PLaD (preference-based LLM distillation framework) pseudo-preference pairs ranking loss sequence likelihood sequence generation tasks