title: Personalised Distillation: Empowering Open-Sourced LLMs with Adaptive
    Learning for Code Generation
summary: With the rise of powerful closed-sourced LLMs (ChatGPT, GPT-4), there are
  increasing interests in distilling the capabilies of close-sourced LLMs to
  smaller open-sourced LLMs. Previous distillation methods usually prompt ChatGPT
  to generate a set of instructions and answers, for the student model to learn.
  However, such standard distillation approach neglects the merits and conditions
  of the student model. Inspired by modern teaching principles, we design a
  personalised distillation process, in which the student attempts to solve a
  task first, then the teacher provides an adaptive refinement for the student to
  improve. Instead of feeding the student with teacher's prior, personalised
  distillation enables personalised learning for the student model, as it only
  learns on examples it makes mistakes upon and learns to improve its own
  solution. On code generation, personalised distillation consistently
  outperforms standard distillation with only one third of the data. With only
  2.5-3K personalised examples that incur a data-collection cost of 4-6$, we
  boost CodeGen-mono-16B by 7% to achieve 36.4% pass@1 and StarCoder by 12.2% to
  achieve 45.8% pass@1 on HumanEval.
paper_authors._id: 65408756e67275697477439f
paper_authors.name: Hailin Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-31T08:42:07.283Z
paper_authors._id: 646c440036505117e22f5b76
paper_authors.avatarUrl: /avatars/a833947c9d452d673d2598eea7363a73.svg
paper_authors.fullname: Hailin Chen
paper_authors.type: user
paper_authors.user: hailche
paper_authors._id: 65408756e6727569747743a0
paper_authors.name: Amrita Saha
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-31T08:42:16.521Z
paper_authors._id: 6461c2905dba83471db3be53
paper_authors.avatarUrl: /avatars/6e36cf86201d590ac729a75d4a439cde.svg
paper_authors.fullname: Amrita Saha
paper_authors.type: user
paper_authors.user: amritasaha87
paper_authors._id: 65408756e6727569747743a1
paper_authors.name: Steven Hoi
paper_authors._id: 65408756e6727569747743a2
paper_authors.name: Shafiq Joty
paper_title: Personalised Distillation: Empowering Open-Sourced LLMs with Adaptive
    Learning for Code Generation
paper_summary: With the rise of powerful closed-sourced LLMs (ChatGPT, GPT-4), there are
  increasing interests in distilling the capabilies of close-sourced LLMs to
  smaller open-sourced LLMs. Previous distillation methods usually prompt ChatGPT
  to generate a set of instructions and answers, for the student model to learn.
  However, such standard distillation approach neglects the merits and conditions
  of the student model. Inspired by modern teaching principles, we design a
  personalised distillation process, in which the student attempts to solve a
  task first, then the teacher provides an adaptive refinement for the student to
  improve. Instead of feeding the student with teacher's prior, personalised
  distillation enables personalised learning for the student model, as it only
  learns on examples it makes mistakes upon and learns to improve its own
  solution. On code generation, personalised distillation consistently
  outperforms standard distillation with only one third of the data. With only
  2.5-3K personalised examples that incur a data-collection cost of 4-6$, we
  boost CodeGen-mono-16B by 7% to achieve 36.4% pass@1 and StarCoder by 12.2% to
  achieve 45.8% pass@1 on HumanEval.
paper_ai_summary: A personalized distillation process improves smaller open-sourced LLMs by focusing on mistakes made by the student model, outperforming standard distillation methods with limited data.
paper_ai_keywords: LLMs ChatGPT GPT-4 distillation methods student model teacher model adaptive refinement code generation HumanEval CodeGen-mono-16B StarCoder