title: Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo
    Labelling
summary: As the size of pre-trained speech recognition models increases, running these
  large models in low-latency or resource-constrained environments becomes
  challenging. In this work, we leverage pseudo-labelling to assemble a
  large-scale open-source dataset which we use to distill the Whisper model into
  a smaller variant, called Distil-Whisper. Using a simple word error rate (WER)
  heuristic, we select only the highest quality pseudo-labels for training. The
  distilled model is 5.8 times faster with 51% fewer parameters, while performing
  to within 1% WER on out-of-distribution test data in a zero-shot transfer
  setting. Distil-Whisper maintains the robustness of the Whisper model to
  difficult acoustic conditions, while being less prone to hallucination errors
  on long-form audio. Distil-Whisper is designed to be paired with Whisper for
  speculative decoding, yielding a 2 times speed-up while mathematically ensuring
  the same outputs as the original model. To facilitate further research in this
  domain, we make our training code, inference code and models publicly
  accessible.
paper_authors._id: 65430c759ce0fa3dfa3c1ec6
paper_authors.name: Sanchit Gandhi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-02T10:08:16.773Z
paper_authors._id: 61f91cf54a8e5a275b2b3e7c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1653243468328-61f91cf54a8e5a275b2b3e7c.jpeg
paper_authors.fullname: Sanchit Gandhi
paper_authors.type: user
paper_authors.user: sanchit-gandhi
paper_authors._id: 65430c759ce0fa3dfa3c1ec7
paper_authors.name: Patrick von Platen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-02T10:07:46.528Z
paper_authors._id: 5dfcb1aada6d0311fd3d5448
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1584435275418-5dfcb1aada6d0311fd3d5448.jpeg
paper_authors.fullname: Patrick von Platen
paper_authors.type: user
paper_authors.user: patrickvonplaten
paper_authors._id: 65430c759ce0fa3dfa3c1ec8
paper_authors.name: Alexander M. Rush
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-02T10:08:05.413Z
paper_authors._id: 5e694f697127cb58dde42fe1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5dd96eb166059660ed1ee413/J21QWWtFOKqhJTF5qIMlD.jpeg
paper_authors.fullname: Sasha Rush
paper_authors.type: user
paper_authors.user: srush
paper_title: Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo
    Labelling
paper_summary: As the size of pre-trained speech recognition models increases, running these
  large models in low-latency or resource-constrained environments becomes
  challenging. In this work, we leverage pseudo-labelling to assemble a
  large-scale open-source dataset which we use to distill the Whisper model into
  a smaller variant, called Distil-Whisper. Using a simple word error rate (WER)
  heuristic, we select only the highest quality pseudo-labels for training. The
  distilled model is 5.8 times faster with 51% fewer parameters, while performing
  to within 1% WER on out-of-distribution test data in a zero-shot transfer
  setting. Distil-Whisper maintains the robustness of the Whisper model to
  difficult acoustic conditions, while being less prone to hallucination errors
  on long-form audio. Distil-Whisper is designed to be paired with Whisper for
  speculative decoding, yielding a 2 times speed-up while mathematically ensuring
  the same outputs as the original model. To facilitate further research in this
  domain, we make our training code, inference code and models publicly
  accessible.
paper_ai_keywords: pre-trained speech recognition models low-latency resource-constrained environments pseudo-labelling open-source dataset distillation Whisper model Distil-Whisper word error rate (WER) zero-shot transfer robustness acoustic conditions hallucination errors long-form audio speculative decoding open-source training code inference code