title: Quantization for OpenAI's Whisper Models: A Comparative Analysis
summary: Automated speech recognition (ASR) models have gained prominence for
  applications such as captioning, speech translation, and live transcription.
  This paper studies Whisper and two model variants: one optimized for live
  speech streaming and another for offline transcription. Notably, these models
  have been found to generate hallucinated content, reducing transcription
  reliability. Furthermore, larger model variants exhibit increased latency and
  pose challenges for deployment on resource-constrained devices. This study
  analyzes the similarities and differences between three Whisper models,
  qualitatively examining their distinct capabilities. Next, this study
  quantifies the impact of model quantization on latency and evaluates its
  viability for edge deployment. Using the open source LibriSpeech dataset, this
  paper evaluates the word error rate (WER) along with latency analysis of
  whispercpp using 3 quantization methods (INT4, INT5, INT8). Results show that
  quantization reduces latency by 19\% and model size by 45\%, while preserving
  transcription accuracy. These findings provide insights into the optimal use
  cases of different Whisper models and edge device deployment possibilities. All
  code, datasets, and implementation details are available in a public GitHub
  repository: https://github.com/allisonandreyev/WhisperQuantization.git
paper_authors._id: 67d393660d51cf27930a5e5d
paper_authors.name: Allison Andreyev
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-03-14T02:25:50.445Z
paper_authors._id: 67d3930e4d3a41ed9f7ac71e
paper_authors.avatarUrl: /avatars/4829b1d58376f7987a8891d41108c9c9.svg
paper_authors.fullname: Allison Andreyev
paper_authors.type: user
paper_authors.user: allisonandreyev
paper_title: Quantization for OpenAI's Whisper Models: A Comparative Analysis
paper_summary: Automated speech recognition (ASR) models have gained prominence for
  applications such as captioning, speech translation, and live transcription.
  This paper studies Whisper and two model variants: one optimized for live
  speech streaming and another for offline transcription. Notably, these models
  have been found to generate hallucinated content, reducing transcription
  reliability. Furthermore, larger model variants exhibit increased latency and
  pose challenges for deployment on resource-constrained devices. This study
  analyzes the similarities and differences between three Whisper models,
  qualitatively examining their distinct capabilities. Next, this study
  quantifies the impact of model quantization on latency and evaluates its
  viability for edge deployment. Using the open source LibriSpeech dataset, this
  paper evaluates the word error rate (WER) along with latency analysis of
  whispercpp using 3 quantization methods (INT4, INT5, INT8). Results show that
  quantization reduces latency by 19\% and model size by 45\%, while preserving
  transcription accuracy. These findings provide insights into the optimal use
  cases of different Whisper models and edge device deployment possibilities. All
  code, datasets, and implementation details are available in a public GitHub
  repository: https://github.com/allisonandreyev/WhisperQuantization.git
paper_ai_keywords: Whisper live speech streaming offline transcription hallucinated content latency deployment quantization LibriSpeech dataset word error rate (WER) whispercpp INT4 INT5 INT8 speech recognition (ASR) transcription accuracy edge deployment