title: Whisper-LM: Improving ASR Models with Language Models for Low-Resource
    Languages
summary: Automatic speech recognition systems have undoubtedly advanced with the
  integration of multilingual and multitask models such as Whisper, which have
  shown a promising ability to understand and process speech across a wide range
  of languages. Despite their robustness, these models often fall short in
  handling the linguistic distinctions of minority languages. This study
  addresses this gap by integrating traditional and novel language models with
  fine-tuned Whisper models to raise their performance in less commonly studied
  languages. Through rigorous fine-tuning and evaluation across multiple
  datasets, we demonstrate substantial improvements in word error rate,
  particularly in low-resource scenarios. Our approach not only does take
  advantage of the extensive data Whisper was pre-trained on, but also
  complements its linguistic adaptability by incorporating language models. We
  obtained improvements up to 51\% for in-distribution datasets and up to 34\%
  for out-of-distribution sentences using statistical language models, while
  large language models provided moderate but consistently robust improvement
  across diverse linguistic contexts. The findings reveal that, while the
  integration reliably benefits all model sizes, the extent of improvement
  varies, highlighting the importance of optimized language model parameters.
  Finally, we emphasize the importance of selecting appropriate evaluation
  parameters when reporting the results using transformer-based ASR models. In
  summary, this research clears the way for more inclusive ASR technologies that
  perform better across languages by enriching their linguistic knowledge. For
  further implementation details of this study, the technical documentation and
  source code are available at http://www.github.com/hitz-zentroa/whisper-lm.
paper_authors._id: 67ef7e3e8d4b8473f157ea59
paper_authors.name: Xabier de Zuazo
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-04-04T10:58:52.768Z
paper_authors._id: 621ff334fa5492893dc03d82
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/621ff334fa5492893dc03d82/EAIr-l3O4OeM10f1boLux.jpeg
paper_authors.fullname: Xabier de Zuazo
paper_authors.type: user
paper_authors.user: zuazo
paper_authors._id: 67ef7e3e8d4b8473f157ea5a
paper_authors.name: Eva Navas
paper_authors._id: 67ef7e3e8d4b8473f157ea5b
paper_authors.name: Ibon Saratxaga
paper_authors._id: 67ef7e3e8d4b8473f157ea5c
paper_authors.name: Inma Hernáez Rioja
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-04-04T06:37:52.727Z
paper_authors._id: 65f47c863462e375c27a8439
paper_authors.avatarUrl: /avatars/b92fd8a8d7741cf27935e98b083d147f.svg
paper_authors.fullname: Inma Hernaez
paper_authors.type: user
paper_authors.user: inmahernaez
paper_title: Whisper-LM: Improving ASR Models with Language Models for Low-Resource
    Languages
paper_summary: Automatic speech recognition systems have undoubtedly advanced with the
  integration of multilingual and multitask models such as Whisper, which have
  shown a promising ability to understand and process speech across a wide range
  of languages. Despite their robustness, these models often fall short in
  handling the linguistic distinctions of minority languages. This study
  addresses this gap by integrating traditional and novel language models with
  fine-tuned Whisper models to raise their performance in less commonly studied
  languages. Through rigorous fine-tuning and evaluation across multiple
  datasets, we demonstrate substantial improvements in word error rate,
  particularly in low-resource scenarios. Our approach not only does take
  advantage of the extensive data Whisper was pre-trained on, but also
  complements its linguistic adaptability by incorporating language models. We
  obtained improvements up to 51\% for in-distribution datasets and up to 34\%
  for out-of-distribution sentences using statistical language models, while
  large language models provided moderate but consistently robust improvement
  across diverse linguistic contexts. The findings reveal that, while the
  integration reliably benefits all model sizes, the extent of improvement
  varies, highlighting the importance of optimized language model parameters.
  Finally, we emphasize the importance of selecting appropriate evaluation
  parameters when reporting the results using transformer-based ASR models. In
  summary, this research clears the way for more inclusive ASR technologies that
  perform better across languages by enriching their linguistic knowledge. For
  further implementation details of this study, the technical documentation and
  source code are available at http://www.github.com/hitz-zentroa/whisper-lm.
paper_ai_summary: The research enhances automatic speech recognition models for minority languages by integrating traditional and novel language models with fine-tuned Whisper models, leading to substantial improvements in word error rate.
paper_ai_keywords: automatic speech recognition multilingual models multitask models Whisper language models fine-tuning word error rate low-resource scenarios in-distribution datasets out-of-distribution sentences statistical language models large language models transformer-based ASR models evaluation parameters