Whisper-LM: Improving ASR Models with Language Models for Low-Resource
  Languages
Automatic speech recognition systems have undoubtedly advanced with the
integration of multilingual and multitask models such as Whisper, which have
shown a promising ability to understand and process speech across a wide range
of languages. Despite their robustness, these models often fall short in
handling the linguistic distinctions of minority languages. This study
addresses this gap by integrating traditional and novel language models with
fine-tuned Whisper models to raise their performance in less commonly studied
languages. Through rigorous fine-tuning and evaluation across multiple
datasets, we demonstrate substantial improvements in word error rate,
particularly in low-resource scenarios. Our approach not only does take
advantage of the extensive data Whisper was pre-trained on, but also
complements its linguistic adaptability by incorporating language models. We
obtained improvements up to 51\% for in-distribution datasets and up to 34\%
for out-of-distribution sentences using statistical language models, while
large language models provided moderate but consistently robust improvement
across diverse linguistic contexts. The findings reveal that, while the
integration reliably benefits all model sizes, the extent of improvement
varies, highlighting the importance of optimized language model parameters.
Finally, we emphasize the importance of selecting appropriate evaluation
parameters when reporting the results using transformer-based ASR models. In
summary, this research clears the way for more inclusive ASR technologies that
perform better across languages by enriching their linguistic knowledge. For
further implementation details of this study, the technical documentation and
source code are available at http://www.github.com/hitz-zentroa/whisper-lm.
67ef7e3e8d4b8473f157ea59
Xabier de Zuazo
extracted_confirmed
2025-04-04T10:58:52.768Z
621ff334fa5492893dc03d82
https://cdn-avatars.huggingface.co/v1/production/uploads/621ff334fa5492893dc03d82/EAIr-l3O4OeM10f1boLux.jpeg
Xabier de Zuazo
user
zuazo
67ef7e3e8d4b8473f157ea5a
Eva Navas
67ef7e3e8d4b8473f157ea5b
Ibon Saratxaga
67ef7e3e8d4b8473f157ea5c
Inma Hernáez Rioja
extracted_pending
2025-04-04T06:37:52.727Z
65f47c863462e375c27a8439
/avatars/b92fd8a8d7741cf27935e98b083d147f.svg
Inma Hernaez
user
inmahernaez
Whisper-LM: Improving ASR Models with Language Models for Low-Resource
  Languages
Automatic speech recognition systems have undoubtedly advanced with the
integration of multilingual and multitask models such as Whisper, which have
shown a promising ability to understand and process speech across a wide range
of languages. Despite their robustness, these models often fall short in
handling the linguistic distinctions of minority languages. This study
addresses this gap by integrating traditional and novel language models with
fine-tuned Whisper models to raise their performance in less commonly studied
languages. Through rigorous fine-tuning and evaluation across multiple
datasets, we demonstrate substantial improvements in word error rate,
particularly in low-resource scenarios. Our approach not only does take
advantage of the extensive data Whisper was pre-trained on, but also
complements its linguistic adaptability by incorporating language models. We
obtained improvements up to 51\% for in-distribution datasets and up to 34\%
for out-of-distribution sentences using statistical language models, while
large language models provided moderate but consistently robust improvement
across diverse linguistic contexts. The findings reveal that, while the
integration reliably benefits all model sizes, the extent of improvement
varies, highlighting the importance of optimized language model parameters.
Finally, we emphasize the importance of selecting appropriate evaluation
parameters when reporting the results using transformer-based ASR models. In
summary, this research clears the way for more inclusive ASR technologies that
perform better across languages by enriching their linguistic knowledge. For
further implementation details of this study, the technical documentation and
source code are available at http://www.github.com/hitz-zentroa/whisper-lm.
The research enhances automatic speech recognition models for minority languages by integrating traditional and novel language models with fine-tuned Whisper models, leading to substantial improvements in word error rate.
automatic speech recognition multilingual models multitask models Whisper language models fine-tuning word error rate low-resource scenarios in-distribution datasets out-of-distribution sentences statistical language models large language models transformer-based ASR models evaluation parameters