Multilingual and Fully Non-Autoregressive ASR with Large Language Model
  Fusion: A Comprehensive Study
In the era of large models, the autoregressive nature of decoding often
results in latency serving as a significant bottleneck. We propose a
non-autoregressive LM-fused ASR system that effectively leverages the
parallelization capabilities of accelerator hardware. Our approach combines the
Universal Speech Model (USM) and the PaLM 2 language model in per-segment
scoring mode, achieving an average relative WER improvement across all
languages of 10.8% on FLEURS and 3.6% on YouTube captioning. Furthermore, our
comprehensive ablation study analyzes key parameters such as LLM size, context
length, vocabulary size, fusion methodology. For instance, we explore the
impact of LLM size ranging from 128M to 340B parameters on ASR performance.
This study provides valuable insights into the factors influencing the
effectiveness of practical large-scale LM-fused speech recognition systems.
65b07feda037572db164a1da
W. Ronny Huang
65b07feda037572db164a1db
Cyril Allauzen
65b07feda037572db164a1dc
Tongzhou Chen
65b07feda037572db164a1dd
Kilol Gupta
65b07feda037572db164a1de
Ke Hu
65b07feda037572db164a1df
James Qin
admin_assigned
2024-01-24T10:12:25.663Z
64b900a17ec316f84f8396f1
/avatars/dc24e805995e88368840874f72041a2f.svg
James Qin
user
JamesQin
65b07feda037572db164a1e0
Yu Zhang
65b07feda037572db164a1e1
Yongqiang Wang
admin_assigned
2024-01-24T10:13:00.049Z
65662ce24f6ec720173f38d9
/avatars/3b9d61ab5fe8c793f8feefb27175e641.svg
Yongqiang Wang
user
yqwangustc
65b07feda037572db164a1e2
Shuo-Yiin Chang
65b07feda037572db164a1e3
Tara N. Sainath
Multilingual and Fully Non-Autoregressive ASR with Large Language Model
  Fusion: A Comprehensive Study
In the era of large models, the autoregressive nature of decoding often
results in latency serving as a significant bottleneck. We propose a
non-autoregressive LM-fused ASR system that effectively leverages the
parallelization capabilities of accelerator hardware. Our approach combines the
Universal Speech Model (USM) and the PaLM 2 language model in per-segment
scoring mode, achieving an average relative WER improvement across all
languages of 10.8% on FLEURS and 3.6% on YouTube captioning. Furthermore, our
comprehensive ablation study analyzes key parameters such as LLM size, context
length, vocabulary size, fusion methodology. For instance, we explore the
impact of LLM size ranging from 128M to 340B parameters on ASR performance.
This study provides valuable insights into the factors influencing the
effectiveness of practical large-scale LM-fused speech recognition systems.
A non-autoregressive LM-fused ASR system using USM and PaLM 2 improves WER across languages and highlights key factors impacting performance.
non-autoregressive LM-fused ASR Universal Speech Model PaLM 2 WER FLEURS YouTube captioning LLM size context length vocabulary size fusion methodology