title: Multilingual and Fully Non-Autoregressive ASR with Large Language Model
    Fusion: A Comprehensive Study
summary: In the era of large models, the autoregressive nature of decoding often
  results in latency serving as a significant bottleneck. We propose a
  non-autoregressive LM-fused ASR system that effectively leverages the
  parallelization capabilities of accelerator hardware. Our approach combines the
  Universal Speech Model (USM) and the PaLM 2 language model in per-segment
  scoring mode, achieving an average relative WER improvement across all
  languages of 10.8% on FLEURS and 3.6% on YouTube captioning. Furthermore, our
  comprehensive ablation study analyzes key parameters such as LLM size, context
  length, vocabulary size, fusion methodology. For instance, we explore the
  impact of LLM size ranging from 128M to 340B parameters on ASR performance.
  This study provides valuable insights into the factors influencing the
  effectiveness of practical large-scale LM-fused speech recognition systems.
paper_authors._id: 65b07feda037572db164a1da
paper_authors.name: W. Ronny Huang
paper_authors._id: 65b07feda037572db164a1db
paper_authors.name: Cyril Allauzen
paper_authors._id: 65b07feda037572db164a1dc
paper_authors.name: Tongzhou Chen
paper_authors._id: 65b07feda037572db164a1dd
paper_authors.name: Kilol Gupta
paper_authors._id: 65b07feda037572db164a1de
paper_authors.name: Ke Hu
paper_authors._id: 65b07feda037572db164a1df
paper_authors.name: James Qin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-24T10:12:25.663Z
paper_authors._id: 64b900a17ec316f84f8396f1
paper_authors.avatarUrl: /avatars/dc24e805995e88368840874f72041a2f.svg
paper_authors.fullname: James Qin
paper_authors.type: user
paper_authors.user: JamesQin
paper_authors._id: 65b07feda037572db164a1e0
paper_authors.name: Yu Zhang
paper_authors._id: 65b07feda037572db164a1e1
paper_authors.name: Yongqiang Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-24T10:13:00.049Z
paper_authors._id: 65662ce24f6ec720173f38d9
paper_authors.avatarUrl: /avatars/3b9d61ab5fe8c793f8feefb27175e641.svg
paper_authors.fullname: Yongqiang Wang
paper_authors.type: user
paper_authors.user: yqwangustc
paper_authors._id: 65b07feda037572db164a1e2
paper_authors.name: Shuo-Yiin Chang
paper_authors._id: 65b07feda037572db164a1e3
paper_authors.name: Tara N. Sainath
paper_title: Multilingual and Fully Non-Autoregressive ASR with Large Language Model
    Fusion: A Comprehensive Study
paper_summary: In the era of large models, the autoregressive nature of decoding often
  results in latency serving as a significant bottleneck. We propose a
  non-autoregressive LM-fused ASR system that effectively leverages the
  parallelization capabilities of accelerator hardware. Our approach combines the
  Universal Speech Model (USM) and the PaLM 2 language model in per-segment
  scoring mode, achieving an average relative WER improvement across all
  languages of 10.8% on FLEURS and 3.6% on YouTube captioning. Furthermore, our
  comprehensive ablation study analyzes key parameters such as LLM size, context
  length, vocabulary size, fusion methodology. For instance, we explore the
  impact of LLM size ranging from 128M to 340B parameters on ASR performance.
  This study provides valuable insights into the factors influencing the
  effectiveness of practical large-scale LM-fused speech recognition systems.
paper_ai_summary: A non-autoregressive LM-fused ASR system using USM and PaLM 2 improves WER across languages and highlights key factors impacting performance.
paper_ai_keywords: non-autoregressive LM-fused ASR Universal Speech Model PaLM 2 WER FLEURS YouTube captioning LLM size context length vocabulary size fusion methodology