title: Zero-AVSR: Zero-Shot Audio-Visual Speech Recognition with LLMs by
    Learning Language-Agnostic Speech Representations
summary: We explore a novel zero-shot Audio-Visual Speech Recognition (AVSR)
  framework, dubbed Zero-AVSR, which enables speech recognition in target
  languages without requiring any audio-visual speech data in those languages.
  Specifically, we introduce the Audio-Visual Speech Romanizer (AV-Romanizer),
  which learns language-agnostic speech representations by predicting Roman text.
  Then, by leveraging the strong multilingual modeling capabilities of Large
  Language Models (LLMs), we propose converting the predicted Roman text into
  language-specific graphemes, forming the proposed Cascaded Zero-AVSR. Taking it
  a step further, we explore a unified Zero-AVSR approach by directly integrating
  the audio-visual speech representations encoded by the AV-Romanizer into the
  LLM. This is achieved through finetuning the adapter and the LLM using our
  proposed multi-task learning scheme. To capture the wide spectrum of phonetic
  and linguistic diversity, we also introduce a Multilingual Audio-Visual
  Romanized Corpus (MARC) consisting of 2,916 hours of audio-visual speech data
  across 82 languages, along with transcriptions in both language-specific
  graphemes and Roman text. Extensive analysis and experiments confirm that the
  proposed Zero-AVSR framework has the potential to expand language support
  beyond the languages seen during the training of the AV-Romanizer.
paper_authors._id: 67cfa7ca62df3125810efc27
paper_authors.name: Jeong Hun Yeo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-11T10:57:57.858Z
paper_authors._id: 65bb389bc60df511e142ba82
paper_authors.avatarUrl: /avatars/5884ea11d2682ed71585a4025a4248c1.svg
paper_authors.fullname: Jeong Hun Yeo
paper_authors.type: user
paper_authors.user: JeongHun0716
paper_authors._id: 67cfa7ca62df3125810efc28
paper_authors.name: Minsu Kim
paper_authors._id: 67cfa7ca62df3125810efc29
paper_authors.name: Chae Won Kim
paper_authors._id: 67cfa7ca62df3125810efc2a
paper_authors.name: Stavros Petridis
paper_authors._id: 67cfa7ca62df3125810efc2b
paper_authors.name: Yong Man Ro
paper_title: Zero-AVSR: Zero-Shot Audio-Visual Speech Recognition with LLMs by
    Learning Language-Agnostic Speech Representations
paper_summary: We explore a novel zero-shot Audio-Visual Speech Recognition (AVSR)
  framework, dubbed Zero-AVSR, which enables speech recognition in target
  languages without requiring any audio-visual speech data in those languages.
  Specifically, we introduce the Audio-Visual Speech Romanizer (AV-Romanizer),
  which learns language-agnostic speech representations by predicting Roman text.
  Then, by leveraging the strong multilingual modeling capabilities of Large
  Language Models (LLMs), we propose converting the predicted Roman text into
  language-specific graphemes, forming the proposed Cascaded Zero-AVSR. Taking it
  a step further, we explore a unified Zero-AVSR approach by directly integrating
  the audio-visual speech representations encoded by the AV-Romanizer into the
  LLM. This is achieved through finetuning the adapter and the LLM using our
  proposed multi-task learning scheme. To capture the wide spectrum of phonetic
  and linguistic diversity, we also introduce a Multilingual Audio-Visual
  Romanized Corpus (MARC) consisting of 2,916 hours of audio-visual speech data
  across 82 languages, along with transcriptions in both language-specific
  graphemes and Roman text. Extensive analysis and experiments confirm that the
  proposed Zero-AVSR framework has the potential to expand language support
  beyond the languages seen during the training of the AV-Romanizer.
paper_ai_keywords: Audio-Visual Speech Recognition (AVSR) Zero-AVSR Audio-Visual Speech Romanizer (AV-Romanizer) language-agnostic speech representations Roman text Large Language Models (LLMs) Cascaded Zero-AVSR adapter multi-task learning scheme Multilingual Audio-Visual Romanized Corpus (MARC) audio-visual speech data language-specific graphemes phonetic diversity linguistic diversity zero-shot framework