title: Scaling and Enhancing LLM-based AVSR: A Sparse Mixture of Projectors
    Approach
summary: Audio-Visual Speech Recognition (AVSR) enhances robustness in noisy
  environments by integrating visual cues. While recent advances integrate Large
  Language Models (LLMs) into AVSR, their high computational cost hinders
  deployment in resource-constrained settings. To address this, we propose
  Llama-SMoP, an efficient Multimodal LLM that employs a Sparse Mixture of
  Projectors (SMoP) module to scale model capacity without increasing inference
  costs. By incorporating sparsely-gated mixture-of-experts (MoE) projectors,
  Llama-SMoP enables the use of smaller LLMs while maintaining strong
  performance. We explore three SMoP configurations and show that Llama-SMoP DEDR
  (Disjoint-Experts, Disjoint-Routers), which uses modality-specific routers and
  experts, achieves superior performance on ASR, VSR, and AVSR tasks. Ablation
  studies confirm its effectiveness in expert activation, scalability, and noise
  robustness.
paper_authors._id: 682f1664cd2f858ae3c8ec33
paper_authors.name: Umberto Cappellazzo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-26T08:18:07.542Z
paper_authors._id: 64903f017b630c141867877f
paper_authors.avatarUrl: /avatars/3c7b248baed446ecc2b6adc2c444320d.svg
paper_authors.fullname: Umberto Cappellazzo
paper_authors.type: user
paper_authors.user: hisoka94
paper_authors._id: 682f1664cd2f858ae3c8ec34
paper_authors.name: Minsu Kim
paper_authors._id: 682f1664cd2f858ae3c8ec35
paper_authors.name: Stavros Petridis
paper_authors._id: 682f1664cd2f858ae3c8ec36
paper_authors.name: Daniele Falavigna
paper_authors._id: 682f1664cd2f858ae3c8ec37
paper_authors.name: Alessio Brutti
paper_title: Scaling and Enhancing LLM-based AVSR: A Sparse Mixture of Projectors
    Approach
paper_summary: Audio-Visual Speech Recognition (AVSR) enhances robustness in noisy
  environments by integrating visual cues. While recent advances integrate Large
  Language Models (LLMs) into AVSR, their high computational cost hinders
  deployment in resource-constrained settings. To address this, we propose
  Llama-SMoP, an efficient Multimodal LLM that employs a Sparse Mixture of
  Projectors (SMoP) module to scale model capacity without increasing inference
  costs. By incorporating sparsely-gated mixture-of-experts (MoE) projectors,
  Llama-SMoP enables the use of smaller LLMs while maintaining strong
  performance. We explore three SMoP configurations and show that Llama-SMoP DEDR
  (Disjoint-Experts, Disjoint-Routers), which uses modality-specific routers and
  experts, achieves superior performance on ASR, VSR, and AVSR tasks. Ablation
  studies confirm its effectiveness in expert activation, scalability, and noise
  robustness.
paper_ai_summary: Llama-SMoP, an efficient multimodal LLM incorporating Sparse Mixture of Projectors, enhances AVSR performance without increasing inference costs through modality-specific routers and experts.
paper_ai_keywords: Large Language Models (LLMs) Multimodal LLM Sparse Mixture of Projectors (SMoP) sparsely-gated mixture-of-experts (MoE) Disjoint-Experts Disjoint-Routers Audio-Visual Speech Recognition (AVSR) Automatic Speech Recognition (ASR) Visual Speech Recognition (VSR)