title: Adaptive Audio-Visual Speech Recognition via Matryoshka-Based Multimodal
    LLMs
summary: Audio-Visual Speech Recognition (AVSR) leverages both audio and visual
  modalities to enhance speech recognition robustness, particularly in noisy
  environments. Recent advancements in Large Language Models (LLMs) have
  demonstrated their effectiveness in speech recognition, including AVSR.
  However, due to the significant length of speech representations, direct
  integration with LLMs imposes substantial computational costs. Prior approaches
  address this by compressing speech representations before feeding them into
  LLMs. However, higher compression ratios often lead to performance degradation,
  necessitating a trade-off between computational efficiency and recognition
  accuracy. To address this challenge, we propose Llama-MTSK, the first
  Matryoshka-based Multimodal LLM for AVSR, which enables flexible adaptation of
  the audio-visual token allocation based on specific computational constraints
  while preserving high performance. Our approach, inspired by Matryoshka
  Representation Learning, encodes audio-visual representations at multiple
  granularities within a single model, eliminating the need to train separate
  models for different compression levels. Moreover, to efficiently fine-tune the
  LLM, we introduce three LoRA-based Matryoshka strategies using global and
  scale-specific LoRA modules. Extensive evaluations on the two largest AVSR
  datasets demonstrate that Llama-MTSK achieves state-of-the-art results,
  matching or surpassing models trained independently at fixed compression
  levels.
paper_authors._id: 67cffd119f703990a8e25925
paper_authors.name: Umberto Cappellazzo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-11T14:19:16.209Z
paper_authors._id: 64903f017b630c141867877f
paper_authors.avatarUrl: /avatars/3c7b248baed446ecc2b6adc2c444320d.svg
paper_authors.fullname: Umberto Cappellazzo
paper_authors.type: user
paper_authors.user: hisoka94
paper_authors._id: 67cffd119f703990a8e25926
paper_authors.name: Minsu Kim
paper_authors._id: 67cffd119f703990a8e25927
paper_authors.name: Stavros Petridis
paper_title: Adaptive Audio-Visual Speech Recognition via Matryoshka-Based Multimodal
    LLMs
paper_summary: Audio-Visual Speech Recognition (AVSR) leverages both audio and visual
  modalities to enhance speech recognition robustness, particularly in noisy
  environments. Recent advancements in Large Language Models (LLMs) have
  demonstrated their effectiveness in speech recognition, including AVSR.
  However, due to the significant length of speech representations, direct
  integration with LLMs imposes substantial computational costs. Prior approaches
  address this by compressing speech representations before feeding them into
  LLMs. However, higher compression ratios often lead to performance degradation,
  necessitating a trade-off between computational efficiency and recognition
  accuracy. To address this challenge, we propose Llama-MTSK, the first
  Matryoshka-based Multimodal LLM for AVSR, which enables flexible adaptation of
  the audio-visual token allocation based on specific computational constraints
  while preserving high performance. Our approach, inspired by Matryoshka
  Representation Learning, encodes audio-visual representations at multiple
  granularities within a single model, eliminating the need to train separate
  models for different compression levels. Moreover, to efficiently fine-tune the
  LLM, we introduce three LoRA-based Matryoshka strategies using global and
  scale-specific LoRA modules. Extensive evaluations on the two largest AVSR
  datasets demonstrate that Llama-MTSK achieves state-of-the-art results,
  matching or surpassing models trained independently at fixed compression
  levels.
paper_ai_summary: Llama-MTSK, a Matryoshka-based Multimodal LLM for AVSR, achieves state-of-the-art results by flexibly adapting audio-visual token allocation and using LoRA-based fine-tuning strategies.
paper_ai_keywords: Large Language Models AVSR audio-visual modalities speech recognition Matryoshka-based multimodal LLM audio-visual token allocation compression levels Matryoshka Representation Learning LoRA-based fine-tuning strategies