Adaptive Audio-Visual Speech Recognition via Matryoshka-Based Multimodal
  LLMs
Audio-Visual Speech Recognition (AVSR) leverages both audio and visual
modalities to enhance speech recognition robustness, particularly in noisy
environments. Recent advancements in Large Language Models (LLMs) have
demonstrated their effectiveness in speech recognition, including AVSR.
However, due to the significant length of speech representations, direct
integration with LLMs imposes substantial computational costs. Prior approaches
address this by compressing speech representations before feeding them into
LLMs. However, higher compression ratios often lead to performance degradation,
necessitating a trade-off between computational efficiency and recognition
accuracy. To address this challenge, we propose Llama-MTSK, the first
Matryoshka-based Multimodal LLM for AVSR, which enables flexible adaptation of
the audio-visual token allocation based on specific computational constraints
while preserving high performance. Our approach, inspired by Matryoshka
Representation Learning, encodes audio-visual representations at multiple
granularities within a single model, eliminating the need to train separate
models for different compression levels. Moreover, to efficiently fine-tune the
LLM, we introduce three LoRA-based Matryoshka strategies using global and
scale-specific LoRA modules. Extensive evaluations on the two largest AVSR
datasets demonstrate that Llama-MTSK achieves state-of-the-art results,
matching or surpassing models trained independently at fixed compression
levels.
67cffd119f703990a8e25925
Umberto Cappellazzo
admin_assigned
2025-03-11T14:19:16.209Z
64903f017b630c141867877f
/avatars/3c7b248baed446ecc2b6adc2c444320d.svg
Umberto Cappellazzo
user
hisoka94
67cffd119f703990a8e25926
Minsu Kim
67cffd119f703990a8e25927
Stavros Petridis
Adaptive Audio-Visual Speech Recognition via Matryoshka-Based Multimodal
  LLMs
Audio-Visual Speech Recognition (AVSR) leverages both audio and visual
modalities to enhance speech recognition robustness, particularly in noisy
environments. Recent advancements in Large Language Models (LLMs) have
demonstrated their effectiveness in speech recognition, including AVSR.
However, due to the significant length of speech representations, direct
integration with LLMs imposes substantial computational costs. Prior approaches
address this by compressing speech representations before feeding them into
LLMs. However, higher compression ratios often lead to performance degradation,
necessitating a trade-off between computational efficiency and recognition
accuracy. To address this challenge, we propose Llama-MTSK, the first
Matryoshka-based Multimodal LLM for AVSR, which enables flexible adaptation of
the audio-visual token allocation based on specific computational constraints
while preserving high performance. Our approach, inspired by Matryoshka
Representation Learning, encodes audio-visual representations at multiple
granularities within a single model, eliminating the need to train separate
models for different compression levels. Moreover, to efficiently fine-tune the
LLM, we introduce three LoRA-based Matryoshka strategies using global and
scale-specific LoRA modules. Extensive evaluations on the two largest AVSR
datasets demonstrate that Llama-MTSK achieves state-of-the-art results,
matching or surpassing models trained independently at fixed compression
levels.
Llama-MTSK, a Matryoshka-based Multimodal LLM for AVSR, achieves state-of-the-art results by flexibly adapting audio-visual token allocation and using LoRA-based fine-tuning strategies.
Large Language Models AVSR audio-visual modalities speech recognition Matryoshka-based multimodal LLM audio-visual token allocation compression levels Matryoshka Representation Learning LoRA-based fine-tuning strategies