Multimodal Large Language Models with Fusion Low Rank Adaptation for Device Directed Speech Detection
<|reference_start|>Multimodal Large Language Models with Fusion Low Rank Adaptation for Device Directed Speech Detection: Although Large Language Models (LLMs) have shown promise for human-like conversations, they are primarily pre-trained on text data. Incorporating audio or video improves performance, but collecting large-scale multimodal data and pre-training multimodal LLMs is challenging. To this end, we propose a Fusion Low Rank Adaptation (FLoRA) technique that efficiently adapts a pre-trained unimodal LLM to consume new, previously unseen modalities via low rank adaptation. For device-directed speech detection, using FLoRA, the multimodal LLM achieves 22% relative reduction in equal error rate (EER) over the text-only approach and attains performance parity with its full fine-tuning (FFT) counterpart while needing to tune only a fraction of its parameters. Furthermore, with the newly introduced adapter dropout, FLoRA is robust to missing data, improving over FFT by 20% lower EER and 56% lower false accept rate. The proposed approach scales well for model sizes from 16M to 3B parameters.<|reference_end|>
@article{palaskar2024multimodal,
    title={Multimodal Large Language Models with Fusion Low Rank Adaptation for
  Device Directed Speech Detection},
    author={Shruti Palaskar, Oggi Rudovic, Sameer Dharur, Florian Pesce, Gautam
  Krishna, Aswin Sivaraman, Jack Berkowitz, Ahmed Hussen Abdelaziz, Saurabh
  Adya and Ahmed Tewfik},
    journal={arXiv preprint arXiv:2406.09617},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.09617},
    primaryClass={cs.CL cs.HC eess.AS}
}