SSR: Alignment-Aware Modality Connector for Speech Language Models
<|reference_start|>SSR: Alignment-Aware Modality Connector for Speech Language Models: Fusing speech into pre-trained language model (SpeechLM) usually suffers from inefficient encoding of long-form speech and catastrophic forgetting of pre-trained text modality. We propose SSR-Connector (Segmented Speech Representation Connector) for better modality fusion. Leveraging speech-text alignments, our approach segments and compresses speech features to match the granularity of text embeddings. Additionally, we introduce a two-stage training pipeline that includes the distillation and fine-tuning phases to mitigate catastrophic forgetting. SSR-Connector outperforms existing mechanism for speech-text modality fusion, consistently achieving better speech understanding (e.g., +10 accuracy on StoryCloze and +20 on Speech-MMLU) while preserving pre-trained text ability.<|reference_end|>
@article{tan2024ssr:,
    title={SSR: Alignment-Aware Modality Connector for Speech Language Models},
    author={Weiting Tan, Hirofumi Inaguma, Ning Dong, Paden Tomasello, Xutai Ma},
    journal={arXiv preprint arXiv:2410.00168},
    year={2024},
    archivePrefix={arXiv},
    eprint={2410.00168},
    primaryClass={cs.CL cs.SD eess.AS}
}