Seal: Advancing Speech Language Models to be Few-Shot Learners
<|reference_start|>Seal: Advancing Speech Language Models to be Few-Shot Learners: Existing auto-regressive language models have demonstrated a remarkable capability to perform a new task with just a few examples in prompt, without requiring any additional training. In order to extend this capability to a multi-modal setting (i.e. speech and language), this paper introduces the Seal model, an abbreviation for speech language model. It incorporates a novel alignment method, in which Kullback-Leibler divergence loss is performed to train a projector that bridges a frozen speech encoder with a frozen language model decoder. The resulting Seal model exhibits robust performance as a few-shot learner on two speech understanding tasks. Additionally, consistency experiments are conducted to validate its robustness on different pre-trained language models.<|reference_end|>
@article{lei2024seal:,
    title={Seal: Advancing Speech Language Models to be Few-Shot Learners},
    author={Shuyu Lei, Lingen Liu, Jiaolong Yang, Yasen Jiao, Yuxiang Yang, Yushu
  Yang and Xiang Guo},
    journal={arXiv preprint arXiv:2407.14875},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.14875},
    primaryClass={cs.CL}
}