CMU's IWSLT 2024 Simultaneous Speech Translation System
<|reference_start|>CMU's IWSLT 2024 Simultaneous Speech Translation System: This paper describes CMU's submission to the IWSLT 2024 Simultaneous Speech Translation (SST) task for translating English speech to German text in a streaming manner. Our end-to-end speech-to-text (ST) system integrates the WavLM speech encoder, a modality adapter, and the Llama2-7B-Base model as the decoder. We employ a two-stage training approach: initially, we align the representations of speech and text, followed by full fine-tuning. Both stages are trained on MuST-c v2 data with cross-entropy loss. We adapt our offline ST model for SST using a simple fixed hold-n policy. Experiments show that our model obtains an offline BLEU score of 31.1 and a BLEU score of 29.5 under 2 seconds latency on the MuST-C-v2 tst-COMMON.<|reference_end|>
@article{xu2024cmu's,
    title={CMU's IWSLT 2024 Simultaneous Speech Translation System},
    author={Xi Xu and Siqi Ouyang and Brian Yan and Patrick Fernandes and William
  Chen and Lei Li and Graham Neubig and Shinji Watanabe},
    journal={arXiv preprint arXiv:2408.07452},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.07452},
    primaryClass={cs.CL cs.AI}
}