A Modular-based Strategy for Mitigating Gradient Conflicts in Simultaneous Speech Translation
<|reference_start|>A Modular-based Strategy for Mitigating Gradient Conflicts in Simultaneous Speech Translation: Simultaneous Speech Translation (SimulST) involves generating target language text while continuously processing streaming speech input, presenting significant real-time challenges. Multi-task learning is often employed to enhance SimulST performance but introduces optimization conflicts between primary and auxiliary tasks, potentially compromising overall efficiency. The existing model-level conflict resolution methods are not well-suited for this task which exacerbates inefficiencies and leads to high GPU memory consumption. To address these challenges, we propose a Modular Gradient Conflict Mitigation (MGCM) strategy that detects conflicts at a finer-grained modular level and resolves them utilizing gradient projection. Experimental results demonstrate that MGCM significantly improves SimulST performance, particularly under medium and high latency conditions, achieving a 0.68 BLEU score gain in offline tasks. Additionally, MGCM reduces GPU memory consumption by over 95\% compared to other conflict mitigation methods, establishing it as a robust solution for SimulST tasks.<|reference_end|>
@article{liu2024a,
    title={A Modular-based Strategy for Mitigating Gradient Conflicts in
  Simultaneous Speech Translation},
    author={Xiaoqian Liu, Yangfan Du, Jianjin Wang, Yuan Ge, Chen Xu, Tong Xiao,
  Guocheng Chen, Jingbo Zhu},
    journal={arXiv preprint arXiv:2409.15911},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.15911},
    primaryClass={cs.CL cs.SD eess.AS}
}