title: Towards Achieving Human Parity on End-to-end Simultaneous Speech
    Translation via LLM Agent
summary: In this paper, we present Cross Language Agent -- Simultaneous
  Interpretation, CLASI, a high-quality and human-like Simultaneous Speech
  Translation (SiST) System. Inspired by professional human interpreters, we
  utilize a novel data-driven read-write strategy to balance the translation
  quality and latency. To address the challenge of translating in-domain
  terminologies, CLASI employs a multi-modal retrieving module to obtain relevant
  information to augment the translation. Supported by LLMs, our approach can
  generate error-tolerated translation by considering the input audio, historical
  context, and retrieved information. Experimental results show that our system
  outperforms other systems by significant margins. Aligned with professional
  human interpreters, we evaluate CLASI with a better human evaluation metric,
  valid information proportion (VIP), which measures the amount of information
  that can be successfully conveyed to the listeners. In the real-world
  scenarios, where the speeches are often disfluent, informal, and unclear, CLASI
  achieves VIP of 81.3% and 78.0% for Chinese-to-English and English-to-Chinese
  translation directions, respectively. In contrast, state-of-the-art commercial
  or open-source systems only achieve 35.4% and 41.6%. On the extremely hard
  dataset, where other systems achieve under 13% VIP, CLASI can still achieve 70%
  VIP.
paper_authors._id: 66aaf0de28277564bd72c56c
paper_authors.name: Shanbo Cheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-01T07:00:27.433Z
paper_authors._id: 632ab0407fb39c2b6350c10a
paper_authors.avatarUrl: /avatars/fb02cad2a017654965486418bf370157.svg
paper_authors.fullname: Cheng
paper_authors.type: user
paper_authors.user: Shanbo
paper_authors._id: 66aaf0de28277564bd72c56d
paper_authors.name: Zhichao Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T15:07:51.358Z
paper_authors._id: 6411981aca1bb1e9253b3527
paper_authors.avatarUrl: /avatars/f180b6def7636a9e6955ebcba744d05a.svg
paper_authors.fullname: Zhichao Huang
paper_authors.type: user
paper_authors.user: iceshzc
paper_authors._id: 66aaf0de28277564bd72c56e
paper_authors.name: Tom Ko
paper_authors._id: 66aaf0de28277564bd72c56f
paper_authors.name: Hang Li
paper_authors._id: 66aaf0de28277564bd72c570
paper_authors.name: Ningxin Peng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T15:08:33.028Z
paper_authors._id: 64ae8b41c21d9baacbaa4d98
paper_authors.avatarUrl: /avatars/82a09c5f390d7fb34b7092e03b793155.svg
paper_authors.fullname: Ningxin Peng
paper_authors.type: user
paper_authors.user: nxpeng9235
paper_authors._id: 66aaf0de28277564bd72c571
paper_authors.name: Lu Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-01T07:00:24.877Z
paper_authors._id: 662c7c0a7b24fe598b317539
paper_authors.avatarUrl: /avatars/9c578f8a1b9e5f85bdcfe0dc7465e0b4.svg
paper_authors.fullname: xuuuluuu
paper_authors.type: user
paper_authors.user: xuuuluuu
paper_authors._id: 66aaf0de28277564bd72c572
paper_authors.name: Qini Zhang
paper_title: Towards Achieving Human Parity on End-to-end Simultaneous Speech
    Translation via LLM Agent
paper_summary: In this paper, we present Cross Language Agent -- Simultaneous
  Interpretation, CLASI, a high-quality and human-like Simultaneous Speech
  Translation (SiST) System. Inspired by professional human interpreters, we
  utilize a novel data-driven read-write strategy to balance the translation
  quality and latency. To address the challenge of translating in-domain
  terminologies, CLASI employs a multi-modal retrieving module to obtain relevant
  information to augment the translation. Supported by LLMs, our approach can
  generate error-tolerated translation by considering the input audio, historical
  context, and retrieved information. Experimental results show that our system
  outperforms other systems by significant margins. Aligned with professional
  human interpreters, we evaluate CLASI with a better human evaluation metric,
  valid information proportion (VIP), which measures the amount of information
  that can be successfully conveyed to the listeners. In the real-world
  scenarios, where the speeches are often disfluent, informal, and unclear, CLASI
  achieves VIP of 81.3% and 78.0% for Chinese-to-English and English-to-Chinese
  translation directions, respectively. In contrast, state-of-the-art commercial
  or open-source systems only achieve 35.4% and 41.6%. On the extremely hard
  dataset, where other systems achieve under 13% VIP, CLASI can still achieve 70%
  VIP.
paper_ai_keywords: Cross Language Agent -- Simultaneous Interpretation (CLASI) Simultaneous Speech Translation (ExposureST) read-write strategy multi-modal retrieving module LLM error-tolerated translation valid information proportion (VIP) disfluent informal unclear