title: Spatial Speech Translation: Translating Across Space With Binaural
    Hearables
summary: Imagine being in a crowded space where people speak a different language and
  having hearables that transform the auditory space into your native language,
  while preserving the spatial cues for all speakers. We introduce spatial speech
  translation, a novel concept for hearables that translate speakers in the
  wearer's environment, while maintaining the direction and unique voice
  characteristics of each speaker in the binaural output. To achieve this, we
  tackle several technical challenges spanning blind source separation,
  localization, real-time expressive translation, and binaural rendering to
  preserve the speaker directions in the translated audio, while achieving
  real-time inference on the Apple M2 silicon. Our proof-of-concept evaluation
  with a prototype binaural headset shows that, unlike existing models, which
  fail in the presence of interference, we achieve a BLEU score of up to 22.01
  when translating between languages, despite strong interference from other
  speakers in the environment. User studies further confirm the system's
  effectiveness in spatially rendering the translated speech in previously unseen
  real-world reverberant environments. Taking a step back, this work marks the
  first step towards integrating spatial perception into speech translation.
paper_authors._id: 68151b71df379a1f7e1ce0ef
paper_authors.name: Tuochao Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-04T10:04:34.322Z
paper_authors._id: 65a9d16f3b9e1f0f307c96c3
paper_authors.avatarUrl: /avatars/882a0746a0adb2d7befe563f7fbc6321.svg
paper_authors.fullname: Tuochao Chen
paper_authors.type: user
paper_authors.user: tuochao
paper_authors._id: 68151b71df379a1f7e1ce0f0
paper_authors.name: Qirui Wang
paper_authors._id: 68151b71df379a1f7e1ce0f1
paper_authors.name: Runlin He
paper_authors._id: 68151b71df379a1f7e1ce0f2
paper_authors.name: Shyam Gollakota
paper_title: Spatial Speech Translation: Translating Across Space With Binaural
    Hearables
paper_summary: Imagine being in a crowded space where people speak a different language and
  having hearables that transform the auditory space into your native language,
  while preserving the spatial cues for all speakers. We introduce spatial speech
  translation, a novel concept for hearables that translate speakers in the
  wearer's environment, while maintaining the direction and unique voice
  characteristics of each speaker in the binaural output. To achieve this, we
  tackle several technical challenges spanning blind source separation,
  localization, real-time expressive translation, and binaural rendering to
  preserve the speaker directions in the translated audio, while achieving
  real-time inference on the Apple M2 silicon. Our proof-of-concept evaluation
  with a prototype binaural headset shows that, unlike existing models, which
  fail in the presence of interference, we achieve a BLEU score of up to 22.01
  when translating between languages, despite strong interference from other
  speakers in the environment. User studies further confirm the system's
  effectiveness in spatially rendering the translated speech in previously unseen
  real-world reverberant environments. Taking a step back, this work marks the
  first step towards integrating spatial perception into speech translation.
paper_ai_summary: A binaural spatial speech translation system achieves high translation accuracy while preserving speaker directional cues, even in noisy environments.
paper_ai_keywords: spatial speech translation blind source separation localization real-time expressive translation binaural rendering real-time inference BLEU score binaural headset spatial perception