Recent years have seen significant advancements in the field of direct speech-to-speech translation
(S2ST), which aims to translate speech in one language directly into speech in another language
Jia et al. [2022a]. One of the major motivations of direct S2ST is the potential to preserve and
translate para-/non-linguistic speech characteristics, such as speaking styles, emotions, emphasis,
phonation, and vocal bursts. Although these audible characteristics are essential aspects of human
verbal communication, they are often lost in traditional cascade speech translation systems, which
typically involve a conversion to text Lavie et al. [1997], Wahlster [2013], Nakamura et al. [2006].
Previous direct S2ST models have demonstrated the ability to preserve some para-/non-linguistic
characteristics, such as speaker’s voice characteristics, via synthesized targets or speaker modeling
Jia et al. [2019b, 2022a], Lee et al. [2022]. However, a significant challenge remains: currently there
Translatotron3