is no publicly available bilingual speech dataset where the target speech has the same para-/non-
linguistic characteristics as or imitation of the source speech. As the target speech in the available
datasets is often either synthesized or spoken without semantic translation of the para-/non-linguistic
characteristics in the source speech, the translated speech by a direct S2ST model trained from these
datasets doesn’t convey para-/non-linguistic characteristics of the source speech.
Unsupervised machine translation (UMT) is a task to translate texts with no bilingual text datasets.
Previous research in UMT has employed techniques such as back-translation Sennrich et al. [2015],
where a synthetic translation of the source language is used as “bilingual text datasets” [Artetxe
et al., 2018b, Lample et al., 2018a]. Previous S2ST research, such as Jia et al. [2022a], Lee et al.
[2021a], primarily used supervised learning that rely on bilingual speech datasets. This dependency
Translatotron3