  This paper proposes a direct text to speech translation system using discrete
acoustic units. This framework employs text in different source languages as
input to generate speech in the target language without the need for text
transcriptions in this language. Motivated by the success of acoustic units in
previous works for direct speech to speech translation systems, we use the same
pipeline to extract the acoustic units using a speech encoder combined with a
clustering algorithm. Once units are obtained, an encoder-decoder architecture
is trained to predict them. Then a vocoder generates speech from units. Our
approach for direct text to speech translation was tested on the new CVSS
corpus with two different text mBART models employed as initialisation. The
systems presented report competitive performance for most of the language pairs
evaluated. Besides, results show a remarkable improvement when initialising our
proposed architecture with a model pre-trained with more languages.

Computation and Language;Machine Learning;Sound;Audio and Speech Processing