BLEU score: The translation performance was measured by BLEU on ASR transcriptions from the
translated speech (in lowercase, excluding punctuation marks except for apostrophes), compared
to the corresponding reference translation text. Because ASR can make errors, such BLEU can be
considered as a lower bound of the translation performance. SQuId MOS: There has been a growing
interest in automatic metrics for speech synthesis, with models such as Patton et al. [2016], Lo et al.
[2019], Huang et al. [2022b]. These models predicts a 5-scale subjective mean opinion score (MOS)
in naturalness given a speech. SQuId Sellam et al. [2022] is one of the latest multilingual models
for MOS prediction based on mSLAM Bapna et al. [2022].Average cosine similarity of speaker
embeddings: We use the speaker encoder of the PnG NAT Text-To-Speech model Morioka et al.
[2022] to compute the speaker embedding of the input and of the translated output. Using cosine
Translatotron3