We present a direct speech-to-speech translation (S2ST) model that translates speech from one language to speech in another language without relying on intermediate text generation. We tackle the problem by first applying a self-supervised discrete speech encoder on the target speech and then training a sequence-to-sequence speech-to-unit translation (S2UT) model to predict the discrete representations of the target speech. When target text transcripts are available, we design a joint speech and text training framework that enables the model to generate dual modality output (speech and text) simultaneously in the same inference pass. Experiments on the Fisher Spanish-English dataset show that the proposed framework yields improvement of 6.7 BLEU compared with a baseline direct S2ST model that predicts spectrogram features. When trained without any text transcripts, our model performance is comparable to models that predict spectrograms and are trained with text supervision, showing the potential of our system for translation between unwritten languages 1 . Speech translation aims at converting speech from one language into speech or text in another language. The technology helps bridge the communication barriers between people speaking different languages and can provide access to multimedia content in different languages. Conventional speech-to-text translation (S2T) systems take a cascaded approach by concatenating automatic speech recognition (ASR) and machine translation (MT). In recent years, end-to-end S2T (Brard et al., 2016) is proposed to alleviate the error propagation issue between ASR and MT. These S2T models can be further combined with text-to-speech (TTS) 1 Audio samples are available at https: //facebookresearch.github.io/speech_translation/direct_s2st_units/index.html synthesis to provide both speech and text translation, which allows the technology to be adopted in a wider range of applications. More recently, researchers have started exploring building direct speech-to-speech translation (S2ST) models without relying on text generation as an intermediate step (Jia et al., 2019b, 2021). Direct S2ST has the benefits of lower computational costs and inference latency as fewer decoding steps are needed compared to cascaded systems. In addition, direct S2ST is a natural approach for supporting translation for languages without a writing system (Tjandra et al., 2019; Zhang et al., 2020). Jia et al. (2019b) first addresses the problem by training an attention-based sequence-to-sequence model that maps source speech spectrograms into target spectrograms. Model training is challenging as it requires the model to learn not only the alignment between two languages but also the acoustic and linguistic characteristics of both languages. As a result, there is a performance gap between the direct S2ST system and an S2T+TTS cascaded system. The recent success in self-supervised learning for speech has demonstrated that speech representations learned from a large unlabelled speech corpus can lead to impressive performance on a variety of downstream tasks (Yang et al., 2021) including ASR (Baevski et al., 2020; Hsu et al., 2021), speaker and language identification (Fan et al., 2020), etc. Moreover, discretized speech units obtained from the clustering of self-supervised speech representations allow researchers to take advantage of existing NLP modeling techniques on speech, such as spoken generative language modeling (Lakhotia et al., 2021). In this work, we tackle the challenge of modeling target speech in direct S2ST by predicting self-supervised discrete representations of the target speech instead of mel-spectrogram features. Compared with spectrogram features, self-supervised 3327 discrete units can disentangle linguistic content from speaker identify or prosodic information in speech (Polyak et al., 2021). With the use of discrete units, we can also apply common practice such as beam search during inference. We investigate direct S2ST with discrete units in the scenarios where the source and target transcripts may or may not be available, the latter case being representative of unwritten languages. For the written languages, we present a framework that jointly generates speech and text output by combining S2ST and S2T tasks through a shared encoder and a partially shared decoder. We resolve the length mismatch issue between the speech and text output during decoding with connectionist temporal classification (CTC) (Graves et al., 2006). Experiments show that with the combination of discrete units prediction, speech and text joint training and beam search, our direct S2ST system matches the performance of a cascaded S2T+TTS system. For the unwritten target languages, we first extend the use of discrete units to text-to-speech translation (Zhang et al., 2020) when there are source text transcripts available. Then we show that with multitask learning using both discrete representations for the source and the target speech, it is possible to train a direct S2ST system without the use of any text transcripts. In addition, we measure the system runtime and memory usage during inference and empirically show that the proposed framework is the most efficient compared to the direct S2ST system that predicts spectrogram features or other cascaded systems. The rest of this paper is organized as follows. After introducing background and related work in the next section, we describe our system in detail in Sec. 3. Following this, we present experimental results including objective evaluation on translation quality, subjective evaluation on speech quality, and system benchmark in Sec. 4. Finally Sec. 5 concludes with a discussion of potential future work. Conventional S2ST systems are built by combining either cascaded or end-to-end S2T models with TTS (Lavie et al., 1997; Nakamura et al., 2006). The majority of the speech translation research has focused on the S2T setup. Studies on ASR+MT systems explore better ways to integrate ASR output lattice to MT models (Matusov et al., 2005) in order to alleviate the error propagation issue between the two. End-to-end S2T (Brard et al., 2016) has the potential to resolve the issue, as long as it is properly trained with multitask learning (Weiss et al., 2017), model pre-training (Bahar et al., 2019; Li et al., 2021) or data augmentation (Jia et al., 2019a) to overcome the data scarcity problem. Studies on TTS for S2ST focus more on synthesizing the paralinguistic information transferred from the source speech, such as prosody (Aguero et al., 2006; Anu-manchipalli et al., 2012) and word-level emphasis (Do et al., 2017). On the other hand, Translatotron (Jia et al., 2019b) is an attention-based sequence-to-sequence framework that directly translates mel-spectrogram of the source speech into spectrogram features of the target speech. Multitask learning is essential in facilitating the model to converge, though there is still a performance gap towards a S2T+TTS cascaded system. The follow-up and concurrent work, Translatotron 2 (Jia et al., 2021), addresses the over-generation issue by conditioning the spectrogram synthesizer directly on the output from the auxiliary target phoneme decoder. Kano et al. (2021) propose to build a single deep-learning framework step-by-step by pre-training ASR, MT and TTS models separately and connecting them with Transcoder layers. However, the inference process requires the ASR and MT decoders to complete decoding a full sequence, and thus it loses the latency advantage of a direct S2ST system. Tjandra et al. (2019); Zhang et al. (2020) both investigate direct S2ST models under the unwritten language setup by transforming the target speech into discrete representations through Variational Auto-Encoder (VAE), training a sequence-to-sequence model for translation into target discrete units, and an inverter for converting the units to speech. In this work, we propose to train a transformer-based speech-to-discrete unit model for direct S2ST. We design a text decoding task conditioned on the intermediate representation of the discrete unit decoder in addition to the auxiliary tasks proposed in (Jia et al., 2019b). We choose to use HuBERT (Hsu et al., 2021) to generate the target self-supervised discrete units, since Yang et al. (2021); Lakhotia et al. (2021); Polyak et al. (2021) have shown its superior performance across ASR, spoken language modeling and speech synthesis, compared to other unsupervised representations, including VAE-based representations used in (Tjan-dra et al., 2019; Zhang et al., 2020). Overall, there exists little work on direct S2ST due to the lack of parallel S2ST training data. While Jia et al. (2019b) performs one set of experiments on in-house real-world S2ST data, Jia et al. (2019b, 2021); Tjandra et al. (2019); Zhang et al. (2020); Kano et al. (2021) all take advantage of TTS services to produce synthetic target speech for model training. We follow the same approach and conduct our experiments with single-speaker synthetic target speech. Our proposed system (Fig. 1) is a transformer-based sequence-to-sequence model with a speech encoder and a discrete unit decoder and incorporates auxiliary tasks (shown in dashed lines) similar to (Jia et al., 2019b) during training to facilitate model learning. For written target languages, we further apply target text CTC decoding conditioned on the intermediate representations from the discrete unit decoder for joint speech and text training and generation. Finally, a vocoder is separately trained to convert discrete units into waveform. HuBERT (Hsu et al., 2021) learns speech representations in a self-supervised manner by leveraging k-means clustering on the model's intermediate representations (or the Mel-frequency cepstral co-(a) stacked efficient features for the first iteration) to generate discrete labels of masked audio segments. A HuBERT model pre-trained on an unlabelled speech corpus of the target language can encode the target speech into continuous representations at every 20-ms frame. A k-means algorithm is applied on the learned representations of the unlabelled speech to generate K cluster centroids (Lakhotia et al., 2021; Polyak et al., 2021), which are used to encode target utterances into sequences of cluster indices at every 20-ms. In the end, a target utterance y is represented as [ z 1 , z 2 , ..., z T ] , z i  { 0 , 1 , ..., K  1 } ,  1  i  T , where T is the number of frames. We build the S2UT model by adapting from the transformer model for MT (Vaswani et al., 2017). A stack of 1D-convolutional layers, each with stride 2 and followed by a gated linear unit activation function, is prepended to the transformer layers in the encoder for downsampling the speech input (Synnaeve et al., 2019). As the target sequence is discrete, we train the S2UT model with cross-entropy loss with label smoothing. We explore two strategies for predicting the discrete unit sequence. In the first strategy (Fig. 2(a), dubbed as  stacked ), we apply the concept of reduction factor, r , from TTS (Wang et al., 2017) and generate a K  r vector at every decoding step for predicting r consecutive discrete units. In the second strategy (Fig. 2(b), dubbed as  reduced ), we collapse a consecutive sequence of the same units into one single unit, resulting a sequence of unique discrete units. Both strategies help speed up training and inference time. We follow the design in (Jia et al., 2019b) to incorporate auxiliary tasks with additional attention and decoder modules conditioned on the intermediate layers of the encoder. The target output of the auxiliary tasks can be either phonemes, characters, subword units or any discrete representations of the source or target utterances. These auxiliary tasks are only used during training and not in inference. For written target languages, we add target text CTC decoding conditioned on an intermediate layer from the discrete unit decoder for the model to generate dual modality output. The use of CTC can mitigate the length mismatch between the speech and text output. However, since it only allows monotonic alignment, we rely on the transformer layers that the CTC decoder conditioned on to take care of the reordering from source to target. During training, we do teacher-forcing with the ground truth target discrete unit sequence and compute CTC loss using the teacher-forced intermediate representations from the decoder. During inference, we can perform discrete unit decoding and CTC decoding for text at each decode step simultaneously. We adopt the modified version of the HiFi-GAN neural vocoder (Kong et al., 2020) proposed in (Polyak et al., 2021) for unit-to-waveform conversion. For the stacked discrete unit output, we train the vocoder with only discrete unit sequence and without extra pitch information as the input. For the reduced discrete unit output, we enhance the vocoder with a lightweight duration prediction module from Fastspeech 2 (Ren et al., 2020), which consists of two 1D-convolutional layers, each with ReLU activation and followed by layer normalization and dropout, and a linear layer. We train the enhanced vocoder by minimizing the mean square error (MSE) between the module prediction and the ground truth duration of each unit segment in logarithmic domain, together with the generator-discriminator loss from HiFi-GAN. We perform our experiments using the Fisher Spanish-English speech translation corpus (Post et al., 2014) as in (Jia et al., 2019b; Zhang et al., 2020). The dataset consists of 139k sentences (ap-proximately 170 hours) from telephone conversa-train dev dev2 test # samples 126k 4k 4k 3.6k source (hrs) 162.5 4.6 4.7 4.5 target (hrs) 139.3 4.0 3.8 3.9 Table 1: Statistics (number of samples and duration) of the Fisher Spanish-English dataset (Post et al., 2014) after pre-processing tions in Spanish, the corresponding Spanish text transcriptions and their English text translation. As in previous studies on direct S2ST (Jia et al., 2019b, 2021; Zhang et al., 2020), we use a high-quality in-house TTS engine to prepare synthetic target speech with a single female voice as the training targets. We perform all experiments, including the baselines, with the synthetic target speech and do not rely on the TTS engine for other uses. We apply the ASR model described in Sec. 4.4 on the synthetic speech and filter out samples with word error rate (WER) greater than 80. Table 1 lists the statistics of the resulting training set, the two development sets and the test set. S2UT model We use the pre-trained HuBERT Base model 2 trained on Librispeech (Panayotov et al., 2015) for two iterations and follow (Hsu et al., 2021; Lakhotia et al., 2021) to perform k-means with K = 100 on representations from the sixth layer of the model for extracting discrete units for all target English speech. We compute 80-dimensional mel-filterbank features at every 10-ms for the source speech as input to the speech encoder and apply cepstral mean and variance normalization and SpecAugment (Park et al., 2019) with the Librispeech basic policy. The downsampling stack in the speech encoder contains two 1D-convolutional layers with kernel size 5 and 1024 channels, resulting in a downsampling factor of 4 on the input speech. The encoder contains 12 transformer layers with embedding size 256, feed-forward network (FFN) embedding size 2048 and 4 attention heads. The decoder consists of 6 transformer layers with the same embedding size and FFN embedding size as the encoder and 8 attention heads. We explore four targets for the auxiliary tasks: source phonemes ( sp ), target phonemes ( tp ), source characters ( sc ) and target characters ( tc ). For sp 2 https://github.com/pytorch/fairseq/ tree/master/examples/hubert 3330 or sc , we append an attention module and a decoder to the sixth layer of the encoder based on preliminary experimentation. For tp or tc , we attach the attention and the decoder to the eighth layer of the encoder. All multihead attention modules have 4 heads and the decoders have 2 transformer layers, 256-dimensional embeddings and a FFN embedding size of 2048. Each auxiliary loss has a constant weight of 8.0 during training. For written target languages, we condition the CTC decoding on the third layer of the discrete unit decoder. The target text for CTC is encoded as 1k unigram subword units (Kudo, 2018) to guarantee that the text sequence length is shorter than the length of the stacked or reduced discrete unit sequence. The weight on the CTC loss is set to 1.6 during training. We train the models for 400k steps using Adam with  1 = 0 . 9 ,  2 = 0 . 98 , (cid:15) = 10  8 , label smoothing 0 . 2 , and apply an inverse square root learning rate decay schedule with 10k warmup steps. All other hyper-parameters, such as dropout and learning rate, are tuned on the development set. All models are implemented using FAIRSEQ S2T (Ott et al., 2019; Wang et al., 2020b) 3 . Unit-based vocoder We follow the same vocoder design and training procedure in (Polyak et al., 2021) and incorporate a duration prediction module from Fastspeech 2 (Ren et al., 2020). The two 1D-convolutional layers in the module have a filter size of 128 and a kernel size of 3. We apply a dropout of 0.5, and the weight on the MSE loss from the duration prediction module is set to 1.0 during training 4 . The vocoder is trained on the synthetic target speech from the Fisher training set. We build two cascaded baselines, ASR+MT+TTS and S2T+TTS, and one direct S2ST baseline that predicts spectrogram features. All models in the cascaded baselines are trained with character input or output. ASR We train the transformer-based Spanish ASR system with the default hyper-parameters and s2t_transformer_s architecture in FAIRSEQ S2T (Wang et al., 2020b). pytorch/fairseq/tree/main/examples/speech_to_speech . 4 Code for vocoder training is available at https://github.com/facebookresearch/speech-resynthesis/tree/main/examples/speech_to_speech_translation MT As the input to the MT model is characters, we follow the default gru_transformer setup in FAIRSEQ (Ott et al., 2019) to prepend a bidirectional recurrent layer with gated recurrent units (GRU) to the transformer encoder to incorporate a larger context (Wang et al., 2020a). S2T We explore both LSTM-based (Weiss et al., 2017) and transformer-based end-to-end S2T systems. The former consists of 8 bidirectional LSTM layers for the encoder and 4 LSTM layers for the decoder. Embedding and hidden state sizes are all 256. The latter has the same model architecture as the S2UT model except that it predicts characters as output. We do not apply pre-training or multitask learning and find that the LSTM-based model works better. TTS The transformer-based TTS model (Li et al., 2019) has 6 transformer layers, 4 attention heads, embedding size 512 and FFN embedding size 2048 for both the encoder and the decoder. We use 32-dimensional layer for the decoder prenet. The model is trained on the English text and the synthetic target speech with a reduction factor of 5 on the output feature frames. The vocoder is a HiFi-GAN model (Kong et al., 2020) fine-tuned on the mel-spectrogram features from teacher-forcing. Transformer Translatotron We implement a transformer-based Translatotron instead of the LSTM architecture in (Jia et al., 2019b) to speed up model training. The model predicts mel-spectrogram features of the target speech and consists of the same speech encoder design as in the S2UT model, the same speech decoder design as in the TTS model for the cascaded baselines, and a fine-tuned HiFi-GAN vocoder (Kong et al., 2020). We use the same auxiliary task setup as in the S2UT model with a constant weight of 0.1 on each auxiliary loss, apply a reduction factor of 5 on the output feature frames and tune the hyper-parameters on the development sets. Preliminary studies show no performance degradation for the transformer-based model compared with our implementation of the LSTM version of the model. We evaluate both the translation quality and the speech quality of the system output. To evaluate the translation quality, we follow the setup in (Jia et al., 2019b; Zhang et al., 2020) to apply ASR on the speech output and compute BLEU scores 3331 BLEU MOS dev dev2 test test ID speech text speech text speech text 1 Synthetic target 88.5 100.0 89.4 100.0 90.5 100.0 3.49  0.14 Cascaded systems: 2 ASR (beam=10) + MT (beam=5) + TTS 42.1 45.1 43.5 46.1 43.9 46.3 3.37  0.15 3 S2T (beam=10) + TTS 38.5 41.1 39.9 42.4 40.2 42.1 3.43  0.14 Direct systems: 4 Transformer Translatotron ( r = 5 , w/ sp, tp ) 25.0 -26.3 -26.2 -5 Transformer Translatotron ( r = 5 , w/ sc, tc ) 32.9 -34.1 -33.2 -3.31  0.11 6 S2UT, no reduction ( r = 1 , w/ sc, tc ) 33.4 -34.6 -34.1 -3.35  0.14 7 S2UT stacked ( r = 5 , w/ sc, tc ) 34.0 -34.5 -34.4 -Direct systems with dual modality output: 8 S2UT stacked + CTC ( r = 5 , w/ sc, tc ) 34.4 36.4 36.4 37.9 34.4 35.8 3.32  0.14 9 S2UT reduced + CTC (w/ sc, tc ), beam=1 36.8 40.0 38.4 41.5 38.5 40.7 10 S2UT reduced + CTC (w/ sc, tc ), beam=10 38.2 41.3 39.5 42.2 39.9 41.9 3.41  0.14 From the literature  : 11 Translatotron (Jia et al., 2019b) 24.8 -26.5 -25.6 -3.69  0.07 12 + pre-trained encoder (Jia et al., 2019b) 30.1 -31.5 -31.1 -13 Translatotron 2 (Jia et al., 2021) --37.0 -3.98  0.08 14 + data augmentation (Jia et al., 2021) --40.3 -3.79  0.09 Table 2: Results from systems using target transcripts during training. We adopt an open-sourced English ASR model 5 built with the combination of wav2vec 2.0 pre-training and self-training (Xu et al., 2021). The model, which is pre-trained on Libri-Light (Kahn et al., 2020) and fine-tuned on full Librispeech (Panayotov et al., 2015), achieves WER of 1.9 and 3.9 on the Librispeech test-clean and other sets, respectively. As the ASR output is in lowercase and without punctuation except apostrophes, we normalize the reference text before computing BLEU using SACREBLEU (Post, 2018) 6 . In addition to measuring the translation quality via an objective metric, we conduct human listening tests to collect mean opinion scores (MOS) to evaluate the naturalness of the speech output. We randomly sample 200 utterances from the test set, and each sample is rated by 8 raters on a scale of 15, with 1 being the worst and 5 being the best. We explore model training under both written and unwritten language scenarios. For the former, we take advantage of text transcriptions of source and target speech during S2UT model training. For the latter, we focus on the cases where the source is in either a written or unwritten language, while the target language is without a writing system. Source & Target Written Table 2 summarizes the experimental results under the written language setup. In the following discussion, we first focus on the translation content quality evaluated by BLEU. We include the results from (Jia et al., 2019b, 2021) as references (11-14). However, as different ASR models are used for evaluation, we should not directly compare the BLEU scores with our experiments. We also list the BLEU scores evaluated on the synthetic target speech (1) to show the impact of the ASR errors on the evaluation metric. First, we explore using different targets for the auxiliary tasks with transformer Translatotron and see that using characters as targets for the auxiliary tasks gives 7 BLEU gain on the test set compared to phonemes (4 vs. 5). In all following experiments, we use characters as the auxiliary task targets. Second, we compare the proposed S2UT model to transformer Translatotron . We start with the stacked strategy as both models have the same reduction ratio of 5. We can see that S2UT stacked  outperforms the transformer Translatotron by 1.2 BLEU on the test set (5 vs. 7), indicating that discrete units are easier to model than continuous-valued mel-spectrogram features. We also experiment with S2UT training using the full discrete unit sequence ( r = 1 ) and see that a larger reduction factor can speed up training and inference and does not hurt the performance (6 vs. 7). Third, we incorporate target text CTC decoding to the S2UT model and evaluate both speech and text output. Joint training with discrete unit loss and text CTC loss brings an average gain of 1.2 BLEU on the dev sets for S2UT stacked  (7 vs. 8), while the performance on the test set remains the same. Moreover, we see that the reduced strategy is more effective than stacked . When decoding with a beam size of 1, we see 1.4 BLEU improvement on speech output and 1.2 BLEU gain on text output on the test set (8 vs. 9). Finally, we apply beam search on the best setup we find, S2UT reduced  with joint speech and text training and auxiliary tasks, and the resulting direct S2ST system performs on par with the S2T+TTS system (3 vs. 10) and bridges 63% of the gap between transformer Translatotron (5) and the three-stage ASR+MT+TTS cascaded system (2). Compared with the cascaded system, the proposed framework has the advantage of being able to generate consistent speech and text output in one inference pass. We also examine the output from the tc auxiliary task, which can serve as another way to generate translated text from the direct S2ST system. By using ASR decoded text from the speech output as reference, we see a character error rate (CER) of 4.5 for the CTC decoded text and 30.3 for the tc decoded text on the dev set, indicating that the former is more aligned with the generated audio. From the MOS results in Table 2, we see that direct S2ST systems that predict all frames, such as Translatotron and S2UT stacked models, tend to have slightly lower MOS than others. The proposed S2UT reduced system has an MOS close to that for synthetic target (1 vs. 10). The latter can be viewed as the upper bound of the best MOS we can get, since the model is trained with the synthetic speech as target. Source Written, Target Unwritten We explore the unwritten target language setup by starting from the scenario where the source speech has a text writing system. Table 3 summarizes the results. First, we build cascaded systems by combining ASR and text-to-speech translation (Zhang et al., 2020). The latter can be built by either training a TTS model that predicts spectrogram features or a text-to-unit model with source text and target speech in two languages. We refer to the first approach as text-to-spectrogram translation (T2ST) and the second as text-to-unit translation (T2UT). We use the same architecture as the transformer TTS model to train the T2ST model with reduction ratio 2, and the same setup as the MT model to train the T2UT model with reduced unit sequences. From Table 3, we see that the model that predicts discrete units outperforms the one that predicts spectrogram features by 15.1 BLEU on the test set (15 vs. 16), which is another evidence showing that discrete units are easier to model as translation targets than continuous spectrogram features. In fact, ASR+T2UT also outperforms S2T+TTS by 0.8 BLEU on the test set (3 vs. 16), which provides another option for building two-stage cascaded systems. Next, we focus on S2UT reduced  based on the findings from the written language setup for direct S2ST. We find that training an S2UT model with sc auxiliary task can already achieve 88% of the performance from a system trained with both 3333 source and target text (10 vs. 17). This is in contrary to the findings in (Jia et al., 2019b) where training Translatotron with only source transcripts attains 28% of the performance of a system trained with both source and target text. Source & Target Unwritten We extend our experiments to a fully unwritten language setup by training models without using any text transcripts (Table 3). Jia et al. (2019b) has pointed out that the model has difficulty in learning to attend to the input speech when trained without auxiliary tasks. Zhang et al. (2020) addresses the challenge by training with discrete unit targets and shows potential, while it uses labelled speech from languages other than the source or the target to guide the VAE learning for the discrete units. When S2UT reduced  is trained without auxiliary tasks, the performance greatly deteriorates (19). We notice that the model can still generate meaningful text. However, the generated speech does not reflect the content in the source speech, and the 7.4 BLEU score is mostly contributed by the function words. This shows that the discrete unit decoder can learn a language model over the unit sequence, while the challenge is in the attention on the encoder output. To facilitate the S2UT model training, we apply the HuBERT model pre-trained on English to extract discrete representations for the source Spanish speech, and the source units ( su ) are used as an auxiliary task target. The resulting S2UT model achieves only a 1.4 BLEU difference on the test set compared with transformer Translatotron trained with both source and target text supervision (5 vs. 20). This shows that source units are effective in guiding the model to properly learn the attention, and the self-supervised discrete representations can capture basic pronunciations that are transferable across languages. In addition to evaluating the quality of the system output, we examine the efficiency of the models during inference by benchmarking the runtime, total number of floating point operations (FLOPs) and max memory on an Intel Xeon Gold 6230 CPU. We conduct the study with three subsets of 500 samples from the Fisher dev set, one with random samples, one with the shortest and the other one with the longest utterances. Fig. 3 shows the comparison of two direct S2ST systems, the proposed S2UT reduced and transformer Translatotron , one two-stage cascaded system (S2T+TTS) and one three-stage cascaded system (ASR+MT+TTS). For each system, we report the runtime and FLOPs measured by timeit and PyPAPI from all stages, and the maximum memory from any single stage measured by memory-profiler . All metrics are averaged by the total number of samples. For cascaded models we only consider the metrics for model inference at different stages and ignore any intermediate data/IO processing overhead. First, we see that TTS is the bottleneck for cascaded systems, as it takes up the largest percentage of runtime ( > 89% in S2T+TTS and > 81% in ASR+MT+TTS) and contributes to the maximum memory used. The runtime may be improved with the use of non-autoregressive TTS systems. We leave the investigation to future work, as it is also possible to apply non-autoregressive translation 3334 with discrete units. Next, the proposed S2UT reduced model is the most efficient among the four systems across all subsets. Compared to S2T+TTS, our direct system runs 1.5X faster and reduces 47% FLOPs and 55% max memory, while maintaining the same level of translation quality (Table 2). This verifies one of the benefits of direct S2ST systems, which is lower computational costs and inference latency. Lastly, the proposed S2UT reduced can not only produce better translation than transformer Translatotron but also run 1.3X faster and reduce 39% FLOPs and 51% max memory. This demonstrates an addition advantage of modeling discrete units instead of spectrogram features. We investigate training direct S2ST models with the use of self-supervised discrete representations as targets. We examine model training under both the written and unwritten language scenarios. For the former, we propose a framework with joint speech and text training that performs on par with an S2T+TTS baseline, yet it can run more effi-ciently. We demonstrate the possibility of translating between two unwritten languages by taking advantage of discrete representations of both the source and the target speech for model training. Our empirical benchmark shows that the proposed direct S2ST system with discrete units is the most efficient during inference compared with a direct S2ST model that predicts spectrogram features or other cascaded systems. With the recent release of large-scale S2S dataset (Wang et al., 2021), we plan to investigate the proposed framework with real data in the future. Another important aspect in generating speech output is the voice and prosody. In our work, we focus on content translation and leave the para-linguistic aspect of speech translation to future work. We use an open-sourced ASR model for evaluation, so the results should be comparable with all future research in the field that follows the same evaluation protocol. We will also release the code for reproducing the experiments. We would like to thank Jade Copet, Emmanuel Dupoux, Evgeny Kharitonov, Kushal Lakhotia, Abdelrahman Mohamed, Tu Anh Nguyen and Morgane Rivire for helpful discussions on discrete
method objective method abstain result abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain method abstain method objective method abstain result abstain result objective method objective other other other other other other other other other other other objective abstain other other other method abstain method abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain method abstain abstain abstain method abstain abstain method abstain method method method abstain method method method abstain other abstain abstain abstain abstain method method abstain abstain abstain method other method abstain abstain method abstain abstain method abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain other abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain result abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain result abstain abstain abstain abstain result result abstain abstain result result result abstain abstain result result abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain objective objective objective abstain abstain method result abstain other