Current research on spoken language translation (SLT) has to confront with the scarcity of sizeable and publicly available training corpora. This problem hinders the adoption of neural end-to-end approaches, which represent the state of the art in the two parent tasks of SLT: automatic speech recognition and machine translation. To fill this gap, we created MuST-C, a multilingual speech translation corpus whose size and quality will facilitate the training of end-to-end systems for SLT from English into 8 languages. For each target language, MuST-C comprises at least 385 hours of audio recordings from English TED Talks, which are automatically aligned at the sentence level with their manual transcriptions and translations. Together with a description of the corpus creation methodology (scalable to add new data and cover new languages), we provide an empirical verification of its quality and SLT results computed with strong baseline system on each language direction. Besides the increased computing power, the recent surge of neural end-to-end approaches to natural language processing tasks has been stoked by the increased availability of data. For instance, when supported by sizeable training corpora, the robustness and the strong generalization capabilities of neural networks led to their dominance over previous paradigms both in automatic speech recognition (ASR (Chiu et al., 2018)) and machine translation (MT (Bojar et al., 2018)). Compared to its two parent research areas, spoken language translation (SLT) has not shown such a steady progress yet. Despite recent claims by big industry players about the effectiveness of end-to-end learning (Weiss et al., 2017; Jia et al., 2018), its adoption does not yet represent the mainstream solution to the SLT task. One of the main obstacles Corpus Languages Hours Niehues et al. (2018) En  De 273 Kocabiyikoglu et al. (2018) En  Fr 236 Tohyama et al. (2005) En  Jp 182 Paulik and Waibel (2009) En  Es 111 Es  En 105 Post et al. (2013) En  Es 38 Stuker et al. (2012) De  En 37 Shimizu et al. (2014) En  Jp 22 Federmann and Lewis (2017) En  Jp/Zh 22 Bendazzoli and Sandrelli (2005) En  It/Es 18 It  Es Berard et al. (2016) Fr  En 17 Federmann and Lewis (2016) En  Fr/De 8 Woldeyohannis et al. (2017) Am  En 7 Godard et al. (2017) Mboshi  Fr 4 Table 1: Publicly available SLT corpora. to a stable dominance of the end-to-end paradigm also in this area is the scarcity of training corpora. While cascade ASR+MT solutions can exploit the wealth of task-specific data available for each of the two tasks, 1 the situation for end-to-end model training is much less favourable. As shown in Table 1, few publicly available corpora exist, their language coverage is rather limited and, most importantly, their size is often too small (less than 100 hours of translated audio) for training data-hungry neural models. 2 To circumvent the problem, neural SLT approaches currently rely on: 1 In resource-rich conditions, ASR and MT training often builds on thousands of hours of transcribed speech and tens of millions of parallel sentences, respectively. 2 Besides the corpora reported in Table 1, several smaller ( < 4 hours) freely-available datasets have been created ( e.g. the IWSLT evaluation campaign development and test sets from 2010 to 2017 and the Griko-Italian corpus by Boito et al. (2018)). (Weiss et al., 2017; Anastasopoulos and Chiang, 2018; Berard et al., 2018), iii) encoder/decoder pre-training (Bansal et al., 2018; Berard et al., 2018), iv) synthesized speech data (Berard et al., 2016), or v) machine-translated target text data (Berard et al., 2018). Though effective, solutions ii) and iii) assume the availability of ASR and MT data, which is not always guaranteed (especially in low-resource language settings). Solutions iv) and v) , instead, rely on training material derived from sub-optimal automatic data creation/augmentation procedures. This situation calls for initiatives towards the creation of large, high-quality multilingual corpora suitable to explore end-to-end SLT in more favorable conditions similar to condition i) . Along this direction, our contributions are:  A large (  400 hours of speech per language) multilingual corpus for SLT from English into 8 languages (German, Spanish, French, Italian, Dutch, Portuguese, Romanian and Russian);  An empirical verification of its quality;  ASR, MT and SLT results computed with strong baseline systems on each language direction. MuST-C is released under a Creative Commons license, Attribution Non Commercial No Derivatives (CC BY NC ND 4.0 International), and is freely downloadable at mustc.fbk.eu 2 Corpus Creation Methodology Must-C was created pursuing high quality as well as large size, speaker variety (male/female, native/non-native) and coverage in terms of topics and languages. To achieve these objectives, similar to (Niehues et al., 2018), we started from English TED Talks, in which a variety of speakers discuss topics spanning from business to science and entertainment. Most importantly, the fact that TED talks are often manually transcribed and translated sets ideal conditions for creating an SLT corpus from high-quality text material. Although the initial data are similar to those used to build the IWSLT18 corpus, our methodology is different. Inspired by Kocabiyikoglu et al. (2018), it exploits automatic alignment procedures, first at the text level (between transcriptions and translations) and then with the corresponding audio segments. More in detail, for each target language L i , the (EnglishL i ) section of MuST-C is created as follows. First, for all the English talks available from the TED website, 3 we download the videos and the HTML files containing the manual transcriptions and their translation into L i . 4 Then, the plain text transcription and the translation of each talk are split at the sentence level based on strong punctuation marks and aligned using the Gargantua sentence alignment tool (Braune and Fraser, 2010). This step produces a bilingual text corpus aligned at the sentence level. In the third step, the English side of this bilingual corpus is aligned to the corresponding audio track extracted from the video. This is done using Gentle, 5 an off-the-shelf English forced-aligner built on the Kaldi ASR toolkit (Povey et al., 2011). Next, the audio-text alignments are processed to create a YAML file containing time information ( i.e. start and duration) for each sentence. In this processing step, two filters are applied to weed out potentially noisy segments, or entire talks, based on the number of words that were not aligned by Gentle. First, entire talks are discarded if the proportion of unrecognized words is equal or greater than 15% of the total. This threshold was determined after a manual analysis of 73 talks (those with the highest percentage of unrecognized words). The analysis showed that these cases are representative of different types of noise like: i) non-English speech, ii) long silences, iii) music, non-transcribed songs and videos played during the talk, and iv) wrong transcriptions ( e.g. captions from other talks in the material downloaded from the TED website). The second rule applies to the single sentences of the talks that passed the first filter, and removes those in which none of the words was aligned by Gentle. 6 In the last step, the log Mel 40-dimensional filter-bank features  commonly used as input representation for ASR (Graves et al., 2013) and SLT (Weiss et al., 2017)  are extracted from the 3 www.ted.com  dump of April 2018. 4 All talks have manual captions, which were also translated into many languages by volunteers. The language coverage of the translations depends on several factors like the age of the talk (the old ones often have more translations), the popularity of its topic and the availability of volunteer translators for a given language. 5 github.com/lowerquality/gentle 6 The effectiveness of this filtering criterion was manually verified on random samples. More aggressive solutions will be explored for future releases of the corpus. Table 2 provides basic statistics for the 8 sections of the MuST-C corpus. Comparing the 4 th column with the numbers reported in Table 1, it is worth noting that, in terms of hours of tran-scribed/translated speech, each section is larger than any existing publicly available SLT resource. In this section we present two sets of experiments, which are respectively aimed to: i) empirically assess the quality of the MuST-C corpus (Section 3.3) and ii) compute baseline ASR, MT, and SLT results for future comparisons (Section 3.4). In these experiments, the audio-transcription alignments of MuST-C are used to train and evaluate ASR models, transcription-translation alignments are used for the MT models, and audio-translation alignments are used for the SLT models. ASR and SLT. For our experiments in ASR and SLT we use the same neural architecture. This setting allows us to use the encoder of the ASR models to initialize the weights of the SLT encoders and achieve a faster convergence (Bansal et al., 2018). Our SLT architecture is a variant of the system proposed by Berard et al. (2018), which we re-implemented in the fairseq toolkit (Gehring et al., 2017). The system relies on an attentional encoder-decoder model that takes in input sequences of audio features and outputs the target sequence at the character level. The encoder processes the input with two consecutive fully-connected layers to expand the size of the representation, followed by two 2D strided convolu-7 github.com/neulab/xnmt tional layers that reduce the sequence length. The output of the convolutions is then processed by three stacked LSTMs (Hochreiter and Schmidhu-ber, 1997). The decoder consists of a two-layered deep transition (Pascanu et al., 2014) LSTM with an attention network based on the general soft attention score (Luong et al., 2015). The final output of the decoder is a function of the concatenation of the LSTM output, the context vector and the previous-character embedding. MT. For the MT experiments we use the open source version of ModernMT. 8 The system is based on the Transformer (Vaswani et al., 2017) architecture, which represents the state of the art in NMT (Bojar et al., 2018). The encoder consists of a stack of 6 layers, each containing a sequence of two sub-layers, a self-attention network based on multi-head attention, and a position-wise feed-forward layer. The decoder layers have an additional sub-layer: between the self attention and the position-wise feed-forward layer they have an encoder-decoder multi-head attention. All the sublayers in both the encoder and decoder are preceded by layer normalization and are followed by residual connections. In our experiments, texts are tokenized and punctuation is normalized. Furthermore, the English texts are lowercased, while the target language texts are split into characters still preserving the word boundaries. For MT, we segment the English words with the BPE algorithm (Sennrich et al., 2015) using a maximum of 30 K merge operations. The output generation of all models is performed using beam search with a beam size of 5 . ASR performance is measured with word error rate (WER) computed on lower-cased, tokenized texts without punctuation. MT and SLT results are computed with BLEU (Papineni et al., 2002). As observed in Section 2, each section of MuST-C is larger than any other existing publicly available SLT corpus. The usefulness of a resource, however, is not only a matter of size but also of quality (in this case, the quality of the audio transcription translation alignments). For an empirical verification of this aspect, we experimented with two comparable datasets. One is 8 www.modernmt.eu the TED-derived English-German IWSLT18 corpus (Niehues et al., 2018), which is built following a pipeline that performs segment extraction and alignment based on time information ( i.e. start and end position of each segment in the SubRip Text (SRT) files) instead of text-level alignments. The other is the English-German subset of MuST-C derived from the same TED Talks used to build the IWSLT18 corpus. On one side (MuST-C), the number of segments, their length, and the overall corpus quality depend on text-level alignments. On the other side (IWSLT18), they depend on matching time stamps. This strategy, however, has some drawbacks. First, as pointed out by (Niehues et al., 2018; Liu et al., 2018; Di Gangi et al., 2018), the use of time information brings some noise in the corpus. Second, it often results in utterance-level alignment (based on speakers' pauses in the original audio). Compared to sentence-level alignment, this level of granularity can be sub-optimal during model training ( e.g. for MT and SLT, learning from complete sentences is easier than learning from phrases). Finally, time information about the recorded speech is not always available: bypassing this need would make the method replicable on other data (not only TED-like). Though initialized with the same set of 1 , 619 talks, the two pipelines produce different corpora. As shown in Table 3, our approach filters out 58 entire talks (  3.6% of the total) but the final number of segments, their corresponding audio duration and their average length (in words) are larger. Corpus #Talk #Sent Hours src w tgt w IWSLT18 1,619 176K 280 2.7M 2.5M MuST-C 1,561 179K 313 3.3M 3.1M Table 3: Statistics of the English-German corpora created by applying the IWSLT18 and MuST-C pipelines to the same initial set of 1 , 619 TED Talks. Each corpus was divided into training, development and test. Development and test contain segments from randomly selected common talks ( i.e. those preserved by the MuST-C pipeline). Their size is respectively 2 . 3 K (from 28 talks) and 2 . 1 K segments (from 26 talks). The test portions were concatenated to create a balanced test set ( 4 . 2 K segments) containing half of the instances from the IWSLT18 corpus and half from MuST-C. The remaining material was used to separately train ASR, MT and SLT models on homogeneous data from either of the two corpora ( i.e. three systems Training set ASR (  ) MT (  ) SLT (  ) IWSLT18 42.15 24.90 8.94 MuST-C 32.05 25.46 12.25 Table 4: Performance of ASR, MT and SLT systems trained with En-De IWSLT18 and MuST-C data. per corpus). All the systems are evaluated on the common test set. Table 4 shows that the models trained on MuST-C data achieve better results on the balanced test set in all the three tasks. In particular: i) a reduction of 10 . 1 WER points in ASR indicates a higher quality of audio transcription alignments, ii) a BLEU increase of 0 . 56 points in MT indicates a similar quality for transcription translation alignments, and iii) a BLEU increase of 3 . 31 points in SLT indicates a higher quality of audio translation alignments. We consider these results as evidence of the reliability of our corpus creation methodology. Being the same for all the language pairs, we expect this procedure to end up in comparable quality for all the 8 sections of MuST-C. We finally present baseline results computed, for all the three tasks, on each section of MuST-C. Also for these experiments, development and test data are created with segments from talks that are common to all the languages. Their size is respectively 1 . 4 K (from 11 talks) and 2 . 5 K segments (from 27 talks). The remaining data (of variable size depending on the language pairs) are used for training. For the sake of replicability, these splits are preserved in the released version of MuST-C. The results in Table 5 lead to the following observations. First, though not directly comparable since they are computed on different test sets, English-German results are in line (actually higher, since they are produced by models built on larger training data) with those presented in Section 3.3. This indicates that the level of quality observed in the previous experiments with a subset of the training data is preserved by the whole material released for this language pair. Second, looking at the other language pairs, ASR, MT and SLT results are comparable with the English-German scores. Besides normal fluctuations in the optimization of the neural models, performance differences are coherent with: i) the relative difficulty of each target language ( e.g. Russian is more dif-ficult due to high inflection) and ii) the variable quantity of training data available ( e.g. French has the largest training set, see Table 2). Overall, these explainable differences suggest that our corpus creation methodology yields homogeneous quality for all the languages covered by MuST-C. We presented MuST-C, a Mu ltilingual S peech T ranslation C orpus built to address the need of resources for training data-hungry neural SLT models. To the best of our knowledge, to date MuST-C is the largest publicly available corpus of this kind. In its current version, it comprises the English transcription and the translations into 8 target languages of at least 385 hours of speech (up to 504 ) per language. Thanks to a scalable corpus creation procedure initialized with constantly expanding TED talks data, future extensions will increase the coverage of the already present target languages and introduce new ones. MuST-C is released under a Creative Commons license, Attribution Non Commercial No Derivatives (CC BY NC ND 4.0 International), and is freely downloadable at mustc.fbk.eu Acknowledgments The authors gratefully acknowledge NVIDIA Corporation for the donation of the Tesla K80 and GeForce GTX 1080 Ti GPUs used for this research.
abstain abstain method abstain objective abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain abstain method abstain abstain objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain