## Multilingual Speech Recognition with Corpus Relatedness Sampling

Xinjian Li, Siddharth Dalmia, Alan W. Black, Florian Metze

Language Technologies Institute, Carnegie Mellon University; Pittsburgh, PA; U.S.A.

{ xinjianl|sdalmia|awb|fmetze @cs.cmu.edu }

## Abstract

Multilingual acoustic models have been successfully applied to low-resource speech recognition. Most existing works have combined many small corpora together, and pretrained a multilingual model by sampling from each corpus uniformly. The model is eventually fine-tuned on each target corpus. This approach, however, fails to exploit the relatedness and similarity among corpora in the training set. For example, the target corpus might benefit more from a corpus in the same domain or a corpus from a close language. In this work, we propose a simple but useful sampling strategy to take advantage of this relatedness. We first compute the corpus-level embeddings and estimate the similarity between each corpus. Next we start training the multilingual model with uniform-sampling from each corpus at first, then we gradually increase the probability to sample from related corpora based on its similarity with the target corpus. Finally the model would be fine-tuned automatically on the target corpus. Our sampling strategy outperforms the baseline multilingual model on 16 low-resource tasks. Additionally, we demonstrate that our corpus embeddings capture the language and domain information of each corpus.

## 1. Introduction

In recent years, Deep Neural Networks (DNNs) have been successfully applied to Automatic Speech Recognition (ASR) for many well-resourced languages including Mandarin and English [1, 2]. However, only a small portion of languages have clean speech labeled corpus. As a result, there is an increasing interest in building speech recognition systems for low-resource languages. To address this issue, researchers have successfully exploited multilingual speech recognition models by taking advantage of labeled corpora in other languages [3, 4]. Multilingual speech recognition enables acoustic models to share parameters across multiple languages, therefore low-resource acoustic models can benefit from rich resources.

While low-resource multilingual works have proposed various acoustic models, those works tend to combine several lowresource corpora together without paying attention to the variety of corpora themselves. One common training approach here is to first pretrain a multilingual model by combining all training corpora, then the pretrained model is fine-tuned on the target corpus [5]. During the training process, each corpus in the training set is treated equally and sampled uniformly. We argue, however, this approach does not take account of the characteristics of each corpus, therefore it fails to take advantage of the relations between them. For example, a conversation corpus might be more beneficial to another conversation corpus rather than an audio book corpus.

In this work, we propose an effective sampling strategy (Corpus Relatedness Sampling) to take advantage of relations among corpora. Firstly, we introduce the corpus-level embedding which can be used to compute the similarity between cor- pora. The embedding can be estimated by being jointly trained with the acoustic model. Next, we compute the similarity between each corpus and the target corpus, the similarity is then used to optimize the model with respect to the target corpus. During the training process, we start by uniformly sampling from each corpus, then the sampling distribution is gradually updated so that more related corpora would be sampled more frequently. Eventually, only the target corpus would be sampled from the training set as the target corpus is the most related corpus to itself. While our approach differs from the pretrained model and the fine-tuned model, we can prove that those models are special cases of our sampling strategy.

To evaluate our sampling strategy, we compare it with the pretrained model and fine-tuned model on 16 different corpora. The results show that our approach outperforms those baselines on all corpora: it achieves 1.6% lower phone error rate on average. Additionally, we demonstrate that our corpus-level embeddings are able to capture the characteristics of each corpus, especially the language and domain information. The main contributions of this paper are as follows:

- 1. Wepropose a corpus-level embedding which can capture the language and domain information of each corpus.
- 2. We introduce the Corpus Relatedness Sampling strategy to train multilingual models. It outperforms the pretrained model and fine-tuned model on all of our test corpora.

## 2. Related Work

Multilingual speech recognition has explored various models to share parameters across languages in different ways. For example, parameters can be shared by using posterior features from other languages [6], applying the same GMM components across different HMM states [7], training shared hidden layers in DNNs [3, 4] or LSTM [5], using language independent bottleneck features [8, 9]. Some models only share their hidden layers, but use separate output layers to predict their phones [3, 4]. Other models have only one shared output layer to predict the universal phone set shared by all languages [10, 11, 12]. While those works proposed the multilingual models in different ways, few of them have explicitly exploited the relatedness across various languages and corpora. In contrast, our work computes the relatedness between different corpora using the embedding representations and exploits them efficiently.

The embedding representations have been heavily used in multiple fields. In particular, embeddings of multiple granularities have been explored in many NLP tasks. To name a few, character embedding [13], subword embedding [14], sentence embedding [15] and document embedding [16]. However, there are few works exploring the corpus level embeddings. The main reason is that the number of corpora involved in most experiments is usually limited and it is not useful to compute corpus embeddings. The only exception is the multitask learning where many tasks and corpora are combined together. For instance, the language level (corpus level) embedding can be generated along with the model in machine translation [17] and speech recognition [18]. However, those embeddings are only used as an auxiliary feature to the model, few works continue to exploit those embeddings themselves.

Another important aspect of our work is that we focused on the sampling strategy for speech recognition. While most of the previous speech works mainly emphasized the acoustic modeling side, there are also some attempts focusing on the sampling strategies. For instance, curriculum learning would train the acoustic model by starting from easy training samples and increasingly adapt it to more difficult samples [1, 19]. Active learning is an approach trying to minimize human costs to collect transcribed speech data [20]. Furthermore, sampling strategies can also be helpful to speed up the training process [21]. However, the goals of most strategies are to improve the acoustic model by modifying the sampling distribution within a single speech corpus for a single language. On the contrary, our approach aims to optimize the multilingual acoustic model by modifying distributions across all the training corpora.

## 3. Approach

In this section, we describe our approach to compute the corpus embedding and our Corpus Relatedness Sampling strategy.

## 3.1. Corpus Embedding

Suppose that C t is the target low-resource corpus, we are interested in optimizing the acoustic model with a much larger training corpora set S = {C 1 , C 2 ... C n } where n is the number of corpora and C t ∈ S . Each corpus C i is a collection of ( x y , ) pairs where x is the input features and y is its target.

Figure 1: The acoustic model to optimize corpus embeddings.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.01060v1-figure-f1967418-978f-47c1-be13-c664925896f8.jpg)

Our purpose here is to compute the embedding e i for each corpus C i where e i is expected to encode information about its corpus C i . Those embeddings can be jointly trained with the standard multilingual model [5]. First, the embedding matrix E for all corpora is initialized, the i -th row of E is corresponding to the embedding e i of the corpus C i . Next, during the training phase, e i can be used to bias the input feature x as follows.

$$h = E n c o d e r ( { \mathbf x } + { \mathbf e } _ { i } ; W, E ) \text{ \quad \ \ } ( 1 )$$

where ( x y , ) ∈ C i is an utterance sampled randomly from S , h is its hidden features, W is the parameter of the acoustic model and Encoder is the stacked bidirectional LSTM as shown in Figure.1. Next, we apply the language specific softmax to compute logits l and optimize them with the CTC objective [30]. The embedding matrix E can be optimized together with the model during the training process.

## 3.2. Corpus Relatedness Sampling

With the embedding e i of each corpus C i , we can compute the similarity score between any two corpora using the cosine similarity.

$$s c o r e ( \mathcal { C } _ { i }, \mathcal { C } _ { j } ) = \frac { \mathrm e _ { i } \cdot \mathrm e _ { j } } { | \mathrm e _ { i } | | \mathrm e _ { j } | } \quad \quad ( 2 )$$

As the similarity reflects the relatedness between corpora in the training set, we would like to sample the training set based on this similarity: those corpora which have a higher similarity with the target corpus C t should be sampled more frequently. Therefore, we assume those similarity scores to be the sampling logits and they should be normalized with softmax.

$$P r ( \mathcal { C } _ { i } ) = \frac { \exp \left ( T \cdot s c o r e ( \mathcal { C } _ { i }, \mathcal { C } _ { t } ) \right ) } { \sum _ { j } \exp ( T \cdot s c o r e ( \mathcal { C } _ { j }, \mathcal { C } _ { t } ) ) } \quad \ \ ( 3 )$$

where Pr ( C i ) is the probability to sample C i from S , and T is the temperature to normalize the distribution during the training phase. We argue that different temperatures could create different training conditions. The model with a lower temperature tends to sample each corpus equally like uniform sampling. In contrast, a higher temperature means that the sampling distribution should be biased toward the target corpus like the finetuning.

Next, we prove that both the pretrained model and the finetuned model can be realized with specific temperatures. In the case of the pretrained model, each corpus should be sampled equally. This can be implemented by setting T to be 0 .

$$P r ( \mathcal { C } _ { i } ) = \lim _ { T \to 0 } \frac { \exp \left ( T \cdot s c o r e ( \mathcal { C } _ { i }, \mathcal { C } _ { t } ) \right ) } { \sum _ { j } \exp ( T \cdot s c o r e ( \mathcal { C } _ { j }, \mathcal { C } _ { t } ) ) } = \frac { 1 } { n } \quad ( 4 )$$

glyph[negationslash]

On the other hand, the fine-tuned model should only consider samples from the target corpus C t , while ignoring all other corpora. We argue that this condition can be approximated by setting T to a very large number. As score ( C t , C t ) = 1 0 . and score ( C i , C t ) &lt; 1 0 . if i = t , we can prove the statement as follows:

glyph[negationslash]

$$P r ( \mathcal { C } _ { i } ) & = \lim _ { T \to \infty } \frac { \exp \left ( T \cdot s c o r e ( \mathcal { C } _ { i }, \mathcal { C } _ { t } ) \right ) } { \sum _ { j } \exp ( T \cdot s c o r e ( \mathcal { C } _ { j }, \mathcal { C } _ { t } ) ) } \\ & = \begin{cases} 1. 0 & i f \, \mathcal { C } _ { i } = \mathcal { C } _ { t } \\ 0. 0 & i f \, \mathcal { C } _ { i } \neq \mathcal { C } _ { t } \end{cases}$$

While both the pretrained model and the fine-tuned model are special cases of our approach, we note that our approach is more flexible to sample from related corpora by interpolating between those two extreme temperatures. In practice, we

Table 1: The collection of training corpora used in the experiment. Both the baseline model and the proposed model are trained and tested with 16 corpora across 10 languages. We assign a corpus id to each corpus after its corpus name so that we can distinguish the corpora sharing the same language.

| Language                        | Corpus Name                                                              | Domain                             | Utterance                      | Language         | Corpus Name                                                                                | Domain                        | Utterance             |
|---------------------------------|--------------------------------------------------------------------------|------------------------------------|--------------------------------|------------------|--------------------------------------------------------------------------------------------|-------------------------------|-----------------------|
| English English English English | TED (ted) [22] Switchboard Librispeech (libri) [26] Fisher (fisher) [27] | broadcast telephone read telephone | 100,000 100,000 100,000 41,403 | Mandarin Bengali | Hkust (hk) [23] SLR18 (s18) [25] LDC98S73 (hub) SLR47 (s47) [28] LDC2016S08 voxforge (vox) | telephone read broadcast read | 100,000 13,388 35,999 |
|                                 | (swbd)[24]                                                               |                                    |                                | Mandarin         |                                                                                            |                               |                       |
|                                 |                                                                          |                                    |                                | Mandarin         |                                                                                            |                               |                       |
|                                 |                                                                          |                                    | 100,000                        | Mandarin         |                                                                                            |                               | 50,384                |
| Amharic                         | LDC2014S06 (babel)                                                       | telephone                          |                                |                  | (babel)                                                                                    | telephone                     | 60,663                |
| Dutch                           | voxforge (vox)                                                           | read                               | 8,492                          | German           |                                                                                            | read                          | 41,146                |
| Spanish                         | LDC98S74 (hub)                                                           | broadcast                          | 31,615                         | Swahili          | LDC2017S05(babel)                                                                          | telephone                     | 44,502                |
| Turkish                         | LDC2012S06 (hub) [29]                                                    | broadcast                          | 97,427                         | Zulu             | LDC2017S19 (babel)                                                                         | telephone                     | 60,835                |

would like to start with a low temperature to sample broadly in the early training phase. Then we gradually increase the temperature so that it can focus more on the related corpora. Eventually, the temperature would be high enough so that the model is automatically fine-tuned on the target corpus. Specifically, in our experiment, we start training with a very low temperature T 0 , and increase its value every epoch k as follows.

$$T _ { k + 1 } = a T _ { k }$$

where T k is the temperature of epoch k and a is a hyperparameter to control the growth rate of the temperature.

## 4. Experiments

To demonstrate that our sampling approach could improve the multilingual model, we conduct experiments on 16 corpora to compare our approach with the pretrained model and fine-tuned model.

## 4.1. Datasets

We first describe our corpus collection. Table.1 lists all corpora we used in the experiments. There are 16 corpora from 10 languages. To increase the variety of corpus, we selected 4 English corpora and 4 Mandarin corpora in addition to the low resource language corpora. As the target of this experiment is low resource speech recognition, we only randomly select 100,000 utterances even if there are more in each corpus. All corpora are available in LDC, voxforge, openSLR or other public websites. Each corpus is manually assigned one domain based on its speech style. Specifically, the domain candidates are telephone , read and broadcast .

## 4.2. Experiment Settings

We use EESEN [31] for the acoustic modeling and epitran [32] as the g2p tool in this work. Every utterance in the corpora is firstly re-sampled into 8000Hz, and then we extract 40 dimension MFCCs features from each audio. We use a recent multilingual CTC model as our acoustic architecture [5]: The architecture is a 6 layer bidirectional LSTM model with 320 cells in each layer. We use this architecture for both the baseline models and the proposed model.

Our baseline model is the fine-tuned model: we first pretrained a model by uniformly sampling from all corpora. After the loss converges, we fine-tune the model on each of our target corpora. To compare it with our sampling approach, we first train an acoustic model to compute the embeddings of all corpora, then the embeddings are used to estimate the similarity as described in the previous section. The initial temperature T 0 is set to 0.01, and the growth rate is 1 5 . . We evaluated all models using the phone error rate (PER) instead of the word error rate (WER). The reason is that we mainly focus on the acoustic model in this experiment. Additionally, some corpora (e.g.: Dutch voxforge) in this experiment have very few amounts of texts, therefore it is difficult to create a reasonable language model without augmenting texts using other corpora, which is beyond the scope of this work.

## 4.3. Results

Table 2: Phone error rate (%PER) of the pretrained model, the baseline model (fine-tuned model) and our CRS(Corpus Relatedness Sampling) approach on all 16 corpora

| Corpus           |   Pretrain PER |   Base PER |   CRS PER |
|------------------|----------------|------------|-----------|
| English (ted)    |           19.2 |       11.6 |      10.3 |
| English (swbd)   |           24.9 |       15.3 |      14.1 |
| English (libri)  |           12.1 |        6.5 |       5.4 |
| English (fisher) |           34.7 |       23.5 |      22.5 |
| Mandarin (hk)    |           32.6 |       16.1 |      14.5 |
| Mandarin (s18)   |            8.5 |        6.6 |       5.6 |
| Mandarin (hub)   |           10.2 |        5.1 |       4.4 |
| Mandarin (s47)   |           13   |       10.9 |       9.2 |
| Amharic (babel)  |           45.4 |       40.7 |      36.9 |
| Bengali (babel)  |           47   |       41.9 |      40   |
| Dutch (vox)      |           27.3 |       21.6 |      18.2 |
| German (vox)     |           23.1 |       12.9 |      10.3 |
| Swahili (babel)  |           17.7 |       16.1 |      14.5 |
| Spanish (hub)    |           14   |        8.4 |       7.5 |
| Turkish (hub)    |           49.2 |       45.6 |      44.9 |
| Zulu (babel)     |           47.8 |       38.5 |      37.9 |
| Average          |           26.7 |       20.1 |      18.5 |

Table.2 shows the results of our evaluation. We compare our approach with the baseline using all corpora. The left-most column of Table.2 shows the corpus we used for each experiment, the remaining columns are corresponding to the phone error rate of the pretrained model, the fine-tuned model and our proposed model. First, we can easily confirm that the fine-tuned model outperforms the pretrained model on all corpora. For instance, the fine-tuned model outperforms the pretrained model by 4.7% on the Amharic corpus. The result is reasonable as the pretrained model is optimized with the entire training set, while the fine-tuned model is further adapted to the target corpus. Next, the table suggests our Corpus Relatedness Sampling approach achieves better results than the fine-tuned model on all test corpora. For instance, the phone error rate is improved from 40.7% to 36.9% on Amharic and is improved from 41.9% to 40.0% on Bengali. On average, our approach outperforms the fine-tuned model by 1.6% phone error rate. The results demonstrate that our sampling approach is more effective at optimizing the acoustic model on the target corpus. We also train baseline models by appending corpus embeddings to input features, but the proposed model outperforms those baselines similarly.

One interesting trend we observed in the table is that the improvements differ across the target corpora. For instance, the improvement on the Dutch corpus is 3.4%, on the other hand, its improvement of 0.6% is relatively smaller on the Zulu dataset. We believe the difference in improvements can be explained by the size of each corpus. The size of Dutch corpus is very small as shown in Table.1, therefore the fine-tuned model is prone to overfit to the dataset very quickly. In contrast, it is less likely for a larger corpus to overfit. Compared with the fine-tuned model, our approach optimizes the model by gradually changing the temperature without quick overfitting. This mechanism could be interpreted as a built-in regularization. As a result, our model can achieve much better performance in small corpora by preventing the overfitting effect.

Table 3: The similarity between training corpora. For each target corpus, we show its most related corpus and second related corpus.

| Target                                                                                                                                                                            | 1st Related Corpus                                                                                                                                                                     | 2nd Related Corpus                                                                                                                                                        |
|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| English (ted) English (swbd) English (libri) English (fisher) Mandarin (hk) Mandarin (s18) Mandarin (s47) Mandarin (hub) Amharic (babel) Bengali (babel) Dutch (vox) German (vox) | English (libri) English (fisher) English (ted) English (swbd) Bengali (babel) Mandarin (s47) Mandarin (s18) Spanish (hub) Bengali (babel) Mandarin (hk) Mandarin (s47) English (libri) | Turkish (hub) Mandarin (hk) German (vox) Mandarin (hk) English (swbd) Mandarin (hub) Dutch (vox) Tkish (hub) Swahili (babel) Amharic (babel) Mandarin(s18) Mandarin (s47) |

To understand how our corpus embeddings contribute to our approach, we rank those embeddings and show the top-2 similar corpora for each corpus in Table.3. We note that the target corpus itself is removed from the rankings because it is the most related corpus to itself. The results of the top half show very clearly that our embeddings can capture the language level information: For most English and Mandarin corpora, the most related corpus is another English or Mandarin corpus. Addi-

Figure 2: Domain plot of 36 corpora, the corpus embeddings are reduced to 2 dimensions by t-SNE

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.01060v1-figure-0a8da4d1-952a-4ef8-a140-194b6c907c20.jpg)

tionally, the bottom half of the table indicates that our embeddings are able to capture domain level information as well. For instance, the top 2 related corpus for Amharic is Bengali and Swahili. According to Table.1, those three corpora belong to the telephone domain. In addition, Dutch is a read corpus, its top 2 related corpora are also from the same domain. This also explains why the 1st related corpus of Mandarin (hk) is Bengali: because both of them are from the same telephone domain.

To further investigate the domain information contained in the corpus embeddings, we train the corpus embeddings with an even larger corpora collection (36 corpora) and plot all of them in Figure.2. To create the plot, the dimension of each corpus embedding is reduced to 2 with t-SNE [33]. The figure demonstrates clearly that our corpus embeddings are capable of capturing the domain information: all corpora with the same domain are clustered together. This result also means that our approach improves the model by sampling more frequently from the corpora of the same speech domain.

## 5. Conclusion

In this work, we propose an approach to compute corpus-level embeddings. We also introduce Corpus Relatedness Sampling approach to train multilingual speech recognition models based on those corpus embeddings. Our experiment shows that our approach outperforms the fine-tuned multilingual models in all 16 test corpora by 1.6 phone error rate on average. Additionally, we demonstrate that our corpus embeddings can capture both language and domain information of each corpus.

## 6. Acknowledgements

This project was sponsored by the Defense Advanced Research Projects Agency (DARPA) Information Innovation Office (I2O), program: Low Resource Languages for Emergent Incidents (LORELEI), issued by DARPA/I2O under Contract No. HR0011-15-C-0114.

## 7. References

- [1] D. Amodei, S. Ananthanarayanan, R. Anubhai, J. Bai, E. Battenberg, C. Case, J. Casper, B. Catanzaro, Q. Cheng, G. Chen et al. , 'Deep speech 2: End-to-end speech recognition in english
- and mandarin,' in International conference on machine learning , 2016, pp. 173-182.
- [2] W. Xiong, J. Droppo, X. Huang, F. Seide, M. Seltzer, A. Stolcke, D. Yu, and G. Zweig, 'Achieving human parity in conversational speech recognition,' arXiv preprint arXiv:1610.05256 , 2016.
- [3] J.-T. Huang, J. Li, D. Yu, L. Deng, and Y. Gong, 'Cross-language knowledge transfer using multilingual deep neural network with shared hidden layers,' in Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on . IEEE, 2013, pp. 7304-7308.
- [4] G. Heigold, V. Vanhoucke, A. Senior, P. Nguyen, M. Ranzato, M. Devin, and J. Dean, 'Multilingual acoustic models using distributed deep neural networks,' in Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on . IEEE, 2013, pp. 8619-8623.
- [5] S. Dalmia, R. Sanabria, F. Metze, and A. W. Black, 'Sequencebased multi-lingual low resource speech recognition,' in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2018, pp. 4909-4913.
- [6] A. Stolcke, F. Grezl, M.-Y. Hwang, X. Lei, N. Morgan, and D. Vergyri, 'Cross-domain and cross-language portability of acoustic features estimated by multilayer perceptrons,' in Acoustics, Speech and Signal Processing, 2006. ICASSP 2006 Proceedings. 2006 IEEE International Conference on , vol. 1. IEEE, 2006, pp. I-I.
- [7] L. Burget, P. Schwarz, M. Agarwal, P. Akyazi, K. Feng, A. Ghoshal, O. Glembek, N. Goel, M. Karafi´ at, D. Povey et al. , 'Multilingual acoustic modeling for speech recognition based on subspace gaussian mixture models,' in Acoustics Speech and Signal Processing (ICASSP), 2010 IEEE International Conference on . IEEE, 2010, pp. 4334-4337.
- [8] K. Vesel` y, M. Karafi´t, F. Gr´zl, M. Janda, and E. Egorova, 'The a e language-independent bottleneck features,' in Spoken Language Technology Workshop (SLT), 2012 IEEE . IEEE, 2012, pp. 336341.
- [9] S. Dalmia, X. Li, F. Metze, and A. W. Black, 'Domain robust feature extraction for rapid low resource asr development,' in 2018 IEEE Spoken Language Technology Workshop (SLT) . IEEE, 2018, pp. 258-265.
- [10] T. Schultz and A. Waibel, 'Language-independent and languageadaptive acoustic modeling for speech recognition,' Speech Communication , vol. 35, no. 1-2, pp. 31-51, 2001.
- [11] S. Tong, P. N. Garner, and H. Bourlard, 'An investigation of deep neural networks for multilingual speech recognition training and adaptation,' Tech. Rep., 2017.
- [12] N. T. Vu and T. Schultz, 'Multilingual multilayer perceptron for rapid language adaptation between and across language families.' in Interspeech , 2013, pp. 515-519.
- [13] Y. Kim, Y. Jernite, D. Sontag, and A. M. Rush, 'Character-aware neural language models,' in Thirtieth AAAI Conference on Artificial Intelligence , 2016.
- [14] P. Bojanowski, E. Grave, A. Joulin, and T. Mikolov, 'Enriching word vectors with subword information,' Transactions of the Association for Computational Linguistics , vol. 5, pp. 135-146, 2017.
- [15] R. Kiros, Y. Zhu, R. R. Salakhutdinov, R. Zemel, R. Urtasun, A. Torralba, and S. Fidler, 'Skip-thought vectors,' in Advances in neural information processing systems , 2015, pp. 3294-3302.
- [16] Q. Le and T. Mikolov, 'Distributed representations of sentences and documents,' in International conference on machine learning , 2014, pp. 1188-1196.
- [17] M. Johnson, M. Schuster, Q. V. Le, M. Krikun, Y. Wu, Z. Chen, N. Thorat, F. Vi´ egas, M. Wattenberg, G. Corrado et al. , 'Googles multilingual neural machine translation system: Enabling zeroshot translation,' Transactions of the Association for Computational Linguistics , vol. 5, pp. 339-351, 2017.

| [18]   | B. Li, T. N. Sainath, K. C. Sim, M. Bacchiani, E. Weinstein, P. Nguyen, Z. Chen, Y. Wu, and K. Rao, 'Multi-dialect speech recognition with a single sequence-to-sequence model,' in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2018, pp. 4749-4753.                             |
|--------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [19]   | S. Braun, D. Neil, and S.-C. Liu, 'A curriculum learning method for improved noise robustness in automatic speech recogni- tion,' in 2017 25th European Signal Processing Conference (EU- SIPCO) . IEEE, 2017, pp. 548-552.                                                                                                         |
| [20]   | G. Riccardi and D. Hakkani-Tur, 'Active learning: Theory and ap- plications to automatic speech recognition,' IEEE transactions on speech and audio processing , vol. 13, no. 4, pp. 504-511, 2005.                                                                                                                                 |
| [21]   | J. Cui, B. Kingsbury, B. Ramabhadran, A. Sethy, K. Audhkhasi, X. Cui, E. Kislal, L. Mangu, M. Nussbaum-Thom, M. Picheny et al. , 'Multilingual representations for low resource speech recognition and keyword search,' in 2015 IEEE Workshop on Au- tomatic Speech Recognition and Understanding (ASRU) . IEEE, 2015, pp. 259-266. |
| [22]   | A. Rousseau, P. Del´glise, e and Y. Esteve, 'TED-LIUM: an auto- matic speech recognition dedicated corpus.' in LREC , 2012, pp. 125-129.                                                                                                                                                                                            |
| [23]   | Y. Liu, P. Fung, Y. Yang, C. Cieri, S. Huang, and D. Graff, 'Hkust/mts: A very large scale mandarin telephone speech cor- pus,' in Chinese Spoken Language Processing . Springer, 2006, pp. 724-735.                                                                                                                                |
| [24]   | J. J. Godfrey, E. C. Holliman, and J. McDaniel, 'SWITCH- BOARD: Telephone speech corpus for research and develop- ment,' in Acoustics, Speech, and Signal Processing, 1992. ICASSP-92., 1992 IEEE International Conference on , vol. 1. IEEE, 1992, pp. 517-520.                                                                    |
| [25]   | Z. Z. Dong Wang, Xuewei Zhang, 'Thchs-30 : A free chinese speech corpus,' 2015. [Online]. Available: http: //arxiv.org/abs/1512.01882                                                                                                                                                                                               |
| [26]   | V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, 'Lib- rispeech: an ASR corpus based on public domain audio books,' in Acoustics, Speech and Signal Processing (ICASSP), 2015 IEEE International Conference on . IEEE, 2015, pp. 5206-5210.                                                                                       |
| [27]   | C. Cieri, D. Miller, and K. Walker, 'The fisher corpus: a resource for the next generations of speech-to-text.' in LREC , vol. 4, 2004, pp. 69-71.                                                                                                                                                                                  |
| [28]   | L. Primewords Information Technology Co., 'Primewords chi- nese corpus set 1,' 2018, https://www.primewords.cn.                                                                                                                                                                                                                     |
| [29]   | M. Saraclar, 'Turkish broadcast news speech and transcripts ldc2012s06,' Philadelphia, Linguistic Data Consortium, Web Download , 2012.                                                                                                                                                                                             |
| [30]   | A. Graves, S. Fern´ndez, a F. Gomez, and J. Schmidhuber, 'Con- nectionist temporal classification: labelling unsegmented se- quence data with recurrent neural networks,' in Proceedings of the 23rd international conference on Machine learning . ACM, 2006, pp. 369-376.                                                         |
| [31]   | Y. Miao, M. Gowayyed, and F. Metze, 'EESEN: End-to-end speech recognition using deep RNN models and WFST-based decoding,' in Automatic Speech Recognition and Understanding (ASRU), 2015 IEEE Workshop on . IEEE, 2015, pp. 167-174.                                                                                                |
| [32]   | D. R. Mortensen, S. Dalmia, and P. Littell, 'Epitran: Precision G2P for many languages.' in LREC , 2018.                                                                                                                                                                                                                            |
| [33]   | L. v. d. Maaten and G. Hinton, 'Visualizing data using t-sne,' Journal of machine learning research , vol. 9, no. Nov, pp. 2579- 2605, 2008.                                                                                                                                                                                        |
Xinjian Li Siddharth Dalmia Alan W. Black Florian Metze
Multilingual Speech Recognition with Corpus Relatedness Sampling
Multilingual acoustic models have been successfully applied to low-resource
speech recognition. Most existing works have combined many small corpora
together and pretrained a multilingual model by sampling from each corpus
uniformly. The model is eventually fine-tuned on each target corpus. This
approach, however, fails to exploit the relatedness and similarity among
corpora in the training set. For example, the target corpus might benefit more
from a corpus in the same domain or a corpus from a close language. In this
work, we propose a simple but useful sampling strategy to take advantage of
this relatedness. We first compute the corpus-level embeddings and estimate the
similarity between each corpus. Next, we start training the multilingual model
with uniform-sampling from each corpus at first, then we gradually increase the
probability to sample from related corpora based on its similarity with the
target corpus. Finally, the model would be fine-tuned automatically on the
target corpus. Our sampling strategy outperforms the baseline multilingual
model on 16 low-resource tasks. Additionally, we demonstrate that our corpus
embeddings capture the language and domain information of each corpus.