We present an unsupervised method for learning speech representations based on a bidirectional contrastive predictive coding that implicitly discovers phonetic structure from large-scale corpora of unlabelled raw audio signals. The representations, which we learn from up to 8000 hours of publicly accessible speech data, are evaluated by looking at their impact on the behaviour of supervised speech recognition systems. First, across a variety of datasets, we find that the features learned from the largest and most diverse pretraining dataset result in significant improvements over standard audio features as well as over features learned from smaller amounts of pretraining data. Second, they significantly improve sample efficiency in low-data scenarios. Finally, the features confer significant robustness advantages to the resulting recognition systems: we see significant improvements in out-of-domain transfer relative to baseline feature sets, and the features likewise provide improvements in four different low-resource African language datasets.
Overview:
This work uses contrastive predictive coding (CPC) to learn unsupervised speech representations on large amounts of unlabelled speech data and then uses the resulting features in downstream speech recognition systems. Unlabelled data is obtained from several sources (spanning different languages). Supervised systems are then built on top of these features and sample-efficiency and cross-domain robustness is investigated using English data sets. Finally, the approach is applied to four African languages.
Strengths:
Firstly, the paper is very clearly written and motivated. Secondly, a very relevant problem is tackled in a systematic way; compared to transcribed resources, unlabelled resources are much easier to collect and more widely available. This paper shows that these unlabelled resources can be of great benefit in downstream tasks and on languages where few resources are available. Thirdly, the experiments are carried out very systematically to support the claims of the paper: that bidirectional CPC-based feature learning improves same efficiency (they show that much less labelled data is required to achieve the same performance as when using more substantial labelled data with conventional features), and that it improves robustness to out-of-domain data. They perform these experiments on both English and truly low-resource languages.
Weaknesses:
There are two main weaknesses to the paper. Firstly, as the authors note themselves, unsupervised CPC-based speech feature learning was developed and considered in previous work, and has also been subsequently investigated by others. The main technical contribution is therefore only in changing the unidirectional architecture to bidirectional. Secondly, the paper does a very poor job of linking this work with previous work. The work in [1] is very related. In Section 5, the ZeroSpeech challenges are mentioned briefly (with a single citation), but over the last decade there has been substantial work in this community specifically looking at exactly the main problem addressed in this paper (unsupervised speech representation learning). It would be of great benefit to situate this work within that context, and I would recommend that the paper at least mention [2] to [9].
Overall assessment:
Although technical novelty is limited (first weakness), I think there is novelty in the paper's systematic experimental investigation, including ASR experiments on truly low-resource languages. The conclusions of this work also has practical implications for the ASR community. The second weakness can be addressed by amending Section 5. I therefore assign a "Weak Accept" to the paper.
Questions, suggestions, typos, grammar and style:
- In Figure 1, it might be useful to indicate the autoregressive nature of the context vectors by adding arrows in-between the $c$ blocks on the top left. (In the text it says an RNN is used.)
- p.7: "... are suitable for driving recognition different languages ...". A typo or grammatically incorrect sentence.
- p. 9: "Tts without t" -> "TTS without T"
- p. 9: "african" -> "African" (check all citations for capitalization)
Missing references:
1. https://arxiv.org/abs/1904.03240
2. A. Jansen et al. A summary of the 2012 JHU CLSP workshop on zero resource speech technologies and models of early language acquisition. ICASSP, 2013.
3. Badino, L., Canevari, C., Fadiga, L., & Metta, G. (2014). An auto-encoder based approach to unsupervised learning of subword units. in ICASSP.
4. Versteegh, M., Anguera, X., Jansen, A. & Dupoux, E. (2016). The Zero Resource Speech Challenge 2015: Proposed Approaches and Results. In SLTU-2016 Procedia Computer Science, 81, (pp 67-72).
5. Renshaw, D et al. (2015). A Comparison of Neural Network Methods for Unsupervised Representation Learning on the Zero Resource Speech Challenge. Interspeech.
6. R. Thiolliere et al. A  hybrid  dynamic  time  warping-deep  neural  network  architecture  forunsupervised acoustic modeling. Interspeech. 2015
7. https://arxiv.org/abs/1811.08284
8. https://arxiv.org/abs/1702.01360
9. https://arxiv.org/abs/1709.07902