We propose vq-wav2vec to learn discrete representations of audio segments through a wav2vec-style self-supervised context prediction task. The algorithm uses either a Gumbel-Softmax or online k-means clustering to quantize the dense representations. Discretization enables the direct application of algorithms from the NLP community which require discrete inputs. Experiments show that BERT pre-training achieves a new state of the art on TIMIT phoneme classification and WSJ speech recognition.
Overview:
This paper considers unsupervised (or self-supervised) discrete representation learning of speech using a combination of a recent vector quantized neural network discritization method and future time step prediction. Discrete representations are fine-tuned by using these as input to a BERT model; the resulting representations are then used instead of conventional speech features as the input to speech recognition models. New state-of-the-art results are achieved on two datasets.
Strengths:
The core strength of this paper is in the results that are achieved on standard speech recognition benchmarks. The results indicate that, while discritization in itself does not give improvements, coupling this with the BERT-objective results in speech features which are better in downstream speech recognition than standard features. I think the main technical novelty is in combining discritization with future time step prediction (but see the weakness below).
Weaknesses:
The main weakness of the paper is that it does not situate itself within existing literature in this area. Over the last few years, researchers in the speech community have invested significant effort in learning better speech representations, and this is not discussed. See e.g. [1]. Even more importantly, very recently there has been a number of papers investigating discrete representations of speech; see the review [2]. Some of these papers specifically use VQ-VAEs [3]. [4] actually compares VQ-VAE and the Gumbel-Softmax approach. These studies should be mentioned. This paper is different in that it incorporates future time step prediction. But context prediction has also been considered before, also for speech [5, 6, 7]. This paper can be situated as a new contribution combining these two strands of research. In the longer run it would be extremely beneficial to the community if this approach is applied to the standard benchmarks as set out in [2].
As a minor weakness, some parts of the paper is not described in enough detail and the motivation is weak or not exactly clear (see detailed comments below).
Overall assessment:
I think the results as well as the new combination of existing approaches in the paper warrants publication. But it should be amended significantly to situate itself within the existing literature. I therefore award a "weak accept".
Detailed questions and suggestions:
- Section 1: As motivation for this work, it is stated that "we aim to make well performing NLP algorithms more widely applicable". As noted above, some NLP-like ideas (such as prediction of future speech segments, stemming from text-based language modelling) have already been considered within the speech community. Rather than motivating the work in this way, it might be helpful to focus the contribution as a combination of future time step prediction and discretization (both of which have been considered in previous work, but not in combination).
- Section 4: Would it be possible to train the vq-wav2vec model jointly with BERT, i.e. as one model? I suspect it would be difficult since, for the masking objective, the discrete units are already required, but maybe there is a scheme where this could work.
- Section 2.2: Similarly to the above question, would there be a way to incorporate the BERT principles directly into an end-to-end model, e.g. by randomly masking some of the continuous input speech?
- Section 3.3: What exactly does "mode collapse" refer to in this context? Would this be using only one codebook entry, for instance?
- Section 6: It seems that in all cases to obtain improvements from discritization, BERT is required on top of the vq-wav2vec discrete symbols. Is it possible that the output acoustic model is simply better-matched to continuous rather than discrete input (direct vq-wav2vec gives discrete while BERT gives continuous)? Would it make sense to train the wav2vec acoustic model on top of the vqvae codebook entries (e) instead of directly on the symbols?
Typos, grammar and style:
- "gumbel" -> "Gumbel" (throughout; or just be consistent in capitalization)
- "which can be mitigated my workarounds" -> "which can be mitigated *by* workarounds"
- "work around" -> "workaround"
Missing references:
1. Versteegh, M., Anguera, X., Jansen, A. & Dupoux, E. (2016). The Zero Resource Speech Challenge 2015: Proposed Approaches and Results. In SLTU-2016 Procedia Computer Science, 81, (pp 67-72).
2. https://arxiv.org/abs/1904.11469
3. https://arxiv.org/abs/1905.11449
4. https://arxiv.org/abs/1904.07556
5. https://arxiv.org/abs/1904.03240
6. https://arxiv.org/abs/1807.03748 (this paper is cited)
7. https://arxiv.org/abs/1803.08976
Edit: Based on the feedback from the authors, I changed my rating from a 'weak accept' to an 'accept'.