Neural Discrete Representation Learning
Learning useful representations without supervision remains a key challenge in machine learning. In this paper, we propose a simple yet powerful generative model that learns such discrete representations. Our model, the Vector QuantisedVariational AutoEncoder (VQ-VAE), differs from VAEs in two key ways: the encoder network outputs discrete, rather than continuous, codes; and the prior is learnt rather than static. In order to learn a discrete latent representation, we incorporate ideas from vector quantisation (VQ). Using the VQ method allows the model to circumvent issues of "posterior collapse" --where the latents are ignored when they are paired with a powerful autoregressive decoder --typically observed in the VAE framework. Pairing these representations with an autoregressive prior, the model can generate high quality images, videos, and speech as well as doing high quality speaker conversion and unsupervised learning of phonemes, providing further evidence of the utility of the learnt representations.
Summary-
This paper describes a VAE model in which the latent space consists of
discrete-valued (multinomial) random variables. VAEs with multinomial latent
variables have been proposed before. This model differs in the way it computes
which state to pick and how it sends gradients back. Previous work has used
sampled softmaxes, and other softmax-based ideas (for example, trying to start
with something more continuous and anneal to hard). In this work, a VQ approach
is used, where a hard decision is made to pick the closest embedding space
vector. The gradients are sent into both into the embedding space and the
encoder.
Strengths
- The model is tested on many different datasets including images, audio, and video datasets.
- The model gets good results which are comparable to those obtained with continuous latent variables.
- The paper is well-written and easy to follow.
Weaknesses-
- Experimental comparisons with other discrete latent variable VAE approaches
are not reported. It is mentioned that the proposed model is the first to
achieve comparable performance with continuous latent variable models, but it
would be useful to provide a quantitative assessment as well.
- The motivation for having discrete latent variables is not entirely
convincing. Even if the input space is discrete, it is not obvious why the
latent space should be. For example, the paper argues that "Language is
inherently discrete, similarly speech is typically represented as a sequence of
symbols" and that this makes discrete representations a natural fit for such
domains. However, the latent semantic representation of words or speech could
still be continuous, which would make it possible to capture subtle inflections
in meaning or tone which might otherwise be lost in a discretized or quantized
representation. The applications to compression are definitely more convincing.
Overall-
The use of VQ to work with discrete latent variables in VAEs is a novel
contribution. The model is shown to work well on a multiple domains and
datasets. While the qualitative results are convincing, the paper can be made
more complete by including some more comparisons, especially with other ways of
producing discrete latent representations.