THE KANERVA MACHINE: A GENERATIVE DISTRIBUTED MEMORY
We present an end-to-end trained memory system that quickly adapts to new data and generates samples like them. Inspired by Kanerva's sparse distributed memory, it has a robust distributed reading and writing mechanism. The memory is analytically tractable, which enables optimal on-line compression via a Bayesian update-rule. We formulate it as a hierarchical conditional generative model, where memory provides a rich data-dependent prior distribution. Consequently, the top-down memory and bottom-up perception are combined to produce the code representing an observation. Empirically, we demonstrate that the adaptive memory significantly improves generative models trained on both the Omniglot and CIFAR datasets. Compared with the Differentiable Neural Computer (DNC) and its variants, our memory model has greater capacity and is significantly easier to train.
The generative model comprises a real-valued matrix M (with a multivariate normal prior) that serves
as the memory for an episode (an unordered set of datapoints). For each datapoint a marginally independent
latent variable y_t is used to index into M and realize a conditional density
of another latent variable z. z_t is used to generate the data.
The proposal of learning with a probabilistic memory is interesting and the framework proposed is elegant and cleanly explained. The model is evaluated on the following tasks:
* Qualitative results on denoising and one-shot generation using the Omniglot dataset.
* Qualitative results on sampling from the model using the CIFAR dataset.
* Likelihood estimation on the Omniglot dataset
Questions and concerns: 
The model appears novel and is interesting, the experiments, however, are lacking in that they
do not compare against other any recently proposed memory augmented deep generative models [Bornschein et al] and [Li et. al] (https://arxiv.org/pdf/1602.07416.pdf). At the very minimum, the paper should include a discussion and a comparison with the latter. Doing so will help better understand what is gained from using retaining a probabilistic form of memory versus a determinstic memory indexed with attention as in [Li et. al].
How does the model perform as a function of varying T (size of episodes) during training? It would be interesting to see how well the model performs in the limiting case of T=1.
What is the task being solved in Section 4.4 by the DNC and the Kanerva machine? Please state this in the main paper.
Training and Evaluation: There is a mismatch in the training and evaluation procedure the implications of which I don't
fully understand yet. The text states that the model was trained where each observation in an episode comprised randomly sampled datapoints. This corresponds to a generative process where (1) a memory is randomly drawn, (2) each observation in the episode is an independent draws from the memory conditioned decoder. During training,
points in an episode are randomly selected. At test time, (if I understand correctly, please correct me if I haven't), the model is evaluated by having multiple copies of the same test point within an episode. Is that correct? If so, doesn't that correspond to evaluating the model under a different generative assumption? Why is this OK?
Likelihood evaluation: Could you expand on how the ELBO of 68.3 is computed under the model for a single test image in the Omniglot dataset? The text says that the likelihood of each data-point was divided by T (the length of the episode considered). This seems at odds with models, such as DRAW, evaluate the likelihood -- once at the end of the generative drawing process. What is the per-pixel likelihood obtained on the CIFAR dataset and what is the likelihood on a model where T=1 (for omniglot/cifar)?
Using Labels: Following up on the previous point, what happens if labelled information from Omniglot or CIFAR is used to define points within an episode during the training procedure? Does this help or hurt performance?
For the denoising comparison, how do the results compare to those obtained if you simulate a Markov Chain (sample latent state conditioned on noisy image, sample latent state, sample denoised observation, repeat using denoised observation) using a VAE?