The Kanerva Machine: A Generative Distributed Memory
We present an end-to-end trained memory system that quickly adapts to new data and generates samples like them. Inspired by Kanerva*s sparse distributed memory, it has a robust distributed reading and writing mechanism. The memory is analytically tractable, which enables optimal on-line compression via a Bayesian update-rule. We formulate it as a hierarchical conditional generative model, where memory provides a rich data-dependent prior distribution. Consequently, the top-down memory and bottom-up perception are combined to produce the code representing an observation. Empirically, we demonstrate that the adaptive memory significantly improves generative models trained on both the Omniglot and CIFAR datasets. Compared with the Differentiable Neural Computer (DNC) and its variants, our memory model has greater capacity and is significantly easier to train.
 The paper presents the Kanerva Machine, extending an interesting older conceptual memory model to modern usage. The review of Kanerva’s sparse distributed memory in the appendix was appreciated. While the analyses and bounds of the original work were only proven when restricted to uniform and binary data, the extensions proposed bring it to modern domain of non-uniform and floating point data. The iterative reading mechanism which provides denoising and reconstruction when within tolerable error bounds, whilst no longer analytically provable, is well shown experimentally. The experiments and results on Omniglot and CIFAR provide an interesting insight to the model*s behaviour with the comparisons to VAE and DNC also seem well constructed. The discussions regarding efficiency and potential optimizations of writing inference model were also interesting and indeed the low rank approximation of U seems an interesting future direction. Overall I found the paper well written and reintroduced + reframed a relatively underutilized but well theoretically founded model for modern use.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
The Kanerva Machine: A Generative Distributed Memory
We present an end-to-end trained memory system that quickly adapts to new data and generates samples like them. Inspired by Kanerva*s sparse distributed memory, it has a robust distributed reading and writing mechanism. The memory is analytically tractable, which enables optimal on-line compression via a Bayesian update-rule. We formulate it as a hierarchical conditional generative model, where memory provides a rich data-dependent prior distribution. Consequently, the top-down memory and bottom-up perception are combined to produce the code representing an observation. Empirically, we demonstrate that the adaptive memory significantly improves generative models trained on both the Omniglot and CIFAR datasets. Compared with the Differentiable Neural Computer (DNC) and its variants, our memory model has greater capacity and is significantly easier to train.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 The paper presents the Kanerva Machine, extending an interesting older conceptual memory model to modern usage. The review of Kanerva’s sparse distributed memory in the appendix was appreciated. While the analyses and bounds of the original work were only proven when restricted to uniform and binary data, the extensions proposed bring it to modern domain of non-uniform and floating point data. The iterative reading mechanism which provides denoising and reconstruction when within tolerable error bounds, whilst no longer analytically provable, is well shown experimentally. The experiments and results on Omniglot and CIFAR provide an interesting insight to the model*s behaviour with the comparisons to VAE and DNC also seem well constructed. The discussions regarding efficiency and potential optimizations of writing inference model were also interesting and indeed the low rank approximation of U seems an interesting future direction. Overall I found the paper well written and reintroduced + reframed a relatively underutilized but well theoretically founded model for modern use.
assistant
 The paper presents the Kanerva Machine, extending an interesting older conceptual memory model to modern usage. The review of Kanerva’s sparse distributed memory in the appendix was appreciated. While the analyses and bounds of the original work were only proven when restricted to uniform and binary data, the extensions proposed bring it to modern domain of non-uniform and floating point data. The iterative reading mechanism which provides denoising and reconstruction when within tolerable error bounds, whilst no longer analytically provable, is well shown experimentally. The experiments and results on Omniglot and CIFAR provide an interesting insight to the model*s behaviour with the comparisons to VAE and DNC also seem well constructed. The discussions regarding efficiency and potential optimizations of writing inference model were also interesting and indeed the low rank approximation of U seems an interesting future direction. Overall I found the paper well written and reintroduced + reframed a relatively underutilized but well theoretically founded model for modern use.