Noise-Enhanced Associative Memories
Recent advances in associative memory design through structured pattern sets and graph-based inference algorithms allow reliable learning and recall of exponential numbers of patterns. Though these designs correct external errors in recall, they assume neurons compute noiselessly, in contrast to highly variable neurons in hippocampus and olfactory cortex. Here we consider associative memories with noisy internal computations and analytically characterize performance. As long as internal noise is less than a specified threshold, error probability in the recall phase can be made exceedingly small. More surprisingly, we show internal noise actually improves performance of the recall phase. Computational experiments lend additional support to our theoretical analysis. This work suggests a functional benefit to noisy neurons in biological neuronal networks.
SUMMARY In this article, the authors study a 
      model for associative memory where the patterns lie in some 
      low-dimensional manifold. The structure of the patterns is captured by the 
      so-called "constraint-neurons". Those constraint-neurons together with the 
      pattern neurons form the memory network. The authors propose a recall 
      algorithm for this network and show that intrinsic noise actually improves 
      recall performance. The paper is well written and as far as I can 
      judge, it is technically sound. MAJOR COMMENTS 1. 
      Biological plausibility. The paper starts with considerations on 
      associative memory and the brain. However, the associative memory model 
      seems very distant to any biological implementation. - For example, 
      how would step 4 in Algorithm 2 be implemented in some biologically 
      realistic network? This step assumes that some memory should be kept over 
      the tmax iterations of Algo 1 (c.f. Step 3 in Algo 3). - How should we 
      think of the integer-valued state of the neuron (are states spike vs 
      non-spike? this binary code (S = 2) would be different from the code in 
      the constraint neurons.) 2. As mentioned by the authors, the 
      architecture is similar to the one of Restricted Boltzman machines (RBM), 
      the pattern neurons being the visible neurons and the constraint neuron 
      being the hidden neurons. Now, with RBM, neurons are also intrinsically 
      noisy and the level of intrinsic noise is adapted (through synaptic 
      strength) such that the marginal distribution over visible units is 
      consistent with the data. So I would also expect that the performance 
      would be poorer with RBM if we forced neurons to be deterministic. 
      Therefore, it is not clear to what extend it is a breaking news that 
      intrinsic noise is good. Furthermore, the authors state that Deep Belief 
      Nets (and therefore RBM as well) are used for classification and not 
      pattern completion, but this is not correct. RBM can be used for pattern 
      completion. 3 Other approaches to associative memories have 
      been proposed. For example in [R1], Lengyel et al. see recall as a 
      Bayesian inference process which exploits knowledge about the structure of 
      the stored patterns (prior) and the knowledge about the learning rule as 
      well as knowledge about the corrupting process of the inputs (the 
      likelihood). In this perspective, the posterior distribution is a 
      deterministic function of the prior and the likelihood and does not 
      require additional noise. So how is it possible that the "best" recall 
      rule (in a Bayesian sense) does not need noise (except if one want to 
      sample from the posterior distribution) and the recall rule presented here 
      needs noise? [R1] Lengyel, M., Kwag, J., Paulsen, O., & Dayan, 
      P. (2005). Matching storage and recall: hippocampal spike timing–dependent 
      plasticity and phase response curves. Nature Neuroscience, 8(12), 
      1677–1683.