Correlations strike back (again): the case of associative memory retrieval
It has long been recognised that statistical dependencies in neuronal activity need to be taken into account when decoding stimuli encoded in a neural population. Less studied, though equally pernicious, is the need to take account of dependencies between synaptic weights when decoding patterns previously encoded in an auto-associative memory. We show that activity-dependent learning generically produces such correlations, and failing to take them into account in the dynamics of memory retrieval leads to catastrophically poor recall. We derive optimal network dynamics for recall in the face of synaptic correlations caused by a range of synaptic plasticity rules. These dynamics involve well-studied circuit motifs, such as forms of feedback inhibition and experimentally observed dendritic nonlinearities. We therefore show how addressing the problem of synaptic correlations leads to a novel functional account of key biophysical features of the neural substrate.
The authors study the role of correlations between 
      synaptic connections in a recurrent network induced by synaptic learning 
      rules. The authors separate the contribution of learning rules by making 
      the simplifying assumption that the activity patterns which drive learning 
      do not themselves have correlations. They proceed to discuss two classes 
      of learning rules, additive and cascade models, derive and examine the 
      differences in the correlations induced by them and their effect on the 
      accuracy, simplicity and plausibility of recall dynamics. The paper 
      follows in the path of several recent papers discussing recall dynamics as 
      probabilistic inference, is generally clearly written, interesting and has 
      substantial results that should be of interest to the general NIPS 
      community. Major comments The first part of the paper overlaps 
      with previously published papers. Though this introduction is important 
      and some relevant papers are referenced by the authors, it must be made 
      much clearer where the novel parts are found and which results are already 
      published. Minor comments In general, one can certainly 
      appreciate trying to write in a more engaging, elevated form but some 
      sentences are a bit too poetical. For instance, “assuaging an old enemy 
      leads to…” is a bit much. Lines 39-41: this sentence is too vague 
      and imprecise. As the authors themselves will state in a few pages, the 
      perfect symmetry of Hopfield networks (an obvious synaptic correlation) is 
      well known, and heavily studied issue. The authors should be more precise 
      with what kinds of correlations have not been explored and in what 
      context, or at least tone down this statement. Line 77-78: “recall 
      is inherently a probabilistic inference…” that is not the only approach, 
      rather the authors can claim that “recall can be seen as a probabilistic…” 
      Footnote page 2 “the the”, one “the” is a typo Line 
      390-392: “Statistical dependencies between synaptic efficacies are a 
      natural consequence of activity dependent synaptic plasticity, and yet 
      their implications for network function have been unexplored.” Again, this 
      statement has to be made more specific considering the Hopfield network 
      literature. Footnote #10: this statement should be put into the 
      discussion at the point where the footnote was referenced. Line 
      414-415: “the deeper is the cascade” should be “the deeper the cascade” 
      Post-feedback response: My comments were mostly minor, the 
      authors addressed them in their response and I am sure will be able to 
      correct everything for the final version.