Working memory requires information about external stimuli to be represented in the brain even after those stimuli go away. This information is encoded in the activities of neurons, and neural activities change over timescales of tens of milliseconds. Information in working memory, however, is retained for tens of seconds, suggesting the question of how time-varying neural activities maintain stable representations. Prior work shows that, if the neural dynamics are in the 'null space' of the representation -so that changes to neural activity do not affect the downstream read-out of stimulus information -then information can be retained for periods much longer than the time-scale of individual-neuronal activities. The prior work, however, requires precisely constructed synaptic connectivity matrices, without explaining how this would arise in a biological neural network. To identify mechanisms through which biological networks can self-organize to learn memory function, we derived biologically plausible synaptic plasticity rules that dynamically modify the connectivity matrix to enable information storing. Networks implementing this plasticity rule can successfully learn to form memory representations even if only 10% of the synapses are plastic, they are robust to synaptic noise, and they can represent information about multiple stimuli.
A neural network model consisting of recurrently connected neurons and one or more readouts is introduced which aims to retain some output over time. A plasticity rule for this goal is derived. Experiments show the robustness of the network with respect to noisy weight updates, number of non-plastic connections, and sparse connectivity. Multiple consecutive runs increase the performance; furthermore, remembering multiple stimuli is possible. Finally, ideas for the biological implementation of the rule are suggested.
While the presentation is generally comprehensible a number of errors and deficits exist (see below). In general, this paper addresses a question that seems only relevant from a neuroscience perspective. Therefore, I wonder whether it is relevant in terms of the scope of this conference. I also think that the model is rather speculative. The authors argue that the resulting learning rule is biologically plausible. But even if this is the case, it does not imply that it is implemented in neuronal circuits in the brain. As far as I can see, there exists no experimental evidence for this rule. 
The paper shows the superiority of the proposed model over the approach of Druckmann & Chkolvskii (2012), however, it lacks in-depth analysis of the network behavior. Specifically, it is not clear how the information is stored. Do neurons show time-varying responses as in Druckmann & Chkolvskii (2012) or do all neuron stabilize within the first 50 ms (as in Fig. 2A, it is not detailed how the neurons shown there have been selected)? Do the weights change continuously within the delay period or do they also converge rapidly? This question is particularily important when considering multiple consecutive trials (cf. Fig. 5) as it seem that a specific but constant network architecture can retain the desired stimulus without further plasticity. Weight histograms should be presented for the different cases and network states. Also, since autapses are allowed, an analysis of their role should be performed. This information is vital to compare the model to findings from neuroscience and judge the biologic realism.
The target used is \hat{s}(t) / \hat{s}(t = 0), this is dubbed "fraction of stimulus retained". In most plots, the values for this measure are <= 1, but in Fig. 3A, the value (for the FEVER network) is > 1. Thus, the name is arguably not well-chosen: how can a fraction of remembrance be greater than one? Also, in a realistic environment, it is not clear that the neuronal activities decay to zero (resulting in \hat{s}(t) also approaching zero). A squared distances measure should therefore be considered.
It is not clear from the paper when and how often weight updates are performed. Therefore, the biologic plausability cannot be assessed, since the learning rule might lead to much more rapid changes of weights than the known learning rules in biological neural networks. Since the goal seems to be biologic realism, generally, spiking neurons should be used for the model. This is important as spiking neural networks are much more fragile than artificial ones in terms of stability.
Further remarks:
- In Sec. 3.2.1, noise is added to weight updates. The absolute values of alpha are hard to interpret since it is not clear in what range the weights, activities, and weight updates typically lie.
- In Sec. 3.2.2 it is shown that 10% plastic synapses is enough for reasonable performance. In this case, it should be investigated whether the full network is essential for the memory task at all (especially since later, it is argued that 100 neurons can store up to 100 stimuli).
- For biologic realism, just assuming that the readout value at t = 0 is the target seems a bit too simple. How does this output arise in the first place? At least, an argument for this choice should be presented.
Remarks on writing:
- Fig. 1A is too small to read.
- The caption of Fig. 4C is missing.
- In Fig. 7AB, q_i and q_j are swapped. Also, it is unclear in the figure to which connection the ds and qs belong.
- In 3.6.1, Fig. 7 is referenced, but in the figure the terminology of Eq. 5 is used, which is only introduced in Sec. 3.6.2. This is confusing.
- The beginning of Sec. 3.6 claims that all information is local except d\hat{s}_k / dt, but this is not the case as d_i is not local (which is explained later).
- The details of the "stimulus presentation" (i.e. it is not performed explicitly) should be emphasised in 2.1. Also, the description of the target \hat{s} is much clearer in 3.4 than in 2.1 (where it should primarily be explained).
- The title of the citation Cowan (2010) is missing.
- In Fig. 2A, the formulas are too small too read in a printed version.
- In Sec. 3.6.1 some sums are given over k, but k is also the index of a neuron in Fig. 7A (which is referenced there), this can be ambiguous and could be changed.