Recurrent neural networks (RNNs) are difficult to train on sequence processing tasks, not only because input noise may be amplified through feedback, but also because any inaccuracy in the weights has similar consequences as input noise. We describe a method for denoising the hidden state during training to achieve more robust representations thereby improving generalization performance. Attractor dynamics are incorporated into the hidden state to 'clean up' representations at each step of a sequence. The attractor dynamics are trained through an auxillary denoising loss to recover previously experienced hidden states from noisy versions of those states. This state-denoised recurrent neural network (SDRNN) performs multiple steps of internal processing for each external sequence step. On a range of tasks, we show that the SDRNN outperforms a generic RNN as well as a variant of the SDRNN with attractor dynamics on the hidden state but without the auxillary loss. We argue that attractor dynamics-and corresponding connectivity constraints-are an essential component of the deep learning arsenal and should be invoked not only for recurrent networks but also for improving deep feedforward nets and intertask transfer.
In this paper the authors develop the clever idea to use attractor networks, inspired by Hopfield nets, to “denoise” a recurrent neural network.  The idea is that for every normal step of an RNN, one induces an additional "dimension" of recurrency in order to create attractor dynamics around that particular hidden state. The authors introduce their idea and run some basic experiments. This paper is well written and the idea is novel (to me) and worthy of exploration.  Unfortunately, the experiments are seriously lacking in my opinion, as I believe *the major focus* of those experiments should be comparisons to other denoising / regularization techniques.
MAJOR
The point is taken that RNNs are susceptible to noise due to iterated application of the function. In my experience, countering noise (in the sense of gaussian noise added) isn’t a huge problem in practice because there are many regularization methodologies to handle it. This leads me to the point that I think the experiments need to compare across a number of regularization techniques.  The paper is motivated by discussion of noise, “noise robustness is a highly desirable property in neural networks”, and the experiments show improved performance on smaller datasets, all of which speak to regularization. So I believe comparisons with regularization techniques are pretty important here. 
MODERATE
There is some motivation at the beginning of this piece, in particular about language, and does not contain citations, but should.
“Training is in complete batches to avoid the noise of mini-batch training.”  Please explain, I guess this is not a type of noise that the method handles? 
What about problems that require graded responses, which is likely anything requiring integration? For example,  what happens in the majority task if the inputs were switched to a non-discrete version, where one must hold analog numbers?
MINOR
Any discussion about the (presumably dramatic) increase in training time due to the attractor dynamics unrolling + additional batching due to noise vectors (if I understood correctly)?
What are your confidence intervals over?  Presumably, we’d like to get confidence over multiple network instantiations.
Pg 1. Articulated neural network? 
QUESTIONS
Does using a the ‘c’ variable as a bias instead of an initial condition really matter? 
How does supervised training via eqn (4) relate to the classic training of Hopfield nets? I assume not at all, but it would be useful to clarify?
What RNN architecture did you use in the Figure 5 simulations (tanh vanilla RNN or GRU?)