Identifying computational mechanisms for memorization and retrieval is a longstanding problem at the intersection of machine learning and neuroscience. In this work, we demonstrate empirically that overparameterized deep neural networks trained using standard optimization methods provide a mechanism for memorization and retrieval of real-valued data. In particular, we show that overparameterized autoencoders store training examples as attractors, and thus, can be viewed as implementations of associative memory with the retrieval mechanism given by iterating the map. We study this phenomenon under a variety of common architectures and optimization methods and construct a network that can recall 500 real-valued images without any apparent spurious attractor states. Lastly, we demonstrate how the same mechanism allows encoding sequences, including movies and audio, instead of individual examples. Interestingly, this appears to provide an even more efficient mechanism for storage and retrieval than autoencoding single instances.
This paper empirically demonstrates that DNNs can be trained to be identity mappings for small quantities of samples. It also demonstrates that for many parameterizations, these identity DNNs also have a small number of attractors, iterative fixed points, and can also learn short circular sequences of examples.
The paper is well written and easy to understand, although the presentation could be improved a bit (see comments). Its contents aren't particularly novel in terms of ideas, but they investigate memorization and attractors much further than previous studies. 
In a way, the memorization results are unsurprising. We know that DNNs can memorize perfectly, including sequences, so it is natural that by increasing capacity, at some point they should be able to memorize entire images (in fact this is what Zhang et al. (2019)'s Figure 1 appears to be showing). 
The more novel and surprising aspect of this is that DNNs would learn such strong (and so few) attractor basins. The fact that deep autoencoders could have attractors centered on the training points has been postulated before (see [1]), but this work makes a stronger case for it.
A crucial aspect that is missing from this paper in order for me to give in an accept is that there is very little about how this paper positions itself in the current literature. There could be much more discussion about related work, and much more discussion about the impacts of these findings.
I have given this paper a 'weak reject' mark but I think with some work this paper could be of interest to many. To reiterate, I am unable to see anything wrong with this paper, but at the same time I am unable to see how impactful these findings are.
Detailed comments:
- It's interesting that DNNs can implement associative memory, but what is the cost of doing that? Should we be using that in practice? Since there is no sense of how costly the presented experiments are, it is hard to tell.
- Again, these results are interesting, but after some time pondering about it, I can't really convince myself that knowing the results of this paper will be beneficial to future research. That being said, there are many areas of Machine Learning that I am unfamiliar with. It should be part of the paper to familiarize readers with areas where these results could be impactful.
- "the function interpolates the training images" not sure what this means. Interpolation means making a prediction for a point `u` that is "between" two points `x,y` with known values
- "black and white" should be "grayscale" if values are in [0,1]
- Figure 2b is interesting, but I wonder what happens if e.g. a perturbed version of e.g. Example 6 is fed. Presumably since Example 6 is not an attractor (Jacbian with an eigeinvalue > 1), it should converge to another example.
- Figure 2b's caption numbers, which say you use 1k example, to not correspond to numbers earlier in the text, which say you use 10k examples.
- "Since overparameterized autoencoders interpolate the training data", again this is a fairly important assumption and it needs to be defined very clearly, because it could mean many things.
- "it is essential that we interpolate to numerical precision", I don't think you are using the word "interpolate" correctly, do you mean "inference"? "train"?
- Adam citation should be "Adam: A Method for Stochastic Optimization, Diederik P. Kingma, Jimmy Ba", not Goodfellow et al., RMSprop should also have a citation, Hinton et al. 2012
- ReLU citation should be "Rectified linear units improve restricted Boltzmann machines, Nair & Hinton", Leaky ReLU should be Maas et al 2013, SELU should be Klambauer et al. 2017.
- The combination of section 3.1 and Figure 3 doesn't make it clear if models trained with Adam and RMSprop have weight decay or not. Can you clarify?
- "Note that a minimum width of 100 is needed to allow for interpolation." Again I think you mean "learning" rather than "interpolation".
- You say that you trained black and white images, but all the images of CIFAR10 in the figures are colored, including the inputs and outputs. Can you clarify why?
- You might be interested in [2], which is much older work about perceptrons, but still relevant to what is studied here.
- The linked supplemental material gives a 404 for me. I replicated the MNIST Figure 6 experiment. I was unable to replicate exactly your results but they were similar enough. In particular, the activation function choice seems to be critical.
[1] The Potential Energy of an Autoencoder, Hanna Kamyshanska, Roland Memisevic
[2] Basins of Attraction in a Perceptron-like Neural Network, Werner Krauth, Marc Mezard, Jean-Pierre Nadal