A UNIFIED THEORY OF EARLY VISUAL REPRESENTA- TIONS FROM RETINA TO CORTEX THROUGH ANATOMI- CALLY CONSTRAINED DEEP CNNS
The vertebrate visual system is hierarchically organized to process visual information in successive stages. Neural representations vary drastically across the first stages of visual processing: at the output of the retina, ganglion cell receptive fields (RFs) exhibit a clear antagonistic center-surround structure, whereas in the primary visual cortex (V1), typical RFs are sharply tuned to a precise orientation. There is currently no unified theory explaining these differences in representations across layers. Here, using a deep convolutional neural network trained on image recognition as a model of the visual system, we show that such differences in representation can emerge as a direct consequence of different neural resource constraints on the retinal and cortical networks, and for the first time we find a single model from which both geometries spontaneously emerge at the appropriate stages of visual processing. The key constraint is a reduced number of neurons at the retinal output, consistent with the anatomy of the optic nerve as a stringent bottleneck. Second, we find that, for simple downstream cortical networks, visual representations at the retinal output emerge as nonlinear and lossy feature detectors, whereas they emerge as linear and faithful encoders of the visual scene for more complex cortical networks. This result predicts that the retinas of small vertebrates (e.g. salamander, frog) should perform sophisticated nonlinear computations, extracting features directly relevant to behavior, whereas retinas of large animals such as primates should mostly encode the visual scene linearly and respond to a much broader range of stimuli. These predictions could reconcile the two seemingly incompatible views of the retina as either performing feature extraction or efficient coding of natural scenes, by suggesting that all vertebrates lie on a spectrum between these two objectives, depending on the degree of neural resources allocated to their visual system.
This paper addresses questions about the representation of visual information in the retina. The authors create a deep neural network model of the visual system in which a single parameter (bandwidth between the “retina” and “visual cortex” parts) is sufficient to qualitatively reproduce retinal receptive fields observed across animals with different brain sizes, which have been hard to reconcile in the past. 
This work is an innovative application of deep neural networks to a long-standing question in visual neuroscience. While I have some questions about the analyses and conclusions, I think that the paper is interesting and of high quality.
My main concern is that the authors only show single examples, without quantification, for some main results (RF structure). For example, for Fig. 2A and 2B, an orientation selectivity index should be shown for all neurons. A similar population analysis should be devised for Fig 2C, e.g. like Fig 3 in [1]
Minor comments:
1. Page 4: “These results suggest that the key constraint ... might be the dimensionality bottleneck..”: The analyses only show that the bottleneck is *sufficient* to explain the differences, but “the key constraint” also implies *necessity*. Either soften the claim or provide control experiments showing that alternative hypotheses (constraint on firing rate etc.) cannot explain this result in your model.
2. I don’t understand most of the arguments about “cell types” (e.g. Fig. 2F and elsewhere). In neuroscience, “cell types” usually refers to cells with completely different connectivity constraints, e.g. excitatory vs. inhibitory cells or somatostatin vs. parvalbumin cells. But you refer to different CNN channels as different “types”. This seems very different than the neuroscience definition. CNN channels just represent different feature maps, i.e. different receptive field shapes, but not fundamentally different connectivity patterns. Therefore, I also don’t quite understand what you are trying to show with the weight-untying experiments (Fig. 2E/F).
3. It is not clear to me what Fig. 3B and the associated paragraph are trying to show. What are the implications of the nonlinearity being due to the first or second stage? 
4. Comment on Fig 3F: The center-surround RFs probably implement a whitening transform (which is linear). Whitened inputs can probably be represented more efficiently in a network trained with L2-regularization and/or SGD. This might explain why the “quasi-linear” retina improves separability later-on.
[1] Cossell, Lee, Maria Florencia Iacaruso, Dylan R. Muir, Rachael Houlton, Elie N. Sader, Ho Ko, Sonja B. Hofer, and Thomas D. Mrsic-Flogel. “Functional Organization of Excitatory Synaptic Strength in Primary Visual Cortex.” Nature 518, no. 7539 (February 19, 2015): 399–403. https://doi.org/10.1038/nature14182.