Neural system identification for large populations separating "what" and "where"
Neuroscientists classify neurons into different types that perform similar computations at different locations in the visual field. Traditional methods for neural system identification do not capitalize on this separation of "what" and "where". Learning deep convolutional feature spaces that are shared among many neurons provides an exciting path forward, but the architectural design needs to account for data limitations: While new experimental techniques enable recordings from thousands of neurons, experimental time is limited so that one can sample only a small fraction of each neuron's response space. Here, we show that a major bottleneck for fitting convolutional neural networks (CNNs) to neural data is the estimation of the individual receptive field locations -a problem that has been scratched only at the surface thus far. We propose a CNN architecture with a sparse readout layer factorizing the spatial (where) and feature (what) dimensions. Our network scales well to thousands of neurons and short recordings and can be trained end-to-end. We evaluate this architecture on ground-truth data to explore the challenges and limitations of CNN-based system identification. Moreover, we show that our network model outperforms current state-of-the art system identification models of mouse primary visual cortex.
I am no expert in this field, so right now I can not fully judge on the potential impact of the paper at hand in this field of science.
So first  I want to ask  a few clarifying questions to the authors to give me a clearer picture. I will also give a preliminary judgement at the end.
Introuction
34-47:  I think it should be mentioned that using more expressive models such like CNNs will come at a cost. Simple models such as GLMs are very interpretable, it also easier to argue for biological plausibility than with neural network trained by backpropagation. 
I do agree with your point about:"Thus, we should be able to learn much more complex nonlinear functions by combining data from many neurons and learning a common feature space from which we can linearly predict the activity of each neuron."
Related Work:
The approaches of McIntosh et al.[17] and Batty et al. [18] seem to be the most similar to yours. What were the practical and theoretical reasons for not comparing to them in your experiments?
3 Learning a common feature space
This section describes a well-known insight from transfer learning in ML. Maybe establishing a link here is useful as you use a ML technique for a neuroscience application.
5 Learning non-linear feature spaces
Is this comparison to GLMs fair? Your proposed architecture seems to share a lot of properties with the ground truth (such as convolutional layers). 
5.2 Application to data from primary visual cortex 
It seems you needed to make a few modifications to your architecture. Can you respond to a possible çritique that says your approach only works better because you have more degrees of freedom to adjust hyperparameters to a given task?  Without these adjustments, were you still able to improve over previous state of the art?
--
The paper is well written and my initial impression is that this is a good contribution. The intuition to combining data from many neurons and learning a common feature space is intuitive and  appears straightforward. I can't fully judge if this is novel or incremental, compared to the works of  McIntosh et al.[17] and Batty et al. and I have some open questions with regards to the experiments.