Neural Networks for Efficient Bayesian Decoding of Natural Images from Retinal Neurons
Decoding sensory stimuli from neural signals can be used to reveal how we sense our physical environment, and is valuable for the design of brain-machine interfaces. However, existing linear techniques for neural decoding may not fully reveal or exploit the fidelity of the neural signal. Here we develop a new approximate Bayesian method for decoding natural images from the spiking activity of populations of retinal ganglion cells (RGCs). We sidestep known computational challenges with Bayesian inference by exploiting artificial neural networks developed for computer vision, enabling fast nonlinear decoding that incorporates natural scene statistics implicitly. We use a decoder architecture that first linearly reconstructs an image from RGC spikes, then applies a convolutional autoencoder to enhance the image. The resulting decoder, trained on natural images and simulated neural responses, significantly outperforms linear decoding, as well as simple point-wise nonlinear decoding. These results provide a tool for the assessment and optimization of retinal prosthesis technologies, and reveal that the retina may provide a more accurate representation of the visual scene than previously appreciated.
The paper describes a method to decode natural images from retinal-like activities, using convolutional neural networks. The retinal-like activities are generated by a constructed lattice of linear-nonlinear-Poisson models (separately fitted to RGC responses to natural scenes in a macaque retina preparation) in response to natural static images. After a simple linear decoding of the images from the retinal-like activities, a convolutional neural network further improves on the reconstruction of the original natural images. The paper is clearly written and the results seem sound. A few comments to clarify the motivation, assumptions and impact of the work:
-the method proposed is compared to a linear decoder and shown to perform substantially better. However, the performance of the decoding stage will most likely depend on the performance of the encoding stage. A more explicit discussion on the fact that the results of the CAE will ultimately depend on the encoding would clarify the scope and limitations of the study;
-on a related note, the authors should be explicit about the reasons for choosing a synthetic encoding stage: why didn’t the authors directly fit their method to the data without the synthetic encoding stage? Too few cells recorded? Too few natural images for fitting the CAE? An attempt at excluding the temporal component of the cells responses, so that the problem reduces to decoding static natural images instead of movies (which would be a more complex task)? This should be more explicitly stated;
-in Macintosh et al. 2017 (https://arxiv.org/abs/1702.01825), a convolutional neural network is shown to be a good encoding model of the RGCs in response to natural movies. The authors should discuss their method in the context of such study; 
-a plot illustrating how the performance of the CAE decoder depends on the amount of data used for training the network would be instrumental for assessing how powerful and practical the approach is; 
-the authors show that the CAE decoder is able to capture phase structure, unlike the linear decoder. Is the CAE better than the linear decoder mostly because it captures the phase structure? How does the performance between CAE and linear decoder compare in phase scrambled images? It would be helpful to comment on this, and perhaps explicitly plot such comparison.
Some typos/minor comments:
-lines 83-84, the authors write “resulting response of each neuron…displayed in Figure 3”, however no neural responses are displayed in Figure 3;
-line 104, the authors write “all tuned through an exhaustive grid-search”. Providing more details about this tuning would improve the clarity of the paper, and allow for a better assessment of the usability of the method presented;
-line 112, the authors write “i = {1,2,…,n}”. However, ’n’ is the number of neurons. Is ’n’ also the number of training epochs?
-- After the rebuttal --
The rebuttal has addressed satisfactorily all my concerns/comments, and therefore I would like to increase my score to a clear accept (score: 8).