Fast amortized inference of neural activity from calcium imaging data with variational autoencoders
Calcium imaging permits optical measurement of neural activity. Since intracellular calcium concentration is an indirect measurement of neural activity, computational tools are necessary to infer the true underlying spiking activity from fluorescence measurements. Bayesian model inversion can be used to solve this problem, but typically requires either computationally expensive MCMC sampling, or faster but approximate maximum-a-posteriori optimization. Here, we introduce a flexible algorithmic framework for fast, efficient and accurate extraction of neural spikes from imaging data. Using the framework of variational autoencoders, we propose to amortize inference by training a deep neural network to perform model inversion efficiently. The recognition network is trained to produce samples from the posterior distribution over spike trains. Once trained, performing inference amounts to a fast single forward pass through the network, without the need for iterative optimization or sampling. We show that amortization can be applied flexibly to a wide range of nonlinear generative models and significantly improves upon the state of the art in computation time, while achieving competitive accuracy. Our framework is also able to represent posterior distributions over spike-trains. We demonstrate the generality of our method by proposing the first probabilistic approach for separating backpropagating action potentials from putative synaptic inputs in calcium imaging of dendritic spines.
This paper presents a variational auto-encoder-style method for extracting spike-trains from two-photon fluorescence microscopy time traces. This is a task that scientists are interested in for the purposes of interpreting one- two- or three- photon microscopy data, which are becoming a staple method in neuroscience. The authors introduce 3 different spike-to-fluorescence models and use one of two neural networks to create an decoder to recover the (discritized) spike trains from the fluorescence. Overall, the method does show some promise. That said, my biggest concern with this work lies with the characterization of the method. Deep-networks just like any other algorithm, need to have an understanding of their trade-offs and limitations. It is especially important in an applications that will be used to inform scientific conclusion to understand aspects of the method such as the the bias and sensitivity to model mis-match. Unfortunately, I feel that the results section fall short of such a characterization and additional simulations or theory needs to be established to instill confidence in this method. I have detailed below specific comments that I feel the authors should address to improve the manuscript.
Comments:
1) Is this problem even solvable in general? In the linear case (the first, most basic model) it seems that spikes can be inferred, however the nonlinearities used in the following models can make spikes indistinguishable without adequate time between spikes. Is there a limit to the inter-spike times that this method can distinguish in the non-linear case? Is this  a reasonable limit for neural data? 
2) In a related note, all the data (and simulations) in this work use 60Hz sampling rates for the fluorescence traces. Much of the fluorescence calcium literature (especially 2-photon imaging, which seems to be becoming dominant) sampling rates as low as 10-15Hz are not unheard of. How does this method perform when the sampling rates fall to 30Hz or 10Hz, when spikes may truly be indistinguishable? Additionally, to have the discrete spike-train output retain a sampling rate on the order of a refractory period, would the proposed networks have an a-symmetry between the input and output rates, or would the network need to be changed to allow for multiple spikes per bins?
3) On line 66, the authors claim to use a "powerful recurrent neural network" model. I assume the authors don't mean 'powerful' in the statistical testing sense, so such embellishment is unnecessary. 
4) The authors often claim that the network learned in one sense can generalize well to other settings (e.g. line 61). All the testing and training were done with i.i.d. spikes constant rates chosen from one of three values. I did not see in the results any testing from the network trained with this constrained data-set on fluorescence traces generated from more complex spike-rates. Do the authors have evidence of this generalization, or is this a statement provided based on the use of VAEs in other contexts? If it is the former, the authors should provide these results to validate their claim. If it is the latter, the authors should verify that this generalization holds true for their application, as properties of VAEs do not necessarily translate across domains. 
5) The authors present three different models with various levels of complexity, however I do not see any results as to how the proposed network inference methodology changes with respect to each of these networks. It would be informative to see how the results, say on a single V1 trace, change with respect to the model used to train the network. 
6) Related to the previous point, sometimes it is unclear which model is being used in which networks for the various results (e.g. what model is used for the V1 results?).
7) In Figure 3, the reconstruction base on the true spike train using (was this using the SCF or SCDF model?) is not as accurate as the results of the spikes inferred using the network trained on that model. How is this possible? This discrepancy warrants at least some discussion. Clearly there is a model-mismatch of some kind. Is the network's ability to achieve better prediction mean that the method has a built-in bias in the estimated spikes that stems from the model mismatch? This seems like an important point that is glossed over. A systematic bias in the inferred spikes can create errors in scientific analyses applied to the resulting spike trains. 
8) The work "amortize" is useless in the context of machine learning and only serves to further obfuscate our field further with respect to outside audiences that may seek to use these methods. I very highly recommend omitting this word from the paper. 
2) Acronyms should be defined where first used (e.g. in all the sub-sub-section titles for the various models)
3) Most of the figures could do with bigger font sizes