LEMONADE: LEARNED MOTIF AND NEURONAL ASSEMBLY DETECTION IN CALCIUM IMAGING VIDEOS
Neuronal assemblies, loosely defined as subsets of neurons with reoccurring spatiotemporally coordinated activation patterns, or "motifs", are thought to be building blocks of neural representations and information processing. We here propose LeMoNADe, a new exploratory data analysis method that facilitates hunting for motifs in calcium imaging videos, the dominant microscopic functional imaging modality in neurophysiology. Our nonparametric method extracts motifs directly from videos, bypassing the difficult intermediate step of spike extraction. Our technique augments variational autoencoders with a discrete stochastic node, and we show in detail how a differentiable reparametrization and relaxation can be used. An evaluation on simulated data, with available ground truth, reveals excellent quantitative performance. In real video data acquired from brain slices, with no ground truth available, LeMoNADe uncovers nontrivial candidate motifs that can help generate hypotheses for more focused biological investigations.
The paper proposes a VAE-style model for identifying motifs from calcium imaging videos. As opposed to standard VAE with Gaussian latent variables it relies on Bernouli variables and hence, requires Gumbel-softmax trick for inference. Compared to methods based on matrix factorization, the proposed method has the advantage of not requiring any preprocessing on the imaging videos. My main comments are as follows:
- How sensitive is the method to the choice of beta and other hyperparameters?  Compared to SCC which has fewer hyperparameters, how robust is the method?
- How does it perform on real data compared to methods based on spike time matrices? Do they generate similar motifs? 
- The application of the method seems quite limited to calcium imaging videos and it does not provide comparison with other deep generative models for videos. Methods such as Johnson et al. NIPS 2016 (Composing graphical models with neural networks for structured representations and fast inference) can also be applied to calcium imaging datasets and can potentially infer the motifs.
I believe the problem of inferring the neural motifs is an interesting problem; however, I think this paper requires more work to it shows its advantages over other deep generative models for video data and also it’s performance on real data compared to SCC (or some other matrix factorization based approach). 
-----------------------------------------------------------------------
The authors have addressed my comments about other deep generative models and hyperparameter sensitivity. However, I still think the paper is more suitable for other venues with readers from the neuroscience community. Hence, I change my rating to 5.