Learning Multi-level Sparse Representations
Bilinear approximation of a matrix is a powerful paradigm of unsupervised learning. In some applications, however, there is a natural hierarchy of concepts that ought to be reflected in the unsupervised analysis. For example, in the neurosciences image sequence considered here, there are the semantic concepts of pixel → neuron → assembly that should find their counterpart in the unsupervised analysis. Driven by this concrete problem, we propose a decomposition of the matrix of observations into a product of more than two sparse matrices, with the rank decreasing from lower to higher levels. In contrast to prior work, we allow for both hierarchical and heterarchical relations of lower-level to higher-level concepts. In addition, we learn the nature of these relations rather than imposing them. Finally, we describe an optimization scheme that allows to optimize the decomposition over all levels jointly, rather than in a greedy level-by-level fashion. The proposed bilevel SHMF (sparse heterarchical matrix factorization) is the first formalism that allows to simultaneously interpret a calcium imaging sequence in terms of the constituent neurons, their membership in assemblies, and the time courses of both neurons and assemblies. Experiments show that the proposed model fully recovers the structure from difficult synthetic data designed to imitate the experimental data. More importantly, bilevel SHMF yields plausible interpretations of real-world Calcium imaging data.
The authors present an algorithm for learning 
      multilevel sparse representations. This is modeled as a (penalized) low 
      rank matrix factorization problem where the number of factors is typically 
      greater than 2 and each factor represents a different entity in the 
      hierarchy. The factors are estimated using standard block co-ordinate 
      descent methods where each factor is optimized while the others are kept 
      first and sparsity promoting penalties are used at each step. The 
      method is discussed in the context of analyzing large scale calcium 
      imaging data where the assumption is that the neurons can be collected 
      into a set of (possibly overlapping) neuronal assemblies giving rise to 
      the hierarchy pixel --> neuron --> assembly. The goal then is to 
      estimate the location of each neuron (in terms of pixels), the membership 
      of each neuronal assembly, and the spikes of each assembly. The 
      authors argue that their method performs better than other methods in 
      artificial (but realistic) data and that it can correctly identify neural 
      assemblies in real data. Originality The ideas 
      presented in this paper are novel to the best of my knowledge. The 
      three-way decomposition approach is interesting and I liked the fact that 
      the authors also keep a standard two-way decomposition in their model and 
      enforce consistency between the two. This extra step can make the 
      algorithm more robust to phenomena such as a neuron in an assembly 
      "misses" a spike etc. Quality My main concern about this 
      paper is efficiency. Large scale calcium imaging experiments typically 
      involve tens of thousands of pixels and thousand of timesteps. Although 
      the multilevel decomposition reduces significantly the dimensionality, and 
      several steps of the algorithm can bee parallelized, I can imagine that 
      the algorithm might still be quite slow and require a large number of 
      iterations. These points should be discussed. I am also not 
      convinced by the usage of neural assemblies. It is not clear that this is 
      something that will often appear in practice. Also, if a subset of neurons 
      have the same spikes then this will be reflected in the temporal component 
      of the spatiotemporal matrix. The rank of the matrix will still be small 
      and I don't see a direct benefit from incorporating the proposed 
      multilevel hierarchical decomposition. The authors would probably need to 
      expand their opinion on this issue. Moreover, the various 
      regularizers \Omega_U, \Omega_A should be clarified. For example, what are 
      the convex cell shape priors that are used? The authors leave to 
      the user the choice of regularization weights. This seems an important 
      task to me since the number of regularizers is large and their use of 
      critical importance. The authors claim that an accurate prior estimation 
      of the rank is not of great importance since the algorithm will shrink non 
      relevant components to zero. Although this sounds intuitive I can imagine 
      that this behavior depends heavily on the right choice of the various 
      sparsity penalties that will shrink the non relevant components. 
      Clarity I think the paper is not clearly written at 
      all and right now is only accessible to an expert of calcium imaging data 
      analysis. For example, the authors mention very briefly in the results 
      section how the neural spikes are translated into calcium activity through 
      a convolution operator and do not discuss this central aspect of spike 
      deconvolution at all in their algorithm. In fact it seems to me that they 
      sidestep this issue "by eliminating all transients that overlap more than 
      20%" (if I understand correctly). This is a topic of active research (e.g. 
      Vogelstein et al. 2010, Grewe et. al. 2010) and I'd like to see how the 
      authors relate to that literature. I found the inclusion of the 
      main equations as a figure a bit unusual, although that is a question of 
      style as well. Finally, I would prefer a more thorough 
      presentation of the real data experiments. Although videos cannot be 
      presented easily in paper, the results as shown in Fig 6 could be more 
      convincing. In general, Fig 6 needs a better 
      explanation.