Select-and-Sample for Spike-and-Slab Sparse Coding
Probabilistic inference serves as a popular model for neural processing. It is still unclear, however, how approximate probabilistic inference can be accurate and scalable to very high-dimensional continuous latent spaces. Especially as typical posteriors for sensory data can be expected to exhibit complex latent dependencies including multiple modes. Here, we study an approach that can efficiently be scaled while maintaining a richly structured posterior approximation under these conditions. As example model we use spike-and-slab sparse coding for V1 processing, and combine latent subspace selection with Gibbs sampling (selectand-sample). Unlike factored variational approaches, the method can maintain large numbers of posterior modes and complex latent dependencies. Unlike pure sampling, the method is scalable to very high-dimensional latent spaces. Among all sparse coding approaches with non-trivial posterior approximations (MAP or ICAlike models), we report the largest-scale results. In applications we firstly verify the approach by showing competitiveness in standard denoising benchmarks. Secondly, we use its scalability to, for the first time, study highly-overcomplete settings for V1 encoding using sophisticated posterior representations. More generally, our study shows that very accurate probabilistic inference for multi-modal posteriors with complex dependencies is tractable, functionally desirable and consistent with models for neural inference.
The paper considers the problem of sparse coding, also known as dictionary learning. It is here expressed as a maximum likelihood problem where one wishes to fit to some training data the parameters of a Bernoulli-Gaussian (“spike and slab”) prior on a sparse latent variable of (potentially large) dimension H, the components W of a dictionary, and the variances of the observation noise. To circumvent the intractability of the problem for large H, a technique is proposed which combines ideas from iterative hard thresholding (via a “selection function” which actually keeps the H’ < <  H components most correlated to the observation given the current parameters) and expectation maximization with Gibbs sampling. Numerical experiments show that, with an appropriate choice of H’, the approach behaves well both in controlled synthetic experiments (in low dimension) and in one denoising experiments with large H. Further, a discussion of the shape and nature of the learned dictionary atoms is provided. 
Overall this is an interesting and fairly clear algorithmic and experimental paper paper presenting an approximate EM technique for dictionary learning in a Bernoulli-Gaussian setting.    The main idea is that EM requires sampling to approximate certain intractable expectations/integrals, and that such sampling is inaccurate or difficult in high dimensions H. It is proposed to perform sampling only after a selection step which consists in finding a subspace (dependent on the considered training vector as well as the currently estimated parameters of the model) of intermediate dimension H’ << H.   As far as I could check the selection step, expressed implicitly in (10)-(11) corresponds to computing certain weighted correlations of the training sample y_n with columns of the current estimate of the dictionary, and to keep the H’ largest correlations. This essentially boils down to one step of hard thresholding, reminiscent of certain recent techniques for dictionary learning (see, e.g., recent work of K. Schnass such as [A]).    Numerical experiments on synthetic data show state of the art performance of the proposed approach. An discussion on the shape of the atoms learned on the “house” image is provided.  On the one hand this is interesting and intriguing, as one can see for example some “curved” atoms similar to those appearing in the work of Jost et al [B], and I am curious see if the approach can be extended to factor invariances (translation, rotation) in the learning stage. On the other hand, it is a bit of a pity that the experiment is only conducted on a single image, since this seriously limits the interpretability of the results.    The paper is concluded by the potentially interesting Figure 3 which aims at presenting a synthetic view on existing algorithms based on two axes: “accuracy of posterior approximation” and “latent dimension” (a measure of scalability). I welcome this type of synthetic representations in principle. However I am a bit bothered by the fact that the positioning on the axes (especially the horizontal one) seem to be subjective rather than objective. It would be really interesting to draw such a picture, but one would need to agree for example on the notion of “accuracy of posterior approximation”.  One way to do it may be to conduct large-scale empirical simulations and use agnostic performance measures.   Since the proposed technique is claimed to be scalable, I would have expected some mention of computation times besides pure log-likelihood figures .   Please carefully revise the english writing, e.g.: the sentence line 21-22 has two verbs, line 43 “to in general study”, line 80 “which have in similar forms be derived …”line 181 “implementation following and execute”, line 206 “obtaine”,line 244 “the for MAP based approaches required choice”.  [A] Schnass, K. (2014, January 24). Local Identification of Overcomplete Dictionaries. arXiv.org. [B] Jost, P., Vandergheynst, P., Lesage, S., & Gribonval, R. (2006). MoTIF : an Efficient Algorithm for Learning Translation Invariant Dictionaries (Vol. 5, pp. V–V). Presented at the Acoustics, Speech and Signal Processing, 2006. ICASSP 2006. IEEE International Conference on, Toulouse, France: IEEE. http://doi.org/10.1109/ICASSP.2006.1661411