What Are the Invariant Occlusive Components of Image Patches? A Probabilistic Generative Approach
We study optimal image encoding based on a generative approach with non-linear feature combinations and explicit position encoding. By far most approaches to unsupervised learning of visual features, such as sparse coding or ICA, account for translations by representing the same features at different positions. Some earlier models used a separate encoding of features and their positions to facilitate invariant data encoding and recognition. All probabilistic generative models with explicit position encoding have so far assumed a linear superposition of components to encode image patches. Here, we for the first time apply a model with non-linear feature superposition and explicit position encoding for patches. By avoiding linear superpositions, the studied model represents a closer match to component occlusions which are ubiquitous in natural images. In order to account for occlusions, the non-linear model encodes patches qualitatively very different from linear models by using component representations separated into mask and feature parameters. We first investigated encodings learned by the model using artificial data with mutually occluding components. We find that the model extracts the components, and that it can correctly identify the occlusive components with the hidden variables of the model. On natural image patches, the model learns component masks and features for typical image components. By using reverse correlation, we estimate the receptive fields associated with the model's hidden units. We find many Gabor-like or globular receptive fields as well as fields sensitive to more complex structures. Our results show that probabilistic models that capture occlusions and invariances can be trained efficiently on image patches, and that the resulting encoding represents an alternative model for the neural encoding of images in the primary visual cortex.
The paper describes a new generative model of images, 
      in which low-level features are first shifted and then combined according 
      to a nonlinear, stochastic, masking process. The authors develop 
      approximate inference and learning algorithms, and demonstrate results on 
      grayscale image patches. The paper is clearly written, well 
      organized, and easy to follow. It introduces a combination of two 
      previously explored ideas (translation invariance and occlusive image 
      generation), so conceptually it is somewhat of an incremental advance, but 
      the approximations to inference of occluding components are novel and lead 
      to a new structure for model parameters (feature weights and mask 
      probabilities). Although the results are very similar to previously 
      reported feature learning algorithms, they seem promising, especially if 
      such a model could be extended hierarchically. My main concern is 
      with the focus of the paper: is the goal to generate predictions and 
      theories for biological processing, or is it to propose a new set of 
      representations more useful for computer vision? If the focus is 
      on computer vision, the authors should explain why this solution to 
      occlusion is better than other occlusive models (including max- rule for 
      feature combinations, dead leaves models, and masked RBM by Le Roux, 
      Heess, Shotton, Winn, 2011), and also why translation invariance makes the 
      model more tractable than convolutional models. As it is, this paper 
      presents another alternative to occlusive and translating models (though 
      it unifies the two computations). If the aim is to provide a 
      theoretical result for neuroscience, the authors should emphasize what 
      kind of predictions this model makes (or what it explains about observed 
      properties of neurons in visual cortex). The prevalence of center-surround 
      receptive fields has been noted and modeled previously. Several theories 
      have been proposed for translation invariance in complex cells, and some 
      models even derive this directly from objective functions like information 
      maximization or temporal stability of the representation. If this model is 
      to be taken seriously in the context of brain processing, specific, novel 
      predictions or explanations should be offered, and aspects of the model 
      that are not biologically plausible (like the complete translation 
      invariance) should be addressed in the discussion. I recommend backing off 
      the neuroscientific claims unless these can be strengthened sufficiently 
      to be useful to experimentalists. Other comments: 
      What is the benefit of the stochastic component assignment over 
      choosing pixel value with a max rule, as in (Puertas et al, NIPS2010)? 
      Also, the all-or- none activation of the features seems like a limitation 
      of the proposed model. Is it possible to relate the (feed-forward) 
      operations in a convolutional neural network to performing approximate 
      inference with expectation truncation? What exactly are the benefits of 
      probabilistic pooling? Why all the work to compute the "estimated 
      Receptive Fields"? For visualizing and interpreting model parameters, the 
      mask-feature product seems to work quite well. As a comparison to biology, 
      the translation invariant receptive field is not very appropriate: complex 
      cells are not "fully translation invariant" as claimed in the Discussion 
      (so it's not a good characterization of a complex cell's behavior), and 
      for simple-like cells, linear receptive fields are estimated using direct 
      regression methods. If model units are to be interpreted as populations of 
      cells, then wouldn't a convolutional network with replicated receptive 
      fields be a better model? As an aside, new methods are being developed to 
      characterize the features encoded by translation invariant neurons (e.g., 
      Eickenberg, Rowekamp, Kouh, Sharpee, 2012; Vintch, Zaharia, Movshon, 
      2012). These might be worth citing, though there isn't much data analyzing 
      large neural populations yet. In the last paragraph, there is a 
      mention of building hierarchical versions of this model. I am curious if 
      the authors have more specific ideas of how multi-layered occlusive models 
      can be constructed, and what kind of features they will extract from 
      natural images. Specifically, would the layering/transparency be 
      interpreted similarly at higher levels of the hierarchy, or would it 
      simply add a nonlinear stochastic component to a deep model? Results 
      presented here are not strikingly different from many other learning 
      algorithms, so it is important to show that extensions to the model have 
      promise. Minor comments: How are image patch 
      boundaries handled during translation? I am assuming the masks are 
      constrained to be nonnegative, but the text does not specify. What 
      is the motivation for prefiltering with center-surround? It's true that 
      this is comparable to the (linear component of the) transformation 
      performed in LGN, but receptive fields are experimentally derived by 
      correlating to pixel stimuli on the screen, not LGN outputs. It 
      would be helpful if a sentence or two in the paper listed all the 
      approximations required to make the model tractable (expectation 
      truncation, independent pixel occlusion). Do you have any insight 
      as to why all globular components have positive centers?