Learning Generative Models with Visual Attention
Attention has long been proposed by psychologists to be important for efficiently dealing with the massive amounts of sensory stimulus in the neocortex. Inspired by the attention models in visual neuroscience and the need for object-centered data for generative models, we propose a deep-learning based generative framework using attention. The attentional mechanism propagates signals from the region of interest in a scene to an aligned canonical representation for generative modeling. By ignoring scene background clutter, the generative model can concentrate its resources on the object of interest. A convolutional neural net is employed to provide good initializations during posterior inference which uses Hamiltonian Monte Carlo. Upon learning images of faces, our model can robustly attend to the face region of novel test subjects. More importantly, our model can learn generative models of new faces from a novel dataset of large images where the face locations are not known.
Summary:This paper proposes a deep learning based spatial attention mechanism based on a probabilistic generative model. The approach enables identifying novel objects in large images, and hence might allow for better exploitation of unlabeled, uncropped data.Main Comments:This paper takes a solid step towards being able to apply deep learning methods to large uncropped, unlabeled images. The motivation is compelling: previous DL approaches have required ‘curated’ training data that sticks the object essentially in the center with minimal clutter/occlusions—overcoming this limitation would truly enable learning from unlimited unlabeled data.The experimental section contains a number of interesting experiments verifying that the approximate inference method is working, and that HMC is still worthwhile beyond this. The network can perform some notable and rarely addressed tasks such as the ability to learn generative models from large images without labels, and the ability to shift attention by conditioning on various target stimuli.The paper claims that the algorithm runs in time O(1) given the size of the image. However initializations farther from the target image are likely to require more approximate inference steps, so there is an implicit dependence on image size. It could be interesting to plot the required number of steps as a function of image size to see this scaling. Also the window patch supplied to the conv net might also need to be increased for large images, so the scaling is not really O(1).The paper is clearly written and easy to follow.