Deep Networks with Internal Selective Attention through Feedback Connections
Traditional convolutional neural networks (CNN) are stationary and feedforward. They neither change their parameters during evaluation nor use feedback from higher to lower layers. Real brains, however, do. So does our Deep Attention Selective Network (dasNet) architecture. DasNet's feedback structure can dynamically alter its convolutional filter sensitivities during classification. It harnesses the power of sequential processing to improve classification performance, by allowing the network to iteratively focus its internal attention on some of its convolutional filters. Feedback is trained through direct policy search in a huge million-dimensional parameter space, through scalable natural evolution strategies (SNES). On the CIFAR-10 and CIFAR-100 datasets, dasNet outperforms the previous state-of-the-art model on unaugmented datasets.
This paper adds selective attention to a convolutional neural network and develops a method for training gating parameters for attention in order to optimize the network's classification performance. It addresses an important set of problems in computer vision -- how to adaptively reconfigure complex, deep processing stages when resources are limited and different tasks need to be solved.  The classification results are impressive, but the whole thing is very much a black box -- it suggests useful directions to pursue but does not reveal why the method works.  The analysis of image representation is not very revealing, and there is no sense of what the learned parameters (mean and variance of the theta distribution) are doing.  Nevertheless, the work is an important step beyond the current paradigm of large, feed-forward networks with parameters fixed after learning. The exposition is quite clear, though some of the equations have typos (or need to be explained better).major comments:I would like to see the authors seriously explore the connection to dynamical systems. The work seems closer to recurrent networks than to Reinforcement Learning.  The reward is observed immediately and the policy is deterministic, so training involves a traversal through the space of recurrent "policy" parameters theta to maximize the observed objective function.I did not find the analysis of the network processing the cat image very helpful in understanding how the new features of the model help recognition.   Why are layer 1 activations all increased at the first iteration?  Does this affect the objective function, or is the output invariant to global scalings?  Perhaps focusing on a smaller region of the image and analyzing the evolution of the gating variables in the region could be instructive.  Also, looking at difficult cases -- those misclassified by the standard Maxout network but correctly labeled by the proposed model, or teasing out the effect of the learned model parameters (mean and variance of theta) by constraining them further would be interesting.minor comments:Algorithm 1 description: where is h_M defined? is that the step of collecting observations?  F[i] and \Theta[i] are inside the loop over images (j), are the image fitness values collected in an array or overwritten on each iteration?Eqn 11: what is the sum over?  Is Eqn 8 for a single image, and should be indexed j?Text after eqn 11: what do 'x' and 'd' refer to?I am not clear on why regularization of theta is necessary, since theta is already sampled from the prior and will tend to small values anyway.  In fact, the last step of the optimization (gradient updates on the parameters after having sampled and evaluated the fitness of thetas) is not explained well (how do you compute the gradients) or justified theoretically (are you adjusting hyper-parameters of the posterior over policies).