Learning Wake-Sleep Recurrent Attention Models
Despite their success, convolutional neural networks are computationally expensive because they must examine all image locations. Stochastic attention-based models have been shown to improve computational efficiency at test time, but they remain difficult to train because of intractable posterior inference and high variance in the stochastic gradient estimates. Borrowing techniques from the literature on training deep generative models, we present the Wake-Sleep Recurrent Attention Model, a method for training stochastic attention networks which improves posterior inference and which reduces the variability in the stochastic gradients. We show that our method can greatly speed up the training time for stochastic attention networks in the domains of image classification and caption generation.
This paper presents a new learning algorithm for stochastic attention models, employing some recent advances from the literature on training generative models: specifically (1) a simultaneously trained inference network; (2) importance sampling to estimate gradients (a la "reweighted wake sleep"); and (3) tailored control-variates to reduce estimator variance.The use of control variates and (to a lesser degree) a trained inference network are shown to give significant speed-ups to training time.Using this learning algorithm, the model is shown to be capable of achieving comparable performance to a recent variational approach [Mnih et al. ICLR2015] (in it's naive form), but with shorter training times.The authors commendably pack a lot of detail into a small number of pages, and the overall body of work is impressive.However, I was left with some minor questions (see below) about details of the results -- which hopefully the authors can address in their feedback. == Overall the paper has good quality, and it uses a nice combination of sophisticated techniques to achieve good results on interesting problem domains.Equation (10) was also informative/interesting; in the limit, the proposed method gives at least as accurate a bound on the objective as previous variational approaches.One area of confusion that I have is in regards to the performance in comparison to the variational method of Mnih et al, with respect to modifications on top of that model that encourage exploration. It seems that when compared that the variational approach *without* exploration the proposed method shows a significant speed-up. However, with Figure 3 it appears the the method of Mnih et al *with* exploration significantly out-performs (in terms of speed) the proposed approach (and the proposed approach does not seem to benefit from exploration). Is this the case? And if so, why might this be the case, and are there analogous exploration-boosting methods that one could/should consider for the case of the Flickr 8K example in Figure 4? Granted space is limited -- but it would be useful if the authors could address these issues more thoroughly.Also, for the learning curves presented, it might also be informative to show testing/validation error progress in addition (or instead of) training progress.-- In terms of clarity, the paper is reasonably clear and well written. However, there are several parts that could be improved in this regard (in addition to the questions above).One general comment here is attention to detail, and precision with notation. There are several parts of section for in particular where notation usage is a little sloppy. With a little thought from the reader, it's possible to figure out the precise meanings but it would be useful if the authors could make everything explicit. (As an example of this, the expectation distribution in (9) and (10) is not specified, however it is in (13), (15), etc. Another example is the use of set notation {a_n} and an unadorned "a" symbol both represent a sequence of actions.Similarly, it's possible to infer the legends of the figures but this could be made clearer and more explicit in several cases.-- This work has moderate originality. It is a sensible and non-trivial combination of models and techniques that have been proposed recently in the literature. -- Likewise, I think the work has moderate significance. It represents (potentially) algorithmic improvements/speed-ups to an interesting model, using techniques that are currently gaining favour in the community. It remains to be seen whether the methods proposed here have real-world impact (improvements) any significant applications, and whether they help consistently move forward the state of the art. It does, however, represent a useful exploration of the overall model/algorithm design space and in this sense I found it quite valuable.