Strategic Attentive Writer for Learning Macro-Actions
We present a novel deep recurrent neural network architecture that learns to build implicit plans in an end-to-end manner purely by interacting with an environment in reinforcement learning setting. The network builds an internal plan, which is continuously updated upon observation of the next input from the environment. It can also partition this internal representation into contiguous sub-sequences by learning for how long the plan can be committed to -i.e. followed without replaning. Combining these properties, the proposed model, dubbed STRategic Attentive Writer (STRAW) can learn high-level, temporally abstracted macro-actions of varying lengths that are solely learnt from data without any prior information. These macro-actions enable both structured exploration and economic computation. We experimentally demonstrate that STRAW delivers strong improvements on several ATARI games by employing temporally extended planning strategies (e.g. Ms. Pacman and Frostbite). It is at the same time a general algorithm that can be applied on any sequence data. To that end, we also show that when trained on text prediction task, STRAW naturally predicts frequent n-grams (instead of macro-actions), demonstrating the generality of the approach.
This paper proposes a way for an agent to learn so-called "macro-actions" (where one macro-action is a sequence of low-level actions), to help long-term planning in reinforcement learning tasks. The core idea consists in predicting the sequence of actions to perform in the next N steps. The agent follows this sequence until a new prediction is made, which occurs at "re-planning" steps also decided by the model. More precisely, a re-planning step triggered at time t consists in predicting for each future time step t+1, ..., t+N both a distribution probability over actions and a probability of triggering another re-planning (using an attention mechanism to actually update predictions only for a small number of time steps). The main experiments are on reinforcement learning tasks (2D maze and Atari Learning Environment (ALE)), showing the benefits of learning such macro-actions. Importantly, better results are obtained by adding noise to the state representation, making agent predictions more robust.
The paper is clearly written (up to a few notation issues detailed below), with proper references to related work. It addresses a very relevant problem in reinforcement learning: automatically learning high-level abstractions of actions to allow better long-term planning. To the best of my knowledge the approach taken here is original, and the ideas presented are sound, making use of several "hot" methods in recent machine learning applications (Asynchronous Advantage Actor-Critic, attention mechanisms, LSTMs, Convolutional Neural Networks...). Experimental results are promising, indeed demonstrating that the proposed algorithm helps the agent perform better in most tasks.  Overall I believe this is very interesting research that might eventually lead to significant advances in the field. It does not however, in my opinion, reach that point in the current state of this submission. Although there are many relevant experimental results shown here (and no room left for more due to the 8-page limit), I still feel after reading the paper that it is not entirely clear why and how it works. Here are some examples of questions that are not answered in the current state of the paper: - How useful is the attention mechanism? It looks like the agent is systematically re-planning only the next few steps anyway (see e.g. Fig. 5 and supplementary video) - In the maze experiments, is the improvement over LSTM brought by the long-term planning part, the noisy state representation, or both? - Is the "action repeat of 4" in ALE still useful for such a model, that is supposed to be able to learn by itself if and when it is good to commit to multiple consecutive actions? (I know it is standard practice in ALE but it would be interesting to see if your algorithm can do better) - Extra computational cost aside, would the agent perform better or worse if it re-planned at each time step at evaluation time? (but not during training as in 5.5) - What would Fig. 6 look like over 500 epochs instead of 100? (there is no justification for looking only at the first 100 epochs) What would it look like across more games? - What are typical macro-actions learned in ALE?  Other small remarks / questions: - Are actions also sampled during evaluation, or is the highest scoring action used? - Similar question for the noise added in STRAWe (is it also added during evaluation?) - "a matrix D of the same size as At, which contains the patch p scaled and positioned according to ψ": with the rest set to 0? - Please clarify earlier (in paragraph of l. 145) how the gradient flows through g_t (instead of waiting until the last sentence of section 4): it is confusing in Alg. 1 because it looks like a useless multiplication by 1 and the link to c_1^t-1 is unclear at that point. - The formatting of eq. 2 makes it hard to read - Eq. 3 is confusing: should it be "t=1" below the sum? Could you please define 1_gt? Should it be zeta instead of phi? What is c_t[t]? (by the way it should be c^t, not c_t) - "The most natural solution for our architecture would be to create value-plan containing the estimates": the wording "would be" makes it sound like you are doing something different, but as far as I can tell it is what is being done here. Just use "... is to create a..."? - l. 254: "Notice how corners and areas close to junctions are highlighted" => not all corners are, maybe add "some"? - l. 263: "every 2 million training steps" => better say "every two epochs" since Fig. 3b is plotted in epochs - What are the blue / black horizontal lines in Fig. 5? - Why does replanning occur in Fig. 5 even though it looks like the next two re-planning bits (last row in the figure) are off in all frames?  Typos and notation issues: - l. 7: "re-planing" - l. 43: "it is committed its action plan" - l. 57: "in genral" - l. 60: "enviroment" - C^t and d^t are used in some places instead of A^t and c^t: in Fig. 1, Fig. 2, l. 143 and Fig. 5 - l. 111: "it used to update" - l. 117: "A grid of K × A of" - l. 118: "co-ordinates" - l. 131: "h is two-layer" - l. 138 the last c_t should be c^t, and the last psi^c should be psi_t^c - l. 219 "STRAW(e)" should be STRAWe - l. 236: "it's" - l. 258: "uses a larger 15 x 15 random mazes" - Fig. 4: "From top to bottom" should be "From left to right" - Fig. 5: "drasticly" and "resotres" - l. 300: "and is beneficial to re-plan often"  Update after author feedback: Thanks for the response. I hope some of the questions I asked above can be answered in future work. Regarding the attention mechanism, thanks for pointing out the differences in planning horizon in the video. I still believe though it would be interesting to better understand how it contributes to the overall agent behavior: do you really need 3 attention parameters (position / stride / width) or would a single planning length be enough? (maybe even fixed for a given game?) Also do the read & write attention parameters need to be the same? Thank you also for mentioning the results when re-planning at each step. It would be interesting to add a remark about it in a final version (but please clarify why internal states differ, as I am not sure why this is the case).