Sparse Attentive Backtracking: Temporal Credit Assignment Through Reminding
Learning long-term dependencies in extended temporal sequences requires credit assignment to events far back in the past. The most common method for training recurrent neural networks, back-propagation through time (BPTT), requires credit information to be propagated backwards through every single step of the forward computation, potentially over thousands or millions of time steps. This becomes computationally expensive or even infeasible when used with long sequences. Importantly, biological brains are unlikely to perform such detailed reverse replay over very long sequences of internal states (consider days, months, or years.) However, humans are often reminded of past memories or mental states which are associated with the current mental state. We consider the hypothesis that such memory associations between past and present could be used for credit assignment through arbitrarily long sequences, propagating the credit assigned to the current state to the associated past state. Based on this principle, we study a novel algorithm which only back-propagates through a few of these temporal skip connections, realized by a learned attention mechanism that associates current states with relevant past states. We demonstrate in experiments that our method matches or outperforms regular BPTT and truncated BPTT in tasks involving particularly longterm dependencies, but without requiring the biologically implausible backward replay through the whole history of states. Additionally, we demonstrate that the proposed method transfers to longer sequences significantly better than LSTMs trained with BPTT and LSTMs trained with full self-attention.
The authors augment an RNN with skip connections in time, that are sparsely gated by learnable attention. This allows to reap the benefits of full BPTT while effectively using only truncated BPTT.  While other forms of attention-gated skip connections in time have been suggested before, to which the authors compare, here the authors looked at sparse (still differentiable) retrieval where only a few top memory entries are selected, enabling the benefits of backpropagating over only a few selected earlier states.  Overall, I think this work is very significant, both for enabling faster implementations of BPTT when considering long time horizons, but also for suggesting future directions for how the brain might perform credit assignment and for pointing out further brain strategies / biases to employ in machine learning. With some clarifications / changes as below, I recommend the acceptance of this article for NIPS.  1. In lines 58-60, the authors say that BPTT would require "playing back these events". However, it is not so simple. The error must be backpropagated using the same recurrent weights, and combined with the activities and derivatives at previous time points to modify the weights. The authors should highlight this, else BPTT in the brain seems a simple matter of replaying past memories which is still bio-plausible, even though requires long time scales, whereas BPTT is far less bio-plausible, from perspectives of architecture and electro-physiology.  2. In lines 176 to 181, the authors show how a hard attention mechanism is implemented retrieving only the k_{top} memories. The authors should clarify how this memory access is differentiable, as the usage of soft-attention by other memory-augmented implementations was partly to enable differentiable access allowing backprop to pass through, and would allow the learning of what to attend to in the past given current hidden state \hat{h}. It is not clear here how the authors maintain differentiability while keeping hard attention.  Minor: l 156: "computationsl" l 223" "pixel-by-pixle"