A major drawback of backpropagation through time (BPTT) is the difficulty of learning long-term dependencies, coming from having to propagate credit information backwards through every single step of the forward computation. This makes BPTT both computationally impractical and biologically implausible. For this reason, full backpropagation through time is rarely used on long sequences, and truncated backpropagation through time is used as a heuristic. However, this usually leads to biased estimates of the gradient in which longer term dependencies are ignored. Addressing this issue, we propose an alternative algorithm, Sparse Attentive Backtracking, which might also be related to principles used by brains to learn long-term dependencies. Sparse Attentive Backtracking learns an attention mechanism over the hidden states of the past and selectively backpropagates through paths with high attention weights. This allows the model to learn long term dependencies while only backtracking for a small number of time steps, not just from the recent past but also from attended relevant past states.
This work proposes Sparse Attentive Backtracking, an attention-based approach to incorporating long-range dependencies into RNNs. Through time, a “macrostate” of previous hidden states is accumulated. An attention mechanism is used to select the states within the macro-state most relevant to the current timestep. A weighted combination of these previous states is then added to the hidden state as computed in the ordinary way. This construction allows gradients to flow backwards quickly across longer time scales via the macrostate. The proposed architecture is compared against LSTMs trained with both BPTT and truncated BPTT.
Pros:
- Novel combination of recurrent skip connections with attention.
- The paper is overall written clearly and structured well.
  
Cons:
- The proposed algorithm is compared against TBPTT but it is unclear the extent to which it is solving the same computational issues TBPTT is designed to solve.
- Design decisions, particularly regarding the attention computation, are not fully explained.
SAB, like TBPTT, allows for more frequent updates to the parameters. However, unlike TBPTT, activations for previous timesteps (even those far in the past) need to be maintained since gradients could flow backwards to them via the macrostate. Thus SAB seems to have higher memory requirements than TBPTT. The empirical results demonstrate that SAB performs slightly better than TBPTT for most tasks in terms of accuracy/CE, but there is no mention of comparing the memory requirements of each. Results demonstrating also whether SAB trains more quickly than the LSTM baselines would be helpful.
The proposed affine form of attention does not appear to actually represent the salience of a microstate and a given time. The second term of the RHS of equation 1 (w_2^T \hat{h}^{(t)}) is canceled out in the subtraction in equation 2, since this term is constant for all i. Thus the attention weights for a given microstate are constant throughout time, which seems undesirable.
The related work discusses skip connections in the context of convolutional nets, but doesn’t mention previous works incorporating skip connections into RNN architectures, such as [1], [2], or [3].
Overall, the combination of recurrent skip connections and attention appears to be novel, but experimental comparisons to other skip connection RNN architectures are missing and thus it is not clear how this work is positioned relative to previous related work. 
[1] Lin, Tsungnan, et al. "Learning long-term dependencies in NARX recurrent neural networks." IEEE Transactions on Neural Networks 7.6 (1996): 1329-1338.
[2] Koutnik, Jan, et al. "A clockwork rnn." International Conference on Machine Learning. 2014.
[3] Chang, Shiyu, et al. "Dilated recurrent neural networks." Advances in Neural Information Processing Systems. 2017.
EDIT: I have read the updated paper and the author's rebuttal. I am satisfied with the update to the attention weight formulation. Overall, I still feel that the proposed SAB approach represents a change to the model structure via skip connections. Therefore SAB should also be compared against other approaches that use skip connections, and not just BPTT / TBPTT, which operate on the standard LSTM. Thus to me the experiments are still lacking. However, I think the approach is quite interesting and as such I am revising my rating from 4 to 5.