Truncated Backpropagation Through Time (truncated BPTT, Jaeger (2005)) is a widespread method for learning recurrent computational graphs. Truncated BPTT keeps the computational benefits of Backpropagation Through Time (BPTT Werbos  (1990)) while relieving the need for a complete backtrack through the whole data sequence at every step. However, truncation favors short-term dependencies: the gradient estimate of truncated BPTT is biased, so that it does not benefit from the convergence guarantees from stochastic gradient theory. We introduce Anticipated Reweighted Truncated Backpropagation (ARTBP), an algorithm that keeps the computational benefits of truncated BPTT, while providing unbiasedness. ARTBP works by using variable truncation lengths together with carefully chosen compensation factors in the backpropagation equation. We check the viability of ARTBP on two tasks. First, a simple synthetic task where careful balancing of temporal dependencies at different scales is needed: truncated BPTT displays unreliable performance, and in worst case scenarios, divergence, while ARTBP converges reliably. Second, on Penn Treebank character-level language modelling Mikolov et al. (2012), ARTBP slightly outperforms truncated BPTT.
Backpropagation Through Time (BPTT) Werbos (1990) is the de facto standard for training recurrent neural networks. However, BPTT has shortcomings when it comes to learning from very long sequences: learning a recurrent network with BPTT requires unfolding the network through time for as many timesteps as there are in the sequence. For long sequences this represents a heavy computational and memory load. This shortcoming is often overcome heuristically, by arbitrarily splitting the initial sequence into subsequences, and only backpropagating on the subsequences. The resulting algorithm is often referred to as Truncated Backpropagation Through Time (truncated BPTT, see for instance Jaeger (2005)). This comes at the cost of losing long term dependencies.
We introduce Anticipated Reweighted Truncated BackPropagation (ARTBP), a variation of truncated BPTT designed to provide an unbiased gradient estimate, accounting for long term dependencies. Like truncated BPTT, ARTBP splits the initial training sequence into subsequences, and only backpropagates on those subsequences. However, unlike truncated BPTT, ARTBP splits the training sequence into variable size subsequences, and suitably modifies the backpropagation equation to obtain unbiased gradients.
Unbiasedness of gradient estimates is the key property that provides convergence to a local optimum in stochastic gradient descent procedures. Stochastic gradient descent with biased estimates, such as the one provided by truncated BPTT, can lead to divergence even in simple situations and even with large truncation lengths (Fig. 3).
ARTBP is experimentally compared to truncated BPTT. On truncated BPTT failure cases, typically when balancing of temporal dependencies is key, ARTBP achieves reliable convergence thanks to unbiasedness. On small-scale but real world data, ARTBP slightly outperforms truncated BPTT on the test case we examined.
ARTBP formalizes the idea that, on a day-to-day basis, we can perform short term optimization, but must reflect on long-term effects once in a while; ARTBP turns this into a provably unbiased overall gradient estimate. Notably, the many short subsequences allow for quick adaptation to the data, while preserving overall balance. 1 RELATED WORK BPTT Werbos (1990) and its truncated counterpart Jaeger (2005) are nearly uncontested in the recurrent learning field. Nevertheless, BPTT is hardly applicable to very long training sequences, as it requires storing and backpropagating through a network with as many layers as there are timesteps Sutskever (2013). Storage issues can be partially addressed as in Gruslys et al. (2016), but at an increased computational cost. Backpropagating through very long sequences also implies performing fewer gradient descent steps, which significantly slows down learning Sutskever (2013).
Truncated BPTT heuristically solves BPTT deficiencies by chopping the initial sequence into evenly sized subsequences. Truncated BPTT truncates gradient flows between contiguous subsequences, but maintains the recurrent hidden state of the network. Truncation biases gradients, removing any theoretical convergence guarantee. Intuitively, truncated BPTT has trouble learning dependencies above the range of truncation.
This is an interesting paper.
It is well known that TBPTT is biased because of a fixed truncation length. The authors propose to make it  unbiased by sampling different truncation lengths and hence changing  the optimization procedure which corresponds to adding noise in the gradient estimates which leads to  unbiased gradients. 
Pros:
- Its a well written and easy to follow paper.
- If I understand correctly, they are changing the optimization procedure so that the proposed approach is able to find a local minima, which was not possible by using truncated backpropagation through time.  
- Its interesting to see in there PTB results that they get better validation score as compared to truncated BPTT.
Cons: 
- Though the approach is interesting, the results are quite preliminary. And given the fact there results are worse than the LSTM baseline (1.40 v/s 1.38). The authors note that it might be because of they are applying without sub-sequence shuffling. 
- I'm not convinced of the approach yet. The authors could do some large scale experiments on datasets like Text8 or speech modelling. 
Few points
- If I'm correct that the proposed approach indeed changes the optimization procedure, than I'd like to know what the authors think about exposure bias issue. Its a well known[1, 2] that we can't sample from RNN's for more number of steps, than what we used for trained (difference b/w teacher forcing and free running RNN). I'd like to know how does there method perform in such a regime (where you sample for more number of steps than you have trained for)
- Another thing, I'd like to see is the results of this model as compared to truncated backpropagation when you increase the sequence length. For example, Lets say you are doing language modelling on PTB, how the result varies when you change the length of the input sequence. I'd like to see a graph where on X axis is the length of the input sequence and on the Y axis is the bpc score (for PTB) and how does it compare to truncated backpropagation through time. 
-  PTB dataset has still not very long term dependencies, so I'm curious what the authors think about using there method for something like speech modelling or some large scale experiments.
- I'd expect the proposed approach to be more computationally expensive as compared to Truncated Back-propagation through time. I dont think the authors mentioned this somewhere in the paper. How much time does a single update takes as compared to Truncated Back-propagation through time ?
- Does the proposed approach help in flow of gradients?  
- In practice for training RNN's people use gradient clipping which also makes the gradient biased. Can the proposed method be used for training longer sequences?  
[1] Scheduled Sampling For Sequence Prediction with RNN's https://arxiv.org/abs/1506.03099
[2] Professor Forcing  https://arxiv.org/abs/1610.09038
Overall, Its an interesting paper which requires some more analysis to be published in this conference. I'd be very happy to increase my score if the authors can provide me results what I have asked for.