Hello,
I’m implementing a recursive network that is going to be trained with very long sequences. I had memory problems when training because of that excessive length and I decided to use a truncated-BPTT algorithm to train it as described here 475, that is,
every k1 steps backpropagate taking k2 back steps
checking some examples 355  I could easily write the case when k1 = k2. However, I haven’t been able to implement the general case yet.
My first idea was to freeze the  gradient graph after the first k2 steps and keep changing the Variables referenced there. Later, I saw that the graph doesn’t directly reference earlier Variables, instead it directly incorporates the gradient graphs from that Variables. So I thought about looking for that copied subgraphs and substituting them by a Variable reference, but I found out that the gradient graph is unwritable and immutable.
The only idea I have left is to recompute the calculations in the overlaps between backpropagations. That’s going to work, but I’d really love to avoid recomputing things.
Any idea on how to implement this efficiently?