Hi! I want to implement next in Pytorch.
For example, for sequenced hidden layers, combine hidden layers in some window, calculate loss and then update the weights.
Slide window for one step, find loss and update the weights in that window.
Repeat.
I think that it can be done while backprop, hold the computation graph as it is without freeing it, and free the computation graph when the hidden layer falls outside the window.
It is possible not to free the computation graph while backward by using retain_graph(), but I don’t know how to free computation graph for held arbitrary part.
Can you help me? Or is there any better solution?
By the way, it is not realistic to recalculate the hidden layers every time I update in terms of data size.
Also, can you tell me which class holds computation graph?
And, is it possible to display the computation graph?
Thank you!