I fear the .detach() functionality does not offer a clean solution in this case.
At this moment, if I want to backpropagate over a window of 3 timesteps, I need to perform the forward pass three times with different variables for which the histories have been deleted.

pytorch_howto.png1482×1183 67.9 KB

# written out without using loops and disregarding the input to keep it clean.
h3 = w*h0_det_2
h3_det = h3.detach()
h2_det_1 = w*h2_det
h1_det_2 = w*h1_det_1

h4 = w*h1_det_2
h4_det = h4.detach()
h3_det_1 = w*h3_det
h2_det_2 = w*h2_det_1

h5 = w*h2_det_2
h5_det = h5.detach()
h4_det_1 = w*h4_det
h3_det_2 = w*h3_det_1

h6 = w*h3_det_2
h6_det = h6.detach()
h5_det_1 = w*h5_det
h4_det_2 = w*h4_det_1

h7 = w*h4_det_2
h7_det = h7.detach()
h6_det_1 = w*h6_det
h5_det_2 = w*h5_det_1

...

# this possible (dirty) solution does not work
h7.grad_fn.next_functions[1][0].next_functions[1][0].next_functions = ()
h7.backward()

This solution is not very useful as it requires the forward pass of the RNN to be performed as many times as the amount of timesteps I want to backpropagate through.
It seems like there must be an easier solution to this.