Why the hidden variables which represents hidden states of a recurrent neural network is at a risk to get affected during backpropagation and should be detached from previous value?

As in the diagram below, the error E3 depends on hidden state s3, which depends on s2, which depends on s1 and so on. At timestep t=100, to compute the gradients we will have to consider last 100 states.
We don’t want that: we want to assume that for some k, the hidden state s[t-k] is constant and does not depend on anything.
To achieve this we take a batch of k words and simply assume that the initial hidden state for this batch is constant and doesn’t depend on anything.
But for pytorch, if you call hidden = model(batch, hidden), all of those hidden states are connected. So at the start of each batch you have to manually tell pytorch: “here’s the hidden state from previous batch, but consider it constant”.
I believe you could simply call hidden.detach_() though, no need to create new Variable. (@smth  ?)

rnn-bptt-with-gradients.png953×550 11 KB


When the hidden variable should be detached from previous value? At the beginning of each batch?

Yes.
Image source and a nice blog post about backprop through time: LINK 201
EDIT Oops, I see @chenzhekl’s already answered that when I was writing. Sorry for duplicating.