[solved] Why we need to detach Variable which contains hidden representation?
<SCODE># Starting each batch, we detach the hidden state from how it was previously produced.
# If we didn't, the model would try backpropagating all the way to start of the dataset.
hidden = repackage_hidden(hidden)
<ECODE> I am not understanding why we need to detach hidden variable from the hidden variable associated with the previous batch of input? When gradients are computed and loss is backpropagated, weight matrices are usually affected by the chain of computation. Why the hidden variables which represents hidden states of a recurrent neural network is at a risk to get affected during backpropagation and should be detached from previous value? When the hidden variable should be detached from previous value? At the beginning of each batch? or beginning of each training epoch?
wasiahmad
This is for doing truncated BPTT. The dependency graph of RNN can be simply viewed as this. <SCODE>c1, h1 -> c2, h2 -> c3, h3 -> ... -> cn, hn
    |        |         |                |
  loss1    loss2     loss3            lossn
<ECODE> If we did not truncate the history of hidden states (c, h), the back-propagated gradients would flow from the loss towards the beginning, which may result in gradient vanishing or exploding. Detailed explanation can be found here.
chenzhekl
Yes.
elanmart
Hello, thanks for the nice answer, it explains a lot! But I still have a question: , where input is tensor of sequences (or a packed sequence), and the initial hidden states h_0, c_0 are initialized automatically. In the example above, the hidden states are not detached manually. I am wondering if the attachment is done automatically here, or BPTT is computed based on all previous hidden states without doing any detachments? Thanks.
RangoHU
In your examples new zero-initialized hidden states are initialized at every call to the lstm, and therefore are not connected to the previous sequence in any way. The language modeling example needs to detach them, beacuse it retains the values of hidden states between training states (but doesn’t want to backprop very far back).
apaszke
Understood. Many thanks.
RangoHU
Hello, I come up with a specific question about the detachment op we discuss here. For example, we have a seq2seq model with an attention layer between the encoder and the decoder. According to the common implementations of attention models, the last hidden state of the encoder (say, ‘hn’) is used as the first hidden state of the decoder. My question is: is it necessary to detach hn from the encoder network?
RangoHU
It depends what you want to do. If you detach it, the encoder won’t get any gradients from that backward.
apaszke
Then I suppose detaching the hidden variable from the graph during the evaluation stage is not necessary, or is it? <SCODE>def evaluate(data_source):
    # Turn on evaluation mode which disables dropout.
    model.eval()
    total_loss = 0
    ntokens = len(corpus.dictionary)
    hidden = model.init_hidden(eval_batch_size)
    for i in range(0, data_source.size(0) - 1, args.bptt):
        data, targets = get_batch(data_source, i, evaluation=True)
        output, hidden = model(data, hidden)
        output_flat = output.view(-1, ntokens)
        total_loss += len(data) * criterion(output_flat, targets).data
        hidden = repackage_hidden(hidden)
    return total_loss[0] / len(data_source)
<ECODE>
ctyuang
During evaluation detaching is not necessary. When you evaluate there is no need to compute the gradients nor backpropagate anything. So, afaik just put your input variable as volatile and Pytorch won’t hesitate to create the backpropagation graph, it will just do a forward pass.
adrianjav
hi Adam, how do we detach the states from the output of GRU layer: <SCODE>def detach_states(self, states):
	if states is None:
		return states
	return [state.detach() for state in states]
<ECODE>  are output from the GRU layer. When I pass them to It gives me following error: … What I am doing wrong. I can put the entire code if this is not clear.
pp18
In fact, if you call at the beginning of each batch during training, then there will be no need to call .
entslscheia
tastyminerals
backpropagating through all seen data (i.e. in PyTorch just don’t detach the hidden state) use only the last input batch?
ptrblck
andrei-rusu
Messiah