I’m an absolute beginner, trying to understand LSTMs.
I’ve read how if you don’t detach the hidden state of an LSTM some graph used for the propagation of gradients gets really big. Why does this not happen with a linear classifier?
I thought LSTMs are unrolled through time and after that are an acyclic computation graph and can be trained as usual, but apparently that’s somehow wrong, how exactly?
Also in the current https://pytorch.org/tutorials/beginner/nlp/sequence_models_tutorial.html 34 they do not seem to do the detaching, at least there is no call to .detach(). What’s going on here? Are they propagating the gradient through the whole sequence?