The hidden state in an LSTM is suppose to serve as “memory”. We start off with an initial hidden state, but this hidden state isn’t suppose to be learned, so we detach it to let the model use those values but to not compute gradients w.r.t.
In the example you linked, it looks like they construct the initial hidden state each time. So there’s no concerns about those parameters being adjusted.
Could you elaborate on what you mean this doesn’t occur with a linear classifier? It could be that there’s a double usage of the word “hidden” which can mean the hidden state of a neural network (so the layers in between) or the hidden state being the memory of an LSTM.