If you don’t detach the hidden states, then each backward will trace all the way to the beginning of that sequence and update parameter gradients along the way. The gradients will be more accurate (correct gradient wrt full sequence) than backward at each time step. In that case, I suggest you directly feed the sequence data to nn.LSTM module as it does the exactly same thing but much faster.
That said, I still am not sure why you want to backward for each single each time step. It may save you some memory, but it takes the same, if not longer, time to do so. And it gives incorrect gradients.
In the context of this question, RNN and LSTM are mostly the same. They are back’ed by cuDNN and supports directly operating on sequence data.