So what happens when you dont detach the hidden state is that the previous computation graph is attached to it (if retain_graph=True). So when you calculate your next hidden state, the previous computation graph is added to the current computation graph. After a couple of iterations the complete graph is so big that you run out of memory.
Not detaching will allow you to backprop through time (I will use BPTT as an abbreviation).
By instead detaching the hidden state you stop BPTT and the computation graph will not grow with each iteration. It is also possible to detach the hidden state at certain intervals to keep some BPTT.
out, self.hidden = self.lstm(out, self.hidden)
# If hidden is similar to LSTM it should be a tuple of two tensors
self.hidden = tuple(state.detach() for state in self.hidden)

This should run without needing to specify retain_graph=True