I see. So, I think that was a correct use of detach.
May you try to reduce the number of linear layers at the end and see is there any difference?
out, _ = self.lstm(x, (h0.detach(), c0.detach())) 
out = self.act(out[:, -1, :])
out = self.act(self.dense1(out))
out = F.softmax(self.act(self.dense2(out)))

to
out, _ = self.lstm(x, (h0.detach(), c0.detach())) 
# do the proper `.view()` of the `out` tensor 
out = F.softmax(self.dense2(out))

Two points are from the bottom to the top of your code:


softmax is an activation function there’s no specific need to do ReLu first,
There’s no particular need to apply ReLU on LSTM's output because nonlinearities are present within LSTM (see here tanh)