Solved by zetyquickly in post #4 
              
              
                I see. So, I think that was a correct use of detach. 
May you try to reduce the number of linear layers at the end and see is there any difference? 
out, _ = self.lstm(x, (h0.detach(), c0.detach())) 
out = self.act(out[:, -1, :])
out = self.act(self.dense1(out))
out = F.softmax(self.act(self.dense2(…