Hm, can’t see anything obvious what might cause the performance issues. Just some generic comments:


Since you initialize h0 and c0 you don’t need to detach them as well. Maybe that’s even bad for the training itself, but here I’m not sure.


I cannot see the point of the lines below. Again, I don’t think they hurt, but in the end you return the last hidden state after being pushed through a linear layer. lstm_out is never used
  lstm_out.size()
  lstm_out[:, -1, :]
  lstm_out = self.linear(lstm_out[:, -1, :])
  lstm_out.size()



I have a implementation of a multiclass LSTM/GRU classifier 14. Maybe you can have a look to see what might be different. The code is a bit verbose to make the classifier configurable.