Solved by quanguet in post #2 
              
              
                Could you give detailed implementation of your model. One possibility is the mismatch between the ground truth labels and the predictions. 
But there is a high chance that you missed a linear layer to convert hidden output of LSTM to class prediction. 
Try the following: 

class Model(nn.Module):

 …