Fixed it using
class_weights = torch.FloatTensor(weight).cuda()
from this reply;

  
    
    
    What is the weight values mean in torch.nn.CrossEntropyLoss? 
  
  
    What should the weights be to get better accuracy ? In a 5 classes data I used something like that and the results got much worst. Can you give an intuition ? 
weights = [1/1016, 1/12852, 1/12888, 1/3380, 1/296] #[ 1 / number of instances for each class]
class_weights = torch.FloatTensor(weights).cuda()

criterion = torch.nn.CrossEntropyLoss(weight=class_weights)
parameters = filter(lambda p: p.requires_grad, model.parameters())
optimizer = torch.optim.Adam(parameters)