I will update my question after this reply. But for the overfitting part, it is not happening to that extent. When I evaluate results on the third set (test set) after the model already is trained, then the resuts are normal. I have printed the confusion matrix and the results look pretty normal. What I think is that somehow I am not doing this part correctly:



 Arturas_Druteika:

model.eval()

with torch.no_grad():
    for batch in valid_loader:
    images = batch[0].to(device)
    labels = batch[1].to(device)

    preds = model(images)
    loss = criterion(preds, labels)
    total_loss_valid += loss.item()



I think somewhere in this part of the code something happens that produces these large loss results, even though my model is not overfitting.