Solved by googlebot in post #8 
              
              
                I missed that loss is indeed incorrect. That one is for raw real scores (and two classes), look at F.cross_entropy or F.nll_loss, and don’t use softmax as they do it themselves.