Solved by nnss in post #10 
              
              
                Oh, I think that would really be an interesting way to approach the problem, I’ll give it a try for sure and update you. 
I think, as you say, the models are indeed equivalent. I did some further inspection and realised that I wasn’t using .zero_grad() on the classification layer separately when tra…