I’m stuck to the older version because of my work group, unfortunately I can’t upgrade to the new version right now.
The loss criterion is out the for loop in my version of the code, I put it there for easy-to-read purposes.
The loss drastical increase and accuracy decrease both happen during exactly one epoch, every time I run the experiment.
My guess is the net just use the pretrained weight during the first epoch (which are ok at their job), giving me good results. Starting from the second epoch the net fresh-starts without any weights, learning from zero.
This should explain why the accuracy drop so much (it’s a segmentation task, all images become almost completely black).