Ok, so it was the batch norm. Now you’re validation curve and training curves are consistent with respect to each other at least.
I’m just not sure to get the difference between the two experiments. So, in the first one you used the flatten trick to apply softmax on the spatial dimensions, and on the second one you applied the softmax across the channels (dim=1), is that correct?