Solved by FreddyJ in post #3 
              
              
                Found the issue. The data was not the same after all (the pipeline was missing the normalization step, and I didn’t notice). 
Let this be a lesson to anyone getting weird logits out of your network: Print the values, don’t plot the image. :v)