Found the issue. The data was not the same after all (the pipeline was missing the normalization step, and I didn’t notice).
Let this be a lesson to anyone getting weird logits out of your network: Print the values, don’t plot the image. :v)