I have in the meantime narrowed down the problem somewhat. Data input is the same, the accuracy (or more precisely the weird probabilities output) is not tied to the batch size. For some reason, the raw logits are just several magnitudes larger (-15 vs -10000 for most cases) when I load the network somewhere else for inference.