I found the problem. It turns out that if you have batch normalization layers, you need to keep track of the running mean and the running variance. These values don’t show up in the model parameter list (they are not parameters) but are important at test time.
They are present in the model state dict, though, which is how I found out they were different for the two models.