I see, that’s weird. So, we need to properly debug this to see where this comes from. We need to see the outputs of the two models in different settings, but using the same input. Can you do the following steps:

Create a single batch of input, any batch-size is fine.
Print out the outputs of the two models when loaded individually (one at a time), but on the same batch of data that was created in step 1.
Print out the outputs of the two model when they are loaded in a loop, using the exact same batch of data that was created step 1.

Note that we want to look at the output of the model, not the loss or the accuracy. For example, if the model returns probabilities of different classes, we want to see these probabilities.