One more piece of update. I compared the weight matrices (only one of the layers but should be enough to test equality) of the model at the end of training and at the start of the inference code, they are exactly equal.