If your stem contains batchnorm layers, note that their running estimates will be updated during the forward pass. This should be visible during the evaluation step, although these stats should eventually converge, if the input samples are drawn from the same distribution and are sufficiently large.
Also, dropout layers might be applied differently, if e.g. the batch size differs.
Could you explain a bit more, what issues you are seeing, and where the difference is visible in which magnitude?