The are different factors than can produce this behavior. First at all, Are you seeding all frameworks that you are using to generate random numbers? Are you setting cudnn to determinisic mode? You can do it using this piece of code:

torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False

Moreover you need to take into account some gpu operations are non-deterministic This link 4 can be useful for you. (I am supposing you are using GPU.)
Answering your last point. As far as I know batchnorm is not updated in evaluation time. You can find more information about batchnorm here 2 .