Realize multi gpu training differs a lot from single-gpu training. Batch norm is not shared across gpus. Gradients resulting from each gpu are averaged. There are differences and some people report the same as you.
Besides, some operations based on nvidia library aren’t deterministic even if you set the seed, for instance, interpolation. There is some flag like torch.cuda.benchmark which is recommended to be set to false.  Lastly, realize each machine may process floating points with different hardware, thus, getting different results.
import torch
a=torch.rand(1)*10
import numpy as np
b=a.cuda(1)**np.pi
c=a**np.pi
print('Cuda {0:10f} CPU {1:10f}'.format(b.item(),c.item()))
Cuda 731.980347 CPU 731.980286