I think this is normal if these are physically different GPUs since cuDNN implements several different algorithms for convolution etc. You can try to set the derministic flag for cuDNN to see if that helps. I.e., set
torch.backends.cudnn.deterministic = True

in the beginning of your code