Are you using model.eval() before executing the prediction step?
If not, then note that the prediction results could depend on the batch size, which might be different on the single-GPU vs. data parallel setup, e.g. due to batchnorm layers.