Based on your experiments it looks like the batch statistics differ a lot between the single GPU run and the data parallel one.
Using .train() or track_running_stats=False will use the current batch estimates, which might be skewed for smaller batch sizes.
You could try SyncBatchNorm 8 with DistributedDataParallel.