Hi Eli,
TL;DR: I suspect the main reason for the disparity you’re observing is that each epoch, the dataloader processes are shutdown and recreated, and that is not free.
I was able to reproduce your issue with your training script, although I had about 95% scaling from the start on my machine (Ubuntu 18.04.2, 28-core Intel Core i9-9940X CPU @ 3.30GHz, 2x Quadro RTX 5000, PyTorch 1.3.1, CUDA 10.0, NCCL 2.4.8) with the parameters
OMP_NUM_THREADS=4 EPOCHS=2 EPOCH_SIZE=3840 BATCH_SIZE=64 NODES=1 GPUS=2

vs.
OMP_NUM_THREADS=4 EPOCHS=1 EPOCH_SIZE=3840 BATCH_SIZE=64 NODES=1 GPUS=1

Looking at the GPU utilization with nvtop, I noticed a dip in GPU usage between the epochs with GPUS=2. I knew that dataloader processes are destroyed and then recreated from scratch every epoch, so GPUS=1 EPOCHS=1 version would only do it once, while GPUS=2 EPOCHS=2 would have to do it twice. So I decided to remove this unfairness: I made the script to always do just 1 epoch, and instead scale the dataset size like:
ds = torchvision.datasets.FakeData(
    int(os.environ['EPOCH_SIZE']) * int(os.environ['EPOCHS']),

This gave me 99% scaling, in fact even more when I set EPOCH_SIZE=38400 (10x). And, the invocations counts became equal between GPUS=1 and 2 for the top 20 functions from your pstats output. (Except the {method 'acquire' of '_thread.lock' objects} which has 4 extra invocations in GPUS=2 case. That one is coming from pin_memory thread, if you set pin_memory=False, all those acquires go away, but training, as expected, gets slower in both cases, although GPUS=1 case suffers more than GPUS=2).
BTW there is a way to not recreate dataloader processes 10 each epoch and just loop over and over.
Hope this helps and you can replicate!


my pstats for GPUS=1
   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
      600   50.489    0.084   50.489    0.084 {method 'run_backward' of 'torch._C._EngineBase' objects}
   193200   37.203    0.000   37.203    0.000 {method 'mul_' of 'torch._C._TensorBase' objects}
   193200   32.657    0.000   32.657    0.000 {method 'add_' of 'torch._C._TensorBase' objects}
    96600   20.367    0.000   20.367    0.000 {method 'sqrt' of 'torch._C._TensorBase' objects}
    96600   19.994    0.000   19.994    0.000 {method 'addcdiv_' of 'torch._C._TensorBase' objects}
      600   15.513    0.026  140.393    0.234 /opt/conda/lib/python3.6/site-packages/torch/optim/adam.py:49(step)
    96600   14.561    0.000   14.561    0.000 {method 'addcmul_' of 'torch._C._TensorBase' objects}
     1200    3.360    0.003    3.360    0.003 {method 'to' of 'torch._C._TensorBase' objects}
    31800    1.367    0.000    1.367    0.000 {built-in method conv2d}
      600    1.046    0.002    1.046    0.002 {built-in method torch.distributed._broadcast_coalesced}
    31800    0.919    0.000    0.919    0.000 {built-in method batch_norm}
    31800    0.704    0.000    1.962    0.000 /opt/conda/lib/python3.6/site-packages/torch/nn/modules/batchnorm.py:58(forward)
     1840    0.496    0.000    0.496    0.000 {method 'acquire' of '_thread.lock' objects}
    96439    0.359    0.000    0.359    0.000 {method 'zero_' of 'torch._C._TensorBase' objects}
    29400    0.307    0.000    0.307    0.000 {built-in method relu_}
110400/600    0.254    0.000    5.682    0.009 /opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py:531(__call__)
     9600    0.223    0.000    4.182    0.000 /opt/conda/lib/python3.6/site-packages/torchvision/models/resnet.py:95(forward)
        5    0.219    0.044    0.219    0.044 {built-in method _posixsubprocess.fork_exec}
   353400    0.175    0.000    0.175    0.000 /opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py:571(__getattr__)
       53    0.116    0.002    0.116    0.002 {built-in method posix.waitpid}
    31800    0.096    0.000    1.057    0.000 /opt/conda/lib/python3.6/site-packages/torch/nn/functional.py:1643(batch_norm)
      600    0.087    0.000    0.485    0.001 /opt/conda/lib/python3.6/site-packages/torch/optim/optimizer.py:159(zero_grad)




my pstats for GPUS=2
   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
      600   50.391    0.084   50.391    0.084 {method 'run_backward' of 'torch._C._EngineBase' objects}
   193200   37.327    0.000   37.327    0.000 {method 'mul_' of 'torch._C._TensorBase' objects}
   193200   32.815    0.000   32.815    0.000 {method 'add_' of 'torch._C._TensorBase' objects}
    96600   20.583    0.000   20.583    0.000 {method 'sqrt' of 'torch._C._TensorBase' objects}
    96600   20.394    0.000   20.394    0.000 {method 'addcdiv_' of 'torch._C._TensorBase' objects}
      600   15.658    0.026  141.634    0.236 /opt/conda/lib/python3.6/site-packages/torch/optim/adam.py:49(step)
    96600   14.755    0.000   14.755    0.000 {method 'addcmul_' of 'torch._C._TensorBase' objects}
     1200    3.549    0.003    3.549    0.003 {method 'to' of 'torch._C._TensorBase' objects}
    31800    1.394    0.000    1.394    0.000 {built-in method conv2d}
      600    1.104    0.002    1.104    0.002 {built-in method torch.distributed._broadcast_coalesced}
    31800    0.924    0.000    0.924    0.000 {built-in method batch_norm}
    31800    0.713    0.000    1.975    0.000 /opt/conda/lib/python3.6/site-packages/torch/nn/modules/batchnorm.py:58(forward)
     1844    0.550    0.000    0.550    0.000 {method 'acquire' of '_thread.lock' objects}
    96439    0.369    0.000    0.369    0.000 {method 'zero_' of 'torch._C._TensorBase' objects}
    29400    0.316    0.000    0.316    0.000 {built-in method relu_}
110400/600    0.261    0.000    5.870    0.010 /opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py:531(__call__)
     9600    0.242    0.000    4.253    0.000 /opt/conda/lib/python3.6/site-packages/torchvision/models/resnet.py:95(forward)
        5    0.215    0.043    0.215    0.043 {built-in method _posixsubprocess.fork_exec}
   353400    0.174    0.000    0.174    0.000 /opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py:571(__getattr__)
       53    0.152    0.003    0.152    0.003 {built-in method posix.waitpid}
      600    0.118    0.000    0.118    0.000 {built-in method addmm}
    31800    0.098    0.000    1.063    0.000 /opt/conda/lib/python3.6/site-packages/torch/nn/functional.py:1643(batch_norm)
      600    0.090    0.000    0.499    0.001 /opt/conda/lib/python3.6/site-packages/torch/optim/optimizer.py:159(zero_grad)