Yes, I’m using one process per GPU.
OMP_NUM_THREADS at 4 doesn’t have much of a difference from 1; leaving it unset has a very slight performance regression.
I am already using synthetic data using the torchvision.datasets.FakeData; each data loader process takes up about 15% CPU with 4 procs, and 60% CPU with one process. The overall scaling jumps to 92% of linear using 1 worker process, but drops to ~65% with num_workers set to zero (so all of the data stuff happens in the main process).
If I get rid of the DataLoader entirely, and just do:
x = torch.rand((batch_size, 3, 224, 224), device='cuda:' + str(gpu_ndx))
y = torch.randint(0, 100, size=(batch_size,), dtype=torch.long, device='cuda:' + str(gpu_ndx))

Inside the training loop, then I see a 6% performance improvement in the single-GPU case, and the two-GPU case jumps to 94% of linear (based off of the improved single GPU perf). The primary causes of slowdown are now basic math methods of torch._C._TensorBase:
0.498, {method 'mul_' of 'torch._C._TensorBase' objects}
0.254, {method 'add_' of 'torch._C._TensorBase' objects}
0.176, {method 'sqrt' of 'torch._C._TensorBase' objects}
0.153, {method 'addcdiv_' of 'torch._C._TensorBase' objects}
0.07, {method 'run_backward' of 'torch._C._EngineBase' objects}
0.064, /home/elis/v/lib/python3.6/site-packages/torch/optim/adam.py:49(step)
0.03, {built-in method conv2d}
0.022, {method 'addcmul_' of 'torch._C._TensorBase' objects}
0.012, {built-in method batch_norm}
0.011, {built-in method zeros_like}

The 94% of linear scaling remains the case even if I move the creation of x and y outside the training loop. Switching from Adam to SGD speeds things up marginally, but doesn’t change the ratio.