That’s odd; I see about 95% using the code you posted.
$ OMP_NUM_THREADS=1 NUM_WORKERS=1 EPOCHS=2 EPOCH_SIZE=3840 BATCH_SIZE=64 NODES=1 GPUS=2 ~/v/bin/python forum_min_ddp.py
2020-04-21 15:18:35.059667 torch.cuda.set_device(1); torch.distributed.init_process_group('nccl', rank=1, world_size=2)
2020-04-21 15:18:35.059667 torch.cuda.set_device(0); torch.distributed.init_process_group('nccl', rank=0, world_size=2)
2020-04-21 15:18:36.918569 Epoch 1, dl: 30
2020-04-21 15:18:36.918801 Epoch 1, dl: 30
2020-04-21 15:18:48.006334 Epoch 2, dl: 30
2020-04-21 15:18:48.007357 Epoch 2, dl: 30
2020-04-21 15:18:59.082558 training loop time: 22.16376233100891 seconds

$ OMP_NUM_THREADS=1 NUM_WORKERS=1 EPOCHS=1 EPOCH_SIZE=3840 BATCH_SIZE=64 NODES=1 GPUS=1 ~/v/bin/python forum_min_ddp.py
2020-04-21 15:19:09.327511 torch.cuda.set_device(0); torch.distributed.init_process_group('nccl', rank=0, world_size=1)
2020-04-21 15:19:11.018629 Epoch 1, dl: 60
2020-04-21 15:19:31.962914 training loop time: 20.944292783737183 seconds

Could you post your output from nvidia-smi -q?