Thank you for your detailed analysis.

In general, when comparing DDP and DP speed, we need to make sure that they run the same model.

I have converted BatchNorm into SyncBatchNorm in DP too, as you can find “convert_model()” in the above code list of DP.
As you pointed out, removing convert_model() from the DP program significantly improves the performance. (2500msec. / itr  ->  1500), However, I could not see such a difference of latency for the DDP program. The improvement observed in my experiment is less than 4%, with a batch size of 4096 and 8 GPUs. I used the same time measurement method as you did.
I can tolerate the 4% difference if I can make my DDP program faster than the DP.
DP with convert_model()                     --------------------------  DP without convert_model()

DDP with convert_sync_batchnorm()        --------- DDP without convert_sync_batchnorm()

I use convert_model(), which converts BatchNorm into a SyncBatchNorm for DP, but it is different from the torch version for DDP. The torch.nn.SyncBatchNorm.convert_sync_batchnorm() supports only DDP.
By the way, I wonder why the latency in your experiment is one digit lower than mine. Are you using the same model (resnet34) and CIfar10?
If there is an example program for image classification using DDP, I will be curious to see the latency. I am trying to test my original optimizer with a practical settings. So far, the test comparing it with Adam has been successful in terms of the number of steps to reach a target accuracy. Now I have to improve the wall clock time and trying to find a way to scale the speed with the batch size, like the experiment you have shown. However, DDP is still not working in my program  for that purpose now.