Hey @nickvu, thanks for sharing the details.  DP is less efficient compared to DDP, as it needs to replicate the model in every iteration. And IIUC, currently there is no plan to improve DP performance. cc @VitalyFedyunin
So if perf is the main concern, DDP might be a better choice.