Sorry for resurrecting this old thread. The answer above made some confusion with some folks I’ve talked to.
Distributed Data Parallel can very much be advantageous perf wise for single node multi-gpu runs. When run in a 1 gpu / process configuration Distributed Data Parallel can be beneficial as CPU based overheads are now spread across multiple processes.
Perf gains will especially be prominent in networks that have many  small layers/operations. I primarily recommend to folks that they use single gpu / process Distributed Data Parallel over Data Parallel even for single node cases if they want to scale past 2 GPUs.