@ioyy on the onset, the difference between Data Parallel and Distributed Data Parallel is that one uses single process and the other multiple processes. Both implement the gradient synchronization process to assimilate gradients trained in different threads/machines.
Please share your code so that we can look further into it. Please ensure similar seeds in all machines in DDD