Solved by cbalioglu in post #2 
              
              
                Regarding your first question. That is correct. DDP should give you the same result as if your training was run on a single process with a single GPU. As you described, DDP by default averages your gradients across all nodes, meaning except any deviations due to floating point arithmetic, mathematic…