Solved by rvarm1 in post #2 
              
              
                DataParallel works via scatter/gather, so the gathering of scattered gradients is implemented in the backwards function of these operators here: pytorch/_functions.py at master · pytorch/pytorch · GitHub