Solved by David_Harvey in post #2 
              
              
                The valid batch size is 16*N. 16 is just the batch size in each GPU. During loss backward, DDP makes all-reduce to average the gradients across all GPUs, so the valid batch size is 16*N.