If you use vanilla multiprocessing you’ll have to do this yourself. If you use it in combination with torch.nn.parallel.DistributedDataParallel 14 then gradient synchronization and averaging is done for you. Also see the documentation on torch.distributed 15 for more information.