Where do I obtain details corresponding to this particular information?

We need to go through some internal approval process to publicly disclose that paper. It will take some time. For now  https://pytorch.org/docs/master/notes/ddp.html is the best place for overall intro. The implementation of DDP is linked below:

  
      github.com
  
  
    pytorch/pytorch/blob/ebd869153c6adb37507d2ecb6a9fe3fd495fbb6e/torch/nn/parallel/distributed.py 1
from contextlib import contextmanager
import copy
import itertools

import torch

import torch.cuda.comm
import torch.distributed as dist

if dist.is_available():
    from torch.distributed.distributed_c10d import _get_default_group

from ..modules import Module
from .replicate import replicate
from .scatter_gather import scatter_kwargs, gather
from .parallel_apply import parallel_apply
from torch.cuda._utils import _get_device_index


def _find_tensors(obj):


  This file has been truncated. show original

  
  
    
    
  
  



  
      github.com
  
  
    pytorch/pytorch/blob/ebd869153c6adb37507d2ecb6a9fe3fd495fbb6e/torch/csrc/distributed/c10d/reducer.cpp 1
#include <torch/csrc/distributed/c10d/reducer.h>

#include <functional>

#include <c10/core/DeviceGuard.h>
#include <c10/util/Exception.h>
#include <torch/csrc/autograd/engine.h>
#include <torch/csrc/autograd/function_hook.h>
#include <torch/csrc/autograd/functions/accumulate_grad.h>
#include <torch/csrc/autograd/profiler.h>
#include <torch/csrc/autograd/utils/lambda_post_hook.h>
#include <torch/csrc/distributed/c10d/comm.h>
#include <torch/csrc/utils/hash.h>
#include <torch/csrc/utils/memory.h>

namespace c10d {
namespace {

inline int64_t current_time_in_nanos() {
  return torch::autograd::profiler::getTime();


  This file has been truncated. show original

  
  
    
    
  
  



Isn’t only  .grad  meant to be communicated and the workers applying the updates individually?

No. Currently at construction time, DDP creates a mapping from parameters to buckets, and always communicate all buckets even if some gradients are not used in one iteration. The reason for doing so is that it is possible process 1 only computes grad A and process 2 only computes grad B. However, AllReduce operation requires all processes to provide the same set of input tensors. So in this case, both process 1 and 2 need to communicate grad A and B. DDP can use another communication to first figure out which grads are used globally. However, if block waiting for this signal, there will be no overlap between communication and computation, which could result in >30% slowdown in some cases.

If my parameters of  theta[t]  has only gradients for the particular task, would this help the case?

It helps to skip computation but not communication. DDP always communicates all parameters in the model you passed to DDP constructor.

I’m reading the  Forward Pass  section of Internal Design , with  find_unused_parameters , it is possible to operate on a subgraph, correct(?)

That flag only allows DDP to skip waiting for grads of those parameters. The communication phase is the same regardless the value of find_unused_parameters.