You can use a flag keyword argument inside forward, noticing that the two functions don’t differ by much. I tried to switch member functions using a flag and the following segment worked for me:

  
      github.com
  
  
    jerinphilip/MaskGAN.pytorch/blob/cdb2a7aa87826464f79273976286e90fbd3845dc/mgan/modules/distributed_model.py#L58-L70 7

def forward(self, *args, **kwargs):
    if 'ppl' not in kwargs:
        kwargs['ppl'] = False


    if kwargs['tag'] == 'g-step':
        if self.pretrain:
            return self._gstep_pretrain(*args, ppl_compute=kwargs['ppl'])
        else:
            return self._gstep(*args, ppl_compute=kwargs['ppl'])
    elif kwargs['tag'] == 'c-step':
        return self._cstep(*args)


    return self._dstep(*args, real=kwargs['real'])



  
  
    
    
  
  


In my case I’m switching whether to use generator, discriminator or critic in an GAN-Actor Critic setup. I’m using tag here to control which sub-model’s forward is being called.
You can see scatter's source code below to understand how args and kwargs are replicated along workers, in case there’s any confusion, which at the time I had:

  
      github.com
  
  
    pytorch/pytorch/blob/4e3c97a0be5c1bba04928de6abbdad31169e62ee/torch/nn/parallel/scatter_gather.py#L5-L31 9

def scatter(inputs, target_gpus, dim=0):
    r"""
    Slices tensors into approximately equal chunks and
    distributes them across given GPUs. Duplicates
    references to objects that are not tensors.
    """
    def scatter_map(obj):
        if isinstance(obj, torch.Tensor):
            return Scatter.apply(target_gpus, None, dim, obj)
        if isinstance(obj, tuple) and len(obj) > 0:
            return list(zip(*map(scatter_map, obj)))
        if isinstance(obj, list) and len(obj) > 0:
            return list(map(list, zip(*map(scatter_map, obj))))
        if isinstance(obj, dict) and len(obj) > 0:
            return list(map(type(obj), zip(*map(scatter_map, obj.items()))))
        return [obj for targets in target_gpus]


    # After scatter_map is called, a scatter_map cell will exist. This cell
    # has a reference to the actual function scatter_map, which has references
    # to a closure that has a reference to the scatter_map cell (because the


  This file has been truncated. show original