For Question 1, I found that the pytorch code aggregating output of each (copied) module is using lock.

  
      github.com
  
  
    pytorch/pytorch/blob/master/torch/nn/parallel/parallel_apply.py#L60 10

def _worker(i, module, input, kwargs, device=None):
    torch.set_grad_enabled(grad_enabled)
    if device is None:
        device = get_a_var(input).get_device()
    try:
        with torch.cuda.device(device):
            # this also avoids accidental slicing of `input` if it is a Tensor
            if not isinstance(input, (list, tuple)):
                input = (input,)
            output = module(*input, **kwargs)
        with lock:
            results[i] = output
    except Exception as e:
        with lock:
            results[i] = e


if len(modules) > 1:
    threads = [threading.Thread(target=_worker,
                                args=(i, module, input, kwargs, device))
               for i, (module, input, kwargs, device) in
               enumerate(zip(modules, inputs, kwargs_tup, devices))]



  
  
    
    
  
  


So the answer would be I have to use threading.lock.