title: How to set random seed when it is in distributed training in PyTorch?
body: <p>Now I am training a model using <code>torch.distributed</code>, but I am not sure how to set the random seeds. For example, this is my current code:</p>
  
  <pre class="lang-py prettyprint-override"><code>def main():
      np.random.seed(args.seed)
      torch.manual_seed(args.seed)
      torch.cuda.manual_seed(args.seed)
  
      cudnn.enabled = True
      cudnn.benchmark = True
      cudnn.deterministic = True 
  
      mp.spawn(main_worker, nprocs=args.ngpus, args=(args,))
  </code></pre>
  
  <p>And should I move the</p>
  
  <pre class="lang-py prettyprint-override"><code>    np.random.seed(args.seed)
      torch.manual_seed(args.seed)
      torch.cuda.manual_seed(args.seed)
  
      cudnn.enabled = True
      cudnn.benchmark = True
      cudnn.deterministic = True 
  </code></pre>
  
  <p>into the function <code>main_worker()</code> to make sure every process has the correct seed and cudnn settings? By the way, I have tried this, and <strong>this behavior will make the training 2 times slower</strong>, which really confused me. </p>
  
  <p>Thank you very much for any help!</p>