I think it’s related to code block inside torch.no_grad() even though stack trace said error is hit in the forward pass. The environment I’m using is ‘1.8.1+cu101’.
Following-up information on previous code by Howard:
If I change the save_img line to be vutil.save_image(imgs, '%e.png' % e, normalize=True), the error changes to
running on rank 0
running on rank 1
Epoch D loss:0.07447701692581177;
Traceback (most recent call last):
  File "debug2.py", line 178, in <module>
    mp.spawn(subprocess_fn, args=(args, ), nprocs=args.num_gpus)
  File "/home/azav/.local/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 230, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
  File "/home/azav/.local/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 188, in start_processes
    while not context.join():
  File "/home/azav/.local/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 150, in join
    raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException: 

-- Process 1 terminated with the following error:
Traceback (most recent call last):
  File "/home/azav/.local/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 59, in _wrap
    fn(i, *args)
  File "/home/azav/Documents/LargeGANs/debug2.py", line 147, in subprocess_fn
    real_pred = discriminator(real_data)
  File "/home/azav/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/azav/.local/lib/python3.6/site-packages/torch/nn/parallel/distributed.py", line 692, in forward
    if self.reducer._rebuild_buckets():
RuntimeError: Out of range variable index specified.