I tried to run the code 1 with gloo backend to check if it is a MPI-only problem, but initially hits the following error:
...
    self._target(*self._args, **self._kwargs)
  File "test.py", line 132, in init_processes
    fn(rank, size)
  File "test.py", line 103, in run
    train_set, bsz = partition_dataset()
  ...
ValueError: batch_size should be a positive integer value, but got batch_size=64.0

After fixing that, hits the error below:
  File "test.py", line 132, in init_processes
    fn(rank, size)
  File "test.py", line 118, in run
    epoch_loss += loss.data[0]
IndexError: invalid index of a 0-dim tensor. Use `tensor.item()` in Python or `tensor.item<T>()` in C++ to convert a 0-dim tensor to a number

Can we have a min example that can reproduce the hang issue? Thanks!