Solved by minsikseo in post #3 
              
              
                Thx!  
In my case, however, I solved using torch.multiprocessing.Pipe as below: 
def main_worker(rank, world_size, conn):
    ...
    conn.send(loss)

if __name__ == "__main__":
    parent_conn, child_conn = mp.Pipe()
    mp.spawn(main_worker, args=(ngpus, child_conn,), nprocs=ngpus, join=True)
    …