Thx! 
In my case, however, I solved using torch.multiprocessing.Pipe as below:
def main_worker(rank, world_size, conn):
    ...
    conn.send(loss)

if __name__ == "__main__":
    parent_conn, child_conn = mp.Pipe()
    mp.spawn(main_worker, args=(ngpus, child_conn,), nprocs=ngpus, join=True)
    losses = []
    while parent_conn.poll():
        losses.append(parent_conn.recv())

Then I can gather all losses from every worker.