Solved by wanchaol in post #4 
              
              
                I see, so you are using one GPU but run two training in parallel. We couldn’t run all_reduce collective manually since there’s only one GPU. If we are on CPU we can leverage tensor.share_memory_(), but since we are on GPU, this is not an option. Yeah so the only option I could think of is the first …