I see, so you are using one GPU but run two training in parallel. We couldn’t run all_reduce collective manually since there’s only one GPU. If we are on CPU we can leverage tensor.share_memory_(), but since we are on GPU, this is not an option. Yeah so the only option I could think of is the first option you mentioned, you can periodically move the two model back to CPU, sync params/grads, then move them back and resume training.