HuYang719:

For the tutorial for the ResNet, seems ResNet has more activations for transmission:


Thanks for all the detailed comments! Just want to confirm my understanding:
Are you saying that, even in the tutorial for the ResNet instead of your own code, you can still observe the memory leak on devices cuda:0 and cuda:1, just not on the master process.
So far, such memory leak is irrelevant to 1) RPC backend (TensorPipe or ProcessGroup), 2) worker’s device (GPU or CPU), or 3) how we launch distributed training (mp.spawn or just run_worker, even within a single host).
The only factor that is relevant to the memory leak is num_worker_threads:

Set the num_worker_threads=2, memory does not increase
Set the num_worker_threads=16, memory first increases to a certain level and stops growing
Set the num_worker_threads=128, memory keeps growing after every forward pass

@mrshenli