Hi @mrshenli,
Thank you very much for the detailed explanations and they are very helpful.
Perhaps I do not understand it correctly or maybe I did not make it clear but I hope if you could explain more on your last comment in your previous reply.



 mrshenli:


The traffic does relate to the batch size * the size of feature maps at the boundaries?

No, batch size is still irrelevant here. This is because, the local autograd engine will only run a function once in the backward pass when all its input gradients are ready. More specifically, there is an input buffer for every autograd function. If a function x have multiple fanout in the forward pass, those fanout will accumulate gradient into the same input buffer and the local autograd engine will only trigger x when all gradients from those fanout are accumulated in the input buffer. So, the amount of data communicated through an RPC in the backward pass should be the same as the total tensor size went through that RPC in the forward pass.


Take a nn.con2d as an example(nn.Conv2d). If this is the one at the RPC boundary and in the forward pass, the total tensor size to be transmitted to the following node should be equal to Batchsize x Channels xW x H, IIUC. So in the backward path, should it be the same size as this? Or it is accumulated as  Channels xW x H?
Thanks.