Hey Garrett,
Thanks for the reply.

Blockquote That is ddp communication hook. It is used during the backward pass.

I see. I guess there should be a variable that represents a layer like this 1. I’m not sure if bucket_index represents a layer though. Are bucket and layer the same thing in this context? I printed the value of bucket_index and noticed that it only takes two values 0 and 1 (the application was training RN20 on CIFAR10).