Solved by rvarm1 in post #4 
              
              
                bucket.get_index() does not necessarily correspond to layer indices. 
We use GradBuckets to store gradients in DDP’s reducer which is passed to the gradient communication hook. Buckets can contain gradients from 1 or more parameters that correspond to 1 or more layers, and the index is just the inde…