bucket.get_index() does not necessarily correspond to layer indices.
We use GradBuckets to store gradients in DDP’s reducer which is passed to the gradient communication hook. Buckets can contain gradients from 1 or more parameters that correspond to 1 or more layers, and the index is just the index of this bucket in the list of all buckets.
In addition, there is an API bucket.get_per_parameter_tensors() (pytorch/powerSGD_hook.py at master · pytorch/pytorch · GitHub 1) that will allow you to get the tensors for a given parameter.