This refers to some of the internals of PyTorch DDP - in each backward pass, DDP must allreduce gradients across all the nodes (4 GPUs in this case) so they are all in sync we reap the benefit of using multiple GPUs. We gather these gradients into buckets (which are 25mb by default), and we initiate the allreduce once the bucket is full. Once during the course of training, these buckets are actually allocated according to when the tensors may receive grads in the backward pass. The log message you saw simply indicates this buck allocation/rebuilding process has taken place, in this case, at the start of training.