Hi! Gloo backend is much slower than NCCL if can do trainings on GPU. You may want to switch to NCCL if possible.
A few relatively easy options on my mind:

It will be much more efficient if you use DDP 1 instead of implementing your own communication by allreduce primitive.
Try no_sync context manager on the tutorial, which can reduce the sync frequency by accumulating gradients locally.
If you can use NCCL, it will be much faster. You can also try DDP communication hooks 1 for gradient compression.