I think calling all_reduce is enough to sync loss manually. For this question, I encourage you to look at document and try cooking up a toy example yourself and see how it works in practice to help understand.
Gradient is averaged over 32 * 8 samples in 8-GPUs case. In single GPU case, your gradient is computed from 32 samples. I don’t think there is any theory saying you have to use 8x learning rate. But intuitively, the averaged gradient over a bigger batch has a smaller variance (think gradient from a sample as a random variable). This paper 2 also gives some theoretic intuition and practical advice.
Regarding the code you referred to, it uses DistributedSampler (check createdataset function in the utils.dataset.py ) to divide dataset into several subsets so that each GPU only sees one subset of the whole dataset.
PS: I think it would be better to start from DDP tutorial and try to write the toy example yourself so that you can understand every component of the DDP through coding yourself.