Thanks for your answer,it helped me a lot. 
One conclusion I got from these materials is that I should set torch.utils.data.DataLoader(batch_size=args.batch_size/world_size）
lr still be 1xlr.
Is this correct?