Hi,
Just to make sure I have understood correctly. If I train on one gpu with batchsize=16 and lr=0.01, what would be the correct lr if I train on two gpus in torch.distributed mode with the same batchsize=16(8 on each gpu)?