From Pytorch documentation, I know that when one use DistributedDataParallel, gradients on multiple GPUs will be averaged(all-reduced), and each model’s parameters will be updated based on the grapdients. So there is no need to sync model’s parameters. But we know float tensor addition might incur presicion error. With the training going on (such as millions of minibatches), will the presicion error affect the consistency of model’s parameters on miltiple GPUs? Thank you!