Hi,
When I use two gpus to train my model, in the backward computation, are the gradients of the two gpus get added or averaged to for us to obtain the gradient for updating the model parameters?
As for the moving average parameters of batchnorm layers, if I do not use sync-bn, how is the moving average status of the final model computed?