Hi,
Each gpu compute the gradients for it’s part of the batch. Then they are accumulated on the “main” model where the weight update is done. Then this “main” model shares it’s weight to all the other gpus so that all models have the same weight.