Yes, each GPU separately computes its own parameters
They don’t, in inference the parameters that the master gpu accumulated are used
This implementation of synchronized batch norm may be of interest if you indeed want the gpus to accumulate parameters between their respective mini-batches http://hangzh.com/PyTorch-Encoding/syncbn.html 273