When the shape of the input to the nnSyncBatchNorm is (1, 128, 32, 32) for each gpu, it is OK to use the SyncBatchNorm if I have more than a sinle GPU.
But, when the input shape is (1, 128, 1, 1), i.e, 1x1 spatial resolution,   each device would still need more than a single value to calculate the local estimates， as mentioned by  @ptrblck.
This phenomenon is based on pytorch1.5.
And when on pytorch1.3, nn.SyncBatchNorm always works regardless of the spatial resolution of input.
Is that a  problem for PyTorch? 1.3 or 1.5? @ptrblck
Thanks in advance.