Not sure if something is missing but isn’t SyncBatchnorm.convert_sync_batchnorm() supposed to convert the module transparently?
However, the following code segment produce ValueError: expected at least 3D input (got 2D input) .
Without the conversion, the forward goes as expected.
Any ideas?
import os
import torch
from torch import nn

module = torch.nn.Sequential(
           torch.nn.Linear(20, 100),
           torch.nn.BatchNorm1d(100)
         ).cuda()

# creating process group (optional)
# process_ids is a list of int identifying rank ids.
os.environ['RANK'] = '0' 
os.environ['WORLD_SIZE'] = '1' 
os.environ['MASTER_ADDR'] = 'localhost'
os.environ['MASTER_PORT'] = '25791'

process_group = torch.distributed.init_process_group(backend='nccl')
module = nn.SyncBatchNorm.convert_sync_batchnorm(module, process_group)

input = torch.randn(2, 20).cuda()
output = module(input)
print(output.shape)

The output:
Traceback (most recent call last):
  File "syncBN.py", line 21, in <module>
    output = module(input)
  File "/home/ml/farleylai/miniconda3/envs/sinet37/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/ml/farleylai/miniconda3/envs/sinet37/lib/python3.7/site-packages/torch/nn/modules/container.py", line 92, in forward
    input = module(input)
  File "/home/ml/farleylai/miniconda3/envs/sinet37/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/ml/farleylai/miniconda3/envs/sinet37/lib/python3.7/site-packages/torch/nn/modules/batchnorm.py", line 429, in forward
    self._check_input_dim(input)
  File "/home/ml/farleylai/miniconda3/envs/sinet37/lib/python3.7/site-packages/torch/nn/modules/batchnorm.py", line 417, in _check_input_dim
    .format(input.dim()))
ValueError: expected at least 3D input (got 2D input)

Expected output as w/o conversion:
torch.Size([2, 100])

Ubuntu 16.04 with PyTorch 1.3 installed through conda.