Hi, I have a problem during implementing distributed training.
The dataset is an M*N matrix and input is a vector.
The dataset is loaded as:
class ReadDataset(data.Dataset):
  def __init__(self, filename):
    self._filename = filename
    self._total_data = 0
    with open(filename, 'r') as f:
      self._total_data = len(f.readlines()) - 1

  def __getitem__(self, idx):
    line = linecache.getline(self._filename, idx + 1)
    return line

  def __len__(self):
    return self._total_data 

Then, read it
dataset = ReadDataset(training_filename)
  dataLoader = data.DataLoader(dataset)

the model is:
model = nn.Sequential( nn.Linear(2000, 150), nn.Linear(150, 2000) )
In distributed training:
train_sampler = torch.utils.data.distributed.DistributedSampler(dataset)
 dataLoader = data.DataLoader(dataset, num_workers=args['world-size'], sampler=train_sampler)

 for epoch in range(10):
    train_sampler.set_epoch(epoch)
    for target in dataLoader:
       in_val = ## a vector, whose size equals to len(target) ##
       out = model(in_val)

The error when training with 2 GPUs is,
RuntimeError: size mismatch, m1: [1 x 1000], m2: [2000x 150] at /opt/conda/conda-bld/pytorch_1524586445097/work/aten/src/THC/generic/THCTensorMathBlas.cu:249.
Something is wrong, but I do not know where to start. Can someone give some suggestions?
Thanks