Hello,
I’d like to train my model on multiple GPUs but unfortunately I’m getting a massive validation error (but not so when only doing one gpu) after even 1 epoch.
I think the reason is that I manually modify some of the model parameters, after the optimization step:
torch.cuda.set_device(args.local_rank)
torch.distributed.init_process_group(backend='nccl', init_method='env://')

...

model = nn.Sequential(...).cuda()
dmodel = nn.DataParallel(model)

...
    loss = criterion(dmodel(x),y)
    loss.backward()
    optimizer.step()
    with torch.no_grad():
        deterministic_modify(model[17]) # I need to manually modify some weights.

I’m guessing there is a sync problem, because if I do this on a single gpu, things work as expected. But on multiple gpu’s I get terrible validation error.
The way I understand nn.DataParallel works (but please correct me if I’m wrong) is that it’s a wrapper, and each gpu has a copy of the model, and nn.DataParallel splits the batch into two, gives each gpu half, computes the gradients, and then, somehow, sync’s the model in both gpus (how?).
Thanks.