How large is your batch size using nn.DataParallel?
Would this batch with the model fit on a single GPU without running the forward and backward pass?
I would assume the same, i.e. that the gather on the “master” GPU might run OOM, but not during the initialization.