@ngimel thank you! Yes when I go from 340 mini batches to 170 minibatches I expect the wall clock time to drop from 22 minutes to 11 minutes. Instead I’m seeing no change.
@smth I think I’m missing something, DataParallel only splits up the forward pass, I would think that in order to see a proper speed up during training I’d need to encapsulate the whole train loop so that all weights from the loss backward pass were averaged and then propagated. This snippet is the logic I’m referring to: http://docs.chainer.org/en/latest/tutorial/gpu.html#data-parallel-computation-on-multiple-gpus-without-trainer 36 Is there some equivalent to their ParallelUpdater that I’m overlooking?
Otherwise, I have 61989982 parameters in my model. I guess that’s too many parameters? Why would DataParallel be rate limited by that? And is it possible I’m doing something wrong such that I’m ending up with more parameters than I mean to have? It’s just 5 level encoder-decoder FCN with 1-3 convolutions at each level and a skip connection from the output of the encoder levels to the level with same resolution and number of channels in the decoder stage. See the diagram from the paper https://github.com/mattmacy/vnet.pytorch/blob/master/images/diagram.png 16 if my description doesn’t make sense.
Thank you for your time.