Thanks. From the blog post, for nn.DataParallel, we should be using loss.mean().backward() instead of loss.backward() ?
The official tutorial does not indicate a need to change the backwards pass code.