@neale With 0.4 working and 1.0 not working this is clearly a regression. But we haven’t significantly modified (if at all) the data parallel wrapper between these versions. Can you check if the regression happened in 1.0.0 or 1.0.1? I have created https://github.com/pytorch/pytorch/issues/19024 60 to track this.