This solution is not very clean, but you can override the replicate and scatter on DataParallel to make it work: https://github.com/pytorch/pytorch/blob/v0.3.1/torch/nn/parallel/data_parallel.py#L76-L80 257  Hope this helps!