This shortcoming is expected in nn.DataParallel and is one reason I generally recommend to use DistributedDataParallel as well as for performance reason. The underlying mechanics and reason of it are explained in this blog post 3.