Solved by deepmo in post #2 
              
              
                Maybe I find the answer. After I see the source code of nn.DataParallel() (version 0.4.1). 
    def __init__(self, module, device_ids=None, output_device=None, dim=0):
        super(DataParallel, self).__init__()

        if not torch.cuda.is_available():
            self.module = module
           …