Solved by ptrblck in post #6 
              
              
                I don’t quite understand the “each element of the list (which is a tensor) has to shared across multiple gpu(in Dataparallel case)”. 
nn.DataParallel will split the input tensor in dim0 and will send each chunk to a GPU. The elements won’t be duplicated, which implies the “sharing”. 
Since the list …