I suspect that you don’t see the keys because of this line 2 which is called from DataParallel.forward. Basically you should not try to access replicas on different GPUs directly, because they are handled in different way than regular modules. Also, we recommend to use DistributedDataParallel, instead of this class, to do multi-GPU training, even if there is only a single node. See: Use nn.parallel.DistributedDataParallel instead of multiprocessing or nn.DataParallel and Distributed Data Parallel.