Solved by thecho7 in post #3 
              
              
                I solved this issue by map_location='cpu’ 
Loading pretrained model 
ckpt = torch.load(path)

automatically allocates the parameters to GPU:0. 
I cannot understand why until now, but it is solved by using map_location while I load the model. 
ckpt = torch.load(path, map_location='cpu')
model_A.load_…