Thanks for your reply.
fake_img is from netG(z), where netG is passed to multiple GPUs. Please see line 46, 58, 63.
 37     device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
 38 
 39     train_set = DatasetFromFolder(TRAIN_DATA_PATH, resize=RESIZE)
 40     val_set = DatasetFromFolder(VAL_DATA_PATH, resize=RESIZE)
 41     train_loader = DataLoader(dataset=train_set, num_workers=TRAIN_NUM_WORKERS, batch_size=TRAIN_BATCH_SIZE, shuffle=True)
 42     val_loader = DataLoader(dataset=val_set, num_workers=4, batch_size=1, shuffle=False)
 43 
 44     writer = SummaryWriter(log_dir='logs/current_model')
 45 
 46     netG = Generator()
 47     print('# generator parameters:', sum(param.numel() for param in netG.parameters()))
 48     netD = Discriminator()
 49     print('# discriminator parameters:', sum(param.numel() for param in netD.parameters()))
 50 
 51     generator_criterion = GeneratorLoss()
 52     # adversarial_criterion = nn.BCELoss()
 53     adversarial_criterion = nn.BCEWithLogitsLoss()
 54     mse_loss = nn.MSELoss()
 55 
 56     if torch.cuda.device_count() > 1:
 57         print("Let's use", torch.cuda.device_count(), "GPUs!")
 58         netG = nn.DataParallel(netG)
 59         netD = nn.DataParallel(netD)
 60         generator_criterion = nn.DataParallel(generator_criterion)
 61         adversarial_criterion = nn.DataParallel(adversarial_criterion)
 62         mse_loss = nn.DataParallel(mse_loss)
 63     netG.to(device)
 64     netD.to(device)
 65     generator_criterion.to(device)
 66     adversarial_criterion.to(device)
 67     mse_loss.to(device)

If netG is running on multiple GPU, isn’t its output - fake_img, also on multiple GPU? Should I declare it again by fake_img.to(device)? I did try, here’re the changes I made and the corresponding error (same error):
142             fake_img.to(device)
143             interpolates = (1 - alpha) * fake_img  
144             # interpolates = alpha * real_img + ((1 - alpha) * fake_img)  
145             interpolates.to(device) 

and
Let's use 4 GPUs!
Reading checkpoint...
RTResNet-MSE model restore success! Start training in RTGAN with  mse  loss!
RTGAN Training Starts!
 0%|                                                                        | 0/632 [00:00<?, ?it/s]
<class 'torch.Tensor'> torch.Size([16, 3, 128, 128])
<class 'torch.Tensor'> torch.Size([16, 3, 128, 128])
<class 'torch.Tensor'> torch.Size([16, 3, 128, 128])
 0%|                                                                        | 0/632 [00:11<?, ?it/s]
Traceback (most recent call last):
    File "train_rtgan.py", line 143, in <module>
        interpolates = (1 - alpha) * fake_img
RuntimeError: expected device cpu but got device cuda:0

Thank you again for helping me. Appreciate