Here is my implementation of CycleGAN 8, where I parallelize training by making use of 4 GPUs.
CycleGAN consists of 4 models: Generator+Discriminator for type A images, and Generator+Discriminator for type B images.
I train the first pair of G and D on one device and the second pair on the other.
netG_B2A = Generator().to(device1)
netD_A = Discriminator().to(device1)
netG_A2B = Generator().to(device2)
netD_B = Discriminator().to(device2)

cycle_loss1 = torch.nn.L1Loss().to(device1)
cycle_loss2 = torch.nn.L1Loss().to(device2)
identity_loss1 = torch.nn.L1Loss().to(device1)
identity_loss2 = torch.nn.L1Loss().to(device2)
adversarial_loss1 = torch.nn.MSELoss().to(device1)
adversarial_loss2 = torch.nn.MSELoss().to(device2)

I also have 2 copies of input data and 2 copies of each loss functions that I calculate on different devise.
real_image_A1 = data[0].to(device1)
real_image_B1 = data[1].to(device1)
real_image_A2 = data[0].to(device2)
real_image_B2 = data[1].to(device2)

Moreover, I have to copy the resulting images (which will be feed into the second pair of G and D) of one G to another device.
# ...
fake_image_B = netG_A2B(real_image_A2)
# ...
recovered_image_A = netG_B2A(fake_image_B.to(device1))

The final loss is calculated on the CPU.
errG = loss_identity_A.cpu() + loss_identity_B.cpu() + loss_GAN_A2B.cpu() + loss_GAN_B2A.cpu() + loss_cycle_ABA.cpu() + loss_cycle_BAB.cpu()

Besides, I have 2 classification model which are allocated on their own devices.