Hello,
I am trying to train a GAN in pytorch, and as training the “discriminator” will require doing based on the discriminator output on “real”, and “fake” samples. I tried doing this in two different ways that I expected to be equivalent, but seems to be only one of them works.
First approach: (Didn’t work) - summing the two criterions and backward the combined loss.
criterion = nn.BCELoss()
d_out_real = model_d(real_input)
real_loss = criterion(d_out_real, real_label)
d_out_fake = model_d(fake_input)
fake_loss = criterion(d_out_fake, fake_label)
total_loss = real_loss + fake_loss

d_optimizer.zero_grad() # Adam optimizer to optimize model_d parameters
total_loss.backward()
d_optimizer.step()

Second approach: (Works fine): backward each criterion loss individually, then taking an optimization step.
criterion = nn.BCELoss()
d_optimizer.zero_grad() # Adam optimizer to optimize model_d parameters
d_out_real = model_d(real_input)
real_loss = criterion(d_out_real, real_label)
real_loss.backward()
d_out_fake = model_d(fake_input)
fake_loss = criterion(d_out_fake, fake_label)
fake_loss.backward()
d_optimizer.step()

Third approach: (Doesnot work): Exactly like second approach, except for delaying the real_loss.backward() until after I compute the model output and loss from fake data.
criterion = nn.BCELoss()
d_optimizer.zero_grad() # Adam optimizer to optimize model_d parameters
d_out_real = model_d(real_input)
real_loss = criterion(d_out_real, real_label)
d_out_fake = model_d(fake_input)
fake_loss = criterion(d_out_fake, fake_label)
real_loss.backward()
fake_loss.backward()
d_optimizer.step()

Notes:

I tried using two BCELoss modules, but it didn’t help.
I also tried computing the loss using the functional APIs torch.nn.functional.binary_crossentropy_loss. Also this changed nothing.

Any explanations ?
Thanks,
Moustafa