So, I am working with a generator and a discriminator in my network.
I wanted to add a new type of loss, which is computed given the features of every layer of the discriminator (these features are computed in the forward() of the discriminator), but contributes to the generator loss.
Let’s call this loss gen_feat_loss.
Therefore what I do is:

call the forward of the discriminator, that returns a list of every discriminator layer output.
compute the discriminator loss with the last element of the list returned by the forward.
compute the gen_feat_loss.
call the forward of the generator, compute all the other losses.
call discriminator.zero_grad()

discriminator_loss.backward()
opt_discriminator.step()
call generator.zero_grad()
generator_loss = loss1 + loss2 + ... +  gen_feat_loss
generator_loss.zero_grad()
generator_loss.backward()
opt_generator.step()

However, when I run the code, I see the following error:
RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.
I tried to change the order of the computations.
I tried to put the generator backward before, and the discriminator backward after.
I then tried to use retain_graph=True as the error suggests, but the RAM finishes.
Do you have any idea of what I am doing wrong?
It must be a problem of the backward of the generator, that tried to compute the gradients on the gen_feat_loss that were, for the last step, already computed by the discriminator.