Hello everyone,
I am trying to train a model constructed of three different modules. An encoder, a decoder, and a discriminator. These three are connected as follows. Each of these three should minimize its own loss function which is different from the others.
image1287×645 15.9 KB
Here is my code snippet:
with torch.autograd.set_detect_anomaly(True):
for epoch in range(num_epochs):
for i, (data, labels) in enumerate(train_loader):
        data = Variable(data.cuda())
        labels = Variable(labels.cuda())

        Encoder_hiddens = Variable((torch.zeros(2 * num_encoder_layers, batch_size, hidden_size)).cuda())
        Disc_hiddens = Variable((torch.zeros(2 * num_disc_layers, batch_size, hidden_size)).cuda())

        masked_data = data
        masked_data[:, 1:, mask_part] = 0

        # forward step
        optimizer_encoder.zero_grad()

        outputs_end, hidden_enc = model_encoder(data, Encoder_hiddens)
        outputs_dec = model_decoder(masked_data, hidden_enc)

        loss_ele = pixelwise_criterion(outputs_dec, data)
        # backward step
        loss_ele.backward()
        # optimization step
        optimizer_encoder.step()

        optimizer_decoder.zero_grad()
        disc_real = model_disc(data, Disc_hiddens).view(-1)
        lossD_real = adversarial_criterion(disc_real, torch.ones_like(disc_real))
        disc_fake = model_disc(outputs_dec, Disc_hiddens).view(-1)
        lossD_fake = adversarial_criterion(disc_fake, torch.zeros_like(disc_fake))
        lossD = (lossD_real + lossD_fake) / 2
        loss_decoder = adv_ratio * lossD + (1 - adv_ratio) * loss_ele
        loss_decoder.backward()
        optimizer_decoder.step()

        optimizer_disc.zero_grad()
        lossD.backward()
        optimizer_disc.step()

I am getting the following error:
RuntimeError: Trying to backward through the graph a second time, but the saved intermediate results have already been freed. Specify retain_graph=True when calling backward the first time.
When I use retain_graph=True (loss_ele.backward(retain_graph=True)), I get the following error:
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [450, 45]] is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later.
Does anybody know how it can be fixed?
Thanks,