Hi everyone!
Currently, I’m trying to replicate a DeblurGanV2 network (link). At the moment, I’m working on performing the training. Here is my current status of my trainings pipeline:

torch.autograd.set_detect_anomaly(mode=True)
total_generator_loss = 0
total_discriminator_loss = 0
psnr_score = 0.0
used_loss_function = ‘wgan_gp_loss’
for epoch in range(n_epochs):
#set to train mode
generator.train(); discriminator.train()
tqdm_bar = tqdm(train_loader, desc=f’Training Epoch {epoch} ', total=int(len(train_loader)))
for batch_idx, imgs in enumerate(tqdm_bar):
#load imgs to cpu
blurred_images = imgs["blurred"].cuda()
sharped_images = imgs["sharp"].cuda()

# generator output
deblurred_img = generator(blurred_images)

# denormalize
with torch.no_grad():
  denormalized_blurred = denormalize(blurred_images)
  denormalized_sharp = denormalize(sharped_images)
  denormalized_deblurred = denormalize(deblurred_img)

# get D's output
sharp_discriminator_out = discriminator(sharped_images)
deblurred_discriminator_out = discriminator(deblurred_img)

# set critic_updates
if used_loss_function== 'wgan_gp_loss':
  critic_updates = 5
else:
    critic_updates = 1

#train discriminator
discriminator_loss = 0
for i in range(critic_updates):
  discriminator_optimizer.zero_grad()
  # train discriminator on real and fake
  if used_loss_function== 'wgan_gp_loss':
    gp_lambda = 10
    alpha = random.random()
    interpolates = alpha * sharped_images + (1 - alpha) * deblurred_img
    interpolates_discriminator_out = discriminator(interpolates)
    kwargs = {'gp_lambda': gp_lambda,
               'interpolates': interpolates,
               'interpolates_discriminator_out': interpolates_discriminator_out,
               'sharp_discriminator_out': sharp_discriminator_out,
               'deblurred_discriminator_out': deblurred_discriminator_out
                }
    wgan_loss_d, gp_d = wgan_gp_loss('D', **kwargs)
    discriminator_loss_per_update = wgan_loss_d + gp_d

  discriminator_loss_per_update.backward(retain_graph=True)
  discriminator_optimizer.step()
  discriminator_loss += discriminator_loss_per_update.item()


But when I run this code, I receive the following error message:

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [1, 512, 4, 4]] is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!


RuntimeError                              Traceback (most recent call last)
 in ()
62 #       discriminator_loss_per_update = gan_loss_d
63
—> 64       discriminator_loss_per_update.backward(retain_graph=True)
65       discriminator_optimizer.step()
66       discriminator_loss += discriminator_loss_per_update.item()
1 frames
/usr/local/lib/python3.7/dist-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph, inputs)
243                 create_graph=create_graph,
244                 inputs=inputs)
 → 245         torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
246
247     def register_hook(self, hook):
/usr/local/lib/python3.7/dist-packages/torch/autograd/init.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables, inputs)
145     Variable.execution_engine.run_backward(
146         tensors, grad_tensors, retain_graph, create_graph, inputs,
 → 147         allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag
148
149

Unfortunately, I can’t really trace the in-place operation that would cause this error. Does anybody maybe has an idea or advice for me? I would appreciate any input