There are a couple of things going on with your code. You should include how you are defining the optimizers exactly. Given that it is not complete, I cannot try and replicate exactly the error you are getting, however:



 Angry_potato:

    loss_fake=loss_fn(dig(gen_im[i],actual_image[i]),label_fake)


You have already applied a step of your optimizer on geng, however, in the above line of code, you are using its outputs gen_im to compute the loss of the differentiator. Even if it not causing any trouble, try detaching them with .detach() or, if you actually want to backpropagate the error to the generator, generate these images once more after og.step().



 Angry_potato:

  gen_loss.backward(retain_graph=True)


Given that you retain graph, it is possible that you are reapplying past gradients.
Hope these help!