Hi
I am building a VAE-GAN (https://miro.medium.com/max/2992/0*KEmfTtghsCDu6UTb.png 3). It has 3 different loss functions for each part of the network (encoder, decoder, discriminator) and each loss function is a function of outputs/network weights of the different networks.
Pytorch throws this error RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation when calling the second backward pass.
I think it is because the gradients of the network are being used to calculate the loss functions after they have already been updated. I have tried freezing the network gradients and then backpropping but to no avail.
What is best practise for dealing with this kind of issue?
My code looks like this
for data in trainLoader:
		i += 1
		X, _ = data
		X = X.view(-1, 28*28)
		X = X.cuda()

		# zero gradients from last step
		enc_opt.zero_grad()
		dec_opt.zero_grad()
		disc_opt.zero_grad()

		mu, std = enc(X) # encode data; mu = E[z]

		X_tilde = dec(mu) # reconstructed image from training data

		_, disc_1_real, disc_2_real = disc(X)
		_, disc_1_fake, disc_2_fake = disc(X_tilde)


		Z = z_sample(mu.clone().detach(), std.clone().detach()) # sample z using reparameterisation trick
		X_p = dec(Z.detach())# randomly generated image

		# define losses
		L_llike =  mse_loss(disc_1_fake, disc_1_real) + mse_loss(disc_2_fake, disc_2_real)# disc featurewise error
		# L_llike = mse_loss(X_tilde, X)
		L_GAN = GAN_loss(X, X_tilde, X_p, disc)
		L_prior = KL_loss(mu, std) # regularization loss


		# define network losses
		L_enc = L_prior + L_llike
		L_dec = gamma*L_llike.clone() - L_GAN
		L_disc = L_GAN.clone()

		set_requires_grad(enc, True)
		set_requires_grad(dec, False)
		set_requires_grad(disc, False)

		# train encoder
		L_enc.backward(retain_graph=True)
		enc_opt.step()

		set_requires_grad(enc, False)
		set_requires_grad(dec, True)
		set_requires_grad(disc, False)

		# train decoder
		L_dec.backward(retain_graph=True)
		dec_opt.step()

		set_requires_grad(enc, False)
		set_requires_grad(dec, False)
		set_requires_grad(disc, True)

		# train discriminator
		L_disc.backward()
		disc_opt.step()#