My code performs a loss minimization from the output of a pretrained GAN. Apart from the GAN parameters I am also trying to optimize the latent variable z.
The code is set up as follows
with torch.no_grad():
   gan_out = G(z_initial)

z_initial.retain_grad = True
loss = MSEloss(gan_out, observed_image)

GAN_optim.zero_grad()
z_optim.zero_grad()
loss.backward()
GAN_optim.step()
z_optim.step()

The optimizers are defined as
GAN_optim = optim.RMSprop(G.parameters(), lr = 3e-5)
z_optim = optim.RMSprop([z_initial], lr = 1e-3)

# where z_initial is 

z_initial = torch.Variable(z_.clone().to(device), requires_grad = True)

After training, I am visualizing z_initial and the GAN weights. While the GAN weights are updating and changing z_initial is not.
The tensorboard visualization is done for every epoch as follows
writer.add_histogram('Z_prior', z_initial)

How can I make sure z_initial updates?