I have auto-encoder. It has two parts encoder and decoder. Its training process is:
latent_space = encoder(x)
y = decoder(latent_space)
latent_reconstruct = encoder(y)
loss = l1 * F.mse_loss(x, y) + l2 * F.mse_loss(latent_space, latent_reconstruct)
loss.backward()
…
Here I want to use encoder as lens (previously i was using VGG). I don’t want to gradients to flow through encoder when used second time.
To make it simple, it can be achieved by copying whole encoder before applying it for second time. But my encoder part is very large and I can’t afford to copy it for every single iteration.