I have AutoEncoder consisting of encoder and decoder. I want to reconstruct image. So I am optimizing parameters of both encoder and decoder but at the same time I need to use encoder layers (Usually you use VGG) for computing reconstruction loss. But for second encoder pass, I dont want to consider gradient. torch.no_grad() completely discards graph. Here is what code should have looked like:
latent = encoder(x)
x2 = decoder(x)
with torch.no_grad():
    latent2 = encoder(x2)
loss = F.mse_loss(x2, x) + F.mse_loss(latent2, latent)
loss.backward()

Problem is not that its not possible. It can be done by just copying encoder but that’s memory and compute inefficient as its being done for each iteration.
latent = encoder(x)
x2 = decoder(x)
latent2 = clone(encoder)(x)
loss = F.mse_loss(x2, x) + F.mse_loss(latent2, latent)
loss.backward()