Yeah, that makes sense. Though in my case the two forwards are identical, this would require, I suppose,  strange workarounds to avoid redundant computation. Was hoping for a more build-in solution.
By the way, the context is nearest-embed layer 13 for the VQ-VAE 12 model.
I actually need two identical outputs: one with the encoder input detached (stop-gradient) and one with the dictionary input detached.