It depends on what your loss term is. Is the loss computed directly with the output of the decoder? If so: Don’t detach the tensor
Detaching a tensor is the same as saying “I don’t want gradients to flow through here”. In this scenario, your loss is computed with the out of the decoder, gradients flow through to the start of the decoder and then stops there. Your main module will never see any gradients.
To answer your optimizer question; it depends. There’s no wrong answer really. You may want to train them jointly so that the decoder can learn to work better with your main module. You can also choose to just let the main module train on its own and adapt itself to the already trained decoder. I would personally suggest letting them train jointly (so model.parameters()) but you wouldn’t be wrong in only training the main module. Gradients have to flow through the decoder anyways (this would be a different story if you wanted to freeze the encoder instead), so I’m guessing speed-wise it’ll be the same.