You could set requires_grad(False) on the encoder or detach the tensor before passing it to the decoder.
Assuming your model has members named encoder and decoder you could try the following:
for param in model.encoder.parameters():
    param.requires_grad_(False)