natank1:

if the loss is only g and the optimizer is allowed to change only the encoder, learning takes place i.e. the decoder’s loss changes the weights of the encoder.


How did you make sure that the encoder doesn’t get gradients? Did you detach Y before feeding it into the decoder?
Also, why is the optimizer allowed to update the encoder, if it shouldn’t be trained?