Hi
I am developing a sort of creature which is similar to Auto encoder or VAE
I want the encoder to have a loss of his own (f) and than take its outputs (Y) and feed them the decoder with his own loss (g)
I want g to have derivatives up to Y (namely I dont want the gradient of g to reach the encoder)
However, when I run the code I can see that even if the loss is only g and the optimizer is allowed to change only the encoder, learning takes place i.e. the decoder’s loss changes the weights of the encoder.
What can I do?