Solved by ayalaa2 in post #2 
              
              
                It depends on what your loss term is. Is the loss computed directly with the output of the decoder? If so: Don’t detach the tensor 
Detaching a tensor is the same as saying “I don’t want gradients to flow through here”. In this scenario, your loss is computed with the out of the decoder, gradients f…