Yes in that case if model2_output does not depend on the decoder then the decoder weights will not have gradients (after loss2.backward()) and as a result optimizer2.step() will not update the weights of the decoder (due to the gradients being either 0 or None).