Since I want to backprop for T, it means T should exist in the 2nd stage of training. Write the gradients with T and code it should have this resolved, but still hope that there can be easier ways of doing this.