Solved by ptrblck in post #2 
              
              
                enc_out would also get gradients, if you pass its output to classifier2 directly and call backward() on the calculated loss. To avoid it, you should detach the tensor: 
class_out2 = classifier2(enc_out.detach())