Solved by MariosOreo in post #2 
              
              
                Hello, from here it seems that you are confused about intermediate value (non-leaf node) and weight of layers (leaf node), by .detach() the weight of layer would not have any effect on the gradiant flow back to the previous layers. If you want, you could .detach() on the intermediate value and the …