Solved by Majid_Al in post #2 
              
              
                Pytorch uses that template to instantiate layers to keep track of functions and their derivatives in forward and backward pass. You can check out how those inner-workings come together in the source code. Also look into “tensorboard” to visualize the graph that you build and how it is updated in eac…