Solved by albanD in post #2 
              
              
                Hi, 
So what you do here is the following. 
I use l1 for l1.weight and l2 for l2.weight for readability. 
a = l1*(l2*inputs) 
grad wrt a: 
inputs.grad = 1 * l1 * l2 
gp = 3 * inputs.grad = 3 * l1 * l2 
grad wrt gp now 
dgp/dl1 = 3*l2 = 9 
dgp/dl2 = 3*l1 = 6 
So I would agree with the autograd result…