The gradients of intermediate variables (in this case, input1,input2,input3) will be cleared by autograd during backpropagation. If you really want to retain the gradient for certain variables (for example, to visually inspect) after backpropagation, use .retain_grad() on those variables.