Sure,
In the usual way, the gradient of the first weight would be:
grad1 = -(t - outputs[1]) * outputs[1]
grad2 = -(t-outputs[1])* model.layer2[0].weight * outputs[0]
For the new loss function in the hidden layer, I have:
L = 0.5*torch.sum( (t1 - outputs[0])*2.0)
and t1 is:
t1 = outputs[0] + (t-outputs[1]) model.layer2[0].weight
Which the algorithm is right
Now, I want the gradient for the first weight is the derivative of L with respect to the weight, which is:
grad = -(t1 - outputs[0]) * outputs[0]
grad = -(t-outputs[1]) * model.layer2[0].weight * outputs[0]
Which is same as the usual way, if I disable create_graph=True, then I get zero gradient, maybe I am understanding wrong this create_graph?