Hello.
In the context of a central server and several nodes that all have a similar model with different weights, I compute and back-propagate the loss of a norm on the weights on the central model given all the updated nodes models. I want to compute the gradient norm of the difference between the central model and the nodes for every node separetly. The thing is that I load the weigths of the nodes one by one, compute the backward pass on the general model, and finally perform the update step:
    optimizer.zero_grad()
    for i, node in enumerate(nodes):
        user_modelname = f'user_models/user_{i}.pth'
        user_model.load_model(name = user_modelname)
        reg = models_diff(simple_model, user_model)
        reg.backward()
    optimizer.step()

My question is the following: when I check the p.grad of the general model parameters, I get the sum of all the gradients, but I would like the gradients individually of every node and I can’t perform one update at a time since the model difference would change at every iteration, is there a way to get the intermediate gradient values of the backward pass?