Are you dealing with a single loss, which contains multiple computation graphs involving a specific parameter?
If so, my suggestion won’t work, as the grad attribute will only be updated once.
Could you post a minimal example?