I am working on an architecture where I experience spurious exploding gradients and I want to find out which operation exactly is causing them. I have already identified the parameters that are affected by these huge gradients and have code that identifies when unusual gradients occur, but I am unsure how I can proceed.
I think I know what causes it for some parameters, but others I have no clue.
Ideally, I would recalculate the gradient while retaining the graph and interactively traverse the gradient-calculation backwards to find out what’s going on.
When i recalculate my gradients using .backward(retain_graph=True), the grad_fn for the parameters is still None and I am not sure how to actually interact with the graph and find the reason for the exploding gradient.
EDIT: Ok, I’ve found out that by using create_graph=True the grad_fn get’s populated, but I am unsure whether it’s possible to interact with them in a meaningful way. I have not found a way to evaluate them or get the context of them.