What kind of code are you currently using to “identify when unusual gradients occur”?
You could register hooks on all parameters and maybe print some debug message, when high gradients are propagated.
Once the step is found, you could check all .grad attributes as well as the parameters to see, which operations causes the high gradients?
Maybe I’ve misunderstood your question and you are already doing exactly this.