Hello community!
I suspect that during my training procedure, a small number of parameters in my model undergo drastic changes from batch to batch in certain training epochs (exploding gradients). In other words it does not happen all the time. Other than simply printing out the gradients at those epochs/batches (which might be unfeasible for large models), is there a simple way to do this? Like, getting a warning if at one point a gradient becomes “larger than usual”?
Best,
PIF