If you’re using something like adam, gradient variance is already tracked, so I’d consider using that information for tracking. For crude fixing, nn.utils has some clipping functions. And IIRC tensorboard can plot gradient densities over epochs (for tracked params/tensors).