I just always do this
def clip_gradient(model, clip):
""“Clip the gradient.”""
if clip is None:
return
totalnorm = 0
for p in model.parameters():
if p.grad is None:
continue
p.grad.data = p.grad.data.clamp(-clip,clip)
and follow it up with a normal step call to my optimizer (my code isn’t formatting properly but I think u get the point)