Responding to my own question in case anyone ever has this question in the future.
I introduced gradient clipping as shown here:
https://stackoverflow.com/questions/54716377/how-to-do-gradient-clipping-in-pytorch
for p in model.parameters():
    p.register_hook(lambda grad: torch.clamp(grad, -clip_value, clip_value))

This appears to have remediated the problem.