I am not sure how to identify/verify exploding gradients, you could try gradient clipping using something like below that will prevent the gradients from going aboard:
torch.nn.utils.clip_grad_norm(model.parameters()) as shown in:
https://github.com/pytorch/examples/blob/master/word_language_model/main.py#L138 613 and see if that makes any difference.