Thank you @splinter , this is very helpful and successfully rid my LSTM of NaNs.
To abbreviate @splinter’s answer a little, the trick is to call torch.nn.utils.clip_grad_norm_ just after loss.backward like this :
loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), 0.5)