Yes, looks like a bug in the PyTorch AddmmBackward causing it to output a nan (or something in the autograd). Unless someone else can see something I’m missing you should probably submit an issue. It all seems to be PyTorch code.
In order to get a reliable reproduction you could try adding code in your training loop to store a reference to the current inputs (making sure to overwrite each batch and not keep old ones around). Then when the error happens you should have the inputs that caused it. You will also want to save the model weights as it will depend on these. Then hopefully you can get a set of inputs and weights that will reliably trigger it.