Assuming that the forward pass does not create invalid outputs, you could register hooks to the parameters of the model and print the gradients during the backward pass in order to isolate which gradient gets the first invalid value.
This could make it easier to debug the issue further and check the operations used to create this gradient (e.g. are you dividing by a small number?).