I guess a division or any other operation might create invalid outputs so you could add debug print statements to the code and check where the first NaN or Inf value is created.
Note that your model might generally blow up and the parameters might become invalid or your custom loss function might also create these invalid values.