One of the first things to check if your loss is NaN is the learning rate of your optimizer vs. the batch size. What are your settings for learning rate and batch size here? You could try decreasing the learning rate (e.g., incrementally by an order of magnitude each time) and see if that affects the training loss curve.