I dont think the info is enough to know whats the problem exactly. Some of the loss ouput before it goes to nan would be helpful, but I assume that the loss is increase during the training and exploding to inf giving not a number error. if thats the case, the first thing is to see if it makes sense to clip your values.