High learning rate cannot ensure the fast training.
Anyway, going to NaN is not good at all. Does your model normalize the features by using Batchnorm or something? And activation functions are needed in most of cases.