I noticed that sometimes at high learning rate, my model produces NaN randomly in the test output:
ValueError: Input contains NaN, infinity or a value too large for dtype('float32').

That is to say, I am able to train it properly without error, but randomly during evaluating on the test set, the prediction from the model contains NaN.
Is this known to happen, and if so, why?