If the loss is going to infinity does making your learning rate smaller help?
Is the loss actually going to infinity? There is an infinity value inf, but nan is something different. For example, if you take the log of a negative number you’ll get a nan, but if you divide 1 by 0 you get inf (inside tensors).