Tou would have to specify what kind of model you are using. To answer as there could be some other cause.
However, from what you are saying it does seem like the learning rate is responsible for this. The reason this occurs is because of the way gradient descent works.

image from here:

  
      stackoverflow.com
  
  
      
    
  

  Gradient Descent: thetas not converging 5



  matrix, machine-learning, octave, linear-regression



  asked by
  
  
    Mark Hodges
  
  on 07:49PM - 19 Feb 17 UTC


  
  
    
    
  
  


Essentially your larger learning rate is causing you to overshoot and you are artificially causing something similar to the exploding gradient problem (which you can read more about here: https://machinelearningmastery.com/exploding-gradients-in-neural-networks/ 20)
In exploding gradient problem errors accumulate as a result of having a deep network and result in large updates which in turn produce infinite values or NaN’s. In your case your large updates are directly a result of having a large learning rate forcing a large update which causes your NaNs. Though if you have a large network it could be a result of both things combined.