I have thought about it for a long time and got the following questions.

There is actually a guy called Leslie N. Smith  5 who created this paper  12.



But in the paper, there is only graph, which learning rate as the x-axis, and accuracy rate as the y-axis. Not like the graph you mentioned, why is the reason?


And at the stage of Choose a good learning rate, there is Sylvain Gugger’s post 47 explaining how the learning rate finder 73 works:



Over an epoch begin your SGD with a very low learning rate (like 10−810−8) but change it (by multiplying it by a certain factor for instance) at each mini-batch until it reaches a very high value (like 1 or 10). Record the loss each time at each iteration and once you’re finished, plot those losses against the learning rate.

My question about here is:


if change it (by multiplying it by a certain factor for instance) at each mini-batch,
the change of loss will because of: 1. input data(each mini-batch data is different); 2. learning rate.
How can we make sure the change of loss is just because of learning rate?


during the learning rate finder 73 working, will the weights in the model be updated?
If yes, The loss value will become smaller because of the model learning itself.
How can we know the change of loss is because of learning rate but not model learning itself?


What is the value of loss in learning rate finder 73? Is just the average of the model output nodes?


I also asked here 72, but I’m not sure if I can get an answer.

  
      
      Deep Learning Course Forums – 17 Oct 19
  
  
    

Learning Rate Finder's mathematical theory 72

My understanding is,  by Learning Rate Finder, we can get an image ‘learning rate’ vs ‘loss’ like this one.    And by observing this image, we can get the best learning rate. Just as described here.  And it said,   For a more intuitive explanation,...