Answers.Body: <p>First of all, I'd say that there is a reason to give Learning Rate (LR) and Exploration Rate (ER) the same decay: they play at the same scale (the number of successive batches you'll train your model on). But if I refine the analysis, I would rather say that it's a reason to choose them in the same range, i.e. close to 1, but not specifically at the same number.</p>
  <ul>
  <li><p>For LR decay, people often choose it very close to one (which can mean really different things like 0.98 or 0.997), because it plays on a large scale, and you don't want the LR to disappear too brutally.</p>
  </li>
  <li><p>However, the choice of ER decay can have more variation from model to model. It depends on the initial value of ER (you don't wanna decay fastly ER if your ER is initially low), and also to the &quot;learning speed&quot; of your model: if your model learns efficiently at the beginning, you could want to fastly decrease ER in order to reduce the noise on the action, supposing that you did enough exploration at the beginning (but I think this last opinion is more controversial). You can find an interesting paper <a href="https://theses.ubn.ru.nl/bitstream/handle/123456789/5216/Nieuwdorp%2C_T._1.pdf?sequence=1" rel="nofollow noreferrer">here</a>, where the author tries different ER decay and finds out that 0.99 is the best, for CartPole environment.</p>
  </li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-11T11:13:05.223Z
Answers.LastActivityDate: 2022-01-07T16:18:03.587Z
Answers.LastEditDate: 2022-01-07T16:18:03.587Z
Answers.UserDisplayName: 16Aghnar
Body: <p>Should I be decaying the learning rate and the exploration rate in the same manner? What's too slow and too fast of an exploration and learning rate decay? Or is it specific from model to model?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-09-12T00:12:53.437Z
Comments.Text: @16Aghnar in what scenario would I use a lower learning rate? I've looked at the ATARI papers and games like Super Mario Bros. They used learning rates of 0.00025. Is it because we want it to not get hooked onto the same decision again and again? As in, with a higher learning rate, it would assume action x would be best. I hope I made sense!
Comments.UserDisplayName: rtz
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-09-12T08:07:06.283Z
Comments.Text: So, a lower LR means a slower convergence but an improved asymptote (limit of the learning curve). So tuning it depends on the time you have, and also on your model. You can begin with, for example, 0.001, see the learning curve, and if you reach quickly the asymptote, you can try with a lower LR, see again the learning curve, and so on. (and notice that the 0.98 - 0.997 values I mentioned in my answer are for the LR decay, not for the LR)
Comments.UserDisplayName: 16Aghnar
Tags: <reinforcement-learning><deep-rl><hyper-parameters><learning-rate><exploration-strategies>
Title: Should I be decaying the learning rate and the exploration rate in the same manner?
UserDisplayName: rtz
ThreadText: # Should I be decaying the learning rate and the exploration rate in the same manner?
  
  (Asked by: username_0 on 2018-09-11)
  
  Should I be decaying the learning rate and the exploration rate in the same manner? What's too slow and too fast of an exploration and learning rate decay? Or is it specific from model to model?
  
  **Tags:** reinforcement-learning, deep-rl, hyper-parameters, learning-rate, exploration-strategies
  
  > **Comments:**
  > **username_0**: @username_1 in what scenario would I use a lower learning rate? I've looked at the ATARI papers and games like Super Mario Bros. They used learning rates of 0.00025. Is it because we want it to not get hooked onto the same decision again and again? As in, with a higher learning rate, it would assume action x would be best. I hope I made sense!  
  > **username_1**: So, a lower LR means a slower convergence but an improved asymptote (limit of the learning curve). So tuning it depends on the time you have, and also on your model. You can begin with, for example, 0.001, see the learning curve, and if you reach quickly the asymptote, you can try with a lower LR, see again the learning curve, and so on. (and notice that the 0.98 - 0.997 values I mentioned in my answer are for the LR decay, not for the LR)  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-09-11. Score: 6 | Accepted answer
  
  First of all, I'd say that there is a reason to give Learning Rate (LR) and Exploration Rate (ER) the same decay: they play at the same scale (the number of successive batches you'll train your model on). But if I refine the analysis, I would rather say that it's a reason to choose them in the same range, i.e. close to 1, but not specifically at the same number.
  
  * For LR decay, people often choose it very close to one (which can mean really different things like 0.98 or 0.997), because it plays on a large scale, and you don't want the LR to disappear too brutally.
  * However, the choice of ER decay can have more variation from model to model. It depends on the initial value of ER (you don't wanna decay fastly ER if your ER is initially low), and also to the "learning speed" of your model: if your model learns efficiently at the beginning, you could want to fastly decrease ER in order to reduce the noise on the action, supposing that you did enough exploration at the beginning (but I think this last opinion is more controversial). You can find an interesting paper [here](https://theses.ubn.ru.nl/bitstream/handle/123456789/5216/Nieuwdorp%2C_T._1.pdf?sequence=1), where the author tries different ER decay and finds out that 0.99 is the best, for CartPole environment.
  
  
  ---
  
  