Answers.Body: <p>Removing the learning rate will likely yield <strong>poor convergence</strong> to the optimal policy and optimal Q-values. Note that the current policy is completely dependent on the Q-values, as we take the action with highest Q-value in a given state (with a few other considerations such as exploration, etc.). If we were to remove the learning rate, then we are making a relatively large change to our Q-values and possibly to our policy as well after only a single update. For example, if the sample rewards have great variance (e.g. in stochastic environments), then drastic updates to a single Q-value may occur simply by chance when a learning rate is not used. Due to the recursive definition of Q-values, a few poor updates can undo the work of many previous updates. If this phenomenon were to occur frequently, then the policy may take a long time to converge to the optimal policy, if at all.</p>
  <p>Underlying the temporal-difference update and many other reinforcement learning updates is the notion of policy iteration in which the estimated value function is updated to match the true value function of the current policy and the current policy is updated to be greedy with respect to the estimated value function. This process proceeds iteratively and <strong>gradually</strong> until convergence to the optimal policy and optimal value function is achieved. Gradual changes such as setting a small learning rate (e.g. <span class="math-container">$\alpha = 0.1$</span>) aim to speed up convergence by lessening the frequency of the phenomenon in the above paragraph. Sutton and Barto make comments on convergence throughout their <a href="http://incompleteideas.net/book/the-book-2nd.html" rel="nofollow noreferrer">book</a>, with the remarks surrounding line 2.7 in Section 2.5 providing a summary.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-27T08:08:56.243Z
Answers.Text: May be worth noting there are some situations in which a learning rate of 1 works quite well. That includes quite a few simple scenarios use to teach concepts, like deterministic grid worlds etc.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-29T16:34:17.550Z
Answers.Text: @NeilSlater Since what you said is great information that would help the answer, am I supposed to edit my answer to include the info, or do I leave it as a comment? If I edit my answer to include it, do I give you attribution? I'm still a little new here and am trying to figure out the unwritten rules - I didn't see this explicitly mentioned in the rules when I just check them.
Answers.UserDisplayName: DeepQZero
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-29T16:47:43.857Z
Answers.Text: The comments attached to a question or answer are generally supposed to be suggestions for improvement. They also may be deleted for any reason. So please do take my suggestion and edit something appropriate into the answer to cover it, if you agree. I don't require attribution. I also think for most cases this spoils the flow of the answer. The ideal is that the new parts are edited in so that the answer stands by itself, so no need to flag changes either. Some Stack Exchange sites do have a different culture around this, and some writers like to credit sources. Up to you here
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-27T04:42:42.193Z
Answers.LastActivityDate: 2020-06-27T04:42:42.193Z
Answers.UserDisplayName: DeepQZero
Body: <p>Why isn't it wise for us to completely erase our old Q value and replace it with the calculated Q value? Why can't we forget the learning rate and temporal difference?</p>
  <p>Here's the update formula.</p>
  <p><a href="https://i.sstatic.net/HcnEI.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/HcnEI.png" alt="The formula" /></a></p>
  
Tags: <reinforcement-learning><q-learning><temporal-difference-methods><value-functions>
Title: Why isn't it wise for us to completely erase our old Q value and replace it with the calculated Q value?
UserDisplayName: Chukwudi
ThreadText: # Why isn't it wise for us to completely erase our old Q value and replace it with the calculated Q value?
  
  (Asked by: username_0 on 2020-06-26)
  
  Why isn't it wise for us to completely erase our old Q value and replace it with the calculated Q value? Why can't we forget the learning rate and temporal difference?
  
  Here's the update formula.
  
  [![The formula](https://i.sstatic.net/HcnEI.png)](https://i.sstatic.net/HcnEI.png)
  
  **Tags:** reinforcement-learning, q-learning, temporal-difference-methods, value-functions
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-06-27. Score: 3 | Accepted answer
  
  Removing the learning rate will likely yield **poor convergence** to the optimal policy and optimal Q-values. Note that the current policy is completely dependent on the Q-values, as we take the action with highest Q-value in a given state (with a few other considerations such as exploration, etc.). If we were to remove the learning rate, then we are making a relatively large change to our Q-values and possibly to our policy as well after only a single update. For example, if the sample rewards have great variance (e.g. in stochastic environments), then drastic updates to a single Q-value may occur simply by chance when a learning rate is not used. Due to the recursive definition of Q-values, a few poor updates can undo the work of many previous updates. If this phenomenon were to occur frequently, then the policy may take a long time to converge to the optimal policy, if at all.
  
  Underlying the temporal-difference update and many other reinforcement learning updates is the notion of policy iteration in which the estimated value function is updated to match the true value function of the current policy and the current policy is updated to be greedy with respect to the estimated value function. This process proceeds iteratively and **gradually** until convergence to the optimal policy and optimal value function is achieved. Gradual changes such as setting a small learning rate (e.g. $\alpha = 0.1$) aim to speed up convergence by lessening the frequency of the phenomenon in the above paragraph. Sutton and Barto make comments on convergence throughout their [book](http://incompleteideas.net/book/the-book-2nd.html), with the remarks surrounding line 2.7 in Section 2.5 providing a summary.
  
  > **Comments:**
  > **username_2**: May be worth noting there are some situations in which a learning rate of 1 works quite well. That includes quite a few simple scenarios use to teach concepts, like deterministic grid worlds etc.  
  > **username_1**: @NeilSlater Since what you said is great information that would help the answer, am I supposed to edit my answer to include the info, or do I leave it as a comment? If I edit my answer to include it, do I give you attribution? I'm still a little new here and am trying to figure out the unwritten rules - I didn't see this explicitly mentioned in the rules when I just check them.  
  > **username_2**: The comments attached to a question or answer are generally supposed to be suggestions for improvement. They also may be deleted for any reason. So please do take my suggestion and edit something appropriate into the answer to cover it, if you agree. I don't require attribution. I also think for most cases this spoils the flow of the answer. The ideal is that the new parts are edited in so that the answer stands by itself, so no need to flag changes either. Some Stack Exchange sites do have a different culture around this, and some writers like to credit sources. Up to you here  
  
  ---
  
  