Answers.Body: <blockquote>
  <p>Suppose we have a small space state and that, after about 2000 episodes, we've accurately explored the environment and known the accurate <span class="math-container">$Q$</span> values. In that case, why do we still leave a small probability for exploration?</p>
  </blockquote>
  <p>It will depend on the goal of the work:</p>
  <ul>
  <li><p>If the learning algorithm is off-policy (e.g. Q learning), it is normal to continue to explore at a moderate-to-low rate because it can accurately estimate an optimal deterministic target policy from a close-to-optimal stochastic behaviour policy.</p>
  </li>
  <li><p>Perhaps it is engineered with a low tolerance and will keep going even when you don't need it to.</p>
  </li>
  <li><p>Perhaps the code is for education and run so long that convergence is easily visible. Or for comparison with other methods which really do take that long to converge, and you would like data on the same axis.</p>
  </li>
  <li><p>For comparison with other methods for sample efficiency whilst learning and measuring <a href="https://www.coursera.org/lecture/practical-rl/regret-measuring-the-quality-of-exploration-mNxVA" rel="nofollow noreferrer">regret</a> (i.e. how much the exploration is costing you).</p>
  </li>
  <li><p>When environment is dynamic and could change, then continuous exploration is potentially useful to discover the changes, as you suggest in the question.</p>
  </li>
  </ul>
  <p>If you do really have an ideal agent, then of course you could just stop and say &quot;job done&quot;. In practice for more interesting problems, you won't usually get small state spaces and perfect solutions inside 2000 episodes (or ever) - as a result if you are reading tutorials in reinforcement learning, they may just skip this point.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-24T13:56:34.363Z
Answers.Text: Ok so due to the randomness of the environment, it’s ideal to always keep exploring in case of new changes to the environment?
Answers.UserDisplayName: Chukwudi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-24T13:58:25.960Z
Answers.Text: And since small space states are rare, it’s also possible that our space state is close to infinite and therefore even 2000 episodes isn’t enough to have accurate Q values ?
Answers.UserDisplayName: Chukwudi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-24T14:15:18.143Z
Answers.Text: @Chukwudi: It all *depends*. However, those are both valid reasons to continue exploring if they are true for your problem.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-24T14:24:55.487Z
Answers.Text: Is there anyway I could have a personal chat with you ?
Answers.UserDisplayName: Chukwudi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-24T14:29:35.667Z
Answers.Text: @Chukwudi: Sure - it's a public chat mind - here - https://chat.stackexchange.com/rooms/info/109789/why-do-we-explore-after?tab=general - I am happy to talk about refinements to the question and answer here, but I don't want to take role of persoanl tutor or debug any personal projects etc.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-24T13:44:26.447Z
Answers.LastActivityDate: 2020-06-24T14:18:28.403Z
Answers.LastEditDate: 2020-06-24T14:18:28.403Z
Answers.UserDisplayName: Neil Slater
Answers.Body: <p>When you are training a system using stochastic gradient descent, your system will converge towards some local minimum. If the local minimum was a good one, we would be fine with it. However, we cannot know how good a found solution is in comparison to other solutions of which we do not know their quality because they have been insufficiently explored. So, continuing to explore is a good way to escape comparatively bad local minima even if training has progressed already for quite a bit.</p>
  <p>Besides that, maybe even more importantly towards the end of training, one also wants the system to perform well, i.e. robustly, in the presence of noise and not just under ideal circumstances. So, introducing some randomness, i.e. noise, into the network's policy can also lead to more robust policies being learned since the agent gets trained on how to best recover failure/unforeseen transitions into unexpected states.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-24T13:57:02.720Z
Answers.Text: So due to the randomness of the environment( for example a new reward might pop up) it’s ideal to always keep exploring
Answers.UserDisplayName: Chukwudi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-24T15:03:32.530Z
Answers.Text: Consider the case where an agent has to move across some environment and occasionally slips out or gets displaces by some unforeseen event, e.g. by being displaced by a user (might happen to your lawn mowing robot if you realize that it notoriously misses to visit one particular spot in your garden). Then, the agent will have to no idea how to behave in that situation/state, except for random guessing, if the weights of the network containing the agent's policy have never been trained on coping with such an unforeseen state suddenly being encountered for the first time after training is done).
Answers.UserDisplayName: Daniel B.
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-24T15:17:13.257Z
Answers.Text: Ok so basically we train our agent , it goes to the real world , and at a particular state it was trained to always go up , now if there’s an external interference, where always going up makes it land in a pit, it wouldn’t know what to do and will be confused right, so it’s best we give some randomness
Answers.UserDisplayName: Chukwudi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-24T15:29:11.037Z
Answers.Text: Yes, because the randomness shall at least somewhat prepare the agent towards how to cope with being in that pit. Maybe the agent hasn't encountered that specific specific situation yet when getting trapped during run time for the first time, but at least it could have encountered comparable situations during training that could enable it to have at least somewhat accurate estimates of Q-values for the novel situation. And that is only achieved if there is at least some exploration constantly throughout training.
Answers.UserDisplayName: Daniel B.
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-24T15:45:18.553Z
Answers.Text: Can we assume the pit it landed in is another state, if it’s another new state it means it has no Idea of what to do in that state since all Q values are 0, so basically giving a chance of exploring is a good thing
Answers.UserDisplayName: Chukwudi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-24T23:33:40.417Z
Answers.Text: Yes, as long as the degree of exploration doesn't interfere with the eventual convergence to a reasonably good policy, exploration is a good thing.
  And yes, a pit would be a new, distinct environmental state. But please note that a pit could be similar to other, similar training situations/states encountered beforehand during training. So, even if a particular state has not explicitly been visited during training, the deep RL agent can still learn how to cope with such a novel state given the generalization ability of the policy-network it employs.
Answers.UserDisplayName: Daniel B.
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-26T11:54:11.400Z
Answers.Text: B I kind of get it , if we’re using a neural network, our function approximator would Be able to provide close to accurate Q values of the new state it encountered, regardless of whether it had visited the state during its training
Answers.UserDisplayName: Chukwudi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-24T13:44:41.203Z
Answers.LastActivityDate: 2020-06-24T13:44:41.203Z
Answers.UserDisplayName: Daniel B.
Body: <p>Suppose we have a small space state and that, after about 2000 episodes, we've accurately explored the environment and known the accurate <span class="math-container">$Q$</span> values. In that case, why do we still leave a small probability for exploration?</p>
  <p>My guess is in the case of a dynamic environment where a bigger reward might pop up in another state. Is my assumption correct?</p>
  
Tags: <reinforcement-learning><q-learning><exploration-exploitation-tradeoff>
Title: Why do we explore after we have an accurate estimate of the value function?
UserDisplayName: Chukwudi
ThreadText: # Why do we explore after we have an accurate estimate of the value function?
  
  (Asked by: username_0 on 2020-06-24)
  
  Suppose we have a small space state and that, after about 2000 episodes, we've accurately explored the environment and known the accurate $Q$ values. In that case, why do we still leave a small probability for exploration?
  
  My guess is in the case of a dynamic environment where a bigger reward might pop up in another state. Is my assumption correct?
  
  **Tags:** reinforcement-learning, q-learning, exploration-exploitation-tradeoff
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2020-06-24. Score: 0 | Accepted answer
  
  > Suppose we have a small space state and that, after about 2000 episodes, we've accurately explored the environment and known the accurate $Q$ values. In that case, why do we still leave a small probability for exploration?
  
  It will depend on the goal of the work:
  
  * If the learning algorithm is off-policy (e.g. Q learning), it is normal to continue to explore at a moderate-to-low rate because it can accurately estimate an optimal deterministic target policy from a close-to-optimal stochastic behaviour policy.
  * Perhaps it is engineered with a low tolerance and will keep going even when you don't need it to.
  * Perhaps the code is for education and run so long that convergence is easily visible. Or for comparison with other methods which really do take that long to converge, and you would like data on the same axis.
  * For comparison with other methods for sample efficiency whilst learning and measuring [regret](https://www.coursera.org/lecture/practical-rl/regret-measuring-the-quality-of-exploration-mNxVA) (i.e. how much the exploration is costing you).
  * When environment is dynamic and could change, then continuous exploration is potentially useful to discover the changes, as you suggest in the question.
  
  If you do really have an ideal agent, then of course you could just stop and say "job done". In practice for more interesting problems, you won't usually get small state spaces and perfect solutions inside 2000 episodes (or ever) - as a result if you are reading tutorials in reinforcement learning, they may just skip this point.
  
  > **Comments:**
  > **username_0**: Ok so due to the randomness of the environment, it’s ideal to always keep exploring in case of new changes to the environment?  
  > **username_0**: And since small space states are rare, it’s also possible that our space state is close to infinite and therefore even 2000 episodes isn’t enough to have accurate Q values ?  
  > **username_1**: @Chukwudi: It all *depends*. However, those are both valid reasons to continue exploring if they are true for your problem.  
  > **username_0**: Is there anyway I could have a personal chat with you ?  
  > **username_1**: @Chukwudi: Sure - it's a public chat mind - here - https://chat.stackexchange.com/rooms/info/109789/why-do-we-explore-after?tab=general - I am happy to talk about refinements to the question and answer here, but I don't want to take role of persoanl tutor or debug any personal projects etc.  
  
  ---
  
  ### Answer by username_2 on 2020-06-24. Score: 0
  
  When you are training a system using stochastic gradient descent, your system will converge towards some local minimum. If the local minimum was a good one, we would be fine with it. However, we cannot know how good a found solution is in comparison to other solutions of which we do not know their quality because they have been insufficiently explored. So, continuing to explore is a good way to escape comparatively bad local minima even if training has progressed already for quite a bit.
  
  Besides that, maybe even more importantly towards the end of training, one also wants the system to perform well, i.e. robustly, in the presence of noise and not just under ideal circumstances. So, introducing some randomness, i.e. noise, into the network's policy can also lead to more robust policies being learned since the agent gets trained on how to best recover failure/unforeseen transitions into unexpected states.
  
  > **Comments:**
  > **username_0**: So due to the randomness of the environment( for example a new reward might pop up) it’s ideal to always keep exploring  
  > **username_2**: Consider the case where an agent has to move across some environment and occasionally slips out or gets displaces by some unforeseen event, e.g. by being displaced by a user (might happen to your lawn mowing robot if you realize that it notoriously misses to visit one particular spot in your garden). Then, the agent will have to no idea how to behave in that situation/state, except for random guessing, if the weights of the network containing the agent's policy have never been trained on coping with such an unforeseen state suddenly being encountered for the first time after training is done).  
  > **username_0**: Ok so basically we train our agent , it goes to the real world , and at a particular state it was trained to always go up , now if there’s an external interference, where always going up makes it land in a pit, it wouldn’t know what to do and will be confused right, so it’s best we give some randomness  
  > **username_2**: Yes, because the randomness shall at least somewhat prepare the agent towards how to cope with being in that pit. Maybe the agent hasn't encountered that specific specific situation yet when getting trapped during run time for the first time, but at least it could have encountered comparable situations during training that could enable it to have at least somewhat accurate estimates of Q-values for the novel situation. And that is only achieved if there is at least some exploration constantly throughout training.  
  > **username_0**: Can we assume the pit it landed in is another state, if it’s another new state it means it has no Idea of what to do in that state since all Q values are 0, so basically giving a chance of exploring is a good thing  
  > **username_2**: Yes, as long as the degree of exploration doesn't interfere with the eventual convergence to a reasonably good policy, exploration is a good thing. And yes, a pit would be a new, distinct environmental state. But please note that a pit could be similar to other, similar training situations/states encountered beforehand during training. So, even if a particular state has not explicitly been visited during training, the deep RL agent can still learn how to cope with such a novel state given the generalization ability of the policy-network it employs.  
  > **username_0**: B I kind of get it , if we’re using a neural network, our function approximator would Be able to provide close to accurate Q values of the new state it encountered, regardless of whether it had visited the state during its training  
  
  ---
  
  