Answers.Body: <p>You can't guarantee that you have taken every action from every state, even with  1000 time steps. There would be multiple outcomes:</p>
  
  <ol>
  <li><p>The episode terminates, either by success or failure before the 1000
  time steps. The agent is trying to maximise reward, if this is achieved by taking less than 1000 steps then it will do. It won't just walk around until it hits an arbitrary number of time steps.</p></li>
  <li><p>If you have more states than time steps then you will
  never be unable to visit all states and so you cannot guarantee that
  the policy you followed was optimal (and hence would still want to explore). Even if you have #states = #time-steps then you will almost certaintly have more state-action pairs than timesteps. The only time this would be equal is if from every state there is only one action, which would be a trivial problem that wouldn't need RL to solve.</p></li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-16T15:17:30.480Z
Answers.Text: Ok this makes sense, it’s more like since we’re not sure it has taken every action possible from every state , we can’t rely on the Q values as they might not be correct, ok suppose we do 2000 episodes or more, can we now make it 100% exploitation
Answers.UserDisplayName: Chukwudi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-16T15:20:30.167Z
Answers.Text: Our estimated Q values will asymptotically approach the optimal Q values (Q^* in most textbooks) as the number of episodes approaches infinite. The exact number of episodes would be a function of the number of state-action pairs however, so we can not say conclusively if 2000 episodes would be enough.
Answers.UserDisplayName: BlueTurtle
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-16T15:21:56.870Z
Answers.Text: We can decay the exploration rate (epsilon in most books) as the number of episodes increases though so after many thousands of episodes the probability of exploration is  neglible.
Answers.UserDisplayName: BlueTurtle
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-16T15:23:37.183Z
Answers.Text: Oh okay, so everything is basically on saying we’re not sure if 2000 episodes would be enough especially if the state space is large, but we’ve gotten better Q values on state action pairs so we can make y 90% exploitation and 10%exploration
Answers.UserDisplayName: Chukwudi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-16T15:26:14.457Z
Answers.Text: Exactly but now think if you continue until 20,000 episodes, your Q values are becoming more accurate and you have explored more of the state space and know that certain state-action pairs are terrible so exploring them even more is pointless, so you reduce exploration to say 9% and so on.
  
  If you have only been to a state a handful of times (especially if the resulting state is stochastic) you are unsure if it's a bad state - so you keep coming back. However, if you've been to a state 100 times and most of the time it sucked, you would be uninterested in going back.
Answers.UserDisplayName: BlueTurtle
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-16T15:42:07.587Z
Answers.Text: On that makes sense, thank you, so if we’re certain that the Q values have converged and are accurate,can we make it 100% exploitation or is there still a need of exploring
Answers.UserDisplayName: Chukwudi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-16T15:11:20.977Z
Answers.LastActivityDate: 2020-06-16T15:11:20.977Z
Answers.UserDisplayName: BlueTurtle
Answers.Body: <p>BlueTurtle's answer is good, but I'd like to add something.</p>
  <p>Your question realistically has nothing to do with Q Learning, in fact, you can ask the same thing about just about any RL algorithm. In fact, even in multi-armed bandits, you can easily see why your proposed method is suboptimal (please don't interpret this as a criticism, because I think your question is a very natural one). My suggestion to you is to read up on multi-armed bandits since they're much simpler to analyze. I think even the Sutton and Barto book deals with your proposed method explicitly, and mathematically proves that other strategies are better.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-17T16:04:35.247Z
Answers.LastActivityDate: 2020-06-17T16:04:35.247Z
Answers.UserDisplayName: harwiltz
Body: <p>During the first episode, it's 100% exploration, because all our Q values are 0. Suppose we have 1000 time steps, and it's terminated by meeting a reward. So, after the first episode, why can't we make it 100% exploitation? Why do we still need exploration?</p>
  
Tags: <reinforcement-learning><q-learning><exploration-exploitation-tradeoff>
Title: Why can't we fully exploit the environment after the first episode in Q-learning?
UserDisplayName: Chukwudi
ThreadText: # Why can't we fully exploit the environment after the first episode in Q-learning?
  
  (Asked by: username_0 on 2020-06-16)
  
  During the first episode, it's 100% exploration, because all our Q values are 0. Suppose we have 1000 time steps, and it's terminated by meeting a reward. So, after the first episode, why can't we make it 100% exploitation? Why do we still need exploration?
  
  **Tags:** reinforcement-learning, q-learning, exploration-exploitation-tradeoff
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2020-06-16. Score: 1 | Accepted answer
  
  You can't guarantee that you have taken every action from every state, even with 1000 time steps. There would be multiple outcomes:
  
  1. The episode terminates, either by success or failure before the 1000
     time steps. The agent is trying to maximise reward, if this is achieved by taking less than 1000 steps then it will do. It won't just walk around until it hits an arbitrary number of time steps.
  2. If you have more states than time steps then you will
     never be unable to visit all states and so you cannot guarantee that
     the policy you followed was optimal (and hence would still want to explore). Even if you have #states = #time-steps then you will almost certaintly have more state-action pairs than timesteps. The only time this would be equal is if from every state there is only one action, which would be a trivial problem that wouldn't need RL to solve.
  
  > **Comments:**
  > **username_0**: Ok this makes sense, it’s more like since we’re not sure it has taken every action possible from every state , we can’t rely on the Q values as they might not be correct, ok suppose we do 2000 episodes or more, can we now make it 100% exploitation  
  > **username_1**: Our estimated Q values will asymptotically approach the optimal Q values (Q^* in most textbooks) as the number of episodes approaches infinite. The exact number of episodes would be a function of the number of state-action pairs however, so we can not say conclusively if 2000 episodes would be enough.  
  > **username_1**: We can decay the exploration rate (epsilon in most books) as the number of episodes increases though so after many thousands of episodes the probability of exploration is  neglible.  
  > **username_0**: Oh okay, so everything is basically on saying we’re not sure if 2000 episodes would be enough especially if the state space is large, but we’ve gotten better Q values on state action pairs so we can make y 90% exploitation and 10%exploration  
  > **username_1**: Exactly but now think if you continue until 20,000 episodes, your Q values are becoming more accurate and you have explored more of the state space and know that certain state-action pairs are terrible so exploring them even more is pointless, so you reduce exploration to say 9% and so on.  If you have only been to a state a handful of times (especially if the resulting state is stochastic) you are unsure if it's a bad state - so you keep coming back. However, if you've been to a state 100 times and most of the time it sucked, you would be uninterested in going back.  
  > **username_0**: On that makes sense, thank you, so if we’re certain that the Q values have converged and are accurate,can we make it 100% exploitation or is there still a need of exploring  
  
  ---
  
  ### Answer by username_2 on 2020-06-17. Score: 2
  
  BlueTurtle's answer is good, but I'd like to add something.
  
  Your question realistically has nothing to do with Q Learning, in fact, you can ask the same thing about just about any RL algorithm. In fact, even in multi-armed bandits, you can easily see why your proposed method is suboptimal (please don't interpret this as a criticism, because I think your question is a very natural one). My suggestion to you is to read up on multi-armed bandits since they're much simpler to analyze. I think even the Sutton and Barto book deals with your proposed method explicitly, and mathematically proves that other strategies are better.
  
  
  ---
  
  