Answers.Body: <blockquote>
  <p>Why can’t we during the first 1000 episodes allow our agent perform only exploration</p>
  </blockquote>
  <p>You can do this. It is fine to do so either to learn the value function of a simple random policy, or when performing off-policy updates. It is quite normal when learning an environment from scratch in a safe way - e.g. in simulation - to collect an initial set of data from behaving completely randomly. The amount of this random data varies, and usually the agent will not switch from fully random to fully deterministic based on calculated values as a single step, but will do so gradually.</p>
  <blockquote>
  <p>this will give a better chance of covering the entire space state</p>
  </blockquote>
  <p>That will depend on the nature of the problem. For really simple problems, it may explore the space sufficiently to learn from. However, for many problems it is just a starting point, and not sufficient to cover parts of the space that are of interest in optimal control.</p>
  <p>When behaving completely randomly, the agent may take a very long time to complete an episode, and may never complete its first episode. So you could be waiting for a long time to collect data for the first 1000 such episodes. An example of this sort of environment would be a large maze - the agent will move back and forth in the maze, revisiting same parts again and again, where in theory it could already be learning not to repeat its mistakes.</p>
  <p>In some environments, behaving completely randomly will result in early failure, and never experiencing postive rewards that are available in the environment. An example of this might be a robot learning to balance on a tightrope and get from one end to the other. It would fall off after a few random actions, gaining very little knowledge for 1000 episodes.</p>
  <p>The state space coverage you are looking for ideally should include the optimal path through the space - at least at some point during learning (not necessarily the start). This does not have to appear in one single perfect episode, because the update rules for value functions in reinforcement learning (RL) will eventually allocate the correct values and find this optimal path in the data. However, the collected data does need to include the information about this optimal path amongst all the alternatives so that the methods in RL can evaluate and select it. In simple environments acting randomly may be enough to gain this data, but becomes highly unlikely when the environments are more complex.</p>
  <blockquote>
  <p>then after the number of episodes, we can then decide to exploit</p>
  </blockquote>
  <p>Again this might work for very simple environments, where you have collected enough information through acting randomly to construct a useful value function. However, if acting randomly does not find enough of the optimal path, then the best that exploitation can do is find some local optimum based on the data that was collected.</p>
  <p>I suggest you experience this difference for yourself: Set up a toy example environment, and use it to compare different approaches for moving between pure exploration and pure exploitation. You will want to run many experiments (probably 100s for each combination, averaged) to smooth out the randomness, and you can plot the results to see how well each approach learns - e.g. how many time steps (count time steps, not episodes, if you are interested in sample efficiency) it takes for the agent to learn, and whether or not it actually finds the correct optimal behaviour. Bear in mind that the specific results will only apply in your selected environment - so you might also want to do this comparison on a small range of environments.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-29T10:14:48.743Z
Answers.Text: Ohh, using exploration wouldn’t be able to improve much of the optimal path, I get it, so the Q value estimates wouldn’t be that good since your agent wouldn’t land on those states as often as they do during exploitation, am I right
Answers.UserDisplayName: Chukwudi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-29T10:17:40.963Z
Answers.Text: @Chukwudi: Yes. But as usual this depends heavily on the nature of the environment. I have successfully trained an RL agent to play Tic Tac Toe 100% optimally, learning purely from random moves (for 50,000 episodes), then exploiting the value table that it built. However, the same approach does not work well at all for Connect 4. The way you word your questions, it seems you are looking for hard rules why things should work or not, but RL covers such a large range of problems, that this is not really possible,
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-29T10:20:50.010Z
Answers.Text: Oh I get it now, it’s not a question that can be given a concrete answer, it all depends on the type of environment, what works for one, might not work on the other
Answers.UserDisplayName: Chukwudi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-29T10:22:45.693Z
Answers.Text: @Chukwudi: Yes. There are lots of questions like that in RL. Often when someone here has said "that won't work" to one of your questions, and gives an example why, they mean "that won't work **in general**, and here's a counter-example".
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-29T10:26:30.493Z
Answers.Text: So the safest way is to gradually decay our exploration rate
Answers.UserDisplayName: Chukwudi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-29T11:50:14.217Z
Answers.Text: Neil, the same user recently asked [this question](https://ai.stackexchange.com/q/22194/2444). Isn't this just a duplicate?
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-29T11:57:11.223Z
Answers.Text: @nbro: Yes it is. Sorry I missed that. The answers there are OK, but given accpeted answer is here, perhaps the other could be closed as duplicate of this?
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-29T19:50:06.663Z
Answers.Text: Close the other one, these answers are more suited to my question
Answers.UserDisplayName: Chukwudi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-29T07:53:26.070Z
Answers.LastActivityDate: 2020-06-29T08:13:05.090Z
Answers.LastEditDate: 2020-06-29T08:13:05.090Z
Answers.UserDisplayName: Neil Slater
Body: <p>Why can't we during the first 1000 episodes allow our agent to perform only exploration?</p>
  <p>This will give a better chance of covering the entire space state. Then, after the number of episodes, we can then decide to exploit.</p>
  
Tags: <reinforcement-learning><exploration-exploitation-tradeoff>
Title: Why is 100% exploration bad during the learning stage in reinforcement learning?
UserDisplayName: Chukwudi
ThreadText: # Why is 100% exploration bad during the learning stage in reinforcement learning?
  
  (Asked by: username_0 on 2020-06-29)
  
  Why can't we during the first 1000 episodes allow our agent to perform only exploration?
  
  This will give a better chance of covering the entire space state. Then, after the number of episodes, we can then decide to exploit.
  
  **Tags:** reinforcement-learning, exploration-exploitation-tradeoff
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-06-29. Score: 3 | Accepted answer
  
  > Why can’t we during the first 1000 episodes allow our agent perform only exploration
  
  You can do this. It is fine to do so either to learn the value function of a simple random policy, or when performing off-policy updates. It is quite normal when learning an environment from scratch in a safe way - e.g. in simulation - to collect an initial set of data from behaving completely randomly. The amount of this random data varies, and usually the agent will not switch from fully random to fully deterministic based on calculated values as a single step, but will do so gradually.
  
  > this will give a better chance of covering the entire space state
  
  That will depend on the nature of the problem. For really simple problems, it may explore the space sufficiently to learn from. However, for many problems it is just a starting point, and not sufficient to cover parts of the space that are of interest in optimal control.
  
  When behaving completely randomly, the agent may take a very long time to complete an episode, and may never complete its first episode. So you could be waiting for a long time to collect data for the first 1000 such episodes. An example of this sort of environment would be a large maze - the agent will move back and forth in the maze, revisiting same parts again and again, where in theory it could already be learning not to repeat its mistakes.
  
  In some environments, behaving completely randomly will result in early failure, and never experiencing postive rewards that are available in the environment. An example of this might be a robot learning to balance on a tightrope and get from one end to the other. It would fall off after a few random actions, gaining very little knowledge for 1000 episodes.
  
  The state space coverage you are looking for ideally should include the optimal path through the space - at least at some point during learning (not necessarily the start). This does not have to appear in one single perfect episode, because the update rules for value functions in reinforcement learning (RL) will eventually allocate the correct values and find this optimal path in the data. However, the collected data does need to include the information about this optimal path amongst all the alternatives so that the methods in RL can evaluate and select it. In simple environments acting randomly may be enough to gain this data, but becomes highly unlikely when the environments are more complex.
  
  > then after the number of episodes, we can then decide to exploit
  
  Again this might work for very simple environments, where you have collected enough information through acting randomly to construct a useful value function. However, if acting randomly does not find enough of the optimal path, then the best that exploitation can do is find some local optimum based on the data that was collected.
  
  I suggest you experience this difference for yourself: Set up a toy example environment, and use it to compare different approaches for moving between pure exploration and pure exploitation. You will want to run many experiments (probably 100s for each combination, averaged) to smooth out the randomness, and you can plot the results to see how well each approach learns - e.g. how many time steps (count time steps, not episodes, if you are interested in sample efficiency) it takes for the agent to learn, and whether or not it actually finds the correct optimal behaviour. Bear in mind that the specific results will only apply in your selected environment - so you might also want to do this comparison on a small range of environments.
  
  > **Comments:**
  > **username_0**: Ohh, using exploration wouldn’t be able to improve much of the optimal path, I get it, so the Q value estimates wouldn’t be that good since your agent wouldn’t land on those states as often as they do during exploitation, am I right  
  > **username_1**: @username_0: Yes. But as usual this depends heavily on the nature of the environment. I have successfully trained an RL agent to play Tic Tac Toe 100% optimally, learning purely from random moves (for 50,000 episodes), then exploiting the value table that it built. However, the same approach does not work well at all for Connect 4. The way you word your questions, it seems you are looking for hard rules why things should work or not, but RL covers such a large range of problems, that this is not really possible,  
  > **username_0**: Oh I get it now, it’s not a question that can be given a concrete answer, it all depends on the type of environment, what works for one, might not work on the other  
  > **username_1**: @username_0: Yes. There are lots of questions like that in RL. Often when someone here has said "that won't work" to one of your questions, and gives an example why, they mean "that won't work **in general**, and here's a counter-example".  
  > **username_0**: So the safest way is to gradually decay our exploration rate  
  > **username_2**: username_1, the same user recently asked [this question](https://ai.stackexchange.com/q/22194/2444). Isn't this just a duplicate?  
  > **username_1**: @username_2: Yes it is. Sorry I missed that. The answers there are OK, but given accpeted answer is here, perhaps the other could be closed as duplicate of this?  
  > **username_0**: Close the other one, these answers are more suited to my question  
  
  ---
  
  