Answers.Body: <blockquote>
  <p>What does it mean when ϵ=0 and ϵ=1? If ϵ=1, does it mean that the agent explores randomly? If this intuition is right, then it will not learn anything - right? On the other hand, if I set ϵ=0, does this imply that the agent doesn't explore?</p>
  </blockquote>
  <p>You are correct, when  ϵ=1 the agent acts randomly. When  ϵ=0, the agent always takes the current greedy actions. Both of these scenarios are not ideal. Always acting greedily will prevent the agent from exploring possibly better parts of the  state space, and instead the agent may get stuck in a local optimum. And always exploring randomly is obviously not ideal as well. Thus, we need to balance between these two. This is often called the balance between exploration and exploitation.</p>
  <blockquote>
  <p>For a typical problem, what is the recommended value for this parameter?</p>
  </blockquote>
  <p>ϵ is a hyper parameter. It is impossible to know in advance what the ideal value is, and it is highly dependent on the problem at hand. There is no general answer to this question.</p>
  <p>That being said, the most common values that I have seen typically range between 0.01 and 0.1. But I want to stress, there is no ideal value that works for all problems. A typical strategy is to try several values and see which one works best. For more information, you might want to look up hyper parameter tuning.</p>
  <p>Another common practice is slowly decaying epsilon over time (often this is called &quot;annealing&quot; or &quot;simulated annealing&quot;). Depending on the algorithm, decaying epsilon to zero may be a requirement for convergence. In some contexts, an algorithm that decays epsilon over time is called a GLIE algorithm. For example, see <a href="https://ai.stackexchange.com/questions/6486/why-is-glie-monte-carlo-control-an-on-policy-control">this</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-26T13:04:03.283Z
Answers.Text: You say "agent explores randomly.", but this is misleading because "to explore" in RL already implies (usually) that it's randomly, i.e. you take a random action. So, I think that this answer can be made more precise. By the way, you can type epsilon in latex here as `$\epsilon$`. Moreover, there may be cases/problems where $\epsilon$ needs to be higher  or smaller. To be honest, right now, no such an example comes to my mind, but it's likely there are some guidelines. It may also be worthing mentioning schedules to decay the this parameter.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-26T13:49:12.807Z
Answers.Text: @nbro. I will update answer. Regarding your final point, one example that comes to mind is in certain on policy algorithms, epsilon must he decayed to guarantee convergence. Sutton calls this GLIE.
Answers.UserDisplayName: chessprogrammer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-26T13:54:32.600Z
Answers.Text: Regarding guidelines for a higher epsilon: it is easy to construct a toy MDP where a higher epsilon converges faster. One classic example is a K armed bandit with very high variance. But its hard to draw a general guideline as to when it should be higher.
Answers.UserDisplayName: chessprogrammer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-26T19:17:16.770Z
Answers.Text: I wouldn't call annealing of the epsilon "simulated annealing" as "SA" refers to a _specific_ algorithm/meta-heuristic, though it's possible that this term is also used in the context of annealing the epsilon. I don't remember that though.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-26T02:09:05.350Z
Answers.LastActivityDate: 2020-11-26T19:02:48.503Z
Answers.LastEditDate: 2020-11-26T19:02:48.503Z
Answers.UserDisplayName: chessprogrammer
Body: <p>I am trying to understand Reinforcement Learning and already explored different Youtube videos, blog posts, and Wikipedia articles.</p>
  <p>What I don't understand is the impact of <span class="math-container">$\epsilon$</span>. What value should it take? <span class="math-container">$0.5$</span>, <span class="math-container">$0.6$</span>, or <span class="math-container">$0.7$</span>?</p>
  <p>What does it mean when <span class="math-container">$\epsilon = 0$</span> and <span class="math-container">$\epsilon = 1$</span>? If <span class="math-container">$\epsilon = 1$</span>, does it mean that the agent explores randomly? If this intuition is right, then it will not learn anything - right? On the other hand, if I set <span class="math-container">$\epsilon = 0$</span>, does this imply that the agent doesn't explore?</p>
  <p>For a typical problem, what is the recommended value for this parameter?</p>
  
Tags: <reinforcement-learning><q-learning><hyperparameter-optimization><hyper-parameters><epsilon-greedy-policy>
Title: What should the value of epsilon be in the Q-learning?
UserDisplayName: Exploring
ThreadText: # What should the value of epsilon be in the Q-learning?
  
  (Asked by: username_0 on 2020-11-26)
  
  I am trying to understand Reinforcement Learning and already explored different Youtube videos, blog posts, and Wikipedia articles.
  
  What I don't understand is the impact of $\epsilon$. What value should it take? $0.5$, $0.6$, or $0.7$?
  
  What does it mean when $\epsilon = 0$ and $\epsilon = 1$? If $\epsilon = 1$, does it mean that the agent explores randomly? If this intuition is right, then it will not learn anything - right? On the other hand, if I set $\epsilon = 0$, does this imply that the agent doesn't explore?
  
  For a typical problem, what is the recommended value for this parameter?
  
  **Tags:** reinforcement-learning, q-learning, hyperparameter-optimization, hyper-parameters, epsilon-greedy-policy
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-11-26. Score: 2
  
  > What does it mean when ϵ=0 and ϵ=1? If ϵ=1, does it mean that the agent explores randomly? If this intuition is right, then it will not learn anything - right? On the other hand, if I set ϵ=0, does this imply that the agent doesn't explore?
  
  You are correct, when ϵ=1 the agent acts randomly. When ϵ=0, the agent always takes the current greedy actions. Both of these scenarios are not ideal. Always acting greedily will prevent the agent from username_0 possibly better parts of the state space, and instead the agent may get stuck in a local optimum. And always username_0 randomly is obviously not ideal as well. Thus, we need to balance between these two. This is often called the balance between exploration and exploitation.
  
  > For a typical problem, what is the recommended value for this parameter?
  
  ϵ is a hyper parameter. It is impossible to know in advance what the ideal value is, and it is highly dependent on the problem at hand. There is no general answer to this question.
  
  That being said, the most common values that I have seen typically range between 0.01 and 0.1. But I want to stress, there is no ideal value that works for all problems. A typical strategy is to try several values and see which one works best. For more information, you might want to look up hyper parameter tuning.
  
  Another common practice is slowly decaying epsilon over time (often this is called "annealing" or "simulated annealing"). Depending on the algorithm, decaying epsilon to zero may be a requirement for convergence. In some contexts, an algorithm that decays epsilon over time is called a GLIE algorithm. For example, see [this](https://ai.stackexchange.com/questions/6486/why-is-glie-monte-carlo-control-an-on-policy-control).
  
  > **Comments:**
  > **username_2**: You say "agent explores randomly.", but this is misleading because "to explore" in RL already implies (usually) that it's randomly, i.e. you take a random action. So, I think that this answer can be made more precise. By the way, you can type epsilon in latex here as `$\epsilon$`. Moreover, there may be cases/problems where $\epsilon$ needs to be higher  or smaller. To be honest, right now, no such an example comes to my mind, but it's likely there are some guidelines. It may also be worthing mentioning schedules to decay the this parameter.  
  > **username_1**: @username_2. I will update answer. Regarding your final point, one example that comes to mind is in certain on policy algorithms, epsilon must he decayed to guarantee convergence. Sutton calls this GLIE.  
  > **username_1**: Regarding guidelines for a higher epsilon: it is easy to construct a toy MDP where a higher epsilon converges faster. One classic example is a K armed bandit with very high variance. But its hard to draw a general guideline as to when it should be higher.  
  > **username_2**: I wouldn't call annealing of the epsilon "simulated annealing" as "SA" refers to a _specific_ algorithm/meta-heuristic, though it's possible that this term is also used in the context of annealing the epsilon. I don't remember that though.  
  
  ---
  
  