Answers.Body: <ol>
  <li><p>How much the <span class="math-container">$Q$</span>-values change does not depend on the value of <span class="math-container">$\epsilon$</span>, rather the value of <span class="math-container">$\epsilon$</span> dictates how likely you are to take a random action and thus take an action that could give rise to a large TD error -- that is a large difference between the returns you expected from taking this action as to what you actually observed. How much the <span class="math-container">$Q$</span>-value changes depends on the magnitude of this TD error.</p>
  </li>
  <li><p><span class="math-container">$Q$</span>-learning is not guaranteed to converge if there is no exploration. Part of the convergence criteria assumes that each state-action pair will be visited infinitely often in an infinite number of episodes, and so if there is no exploration then this will not happen.</p>
  </li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-01T23:52:32.390Z
Answers.LastActivityDate: 2020-10-01T23:52:32.390Z
Answers.UserDisplayName: David
Body: <p>I'm new to reinforcement learning.</p>
  <p>As it is common in RL, <span class="math-container">$\epsilon$</span>-greedy search for the behavior/exploration is used. So, at the beginning of the training, <span class="math-container">$\epsilon$</span> is high, and therefore a lot of random actions are chosen. With time, <span class="math-container">$\epsilon$</span> decreases and we often choose the best action.</p>
  <ol>
  <li><p>I was wondering, e.g. in Q-Learning, if <span class="math-container">$\epsilon$</span> is small, e.g. 0.1 or 0.01, do the Q-values really still change? Do they just change their direction, i.e. the best action remains the best action but the Q-values diverge further, or do the values really change again so that the best action always changes for a given state?</p>
  </li>
  <li><p>If the Q-values really do still change strongly, is it because of the remaining random actions, which we still have at <span class="math-container">$\epsilon&gt;0$</span> or would it still change at <span class="math-container">$\epsilon=0$</span>?</p>
  </li>
  </ol>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-02T10:37:02.643Z
Comments.Text: The question in the title is different than the questions in the body, although they are related. I suggest that you connect the title's question with the body's questions also in the body, so that we understand what your main concern/question is. For example, the current only answer does not address the question in the body, and that's probably because there's no connection in the body.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-02T11:09:53.513Z
Comments.Text: @nbro yes, exactly why I didn't answer the title of the question. If OP confirms whether his title is his actual question then I can answer that too.
Comments.UserDisplayName: David
Tags: <reinforcement-learning><q-learning><value-functions><exploration-exploitation-tradeoff><epsilon-greedy-policy>
Title: Can we stop training as soon as epsilon is small?
UserDisplayName: Micha Christ
ThreadText: # Can we stop training as soon as epsilon is small?
  
  (Asked by: username_0 on 2020-10-01)
  
  I'm new to reinforcement learning.
  
  As it is common in RL, $\epsilon$-greedy search for the behavior/exploration is used. So, at the beginning of the training, $\epsilon$ is high, and therefore a lot of random actions are chosen. With time, $\epsilon$ decreases and we often choose the best action.
  
  1. I was wondering, e.g. in Q-Learning, if $\epsilon$ is small, e.g. 0.1 or 0.01, do the Q-values really still change? Do they just change their direction, i.e. the best action remains the best action but the Q-values diverge further, or do the values really change again so that the best action always changes for a given state?
  2. If the Q-values really do still change strongly, is it because of the remaining random actions, which we still have at $\epsilon>0$ or would it still change at $\epsilon=0$?
  
  **Tags:** reinforcement-learning, q-learning, value-functions, exploration-exploitation-tradeoff, epsilon-greedy-policy
  
  > **Comments:**
  > **username_1**: The question in the title is different than the questions in the body, although they are related. I suggest that you connect the title's question with the body's questions also in the body, so that we understand what your main concern/question is. For example, the current only answer does not address the question in the body, and that's probably because there's no connection in the body.  
  > **username_2**: @username_1 yes, exactly why I didn't answer the title of the question. If OP confirms whether his title is his actual question then I can answer that too.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-10-01. Score: 2
  
  1. How much the $Q$-values change does not depend on the value of $\epsilon$, rather the value of $\epsilon$ dictates how likely you are to take a random action and thus take an action that could give rise to a large TD error -- that is a large difference between the returns you expected from taking this action as to what you actually observed. How much the $Q$-value changes depends on the magnitude of this TD error.
  2. $Q$-learning is not guaranteed to converge if there is no exploration. Part of the convergence criteria assumes that each state-action pair will be visited infinitely often in an infinite number of episodes, and so if there is no exploration then this will not happen.
  
  
  ---
  
  