Answers.Body: <p>I believe, that you are correct in saying the sentence is incorrect. Since, as you mention, the epsilon greedy action selection will just allow for the Q values to converge to the optimal q* value since the actions of the policy just provide more observations for Q which will allow for the q* to be converged to. However, this is assuming that the updating of the Q values is in such a way that each new observation has less of an effect on the Q(a). Though it is not the case that the epsilon greedy will converge to the optimal policy/action due to the randomness mentioned by the answer.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-18T15:33:15.557Z
Answers.LastActivityDate: 2022-10-18T15:33:15.557Z
Answers.UserDisplayName: chs
Body: <p>Is the following sentence correct?</p>
  <p>The estimated values <span class="math-container">$Q(a)$</span> do not converge to the true values <span class="math-container">$q_*(a)$</span> because <span class="math-container">$\epsilon$</span>-greedy action selection behaves randomly from time to time.</p>
  <p>My Answer: The sentence is wrong since eventually the estimated value <span class="math-container">$Q(a)$</span> will converge to <span class="math-container">$q_*(a)$</span> and the randomness of epsilon greedy action selection won't be able to influence it if we run it for a long time.
  Is my answer correct?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-10-18T15:23:35.010Z
Comments.Text: @Faizy Thanks but could you please explain why? I don't get why randomness is a problem for the convergence of estimated Q to the true value q.
Comments.UserDisplayName: Gunners 
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-10-18T18:22:34.383Z
Comments.Text: i think you correct.... Running the epsilon greedy algorithm **for a long time** will minimize the influence of randomness on the estimated value $Q(a)$. The epsilon greedy action randomly selects actions with probability epsilon and otherwise acts greedily with respect to the current estimated value of $Q(a)$. **Over time**, the algorithm will tend to converge on the true values of $q_∗(a)$, since the random actions will be less influential.
Comments.UserDisplayName: Faizy
Tags: <epsilon-greedy-policy>
Title: Epsilon-greedy action selection question
UserDisplayName: Gunners 
ThreadText: # Epsilon-greedy action selection question
  
  (Asked by: username_0 on 2022-10-18)
  
  Is the following sentence correct?
  
  The estimated values $Q(a)$ do not converge to the true values $q\_\*(a)$ because $\epsilon$-greedy action selection behaves randomly from time to time.
  
  My Answer: The sentence is wrong since eventually the estimated value $Q(a)$ will converge to $q\_\*(a)$ and the randomness of epsilon greedy action selection won't be able to influence it if we run it for a long time.
  Is my answer correct?
  
  **Tags:** epsilon-greedy-policy
  > **Comments:**
  > **username_0**: @Faizy Thanks but could you please explain why? I don't get why randomness is a problem for the convergence of estimated Q to the true value q.  
  > **username_1**: i think you correct.... Running the epsilon greedy algorithm **for a long time** will minimize the influence of randomness on the estimated value $Q(a)$. The epsilon greedy action randomly selects actions with probability epsilon and otherwise acts greedily with respect to the current estimated value of $Q(a)$. **Over time**, the algorithm will tend to converge on the true values of $q_∗(a)$, since the random actions will be less influential.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-10-18. Score: 1
  
  I believe, that you are correct in saying the sentence is incorrect. Since, as you mention, the epsilon greedy action selection will just allow for the Q values to converge to the optimal q\* value since the actions of the policy just provide more observations for Q which will allow for the q\* to be converged to. However, this is assuming that the updating of the Q values is in such a way that each new observation has less of an effect on the Q(a). Though it is not the case that the epsilon greedy will converge to the optimal policy/action due to the randomness mentioned by the answer.
  
  
  ---
  
  