Answers.Body: <p>Q learning is very similar to value iteration. They are based on the same principles. A key similarity is that both assume a greedy action choice on the bootstrap next state value.</p>
  <p>The big difference is that value iteration is based on processing all states and possible actions whilst knowing the MDP model in detail. For Q learning, the value function is instead learned by sampling from experience, and it does not require knowledge of the MDP model, beyond the state and action representations.</p>
  <p>Q learning typically uses an <span class="math-container">$\epsilon$</span>-greedy policy derived from its current action value function estimate, in order to drive exploration and take samples. Whilst value iteration neither explores nor exploits nor samples, instead resolving based on the model, processing all states and actions. Value iteration can be thought of as a kind of planning algorithm.</p>
  <blockquote>
  <p>Or do I have to perform generalized policy iteration where q learning is for evaluation and some other method for improvement till convergence?</p>
  </blockquote>
  <p>Q learning <em>is</em> already a form of generalised policy iteration (GPI). GPI is the underlying principle behind value-based control methods. You can think of policy iteration (and value iteration) as the &quot;standard&quot; for accuracy, but it doesn't scale well or apply directly when the model is not known. GPI is an abstract view of how learning values and then basing policies on greedily selecting on those values should eventually converge on an optimal policy.</p>
  <p>The difference between using off policy TD learning for evaluation or control is in how you select policies. For evaluation you would have a fixed target policy used to calculate bootstrap values (for TD error or TD target). For control scenarios, the target policy is always the greedy action choice based on current value estimates, which is a policy that changes over time. Q learning is exclusively a TD control method. It is easy to adapt it to create an evaluation method, but AFAIK that would not be called Q learning.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-29T15:29:51.507Z
Answers.LastActivityDate: 2023-11-30T08:36:31.690Z
Answers.LastEditDate: 2023-11-30T08:36:31.690Z
Answers.UserDisplayName: Neil Slater
Body: <p><a href="https://i.sstatic.net/QnqNM.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/QnqNM.png" alt="Value iteration" /></a></p>
  <p>This is the explanation of value iteration in our notes where you keep applying bellman optimality equation till it stops changing and then acting greedily wrt the value function gives the optimal policy.</p>
  <p>Is using Q learning for control the same for finding optimal policy? Or do I have to perform generalized policy iteration where q learning is for evaluation and some other method for improvement till convergence?</p>
  
Tags: <reinforcement-learning><comparison><q-learning><value-iteration>
Title: What are the similarities between Q-learning and Value Iteration?
UserDisplayName: ace239
ThreadText: # What are the similarities between Q-learning and Value Iteration?
  
  (Asked by: username_0 on 2023-11-29)
  
  [![Value iteration](https://i.sstatic.net/QnqNM.png)](https://i.sstatic.net/QnqNM.png)
  
  This is the explanation of value iteration in our notes where you keep applying bellman optimality equation till it stops changing and then acting greedily wrt the value function gives the optimal policy.
  
  Is using Q learning for control the same for finding optimal policy? Or do I have to perform generalized policy iteration where q learning is for evaluation and some other method for improvement till convergence?
  
  **Tags:** reinforcement-learning, comparison, q-learning, value-iteration
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-11-29. Score: 3
  
  Q learning is very similar to value iteration. They are based on the same principles. A key similarity is that both assume a greedy action choice on the bootstrap next state value.
  
  The big difference is that value iteration is based on processing all states and possible actions whilst knowing the MDP model in detail. For Q learning, the value function is instead learned by sampling from experience, and it does not require knowledge of the MDP model, beyond the state and action representations.
  
  Q learning typically uses an $\epsilon$-greedy policy derived from its current action value function estimate, in order to drive exploration and take samples. Whilst value iteration neither explores nor exploits nor samples, instead resolving based on the model, processing all states and actions. Value iteration can be thought of as a kind of planning algorithm.
  
  > Or do I have to perform generalized policy iteration where q learning is for evaluation and some other method for improvement till convergence?
  
  Q learning *is* already a form of generalised policy iteration (GPI). GPI is the underlying principle behind value-based control methods. You can think of policy iteration (and value iteration) as the "standard" for accuracy, but it doesn't scale well or apply directly when the model is not known. GPI is an abstract view of how learning values and then basing policies on greedily selecting on those values should eventually converge on an optimal policy.
  
  The difference between using off policy TD learning for evaluation or control is in how you select policies. For evaluation you would have a fixed target policy used to calculate bootstrap values (for TD error or TD target). For control scenarios, the target policy is always the greedy action choice based on current value estimates, which is a policy that changes over time. Q learning is exclusively a TD control method. It is easy to adapt it to create an evaluation method, but AFAIK that would not be called Q learning.
  
  
  ---
  
  