Answers.Body: <p>For normal <a href="https://en.wikipedia.org/wiki/Markov_decision_process#Value_iteration" rel="nofollow noreferrer">value iteration</a>, you need to have the model, i.e. the transition probability, denoted by <span class="math-container">$P(s' \mid s,a)$</span>.</p>
  <p>With Q-learning, you use the current reward and the already stored Q value:</p>
  <p><img src="https://wikimedia.org/api/rest_v1/media/math/render/svg/3d58e03dd47844bb627b83e1265163dcfab3961d" alt="Q value update" /></p>
  <p>The relation between the value function <span class="math-container">$V(s)$</span> and the <span class="math-container">$Q$</span> function <span class="math-container">$Q(s, a)$</span> is that the <span class="math-container">$V(s)$</span> function is simply the value of the action <span class="math-container">$a$</span>, such that <span class="math-container">$Q(s, a)$</span> is the highest, that is, <span class="math-container">$V(s) = \max_a Q(s, a)$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2016-11-07T19:00:20.607Z
Answers.LastActivityDate: 2020-09-27T22:50:52.963Z
Answers.LastEditDate: 2020-09-27T22:50:52.963Z
Answers.UserDisplayName: agold
Body: <p>I don't understand why we can't apply <em>value iteration</em> when don't know the reward and transition probabilities. In <a href="https://www.youtube.com/watch?v=ifma8G7LegE&amp;feature=youtu.be&amp;t=3431&amp;ab_channel=CS188Spring2013" rel="nofollow noreferrer">this lecture</a>, the lecturer says it has to do with not being able to take max with samples, but what does this mean?</p>
  <p>Why does Q-learning not need to know the reward and transition functions? In Q-learning, we also have a max, so I am not understanding.</p>
  
Tags: <reinforcement-learning><q-learning><markov-decision-process><value-iteration>
Title: Why can't we apply value iteration when we do not know the reward and transition functions, and how does Q-learning solve this issue?
UserDisplayName: Abhishek Bhatia
ThreadText: # Why can't we apply value iteration when we do not know the reward and transition functions, and how does Q-learning solve this issue?
  
  (Asked by: username_0 on 2016-10-30)
  
  I don't understand why we can't apply *value iteration* when don't know the reward and transition probabilities. In [this lecture](https://www.youtube.com/watch?v=ifma8G7LegE&feature=youtu.be&t=3431&ab_channel=CS188Spring2013), the lecturer says it has to do with not being able to take max with samples, but what does this mean?
  
  Why does Q-learning not need to know the reward and transition functions? In Q-learning, we also have a max, so I am not understanding.
  
  **Tags:** reinforcement-learning, q-learning, markov-decision-process, value-iteration
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2016-11-07. Score: 3
  
  For normal [value iteration](https://en.wikipedia.org/wiki/Markov_decision_process#Value_iteration), you need to have the model, i.e. the transition probability, denoted by $P(s' \mid s,a)$.
  
  With Q-learning, you use the current reward and the already stored Q value:
  
  ![Q value update](https://wikimedia.org/api/rest_v1/media/math/render/svg/3d58e03dd47844bb627b83e1265163dcfab3961d)
  
  The relation between the value function $V(s)$ and the $Q$ function $Q(s, a)$ is that the $V(s)$ function is simply the value of the action $a$, such that $Q(s, a)$ is the highest, that is, $V(s) = \max\_a Q(s, a)$.
  
  
  ---
  
  