Answers.Body: <p>No, your second statement does not correctly implement the Q-learning update rule, which the first statement correctly implements.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-18T14:31:30.960Z
Answers.LastActivityDate: 2019-02-18T14:31:30.960Z
Answers.UserDisplayName: nbro
Answers.Body: <p>Your second code snippet is equivalent to this:</p>
  
  <p><span class="math-container">$$Q_{k+1}(s,a)  \leftarrow r + \alpha \text{max}_{a'} Q_k(s', a')$$</span></p>
  
  <p>This looks like a simplified Value Iteration update to me, where you have incorrectly switched <span class="math-container">$\alpha$</span> (the learning rate) for <span class="math-container">$\gamma$</span> (the discount rate).</p>
  
  <p>The full Value Iteration update based on action values looks like this:</p>
  
  <p><span class="math-container">$$Q_{k+1}(s,a)  \leftarrow \sum_{r,s'} p(r,s'|s,a)(r + \gamma \text{max}_{a'} Q_k(s', a'))$$</span></p>
  
  <p>This is almost the same as your equation when you have a deterministic environment (so you can directly predict single values <span class="math-container">$r$</span> and <span class="math-container">$s'$</span> from <span class="math-container">$s, a$</span>)</p>
  
  <p>As such, it will sort of work with certain assumptions:</p>
  
  <ul>
  <li><p>You want a specific discount rate, or don't particularly care about predicting values, just finding a close-to-optimal policy</p></li>
  <li><p>The environment is deterministic</p></li>
  </ul>
  
  <p>The further away you are from those assumptions, the worse fit the simpler update method will be to your problem. It is definitely not Q-learning either way.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-18T15:48:24.053Z
Answers.Text: Oh, ok. Not sure why they use this formula in the example though. Either way, don't understand why I get good results using this formula but the actually correct one doesn't work! Guess I'll have to keep trying to find out some more, thanks!
Answers.UserDisplayName: Sergio
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-18T15:56:26.103Z
Answers.Text: @Sergio Maybe you're not using the correct variables when indexing or somewhere else. It would have been better to ask a specific question with your code (maybe on [Stack Overflow](https://stackoverflow.com/questions/ask)).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-18T16:05:40.037Z
Answers.Text: @nbro and Neil: I was using a discount factor of 1 the whole time, is that bad? I just changed it to 0.1 and now I'm getting very good results with the actual Q-Learning formula. Guess I was missing that detail.
Answers.UserDisplayName: Sergio
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-18T16:10:42.747Z
Answers.Text: @Sergio A discount factor of 1 means that you're valuing the future rewards in the same way that you're valuing the immediate reward (the one you received after having taken the action at the current step). You often don't want this behaviour, in general, because, intuitively, the future is uncertain, so you want to give less weight to those future rewards. See also: https://stats.stackexchange.com/q/221402/82135.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-18T20:50:32.960Z
Answers.Text: @Sergio A discount factor of $0.1$ seems awefully low though. Much more common choices are like... $\gamma = 0.99$ or $\gamma = 0.9$.
Answers.UserDisplayName: Dennis Soemers
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-18T22:45:34.910Z
Answers.Text: I think he's mixing up discount factor with step size parameter
Answers.UserDisplayName: Brale
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-20T00:47:39.690Z
Answers.Text: @DennisSoemers Apologies for the late answer. I tried 0.9 and it works well too. I'm new to Reinforcement Learning, so I'm still trying to get to grips with it! Thanks for the help, appreciate it. Is 0.1 _wrong_ though, or just an uncommon choice? It just means I'm giving less emphasis to future rewards so, for example, if it returns better results, would it not be better to use that?
Answers.UserDisplayName: Sergio
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-20T09:05:36.480Z
Answers.Text: @Sergio It's not necessarily *wrong*... in theory, anything that is between $0$ and $1$ is fine. It's so low though that you get very close to only caring about immediate one-step rewards, and almost ignore any future rewards (even if they're very close in the future, only a couple of steps away). But yeah sure, if it works, it works.
Answers.UserDisplayName: Dennis Soemers
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-18T14:32:33.270Z
Answers.LastActivityDate: 2019-02-18T14:32:33.270Z
Answers.UserDisplayName: Neil Slater
Body: <p>I asked a question a while ago here and since then I've been solving the issues within my code but I have just one question... This is the formula for updating the Q-Matrix in Q-Learning:</p>
  
  <p><span class="math-container">$$Q(s_t, a_t) = Q(s_t, a_t) + \alpha \times (R+Q(s_{t+1}, max_a)-Q(s_t, a_t))$$</span></p>
  
  <p>However, I saw a Q-Learning example that uses a different formula, which I'm applying to my own problem and I'm getting good results:</p>
  
  <p><span class="math-container">$$Q(s_t, a_t) = R(s_t,a_t) + \alpha \times Q(s_{t+1}, max_a)$$</span></p>
  
  <p>Is this valid?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-18T15:36:01.053Z
Comments.Text: Where did you see the second implementation?  Perhaps there is a context being missed.
Comments.UserDisplayName: SeeDerekEngineer
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-18T15:55:50.967Z
Comments.Text: @SeeDerekEngineer it was to solve the basic problem where there's 7 nodes and you wanna go from 2 to 5 or something equivalent. Does that make it different?
Comments.UserDisplayName: Sergio
Tags: <reinforcement-learning><python><q-learning>
Title: Is there more than one Q-matrix update formula?
UserDisplayName: Sergio
ThreadText: # Is there more than one Q-matrix update formula?
  
  (Asked by: username_0 on 2019-02-18)
  
  I asked a question a while ago here and since then I've been solving the issues within my code but I have just one question... This is the formula for updating the Q-Matrix in Q-Learning:
  
  $$Q(s\_t, a\_t) = Q(s\_t, a\_t) + \alpha \times (R+Q(s\_{t+1}, max\_a)-Q(s\_t, a\_t))$$
  
  However, I saw a Q-Learning example that uses a different formula, which I'm applying to my own problem and I'm getting good results:
  
  $$Q(s\_t, a\_t) = R(s\_t,a\_t) + \alpha \times Q(s\_{t+1}, max\_a)$$
  
  Is this valid?
  
  **Tags:** reinforcement-learning, python, q-learning
  
  > **Comments:**
  > **username_1**: Where did you see the second implementation?  Perhaps there is a context being missed.  
  > **username_0**: @username_1 it was to solve the basic problem where there's 7 nodes and you wanna go from 2 to 5 or something equivalent. Does that make it different?  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2019-02-18. Score: 3
  
  No, your second statement does not correctly implement the Q-learning update rule, which the first statement correctly implements.
  
  
  ---
  
  ### Answer by username_3 on 2019-02-18. Score: 1
  
  Your second code snippet is equivalent to this:
  
  $$Q\_{k+1}(s,a) \leftarrow r + \alpha \text{max}\_{a'} Q\_k(s', a')$$
  
  This looks like a simplified Value Iteration update to me, where you have incorrectly switched $\alpha$ (the learning rate) for $\gamma$ (the discount rate).
  
  The full Value Iteration update based on action values looks like this:
  
  $$Q\_{k+1}(s,a) \leftarrow \sum\_{r,s'} p(r,s'|s,a)(r + \gamma \text{max}\_{a'} Q\_k(s', a'))$$
  
  This is almost the same as your equation when you have a deterministic environment (so you can directly predict single values $r$ and $s'$ from $s, a$)
  
  As such, it will sort of work with certain assumptions:
  
  * You want a specific discount rate, or don't particularly care about predicting values, just finding a close-to-optimal policy
  * The environment is deterministic
  
  The further away you are from those assumptions, the worse fit the simpler update method will be to your problem. It is definitely not Q-learning either way.
  
  > **Comments:**
  > **username_0**: Oh, ok. Not sure why they use this formula in the example though. Either way, don't understand why I get good results using this formula but the actually correct one doesn't work! Guess I'll have to keep trying to find out some more, thanks!  
  > **username_2**: @username_0 Maybe you're not using the correct variables when indexing or somewhere else. It would have been better to ask a specific question with your code (maybe on [Stack Overflow](https://stackoverflow.com/questions/ask)).  
  > **username_0**: @username_2 and username_3: I was using a discount factor of 1 the whole time, is that bad? I just changed it to 0.1 and now I'm getting very good results with the actual Q-Learning formula. Guess I was missing that detail.  
  > **username_2**: @username_0 A discount factor of 1 means that you're valuing the future rewards in the same way that you're valuing the immediate reward (the one you received after having taken the action at the current step). You often don't want this behaviour, in general, because, intuitively, the future is uncertain, so you want to give less weight to those future rewards. See also: https://stats.stackexchange.com/q/221402/82135.  
  > **username_4**: @username_0 A discount factor of $0.1$ seems awefully low though. Much more common choices are like... $\gamma = 0.99$ or $\gamma = 0.9$.  
  > **username_5**: I think he's mixing up discount factor with step size parameter  
  > **username_0**: @username_4 Apologies for the late answer. I tried 0.9 and it works well too. I'm new to Reinforcement Learning, so I'm still trying to get to grips with it! Thanks for the help, appreciate it. Is 0.1 _wrong_ though, or just an uncommon choice? It just means I'm giving less emphasis to future rewards so, for example, if it returns better results, would it not be better to use that?  
  > **username_4**: @username_0 It's not necessarily *wrong*... in theory, anything that is between $0$ and $1$ is fine. It's so low though that you get very close to only caring about immediate one-step rewards, and almost ignore any future rewards (even if they're very close in the future, only a couple of steps away). But yeah sure, if it works, it works.  
  
  ---
  
  