Answers.Body: <blockquote>
    <p>How does Q learning handle this? Is the Q function only used during the training process, where the future states are known? And is the Q function still used afterwards, if that is the case?</p>
  </blockquote>
  
  <p>The learned <span class="math-container">$Q$</span>-function is not only used during training, but also after training (in what we may call "deployment", when we expect a trained agent to behave according to what it has learned).</p>
  
  <p>However, the reliance on future states is only there during training, it is no longer required for deployment.</p>
  
  <p>During training, we use the following <span class="math-container">$Q$</span>-learning update rule:</p>
  
  <p><span class="math-container">$$Q(s, a) \gets (1 - \alpha) Q(s, a) + \alpha \left( R + \gamma \color{red}{\max_{a'} Q(s', a')} \right),$$</span></p>
  
  <p>where <span class="math-container">$s'$</span> is the state we reach after executing <span class="math-container">$a$</span> in <span class="math-container">$s$</span>. Here, the <span class="math-container">$\color{red}{\text{red}}$</span> part is the part where we rely on knowledge of the future <span class="math-container">$s'$</span>. This is available in training because we can simply pick <span class="math-container">$a$</span>, execute it in <span class="math-container">$s$</span>, observe <span class="math-container">$s'$</span>, and only then trigger our update step.</p>
  
  <hr>
  
  <p>Outside of training (and actually also during training), we also rely on our <span class="math-container">$Q(s, \cdot)$</span> function for the selection of actions. We typically select an action <span class="math-container">$a$</span> according to <span class="math-container">$a = \arg\max_a Q(s, a)$</span>; we select the action <span class="math-container">$a$</span> that maximises <span class="math-container">$Q(s, a)$</span> in our current state <span class="math-container">$s$</span>. <strong>The important thing to note here is that there is no <span class="math-container">$s'$</span> term in this description of how we select actions: we do not require knowledge of our future state</strong>.</p>
  
  <hr>
  
  <p><strong>Note:</strong> in my answer I decided to answer the question literally as it is written, i.e. I'm explaining how <span class="math-container">$Q$</span>-learning can still work in the described setting where actions <span class="math-container">$a$</span> have no influence whatsoever on the future state reached. </p>
  
  <p>In practice, I would never recommend actually using <span class="math-container">$Q$</span>-learning in such a setting, and instead refer to Neil Slater's comment about Contextual Multi-Armed Bandit algorithms likely providing a better solution.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-07T16:23:26.717Z
Answers.Text: What if the action a has influence on the future state but in a nondeterministic way? (eg applying q learning to tic tac toe, the opponent action is non deterministic)
Answers.UserDisplayName: Umberto Fontanazza
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-07T16:50:03.343Z
Answers.Text: If the opponent is stationary (does not change its strategy / is not learning itself), then this is still fine, because the opponent can simply be viewed as being "a part of" a stochastic environment, rather than an actual intelligent agent. And stochastic environments are fine, the assumption that the environment may be stochastic is very standard in MDPs. $Q$-learning can deal with that. If the opponent is itself also a learning agent which changes while you train, both agents can get stuck cycling through suboptimal strategies with algorithms like $Q$-learning.
Answers.UserDisplayName: Dennis Soemers
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-14T10:52:00.907Z
Answers.LastActivityDate: 2019-07-14T10:52:00.907Z
Answers.UserDisplayName: Dennis Soemers
Body: <p>The Q function uses the (current and future) states to determine the action that gets the highest reward. </p>
  
  <p>However, in a stochastic environment, the current action (at the current state) does not determine the next state.</p>
  
  <p>How does Q learning handle this? Is the Q function only used during the training process, where the future states are known? And is the Q function still used afterwards, if that is the case?</p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-03-31T12:20:17.863Z
Comments.Text: Could you clarify: "In a stochastic environment, where the action doesn't influence state" - do you mean that the state is not 100% predictable, and the action doesn't *fully determine* the state? E.g. deciding to move forward, and a game then moves you 1d6 steps forward. Or do you really mean that the action choice has absolutely no influence on the resulting state - and the state will evolve by rules independent of the action choice? E.g. deciding to buy or sell, getting some reward, then the state just changes randomly. The answer you need is very different depending on which case you mean.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-04-02T19:00:31.020Z
Comments.Text: @NeilSlater, Thanks for your response. In this case the action has no influence at all on the next state. The state will evolve by rules independent of the action choice.
Comments.UserDisplayName: redlum
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-04-02T19:11:02.360Z
Comments.Text: In which case, although Q learning should work without errors, you would be expending resources for the agent to learn the fact that the action does not influence state. Your situation is more commonly known as "contextual bandit" and there a variety of solvers out there. The positive news is that good old supervised learning should be fine to learn from the historical data. Then it's just a matter of how online your algorithm needs to be, and how risk averse whilst making online decisions.
Comments.UserDisplayName: Neil Slater
Tags: <reinforcement-learning><q-learning><environment>
Title: How does Q-learning work in stochastic environments?
UserDisplayName: redlum
ThreadText: # How does Q-learning work in stochastic environments?
  
  (Asked by: username_0 on 2018-03-29)
  
  The Q function uses the (current and future) states to determine the action that gets the highest reward.
  
  However, in a stochastic environment, the current action (at the current state) does not determine the next state.
  
  How does Q learning handle this? Is the Q function only used during the training process, where the future states are known? And is the Q function still used afterwards, if that is the case?
  
  **Tags:** reinforcement-learning, q-learning, environment
  > **Comments:**
  > **username_1**: Could you clarify: "In a stochastic environment, where the action doesn't influence state" - do you mean that the state is not 100% predictable, and the action doesn't *fully determine* the state? E.g. deciding to move forward, and a game then moves you 1d6 steps forward. Or do you really mean that the action choice has absolutely no influence on the resulting state - and the state will evolve by rules independent of the action choice? E.g. deciding to buy or sell, getting some reward, then the state just changes randomly. The answer you need is very different depending on which case you mean.  
  > **username_0**: @NeilSlater, Thanks for your response. In this case the action has no influence at all on the next state. The state will evolve by rules independent of the action choice.  
  > **username_1**: In which case, although Q learning should work without errors, you would be expending resources for the agent to learn the fact that the action does not influence state. Your situation is more commonly known as "contextual bandit" and there a variety of solvers out there. The positive news is that good old supervised learning should be fine to learn from the historical data. Then it's just a matter of how online your algorithm needs to be, and how risk averse whilst making online decisions.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2019-07-14. Score: 7
  
  > How does Q learning handle this? Is the Q function only used during the training process, where the future states are known? And is the Q function still used afterwards, if that is the case?
  
  The learned $Q$-function is not only used during training, but also after training (in what we may call "deployment", when we expect a trained agent to behave according to what it has learned).
  
  However, the reliance on future states is only there during training, it is no longer required for deployment.
  
  During training, we use the following $Q$-learning update rule:
  
  $$Q(s, a) \gets (1 - \alpha) Q(s, a) + \alpha \left( R + \gamma \color{red}{\max\_{a'} Q(s', a')} \right),$$
  
  where $s'$ is the state we reach after executing $a$ in $s$. Here, the $\color{red}{\text{red}}$ part is the part where we rely on knowledge of the future $s'$. This is available in training because we can simply pick $a$, execute it in $s$, observe $s'$, and only then trigger our update step.
  
  ---
  
  Outside of training (and actually also during training), we also rely on our $Q(s, \cdot)$ function for the selection of actions. We typically select an action $a$ according to $a = \arg\max\_a Q(s, a)$; we select the action $a$ that maximises $Q(s, a)$ in our current state $s$. **The important thing to note here is that there is no $s'$ term in this description of how we select actions: we do not require knowledge of our future state**.
  
  ---
  
  **Note:** in my answer I decided to answer the question literally as it is written, i.e. I'm explaining how $Q$-learning can still work in the described setting where actions $a$ have no influence whatsoever on the future state reached.
  
  In practice, I would never recommend actually using $Q$-learning in such a setting, and instead refer to Neil Slater's comment about Contextual Multi-Armed Bandit algorithms likely providing a better solution.
  
  > **Comments:**
  > **username_3**: What if the action a has influence on the future state but in a nondeterministic way? (eg applying q learning to tic tac toe, the opponent action is non deterministic)  
  > **username_2**: If the opponent is stationary (does not change its strategy / is not learning itself), then this is still fine, because the opponent can simply be viewed as being "a part of" a stochastic environment, rather than an actual intelligent agent. And stochastic environments are fine, the assumption that the environment may be stochastic is very standard in MDPs. $Q$-learning can deal with that. If the opponent is itself also a learning agent which changes while you train, both agents can get stuck cycling through suboptimal strategies with algorithms like $Q$-learning.  
  
  ---
  
  