Answers.Body: <blockquote>
  <p>Let's say <span class="math-container">$S$</span> denotes the stock price, does it mean we are using future information <span class="math-container">$S_{t+1}$</span> in the Q-learning process?</p>
  </blockquote>
  <p>It means you are using the information from a time step <span class="math-container">$t+1$</span> to update the value you associate with a state or state action pair seen on a previous time step <span class="math-container">$t$</span>. When training, this is <em>not</em> future data, it is past data, and there does not need to be any special consideration of the value of <span class="math-container">$t$</span>. It might be something that has just been observed, or could even be from other experience from years ago. The important thing is that you must know the state and action from time <span class="math-container">$t$</span>, and the reward and next state at time <span class="math-container">$t+1$</span> (in general, you <em>might</em> take longer trajectories up to some time <span class="math-container">$t+n$</span>).</p>
  <p>The term <em>backup</em> is used to describe the process of using information from a range of time steps to improve estimates made in earlier ones, and it is a key part of how all reinforcement learning (RL) methods work.</p>
  <blockquote>
  <p>After the training process and bringing into implementation, how is it possible to apply the strategy without knowing the future price</p>
  </blockquote>
  <p>In production, you make use of the estimates that you refined during training by referring to the learned value function.</p>
  <p>In your example, if you are in a production system and are at time step <span class="math-container">$t$</span>, then you know the current state <span class="math-container">$S_t$</span>. Your policy will be:</p>
  <p><span class="math-container">$$\pi(S_t) = \text{argmax}_a Q(S_t, a)$$</span></p>
  <p>This is the <em>greedy</em> policy, and doesn't use any future information, but it does rely on a training process that was allowed to &quot;look ahead&quot; to learn about results of taking different actions.</p>
  <p>It is common to use a variation of this decision rule during training too, because at the point of making a decision on how to act, the agent <em>does not know</em> what the outcome will be. A common variation is that sometimes the agent will act randomly, and sometimes it will act greedily with respect to action values - that is the <span class="math-container">$\epsilon$</span>-greedy policy.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-04-20T07:22:09.047Z
Answers.LastActivityDate: 2022-04-20T07:30:31.330Z
Answers.LastEditDate: 2022-04-20T07:30:31.330Z
Answers.UserDisplayName: Neil Slater
Body: <p>In Q-learning, <span class="math-container">$Q(S_t,a)$</span> is updated by the Bellman equation. <span class="math-container">$Q(S_t,a) = r + \max_{a'}(Q(S_{t+1},a'))$</span> where <span class="math-container">$S_{t+1}$</span> is the future state.</p>
  <p>Let's say <span class="math-container">$S$</span> denotes the stock price, does it mean we are using future information <span class="math-container">$S_{t+1}$</span> in the Q-learning process?</p>
  <p>After the training process and bringing into implementation, how is it possible to apply the strategy without knowing the future price (the future state)?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-04-23T09:07:19.193Z
Comments.Text: You should be careful with the equations that you use. Is that really the equation or update rule used by Q-learning? Maybe you're referring to [the target (in DQN)](https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf)? In any case, be aware that the target is not the update rule.
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><q-learning><markov-decision-process><testing><algorithmic-trading>
Title: Does $S_{t+1}$ denote the future information in Q-learning?
UserDisplayName: L.Chau
ThreadText: # Does $S_{t+1}$ denote the future information in Q-learning?
  
  (Asked by: username_0 on 2022-04-20)
  
  In Q-learning, $Q(S\_t,a)$ is updated by the Bellman equation. $Q(S\_t,a) = r + \max\_{a'}(Q(S\_{t+1},a'))$ where $S\_{t+1}$ is the future state.
  
  Let's say $S$ denotes the stock price, does it mean we are using future information $S\_{t+1}$ in the Q-learning process?
  
  After the training process and bringing into implementation, how is it possible to apply the strategy without knowing the future price (the future state)?
  
  **Tags:** reinforcement-learning, q-learning, markov-decision-process, testing, algorithmic-trading
  
  > **Comments:**
  > **username_1**: You should be careful with the equations that you use. Is that really the equation or update rule used by Q-learning? Maybe you're referring to [the target (in DQN)](https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf)? In any case, be aware that the target is not the update rule.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-04-20. Score: 1
  
  > Let's say $S$ denotes the stock price, does it mean we are using future information $S\_{t+1}$ in the Q-learning process?
  
  It means you are using the information from a time step $t+1$ to update the value you associate with a state or state action pair seen on a previous time step $t$. When training, this is *not* future data, it is past data, and there does not need to be any special consideration of the value of $t$. It might be something that has just been observed, or could even be from other experience from years ago. The important thing is that you must know the state and action from time $t$, and the reward and next state at time $t+1$ (in general, you *might* take longer trajectories up to some time $t+n$).
  
  The term *backup* is used to describe the process of using information from a range of time steps to improve estimates made in earlier ones, and it is a key part of how all reinforcement learning (RL) methods work.
  
  > After the training process and bringing into implementation, how is it possible to apply the strategy without knowing the future price
  
  In production, you make use of the estimates that you refined during training by referring to the learned value function.
  
  In your example, if you are in a production system and are at time step $t$, then you know the current state $S\_t$. Your policy will be:
  
  $$\pi(S\_t) = \text{argmax}\_a Q(S\_t, a)$$
  
  This is the *greedy* policy, and doesn't use any future information, but it does rely on a training process that was allowed to "look ahead" to learn about results of taking different actions.
  
  It is common to use a variation of this decision rule during training too, because at the point of making a decision on how to act, the agent *does not know* what the outcome will be. A common variation is that sometimes the agent will act randomly, and sometimes it will act greedily with respect to action values - that is the $\epsilon$-greedy policy.
  
  
  ---
  
  