Answers.Body: <blockquote>
  <p>As I understand it, the Bellman equation assumes the setting to be deterministic, meaning that, if you're in state <span class="math-container">$s_t$</span> as you take action <span class="math-container">$a_1$</span>, you should always reach the same <span class="math-container">$s_{t+1}$</span>.</p>
  </blockquote>
  <p>This is not correct, which is a good thing for you. The Bellman equation for action values for an arbitrary policy <span class="math-container">$\pi(a|s): \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R} = \mathbf{Pr}\{A_t=a|S_t=s \}$</span> looks like this:</p>
  <p><span class="math-container">$$q_\pi(s,a) = \sum_{r,s'}p(r,s'|s,a)(r+\gamma\sum_{a'}\pi(s',a')q_{\pi}(s',a'))$$</span></p>
  <p>The environment model in this equation is <span class="math-container">$p(r,s'|s,a)$</span> which is the probability of observing reward <span class="math-container">$r$</span> and next state <span class="math-container">$s'$</span>, given current state <span class="math-container">$s$</span> and action <span class="math-container">$a$</span>. This is stochastic.</p>
  <p>When using a model-free approach learning from experience, you cannot use the full equation each time, because you only have <em>one sample</em> and do not know anything about <span class="math-container">$p()$</span>. However, things will still work, because if you take many samples, on average they will follow the probability distribution for <span class="math-container">$p()$</span>, and your update rule will converge to the average/expected return for each action value.</p>
  <p>The same sampling also applies to the policy, which means the above Bellman equation can be expressed as:</p>
  <p><span class="math-container">$$q_\pi(s,a) = \mathbb{E}_{A_{t+1} \sim \pi}[R_{t+1} + \gamma q_{\pi}(S_{t+1},A_{t+1})|S_t=s, A_t=a]$$</span></p>
  <p>which in turn means you can use <em>samples</em> of <span class="math-container">$r, s'$</span> and <span class="math-container">$a'$</span> (or for Q-learning, the maximising <span class="math-container">$a'$</span>), and an update rule that averages between the values you observe. You can do this with both deterministic and stochastic environments, because the theory behind it works with both.</p>
  <blockquote>
  <p>Would it somehow ruin the training, if I then just hardcoded it, such that the next state reverted to state <span class="math-container">$s_{10}$</span> if the policy network picked action <span class="math-container">$a1$</span>? Are there any counter-productive consequences to this?</p>
  </blockquote>
  <p>For some environments, and some setups of the hardcoding, it may work. However, you should not do this because it is not necessary, and in general it is a bad idea, as it may impact the sampling in a way that causes incorrect learning.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-25T13:05:35.833Z
Answers.LastActivityDate: 2021-12-25T13:05:35.833Z
Answers.UserDisplayName: Neil Slater
Body: <p>Say I have a game of blackjack, and I am trying to teach a single forward-pass neural network to approximate the Q value of the current state and action.</p>
  <p>There are 3 inputs: The current card in hand, the cards in the deck, and the cards in the pile. It outputs the Q-value of two actions, namely, holding or adding the current card to the pile.</p>
  <p>My loss function is <span class="math-container">$$L(Q,Q_E)=  \sum({Q(s,a_i)- Q_E(s,a_i) )^2},$$</span></p>
  <p>where <span class="math-container">$Q_E$</span> is the estimated Q-value of the current state from the policy network. And <span class="math-container">$Q$</span> is the target function, which is calculated using the Bellman equation.</p>
  <p>As I understand it, the Bellman equation assumes the setting to be deterministic, meaning that, if you're in state <span class="math-container">$s_t$</span> as you take action <span class="math-container">$a_1$</span>, you should always reach the same <span class="math-container">$s_{t+1}$</span>.</p>
  <p>Of course, in blackjack, this is not the case, as the state <span class="math-container">$s_{t+1}$</span> is purely dependent on the card you draw, which is a stochastic process.</p>
  <p>Would it be possible to omit some of this noise or &quot;stochasticity&quot; by enforcing the same <span class="math-container">$s_{t + 1}$</span> between the model's estimate and the target function's Q-value?</p>
  <p>In other words, say we're in state <span class="math-container">$s_t$</span>, and the target function picks action <span class="math-container">$a_1$</span> and draws a 10 reaching state <span class="math-container">$s_{10}$</span> as the next state. For the training of the policy network, it loads the state <span class="math-container">$s_t$</span> from the experience replay, it also picks action <span class="math-container">$a_1$</span> and draws a 7 reaching state <span class="math-container">$s_7$</span> as the next state.</p>
  <p>Would it somehow ruin the training, if I then just hardcoded it, such that the next state reverted to state <span class="math-container">$s_{10}$</span> if the policy network picked action <span class="math-container">$a1$</span>? Are there any counter-productive consequences to this?</p>
  
Tags: <reinforcement-learning><dqn><bellman-equations>
Title: Would it be possible to enforce the same $s_{t + 1}$ between the model's estimate and the target function's Q-value?
UserDisplayName: GeorgeWTrump
ThreadText: # Would it be possible to enforce the same $s_{t + 1}$ between the model's estimate and the target function's Q-value?
  
  (Asked by: username_0 on 2021-12-22)
  
  Say I have a game of blackjack, and I am trying to teach a single forward-pass neural network to approximate the Q value of the current state and action.
  
  There are 3 inputs: The current card in hand, the cards in the deck, and the cards in the pile. It outputs the Q-value of two actions, namely, holding or adding the current card to the pile.
  
  My loss function is $$L(Q,Q\_E)= \sum({Q(s,a\_i)- Q\_E(s,a\_i) )^2},$$
  
  where $Q\_E$ is the estimated Q-value of the current state from the policy network. And $Q$ is the target function, which is calculated using the Bellman equation.
  
  As I understand it, the Bellman equation assumes the setting to be deterministic, meaning that, if you're in state $s\_t$ as you take action $a\_1$, you should always reach the same $s\_{t+1}$.
  
  Of course, in blackjack, this is not the case, as the state $s\_{t+1}$ is purely dependent on the card you draw, which is a stochastic process.
  
  Would it be possible to omit some of this noise or "stochasticity" by enforcing the same $s\_{t + 1}$ between the model's estimate and the target function's Q-value?
  
  In other words, say we're in state $s\_t$, and the target function picks action $a\_1$ and draws a 10 reaching state $s\_{10}$ as the next state. For the training of the policy network, it loads the state $s\_t$ from the experience replay, it also picks action $a\_1$ and draws a 7 reaching state $s\_7$ as the next state.
  
  Would it somehow ruin the training, if I then just hardcoded it, such that the next state reverted to state $s\_{10}$ if the policy network picked action $a1$? Are there any counter-productive consequences to this?
  
  **Tags:** reinforcement-learning, dqn, bellman-equations
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-12-25. Score: 0 | Accepted answer
  
  > As I understand it, the Bellman equation assumes the setting to be deterministic, meaning that, if you're in state $s\_t$ as you take action $a\_1$, you should always reach the same $s\_{t+1}$.
  
  This is not correct, which is a good thing for you. The Bellman equation for action values for an arbitrary policy $\pi(a|s): \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R} = \mathbf{Pr}\{A\_t=a|S\_t=s \}$ looks like this:
  
  $$q\_\pi(s,a) = \sum\_{r,s'}p(r,s'|s,a)(r+\gamma\sum\_{a'}\pi(s',a')q\_{\pi}(s',a'))$$
  
  The environment model in this equation is $p(r,s'|s,a)$ which is the probability of observing reward $r$ and next state $s'$, given current state $s$ and action $a$. This is stochastic.
  
  When using a model-free approach learning from experience, you cannot use the full equation each time, because you only have *one sample* and do not know anything about $p()$. However, things will still work, because if you take many samples, on average they will follow the probability distribution for $p()$, and your update rule will converge to the average/expected return for each action value.
  
  The same sampling also applies to the policy, which means the above Bellman equation can be expressed as:
  
  $$q\_\pi(s,a) = \mathbb{E}\_{A\_{t+1} \sim \pi}[R\_{t+1} + \gamma q\_{\pi}(S\_{t+1},A\_{t+1})|S\_t=s, A\_t=a]$$
  
  which in turn means you can use *samples* of $r, s'$ and $a'$ (or for Q-learning, the maximising $a'$), and an update rule that averages between the values you observe. You can do this with both deterministic and stochastic environments, because the theory behind it works with both.
  
  > Would it somehow ruin the training, if I then just hardcoded it, such that the next state reverted to state $s\_{10}$ if the policy network picked action $a1$? Are there any counter-productive consequences to this?
  
  For some environments, and some setups of the hardcoding, it may work. However, you should not do this because it is not necessary, and in general it is a bad idea, as it may impact the sampling in a way that causes incorrect learning.
  
  
  ---
  
  