Answers.Body: <blockquote>
  <p>We see the existence of both <span class="math-container">$a'$</span> and <span class="math-container">$s'$</span>, and <span class="math-container">$s'$</span> could be unseen, for example, on the <em>very first</em> update, where we are at <span class="math-container">$s$</span>, take action <span class="math-container">$a$</span>, and could arrive at any state <span class="math-container">$s'$</span>.</p>
  </blockquote>
  <p>The very first update is made <em>after</em> taking action <span class="math-container">$a$</span> in state <span class="math-container">$s$</span> and observing reward <span class="math-container">$r$</span> plus next state <span class="math-container">$s'$</span>. There is no other way in Q learning of knowing what the next state is in order to process the update. So <span class="math-container">$s'$</span> in not unseen, it has been observed.</p>
  <p>Another way to put this: In model-free methods, the update to Q value estimates for state <span class="math-container">$S_t$</span> action <span class="math-container">$A_t$</span> on time step <span class="math-container">$t$</span> is always made on or after time step <span class="math-container">$t+1$</span>, when <span class="math-container">$R_{t+1}$</span> and <span class="math-container">$S_{t+1}$</span> are known.</p>
  <p>However, on that same time step, the action <span class="math-container">$a'$</span> does not yet need to have been taken. The value of <span class="math-container">$a'$</span> is not taken from <span class="math-container">$A_{t+1}$</span>, but is evaluated for all possibilities. Even after <span class="math-container">$A_{t+1}$</span> is known and has been taken, the need to process all possible actions in the state <span class="math-container">$s'$</span> (in order to update <span class="math-container">$Q(s,a)$</span> or <span class="math-container">$Q(S_t, A_t)$</span>) can often lead to needing action value estimates for never seen state/action pairs.</p>
  <p>The update step you quoted includes the <em>expected reward</em> (or possibly a custom reward function known to the agent) <span class="math-container">$R(s,a)$</span>, which is not standard for Q learning. It would be more usual to use the observed reward <span class="math-container">$r$</span>.  However, it may be ok because a lot of the time the developer/researcher is in charge of the reward signal and can provide <span class="math-container">$R(s,a)$</span> to the agent, even when the full model is not used.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-07T19:22:06.827Z
Answers.Text: Re the last point, my notation is incorrect, I meant to say r(s,a) for actual reward. I had also misremembered the algorithm, and added an expectation where it does not belong. Thanks for the answer.
Answers.UserDisplayName: Snowball
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-07T19:30:43.797Z
Answers.Text: @Snowball The actual reward just $r$ - it is not a function of anything else, but an observed value. The random variable would be $R_{t+1}$
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-07T15:47:44.143Z
Answers.LastActivityDate: 2021-09-07T15:52:59.327Z
Answers.LastEditDate: 2021-09-07T15:52:59.327Z
Answers.UserDisplayName: Neil Slater
Answers.Body: <p>In the offline dataset <span class="math-container">$\mathcal{D}$</span>, each experience is of the form <span class="math-container">$(s, a, s^{\prime}, r)$</span> (state, action, next state, reward). So <span class="math-container">$s^{\prime}$</span> is in your dataset</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-30T22:22:37.067Z
Answers.LastActivityDate: 2022-10-30T22:22:37.067Z
Answers.UserDisplayName: hongshan.li
Body: <p>In the paper <a href="https://proceedings.neurips.cc/paper/2020/file/0d2b2061826a5df3221116a5085a6052-Paper.pdf" rel="nofollow noreferrer">Conservative Q-Learning for Offline Reinforcement Learning</a>, it is stated (section 3.1, page 3) that</p>
  <blockquote>
  <p>standard Q-function training does not query the Q-function value at unobserved states, but queries the Q-function at unseen actions</p>
  </blockquote>
  <p>I don't see how this is true. For every <span class="math-container">$(s,a)$</span> pair, we need to update <span class="math-container">$Q(s,a)$</span> to reduce the value <span class="math-container">$|Q(s,a) - R(s,a) - \gamma E[\max_{a'}Q(s',a')]|$</span> until it converges to zero.</p>
  <p>We see the existence of both <span class="math-container">$a'$</span> and <span class="math-container">$s'$</span>, and <span class="math-container">$s'$</span> could be unseen, for example, on the <em>very first</em> update, where we are at <span class="math-container">$s$</span>, take action <span class="math-container">$a$</span>, and could arrive at any state <span class="math-container">$s'$</span>.</p>
  <p>Can someone explain this?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-07T15:44:56.763Z
Comments.Text: I believe they are referring to the fact that we only update the Q-function for state values that we have seen. This is true as typically we use a replay buffer of stored experience to update the Q-function -- all of these values have been 'seen' as we have experienced them.
Comments.UserDisplayName: David
Tags: <reinforcement-learning><q-learning><papers>
Title: Why does Q-function training not query the Q-function value at unobserved states?
UserDisplayName: Snowball
ThreadText: # Why does Q-function training not query the Q-function value at unobserved states?
  
  (Asked by: username_0 on 2021-09-07)
  
  In the paper [Conservative Q-Learning for Offline Reinforcement Learning](https://proceedings.neurips.cc/paper/2020/file/0d2b2061826a5df3221116a5085a6052-Paper.pdf), it is stated (section 3.1, page 3) that
  
  > standard Q-function training does not query the Q-function value at unobserved states, but queries the Q-function at unseen actions
  
  I don't see how this is true. For every $(s,a)$ pair, we need to update $Q(s,a)$ to reduce the value $|Q(s,a) - R(s,a) - \gamma E[\max\_{a'}Q(s',a')]|$ until it converges to zero.
  
  We see the existence of both $a'$ and $s'$, and $s'$ could be unseen, for example, on the *very first* update, where we are at $s$, take action $a$, and could arrive at any state $s'$.
  
  Can someone explain this?
  
  **Tags:** reinforcement-learning, q-learning, papers
  
  > **Comments:**
  > **username_1**: I believe they are referring to the fact that we only update the Q-function for state values that we have seen. This is true as typically we use a replay buffer of stored experience to update the Q-function -- all of these values have been 'seen' as we have experienced them.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2021-09-07. Score: 0
  
  > We see the existence of both $a'$ and $s'$, and $s'$ could be unseen, for example, on the *very first* update, where we are at $s$, take action $a$, and could arrive at any state $s'$.
  
  The very first update is made *after* taking action $a$ in state $s$ and observing reward $r$ plus next state $s'$. There is no other way in Q learning of knowing what the next state is in order to process the update. So $s'$ in not unseen, it has been observed.
  
  Another way to put this: In model-free methods, the update to Q value estimates for state $S\_t$ action $A\_t$ on time step $t$ is always made on or after time step $t+1$, when $R\_{t+1}$ and $S\_{t+1}$ are known.
  
  However, on that same time step, the action $a'$ does not yet need to have been taken. The value of $a'$ is not taken from $A\_{t+1}$, but is evaluated for all possibilities. Even after $A\_{t+1}$ is known and has been taken, the need to process all possible actions in the state $s'$ (in order to update $Q(s,a)$ or $Q(S\_t, A\_t)$) can often lead to needing action value estimates for never seen state/action pairs.
  
  The update step you quoted includes the *expected reward* (or possibly a custom reward function known to the agent) $R(s,a)$, which is not standard for Q learning. It would be more usual to use the observed reward $r$. However, it may be ok because a lot of the time the developer/researcher is in charge of the reward signal and can provide $R(s,a)$ to the agent, even when the full model is not used.
  
  > **Comments:**
  > **username_0**: Re the last point, my notation is incorrect, I meant to say r(s,a) for actual reward. I had also misremembered the algorithm, and added an expectation where it does not belong. Thanks for the answer.  
  > **username_2**: @username_0 The actual reward just $r$ - it is not a function of anything else, but an observed value. The random variable would be $R_{t+1}$  
  
  ---
  
  ### Answer by username_3 on 2022-10-30. Score: 0
  
  In the offline dataset $\mathcal{D}$, each experience is of the form $(s, a, s^{\prime}, r)$ (state, action, next state, reward). So $s^{\prime}$ is in your dataset
  
  
  ---
  
  