Answers.Body: <blockquote>
    <p>Is the main difference between these two problems, and hence why one is regression and the other is kernel density estimation, because with the reward we are mainly concerned with the expected reward (hence regression) whereas with the state transitioning, we want to be able to simulate this so we need the estimated density?</p>
  </blockquote>
  
  <p>Yes.</p>
  
  <p>An expected reward function from <span class="math-container">$s,a$</span> is all you need to construct valid Bellman equations for value functions. For example</p>
  
  <p><span class="math-container">$$q_{\pi}(s,a) = r(s,a) + \gamma\sum_{s'}p(s'|s,a)\sum_{a'}\pi(a'|s')q(s',a')$$</span></p>
  
  <p>is a valid way of writing the Bellman equation for action values. You can derive this from <span class="math-container">$r(s,a) = \sum_{r,s'}rp(r,s'|s,a)$</span> and <span class="math-container">$q_{\pi}(s,a) = \sum_{r,s'}p(r,s'|s,a)(r + \gamma\sum_{a'}\pi(a'|s')q(s',a'))$</span> if you have the equations in that form.</p>
  
  <p>However, in general there is no such thing as an "expected state" when there is more than one possible outcome (i.e. in environments with stochastic state transitions). You can take a mean of the state vector representations over the samples you see for <span class="math-container">$s'$</span> but that is not the same thing at all and could easily be a representation of an unreachable/nonsense state.</p>
  
  <p>In some cases, the expectation <span class="math-container">$\mathbb{E}_{\pi}[x(S_{t+1})|S_t=s, A_t=a]$</span> where <span class="math-container">$x(s)$</span> creates a feature vector from any given state <span class="math-container">$s$</span>, <span class="math-container">$x(s): \mathcal{S} \rightarrow \mathbb{R}^d$</span>, can be meaningful. The broadest and most trivial example of this is for deterministic environments. You may be able to construct stochastic environments where there is a good interpretation of such a vector, even if it does not represent any reachable state. </p>
  
  <p>Simple one-hot encoded states could maybe made to work like this by representing a probability distribution over states (this would also require re-interpretations of expected reward function and value functions). That is effectively a kernel density function over discrete state space.</p>
  
  <p>In general knowing this <span class="math-container">$\mathbb{E}_{\pi}[x(S_{t+1})|S_t=s, A_t=a]$</span> expected value does not help resolve future rewards, as they can depend arbitrarily on specific state transitions.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-29T12:24:36.427Z
Answers.Text: Thanks - I understood why you wouldn't want to take the expected value of the state distribution, I just wanted to double check why we weren't interested in the full distribution of the rewards, just the expected value.
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-29T12:30:00.660Z
Answers.Text: @DavidIreland: I probably focussed too much on the state distribution issue and not enough on expected reward then. I may come back and address that
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-29T12:42:31.267Z
Answers.Text: For me the comment you made about only needing expected reward to be able to construct the Bellman equations for enough but the new edit is very clear.
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-29T12:14:17.047Z
Answers.LastActivityDate: 2020-05-29T12:35:51.867Z
Answers.LastEditDate: 2020-05-29T12:35:51.867Z
Answers.UserDisplayName: Neil Slater
Body: <p>In <a href="https://youtu.be/ItMutbeOHtc?t=1542" rel="nofollow noreferrer">David Silver's 8th lecture</a> he talks about model learning and says that learning <span class="math-container">$r$</span> from <span class="math-container">$s,a$</span> is a regression problem whereas learning <span class="math-container">$s'$</span> from <span class="math-container">$s,a$</span> is a kernel density estimation. His explanation for the difference is that if we are in a stochastic environment and we are in the tuple <span class="math-container">$s,a$</span> then there might be a 30% chance the wind will blow me left, and a 70% chance the wind will blow me right, so we want to estimate these probabilities. </p>
  
  <p>Is the main difference between these two problems, and hence why one is regression and the other is kernel density estimation, because with the reward we are mainly concerned with the <em>expected</em> reward (hence regression) whereas with the state transitioning, we want to be able to simulate this so we need the estimated density? </p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-29T11:49:07.300Z
Comments.Text: NB the link should take you to roughly where he mentions this.
Comments.UserDisplayName: David
Tags: <reinforcement-learning><markov-decision-process><model-based-methods>
Title: Why is learning $s'$ from $s,a$ a kernel density estimation problem but learning $r$ from $s,a$ is just regression?
UserDisplayName: David
ThreadText: # Why is learning $s'$ from $s,a$ a kernel density estimation problem but learning $r$ from $s,a$ is just regression?
  
  (Asked by: username_0 on 2020-05-29)
  
  In [username_0 Silver's 8th lecture](https://youtu.be/ItMutbeOHtc?t=1542) he talks about model learning and says that learning $r$ from $s,a$ is a regression problem whereas learning $s'$ from $s,a$ is a kernel density estimation. His explanation for the difference is that if we are in a stochastic environment and we are in the tuple $s,a$ then there might be a 30% chance the wind will blow me left, and a 70% chance the wind will blow me right, so we want to estimate these probabilities.
  
  Is the main difference between these two problems, and hence why one is regression and the other is kernel density estimation, because with the reward we are mainly concerned with the *expected* reward (hence regression) whereas with the state transitioning, we want to be able to simulate this so we need the estimated density?
  
  **Tags:** reinforcement-learning, markov-decision-process, model-based-methods
  
  > **Comments:**
  > **username_0**: NB the link should take you to roughly where he mentions this.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-05-29. Score: 2 | Accepted answer
  
  > Is the main difference between these two problems, and hence why one is regression and the other is kernel density estimation, because with the reward we are mainly concerned with the expected reward (hence regression) whereas with the state transitioning, we want to be able to simulate this so we need the estimated density?
  
  Yes.
  
  An expected reward function from $s,a$ is all you need to construct valid Bellman equations for value functions. For example
  
  $$q\_{\pi}(s,a) = r(s,a) + \gamma\sum\_{s'}p(s'|s,a)\sum\_{a'}\pi(a'|s')q(s',a')$$
  
  is a valid way of writing the Bellman equation for action values. You can derive this from $r(s,a) = \sum\_{r,s'}rp(r,s'|s,a)$ and $q\_{\pi}(s,a) = \sum\_{r,s'}p(r,s'|s,a)(r + \gamma\sum\_{a'}\pi(a'|s')q(s',a'))$ if you have the equations in that form.
  
  However, in general there is no such thing as an "expected state" when there is more than one possible outcome (i.e. in environments with stochastic state transitions). You can take a mean of the state vector representations over the samples you see for $s'$ but that is not the same thing at all and could easily be a representation of an unreachable/nonsense state.
  
  In some cases, the expectation $\mathbb{E}\_{\pi}[x(S\_{t+1})|S\_t=s, A\_t=a]$ where $x(s)$ creates a feature vector from any given state $s$, $x(s): \mathcal{S} \rightarrow \mathbb{R}^d$, can be meaningful. The broadest and most trivial example of this is for deterministic environments. You may be able to construct stochastic environments where there is a good interpretation of such a vector, even if it does not represent any reachable state.
  
  Simple one-hot encoded states could maybe made to work like this by representing a probability distribution over states (this would also require re-interpretations of expected reward function and value functions). That is effectively a kernel density function over discrete state space.
  
  In general knowing this $\mathbb{E}\_{\pi}[x(S\_{t+1})|S\_t=s, A\_t=a]$ expected value does not help resolve future rewards, as they can depend arbitrarily on specific state transitions.
  
  > **Comments:**
  > **username_0**: Thanks - I understood why you wouldn't want to take the expected value of the state distribution, I just wanted to double check why we weren't interested in the full distribution of the rewards, just the expected value.  
  > **username_1**: @DavidIreland: I probably focussed too much on the state distribution issue and not enough on expected reward then. I may come back and address that  
  > **username_0**: For me the comment you made about only needing expected reward to be able to construct the Bellman equations for enough but the new edit is very clear.  
  
  ---
  
  