Answers.Body: <p><span class="math-container">$$R_{s}=\mathbb{E}[R_{t}|S_{t}=s]$$</span></p>
  <p>is the <strong>expected reward</strong> at time step <span class="math-container">$t$</span> given that the state at time <span class="math-container">$t$</span> is <span class="math-container">$s$</span>, where</p>
  <ul>
  <li><span class="math-container">$R_{t}$</span> and <span class="math-container">$S_t$</span> are random variables that represent the reward and state at time <span class="math-container">$t$</span>, respectively,</li>
  <li><span class="math-container">$S_{t}=s$</span> is an event, and</li>
  <li><span class="math-container">$\mathbb{E}[X]$</span> denotes the expected value of the random variable <span class="math-container">$X$</span>.</li>
  </ul>
  <p>It does not matter how you entered <span class="math-container">$S_t = s$</span>. The only thing that matters is that the current state is <span class="math-container">$s$</span>. So, the answer to your question in the title is - no. <span class="math-container">$R_s$</span> is defined as an expectation (average) of the reward for being in a state, and, in this case, it doesn't take into account next states.</p>
  <p>Suppose that the reward space <span class="math-container">$\mathcal{R} \subset \mathbb{R}$</span> is a discrete set, then we can write <span class="math-container">$R_s$</span> as follows</p>
  <p><span class="math-container">$$R_{s}=\sum_{r \in \mathcal{R}} r p(r \mid S_{t}=s),$$</span></p>
  <p>where <span class="math-container">$p(r \mid S_{t}=s)$</span> is a conditional probability distribution that describes how the reward is distributed in a given state. If you always get the same reward, i.e. <span class="math-container">$p(r \mid S_{t}=s) = 1$</span> for a particular <span class="math-container">$r$</span> and <span class="math-container">$0$</span> for all others, then you have a deterministic reward function, i.e. <span class="math-container">$R_s$</span> is equal to the <span class="math-container">$r$</span> for which <span class="math-container">$p(r \mid S_{t}=s) = 1$</span>. In most RL examples, reward functions are deterministic.</p>
  <p>Note that it is probably more common to define the reward function as a function of the state <span class="math-container">$S_t = s$</span> and action taken in <span class="math-container">$s$</span>, <span class="math-container">$A_t =a$</span>, and write it as <span class="math-container">$R(s, a)$</span> or <span class="math-container">$r(s, a)$</span>, i.e.</p>
  <p><span class="math-container">\begin{align}
  R(s, a)
  &amp;=\mathbb{E}[R_{t}|S_{t}=s, A_t = a]\\
  &amp;=\sum_{r \in \mathcal{R}} r p(r \mid S_{t}=s, A_t = a)
  \end{align}</span></p>
  <p>Note that we're now using <span class="math-container">$p(r \mid S_{t}=s, A_t = a)$</span> and not <span class="math-container">$p(r \mid S_{t}=s)$</span>.</p>
  <p>You could also define <span class="math-container">$R(s, a, s')$</span>. See <a href="https://ai.stackexchange.com/q/10442/2444">this</a> or Sutton &amp; Barto's book for more details.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-04-17T01:05:57.490Z
Answers.Text: I see, thank you for clarifying this and for providing the link.
Answers.UserDisplayName: Daviiid
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-04-16T23:34:16.160Z
Answers.LastActivityDate: 2022-04-16T23:34:16.160Z
Answers.UserDisplayName: nbro
Body: <p>I'm trying to understand correctly what each &quot;variable&quot; in RL is and I'm not sure about <span class="math-container">$R_{s}$</span> the reward function. I used to think that it's the reward we may expect on average after taking an action <span class="math-container">$A_{t}$</span> at state <span class="math-container">$S_{t}=s$</span> and at time step <span class="math-container">$s$</span> but thinking about it more I think I was wrong.</p>
  <p>If we consider that the agent at time step <span class="math-container">$t$</span> receives from the environment both an observation <span class="math-container">$O_{t}$</span> and a reward <span class="math-container">$R_{t}$</span> which will make up the agent's state <span class="math-container">$S_{t}=s$</span> depending on what function of the history we take, then it means that we got to state <span class="math-container">$S_{t}=s$</span> from a state <span class="math-container">$S_{t-1}$</span> taking action an action <span class="math-container">$A_{t-1}$</span> right?</p>
  <p>I'd like to ask you to confirm or not my claim.</p>
  
Tags: <reinforcement-learning><markov-decision-process><notation><reward-functions><state-spaces>
Title: Does $R_{s}=E[R_{t}|S_{t}=s]$ indicate the reward we might expect on getting on average moving from any other state to $s$?
UserDisplayName: Daviiid
ThreadText: # Does $R_{s}=E[R_{t}|S_{t}=s]$ indicate the reward we might expect on getting on average moving from any other state to $s$?
  
  (Asked by: username_0 on 2022-04-16)
  
  I'm trying to understand correctly what each "variable" in RL is and I'm not sure about $R\_{s}$ the reward function. I used to think that it's the reward we may expect on average after taking an action $A\_{t}$ at state $S\_{t}=s$ and at time step $s$ but thinking about it more I think I was wrong.
  
  If we consider that the agent at time step $t$ receives from the environment both an observation $O\_{t}$ and a reward $R\_{t}$ which will make up the agent's state $S\_{t}=s$ depending on what function of the history we take, then it means that we got to state $S\_{t}=s$ from a state $S\_{t-1}$ taking action an action $A\_{t-1}$ right?
  
  I'd like to ask you to confirm or not my claim.
  
  **Tags:** reinforcement-learning, markov-decision-process, notation, reward-functions, state-spaces
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-04-16. Score: 1 | Accepted answer
  
  $$R\_{s}=\mathbb{E}[R\_{t}|S\_{t}=s]$$
  
  is the **expected reward** at time step $t$ given that the state at time $t$ is $s$, where
  
  * $R\_{t}$ and $S\_t$ are random variables that represent the reward and state at time $t$, respectively,
  * $S\_{t}=s$ is an event, and
  * $\mathbb{E}[X]$ denotes the expected value of the random variable $X$.
  
  It does not matter how you entered $S\_t = s$. The only thing that matters is that the current state is $s$. So, the answer to your question in the title is - no. $R\_s$ is defined as an expectation (average) of the reward for being in a state, and, in this case, it doesn't take into account next states.
  
  Suppose that the reward space $\mathcal{R} \subset \mathbb{R}$ is a discrete set, then we can write $R\_s$ as follows
  
  $$R\_{s}=\sum\_{r \in \mathcal{R}} r p(r \mid S\_{t}=s),$$
  
  where $p(r \mid S\_{t}=s)$ is a conditional probability distribution that describes how the reward is distributed in a given state. If you always get the same reward, i.e. $p(r \mid S\_{t}=s) = 1$ for a particular $r$ and $0$ for all others, then you have a deterministic reward function, i.e. $R\_s$ is equal to the $r$ for which $p(r \mid S\_{t}=s) = 1$. In most RL examples, reward functions are deterministic.
  
  Note that it is probably more common to define the reward function as a function of the state $S\_t = s$ and action taken in $s$, $A\_t =a$, and write it as $R(s, a)$ or $r(s, a)$, i.e.
  
  \begin{align}
  R(s, a)
  &=\mathbb{E}[R\_{t}|S\_{t}=s, A\_t = a]\\
  &=\sum\_{r \in \mathcal{R}} r p(r \mid S\_{t}=s, A\_t = a)
  \end{align}
  
  Note that we're now using $p(r \mid S\_{t}=s, A\_t = a)$ and not $p(r \mid S\_{t}=s)$.
  
  You could also define $R(s, a, s')$. See [this](https://ai.stackexchange.com/q/10442/2444) or Sutton & Barto's book for more details.
  
  > **Comments:**
  > **username_0**: I see, thank you for clarifying this and for providing the link.  
  
  ---
  
  