Answers.Body: <p>I don't know that I've seen that notation but one must remember that an MDP does not guarantee a transition from <span class="math-container">$S_1$</span> to <span class="math-container">$S_2$</span> via action <span class="math-container">$A_0$</span>.  There may be some stochasity in the transition causing the agent to stay in <span class="math-container">$S_1$</span>, successfully transition to <span class="math-container">$S_2$</span> or instead go to <span class="math-container">$S_3$</span>.  Think of a grid world with a slippery floor that may cause the agent not to move or to over move.</p>
  <p>The reward is not for taking action <span class="math-container">$A_0$</span> in <span class="math-container">$S_1$</span> but for the real resulting state.  Therefore the reward at time t is always for <span class="math-container">$S_{t+1}$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-25T04:38:22.410Z
Answers.LastActivityDate: 2024-01-25T04:38:22.410Z
Answers.UserDisplayName: foreverska
Answers.Body: <p>Your question didn't specify the reference of your picture at all which may have additional context in terms of the specific meaning of your reward function such as whether it's a random variable or an expected value. For instance, your picture very much resembles Figure 3.1 of Sutton's RL book where the reward <span class="math-container">$\mathcal{R}$</span> is a random variable and they never wrote or needed <span class="math-container">$\mathcal{R}(s_t, a_t, s_{t+1})$</span>, but instead, its expected value <span class="math-container">$r(s_t, a_t, s_{t+1})$</span> is defined on page 49 due to possible stochastic environment unlike a normal grid world game:</p>
  <blockquote>
  <p>We can also compute the expected rewards for state–action pairs as a two-argument function <span class="math-container">$r : \mathcal{S} \times \mathcal{A} \to \mathbb{R}$</span>:
  <span class="math-container">$$r(s,a) = \sum_{r \in \mathcal{R}} \sum_{s' \in \mathcal{S}} rp(s',r|s,a)$$</span>
  and the expected rewards for state–action–next-state triples as a three-argument function <span class="math-container">$r : \mathcal{S} \times \mathcal{A} \times \mathcal{S} \to \mathbb{R}$</span>,
  <span class="math-container">$$r(s,a,s') = \sum_{r \in \mathcal{R}}r\frac{p(s',r|s,a)}{p(s'|s,a)}$$</span></p>
  </blockquote>
  <p>In MDP RL there's always supposed to be an explicit model of the environment including stationary state and reward transition dynamics, thus as you claimed there's no necessary need for the (expected) reward function given by the environment to ever have explicit dependence on the next entering state <span class="math-container">$s_{t+1}$</span> to solve any problem. In fact in Sutton's book the only place the 3-argument expected reward is explicitly used appears in a nondeterministic finite state diagram for a recycling robot example on page 52:</p>
  <p><a href="https://i.sstatic.net/5UpVs.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/5UpVs.png" alt="enter image description here" /></a></p>
  <p>You see in such a diagram the expected reward function needs the explicit dependence of the entering state to conveniently express its value based on its entering state for each of the possibly branching state transition due to its random environment characterized by the above Markovian transition dynamics <span class="math-container">$p(s',r|s,a)$</span>, and they're obviously mathematically compatible with the above usual 2-argument reward function.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-25T04:55:24.457Z
Answers.LastActivityDate: 2024-01-30T21:43:37.937Z
Answers.LastEditDate: 2024-01-30T21:43:37.937Z
Answers.UserDisplayName: cinch
Answers.Body: <p>There is a free choice when deciding the indexing of rewards in reinforcement learning. You can decide to associate the immediate reward with the time step of state and action that generated it, or with the time step of the observed next state. Both conventions are self-consistent and unambiguous, they differ only in the notation used in expressions.</p>
  <p>The most common convention is to associate reward with next time step, leading to trajectories that look like this:</p>
  <p><span class="math-container">$$s_0, a_0, r_1, s_1, a_1, r_2 . . . s_{T-1}, a_{T-1}, r_T, s_T$$</span></p>
  <p>The alternative convention, that it looks like your excerpt is using*, looks like this:</p>
  <p><span class="math-container">$$s_0, a_0, r_0, s_1, a_1, r_1 . . . s_{T-1}, a_{T-1}, r_{T-1}, s_T$$</span></p>
  <p>Whether you use 0 as first index or 1 is a similar convention.</p>
  <p>In practice, the next state and reward that result from a state/action pair are part of an environment response where they can be calculated in any order. In some cases, <span class="math-container">$s_{t+1}$</span> is required before the associated reward, because the value of the reward depends on it, yet usually reward is shown in trajectories <em>before</em> the next state so that the listing has the same pattern <span class="math-container">$s,a,r$</span> throughout. It doesn't matter how the notation tracks that, as long as it is consistent. The timing of events is not different in the different conventions, only the index numbering.</p>
  <hr />
  <p>* Note that the diagram that goes with the equations does show reward and state sharing time step indexing. This implies that there could be a typo and the author <em>meant</em> to write <span class="math-container">$r_{t+1}$</span> where they have put <span class="math-container">$r_t$</span> in equation 1.3. Or perhaps they have used two sources which use different conventions and not noticed.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-01T02:53:32.150Z
Answers.Text: So, you also agree that the equations and the diagram are not consistent? That was in fact my question.
Answers.UserDisplayName: DSPinfinity
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-01T08:16:51.530Z
Answers.Text: @DSPinfinity yes I agree, something looks wrong there. However in isolation neither is incorrect
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-31T08:25:39.517Z
Answers.LastActivityDate: 2024-01-31T11:56:36.747Z
Answers.LastEditDate: 2024-01-31T11:56:36.747Z
Answers.UserDisplayName: Neil Slater
Body: <p><a href="https://i.sstatic.net/SAvqo.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/SAvqo.png" alt="enter image description here" /></a></p>
  <p>The above picture is from a textbook on RL.
  I am confused with the notation <span class="math-container">$r_t=\mathcal{R}(s_t, a_t, s_{t+1})$</span>. How can reward at time
  step <span class="math-container">$t$</span> be a function of a state at time step <span class="math-container">$t+1$</span>?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-01-25T12:39:03.110Z
Comments.Text: Does this answer your question? [How are the reward functions $R(s)$, $R(s, a)$ and $R(s, a, s')$ equivalent?](https://ai.stackexchange.com/questions/10442/how-are-the-reward-functions-rs-rs-a-and-rs-a-s-equivalent)
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning>
Title: How can reward at time step $t$ can be a function of a state at time step $t+1$?
UserDisplayName: DSPinfinity
ThreadText: # How can reward at time step $t$ can be a function of a state at time step $t+1$?
  
  (Asked by: username_0 on 2024-01-25)
  
  [![enter image description here](https://i.sstatic.net/SAvqo.png)](https://i.sstatic.net/SAvqo.png)
  
  The above picture is from a textbook on RL.
  I am confused with the notation $r\_t=\mathcal{R}(s\_t, a\_t, s\_{t+1})$. How can reward at time
  step $t$ be a function of a state at time step $t+1$?
  
  **Tags:** reinforcement-learning
  > **Comments:**
  > **username_1**: Does this answer your question? [How are the reward functions $R(s)$, $R(s, a)$ and $R(s, a, s')$ equivalent?](https://ai.stackexchange.com/questions/10442/how-are-the-reward-functions-rs-rs-a-and-rs-a-s-equivalent)  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_4 on 2024-01-31. Score: 2 | Accepted answer
  
  There is a free choice when deciding the indexing of rewards in reinforcement learning. You can decide to associate the immediate reward with the time step of state and action that generated it, or with the time step of the observed next state. Both conventions are self-consistent and unambiguous, they differ only in the notation used in expressions.
  
  The most common convention is to associate reward with next time step, leading to trajectories that look like this:
  
  $$s\_0, a\_0, r\_1, s\_1, a\_1, r\_2 . . . s\_{T-1}, a\_{T-1}, r\_T, s\_T$$
  
  The alternative convention, that it looks like your excerpt is using\*, looks like this:
  
  $$s\_0, a\_0, r\_0, s\_1, a\_1, r\_1 . . . s\_{T-1}, a\_{T-1}, r\_{T-1}, s\_T$$
  
  Whether you use 0 as first index or 1 is a similar convention.
  
  In practice, the next state and reward that result from a state/action pair are part of an environment response where they can be calculated in any order. In some cases, $s\_{t+1}$ is required before the associated reward, because the value of the reward depends on it, yet usually reward is shown in trajectories *before* the next state so that the listing has the same pattern $s,a,r$ throughout. It doesn't matter how the notation tracks that, as long as it is consistent. The timing of events is not different in the different conventions, only the index numbering.
  
  ---
  
  \* Note that the diagram that goes with the equations does show reward and state sharing time step indexing. This implies that there could be a typo and the author *meant* to write $r\_{t+1}$ where they have put $r\_t$ in equation 1.3. Or perhaps they have used two sources which use different conventions and not noticed.
  
  > **Comments:**
  > **username_0**: So, you also agree that the equations and the diagram are not consistent? That was in fact my question.  
  > **username_4**: @DSPinfinity yes I agree, something looks wrong there. However in isolation neither is incorrect  
  
  ---
  
  ### Answer by username_2 on 2024-01-25. Score: -1
  
  I don't know that I've seen that notation but one must remember that an MDP does not guarantee a transition from $S\_1$ to $S\_2$ via action $A\_0$. There may be some stochasity in the transition causing the agent to stay in $S\_1$, successfully transition to $S\_2$ or instead go to $S\_3$. Think of a grid world with a slippery floor that may cause the agent not to move or to over move.
  
  The reward is not for taking action $A\_0$ in $S\_1$ but for the real resulting state. Therefore the reward at time t is always for $S\_{t+1}$.
  
  
  ---
  
  ### Answer by username_3 on 2024-01-25. Score: -1
  
  Your question didn't specify the reference of your picture at all which may have additional context in terms of the specific meaning of your reward function such as whether it's a random variable or an expected value. For instance, your picture very much resembles Figure 3.1 of Sutton's RL book where the reward $\mathcal{R}$ is a random variable and they never wrote or needed $\mathcal{R}(s\_t, a\_t, s\_{t+1})$, but instead, its expected value $r(s\_t, a\_t, s\_{t+1})$ is defined on page 49 due to possible stochastic environment unlike a normal grid world game:
  
  > We can also compute the expected rewards for state–action pairs as a two-argument function $r : \mathcal{S} \times \mathcal{A} \to \mathbb{R}$:
  > $$r(s,a) = \sum\_{r \in \mathcal{R}} \sum\_{s' \in \mathcal{S}} rp(s',r|s,a)$$
  > and the expected rewards for state–action–next-state triples as a three-argument function $r : \mathcal{S} \times \mathcal{A} \times \mathcal{S} \to \mathbb{R}$,
  > $$r(s,a,s') = \sum\_{r \in \mathcal{R}}r\frac{p(s',r|s,a)}{p(s'|s,a)}$$
  
  In MDP RL there's always supposed to be an explicit model of the environment including stationary state and reward transition dynamics, thus as you claimed there's no necessary need for the (expected) reward function given by the environment to ever have explicit dependence on the next entering state $s\_{t+1}$ to solve any problem. In fact in Sutton's book the only place the 3-argument expected reward is explicitly used appears in a nondeterministic finite state diagram for a recycling robot example on page 52:
  
  [![enter image description here](https://i.sstatic.net/5UpVs.png)](https://i.sstatic.net/5UpVs.png)
  
  You see in such a diagram the expected reward function needs the explicit dependence of the entering state to conveniently express its value based on its entering state for each of the possibly branching state transition due to its random environment characterized by the above Markovian transition dynamics $p(s',r|s,a)$, and they're obviously mathematically compatible with the above usual 2-argument reward function.
  
  
  ---
  
  