Answers.Body: <p>The equation you presented is the <em>expected</em> reward function, which will always produce the same scalar value for a specific <span class="math-container">$(s, a, s')$</span> tuple. An essential concern when designing the reward signal for an RL environment is that sampling the reward signal should be relatively easy; computing the expected reward is not usually a consideration. A sample reward is still a scalar, but two sample rewards may be different even for the same <span class="math-container">$(s, a, s')$</span> tuple due to stochasticity in the dynamics function. The dynamics function details the probability of sampling a next state <span class="math-container">$s'$</span> and reward <span class="math-container">$r$</span> given the current state <span class="math-container">$s$</span> and action <span class="math-container">$a$</span>, as follows:
  <span class="math-container">\begin{align}
  &amp;p:\mathcal{S} \times \mathcal{R} \times \mathcal{S} \times \mathcal{A} \rightarrow [0, 1],\\
  &amp;p(s', r|s, a) = \mbox{Pr}(S_t{=}s', R_t{=}r\ |\ S_{t-1}{=}s, A_{t-1}{=}a).
  \end{align}</span>
  Clearly, the probability of sampling reward <span class="math-container">$r$</span> is dependent on current state <span class="math-container">$s$</span> and the action <span class="math-container">$a$</span> from the conditional notation. Sutton &amp; Barto clarify in section 3.8 remark 3.1 that the dynamics function was written as a joint probability to emphasize that the probabilities of sampling <span class="math-container">$s'$</span> and <span class="math-container">$r$</span> are <em>jointly</em> determined (not <em>independently</em> determined). Therefore, the probability of sampling reward <span class="math-container">$r$</span> generally can depend on the new state <span class="math-container">$s'$</span> in addition to the current state <span class="math-container">$s$</span> and action <span class="math-container">$a$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-06T20:39:58.803Z
Answers.LastActivityDate: 2024-09-06T20:39:58.803Z
Answers.UserDisplayName: DeepQZero
Answers.Body: <p>Yes, it is valid within the MDP framework to base immediate reward on current state, action and next state, plus a random factor. Or any subset and combination.</p>
  <p>The important detail is that the starting state and selected action must be the only non-stochastic influences on both the immediate reward <em>and</em> the next state. So for your three part function to work within the theory, the state transition rules must also follow the <span class="math-container">$p(s'|s,a)$</span> formulation strictly. If they do so, then the factor of which next state <span class="math-container">$s'$</span> we end up in before determining <span class="math-container">$r$</span> is just a fixed probability tree, and you can combined same <span class="math-container">$r$</span> values as necessary then calculate the distribution and expectations of <span class="math-container">$r$</span> independently of knowing <span class="math-container">$s'$</span>. The values you get will still <em>depend</em> on whether you know <span class="math-container">$s'$</span>, but doing the calculations for <span class="math-container">$r$</span> do not require you to know it - importantly this means you can select an optimal/greedy action based on knowing <span class="math-container">$s, a$</span> only and predicting the expected returns, without needing to know <span class="math-container">$s'$</span> in advance.</p>
  <p>Another way to put this: The values of <span class="math-container">$s'$</span> and <span class="math-container">$r$</span> can be correlated without breaking any of the rules of MDPs that are used to prove reinforcement learning works.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-06T20:23:37.797Z
Answers.LastActivityDate: 2024-09-06T20:28:59.447Z
Answers.LastEditDate: 2024-09-06T20:28:59.447Z
Answers.UserDisplayName: Neil Slater
Body: <p>I'm structuring the reward function for my RL agent and considering a combination of both the current state and the new state after taking an action. From what I understand, this is possible based on Sutton &amp; Barto, particularly Equation 3.6, where the reward is a function of the state-action pair and the resulting state. I want to ensure that my approach is in line with RL theory.</p>
  <p>Could anyone confirm if this is valid or share insights?</p>
  <p><span class="math-container">$$r(s, a, s') \doteq \mathbb{E} [R_t \mid S_{t-1} = s, A_{t-1} = a, S_t = s'] = \sum_{r \in \mathcal{R}} r \frac{p(s', r \mid s, a)}{p(s' \mid s, a)}$$</span></p>
  
Tags: <reinforcement-learning><deep-rl><markov-decision-process><rewards>
Title: Can reinforcement learning rewards be a combination of current and new state?
UserDisplayName: RookieScientist
ThreadText: # Can reinforcement learning rewards be a combination of current and new state?
  
  (Asked by: username_0 on 2024-09-06)
  
  I'm structuring the reward function for my RL agent and considering a combination of both the current state and the new state after taking an action. From what I understand, this is possible based on Sutton & Barto, particularly Equation 3.6, where the reward is a function of the state-action pair and the resulting state. I want to ensure that my approach is in line with RL theory.
  
  Could anyone confirm if this is valid or share insights?
  
  $$r(s, a, s') \doteq \mathbb{E} [R\_t \mid S\_{t-1} = s, A\_{t-1} = a, S\_t = s'] = \sum\_{r \in \mathcal{R}} r \frac{p(s', r \mid s, a)}{p(s' \mid s, a)}$$
  
  **Tags:** reinforcement-learning, deep-rl, markov-decision-process, rewards
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2024-09-06. Score: 4
  
  The equation you presented is the *expected* reward function, which will always produce the same scalar value for a specific $(s, a, s')$ tuple. An essential concern when designing the reward signal for an RL environment is that sampling the reward signal should be relatively easy; computing the expected reward is not usually a consideration. A sample reward is still a scalar, but two sample rewards may be different even for the same $(s, a, s')$ tuple due to stochasticity in the dynamics function. The dynamics function details the probability of sampling a next state $s'$ and reward $r$ given the current state $s$ and action $a$, as follows:
  \begin{align}
  &p:\mathcal{S} \times \mathcal{R} \times \mathcal{S} \times \mathcal{A} \rightarrow [0, 1],\\
  &p(s', r|s, a) = \mbox{Pr}(S\_t{=}s', R\_t{=}r\ |\ S\_{t-1}{=}s, A\_{t-1}{=}a).
  \end{align}
  Clearly, the probability of sampling reward $r$ is dependent on current state $s$ and the action $a$ from the conditional notation. Sutton & Barto clarify in section 3.8 remark 3.1 that the dynamics function was written as a joint probability to emphasize that the probabilities of sampling $s'$ and $r$ are *jointly* determined (not *independently* determined). Therefore, the probability of sampling reward $r$ generally can depend on the new state $s'$ in addition to the current state $s$ and action $a$.
  
  
  ---
  
  ### Answer by username_2 on 2024-09-06. Score: 4
  
  Yes, it is valid within the MDP framework to base immediate reward on current state, action and next state, plus a random factor. Or any subset and combination.
  
  The important detail is that the starting state and selected action must be the only non-stochastic influences on both the immediate reward *and* the next state. So for your three part function to work within the theory, the state transition rules must also follow the $p(s'|s,a)$ formulation strictly. If they do so, then the factor of which next state $s'$ we end up in before determining $r$ is just a fixed probability tree, and you can combined same $r$ values as necessary then calculate the distribution and expectations of $r$ independently of knowing $s'$. The values you get will still *depend* on whether you know $s'$, but doing the calculations for $r$ do not require you to know it - importantly this means you can select an optimal/greedy action based on knowing $s, a$ only and predicting the expected returns, without needing to know $s'$ in advance.
  
  Another way to put this: The values of $s'$ and $r$ can be correlated without breaking any of the rules of MDPs that are used to prove reinforcement learning works.
  
  
  ---
  
  