Answers.Body: <p>Ok, so I was able to find the answer to this question by myself. So I'm sharing it with everyone. By the law of non-exclusive events, <span class="math-container">$P(A|B) = \frac{P(A,B)}{P(B)}$</span> where <span class="math-container">$P(B) &gt; 0$</span>. This can also be written as, <span class="math-container">$P(A|B,C,D) = \frac{P(A,B|C,D)}{P(B|C,D)}$</span> (just adding C,D as events that are already happened). In this case we have, <span class="math-container">$ p(r|s',s,a) = \frac{p(s',r|s,a)}{p(s'|s,a)}$</span> where by definition <span class="math-container">$p(s'|s,a) &gt; 0$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-25T16:53:52.857Z
Answers.LastActivityDate: 2024-02-25T16:53:52.857Z
Answers.UserDisplayName: Jahid Chowdhury Choton
Answers.Body: <p>You've correctly figured out the most part of your confusion. The only thing I want to add is that by the <em>finite</em> MDP definition here it's certainly possible <span class="math-container">$p(s'|s,a)=0$</span> for some <span class="math-container">$s',s$</span>. The reason you can avoid infinite reward lies in the implicit logic of conditional probability, you cannot arrive at a <em>definite</em> posterior updated <span class="math-container">$P(A|B,C,D)$</span> via your above equation when you already know <span class="math-container">$P(B|C,D)=0$</span>. Since the event <span class="math-container">$B$</span> here is like a premise or assumption, and if it's already false (no chance to happen) then any derivation based on it can be true, that is, <span class="math-container">$P(A|B,C,D)$</span> cannot be definite which certainly is useless for any conceivable application and you have to exclude such priors in deriving expected reward above.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-26T20:42:02.020Z
Answers.LastActivityDate: 2024-02-26T20:42:02.020Z
Answers.UserDisplayName: cinch
Body: <p>In chapter 2 of Barto and Sutton's RL book, the four argument probability function <span class="math-container">$p: S \times R \times S \times A \to [0,1]$</span> is reduced to three arguments <span class="math-container">$p: S \times S \times A \to [0,1]$</span> as follows,
  <span class="math-container">$$ P(s'|s,a) = \sum_{r\in R}p(s',r|s,a) $$</span>
  Therefore, the expected rewards for state-action pairs are defined as a two-argument function <span class="math-container">$r: S \times A \to \mathbb{R} $</span>:
  <span class="math-container">$$ r(s,a) = \sum_{r \in R} r \sum_{s'\in S} p(s',r|s,a) $$</span></p>
  <p>Up to this I understood everything and have no problem. My problem arises when in the next line they defined the expected rewards for state-action-next state triples as <span class="math-container">$r: S \times A \times S \to \mathbb{R} $</span> where,</p>
  <p><span class="math-container">$$ r(s,a,s') = \sum_{r\in R} r \frac{p(s',r|s,a)}{p(s'|s,a)} $$</span></p>
  <p>I do not know how to derive this and it seems very counter-intuitive for me. It is because what if the state <span class="math-container">$s'$</span> is unreachable? Meaning there are no arrows in the MDP from s to s' (<span class="math-container">$p(s′|s,a)=0$</span>). Does it mean that the reward would be infinity? <span class="math-container">$r(s,a,s′)=\infty$</span>? Or is it undefined for this? As in RL we always try to maximize rewards, taking an infinity reward will always makes it maximum (which makes no sense). It would be very nice if somebody can explain this equation with step-by-step derivation.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-02-24T22:43:24.293Z
Comments.Text: $$ r(s',s,a) = \sum_{r\in R}rp(r|s',s,a) $$. also assume that p(s'|s,a)>0. for conciseness some times notations are simplified. for instance as mentioned in the previous page of the book A is used instead of A(s).
Comments.UserDisplayName: Reza_va
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-02-25T16:54:27.850Z
Comments.Text: You're right. I missed that by definition $p(s'|s,a)>0$.
Comments.UserDisplayName: Jahid Chowdhury Choton
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-02-26T10:52:36.800Z
Comments.Text: Does this answer your question? [Why is the equation $r(s', a, s') =\sum_{r \in \mathcal{R}} r \frac{p\left(s^{\prime}, r \mid s, a\right)}{p\left(s^{\prime} \mid s, a\right)}$true?](https://ai.stackexchange.com/questions/8514/why-is-the-equation-rs-a-s-sum-r-in-mathcalr-r-fracp-lefts-p) It might not be an exact duplicate, but the questions are very similar and it might be a good idea to at least link them.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-03-01T16:36:47.787Z
Comments.Text: yes it does. Many thanks for referring to it!
Comments.UserDisplayName: Jahid Chowdhury Choton
Tags: <reinforcement-learning><markov-decision-process><reward-functions><probability-distribution>
Title: If $p(s'|s,a) = 0$, would the reward the reward $r(s,a,s')$ be infinite?
UserDisplayName: Jahid Chowdhury Choton
ThreadText: # If $p(s'|s,a) = 0$, would the reward the reward $r(s,a,s')$ be infinite?
  
  (Asked by: username_0 on 2024-02-24)
  
  In chapter 2 of Barto and Sutton's RL book, the four argument probability function $p: S \times R \times S \times A \to [0,1]$ is reduced to three arguments $p: S \times S \times A \to [0,1]$ as follows,
  $$ P(s'|s,a) = \sum\_{r\in R}p(s',r|s,a) $$
  Therefore, the expected rewards for state-action pairs are defined as a two-argument function $r: S \times A \to \mathbb{R} $:
  $$ r(s,a) = \sum\_{r \in R} r \sum\_{s'\in S} p(s',r|s,a) $$
  
  Up to this I understood everything and have no problem. My problem arises when in the next line they defined the expected rewards for state-action-next state triples as $r: S \times A \times S \to \mathbb{R} $ where,
  
  $$ r(s,a,s') = \sum\_{r\in R} r \frac{p(s',r|s,a)}{p(s'|s,a)} $$
  
  I do not know how to derive this and it seems very counter-intuitive for me. It is because what if the state $s'$ is unreachable? Meaning there are no arrows in the MDP from s to s' ($p(s′|s,a)=0$). Does it mean that the reward would be infinity? $r(s,a,s′)=\infty$? Or is it undefined for this? As in RL we always try to maximize rewards, taking an infinity reward will always makes it maximum (which makes no sense). It would be very nice if somebody can explain this equation with step-by-step derivation.
  
  **Tags:** reinforcement-learning, markov-decision-process, reward-functions, probability-distribution
  > **Comments:**
  > **username_1**: $$ r(s',s,a) = \sum_{r\in R}rp(r|s',s,a) $$. also assume that p(s'|s,a)>0. for conciseness some times notations are simplified. for instance as mentioned in the previous page of the book A is used instead of A(s).  
  > **username_0**: You're right. I missed that by definition $p(s'|s,a)>0$.  
  > **username_2**: Does this answer your question? [Why is the equation $r(s', a, s') =\sum_{r \in \mathcal{R}} r \frac{p\left(s^{\prime}, r \mid s, a\right)}{p\left(s^{\prime} \mid s, a\right)}$true?](https://ai.stackexchange.com/questions/8514/why-is-the-equation-rs-a-s-sum-r-in-mathcalr-r-fracp-lefts-p) It might not be an exact duplicate, but the questions are very similar and it might be a good idea to at least link them.  
  > **username_0**: yes it does. Many thanks for referring to it!  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_0 on 2024-02-25. Score: 1 | Accepted answer
  
  Ok, so I was able to find the answer to this question by myself. So I'm sharing it with everyone. By the law of non-exclusive events, $P(A|B) = \frac{P(A,B)}{P(B)}$ where $P(B) > 0$. This can also be written as, $P(A|B,C,D) = \frac{P(A,B|C,D)}{P(B|C,D)}$ (just adding C,D as events that are already happened). In this case we have, $ p(r|s',s,a) = \frac{p(s',r|s,a)}{p(s'|s,a)}$ where by definition $p(s'|s,a) > 0$.
  
  
  ---
  
  ### Answer by username_3 on 2024-02-26. Score: 2
  
  You've correctly figured out the most part of your confusion. The only thing I want to add is that by the *finite* MDP definition here it's certainly possible $p(s'|s,a)=0$ for some $s',s$. The reason you can avoid infinite reward lies in the implicit logic of conditional probability, you cannot arrive at a *definite* posterior updated $P(A|B,C,D)$ via your above equation when you already know $P(B|C,D)=0$. Since the event $B$ here is like a premise or assumption, and if it's already false (no chance to happen) then any derivation based on it can be true, that is, $P(A|B,C,D)$ cannot be definite which certainly is useless for any conceivable application and you have to exclude such priors in deriving expected reward above.
  
  
  ---
  
  