Answers.Body: <p><span class="math-container">$O(a, s', z) = \mathbb{P}(z \mid a, s')$</span> is a <a href="https://en.wikipedia.org/wiki/Conditional_probability_distribution" rel="nofollow noreferrer">conditional probability distribution</a>, so it always needs to sum up to <span class="math-container">$1$</span>. You should interpret <span class="math-container">$O(a, s', z)$</span> as the probability of observation <span class="math-container">$z$</span>, given that the agent took action <span class="math-container">$a$</span> and landed in state <span class="math-container">$s'$</span>. </p>
  
  <p><span class="math-container">$O(a, s', z)$</span> is thus <strong>not</strong> a <a href="https://en.wikipedia.org/wiki/Joint_probability_distribution" rel="nofollow noreferrer">joint distribution</a>, even though the notation <span class="math-container">$O(a, s', z)$</span> might suggest it. In this case, <span class="math-container">$O(a, s', z)$</span> simply means that <span class="math-container">$O$</span> is a function of <span class="math-container">$a$</span>, <span class="math-container">$z$</span> and <span class="math-container">$s'$</span>.</p>
  
  <p>If you want to see a proof that conditional probability distributions sum up to 1, have a look at <a href="https://math.stackexchange.com/q/2084313/168764">this post</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-28T17:22:54.353Z
Answers.Text: But does it still mean, that I have to observe a state somehow, even the information might not be accurate?
Answers.UserDisplayName: Bryan McGill
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-28T17:25:10.193Z
Answers.Text: @BryanMcGill You don't observe any state (in POMDPs). You get "observations", which you use to model your belief in which state you might be in.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-28T17:32:55.213Z
Answers.Text: That clarifies everythinga. I came along some illustrations where "observations" were connected to states and I got confused. Maybe you could edit your answer regarding that one sentence in the beginning.
Answers.UserDisplayName: Bryan McGill
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-08T21:39:28.627Z
Answers.LastActivityDate: 2019-03-08T21:51:46.307Z
Answers.LastEditDate: 2019-03-08T21:51:46.307Z
Answers.UserDisplayName: nbro
Body: <p>I was reading in the article <em><a href="https://www.sciencedirect.com/science/article/abs/pii/S0022249609000042" rel="nofollow noreferrer">A tutorial on partially observable Markov decision processes</a></em> (p. 120), by Michael L. Littman, that <span class="math-container">$\sum_{z \in Z}O(a, s',z) =1$</span>, where <span class="math-container">$a$</span> is the action, <span class="math-container">$s'$</span> the next possible state and <span class="math-container">$z$</span> a certain/specific observation.</p>
  
  <p>How come that the observation function <span class="math-container">$O(a, s', z)$</span> adds up to <span class="math-container">$1$</span> in POMDP?</p>
  
Tags: <reinforcement-learning><markov-decision-process><pomdp>
Title: Does the observation function for POMDP always add up to 1?
UserDisplayName: Bryan McGill
ThreadText: # Does the observation function for POMDP always add up to 1?
  
  (Asked by: username_0 on 2019-03-08)
  
  I was reading in the article *[A tutorial on partially observable Markov decision processes](https://www.sciencedirect.com/science/article/abs/pii/S0022249609000042)* (p. 120), by Michael L. Littman, that $\sum\_{z \in Z}O(a, s',z) =1$, where $a$ is the action, $s'$ the next possible state and $z$ a certain/specific observation.
  
  How come that the observation function $O(a, s', z)$ adds up to $1$ in POMDP?
  
  **Tags:** reinforcement-learning, markov-decision-process, pomdp
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-03-08. Score: 2 | Accepted answer
  
  $O(a, s', z) = \mathbb{P}(z \mid a, s')$ is a [conditional probability distribution](https://en.wikipedia.org/wiki/Conditional_probability_distribution), so it always needs to sum up to $1$. You should interpret $O(a, s', z)$ as the probability of observation $z$, given that the agent took action $a$ and landed in state $s'$.
  
  $O(a, s', z)$ is thus **not** a [joint distribution](https://en.wikipedia.org/wiki/Joint_probability_distribution), even though the notation $O(a, s', z)$ might suggest it. In this case, $O(a, s', z)$ simply means that $O$ is a function of $a$, $z$ and $s'$.
  
  If you want to see a proof that conditional probability distributions sum up to 1, have a look at [this post](https://math.stackexchange.com/q/2084313/168764).
  
  > **Comments:**
  > **username_0**: But does it still mean, that I have to observe a state somehow, even the information might not be accurate?  
  > **username_1**: @username_0 You don't observe any state (in POMDPs). You get "observations", which you use to model your belief in which state you might be in.  
  > **username_0**: That clarifies everythinga. I came along some illustrations where "observations" were connected to states and I got confused. Maybe you could edit your answer regarding that one sentence in the beginning.  
  
  ---
  
  