Body: <p>How can I update the observation probability for a POMDP (or HMM), in order to have a more accurate prediction model?</p>
  <p>The POMDP relies on observation probabilities that match an observation to a state. This poses an issue as the probabilities are not exactly known. However, the idea is to make them more accurate over time. The simplest idea would be to count the appeared observation as well as the states and use Naive Bayes estimators.</p>
  <p>For example, <span class="math-container">$P(s' \mid a,s)$</span> is the probability that a subsequent state <span class="math-container">$s'$</span> is reached, given that the action <span class="math-container">$a$</span> and the previous state <span class="math-container">$s$</span> are known: In that simple case I can just count and then apply e.g. Naive Bayes estimators.</p>
  <p>But, if I have an observation probability <span class="math-container">$P(z \mid s')$</span> (where <span class="math-container">$z$</span> is the observation) depending on a state, it's not as trivial to just count up the observation and the states, as I can not say that a state really was reached (Maybe I made an observation, but I was in a different state than wanted). I can just make an observation and hope I was in a certain state. But I can not say if e.g. I was in <span class="math-container">$s_1$</span> or maybe <span class="math-container">$s_2$</span>. I think the update of the observation probability is only possible in the late aftermath.</p>
  <p>So, what are good approaches to estimate my state?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-08T00:05:24.263Z
Comments.Text: Ok, now I think I fully understand this post. You want to estimate the probability distribution over the observations given e.g. the next states, $P(z \mid s')$, so that you can perform more accurate calculations (e.g. of the belief states) when that probability distribution is required. However, you don't know exactly in which state you are in, so you think that you cannot estimate $P(z \mid s')$, and that's why you're asking how you can estimate the state you're in. This looks like a circular problem.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-08T10:27:09.787Z
Comments.Text: Exactly, it is some sort of a circular problem, because the reliability of the observation is crucial to estimate the state. But if I can't even trust the observation (if I can't even trust my eyes) how will I determine that I was in a certain state. But I think, there must be a way to handle that problem, because how will ever a POMDP work, when I can't estimate my observation probability or if I can't have a more accurate propability distribution that matches the real systems behaviour.
Comments.UserDisplayName: Pluxyy
Tags: <reinforcement-learning><pomdp><state-spaces><conditional-probability>
Title: How to update the observation probabilities in a POMDP?
UserDisplayName: Pluxyy
ThreadText: # How to update the observation probabilities in a POMDP?
  
  (Asked by: username_0 on 2021-01-07)
  
  How can I update the observation probability for a POMDP (or HMM), in order to have a more accurate prediction model?
  
  The POMDP relies on observation probabilities that match an observation to a state. This poses an issue as the probabilities are not exactly known. However, the idea is to make them more accurate over time. The simplest idea would be to count the appeared observation as well as the states and use Naive Bayes estimators.
  
  For example, $P(s' \mid a,s)$ is the probability that a subsequent state $s'$ is reached, given that the action $a$ and the previous state $s$ are known: In that simple case I can just count and then apply e.g. Naive Bayes estimators.
  
  But, if I have an observation probability $P(z \mid s')$ (where $z$ is the observation) depending on a state, it's not as trivial to just count up the observation and the states, as I can not say that a state really was reached (Maybe I made an observation, but I was in a different state than wanted). I can just make an observation and hope I was in a certain state. But I can not say if e.g. I was in $s\_1$ or maybe $s\_2$. I think the update of the observation probability is only possible in the late aftermath.
  
  So, what are good approaches to estimate my state?
  
  **Tags:** reinforcement-learning, pomdp, state-spaces, conditional-probability
  
  > **Comments:**
  > **username_1**: Ok, now I think I fully understand this post. You want to estimate the probability distribution over the observations given e.g. the next states, $P(z \mid s')$, so that you can perform more accurate calculations (e.g. of the belief states) when that probability distribution is required. However, you don't know exactly in which state you are in, so you think that you cannot estimate $P(z \mid s')$, and that's why you're asking how you can estimate the state you're in. This looks like a circular problem.  
  > **username_0**: Exactly, it is some sort of a circular problem, because the reliability of the observation is crucial to estimate the state. But if I can't even trust the observation (if I can't even trust my eyes) how will I determine that I was in a certain state. But I think, there must be a way to handle that problem, because how will ever a POMDP work, when I can't estimate my observation probability or if I can't have a more accurate propability distribution that matches the real systems behaviour.  
  
  ---
  
  ## 0 answers
  
  