Answers.Body: <p>After looking more into this, I think I have a better understanding.</p>
  <p>In <strong>Bayesian RL</strong>, one has uncertainty over the transition function of the environment. For example, we might know that our robot can slip when transitioning state, but we don't know what the probability of slip <span class="math-container">$\theta$</span> is (although we have some prior over it).</p>
  <p>In <strong>POMDPs</strong>, one has uncertainty over the current state. For example, we might not know exactly where the robot is located, and we try to understand that from observations (which are a function of the state, e.g. sensor readings).</p>
  <p>In both POMDPs and Bayesian RL, a similar trick applies: one can &quot;tuck one's uncertainty into the state&quot;, reducing both problems to MDPs. This is nice because we already have a bunch of methods developed to solve MDPs, so we don't have to reinvent the wheel. However, these are a particular type of MDP: in which each state corresponds to a specific <strong>&quot;belief-state&quot;</strong>.</p>
  <ul>
  <li>In <strong>POMDPs</strong>, each belief-state will only contain a belief over which state one is (e.g. a distribution over x-y locations).</li>
  <li>In <strong>Bayesian RL</strong>, each belief-state will contain both a state <em>and</em> a belief over the unknown parameters of the transition dynamics (e.g. a Beta distribution over <span class="math-container">$\theta$</span>).</li>
  </ul>
  <p>How the transition dynamics are computed in POMDPs and Bayesian RL are obtained is also subtly different, but the idea is similar: one simply incorporates the latest information one gains through observations, e.g. &quot;did the robot slip this last timestep?&quot;, or &quot;what were my last sensor readings this last timestep?&quot;.</p>
  <p>For more information on POMDPs, see &quot;Planning and acting in partially observable stochastic domains&quot; by Kaelbling. For more info on Bayesian RL, <a href="https://www.youtube.com/watch?v=ar9RLwgUvVQ" rel="nofollow noreferrer">this lecture</a> seems like a good starting point.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-06T00:46:28.110Z
Answers.LastActivityDate: 2022-08-06T00:46:28.110Z
Answers.UserDisplayName: mdc
Body: <p><a href="https://www.youtube.com/watch?v=ar9RLwgUvVQ" rel="nofollow noreferrer">Bayesian RL</a> seems concerned with having uncertainty over the transition function of the environment, while POMDPs try to capture uncertainty over the state one is currently in. However, both end up reducing to something like a belief MDP: an MDP in which everything is known, but the states are points in belief space.</p>
  <p>Could anyone explain what the differences between these are in more detail?</p>
  
Tags: <reinforcement-learning><markov-decision-process><pomdp><bayesian-reinforcement-learning>
Title: What's the relationship between Bayesian RL and POMDPs?
UserDisplayName: mdc
ThreadText: # What's the relationship between Bayesian RL and POMDPs?
  
  (Asked by: username_0 on 2022-08-06)
  
  [Bayesian RL](https://www.youtube.com/watch?v=ar9RLwgUvVQ) seems concerned with having uncertainty over the transition function of the environment, while POMDPs try to capture uncertainty over the state one is currently in. However, both end up reducing to something like a belief MDP: an MDP in which everything is known, but the states are points in belief space.
  
  Could anyone explain what the differences between these are in more detail?
  
  **Tags:** reinforcement-learning, markov-decision-process, pomdp, bayesian-reinforcement-learning
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2022-08-06. Score: 3 | Accepted answer
  
  After looking more into this, I think I have a better understanding.
  
  In **Bayesian RL**, one has uncertainty over the transition function of the environment. For example, we might know that our robot can slip when transitioning state, but we don't know what the probability of slip $\theta$ is (although we have some prior over it).
  
  In **POMDPs**, one has uncertainty over the current state. For example, we might not know exactly where the robot is located, and we try to understand that from observations (which are a function of the state, e.g. sensor readings).
  
  In both POMDPs and Bayesian RL, a similar trick applies: one can "tuck one's uncertainty into the state", reducing both problems to MDPs. This is nice because we already have a bunch of methods developed to solve MDPs, so we don't have to reinvent the wheel. However, these are a particular type of MDP: in which each state corresponds to a specific **"belief-state"**.
  
  * In **POMDPs**, each belief-state will only contain a belief over which state one is (e.g. a distribution over x-y locations).
  * In **Bayesian RL**, each belief-state will contain both a state *and* a belief over the unknown parameters of the transition dynamics (e.g. a Beta distribution over $\theta$).
  
  How the transition dynamics are computed in POMDPs and Bayesian RL are obtained is also subtly different, but the idea is similar: one simply incorporates the latest information one gains through observations, e.g. "did the robot slip this last timestep?", or "what were my last sensor readings this last timestep?".
  
  For more information on POMDPs, see "Planning and acting in partially observable stochastic domains" by Kaelbling. For more info on Bayesian RL, [this lecture](https://www.youtube.com/watch?v=ar9RLwgUvVQ) seems like a good starting point.
  
  
  ---
  
  