Answers.Body: <p>Both <strong>Belief-MDPs</strong> and <strong>Bayes-Adaptive MDPs (BAMDPs)</strong> are special cases of <strong>POMDPs</strong> and their state space is <strong>augmented with a belief</strong> over their unobserved/hidden variables.</p>
  <p>In a belief-MDP, the hidden variables <strong>can change</strong> over the course of an episode. (Eg. Both the position and the uncertainty in the position of the robot can vary during an episode).</p>
  <p>In a  BAMDP, the hidden variables are usually the attributes of the transition/reward function and are <strong>held constant</strong> during an episode. (Eg. In a robot locomotion task, ground friction or load - dynamics attributes, current goal location - reward function attributes. Though variables defining these attributes are unknown to the agent and the agent has to infer them, the actual variables remain unchanged during an episode)</p>
  <p><strong>References:</strong></p>
  <ol>
  <li><p>M Ghavamzadeh, S Mannor, J Pineau, Aviv Tamar - <a href="https://arxiv.org/pdf/1609.04436.pdf" rel="nofollow noreferrer">Bayesian reinforcement learning: A survey</a>. Foundations and Trends in Machine Learning, 8(5-6):359-483, 2015.</p>
  </li>
  <li><p>L Zintgraf, K Shiarlis, M Igl, S Schulze, Y Gal, K Hofmann, and S Whiteson. <a href="https://arxiv.org/pdf/1910.08348.pdf" rel="nofollow noreferrer">Varibad: A very good method for bayes-adaptive deep rl via meta-learning</a>. arXiv preprint arXiv:1910.08348, 2019.</p>
  </li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-21T17:43:55.997Z
Answers.LastActivityDate: 2021-02-21T23:00:25.560Z
Answers.LastEditDate: 2021-02-21T23:00:25.560Z
Answers.UserDisplayName: Mohith Kaameswaran Sakthivel
Body: <p>I have been reading a few papers in this area recently and I keep coming across these two terms. As far as I'm aware, Belief-MDPs are when you cast a POMDP as a regular MDP with a continuous state space where the state is a belief (distribution) with some unknown parameters.</p>
  <p>Are they not the same thing?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-27T13:14:36.123Z
Comments.Text: Hi. Maybe, for completeness, you should also explain where you heard the term "Bayes-adaptive MDP".
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><comparison><markov-decision-process><pomdp>
Title: What is the difference between Bayes-adaptive MDP and a Belief-MDP in Reinforcement Learning?
UserDisplayName: gauzah
ThreadText: # What is the difference between Bayes-adaptive MDP and a Belief-MDP in Reinforcement Learning?
  
  (Asked by: username_0 on 2020-07-27)
  
  I have been reading a few papers in this area recently and I keep coming across these two terms. As far as I'm aware, Belief-MDPs are when you cast a POMDP as a regular MDP with a continuous state space where the state is a belief (distribution) with some unknown parameters.
  
  Are they not the same thing?
  
  **Tags:** reinforcement-learning, comparison, markov-decision-process, pomdp
  
  > **Comments:**
  > **username_1**: Hi. Maybe, for completeness, you should also explain where you heard the term "Bayes-adaptive MDP".  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-02-21. Score: 2
  
  Both **Belief-MDPs** and **Bayes-Adaptive MDPs (BAMDPs)** are special cases of **POMDPs** and their state space is **augmented with a belief** over their unobserved/hidden variables.
  
  In a belief-MDP, the hidden variables **can change** over the course of an episode. (Eg. Both the position and the uncertainty in the position of the robot can vary during an episode).
  
  In a BAMDP, the hidden variables are usually the attributes of the transition/reward function and are **held constant** during an episode. (Eg. In a robot locomotion task, ground friction or load - dynamics attributes, current goal location - reward function attributes. Though variables defining these attributes are unknown to the agent and the agent has to infer them, the actual variables remain unchanged during an episode)
  
  **References:**
  
  1. M Ghavamzadeh, S Mannor, J Pineau, Aviv Tamar - [Bayesian reinforcement learning: A survey](https://arxiv.org/pdf/1609.04436.pdf). Foundations and Trends in Machine Learning, 8(5-6):359-483, 2015.
  2. L Zintgraf, K Shiarlis, M Igl, S Schulze, Y Gal, K Hofmann, and S Whiteson. [Varibad: A very good method for bayes-adaptive deep rl via meta-learning](https://arxiv.org/pdf/1910.08348.pdf). arXiv preprint arXiv:1910.08348, 2019.
  
  
  ---
  
  