Answers.Body: <p>The main difference between an MDP and contextual bandit setting is time steps and state progression. If those are important to the problem you want to solve, then it is not possible to convert.</p>
  <p>Essentially MDPs are a strict generalisation of contextual bandits. You can model a CB as an MDP but not vice-versa.</p>
  <p>In some very specific cases you can convert MDP to CB - any one of these situations means that the MDP can be simplified to CB, and then you could use bandit solving algorithms to optimise it:</p>
  <ul>
  <li><p>When there is only one time step for an episodic problem.</p>
  </li>
  <li><p>When the discount factor is zero.</p>
  </li>
  <li><p>When state transition rules are completely independent of action choice, but reward is not.</p>
  </li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-18T13:37:44.127Z
Answers.Text: Thanks for your reply. If you have criteria #2 and #3, does #1 even matter any more? - I think that basically makes it a non-episodic problem? - Also MAB actually can have a non-zero discount factor.
Answers.UserDisplayName: peidaqi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-18T15:10:40.720Z
Answers.Text: It is any one of the criteria. The discount factor in MAB is not the same thing as in MDP
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-17T10:44:18.813Z
Answers.LastActivityDate: 2020-08-19T17:14:45.487Z
Answers.LastEditDate: 2020-08-19T17:14:45.487Z
Answers.UserDisplayName: Neil Slater
Body: <p>I'm trying to get a better understanding of Multi-Arm Bandits, Contextual Multi-Arm Bandits and Markov Decision Process.</p>
  <p>Basically, Multi-Arm Bandits is a special case of Contextual Multi-Arm Bandits where there is no state(features/context). And Contextual Multi-Arm Bandits is a special case of Markov Decision Process, where there is only one state (features, but no transitions).</p>
  <p>However, since MDP has Markov property, I wonder if every MDP problem can also be converted into a Contextual Multi-Arm Bandits problem, if we simply treat each state as a different input context (features)?</p>
  
Tags: <reinforcement-learning><comparison><markov-decision-process><multi-armed-bandits><contextual-bandits>
Title: Can you convert a MDP problem to a Contextual Multi-Arm Bandits problem?
UserDisplayName: peidaqi
ThreadText: # Can you convert a MDP problem to a Contextual Multi-Arm Bandits problem?
  
  (Asked by: username_0 on 2020-08-17)
  
  I'm trying to get a better understanding of Multi-Arm Bandits, Contextual Multi-Arm Bandits and Markov Decision Process.
  
  Basically, Multi-Arm Bandits is a special case of Contextual Multi-Arm Bandits where there is no state(features/context). And Contextual Multi-Arm Bandits is a special case of Markov Decision Process, where there is only one state (features, but no transitions).
  
  However, since MDP has Markov property, I wonder if every MDP problem can also be converted into a Contextual Multi-Arm Bandits problem, if we simply treat each state as a different input context (features)?
  
  **Tags:** reinforcement-learning, comparison, markov-decision-process, multi-armed-bandits, contextual-bandits
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-08-17. Score: 4
  
  The main difference between an MDP and contextual bandit setting is time steps and state progression. If those are important to the problem you want to solve, then it is not possible to convert.
  
  Essentially MDPs are a strict generalisation of contextual bandits. You can model a CB as an MDP but not vice-versa.
  
  In some very specific cases you can convert MDP to CB - any one of these situations means that the MDP can be simplified to CB, and then you could use bandit solving algorithms to optimise it:
  
  * When there is only one time step for an episodic problem.
  * When the discount factor is zero.
  * When state transition rules are completely independent of action choice, but reward is not.
  
  > **Comments:**
  > **username_0**: Thanks for your reply. If you have criteria #2 and #3, does #1 even matter any more? - I think that basically makes it a non-episodic problem? - Also MAB actually can have a non-zero discount factor.  
  > **username_1**: It is any one of the criteria. The discount factor in MAB is not the same thing as in MDP  
  
  ---
  
  