Answers.Body: <p>It's not totally clear from your description, but it sounds like you may be onto something like an <a href="https://en.wikipedia.org/wiki/Additive_Markov_chain" rel="nofollow noreferrer">Additive Markov Chain</a>.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-03-11T20:38:34.017Z
Answers.LastActivityDate: 2017-03-11T20:38:34.017Z
Answers.UserDisplayName: mindcrime
Answers.Body: <p>MPD are not independent from the past, but future actions starting from current state are independent from the past i.e. the probability of the next state given <strong>all</strong> previous states is the same as the probability of the next state given <strong>the</strong> previous state.</p>
  <p>Any state representation which is composed by the full history is a MDP, because looking at the history (coded in your state) is not the same as looking back previous states, so Markov property holds. The problem here is that you will have an explosion of states, since you need to code in the state any possible trajectory, and it is unfeasible most of the times.</p>
  <blockquote>
  <p>What if I define my state as the current &quot;main&quot; state + previous decisions?</p>
  <p>For Example in Poker the &quot;main&quot; state would be my cards and the pot + all previous information about the game.</p>
  </blockquote>
  <p>Yes it is a <strong>Markov Decision Problem</strong>.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-01-17T22:37:58.363Z
Answers.LastActivityDate: 2018-01-17T22:37:58.363Z
Answers.LastEditDate: 2020-06-17T09:57:20.710Z
Answers.UserDisplayName: Marcelo Fornet
Body: <p>As far as I know MDP are independent from the past. But the definition says that the same policy should always take the same action depending on the state.</p>
  
  <p>What if I define my state as the current "main" state + previous decisions?</p>
  
  <p>For Example in Poker the "main" state would be my cards and the pot + all previous information about the game.</p>
  
  <p>Would this still be a MDP or not? </p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-01-17T05:51:47.353Z
Comments.Text: Yeah, it's possible. If we can abstract past state into current state then MDP will still holds good.
Comments.UserDisplayName: rai.skumar
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-05-24T05:46:29.213Z
Comments.Text: But the definition says that the same policy should always take the same action depending on the state. --> this seems not right, considering stochastic policies, such as epsilon-greedy.
Comments.UserDisplayName: heyzude
Tags: <definitions><markov-chain>
Title: Can an Markov decision process be dependent on the past?
UserDisplayName: Miemels
ThreadText: # Can an Markov decision process be dependent on the past?
  
  (Asked by: username_0 on 2017-03-11)
  
  As far as I know MDP are independent from the past. But the definition says that the same policy should always take the same action depending on the state.
  
  What if I define my state as the current "main" state + previous decisions?
  
  For Example in Poker the "main" state would be my cards and the pot + all previous information about the game.
  
  Would this still be a MDP or not?
  
  **Tags:** definitions, markov-chain
  
  > **Comments:**
  > **username_1**: Yeah, it's possible. If we can abstract past state into current state then MDP will still holds good.  
  > **username_2**: But the definition says that the same policy should always take the same action depending on the state. --> this seems not right, considering stochastic policies, such as epsilon-greedy.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_4 on 2018-01-17. Score: 3 | Accepted answer
  
  MPD are not independent from the past, but future actions starting from current state are independent from the past i.e. the probability of the next state given **all** previous states is the same as the probability of the next state given **the** previous state.
  
  Any state representation which is composed by the full history is a MDP, because looking at the history (coded in your state) is not the same as looking back previous states, so Markov property holds. The problem here is that you will have an explosion of states, since you need to code in the state any possible trajectory, and it is unfeasible most of the times.
  
  > What if I define my state as the current "main" state + previous decisions?
  >
  > For Example in Poker the "main" state would be my cards and the pot + all previous information about the game.
  
  Yes it is a **Markov Decision Problem**.
  
  
  ---
  
  ### Answer by username_3 on 2017-03-11. Score: 3
  
  It's not totally clear from your description, but it sounds like you may be onto something like an [Additive Markov Chain](https://en.wikipedia.org/wiki/Additive_Markov_chain).
  
  
  ---
  
  