Answers.Body: <p>A <a href="https://en.wikipedia.org/wiki/Stochastic_process" rel="nofollow noreferrer">stochastic process</a> has the <a href="https://en.wikipedia.org/wiki/Markov_property" rel="nofollow noreferrer">Markov property</a> if the probability distribution of future states conditioned on both the present and past states depends only on the present state or, more formally, the following equality holds.</p>
  
  <p><span class="math-container">$$
  p(s_{t+1} \mid s_{t}, s_{t-1:1}) = p(s_{t+1} \mid s_{t}), \forall t
  $$</span></p>
  
  <p>The <a href="https://www.ece.ucsb.edu/Faculty/Rabiner/ece259/Reprints/tutorial%20on%20hmm%20and%20applications.pdf" rel="nofollow noreferrer">hidden Markov model (HMM)</a> is an example of a model where the Markov property is often assumed to hold. In other words, the Markov assumption is made in the case of the HMM.</p>
  
  <p>There are also the <a href="https://en.wikipedia.org/wiki/Variable-order_Markov_model" rel="nofollow noreferrer">variable-order</a> (or higher-order) Markov models, where the future state can depend on the previously <span class="math-container">$n$</span> states or, more formally, the following equality holds.</p>
  
  <p><span class="math-container">$$
  p(s_{t+1} \mid s_{t}, s_{t-1:1}) = p(s_{t+1} \mid s_{t:t-n}), \forall t
  $$</span></p>
  
  <p>In this context, a hidden Markov model is called a <strong>first-order</strong> Markov model (<span class="math-container">$n=0$</span>). Therefore, there can also be second-order (<span class="math-container">$n=1$</span>), third-order (<span class="math-container">$n=2$</span>), etc., Markov models. In fact, there are also <a href="https://arxiv.org/pdf/1805.12201.pdf" rel="nofollow noreferrer">higher-order hidden Markov models</a>.</p>
  
  <p>To conclude, the expressions <em>Markov property</em> and <em>Markov assumption</em> are not exactly interchangeable. The Markov property is an attribute that a stochastic process can be assumed to possess. In that case, the Markov assumption is made. The expression <em>Markov property</em> usually refers to a first-order Markov property, but it can more generally refer to a higher-order Markov property. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-20T15:17:35.963Z
Answers.LastActivityDate: 2019-11-20T18:32:43.050Z
Answers.LastEditDate: 2019-11-20T18:32:43.050Z
Answers.UserDisplayName: nbro
Body: <p>Does the Markov assumption say that the conditional probability of the next state only depends on the current state or does it say that the conditional probability depends on a fixed finite number of previous states?</p>
  
  <p>As far as I understand from <a href="https://en.wikipedia.org/wiki/Markov_property#Introduction" rel="nofollow noreferrer">the related Wikipedia article</a>, the probability of the next state <span class="math-container">$s'$</span> to appear only depends on the current state <span class="math-container">$s$</span>.</p>
  
  <p>However, in the book "Artificial Intelligence: A Modern Approach" by Russell and Norvig, on page 568, they say: "<strong>Markov assumption</strong> — that the current state depends on only a <em>finite fixed number</em> of previous states". </p>
  
  <p>To me, the second statement seems contradictory to the first, because it may mean that a state can depend on the history of states as long as the number is fixed a finite. For example, the current state depended on the last state and the state before the last state, which is 2 sequential previous states (a finite number of states).</p>
  
  <p>Is Markov assumption and Markov property the same?</p>
  
Tags: <reinforcement-learning><math><definitions><norvig-russell><markov-property>
Title: What does the Markov assumption say about the history of state sequences?
UserDisplayName: MScott
ThreadText: # What does the Markov assumption say about the history of state sequences?
  
  (Asked by: username_0 on 2019-11-20)
  
  Does the Markov assumption say that the conditional probability of the next state only depends on the current state or does it say that the conditional probability depends on a fixed finite number of previous states?
  
  As far as I understand from [the related Wikipedia article](https://en.wikipedia.org/wiki/Markov_property#Introduction), the probability of the next state $s'$ to appear only depends on the current state $s$.
  
  However, in the book "Artificial Intelligence: A Modern Approach" by Russell and Norvig, on page 568, they say: "**Markov assumption** — that the current state depends on only a *finite fixed number* of previous states".
  
  To me, the second statement seems contradictory to the first, because it may mean that a state can depend on the history of states as long as the number is fixed a finite. For example, the current state depended on the last state and the state before the last state, which is 2 sequential previous states (a finite number of states).
  
  Is Markov assumption and Markov property the same?
  
  **Tags:** reinforcement-learning, math, definitions, norvig-russell, markov-property
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-11-20. Score: 6 | Accepted answer
  
  A [stochastic process](https://en.wikipedia.org/wiki/Stochastic_process) has the [Markov property](https://en.wikipedia.org/wiki/Markov_property) if the probability distribution of future states conditioned on both the present and past states depends only on the present state or, more formally, the following equality holds.
  
  $$
  p(s\_{t+1} \mid s\_{t}, s\_{t-1:1}) = p(s\_{t+1} \mid s\_{t}), \forall t
  $$
  
  The [hidden Markov model (HMM)](https://www.ece.ucsb.edu/Faculty/Rabiner/ece259/Reprints/tutorial%20on%20hmm%20and%20applications.pdf) is an example of a model where the Markov property is often assumed to hold. In other words, the Markov assumption is made in the case of the HMM.
  
  There are also the [variable-order](https://en.wikipedia.org/wiki/Variable-order_Markov_model) (or higher-order) Markov models, where the future state can depend on the previously $n$ states or, more formally, the following equality holds.
  
  $$
  p(s\_{t+1} \mid s\_{t}, s\_{t-1:1}) = p(s\_{t+1} \mid s\_{t:t-n}), \forall t
  $$
  
  In this context, a hidden Markov model is called a **first-order** Markov model ($n=0$). Therefore, there can also be second-order ($n=1$), third-order ($n=2$), etc., Markov models. In fact, there are also [higher-order hidden Markov models](https://arxiv.org/pdf/1805.12201.pdf).
  
  To conclude, the expressions *Markov property* and *Markov assumption* are not exactly interchangeable. The Markov property is an attribute that a stochastic process can be assumed to possess. In that case, the Markov assumption is made. The expression *Markov property* usually refers to a first-order Markov property, but it can more generally refer to a higher-order Markov property.
  
  
  ---
  
  