Answers.Body: <p>The highlighted derivation here is simplifying the originally defined conditional probability via the state-space model (5-1), where we can know <span class="math-container">$s_1 = f(s_0, a_0, \xi_0)$</span>. Then you need to know the reason why the author here first replace <span class="math-container">$s_{t+1}$</span> with <span class="math-container">$f(s_t, a_t, \xi_t)$</span> and then revert back at the last step.</p>
  <p>Once a specifically structured environment function <span class="math-container">$f(.)$</span> is given, the probability of the function value of a any fixed input vector, say <span class="math-container">$f(s_t, a_t, \xi_t)$</span>, would not depend on any other function value <span class="math-container">$f(s_0, a_0, \xi_0)$</span> <em>as long as</em> the three parameters are <em>mutually</em> pairwise independent, which is the case here since as stated in your reference the random noise <span class="math-container">$\xi_t$</span> is assumed to be independent of its historical counterparts and initial state <span class="math-container">$s_0$</span>, and each action <span class="math-container">$a_t$</span> is arbitrary and independent of historical states and random noises. This is also why the author needs to weirdly replace <span class="math-container">$s_2$</span> as nested <span class="math-container">$f(f(s_0, a_0, \xi_0), a_1, \xi_1)$</span> to reduce to the initial state <span class="math-container">$s_0$</span> which is a <em>constant</em> and all noises are independent on it as already assumed explicitly above. This process continues until you have the obvious state <span class="math-container">$s_t$</span> dependence on itself in the penultimate step.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-13T22:41:24.800Z
Answers.LastActivityDate: 2023-11-13T22:52:09.577Z
Answers.LastEditDate: 2023-11-13T22:52:09.577Z
Answers.UserDisplayName: cinch
Body: <p>Below is the derivation for the Markov property of a general state-space model.
  The red part is not clear. Could someone please explain the steps in the sequential derivation for the red part?</p>
  <p><a href="https://i.sstatic.net/NiLrP.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/NiLrP.png" alt="enter image description here" /></a></p>
  
Tags: <markov-decision-process><dynamic-programming><markov-property>
Title: How is the Markov property of a general state-space model derived?
UserDisplayName: DSPinfinity
ThreadText: # How is the Markov property of a general state-space model derived?
  
  (Asked by: username_0 on 2023-06-03)
  
  Below is the derivation for the Markov property of a general state-space model.
  The red part is not clear. Could someone please explain the steps in the sequential derivation for the red part?
  
  [![enter image description here](https://i.sstatic.net/NiLrP.png)](https://i.sstatic.net/NiLrP.png)
  
  **Tags:** markov-decision-process, dynamic-programming, markov-property
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-11-13. Score: 0
  
  The highlighted derivation here is simplifying the originally defined conditional probability via the state-space model (5-1), where we can know $s\_1 = f(s\_0, a\_0, \xi\_0)$. Then you need to know the reason why the author here first replace $s\_{t+1}$ with $f(s\_t, a\_t, \xi\_t)$ and then revert back at the last step.
  
  Once a specifically structured environment function $f(.)$ is given, the probability of the function value of a any fixed input vector, say $f(s\_t, a\_t, \xi\_t)$, would not depend on any other function value $f(s\_0, a\_0, \xi\_0)$ *as long as* the three parameters are *mutually* pairwise independent, which is the case here since as stated in your reference the random noise $\xi\_t$ is assumed to be independent of its historical counterparts and initial state $s\_0$, and each action $a\_t$ is arbitrary and independent of historical states and random noises. This is also why the author needs to weirdly replace $s\_2$ as nested $f(f(s\_0, a\_0, \xi\_0), a\_1, \xi\_1)$ to reduce to the initial state $s\_0$ which is a *constant* and all noises are independent on it as already assumed explicitly above. This process continues until you have the obvious state $s\_t$ dependence on itself in the penultimate step.
  
  
  ---
  
  