Answers.Body: <blockquote>
  <p>all definitions I find say that the limiting distribution does not depend on the initial distribution</p>
  </blockquote>
  <p>The proof of this trait of the limit relies on the asumption of <a href="https://ai.stackexchange.com/questions/27196/what-is-ergodicity-in-a-markov-decision-process-mdp">ergodicity</a>.</p>
  <p>You can define an MDP where there are non-interacting splits in trajectory (i.e. a branch in trajectory with no way of reaching outside of a subset of states), with two or more recurrent loops, plus an initial distribution where it is possible to reach at least two split points. This is essentially multiple ergodic MDPs in a branching structure. In that case, <em>within each split</em>, the limiting distribution is independent of the starting distribution.</p>
  <p>In your example case, everything ends up in B &amp; C or D &amp; E. The ratio of times spent between BC and DE depend on the starting distribution. Within those groups, the ratios of time B:C and D:E are fixed and converge independently of any starting distribution.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-24T13:47:13.157Z
Answers.Text: Can a limiting distribution independent of the initial state exist in a non-ergodic chain? Because I usually read "a limting distribution is the limit of the distribution, independent of the initial state". And then "if the chain is ergodic, the limiting distribution exists and it's unique". The definition of limiting distribution seems unrelated to the one of ergodicity, and one that could exist in non-ergodic chains. But you said it depends on ergodicity, which confuses me (makes sense in terms of independency of the initial state, just doesn't seem to fit the literature? Any ref?).
Answers.UserDisplayName: Simon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-24T19:00:41.193Z
Answers.Text: @Simon as you said, ergodicity gives you guarantees that it exists and that is unique… in your example, it exists but it’s not unique, and furthermore depends on the initial state
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-24T21:02:17.960Z
Answers.Text: @Alberto Right, so the limiting distribution, by definition, can be not unique, right? Because I often read "if it exists, it is unique" in its definition. That's why I was confused.
Answers.UserDisplayName: Simon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-25T02:07:12.663Z
Answers.Text: @Simon the fact that the limiting distribution depends on the initial state implies that the couple MC+initial state has a limiting distribution, though you probably cannot prove that it is always the case... generally speaking however, a limiting distribution exists if and only if it does not depend on the initial state (by definition)
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-24T09:20:51.180Z
Answers.LastActivityDate: 2024-11-24T19:18:42.643Z
Answers.LastEditDate: 2024-11-24T19:18:42.643Z
Answers.UserDisplayName: Neil Slater
Answers.Body: <p>Indeed the limiting behavior of a non-ergodic Markov chain which is not irreducible with multiple mean positive recurrent communicating classes depends on its initial distribution, since apparently the state space can be decomposed into multiple communicating classes thus the limiting distribution <em>if exists</em> depends on which class is activated by the initial distribution and typically there's one limiting distribution per recurrent communicating class <em>if</em> all states within the said communicating class are aperiodic.</p>
  <p>In summary without irreducibility the limiting distribution if exists is not unique, and without aperiodicity a mere <em>stationary</em> distribution may not be converged as a <em>limiting</em> distribution as illustrated in the answer to your recent <a href="https://ai.stackexchange.com/questions/47263/conditions-for-the-existence-of-stationary-state-distribution">post</a> here. There's conceptual difference between a limiting and stationary distribution of a MC.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-25T08:18:00.380Z
Answers.Text: Thanks! Your reply helped understanding Neil's better. I think both answers are useful, I accepted Neil's just because he replied first.
Answers.UserDisplayName: Simon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-24T23:19:33.467Z
Answers.LastActivityDate: 2024-11-24T23:19:33.467Z
Answers.UserDisplayName: cinch
Body: <p>I am a bit confused about the definition of limiting distribution in Markov chains.</p>
  <p>My understanding is that it represents the behavior of the chain in-the-limit. That is, I start from the initial distribution <span class="math-container">$\mu_0(s)$</span> and repeat <span class="math-container">$\mu_1(s) = \mu_0(s) P$</span>, <span class="math-container">$\mu_2(s) = \mu_1(s)P, \ldots$</span> and at some point the distribution will not change anymore. This seems also what <a href="https://math.stackexchange.com/questions/1194294/definition-of-limiting-distribution-in-a-markov-chain-why-do-we-condition-on">this question/answer says</a>.</p>
  <p>However, all definitions I find say that <em>the limiting distribution does not depend on the initial distribution</em>. But I can easily make a non-irreducible chain with two communicating classes like the one below, where the limit will converge but depends on the initial distribution.</p>
  <p><a href="https://i.sstatic.net/yJI7820w.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/yJI7820w.png" alt="enter image description here" /></a></p>
  <p><strong>So, can the limiting distribution depend on the initial distribution? If not, how can I find it?</strong></p>
  <p><a href="https://math.stackexchange.com/questions/1595606/computational-methods-for-the-limiting-distribution-of-a-finite-ergodic-markov-c">This answer</a> shows a few methods and one is what I am already doing below, i.e., a simple loop to simulate the chain. But, if I run it I converge to different distributions depending on the initial one.</p>
  <pre><code>import numpy as np
  np.set_printoptions(precision=3, suppress=True)
  
  P = np.array([
      [0.0, 0.5, 0.0, 0.5, 0.0],
      [0.0, 0.0, 1.0, 0.0, 0.0],
      [0.0, 0.5, 0.5, 0.0, 0.0],
      [0.0, 0.0, 0.0, 0.5, 0.5],
      [0.0, 0.0, 0.0, 1.0, 0.0],
  ])
  
  MU = [
      np.array([0.3, 0.1, 0.2, 0.4, 0.0]),
      np.array([1.0, 0., 0., 0., 0.0]),
      np.random.rand(5),
  ]
  
  for mu_t in MU:
      mu_t /= mu_t.sum()
      for _ in range(1000):
          mu_t = np.dot(mu_t, P)
      print(mu_t)
      print(np.dot(mu_t, P))
      print()
  </code></pre>
  <p>I know that an ergodic chain has only one unique limiting distribution and that my example is not ergodic, so I am considering the general case.</p>
  
Tags: <markov-decision-process><probability-distribution><markov-chain><ergodicity>
Title: Can the limiting distribution depend on the initial distribution?
UserDisplayName: Simon
ThreadText: # Can the limiting distribution depend on the initial distribution?
  
  (Asked by: username_0 on 2024-11-24)
  
  I am a bit confused about the definition of limiting distribution in Markov chains.
  
  My understanding is that it represents the behavior of the chain in-the-limit. That is, I start from the initial distribution $\mu\_0(s)$ and repeat $\mu\_1(s) = \mu\_0(s) P$, $\mu\_2(s) = \mu\_1(s)P, \ldots$ and at some point the distribution will not change anymore. This seems also what [this question/answer says](https://math.stackexchange.com/questions/1194294/definition-of-limiting-distribution-in-a-markov-chain-why-do-we-condition-on).
  
  However, all definitions I find say that *the limiting distribution does not depend on the initial distribution*. But I can easily make a non-irreducible chain with two communicating classes like the one below, where the limit will converge but depends on the initial distribution.
  
  [![enter image description here](https://i.sstatic.net/yJI7820w.png)](https://i.sstatic.net/yJI7820w.png)
  
  **So, can the limiting distribution depend on the initial distribution? If not, how can I find it?**
  
  [This answer](https://math.stackexchange.com/questions/1595606/computational-methods-for-the-limiting-distribution-of-a-finite-ergodic-markov-c) shows a few methods and one is what I am already doing below, i.e., a simple loop to simulate the chain. But, if I run it I converge to different distributions depending on the initial one.
  
  ```
  import numpy as np
  np.set_printoptions(precision=3, suppress=True)
  
  P = np.array([
      [0.0, 0.5, 0.0, 0.5, 0.0],
      [0.0, 0.0, 1.0, 0.0, 0.0],
      [0.0, 0.5, 0.5, 0.0, 0.0],
      [0.0, 0.0, 0.0, 0.5, 0.5],
      [0.0, 0.0, 0.0, 1.0, 0.0],
  ])
  
  MU = [
      np.array([0.3, 0.1, 0.2, 0.4, 0.0]),
      np.array([1.0, 0., 0., 0., 0.0]),
      np.random.rand(5),
  ]
  
  for mu_t in MU:
      mu_t /= mu_t.sum()
      for _ in range(1000):
          mu_t = np.dot(mu_t, P)
      print(mu_t)
      print(np.dot(mu_t, P))
      print()
  
  ```
  
  I know that an ergodic chain has only one unique limiting distribution and that my example is not ergodic, so I am considering the general case.
  
  **Tags:** markov-decision-process, probability-distribution, markov-chain, ergodicity
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2024-11-24. Score: 1 | Accepted answer
  
  > all definitions I find say that the limiting distribution does not depend on the initial distribution
  
  The proof of this trait of the limit relies on the asumption of [ergodicity](https://ai.stackexchange.com/questions/27196/what-is-ergodicity-in-a-markov-decision-process-mdp).
  
  You can define an MDP where there are non-interacting splits in trajectory (i.e. a branch in trajectory with no way of reaching outside of a subset of states), with two or more recurrent loops, plus an initial distribution where it is possible to reach at least two split points. This is essentially multiple ergodic MDPs in a branching structure. In that case, *within each split*, the limiting distribution is independent of the starting distribution.
  
  In your example case, everything ends up in B & C or D & E. The ratio of times spent between BC and DE depend on the starting distribution. Within those groups, the ratios of time B:C and D:E are fixed and converge independently of any starting distribution.
  
  > **Comments:**
  > **username_0**: Can a limiting distribution independent of the initial state exist in a non-ergodic chain? Because I usually read "a limting distribution is the limit of the distribution, independent of the initial state". And then "if the chain is ergodic, the limiting distribution exists and it's unique". The definition of limiting distribution seems unrelated to the one of ergodicity, and one that could exist in non-ergodic chains. But you said it depends on ergodicity, which confuses me (makes sense in terms of independency of the initial state, just doesn't seem to fit the literature? Any ref?).  
  > **username_2**: @username_0 as you said, ergodicity gives you guarantees that it exists and that is unique… in your example, it exists but it’s not unique, and furthermore depends on the initial state  
  > **username_0**: @username_2 Right, so the limiting distribution, by definition, can be not unique, right? Because I often read "if it exists, it is unique" in its definition. That's why I was confused.  
  > **username_2**: @username_0 the fact that the limiting distribution depends on the initial state implies that the couple MC+initial state has a limiting distribution, though you probably cannot prove that it is always the case... generally speaking however, a limiting distribution exists if and only if it does not depend on the initial state (by definition)  
  
  ---
  
  ### Answer by username_3 on 2024-11-24. Score: 2
  
  Indeed the limiting behavior of a non-ergodic Markov chain which is not irreducible with multiple mean positive recurrent communicating classes depends on its initial distribution, since apparently the state space can be decomposed into multiple communicating classes thus the limiting distribution *if exists* depends on which class is activated by the initial distribution and typically there's one limiting distribution per recurrent communicating class *if* all states within the said communicating class are aperiodic.
  
  In summary without irreducibility the limiting distribution if exists is not unique, and without aperiodicity a mere *stationary* distribution may not be converged as a *limiting* distribution as illustrated in the answer to your recent [post](https://ai.stackexchange.com/questions/47263/conditions-for-the-existence-of-stationary-state-distribution) here. There's conceptual difference between a limiting and stationary distribution of a MC.
  
  > **Comments:**
  > **username_0**: Thanks! Your reply helped understanding username_1's better. I think both answers are useful, I accepted username_1's just because he replied first.  
  
  ---
  
  