Answers.Body: <p>In general a <em>stationary distribution</em> is a distribution over something (e.g. actions, states, etc) that doesn't change over time, for example.</p>
  <p>Example of non-stationary distributions can be easily found in multi-agent RL, when modelling the policy of some other agent: in this case the opponent's policy can be non-stationary since it can adapt itself to the other agents, and so on. In such case, that distribution will change with time, or, better said, with novel gathered experience from the environment and other agents.</p>
  <p>In you case the distribution <span class="math-container">$d_\pi(s)$</span> is stationary over states: giving a fixed policy <span class="math-container">$\pi$</span> (that doesn't change), the distribution of states that the policy visits does not change too, e.g. with time. In simpler words, <span class="math-container">$\pi$</span> always visits the same states at each rollout. As the next line in the text says &quot;the stationary distribution varies with the learned policy&quot;, means that if you update <span class="math-container">$\pi$</span> then <span class="math-container">$d_\pi$</span> will change consequently assigning a diverse probability mass over <span class="math-container">$s$</span>.</p>
  <p>I think that <span class="math-container">$d_\pi$</span> is also related to the transition dynamics of the environment: maybe the text is implicitly assuming that neither the underlying environment will change with time, so that fixing both the env and policy the visited/reached states and their probability are the same across timesteps/transitions.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-06-06T20:33:46.710Z
Answers.Text: Worth mention ergodicity assumption and linking https://ai.stackexchange.com/questions/27196/what-is-ergodicity-in-a-markov-decision-process-mdp - which is what the quote from the question is about
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-06-06T17:34:02.367Z
Answers.LastActivityDate: 2023-06-06T17:34:02.367Z
Answers.UserDisplayName: Luca Anzalone
Body: <p>What does stationary state
  distribution mean in reinforcement learning?</p>
  <p>The context it is used is as below.</p>
  <p><a href="https://i.sstatic.net/5qnx8.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/5qnx8.png" alt="enter image description here" /></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-06-04T19:17:51.497Z
Comments.Text: Could you give some context where you saw or heard this please? There are a couple of things it could mean, which depends whether we are talking about this as a trait of an environment, or a trait of a policy (the latter typically in a continuing/non-episodic environment). There may be other contexts that I don't know as well.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-06-06T09:34:55.627Z
Comments.Text: I added the context it was used. I updated the question.
Comments.UserDisplayName: DSPinfinity
Tags: <reinforcement-learning>
Title: What does stationary state distribution mean in reinforcement learning?
UserDisplayName: DSPinfinity
ThreadText: # What does stationary state distribution mean in reinforcement learning?
  
  (Asked by: username_0 on 2023-06-04)
  
  What does stationary state
  distribution mean in reinforcement learning?
  
  The context it is used is as below.
  
  [![enter image description here](https://i.sstatic.net/5qnx8.png)](https://i.sstatic.net/5qnx8.png)
  
  **Tags:** reinforcement-learning
  > **Comments:**
  > **username_1**: Could you give some context where you saw or heard this please? There are a couple of things it could mean, which depends whether we are talking about this as a trait of an environment, or a trait of a policy (the latter typically in a continuing/non-episodic environment). There may be other contexts that I don't know as well.  
  > **username_0**: I added the context it was used. I updated the question.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-06-06. Score: 1 | Accepted answer
  
  In general a *stationary distribution* is a distribution over something (e.g. actions, states, etc) that doesn't change over time, for example.
  
  Example of non-stationary distributions can be easily found in multi-agent RL, when modelling the policy of some other agent: in this case the opponent's policy can be non-stationary since it can adapt itself to the other agents, and so on. In such case, that distribution will change with time, or, better said, with novel gathered experience from the environment and other agents.
  
  In you case the distribution $d\_\pi(s)$ is stationary over states: giving a fixed policy $\pi$ (that doesn't change), the distribution of states that the policy visits does not change too, e.g. with time. In simpler words, $\pi$ always visits the same states at each rollout. As the next line in the text says "the stationary distribution varies with the learned policy", means that if you update $\pi$ then $d\_\pi$ will change consequently assigning a diverse probability mass over $s$.
  
  I think that $d\_\pi$ is also related to the transition dynamics of the environment: maybe the text is implicitly assuming that neither the underlying environment will change with time, so that fixing both the env and policy the visited/reached states and their probability are the same across timesteps/transitions.
  
  > **Comments:**
  > **username_1**: Worth mention ergodicity assumption and linking https://ai.stackexchange.com/questions/27196/what-is-ergodicity-in-a-markov-decision-process-mdp - which is what the quote from the question is about  
  
  ---
  
  