Answers.Body: <p>A <a href="http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.516.4121&amp;rep=rep1&amp;type=pdf" rel="nofollow noreferrer"><strong>stationary policy</strong></a>, <span class="math-container">$\pi_t$</span>, is a policy that does not change over time, that is, <span class="math-container">$\pi_t = \pi, \forall t \geq 0$</span>, where <span class="math-container">$\pi$</span> can either be a function, <span class="math-container">$\pi: S \rightarrow A$</span> (a deterministic policy), or a conditional density, <span class="math-container">$\pi(A \mid S)$</span> (a stochastic policy). A <strong>non-stationary policy</strong> is a policy that is <strong>not</strong> stationary. More precisely, <span class="math-container">$\pi_i$</span> may not be equal to <span class="math-container">$\pi_j$</span>, for <span class="math-container">$i \neq j \geq 0$</span>, where <span class="math-container">$i$</span> and <span class="math-container">$j$</span> are thus two different time steps.</p>
  <p>There are problems where a stationary optimal policy is guaranteed to exist. For example, in the case of a stochastic (there is a probability density that models the dynamics of the environment, that is, the transition function and the reward function) and discrete-time Markov decision process (MDP) with finite numbers of states and actions, and bounded rewards, where the <em>objective</em> is the long-run average reward, a stationary optimal policy exists. The proof of this fact is in the book <a href="https://onlinelibrary.wiley.com/doi/book/10.1002/9780470316887" rel="nofollow noreferrer"><em>Markov Decision Processes: Discrete Stochastic Dynamic Programming</em></a> (1994), by Martin L. Puterman, which <em>apparently</em> is not freely available on the web.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-18T14:17:55.147Z
Answers.Text: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/116369/discussion-on-answer-by-nbro-what-is-the-difference-between-a-stationary-and-a-n).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-13T20:09:36.950Z
Answers.LastActivityDate: 2021-02-25T00:17:37.457Z
Answers.LastEditDate: 2021-02-25T00:17:37.457Z
Answers.UserDisplayName: nbro
Body: <p>In reinforcement learning, there are <a href="https://ai.stackexchange.com/q/12274/2444">deterministic and non-deterministic (or stochastic) policies</a>, but there are also <em>stationary</em> and <em>non-stationary</em> policies. </p>
  
  <p>What is the difference between a stationary and a non-stationary policy? How do you formalize both? Which problems (or environments) require a stationary policy as opposed to a non-stationary one (and vice-versa)?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-21T15:22:44.500Z
Comments.Text: [This Wikipedia article](https://en.wikipedia.org/wiki/Stationary_process) should also be useful.
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><comparison><policies><stationary-policy>
Title: What is the difference between a stationary and a non-stationary policy?
UserDisplayName: nbro
ThreadText: # What is the difference between a stationary and a non-stationary policy?
  
  (Asked by: username_0 on 2019-06-27)
  
  In reinforcement learning, there are [deterministic and non-deterministic (or stochastic) policies](https://ai.stackexchange.com/q/12274/2444), but there are also *stationary* and *non-stationary* policies.
  
  What is the difference between a stationary and a non-stationary policy? How do you formalize both? Which problems (or environments) require a stationary policy as opposed to a non-stationary one (and vice-versa)?
  
  **Tags:** reinforcement-learning, comparison, policies, stationary-policy
  > **Comments:**
  > **username_0**: [This Wikipedia article](https://en.wikipedia.org/wiki/Stationary_process) should also be useful.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2019-09-13. Score: 6 | Accepted answer
  
  A [**stationary policy**](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.516.4121&rep=rep1&type=pdf), $\pi\_t$, is a policy that does not change over time, that is, $\pi\_t = \pi, \forall t \geq 0$, where $\pi$ can either be a function, $\pi: S \rightarrow A$ (a deterministic policy), or a conditional density, $\pi(A \mid S)$ (a stochastic policy). A **non-stationary policy** is a policy that is **not** stationary. More precisely, $\pi\_i$ may not be equal to $\pi\_j$, for $i \neq j \geq 0$, where $i$ and $j$ are thus two different time steps.
  
  There are problems where a stationary optimal policy is guaranteed to exist. For example, in the case of a stochastic (there is a probability density that models the dynamics of the environment, that is, the transition function and the reward function) and discrete-time Markov decision process (MDP) with finite numbers of states and actions, and bounded rewards, where the *objective* is the long-run average reward, a stationary optimal policy exists. The proof of this fact is in the book [*Markov Decision Processes: Discrete Stochastic Dynamic Programming*](https://onlinelibrary.wiley.com/doi/book/10.1002/9780470316887) (1994), by Martin L. Puterman, which *apparently* is not freely available on the web.
  
  > **Comments:**
  > **username_0**: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/116369/discussion-on-answer-by-nbro-what-is-the-difference-between-a-stationary-and-a-n).  
  
  ---
  
  