Answers.Body: <p>A stationary policy is the one that does not depend on time. Meaning that the agent will take the same decision whenever certain conditions are met. This stationary policy may be probabilistic which implies that the probability of choosing an action remains the same. It may take different decisions but the probability remains the same. </p>
  
  <p>A Stationary environment refers to the static model of the system. The model comprises of a Reward function and Transition probabilities. So, in a stationary environment, the reward function and transition probabilities remain constant or the changes are slow enough that the agent finds enough training time to learn the changes done in the environment. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-17T06:14:27.103Z
Answers.LastActivityDate: 2020-04-17T06:14:27.103Z
Answers.UserDisplayName: Khalid Ibrahim
Answers.Body: <p>You are right: a stationary policy is independent of time. It is basically a mapping from states to actions (or probability distributions over actions). Regardless of the point in time in which the agent observes the state <span class="math-container">$s$</span> it will select an action <span class="math-container">$a$</span> (or select a probability <span class="math-container">$\pi(a \vert s)$</span> for every action <span class="math-container">$a$</span>).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-20T14:18:28.650Z
Answers.LastActivityDate: 2021-01-21T17:57:58.817Z
Answers.LastEditDate: 2021-01-21T17:57:58.817Z
Answers.UserDisplayName: D. S.
Answers.Body: <p><strong>There are two kinds of problem</strong> </p>
  
  <h2>Stationary and non-stationary</h2>
  
  <p>Stationary  problems are those whose reward value is static, dost not change
  and on other hand non-stationary problems are those whose reward value change with time </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-16T11:38:37.987Z
Answers.LastActivityDate: 2020-04-16T11:38:37.987Z
Answers.UserDisplayName: Laeeq Khan Niazi
Answers.Body: <p>A <em>stationary policy</em> is a policy that does not change. Although strictly that is a time-dependent issue, that is not what the distinction refers to in reinforcement learning. It generally means that the policy is not being updated by a learning algorithm.</p>
  
  <p>If you are working with a stationary policy in reinforcement learning (RL), typically that is because you are trying to learn its value function. Many RL techniques - including Monte Carlo, Temporal Difference, Dynamic Programming - can be used to evaluate a given policy, as well as used to search for a better or optimal policy.</p>
  
  <p><em>Stationary dynamics</em> refers to the environment, and is an assumption that the rules of the environment do not change over time. The rules of the environment are often represented as an MDP model, which consists of all the state transition probabilities and reward distributions. Reinforcement learning algorithms that work <em>online</em> can usually cope and adjust policies to match non-stationary environments, provided the changes do not happen too often, or enough learning/exploring time is allowed between more radical changes. Most RL algorithms have at least some online component, it is also important to keep exploring non-optimal actions in environments with this trait (in order to spot when they may become optimal).</p>
  
  <p><em>Stationary data</em> is not a RL-specific term, but also relates to the need for an online algorithm, or at least plans for discarding older data and re-training existing models over time. You might have non-stationary data in any ML, including supervised learning - prediction problems that work with data about people and their behaviour often have this issue as population norms change over timescales of months and years.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-09T15:40:08.440Z
Answers.Text: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/105357/discussion-on-answer-by-neil-slater-what-does-stationary-mean-in-the-context-o).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-21T15:12:39.037Z
Answers.Text: I had upvoted this answer, but now that I come back to it I'm wondering about the validity of your claim "It generally means that the policy is not being updated by a learning algorithm.". Can you please cite 1-2 papers or books (or whatever) where "stationary" is being used to indicate that the RL algorithm is not changing the policy? One could use that terminology, but I'm not sure I've ever seen it. I would also like to see something that supports this claim "Reinforcement learning algorithms that work online can usually cope and adjust policies to match non-stationary environments".
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-21T15:14:21.000Z
Answers.Text: Moreover, when you say "Most RL algorithms have at least some online component", I mean most RL algorithms are online learning algorithms. Of course, now, with all the experience replays that may not be true anymore. In any case, that claim is also a bit misleading, in my opinion, because it seems that you're saying that most RL algorithms learn offline, but usually it's the opposite. Of course, I'm using the definition "online learning" as learning with one sample at a time as it arrives from some data-generating process (in this case, the environment and the interaction with it).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-21T08:36:38.220Z
Answers.LastActivityDate: 2019-02-15T11:43:29.090Z
Answers.LastEditDate: 2019-02-15T11:43:29.090Z
Answers.UserDisplayName: Neil Slater
Body: <p>I think I've seen the expressions "stationary data", "stationary dynamics" and "stationary policy", among others, in the context of reinforcement learning. What does it mean? I think stationary policy means that the policy does not depend on time, and only on state. But isn't that a unnecessary distinction? If the policy depends on time and not only on the state, then strictly speaking time should also be part of the state.</p>
  
Tags: <reinforcement-learning><terminology><policies><stationary-policy>
Title: What does "stationary" mean in the context of reinforcement learning?
UserDisplayName: Paula Vega
ThreadText: # What does "stationary" mean in the context of reinforcement learning?
  
  (Asked by: username_0 on 2018-08-20)
  
  I think I've seen the expressions "stationary data", "stationary dynamics" and "stationary policy", among others, in the context of reinforcement learning. What does it mean? I think stationary policy means that the policy does not depend on time, and only on state. But isn't that a unnecessary distinction? If the policy depends on time and not only on the state, then strictly speaking time should also be part of the state.
  
  **Tags:** reinforcement-learning, terminology, policies, stationary-policy
  
  ---
  
  ## 4 answers
  
  ### Answer by username_4 on 2018-08-21. Score: 10 | Accepted answer
  
  A *stationary policy* is a policy that does not change. Although strictly that is a time-dependent issue, that is not what the distinction refers to in reinforcement learning. It generally means that the policy is not being updated by a learning algorithm.
  
  If you are working with a stationary policy in reinforcement learning (RL), typically that is because you are trying to learn its value function. Many RL techniques - including Monte Carlo, Temporal Difference, Dynamic Programming - can be used to evaluate a given policy, as well as used to search for a better or optimal policy.
  
  *Stationary dynamics* refers to the environment, and is an assumption that the rules of the environment do not change over time. The rules of the environment are often represented as an MDP model, which consists of all the state transition probabilities and reward distributions. Reinforcement learning algorithms that work *online* can usually cope and adjust policies to match non-stationary environments, provided the changes do not happen too often, or enough learning/exploring time is allowed between more radical changes. Most RL algorithms have at least some online component, it is also important to keep exploring non-optimal actions in environments with this trait (in order to spot when they may become optimal).
  
  *Stationary data* is not a RL-specific term, but also relates to the need for an online algorithm, or at least plans for discarding older data and re-training existing models over time. You might have non-stationary data in any ML, including supervised learning - prediction problems that work with data about people and their behaviour often have this issue as population norms change over timescales of months and years.
  
  > **Comments:**
  > **username_5**: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/105357/discussion-on-answer-by-neil-slater-what-does-stationary-mean-in-the-context-o).  
  > **username_5**: I had upvoted this answer, but now that I come back to it I'm wondering about the validity of your claim "It generally means that the policy is not being updated by a learning algorithm.". Can you please cite 1-2 papers or books (or whatever) where "stationary" is being used to indicate that the RL algorithm is not changing the policy? One could use that terminology, but I'm not sure I've ever seen it. I would also like to see something that supports this claim "Reinforcement learning algorithms that work online can usually cope and adjust policies to match non-stationary environments".  
  > **username_5**: Moreover, when you say "Most RL algorithms have at least some online component", I mean most RL algorithms are online learning algorithms. Of course, now, with all the experience replays that may not be true anymore. In any case, that claim is also a bit misleading, in my opinion, because it seems that you're saying that most RL algorithms learn offline, but usually it's the opposite. Of course, I'm using the definition "online learning" as learning with one sample at a time as it arrives from some data-generating process (in this case, the environment and the interaction with it).  
  
  ---
  
  ### Answer by username_1 on 2020-04-17. Score: 3
  
  A stationary policy is the one that does not depend on time. Meaning that the agent will take the same decision whenever certain conditions are met. This stationary policy may be probabilistic which implies that the probability of choosing an action remains the same. It may take different decisions but the probability remains the same.
  
  A Stationary environment refers to the static model of the system. The model comprises of a Reward function and Transition probabilities. So, in a stationary environment, the reward function and transition probabilities remain constant or the changes are slow enough that the agent finds enough training time to learn the changes done in the environment.
  
  
  ---
  
  ### Answer by username_2 on 2018-08-20. Score: 2
  
  You are right: a stationary policy is independent of time. It is basically a mapping from states to actions (or probability distributions over actions). Regardless of the point in time in which the agent observes the state $s$ it will select an action $a$ (or select a probability $\pi(a \vert s)$ for every action $a$).
  
  
  ---
  
  ### Answer by username_3 on 2020-04-16. Score: 0
  
  **There are two kinds of problem**
  
  ## Stationary and non-stationary
  
  Stationary problems are those whose reward value is static, dost not change
  and on other hand non-stationary problems are those whose reward value change with time
  
  
  ---
  
  