Answers.Body: <blockquote>
    <p>What is the difference between the definition of a stationary policy in reinforcement learning and contextual bandit?</p>
  </blockquote>
  
  <p>There is no difference. A policy decides which action to take in each state. This is usually split into deterministic policies of the form <span class="math-container">$\pi(s) : \mathcal{S} \rightarrow \mathcal{A}$</span> and stochastic policies of the form <span class="math-container">$\pi(s|a) : \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}, \text{Pr}[A_t=a |S_t=s]$</span>. </p>
  
  <p>When we say that a policy is static, it means that the mapping from state to action - or distribution of actions - does not change over the time that we are interested in. This definition applies equally in the Reinforcement Learning (RL) and Contextual Bandit settings.</p>
  
  <blockquote>
    <p>But in a reinforcement learning problem, the history can be used to define a state. </p>
  </blockquote>
  
  <p>It can be used this way, but is not required. What is required is the <a href="https://en.wikipedia.org/wiki/Markov_property" rel="nofollow noreferrer">Markov property</a> i.e. that identifying the state also determines all the allowed transitions and rewards, plus their probabilities of occurring given an action. </p>
  
  <blockquote>
    <p>In this case, does the history include the rewards revealed thus far as well? </p>
  </blockquote>
  
  <p>If those can affect the future state transitions and rewards yes. If they generally do not, then you would usually exclude them from the state description.</p>
  
  <blockquote>
    <p>If so, the policy is not stationary anymore I guess.</p>
  </blockquote>
  
  <p>What you are proposing is a system that changes its state depending on rewards seen so far. As above, this is not necessary to define a RL problem, but it is allowed in RL. Whilst in a contextual bandit you assume no rules apply to state transitions, and this is a key difference between contextual bandit and RL settings.</p>
  
  <p>The policy is stationary if its mapping rules remain unchanged.</p>
  
  <p>Your proposed addition to the state does not require changing the policy. The policy is a function of the state. It may choose a different action depending on this historical aspect of the state, but it can still remain a fixed function - its input may change, but the policy function itself does not need to change to account for that.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-03T20:00:04.150Z
Answers.LastActivityDate: 2019-10-03T20:25:52.283Z
Answers.LastEditDate: 2019-10-03T20:25:52.283Z
Answers.UserDisplayName: Neil Slater
Body: <p>A stationary policy is a function that maps a state to a probability distribution of actions. </p>
  
  <p>In a contextual bandit problem, a state itself does not include the history. But in a reinforcement learning problem, the history can be used to define a state. In this case, does the history include the rewards revealed thus far as well? If so, the policy is not stationary anymore I guess. </p>
  
  <p>As the title says, I think I am confused about the difference in the definitions of stationary (and/or non-stationary) policy between reinforcement learning and contextual bandit.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-10-03T19:04:40.770Z
Comments.Text: A [stationary policy](https://ai.stackexchange.com/q/13088/2444) is a policy that is fixed. So, it is not necessarily a function that maps a state to a _probability distribution over actions_ (which is called a stochastic or non-deterministic policy), but it can map a state to a _single action_ (a deterministic policy). So, both [deterministic and stochastic policies](https://ai.stackexchange.com/q/12274/2444) can (or not) be stationary.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-10-03T19:09:05.797Z
Comments.Text: What is your definition of a stationary policy in the context of the _contextual bandit problem_?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-10-04T01:50:35.300Z
Comments.Text: @nbro as Neil defined in the answer below, if it is deterministic, a stationary policy is a function that maps a state (context in contextual bandit) to an action, and if it is stochastic, it maps a state to probability distribution of actions.
Comments.UserDisplayName: Hunnam 
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-10-04T13:04:24.793Z
Comments.Text: Exactly. You just repeated what I said.
Comments.UserDisplayName: nbro
Tags: <machine-learning><reinforcement-learning><comparison><definitions><stationary-policy>
Title: What is the difference between the definition of a stationary policy in reinforcement learning and contextual bandit?
UserDisplayName: Hunnam 
ThreadText: # What is the difference between the definition of a stationary policy in reinforcement learning and contextual bandit?
  
  (Asked by: username_0 on 2019-10-03)
  
  A stationary policy is a function that maps a state to a probability distribution of actions.
  
  In a contextual bandit problem, a state itself does not include the history. But in a reinforcement learning problem, the history can be used to define a state. In this case, does the history include the rewards revealed thus far as well? If so, the policy is not stationary anymore I guess.
  
  As the title says, I think I am confused about the difference in the definitions of stationary (and/or non-stationary) policy between reinforcement learning and contextual bandit.
  
  **Tags:** machine-learning, reinforcement-learning, comparison, definitions, stationary-policy
  
  > **Comments:**
  > **username_1**: A [stationary policy](https://ai.stackexchange.com/q/13088/2444) is a policy that is fixed. So, it is not necessarily a function that maps a state to a _probability distribution over actions_ (which is called a stochastic or non-deterministic policy), but it can map a state to a _single action_ (a deterministic policy). So, both [deterministic and stochastic policies](https://ai.stackexchange.com/q/12274/2444) can (or not) be stationary.  
  > **username_1**: What is your definition of a stationary policy in the context of the _contextual bandit problem_?  
  > **username_0**: @username_1 as username_2 defined in the answer below, if it is deterministic, a stationary policy is a function that maps a state (context in contextual bandit) to an action, and if it is stochastic, it maps a state to probability distribution of actions.  
  > **username_1**: Exactly. You just repeated what I said.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2019-10-03. Score: 3 | Accepted answer
  
  > What is the difference between the definition of a stationary policy in reinforcement learning and contextual bandit?
  
  There is no difference. A policy decides which action to take in each state. This is usually split into deterministic policies of the form $\pi(s) : \mathcal{S} \rightarrow \mathcal{A}$ and stochastic policies of the form $\pi(s|a) : \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}, \text{Pr}[A\_t=a |S\_t=s]$.
  
  When we say that a policy is static, it means that the mapping from state to action - or distribution of actions - does not change over the time that we are interested in. This definition applies equally in the Reinforcement Learning (RL) and Contextual Bandit settings.
  
  > But in a reinforcement learning problem, the history can be used to define a state.
  
  It can be used this way, but is not required. What is required is the [Markov property](https://en.wikipedia.org/wiki/Markov_property) i.e. that identifying the state also determines all the allowed transitions and rewards, plus their probabilities of occurring given an action.
  
  > In this case, does the history include the rewards revealed thus far as well?
  
  If those can affect the future state transitions and rewards yes. If they generally do not, then you would usually exclude them from the state description.
  
  > If so, the policy is not stationary anymore I guess.
  
  What you are proposing is a system that changes its state depending on rewards seen so far. As above, this is not necessary to define a RL problem, but it is allowed in RL. Whilst in a contextual bandit you assume no rules apply to state transitions, and this is a key difference between contextual bandit and RL settings.
  
  The policy is stationary if its mapping rules remain unchanged.
  
  Your proposed addition to the state does not require changing the policy. The policy is a function of the state. It may choose a different action depending on this historical aspect of the state, but it can still remain a fixed function - its input may change, but the policy function itself does not need to change to account for that.
  
  
  ---
  
  