Answers.Body: <p>It is, I suppose, a philosophical question whether data that <em>describes</em> a whole episode and does not respond to events within it is part of the state, or is part of some other structure.</p>
  <p>However, the practical response is to view such descriptive data as defining an <em>instance</em> of a class of related environments, and to include it in the state features. This may be done for two main reasons:</p>
  <ul>
  <li><p>The static data is a relevant parameter of the environment, affecting state transitions and rewards.</p>
  </li>
  <li><p>It is possible to generalise over the population of all values that the parameters can take.</p>
  </li>
  </ul>
  <p>In simple environments, generalisation might only be that the same agent can learn about all variations in a single combined training session. You could use a tabular RL method, starting randomly with one of the possible variations until all were sufficiently covered.</p>
  <p>In more complex environments, generalisation may also occur through functon approximation, in a similar manner to contextual bandits. In your personalisation example, you are not expecting to train the agent for all possible user descriptions, but hope that people with similar age, gender etc descriptions will respond similarly to an agent that personalises content.</p>
  <p>Philosophically, the contextual data is either part of a larger state space (with a restriction that transitions between different contexts do not happen within an episode), or it is metadata that impacts the &quot;real&quot; state transitions and rewards. Pragmatically, to allow the data to influence value functions and policies, it is necessary to use it in the arguments of those functions. Whether you then view it as part of the state feature vector or as something that is concatenated to state features is a personal choice. Most of the literature I have seen assumes without comment that it is part of the state.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-03T07:27:34.393Z
Answers.LastActivityDate: 2021-01-03T10:35:53.997Z
Answers.LastEditDate: 2021-01-03T10:35:53.997Z
Answers.UserDisplayName: Neil Slater
Body: <p>Does it make sense to incorporate constant states in the Markov Decision Process and employ a reinforcement learning algorithm to solve it?</p>
  <p>For instance, for applications of personalization, I would like to include users' attributes, like age and gender, as states, along with other changing states. Does it make sense to do so? I have a sequential problem, so I assume the contextual bandit does not fit here.</p>
  
Tags: <reinforcement-learning><state-spaces>
Title: Does it make sense to include constant states into reinforcement learning formulation?
UserDisplayName: ycenycute
ThreadText: # Does it make sense to include constant states into reinforcement learning formulation?
  
  (Asked by: username_0 on 2021-01-03)
  
  Does it make sense to incorporate constant states in the Markov Decision Process and employ a reinforcement learning algorithm to solve it?
  
  For instance, for applications of personalization, I would like to include users' attributes, like age and gender, as states, along with other changing states. Does it make sense to do so? I have a sequential problem, so I assume the contextual bandit does not fit here.
  
  **Tags:** reinforcement-learning, state-spaces
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-01-03. Score: 0 | Accepted answer
  
  It is, I suppose, a philosophical question whether data that *describes* a whole episode and does not respond to events within it is part of the state, or is part of some other structure.
  
  However, the practical response is to view such descriptive data as defining an *instance* of a class of related environments, and to include it in the state features. This may be done for two main reasons:
  
  * The static data is a relevant parameter of the environment, affecting state transitions and rewards.
  * It is possible to generalise over the population of all values that the parameters can take.
  
  In simple environments, generalisation might only be that the same agent can learn about all variations in a single combined training session. You could use a tabular RL method, starting randomly with one of the possible variations until all were sufficiently covered.
  
  In more complex environments, generalisation may also occur through functon approximation, in a similar manner to contextual bandits. In your personalisation example, you are not expecting to train the agent for all possible user descriptions, but hope that people with similar age, gender etc descriptions will respond similarly to an agent that personalises content.
  
  Philosophically, the contextual data is either part of a larger state space (with a restriction that transitions between different contexts do not happen within an episode), or it is metadata that impacts the "real" state transitions and rewards. Pragmatically, to allow the data to influence value functions and policies, it is necessary to use it in the arguments of those functions. Whether you then view it as part of the state feature vector or as something that is concatenated to state features is a personal choice. Most of the literature I have seen assumes without comment that it is part of the state.
  
  
  ---
  
  