Answers.Body: <blockquote>
  <p>In those cases the uncontrollable variables can be included as part of the observation space alongside other controllable ones, but that feels sub-optimal as the model will need to learn which dimensions are controllable and which aren't as a result of training, even though it is known before-hand.</p>
  </blockquote>
  <p>This is not usually a problem. In normal RL the agent does not attempt to learn how its actions directly impact the next state. So it doesn't really matter which parts of the observation change for which reason. In model-free RL methods, the agent simply needs to receive fairly sampled observations of reward and next state. If some variables are critical, but don't change depending on action, then the main design consideration you need is ensuring enough of a range of different values is experienced by the agent during training*.</p>
  <p>The exception would be if you are using a technique that tries to learn the environment model, and directly learn/represent <span class="math-container">$p(r, s'|s,a)$</span> or the useful but less complete <span class="math-container">$p(s'|s,a)$</span>. You <em>might</em> want to learn this model if you are writing a model-based agent with planning capability. You may also want to do this as a kind of regularisation of a shared representation that can predict multiple things (e.g. a neural network could be trained under Actor/Critic with 3 output &quot;heads&quot; - one for action values, one for optimal policy choice and another for predicted next state).</p>
  <p>If you <em>are</em> trying to learn the environment model, you can insert your knowldge of which state variables are static or progress mathematically independently of the action choice by <em>not</em> having the neural network attempt to learn those parts of the model, instead adding values calculated according to the fixed rule. So you can directly split the feature vector into &quot;action influenced&quot; and &quot;other&quot; parts, for RL training purposes only bother learning the &quot;action influenced&quot; part, and then recombine the split parts if you actually need a prediction for a planning algorithm. You don't have to do this - if state progression of any feature is complex enough, even if it is not influenced by the action choice, then you may be better off having the nn predict the next value anyway.</p>
  <p>Regardless of whether you are trying to get the agent to the learn the environment model or not, then during training you <em>do</em> have the responsibility to control the added variables somehow. Any observational feature which may change expected reward for different actions, needs to be observed at multiple possible values, in order for the agent to learn to behave optimally as the feature changes over time - whether that is fixed in different scenarios, or changes faster than that, independently of the agent's action, the action value predictions and optimal policy predictions need to see a range of combined values in order to train successfully.</p>
  <p>The &quot;control&quot; on the added environment variables can be as simple as waiting until multiple possible combinations have been experienced. However, in simulations you may have the ability to provide more direct control by varying scenarios according to a useful distribution for training.</p>
  <hr />
  <p>* If <em>all</em> your state variables progress without regard to the action taken, you might be able to simplify the problem from Reinforcement Learning to Contextual Bandit.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-09T17:18:24.450Z
Answers.LastActivityDate: 2023-09-09T17:33:02.580Z
Answers.LastEditDate: 2023-09-09T17:33:02.580Z
Answers.UserDisplayName: Neil Slater
Body: <p>Typically training an RL model requires an action and an observation space, and the agent learns how its actions affect the observations. Even though there are cases where the observation space contains variables that do not change as a result of the agent actions. For instance in a model to navigate a ship actions would be influenced by uncontrollable environmental conditions such as wind speed, weather conditions, etc. as well as controllable ones such as direction or engine speed.</p>
  <p>In those cases the uncontrollable variables can be included as part of the observation space alongside other controllable ones, but that feels sub-optimal as the model will need to learn which dimensions are controllable and which aren't as a result of training, even though it is known before-hand.</p>
  <p>Is there a better method to differentiate both kinds of observations before providing them as input to train an RL model?</p>
  
Tags: <neural-networks><reinforcement-learning><machine-learning><stable-baselines><observation-spaces>
Title: Training an RL model with an environment where some of the variables do not change as a result of the agent actions
UserDisplayName: Jesuspc
ThreadText: # Training an RL model with an environment where some of the variables do not change as a result of the agent actions
  
  (Asked by: username_0 on 2023-09-09)
  
  Typically training an RL model requires an action and an observation space, and the agent learns how its actions affect the observations. Even though there are cases where the observation space contains variables that do not change as a result of the agent actions. For instance in a model to navigate a ship actions would be influenced by uncontrollable environmental conditions such as wind speed, weather conditions, etc. as well as controllable ones such as direction or engine speed.
  
  In those cases the uncontrollable variables can be included as part of the observation space alongside other controllable ones, but that feels sub-optimal as the model will need to learn which dimensions are controllable and which aren't as a result of training, even though it is known before-hand.
  
  Is there a better method to differentiate both kinds of observations before providing them as input to train an RL model?
  
  **Tags:** neural-networks, reinforcement-learning, machine-learning, stable-baselines, observation-spaces
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-09-09. Score: 2
  
  > In those cases the uncontrollable variables can be included as part of the observation space alongside other controllable ones, but that feels sub-optimal as the model will need to learn which dimensions are controllable and which aren't as a result of training, even though it is known before-hand.
  
  This is not usually a problem. In normal RL the agent does not attempt to learn how its actions directly impact the next state. So it doesn't really matter which parts of the observation change for which reason. In model-free RL methods, the agent simply needs to receive fairly sampled observations of reward and next state. If some variables are critical, but don't change depending on action, then the main design consideration you need is ensuring enough of a range of different values is experienced by the agent during training\*.
  
  The exception would be if you are using a technique that tries to learn the environment model, and directly learn/represent $p(r, s'|s,a)$ or the useful but less complete $p(s'|s,a)$. You *might* want to learn this model if you are writing a model-based agent with planning capability. You may also want to do this as a kind of regularisation of a shared representation that can predict multiple things (e.g. a neural network could be trained under Actor/Critic with 3 output "heads" - one for action values, one for optimal policy choice and another for predicted next state).
  
  If you *are* trying to learn the environment model, you can insert your knowldge of which state variables are static or progress mathematically independently of the action choice by *not* having the neural network attempt to learn those parts of the model, instead adding values calculated according to the fixed rule. So you can directly split the feature vector into "action influenced" and "other" parts, for RL training purposes only bother learning the "action influenced" part, and then recombine the split parts if you actually need a prediction for a planning algorithm. You don't have to do this - if state progression of any feature is complex enough, even if it is not influenced by the action choice, then you may be better off having the nn predict the next value anyway.
  
  Regardless of whether you are trying to get the agent to the learn the environment model or not, then during training you *do* have the responsibility to control the added variables somehow. Any observational feature which may change expected reward for different actions, needs to be observed at multiple possible values, in order for the agent to learn to behave optimally as the feature changes over time - whether that is fixed in different scenarios, or changes faster than that, independently of the agent's action, the action value predictions and optimal policy predictions need to see a range of combined values in order to train successfully.
  
  The "control" on the added environment variables can be as simple as waiting until multiple possible combinations have been experienced. However, in simulations you may have the ability to provide more direct control by varying scenarios according to a useful distribution for training.
  
  ---
  
  \* If *all* your state variables progress without regard to the action taken, you might be able to simplify the problem from Reinforcement Learning to Contextual Bandit.
  
  
  ---
  
  