Answers.Body: <blockquote>
  <p>In this scenario, is it preferable to create two separate environments based on the priority (input parameter) of the alert?</p>
  </blockquote>
  <p>It is difficult to make a hard rule here.</p>
  <p>If the resulting environments can be cleanly sorted into a few different categories, and the ideal behaviour and/or the states visited are radically different in each scenario, then maybe a few different agents optimised for each scenario could work well.</p>
  <p>A more general approach however, is to include the episode start data as part of the state that the agent observes on each time step. A single agent can then in theory learn the different behaviours required depending on the initial values, plus still generalise from anything shared between the multiple scenarios.</p>
  <blockquote>
  <p>The alert has two priorities &quot;HIGH&quot; and &quot;LOW&quot;, when the priority is &quot;HIGH&quot; the stopping condition is a reward of &quot;100&quot; and when the priority is &quot;LOW&quot;, the stopping condition is a reward of &quot;1000&quot;.</p>
  </blockquote>
  <p>This may work against you. RL agents do not respond to the absolute values of rewards, other than how they compare to other rewards also available within the same episode (or continuing environment).</p>
  <p>If there is only ever one issue to solve at a time, and no conflict between solving either of the &quot;HIGH&quot; or &quot;LOW&quot; priority problems (such as splittig resources or effort between them), the different reward system seems redundant. Solved is solved. You might rate the <em>usefulness</em> of an agent that solves the &quot;LOW&quot; priority issue well higher, but it seems to me that this describes what you should work on first, not the goals of the agent. To influence the goals of the agent, <em>both</em> rewards would need to be available within the same episode or continuing environment, requiring the agent to make a choice between them.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-24T17:58:22.440Z
Answers.Text: Thanks @neil-slater, I should have clarified the Alert can only be one priority in the example I gave e.g., "HIGH" or "LOW", such that the RL agent can learn what is the best policy (and therefore series of steps to be taken) based on this priority. 
  
  I feel myself leaning towards the first example you gave, but of course if one increases the number of parameters  in the inputs e.g., Alert Priority, System Criticality, Alert Triage Time (where there is a maximum time that the episode can run for), then, the separate environments for the different combinations likely wouldn't scale well.
Answers.UserDisplayName: RL_NOOB
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-24T19:41:31.387Z
Answers.Text: with this approach "A more general approach however, is to include the episode start data as part of the state that the agent observes on each time step. A single agent can then in theory learn the different behaviours required depending on the initial values, plus still generalise from anything shared between the multiple scenarios." <- conceptually how might one go about implementing this? Are there any toy examples you can point me in the direction of?
Answers.UserDisplayName: RL_NOOB
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-24T19:52:32.720Z
Answers.Text: @RL_NOOB: Add extra features to describe the nature of the goal (perhaps a boolean, or a one-hot-encoded class), and train agents as normal. Make sure to train with all possible variations of the goal. If you want the agent to *generalise* then you will need to use some kind of approximator - e.g. neural networks are popular in agents like DQN. Using an approximator will generalise behaviour to work in previously unseen states (caveat: as well as e.g. neural networks usually do). I assumed you were using something like DQN in the answer but if your state is simple enough you might not need to
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-24T19:53:40.637Z
Answers.Text: @RL_NOOB: A clasic toy example where goal state is moved around is Taxi = https://gym.openai.com/envs/Taxi-v3/
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-24T20:36:07.857Z
Answers.Text: Thanks @neil-slater your comments and insight have been really helpful :)
Answers.UserDisplayName: RL_NOOB
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-24T17:32:23.470Z
Answers.LastActivityDate: 2020-11-24T17:32:23.470Z
Answers.UserDisplayName: Neil Slater
Body: <p>I would like some guidance on how to design an Environment for a Reinforcement Learning agent where the stopping conditions and rewards for the environment change based on an initial set of input parameters.</p>
  <p>For example, let's say that a system generated alert triggers the instantiation of the RL environment, whereby the RL agent is launched to make decisions in the environment, based on the alert. The alert has two priorities &quot;HIGH&quot; and &quot;LOW&quot;, when the priority is &quot;HIGH&quot; the stopping condition is a reward of &quot;100&quot; and when the priority is &quot;LOW&quot;, the stopping condition is a reward of &quot;1000&quot;.</p>
  <p>In this scenario, is it preferable to create two separate environments based on the priority (input parameter) of the alert? Or is this a common requirement that should be designed into the environment/agent? If so, how? Note that I have simplified the scenario, so there could be multiple conditions (e.g., alert, system type, etc), but I am just trying to find a basic solution for the general case.</p>
  
Tags: <reinforcement-learning><environment><reward-design><reward-functions>
Title: If the reward function of an environment depends on some initial conditions, should I create a separate environment for each condition?
UserDisplayName: RL_NOOB
ThreadText: # If the reward function of an environment depends on some initial conditions, should I create a separate environment for each condition?
  
  (Asked by: username_0 on 2020-11-24)
  
  I would like some guidance on how to design an Environment for a Reinforcement Learning agent where the stopping conditions and rewards for the environment change based on an initial set of input parameters.
  
  For example, let's say that a system generated alert triggers the instantiation of the RL environment, whereby the RL agent is launched to make decisions in the environment, based on the alert. The alert has two priorities "HIGH" and "LOW", when the priority is "HIGH" the stopping condition is a reward of "100" and when the priority is "LOW", the stopping condition is a reward of "1000".
  
  In this scenario, is it preferable to create two separate environments based on the priority (input parameter) of the alert? Or is this a common requirement that should be designed into the environment/agent? If so, how? Note that I have simplified the scenario, so there could be multiple conditions (e.g., alert, system type, etc), but I am just trying to find a basic solution for the general case.
  
  **Tags:** reinforcement-learning, environment, reward-design, reward-functions
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-11-24. Score: 1 | Accepted answer
  
  > In this scenario, is it preferable to create two separate environments based on the priority (input parameter) of the alert?
  
  It is difficult to make a hard rule here.
  
  If the resulting environments can be cleanly sorted into a few different categories, and the ideal behaviour and/or the states visited are radically different in each scenario, then maybe a few different agents optimised for each scenario could work well.
  
  A more general approach however, is to include the episode start data as part of the state that the agent observes on each time step. A single agent can then in theory learn the different behaviours required depending on the initial values, plus still generalise from anything shared between the multiple scenarios.
  
  > The alert has two priorities "HIGH" and "LOW", when the priority is "HIGH" the stopping condition is a reward of "100" and when the priority is "LOW", the stopping condition is a reward of "1000".
  
  This may work against you. RL agents do not respond to the absolute values of rewards, other than how they compare to other rewards also available within the same episode (or continuing environment).
  
  If there is only ever one issue to solve at a time, and no conflict between solving either of the "HIGH" or "LOW" priority problems (such as splittig resources or effort between them), the different reward system seems redundant. Solved is solved. You might rate the *usefulness* of an agent that solves the "LOW" priority issue well higher, but it seems to me that this describes what you should work on first, not the goals of the agent. To influence the goals of the agent, *both* rewards would need to be available within the same episode or continuing environment, requiring the agent to make a choice between them.
  
  > **Comments:**
  > **username_0**: Thanks @username_1, I should have clarified the Alert can only be one priority in the example I gave e.g., "HIGH" or "LOW", such that the RL agent can learn what is the best policy (and therefore series of steps to be taken) based on this priority.   I feel myself leaning towards the first example you gave, but of course if one increases the number of parameters  in the inputs e.g., Alert Priority, System Criticality, Alert Triage Time (where there is a maximum time that the episode can run for), then, the separate environments for the different combinations likely wouldn't scale well.  
  > **username_0**: with this approach "A more general approach however, is to include the episode start data as part of the state that the agent observes on each time step. A single agent can then in theory learn the different behaviours required depending on the initial values, plus still generalise from anything shared between the multiple scenarios." <- conceptually how might one go about implementing this? Are there any toy examples you can point me in the direction of?  
  > **username_1**: @username_0: Add extra features to describe the nature of the goal (perhaps a boolean, or a one-hot-encoded class), and train agents as normal. Make sure to train with all possible variations of the goal. If you want the agent to *generalise* then you will need to use some kind of approximator - e.g. neural networks are popular in agents like DQN. Using an approximator will generalise behaviour to work in previously unseen states (caveat: as well as e.g. neural networks usually do). I assumed you were using something like DQN in the answer but if your state is simple enough you might not need to  
  > **username_1**: @username_0: A clasic toy example where goal state is moved around is Taxi = https://gym.openai.com/envs/Taxi-v3/  
  > **username_0**: Thanks @username_1 your comments and insight have been really helpful :)  
  
  ---
  
  