Answers.Body: <blockquote>
  <p>Should the constraints reward also be normalized for all 10 constraints?</p>
  </blockquote>
  <p>You should choose a &quot;natural&quot; balance between rewards where possible.</p>
  <p>If you have many separate goals to take account of, ideally you should convert them all into some comparable metric that is meangful to the success of the agent. Such as a financial gain/loss, or energy gain/loss, or similar. You can normalise them after this, but the ratios between the values should be kept the same.</p>
  <p>This is not always possible with constraints.</p>
  <p>For strict constraints, you should ideally ensure that breaking any constraint will score worse than not breaking the constraint but scoring very badly at everything else. If your system is gaining positive rewards from operating within bounds (and it seems that it is from the description), then one simple way to achieve this is to terminate the episode early and score <span class="math-container">$0$</span> for breaking the constraint. If the constraint relates to an ongoing state measurement this may be the best option. That is because the agent learning how to &quot;escape&quot; from an unreachable state may not be useful.</p>
  <p>For soft constraints, you need to decide a relative cost. For example your constraint:</p>
  <blockquote>
  <p>the agent should not start and stop heating up frequently (like around four starts of the device daily is desirable)</p>
  </blockquote>
  <p>Looks very much like a soft constraint - using words like &quot;around&quot; and &quot;desirable&quot;. For something like that, I would probably allow four starts, then add largish penalty for each start beyond that. What that value is, should be related back to the natural balance between rewards and why you want this constraint.</p>
  <p>As an aside, in order for the agent to learn about this constraint, you <em>must</em> add the number of starts so far for each device to the state. This is true for all constraints - there must be data inside the current state that the agent could use to predict that the constraint will come into play. It doesn't need to know the limit you are applying, but does need to know the current value of any variables used to decide whether a limit should be enforced.</p>
  <p>I think you will also want to store what the last action was, or which device is currently on, so that the agent knows to keep it on in order not to waste one of its four uses of the device per day.</p>
  <blockquote>
  <p>Is it also possible to tell the Reinforcement Learning agent some rules directly without any constraints?</p>
  </blockquote>
  <p>Yes for absolute rules that should prevent the agent taking a specific action in the first place, and that will also be enforced in any production system. For example this rule:</p>
  <blockquote>
  <p>I have two storage systems, and the agent is only allowed to heat up 1 for every time-slot.</p>
  </blockquote>
  <p>Can be easily expressed by having three actions:</p>
  <ul>
  <li><span class="math-container">$a_0$</span> no heating</li>
  <li><span class="math-container">$a_1$</span> heat device 1</li>
  <li><span class="math-container">$a_2$</span> heat device 2</li>
  </ul>
  <p>Although you may want:</p>
  <ul>
  <li><span class="math-container">$a_0$</span> do nothing</li>
  <li><span class="math-container">$a_1$</span> start device 1 (and stop device 2 if running)</li>
  <li><span class="math-container">$a_2$</span> start device 2 (and stop device 1 if running)</li>
  <li><span class="math-container">$a_3$</span> stop whichever device is running</li>
  </ul>
  <p>This second list may help the agent from flip/flopping devices during early stages of learning, and speed things up a little.</p>
  <p>Other action sets could work too. The important thing is that there is nothing to gain by presenting the agent with action choices that you are ruling out as not possible immediately. It would mean adding more negative rewards and constraints to what already looks like a complex problem.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-11T15:50:22.910Z
Answers.Text: Thanks a lot Neil for your answer. I have 3 questions on that. 1) About Hard constraints you wrote "terminate the episode early and score 0 for breaking the constraint." --> What do you mean by terminating the episode? To finish the RL and give a reward of 0? I think in my case this would happen many times such that the learning might suffer from that. 2) About the hard constraints you wrote "That is because the agent learning how to "escape" from an unreachable state may not be useful." --> What do you mean by that?
Answers.UserDisplayName: PeterBe
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-11T15:50:27.863Z
Answers.Text: 3) About the absolute rules I have not 3 but 2 actions (one for each story). Can I just directly change the actions in each iteration of the learning such that this constraint is not violated? For example if the RL agent wanted to assign a positive value for each 2 actions (which is not possible) I would just overwrite these proposed actions from RL with my own rule? Where would I have to do this? In the step function of the environment or shall I rather define a new separate control class that can overrule the actions of the RL agent.
Answers.UserDisplayName: PeterBe
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-11T20:13:39.743Z
Answers.Text: @PeterBe: For (1) Yes that's what I mean. For (2) I mean that if a state is not reachable, learning a policy decision for it may be a waste of time. For (3) I don't understand your comment, but I _think_ you are saying that you have a continuous action vector, or at the least multiple settings for each heater. In either case this is *not* two actions, but more - potentially a lot more. However, you can still expand on the idea in the answer - the point is not to *adjust* things afterwards, but to construct the action space so that only valid actions are possible.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-12T13:14:11.240Z
Answers.Text: Thanks Neil for your answer and effort. I really appreciate it. Regarding your answer to 2) " I mean that if a state is not reachable, learning a policy decision for it may be a waste of time." --> how can I make sure that this is not happening? Regarding your answer to 3) "the point is not to adjust things afterwards, but to construct the action space so that only valid actions are possible." --> how can I do that? I have two continuous variables x(t) and y(t). Now IF x(t) is >0 THEN y(t) has to be 0 and vice versa with x(t) and y(t) element 0 or [0.25,1]. How can tell this to the agent?
Answers.UserDisplayName: PeterBe
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-12T16:57:28.983Z
Answers.Text: For (2) the assumption is that you have a state space constraint. So for certain don't train on data that starts from unreachable states according to your constraint. For (3) I think that would be a good separate question on the site. You can also choose to interpret the lower figure as zero regardless in terms of effect. I don't know whether that would be better than your options for remodelling the action space to force only valid actions.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-13T12:00:51.067Z
Answers.Text: Thanks Neil for your answer and effort. I really appreciate it. Regarding (3) I have aske a separate question (https://ai.stackexchange.com/questions/32042/is-it-possible-to-tell-the-reinforcement-learning-agent-some-rules-directly-with). Bur regarding (2) I still do not understand your answer. How can I make sure "that if a state is not reachable, learning a policy decision for it may be a waste of time." that this is not happening?
Answers.UserDisplayName: PeterBe
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-13T12:10:58.430Z
Answers.Text: For (2) by not training your system on the unreachable states as starting states. You have  already decided a constraint that some state is not reachable, and you can detect this (otherwise you would not be able to calculate any penalty for reaching that state). So skip memorising and/or training for any (s,a) pair where s is outside your constraints. You will still want to train for (s,a,r,s') wheer s' is unreachable in orderto apply the constraint penalty and stop the agent going there.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-13T12:22:23.393Z
Answers.Text: Thanks Neil for your answer. Actually the control system always begins at the beginning of the day and the states there are all set to valid values. So the starting states are always valid states in my example. Thus it is not possible to start the training from unreachable states. I don't really understand the second part of your comment. The training always beginns with valid states but it can happen that the states violate some constraints. For this I use a penalty such that the agent gets bad rewards when beeing in such a state. If the agent is in such a state it should try to get out of it
Answers.UserDisplayName: PeterBe
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-13T12:50:36.097Z
Answers.Text: @PeterBe Yes that's fine if it is a soft constraint and that state can actually occur physically in the real system. My answer about excluding states from training is regarding strict constraints. Your question includes examples ranging from soft constraints to strict/hard constraints, so I have done my best to cover both in the answer. Only you can say when looking at a constraint whether it is a hard or soft constraint for the system
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-13T12:52:57.480Z
Answers.Text: In general, RL will cope better with soft constraints for state values, with negative rewards, but it can also cope OK with failure/early termination as a solution for hard constraints. You also asked about avoiding termination - my answers there are with regard to hard constraints only.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-15T12:32:20.190Z
Answers.Text: Thanks Neil for your answers. I accepted and upvoted your answer.
Answers.UserDisplayName: PeterBe
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-24T18:50:52.697Z
Answers.LastActivityDate: 2021-09-24T18:56:10.123Z
Answers.LastEditDate: 2021-09-24T18:56:10.123Z
Answers.UserDisplayName: Neil Slater
Body: <p>I have a control problem for a heating device of a building with the goal to minimize the electricity costs for one day under a varying price for electricity in every hour.
  (more details can be seen here as well: <a href="https://ai.stackexchange.com/questions/28888/reinforcement-learning-applicable-to-a-scheduling-problem">Reinforcement learning applicable to a scheduling problem</a>?).</p>
  <p>I also want to test two further goals (minimize peak load and maximize PV self-consumption rate).</p>
  <p>My problem also has about 10 constraints that should not be violated. I have two main questions about how to integrate the constraints into the Reinforcement Learning agent:</p>
  <p>Here are my two main questions (with following minor questions):</p>
  <p>(1) Basically I have three goals with normalized rewards between 0 and 1 for every time-slot and I have 10 constraints.</p>
  <p>  Should the constraints reward also be normalized for all 10 constraints? And then should I choose a higher weight for the most important constraint than for all three goals combined such that a constraint violation is more crucial than getting a better objective value for all the three goals?</p>
  <p>(2) Is it also possible to tell the Reinforcement Learning agent some rules directly without any constraints?</p>
  <p>  E.g. I have two storage systems, and the agent is only allowed to heat up 1 for every time-slot. Further, the agent should not start and stop heating up frequently (like around four starts of the device daily is desirable).</p>
  <p>  Can I explicitly tell these rules to the agent? Or do I have to do it indirectly by calculating a reward for every of these constraints and incorporate the weighted reward into the overall reward function of the agent?</p>
  <p>I'll appreciate any suggestion and comment.</p>
  
Tags: <reinforcement-learning><control-problem>
Title: How to Weigt Constraints in A Control Problem with Reinforcement Learning
UserDisplayName: PeterBe
ThreadText: # How to Weigt Constraints in A Control Problem with Reinforcement Learning
  
  (Asked by: username_0 on 2021-08-31)
  
  I have a control problem for a heating device of a building with the goal to minimize the electricity costs for one day under a varying price for electricity in every hour.
  (more details can be seen here as well: [Reinforcement learning applicable to a scheduling problem](https://ai.stackexchange.com/questions/28888/reinforcement-learning-applicable-to-a-scheduling-problem)?).
  
  I also want to test two further goals (minimize peak load and maximize PV self-consumption rate).
  
  My problem also has about 10 constraints that should not be violated. I have two main questions about how to integrate the constraints into the Reinforcement Learning agent:
  
  Here are my two main questions (with following minor questions):
  
  (1) Basically I have three goals with normalized rewards between 0 and 1 for every time-slot and I have 10 constraints.
  
    Should the constraints reward also be normalized for all 10 constraints? And then should I choose a higher weight for the most important constraint than for all three goals combined such that a constraint violation is more crucial than getting a better objective value for all the three goals?
  
  (2) Is it also possible to tell the Reinforcement Learning agent some rules directly without any constraints?
  
    E.g. I have two storage systems, and the agent is only allowed to heat up 1 for every time-slot. Further, the agent should not start and stop heating up frequently (like around four starts of the device daily is desirable).
  
    Can I explicitly tell these rules to the agent? Or do I have to do it indirectly by calculating a reward for every of these constraints and incorporate the weighted reward into the overall reward function of the agent?
  
  I'll appreciate any suggestion and comment.
  
  **Tags:** reinforcement-learning, control-problem
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-09-24. Score: 1 | Accepted answer
  
  > Should the constraints reward also be normalized for all 10 constraints?
  
  You should choose a "natural" balance between rewards where possible.
  
  If you have many separate goals to take account of, ideally you should convert them all into some comparable metric that is meangful to the success of the agent. Such as a financial gain/loss, or energy gain/loss, or similar. You can normalise them after this, but the ratios between the values should be kept the same.
  
  This is not always possible with constraints.
  
  For strict constraints, you should ideally ensure that breaking any constraint will score worse than not breaking the constraint but scoring very badly at everything else. If your system is gaining positive rewards from operating within bounds (and it seems that it is from the description), then one simple way to achieve this is to terminate the episode early and score $0$ for breaking the constraint. If the constraint relates to an ongoing state measurement this may be the best option. That is because the agent learning how to "escape" from an unreachable state may not be useful.
  
  For soft constraints, you need to decide a relative cost. For example your constraint:
  
  > the agent should not start and stop heating up frequently (like around four starts of the device daily is desirable)
  
  Looks very much like a soft constraint - using words like "around" and "desirable". For something like that, I would probably allow four starts, then add largish penalty for each start beyond that. What that value is, should be related back to the natural balance between rewards and why you want this constraint.
  
  As an aside, in order for the agent to learn about this constraint, you *must* add the number of starts so far for each device to the state. This is true for all constraints - there must be data inside the current state that the agent could use to predict that the constraint will come into play. It doesn't need to know the limit you are applying, but does need to know the current value of any variables used to decide whether a limit should be enforced.
  
  I think you will also want to store what the last action was, or which device is currently on, so that the agent knows to keep it on in order not to waste one of its four uses of the device per day.
  
  > Is it also possible to tell the Reinforcement Learning agent some rules directly without any constraints?
  
  Yes for absolute rules that should prevent the agent taking a specific action in the first place, and that will also be enforced in any production system. For example this rule:
  
  > I have two storage systems, and the agent is only allowed to heat up 1 for every time-slot.
  
  Can be easily expressed by having three actions:
  
  * $a\_0$ no heating
  * $a\_1$ heat device 1
  * $a\_2$ heat device 2
  
  Although you may want:
  
  * $a\_0$ do nothing
  * $a\_1$ start device 1 (and stop device 2 if running)
  * $a\_2$ start device 2 (and stop device 1 if running)
  * $a\_3$ stop whichever device is running
  
  This second list may help the agent from flip/flopping devices during early stages of learning, and speed things up a little.
  
  Other action sets could work too. The important thing is that there is nothing to gain by presenting the agent with action choices that you are ruling out as not possible immediately. It would mean adding more negative rewards and constraints to what already looks like a complex problem.
  
  > **Comments:**
  > **username_0**: Thanks a lot Neil for your answer. I have 3 questions on that. 1) About Hard constraints you wrote "terminate the episode early and score 0 for breaking the constraint." --> What do you mean by terminating the episode? To finish the RL and give a reward of 0? I think in my case this would happen many times such that the learning might suffer from that. 2) About the hard constraints you wrote "That is because the agent learning how to "escape" from an unreachable state may not be useful." --> What do you mean by that?  
  > **username_0**: 3) About the absolute rules I have not 3 but 2 actions (one for each story). Can I just directly change the actions in each iteration of the learning such that this constraint is not violated? For example if the RL agent wanted to assign a positive value for each 2 actions (which is not possible) I would just overwrite these proposed actions from RL with my own rule? Where would I have to do this? In the step function of the environment or shall I rather define a new separate control class that can overrule the actions of the RL agent.  
  > **username_1**: @PeterBe: For (1) Yes that's what I mean. For (2) I mean that if a state is not reachable, learning a policy decision for it may be a waste of time. For (3) I don't understand your comment, but I _think_ you are saying that you have a continuous action vector, or at the least multiple settings for each heater. In either case this is *not* two actions, but more - potentially a lot more. However, you can still expand on the idea in the answer - the point is not to *adjust* things afterwards, but to construct the action space so that only valid actions are possible.  
  > **username_0**: Thanks Neil for your answer and effort. I really appreciate it. Regarding your answer to 2) " I mean that if a state is not reachable, learning a policy decision for it may be a waste of time." --> how can I make sure that this is not happening? Regarding your answer to 3) "the point is not to adjust things afterwards, but to construct the action space so that only valid actions are possible." --> how can I do that? I have two continuous variables x(t) and y(t). Now IF x(t) is >0 THEN y(t) has to be 0 and vice versa with x(t) and y(t) element 0 or [0.25,1]. How can tell this to the agent?  
  > **username_1**: For (2) the assumption is that you have a state space constraint. So for certain don't train on data that starts from unreachable states according to your constraint. For (3) I think that would be a good separate question on the site. You can also choose to interpret the lower figure as zero regardless in terms of effect. I don't know whether that would be better than your options for remodelling the action space to force only valid actions.  
  > **username_0**: Thanks Neil for your answer and effort. I really appreciate it. Regarding (3) I have aske a separate question (https://ai.stackexchange.com/questions/32042/is-it-possible-to-tell-the-reinforcement-learning-agent-some-rules-directly-with). Bur regarding (2) I still do not understand your answer. How can I make sure "that if a state is not reachable, learning a policy decision for it may be a waste of time." that this is not happening?  
  > **username_1**: For (2) by not training your system on the unreachable states as starting states. You have  already decided a constraint that some state is not reachable, and you can detect this (otherwise you would not be able to calculate any penalty for reaching that state). So skip memorising and/or training for any (s,a) pair where s is outside your constraints. You will still want to train for (s,a,r,s') wheer s' is unreachable in orderto apply the constraint penalty and stop the agent going there.  
  > **username_0**: Thanks Neil for your answer. Actually the control system always begins at the beginning of the day and the states there are all set to valid values. So the starting states are always valid states in my example. Thus it is not possible to start the training from unreachable states. I don't really understand the second part of your comment. The training always beginns with valid states but it can happen that the states violate some constraints. For this I use a penalty such that the agent gets bad rewards when beeing in such a state. If the agent is in such a state it should try to get out of it  
  > **username_1**: @PeterBe Yes that's fine if it is a soft constraint and that state can actually occur physically in the real system. My answer about excluding states from training is regarding strict constraints. Your question includes examples ranging from soft constraints to strict/hard constraints, so I have done my best to cover both in the answer. Only you can say when looking at a constraint whether it is a hard or soft constraint for the system  
  > **username_1**: In general, RL will cope better with soft constraints for state values, with negative rewards, but it can also cope OK with failure/early termination as a solution for hard constraints. You also asked about avoiding termination - my answers there are with regard to hard constraints only.  
  > **username_0**: Thanks Neil for your answers. I accepted and upvoted your answer.  
  
  ---
  
  