Body: <p>I'm working on a project where I train a Q-learning agent to learn an optimal control policy for a water heater. I've set up a simulation that allows the agent to explore for one year. I then examine the results of the agent's performance exploiting its optimal policy for the following year. The agent can perform the following actions (available actions depend on the state of the environment):</p>
  <ul>
  <li>Turn the electrical heating element on.</li>
  <li>Turn the electrical heating element off.</li>
  <li>Turn gas heating on.</li>
  <li>Turn gas heating off.</li>
  <li>Do nothing.</li>
  </ul>
  <p>The goal of the agent reach the target temperature (50 deg C) when hot water is scheduled. The agent is rewarded for choosing actions that produce the lowest CO2 emissions (the CO2 emissions produced from electricity vary over time).</p>
  <p>One of the issues I have noticed is that during the exploration phase, the agent tries a lot of weighted random actions which causes the water heater to overheat (&gt;80 deg C). When the water heater overheats, it is not possible for the agent to perform further actions other than switching off the heating and doing nothing. The agent is also punished for reaching the overheating tank state. The tank may remain in the overheated state for some time. It seems as if the tendency to overheat the tank during exploration is negatively impacting how the agent learns its policy as it reduces the number of experiences in other states.</p>
  <p>Is there a term for this kind of situation during exploration in reinforcement learning? During exploration, the agent uses a chooses a softmax weighted random action. Are there alternative ways of choosing actions that may still allow for exploration while not reaching the overheating state?</p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-05-10T22:21:26.253Z
Comments.Text: what is that you want the agent to learn? isn't switching off/doing nothing when the desired temperature is attained the end goal?
Comments.UserDisplayName: okkhoy
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-05-10T22:40:38.323Z
Comments.Text: @okkhoy I want the agent to learn how to perform a set of actions which leads to the tank being at the target temperature when hot water is required. The problem is that during the exploration phase, the agent spends far more time in the negative overheating state. Once the water heater overheats, it can take some time for it to cool down to a point where the agent can perform other actions/begin to reach the target temperature. 
  
  I've updated my post to clarify the goal of the agent.
Comments.UserDisplayName: Darkphenom
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-05-11T10:19:02.480Z
Comments.Text: @Darkphenom Your project concept scenario does not require an agent,it's just require you to re-invent the wheel...in this case you don't require an agent..I did that project in c++[embedded systems programming]
Comments.UserDisplayName: quintumnia
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-05-11T10:23:51.517Z
Comments.Text: @Darkphenom the way I see it, during the exploration phase, learning to do nothing when it has over heated is also a part of the learning process. if the tank got over heated, then the agent should know it should allow it to cool down. this way not doing anything is also useful, no? is there some detail missing which I am not grasping?
Comments.UserDisplayName: okkhoy
Tags: <machine-learning><reinforcement-learning>
Title: Agent exploration which leads to a negative state where actions are limited
UserDisplayName: Darkphenom
ThreadText: # Agent exploration which leads to a negative state where actions are limited
  
  (Asked by: username_0 on 2017-05-10)
  
  I'm working on a project where I train a Q-learning agent to learn an optimal control policy for a water heater. I've set up a simulation that allows the agent to explore for one year. I then examine the results of the agent's performance exploiting its optimal policy for the following year. The agent can perform the following actions (available actions depend on the state of the environment):
  
  * Turn the electrical heating element on.
  * Turn the electrical heating element off.
  * Turn gas heating on.
  * Turn gas heating off.
  * Do nothing.
  
  The goal of the agent reach the target temperature (50 deg C) when hot water is scheduled. The agent is rewarded for choosing actions that produce the lowest CO2 emissions (the CO2 emissions produced from electricity vary over time).
  
  One of the issues I have noticed is that during the exploration phase, the agent tries a lot of weighted random actions which causes the water heater to overheat (>80 deg C). When the water heater overheats, it is not possible for the agent to perform further actions other than switching off the heating and doing nothing. The agent is also punished for reaching the overheating tank state. The tank may remain in the overheated state for some time. It seems as if the tendency to overheat the tank during exploration is negatively impacting how the agent learns its policy as it reduces the number of experiences in other states.
  
  Is there a term for this kind of situation during exploration in reinforcement learning? During exploration, the agent uses a chooses a softmax weighted random action. Are there alternative ways of choosing actions that may still allow for exploration while not reaching the overheating state?
  
  **Tags:** machine-learning, reinforcement-learning
  
  > **Comments:**
  > **username_1**: what is that you want the agent to learn? isn't switching off/doing nothing when the desired temperature is attained the end goal?  
  > **username_0**: @username_1 I want the agent to learn how to perform a set of actions which leads to the tank being at the target temperature when hot water is required. The problem is that during the exploration phase, the agent spends far more time in the negative overheating state. Once the water heater overheats, it can take some time for it to cool down to a point where the agent can perform other actions/begin to reach the target temperature.   I've updated my post to clarify the goal of the agent.  
  > **username_2**: @username_0 Your project concept scenario does not require an agent,it's just require you to re-invent the wheel...in this case you don't require an agent..I did that project in c++[embedded systems programming]  
  > **username_1**: @username_0 the way I see it, during the exploration phase, learning to do nothing when it has over heated is also a part of the learning process. if the tank got over heated, then the agent should know it should allow it to cool down. this way not doing anything is also useful, no? is there some detail missing which I am not grasping?  
  
  ---
  
  ## 0 answers
  
  