Body: <p>There are several occasion that reinforcement learning can be used as a control mean.
  The action is for example the set target temperature (which in many occasions change with time) and the state is for example the current temperature and other variables. The policy is then the control mean that is going to be learnt using the reinforcement learning.</p>
  
  <p>As there is a dead time (input lag) and time delay in the real world, how can one propose to tackle this problem when using reinforcement learning as a control mean? Thank you.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-10-30T15:29:07.977Z
Comments.Text: Could you explain why you think a delay between action and observable effects of the action would be a problem? Generally in reinforcement learning it is not an issue, but maybe there is some special case for your example that I am missing.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-10-30T15:31:33.073Z
Comments.Text: Related (maybe a duplicate?) https://ai.stackexchange.com/questions/8267/dealing-with-lags-in-reinforcement-learning
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-10-30T16:16:04.383Z
Comments.Text: I think you are right. This is the same problem that I ask here. I didn't find it before, but thank you.
Comments.UserDisplayName: JianNius
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-10-30T16:17:16.627Z
Comments.Text: Possible duplicate of [Dealing with Lags in Reinforcement Learning](https://ai.stackexchange.com/questions/8267/dealing-with-lags-in-reinforcement-learning)
Comments.UserDisplayName: JianNius
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-10-30T16:41:34.217Z
Comments.Text: OK. If lag effects are extreme, such that observable state does not change at all, but "momentum" is building due to actions taken on each time step, and the target may overshoot, then you may need to make some changes to the state representation. If you think that applies in your case, then it may not be a duplicate. If so, add that detail to the question  . . . otherwise it is good to have duplicates because it is another way to word the same question, so more chance of someone else looking to find the same answer.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-10-30T17:36:52.477Z
Comments.Text: Yes, it is right. The lag effects are so huge that after a certain time steps, it will be noticable but it will not overshoot in my case.
Comments.UserDisplayName: JianNius
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-10-30T17:39:08.067Z
Comments.Text: You need lag+ "momentum" for this to be a worry. If the subsystem that the agent is controlling is a traditional control with damping already built in, the lag won't matter. If the agent is directly controlling a heating element a few rooms away though, it may matter more.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-10-30T17:44:50.250Z
Comments.Text: According to your terminology maybe you are right to add "momentum". I used the term dead-time  in control system because normally a system has this property and when the dead-time is big then it poses a problem while using reinforcement learning to control it. Time delay corresponds to the delay due to the dynamics of the system in my terminology. Thanks
Comments.UserDisplayName: JianNius
Tags: <reinforcement-learning><control-problem><control-theory>
Title: Solving the dead time problem for control using reinforcement learning
UserDisplayName: JianNius
ThreadText: # Solving the dead time problem for control using reinforcement learning
  
  (Asked by: username_0 on 2019-10-30)
  
  There are several occasion that reinforcement learning can be used as a control mean.
  The action is for example the set target temperature (which in many occasions change with time) and the state is for example the current temperature and other variables. The policy is then the control mean that is going to be learnt using the reinforcement learning.
  
  As there is a dead time (input lag) and time delay in the real world, how can one propose to tackle this problem when using reinforcement learning as a control mean? Thank you.
  
  **Tags:** reinforcement-learning, control-problem, control-theory
  > **Comments:**
  > **username_1**: Could you explain why you think a delay between action and observable effects of the action would be a problem? Generally in reinforcement learning it is not an issue, but maybe there is some special case for your example that I am missing.  
  > **username_1**: Related (maybe a duplicate?) https://ai.stackexchange.com/questions/8267/dealing-with-lags-in-reinforcement-learning  
  > **username_0**: I think you are right. This is the same problem that I ask here. I didn't find it before, but thank you.  
  > **username_0**: Possible duplicate of [Dealing with Lags in Reinforcement Learning](https://ai.stackexchange.com/questions/8267/dealing-with-lags-in-reinforcement-learning)  
  > **username_1**: OK. If lag effects are extreme, such that observable state does not change at all, but "momentum" is building due to actions taken on each time step, and the target may overshoot, then you may need to make some changes to the state representation. If you think that applies in your case, then it may not be a duplicate. If so, add that detail to the question  . . . otherwise it is good to have duplicates because it is another way to word the same question, so more chance of someone else looking to find the same answer.  
  > **username_0**: Yes, it is right. The lag effects are so huge that after a certain time steps, it will be noticable but it will not overshoot in my case.  
  > **username_1**: You need lag+ "momentum" for this to be a worry. If the subsystem that the agent is controlling is a traditional control with damping already built in, the lag won't matter. If the agent is directly controlling a heating element a few rooms away though, it may matter more.  
  > **username_0**: According to your terminology maybe you are right to add "momentum". I used the term dead-time  in control system because normally a system has this property and when the dead-time is big then it poses a problem while using reinforcement learning to control it. Time delay corresponds to the delay due to the dynamics of the system in my terminology. Thanks  
  
  ---
  
  ## 0 answers
  
  