Answers.Body: <p>RL is a generic technique that can be applied to any MDP system. From the looks of it you have data to produce a state-space model of your system (system identification, excluding your existing control loops) and then you can use that to drive a simulated exploration of your process and discover a control policy. As this is a continuous process, some discretization will be required</p>
  
  <p>so yes, your quest is feasible, but not trivial! </p>
  
  <p>I experimented sometime ago with the cartpole balancing system, which is a toy, but I found that a good old PID was much better than AI :)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-27T19:03:12.193Z
Answers.LastActivityDate: 2020-04-27T19:03:12.193Z
Answers.UserDisplayName: nikos
Body: <p>I am new to RL and wish to realize a RL control for an industrial process. The goal is to control the temperature and humidity in a vegetal food production chamber. </p>
  
  <p><strong>States:</strong> External temperature and humidity, internal temperature and humidity, percentage of the proportional valves controlling heater, cooler and steam for humidity. The goal is to keep temperature and humidity in the chamber (measures) as close as possible to the desired values (the setpoints). </p>
  
  <p><strong>Agent actions:</strong> Increase/decrease the percentage of the proportional valves controlling the actuators. </p>
  
  <p><strong>Rewards:</strong> Deviation between measure and setpoints (small deviation => high reward, high deviation => low reward).</p>
  
  <p>I have data available, the history of states and actions from a real system. The actions are made by several PID controllers (some of them in cascade). So far I have about 3 month every minutes (with some stops sometimes when a chamber is for example cleaned). The data are continuously logged and every month I get more data. The data includes bad/unwanted states. </p>
  
  <p>For training the RL agent, I am planning to simulate the environment using a supervised learning model (with the predict function), probably XGboost. Is it feasible, are there pitfalls to avoid in this case?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-19T23:38:42.417Z
Comments.Text: Are you asking if it's feasible to simulate the environment using SL? Anyway, it's not clear  (to me) how you want to simulate the environment given only a labelled training set.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-20T10:41:18.560Z
Comments.Text: @nbro: OP could train a predictive model for $p(s'|s,a)$ - or maybe $p(s', r|s,a)$, depending on whether reward is imposed by starting goals, or is some measure of output of the industrial process. The simplest environment "simulation" is pretty much experience replay, you would use SL instead if you hoped to generalise from the data set, separately from simply training e.g. Q-learning. Choices such as trying to create a sampling model vs distribution model, and limits of approximate/learned envirobment models might be useful parts of any answer. Also, whether there is any benefit to the idea
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-20T11:23:57.730Z
Comments.Text: @NeilSlater Ha, right. Good points! But I think the OP should be more specific and ask a clearer and more specific answer.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-20T14:26:21.543Z
Comments.Text: @nbro Thanks for your answer. Yes the question is is it feasible.
Comments.UserDisplayName: brz
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-20T14:50:13.760Z
Comments.Text: @brz OP refers to the asker of the question. It's an acronym for "Original Poster" and in this case refers to you, so it's nothing to do with RL. Anyway, I think you should clarify how your dataset looks like and try to be more specific regarding what you want to be feasible.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-20T14:53:19.530Z
Comments.Text: OP = Original Poster i.e. you. I don't understand your description of the reward. I am not sure it is important for the purpose of this question, but I do suggest you add more details about the environment and what you are trying to do. Also, why you want to create a simulator using supervised learning - what is your motivation for that?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-20T15:14:31.173Z
Comments.Text: The goal is to control the temperature and humidity in a vegetal food production chamber. States: External temperature and humidity, internal temperature and humidity, percentage of the proportional valves controlling heater, cooler and steam for humidity. The goal is to keep temperature and humidity in the chamber (measures) as close as possible to the desired values (the setpoints). Agent actions: increase/decrease the percentage of the proportional valves controlling the actuators. Rewards: deviation between measure and setpoints (small deviation=> high reward, high deviation => low reward.
Comments.UserDisplayName: brz
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-20T15:21:03.190Z
Comments.Text: @NeilSlater I have no specific motivation for SL, just wish to compare RL algorithm and train agent before it goes in the real world. It is not clear in my previous post but the two setpoints are also part of the state space.
Comments.UserDisplayName: brz
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-20T15:52:06.727Z
Comments.Text: @NeilSlater You said. "OP could train a predictive model for p(s′|s,a) - or maybe p(s′,r|s,a)". Is it possible with a continuous space state ?
Comments.UserDisplayName: brz
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-20T15:56:42.497Z
Comments.Text: @brz: It is maybe possible to create a distributional model from your data, but might not be the best approach. It definitely is not the simplest approach. Could you please explain what you mean by "I have data available" in more detail - I would guess you have historical data for state and actions (where the actions have been taken by another controller, perhaps a human expert?). How much of this data do you have, and how comprehensively does it cover the state and action space? Does your data include bad/unwanted scenarios that the simulator can learn, and the agent can learn to avoid?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-20T16:49:03.930Z
Comments.Text: @NeilSlater Yes the data are historical of states and actions. The actions are now made by several PID controllers (some of them in cascade). So far I have about 3 month every minutes (with some stops sometimes when a chamber is for example cleaned). The data are continuously logged and every month I get more data. Yes the data are containing bad/unwanted states.
Comments.UserDisplayName: brz
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-20T17:07:54.363Z
Comments.Text: Thank you. I have edited that information into the question for you. For reference if you ask another question or want to add more details here, you can also do that yourself using the [edit] link on the left under your question text.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-09-24T22:31:54.327Z
Comments.Text: This is a very hard problem as there exist time dependencies and you don't have an ability to actually simulate this.
Comments.UserDisplayName: FourierFlux
Tags: <reinforcement-learning><supervised-learning>
Title: Reinforcement learning with industrial continuous process
UserDisplayName: brz
ThreadText: # Reinforcement learning with industrial continuous process
  
  (Asked by: username_0 on 2020-04-19)
  
  I am new to RL and wish to realize a RL control for an industrial process. The goal is to control the temperature and humidity in a vegetal food production chamber.
  
  **States:** External temperature and humidity, internal temperature and humidity, percentage of the proportional valves controlling heater, cooler and steam for humidity. The goal is to keep temperature and humidity in the chamber (measures) as close as possible to the desired values (the setpoints).
  
  **Agent actions:** Increase/decrease the percentage of the proportional valves controlling the actuators.
  
  **Rewards:** Deviation between measure and setpoints (small deviation => high reward, high deviation => low reward).
  
  I have data available, the history of states and actions from a real system. The actions are made by several PID controllers (some of them in cascade). So far I have about 3 month every minutes (with some stops sometimes when a chamber is for example cleaned). The data are continuously logged and every month I get more data. The data includes bad/unwanted states.
  
  For training the RL agent, I am planning to simulate the environment using a supervised learning model (with the predict function), probably XGboost. Is it feasible, are there pitfalls to avoid in this case?
  
  **Tags:** reinforcement-learning, supervised-learning
  > **Comments:**
  > **username_1**: Are you asking if it's feasible to simulate the environment using SL? Anyway, it's not clear  (to me) how you want to simulate the environment given only a labelled training set.  
  > **username_2**: @nbro: OP could train a predictive model for $p(s'|s,a)$ - or maybe $p(s', r|s,a)$, depending on whether reward is imposed by starting goals, or is some measure of output of the industrial process. The simplest environment "simulation" is pretty much experience replay, you would use SL instead if you hoped to generalise from the data set, separately from simply training e.g. Q-learning. Choices such as trying to create a sampling model vs distribution model, and limits of approximate/learned envirobment models might be useful parts of any answer. Also, whether there is any benefit to the idea  
  > **username_1**: @NeilSlater Ha, right. Good points! But I think the OP should be more specific and ask a clearer and more specific answer.  
  > **username_0**: @nbro Thanks for your answer. Yes the question is is it feasible.  
  > **username_1**: @brz OP refers to the asker of the question. It's an acronym for "Original Poster" and in this case refers to you, so it's nothing to do with RL. Anyway, I think you should clarify how your dataset looks like and try to be more specific regarding what you want to be feasible.  
  > **username_2**: OP = Original Poster i.e. you. I don't understand your description of the reward. I am not sure it is important for the purpose of this question, but I do suggest you add more details about the environment and what you are trying to do. Also, why you want to create a simulator using supervised learning - what is your motivation for that?  
  > **username_0**: The goal is to control the temperature and humidity in a vegetal food production chamber. States: External temperature and humidity, internal temperature and humidity, percentage of the proportional valves controlling heater, cooler and steam for humidity. The goal is to keep temperature and humidity in the chamber (measures) as close as possible to the desired values (the setpoints). Agent actions: increase/decrease the percentage of the proportional valves controlling the actuators. Rewards: deviation between measure and setpoints (small deviation=> high reward, high deviation => low reward.  
  > **username_0**: @NeilSlater I have no specific motivation for SL, just wish to compare RL algorithm and train agent before it goes in the real world. It is not clear in my previous post but the two setpoints are also part of the state space.  
  > **username_0**: @NeilSlater You said. "OP could train a predictive model for p(s′|s,a) - or maybe p(s′,r|s,a)". Is it possible with a continuous space state ?  
  > **username_2**: @brz: It is maybe possible to create a distributional model from your data, but might not be the best approach. It definitely is not the simplest approach. Could you please explain what you mean by "I have data available" in more detail - I would guess you have historical data for state and actions (where the actions have been taken by another controller, perhaps a human expert?). How much of this data do you have, and how comprehensively does it cover the state and action space? Does your data include bad/unwanted scenarios that the simulator can learn, and the agent can learn to avoid?  
  > **username_0**: @NeilSlater Yes the data are historical of states and actions. The actions are now made by several PID controllers (some of them in cascade). So far I have about 3 month every minutes (with some stops sometimes when a chamber is for example cleaned). The data are continuously logged and every month I get more data. Yes the data are containing bad/unwanted states.  
  > **username_2**: Thank you. I have edited that information into the question for you. For reference if you ask another question or want to add more details here, you can also do that yourself using the [edit] link on the left under your question text.  
  > **username_3**: This is a very hard problem as there exist time dependencies and you don't have an ability to actually simulate this.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_4 on 2020-04-27. Score: 0
  
  RL is a generic technique that can be applied to any MDP system. From the looks of it you have data to produce a state-space model of your system (system identification, excluding your existing control loops) and then you can use that to drive a simulated exploration of your process and discover a control policy. As this is a continuous process, some discretization will be required
  
  so yes, your quest is feasible, but not trivial!
  
  I experimented sometime ago with the cartpole balancing system, which is a toy, but I found that a good old PID was much better than AI :)
  
  
  ---
  
  