Body: <p>I want to create a <strong>continuing (non-episodic)</strong> reinforcement learning environment for chiller plant operation. There are some tutorials focusing on creating environments for the episodic case, however I couldn't find one for the non-episodic cases.</p>
  <p>My problem is involving scheduling 3 chiller plant for energy optimisation. The state, action spaces and reward function are something like what is defined below.</p>
  <p>S : {temperature, demand}</p>
  <p>A : {c1 condensor, c1 evaporator, c1 compressor, c2 condensor, ... , c3 compressor} - where c1, c2 and c3 refer to the 3 chillers</p>
  <p>R : f(suppy, load) - some function defined based on supply and load</p>
  <p>As my environment is sitting in a Simulink model I need to solve the RL problem in MATLAB (RLToolBox). However any method of creating an environment in OpenAI gym is fine as well since I believe all these frameworks work in a similar fashion.</p>
  <p>The main issue I am having is how we define the &quot;done&quot; (&quot;isdone&quot; in MATLAB) variable, as this case is non-episodic. The general procedure for the episodic case is ...</p>
  <p><code>next_state, reward, done, info = env.step(action)</code> (Gym implementation)</p>
  <p>I believe we just don't pass anything for &quot;done&quot; variable in the non episodic case. Am I correct?.
  Further if we don't pass anything to the &quot;done&quot; variable how would the algorithm eventually terminate. By this what I mean is for practical purposes even for non-episodic cases must eventually terminate (I think, maybe I am incorrect) so that things like performance criteria can be measured.</p>
  <p>Another issue is what would you pass to the environment reset function? Would it be <code>None</code> value?</p>
  <p>If so how do define a termination criteria in non-episodic cases? or does it happen from the algorithm side (i.e DDPG implementation in RLToolBox/stable baseline)?. When something like the RLAgent finding the optimal policy (q -&gt; q*) happens?</p>
  <p>Any knowledge on this much appreciated as I am just starting out RL.</p>
  <p>Thanks</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-06T06:06:51.493Z
Comments.Text: Welcome to SE.AI!  It looks like a lot of what you're asking about are straightforward programming-questions -- those questions might work on StackOverflow.  You could probably just focus on asking what the signatures and call-conventions for the relevant Matlab's RLToolBox methods are, without needing to discuss the background.
Comments.UserDisplayName: Nat
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-06T06:15:02.913Z
Comments.Text: BTW, I don't immediately recall having played with RLToolBox, but I'd guess that you could probably just do a coupled (non-episodic) environment the same way that you'd do an $n=1$ series of de-coupled (episodic) environments.
Comments.UserDisplayName: Nat
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-06T11:05:16.907Z
Comments.Text: Sure thanks, I'll post it in stackoverflow
Comments.UserDisplayName: imantha
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-06T13:53:05.820Z
Comments.Text: Side-note: Careful with that setup. Your environment has hidden state (delays, thermal mass, presumably with a different temperature you don't measure), but many RL algorithms assume a MDP, where the optimal action only depends on the visible state. E.g. they will assume past actions are irrelevant, given the state.
Comments.UserDisplayName: maxy
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-06T18:03:36.797Z
Comments.Text: Notice that in newer versions of both OpenAI Gym and Gymnasium, the `done` flag has been replaced by two variables `terminated` and truncated` where: `terminated` indicates whether or not the new state is terminal, and `truncated` instead says whether the episode have been artificially ended, by a time limit for example (which may be useful in your case.)
Comments.UserDisplayName: Luca Anzalone
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-07T06:14:07.237Z
Comments.Text: Thanks , does any one also know anything regarding how continuing environments eventually terminate (since we want at some point to use the learnt policy instead training forever)? Is it when the optimal policy (v*/q*) is found?
Comments.UserDisplayName: imantha
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-09T17:49:36.327Z
Comments.Text: You can set a *performance threshold* that when it's surpassed by the agent the training terminates. But the simplest approach is to use a *maximum number of training steps* (or updates) criterion that terminates the training when exceeded. Otherwise, you can terminate when the policy converges to some value (which may be suboptimal, e.g. due wrong hyper-parameters.)
Comments.UserDisplayName: Luca Anzalone
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-13T20:14:47.843Z
Comments.Text: Can you please put your **specific question** in the title?
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><environment>
Title: Designing Reinforcement Learning Environments for Continuing Tasks (Non-Episodic)
UserDisplayName: imantha
ThreadText: # Designing Reinforcement Learning Environments for Continuing Tasks (Non-Episodic)
  
  (Asked by: username_0 on 2023-04-06)
  
  I want to create a **continuing (non-episodic)** reinforcement learning environment for chiller plant operation. There are some tutorials focusing on creating environments for the episodic case, however I couldn't find one for the non-episodic cases.
  
  My problem is involving scheduling 3 chiller plant for energy optimisation. The state, action spaces and reward function are something like what is defined below.
  
  S : {temperature, demand}
  
  A : {c1 condensor, c1 evaporator, c1 compressor, c2 condensor, ... , c3 compressor} - where c1, c2 and c3 refer to the 3 chillers
  
  R : f(suppy, load) - some function defined based on supply and load
  
  As my environment is sitting in a Simulink model I need to solve the RL problem in MATLAB (RLToolBox). However any method of creating an environment in OpenAI gym is fine as well since I believe all these frameworks work in a similar fashion.
  
  The main issue I am having is how we define the "done" ("isdone" in MATLAB) variable, as this case is non-episodic. The general procedure for the episodic case is ...
  
  `next_state, reward, done, info = env.step(action)` (Gym implementation)
  
  I believe we just don't pass anything for "done" variable in the non episodic case. Am I correct?.
  Further if we don't pass anything to the "done" variable how would the algorithm eventually terminate. By this what I mean is for practical purposes even for non-episodic cases must eventually terminate (I think, maybe I am incorrect) so that things like performance criteria can be measured.
  
  Another issue is what would you pass to the environment reset function? Would it be `None` value?
  
  If so how do define a termination criteria in non-episodic cases? or does it happen from the algorithm side (i.e DDPG implementation in RLToolBox/stable baseline)?. When something like the RLAgent finding the optimal policy (q -> q\*) happens?
  
  Any knowledge on this much appreciated as I am just starting out RL.
  
  Thanks
  
  **Tags:** reinforcement-learning, environment
  
  > **Comments:**
  > **username_1**: Welcome to SE.AI!  It looks like a lot of what you're asking about are straightforward programming-questions -- those questions might work on StackOverflow.  You could probably just focus on asking what the signatures and call-conventions for the relevant Matlab's RLToolBox methods are, without needing to discuss the background.  
  > **username_1**: BTW, I don't immediately recall having played with RLToolBox, but I'd guess that you could probably just do a coupled (non-episodic) environment the same way that you'd do an $n=1$ series of de-coupled (episodic) environments.  
  > **username_0**: Sure thanks, I'll post it in stackoverflow  
  > **username_2**: Side-note: Careful with that setup. Your environment has hidden state (delays, thermal mass, presumably with a different temperature you don't measure), but many RL algorithms assume a MDP, where the optimal action only depends on the visible state. E.g. they will assume past actions are irrelevant, given the state.  
  > **username_3**: Notice that in newer versions of both OpenAI Gym and Gymnasium, the `done` flag has been replaced by two variables `terminated` and truncated` where: `terminated` indicates whether or not the new state is terminal, and `truncated` instead says whether the episode have been artificially ended, by a time limit for example (which may be useful in your case.)  
  > **username_0**: Thanks , does any one also know anything regarding how continuing environments eventually terminate (since we want at some point to use the learnt policy instead training forever)? Is it when the optimal policy (v*/q*) is found?  
  > **username_3**: You can set a *performance threshold* that when it's surpassed by the agent the training terminates. But the simplest approach is to use a *maximum number of training steps* (or updates) criterion that terminates the training when exceeded. Otherwise, you can terminate when the policy converges to some value (which may be suboptimal, e.g. due wrong hyper-parameters.)  
  > **username_4**: Can you please put your **specific question** in the title?  
  
  ---
  
  ## 0 answers
  
  