Answers.Body: <p>I'm not sure why you need a continuing environment, but actually you can make most (if not all) OpenAI Gym environments continuing. When you perform a step, you receive information about the next state, the reward, a termination signal and a dictionary with additional information. Simply ignore the termination signal if you want the episodes to continue indefinitely. In some cases you will need to modify a variable of the environments called <span class="math-container">$\texttt {_max_episode_steps}$</span>, which may force the simulation to stop or to restart.</p>
  
  <p>About your second question, check a resource called <a href="https://spinningup.openai.com/en/latest/index.html" rel="nofollow noreferrer">Spinning Up</a>, from OpenAI too. They explain several methods and their implementation. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-09T07:49:32.480Z
Answers.Text: For most OpenAI gym environments, your suggested modifications will unconstrain the episode lengths, but that is not the same as making them continuous. E.g. this won't make MountainCar or LunarLander continuous - they are still goal driven to a terminal state, and any successful agent will solve them in that way.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-09T21:47:24.860Z
Answers.Text: I don't understand your point. The inverted pendulum is also driven to a terminal state, but can be understood as a continuing problem. What is the difference? Nonetheless, if you do these modifications to MuJoCo environments, they certainly will be continuing environments with no terminal state.
Answers.UserDisplayName: Diego Gomez
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-10T04:17:26.567Z
Answers.Text: Thanks! Spinning Up is cool, why on earth don't they point to it on the OpenAI Gym main pages?
Answers.UserDisplayName: user118967
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-10T04:18:13.743Z
Answers.Text: I need a continuing environment because I am working on a problem with one, although now I realize that I can still train them using a time limit and the use them in the continuing scenario.
Answers.UserDisplayName: user118967
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-10T04:20:44.747Z
Answers.Text: Regarding Neil Slater's point, I think he means using your trick on environments that have a natural termination may be tricky. For example, once the mountain car is on top or the lunar lander is crashed, what does even mean to continue the episode? I was asking about environments that are naturally continuing and that can keep going indefinitely without large quantitative changes like reaching the mountain top or landing.
Answers.UserDisplayName: user118967
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-10T06:52:51.803Z
Answers.Text: @DiegoGomez Yes user118967 has stated the same concern, not this is not addressed in your answer. I.e. you cannot turn an episodic problem into a continuous one simply by allowing more time steps.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-09T03:15:31.753Z
Answers.LastActivityDate: 2019-12-09T03:15:31.753Z
Answers.UserDisplayName: Diego Gomez
Body: <p>I would like to use OpenAI Gym to solve a <em>continuing</em> environment, that is, a problem with a single, never-ending episode (please note I don't mean a <em>continuous</em> environment with continuous state and actions).</p>
  
  <p>The only continuing environment I found in their repository was the classic <a href="https://gym.openai.com/envs/Pendulum-v0/" rel="nofollow noreferrer">inverted pendulum problem</a>, and I found no baseline methods (algorithms) that don't require episodic environments.</p>
  
  <p>So I have two questions:</p>
  
  <ul>
  <li><p>are there any continuing environments other than the inverted pendulum one?</p></li>
  <li><p>is there an OpenAI Gym baseline method that I can use to solve the inverted pendulum problem as well as other continuing environments?</p></li>
  </ul>
  
Tags: <reinforcement-learning><open-ai><gym>
Title: Are there OpenAI Gym continuing environments (other than inverted pendulum) and baselines?
UserDisplayName: user118967
ThreadText: # Are there OpenAI Gym continuing environments (other than inverted pendulum) and baselines?
  
  (Asked by: username_0 on 2019-12-05)
  
  I would like to use OpenAI Gym to solve a *continuing* environment, that is, a problem with a single, never-ending episode (please note I don't mean a *continuous* environment with continuous state and actions).
  
  The only continuing environment I found in their repository was the classic [inverted pendulum problem](https://gym.openai.com/envs/Pendulum-v0/), and I found no baseline methods (algorithms) that don't require episodic environments.
  
  So I have two questions:
  
  * are there any continuing environments other than the inverted pendulum one?
  * is there an OpenAI Gym baseline method that I can use to solve the inverted pendulum problem as well as other continuing environments?
  
  **Tags:** reinforcement-learning, open-ai, gym
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-12-09. Score: 1
  
  I'm not sure why you need a continuing environment, but actually you can make most (if not all) OpenAI Gym environments continuing. When you perform a step, you receive information about the next state, the reward, a termination signal and a dictionary with additional information. Simply ignore the termination signal if you want the episodes to continue indefinitely. In some cases you will need to modify a variable of the environments called $\texttt {\_max\_episode\_steps}$, which may force the simulation to stop or to restart.
  
  About your second question, check a resource called [Spinning Up](https://spinningup.openai.com/en/latest/index.html), from OpenAI too. They explain several methods and their implementation.
  
  > **Comments:**
  > **username_2**: For most OpenAI gym environments, your suggested modifications will unconstrain the episode lengths, but that is not the same as making them continuous. E.g. this won't make MountainCar or LunarLander continuous - they are still goal driven to a terminal state, and any successful agent will solve them in that way.  
  > **username_1**: I don't understand your point. The inverted pendulum is also driven to a terminal state, but can be understood as a continuing problem. What is the difference? Nonetheless, if you do these modifications to MuJoCo environments, they certainly will be continuing environments with no terminal state.  
  > **username_0**: Thanks! Spinning Up is cool, why on earth don't they point to it on the OpenAI Gym main pages?  
  > **username_0**: I need a continuing environment because I am working on a problem with one, although now I realize that I can still train them using a time limit and the use them in the continuing scenario.  
  > **username_0**: Regarding username_2 username_2's point, I think he means using your trick on environments that have a natural termination may be tricky. For example, once the mountain car is on top or the lunar lander is crashed, what does even mean to continue the episode? I was asking about environments that are naturally continuing and that can keep going indefinitely without large quantitative changes like reaching the mountain top or landing.  
  > **username_2**: @username_1 Yes username_0 has stated the same concern, not this is not addressed in your answer. I.e. you cannot turn an episodic problem into a continuous one simply by allowing more time steps.  
  
  ---
  
  