Answers.Body: <p>Almost certainly, there is no such paper since that would be a trivial problem. The pole lying flat is the definition of failure, hence game over. If you started in that position, you would be permanently in the game-over state and you would never learn anything. </p>
  
  <p>The reason is that if the pole is lying flat, then, if you apply a force on the cart (in the same direction as the pole is pointing, say), the pole head moves in exactly the same direction as the cart (i.e. the directional vectors of the cart and pole head movements are identical). Hence, the pole head never moves upwards. </p>
  
  <p>In fact, I am fairly certain, that below certain angle with the surface, the pole can no longer be stabilized. This should be possible to prove from the dynamic equations governing the movement of the cart and the pole. This may not be easy, though, and definitely not easy for me (these are second-order differential equations). Anyway, with this in mind, you can see why one needs to start close enough to the stationary point for the problem to have a solution. </p>
  
  <p>If the pole could go below the surface and swing, however, as <a href="https://gereshes.com/2019/08/19/swing-up-of-an-inverted-pendulum-on-a-cart/" rel="nofollow noreferrer">here</a>, it would be similar to the acrobot problem mentioned by Neil and you could start anywhere. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-29T07:15:12.530Z
Answers.Text: "Almost certainly, there is no such paper since that would be a trivial problem." it might seem trivial after it has been solved. I have in the meantime found several papers: https://scholar.google.com/scholar?hl=nl&as_sdt=0%2C5&q=swing-up+cart&btnG=. There is also no need to start near the top, the bottom is fine. But yes, it is similar to the acrobat problem.
Answers.UserDisplayName: Thomas Wagenaar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-29T16:54:10.117Z
Answers.Text: @ThomasW Trivial in mathematical sense, that was not a reference to anyone's intelligence. If you know that the pole does not move when you apply the force, then the solution is trivial (i.e. any action yields the same outcome).
Answers.UserDisplayName: PetrH
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-29T16:57:05.380Z
Answers.Text: @ThomasW The references you point to analyze the problem, where the pendulum can rotate 360 degrees. That I mentioned in my last sentence as a side note. My answer was about the pole-balancing problem where the pole can only deviate from the upright position by 90 degrees on either side, since this is the problem which you asked about.
Answers.UserDisplayName: PetrH
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-27T14:31:49.907Z
Answers.LastActivityDate: 2020-05-27T14:31:49.907Z
Answers.UserDisplayName: PetrH
Answers.Body: <p>It is difficult to prove a negative, but I doubt there will be a paper on that specific problem. It should be relatively easy to adjust the environment or write a new one that does this if you wished though.</p>
  
  <p>A very similar environment that does have a <em>lot</em> more written about it is Acrobot, <a href="https://gym.openai.com/envs/Acrobot-v1/" rel="nofollow noreferrer">which does have a OpenAI Gym version</a>. Instead of a cart on a track with forces applied and a free join to the pole, there is a longer pole fixed to a free joint, with an active joint in the middle (that the agent can apply forces to). It can be thought of as a very simple model of an acrobat on a trapeze swing (with poles instead of chains, so the swing is stable balanced upside-down)</p>
  
  <p>The degrees of freedom and difficulty of the Acrobot task are similar to CartPole - I would rate it as a harder problem overall, but if you started CartPole with the pole hanging down in fact the two problems are very similar. Usually the joint motor is made too weak to achieve the goal of balancing in a single clean action, and the agent must learn to build momentum over a few swings, before moving to the balance point. That also makes both your alternate start position CartPole and Acrobot similar to <a href="https://gym.openai.com/envs/MountainCar-v0/" rel="nofollow noreferrer">MountainCar</a>.</p>
  
  <p>For a low-effort look at a very similar problem you could try Acrobot. Otherwise you will likely need to do some custom work on CartPole.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-27T11:57:22.287Z
Answers.LastActivityDate: 2020-05-27T11:57:22.287Z
Answers.UserDisplayName: Neil Slater
Body: <p>There are a lot of examples of balancing a pole (see image below) using reinforcement learning, but I find that almost all examples start close to the upright position.</p>
  
  <p>Is there any good source (or paper) for when the pole actually starts all the way at the bottom? </p>
  
  <p><a href="https://i.sstatic.net/9NbPg.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/9NbPg.png" alt="enter image description here"></a></p>
  
Tags: <reinforcement-learning><reference-request>
Title: Is there any good source for when the pole actually starts all the way at the bottom, in the cartpole problem?
UserDisplayName: Thomas Wagenaar
ThreadText: # Is there any good source for when the pole actually starts all the way at the bottom, in the cartpole problem?
  
  (Asked by: username_0 on 2020-05-27)
  
  There are a lot of examples of balancing a pole (see image below) using reinforcement learning, but I find that almost all examples start close to the upright position.
  
  Is there any good source (or paper) for when the pole actually starts all the way at the bottom?
  
  [![enter image description here](https://i.sstatic.net/9NbPg.png)](https://i.sstatic.net/9NbPg.png)
  
  **Tags:** reinforcement-learning, reference-request
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2020-05-27. Score: 2
  
  Almost certainly, there is no such paper since that would be a trivial problem. The pole lying flat is the definition of failure, hence game over. If you started in that position, you would be permanently in the game-over state and you would never learn anything.
  
  The reason is that if the pole is lying flat, then, if you apply a force on the cart (in the same direction as the pole is pointing, say), the pole head moves in exactly the same direction as the cart (i.e. the directional vectors of the cart and pole head movements are identical). Hence, the pole head never moves upwards.
  
  In fact, I am fairly certain, that below certain angle with the surface, the pole can no longer be stabilized. This should be possible to prove from the dynamic equations governing the movement of the cart and the pole. This may not be easy, though, and definitely not easy for me (these are second-order differential equations). Anyway, with this in mind, you can see why one needs to start close enough to the stationary point for the problem to have a solution.
  
  If the pole could go below the surface and swing, however, as [here](https://gereshes.com/2019/08/19/swing-up-of-an-inverted-pendulum-on-a-cart/), it would be similar to the acrobot problem mentioned by username_2 and you could start anywhere.
  
  > **Comments:**
  > **username_0**: "Almost certainly, there is no such paper since that would be a trivial problem." it might seem trivial after it has been solved. I have in the meantime found several papers: https://scholar.google.com/scholar?hl=nl&as_sdt=0%2C5&q=swing-up+cart&btnG=. There is also no need to start near the top, the bottom is fine. But yes, it is similar to the acrobat problem.  
  > **username_1**: @ThomasW Trivial in mathematical sense, that was not a reference to anyone's intelligence. If you know that the pole does not move when you apply the force, then the solution is trivial (i.e. any action yields the same outcome).  
  > **username_1**: @ThomasW The references you point to analyze the problem, where the pendulum can rotate 360 degrees. That I mentioned in my last sentence as a side note. My answer was about the pole-balancing problem where the pole can only deviate from the upright position by 90 degrees on either side, since this is the problem which you asked about.  
  
  ---
  
  ### Answer by username_2 on 2020-05-27. Score: 2
  
  It is difficult to prove a negative, but I doubt there will be a paper on that specific problem. It should be relatively easy to adjust the environment or write a new one that does this if you wished though.
  
  A very similar environment that does have a *lot* more written about it is Acrobot, [which does have a OpenAI Gym version](https://gym.openai.com/envs/Acrobot-v1/). Instead of a cart on a track with forces applied and a free join to the pole, there is a longer pole fixed to a free joint, with an active joint in the middle (that the agent can apply forces to). It can be thought of as a very simple model of an acrobat on a trapeze swing (with poles instead of chains, so the swing is stable balanced upside-down)
  
  The degrees of freedom and difficulty of the Acrobot task are similar to CartPole - I would rate it as a harder problem overall, but if you started CartPole with the pole hanging down in fact the two problems are very similar. Usually the joint motor is made too weak to achieve the goal of balancing in a single clean action, and the agent must learn to build momentum over a few swings, before moving to the balance point. That also makes both your alternate start position CartPole and Acrobot similar to [MountainCar](https://gym.openai.com/envs/MountainCar-v0/).
  
  For a low-effort look at a very similar problem you could try Acrobot. Otherwise you will likely need to do some custom work on CartPole.
  
  
  ---
  
  