Answers.Body: <p>In general, you <em>can</em> use a simulation to prepare and train a controller for a real world application. A good example of this being done for robotics is in the paper <a href="https://people.eecs.berkeley.edu/~jordan/papers/ng-etal03.pdf" rel="nofollow noreferrer">Autonomous helicopter flight via reinforcement learning</a> where a Reinforcement Learning agent was trained on a model of helicopter dynamics before being used in reality. Often, as in this case, such work is done to avoid expensive failures due to the trial and error nature of RL - if an error is expensive, such as crashing a helicopter, then ideally the agent performs the checks to avoid it in simulation, by planning or some other virtual environment as opposed to in the real world.</p>
  
  <p>The main hurdle to completing training in simulation then transfering to the real world, is the fidelity of the simulation. The simulation's physics, including measurements of physical quantities, the size of time steps, amount of randomness/noise, should match between the simulation and the target real-world environment. If they do not match, then a learning agent could generate a policy that works in simulation, but that fails in reality.</p>
  
  <p>For the autonomous helicopter, the researchers used data from a human operator controlling the real helicopter, to help generate a predictive model that was used in the simulation. </p>
  
  <p>Can you do the same with Open AI Gym environments? Probably not, unfortunately. The main issue is that the units used are fixed in most environments, and are unlikely to closely relate to any specific real world implementation of the same kind of system. In addition, the physics is often simplified - probably a minor issue for CartPole, but a more major one for environments like LunarLander which ignores weight of fuel used and is a 2D simulation of a 3D environment.</p>
  
  <p>So, for instance, in CartPole environments, the following values are fixed:</p>
  
  <ul>
  <li>Size of time step</li>
  <li>Mass of cart</li>
  <li>Mass and length of pole to be balanced</li>
  <li>Force that cart motor pushes with</li>
  </ul>
  
  <p>There are a couple of approaches you could use to work around this:</p>
  
  <ol>
  <li><p>Make a new version of the environment and adjust it so that values match to a real environment you want to train for. Note this may still be limited, as the physics model is still quite simple, and may not allow for the real operating characteristics of the cart motor.</p></li>
  <li><p>Use the CartPole environment as-is, not to train a controller directly, but to select hyper parameters, such as neural network size, learning rate etc. That will result in a learning agent that you are reasonably confident <em>can</em> learn policies with the state representation and general behaviour of your target system. You then train "for real" again in the physical system.</p></li>
  </ol>
  
  <p>You can combine these ideas, creating a best-guess controller from simulation, then refining it in a real environment by continuing the training on a real system.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-19T14:31:21.817Z
Answers.Text: Thanks. Since I am new to machine learning, I don't know much about it. In ordinary control systems that I am familiar with, we use PID to tune the controller.  I've seen some videos of an inverted pendulum using reinforcement learning.  How does reinforcement learning work? Does it learn to tune the values of PID automatically? Or is it a different algorithm altogether?
Answers.UserDisplayName: blazingcannon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-19T16:20:31.043Z
Answers.Text: @blazingcannon: I cannot answer that in comments with any level of detail, and suggest you ask a new question, but take care not to make it too broad. Reinforcement learning basically uses trial and error to tune a *policy* which maps observed state to a controler output (an "action"). Although it uses different terminology to classic control, a lot of the core ideas are the same. RL can solve wider class of problems than PID, but at the expense of being less efficient where PID works well.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-19T16:46:23.640Z
Answers.Text: Thanks, I will consider asking another question. But since you said PID is more efficient than reinforcement learning, I am prompted to ask whether PID can outperform RL in situations where the environment keeps changing?
Answers.UserDisplayName: blazingcannon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-19T16:56:49.833Z
Answers.Text: @blazingcannon: That will all depend on details, such as the nature of the change and the orginal control task. You won't get any PID to play a game of Go at all, whilst the best RL agent (Alpha Zero) should cope just fine if the opponent it faced changed every turn
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-19T13:00:52.530Z
Answers.LastActivityDate: 2019-11-19T13:58:32.770Z
Answers.LastEditDate: 2019-11-19T13:58:32.770Z
Answers.UserDisplayName: Neil Slater
Body: <p>I know that classical control systems have been used to solve the problem of the inverted pendulum - <a href="https://www.youtube.com/watch?v=D3bblng-Kcc" rel="nofollow noreferrer">inverted pendulum</a>.</p>
  
  <p>But I've seen that people have also used machine learning techniques to solve this nowadays - <a href="https://www.youtube.com/watch?v=Lt-KLtkDlh8" rel="nofollow noreferrer">machine learning of inverted pendulum</a>.</p>
  
  <p>I came across a video on how to apply a machine learning technique called reinforcement learning on openAI gym - <a href="https://www.youtube.com/watch?v=C2rpcQyv5bk" rel="nofollow noreferrer">OpenAI gym reinforcement learning</a>.</p>
  
  <p>My question is, can I use this simulation and use it to train a controller for a real-world application of inverted pendulum?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-11-18T18:57:44.383Z
Comments.Text: Hi, welcome to AI Stack Exchange. I'm not sure I understand your last sentence - do you mean you want to build a controller (maybe trained using RL) using the simulation, then attempt to use it to interface to and control a physical cart pole system that you have been supplied or intend to build?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-11-19T04:31:43.710Z
Comments.Text: Yes,I was thinking I could do that.
Comments.UserDisplayName: blazingcannon
Tags: <machine-learning><reinforcement-learning><applications><open-ai><control-theory>
Title: Can OpenAI simulations be used in real world applications?
UserDisplayName: blazingcannon
ThreadText: # Can OpenAI simulations be used in real world applications?
  
  (Asked by: username_0 on 2019-11-18)
  
  I know that classical control systems have been used to solve the problem of the inverted pendulum - [inverted pendulum](https://www.youtube.com/watch?v=D3bblng-Kcc).
  
  But I've seen that people have also used machine learning techniques to solve this nowadays - [machine learning of inverted pendulum](https://www.youtube.com/watch?v=Lt-KLtkDlh8).
  
  I came across a video on how to apply a machine learning technique called reinforcement learning on openAI gym - [OpenAI gym reinforcement learning](https://www.youtube.com/watch?v=C2rpcQyv5bk).
  
  My question is, can I use this simulation and use it to train a controller for a real-world application of inverted pendulum?
  
  **Tags:** machine-learning, reinforcement-learning, applications, open-ai, control-theory
  > **Comments:**
  > **username_1**: Hi, welcome to AI Stack Exchange. I'm not sure I understand your last sentence - do you mean you want to build a controller (maybe trained using RL) using the simulation, then attempt to use it to interface to and control a physical cart pole system that you have been supplied or intend to build?  
  > **username_0**: Yes,I was thinking I could do that.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-11-19. Score: 1 | Accepted answer
  
  In general, you *can* use a simulation to prepare and train a controller for a real world application. A good example of this being done for robotics is in the paper [Autonomous helicopter flight via reinforcement learning](https://people.eecs.berkeley.edu/~jordan/papers/ng-etal03.pdf) where a Reinforcement Learning agent was trained on a model of helicopter dynamics before being used in reality. Often, as in this case, such work is done to avoid expensive failures due to the trial and error nature of RL - if an error is expensive, such as crashing a helicopter, then ideally the agent performs the checks to avoid it in simulation, by planning or some other virtual environment as opposed to in the real world.
  
  The main hurdle to completing training in simulation then transfering to the real world, is the fidelity of the simulation. The simulation's physics, including measurements of physical quantities, the size of time steps, amount of randomness/noise, should match between the simulation and the target real-world environment. If they do not match, then a learning agent could generate a policy that works in simulation, but that fails in reality.
  
  For the autonomous helicopter, the researchers used data from a human operator controlling the real helicopter, to help generate a predictive model that was used in the simulation.
  
  Can you do the same with Open AI Gym environments? Probably not, unfortunately. The main issue is that the units used are fixed in most environments, and are unlikely to closely relate to any specific real world implementation of the same kind of system. In addition, the physics is often simplified - probably a minor issue for CartPole, but a more major one for environments like LunarLander which ignores weight of fuel used and is a 2D simulation of a 3D environment.
  
  So, for instance, in CartPole environments, the following values are fixed:
  
  * Size of time step
  * Mass of cart
  * Mass and length of pole to be balanced
  * Force that cart motor pushes with
  
  There are a couple of approaches you could use to work around this:
  
  1. Make a new version of the environment and adjust it so that values match to a real environment you want to train for. Note this may still be limited, as the physics model is still quite simple, and may not allow for the real operating characteristics of the cart motor.
  2. Use the CartPole environment as-is, not to train a controller directly, but to select hyper parameters, such as neural network size, learning rate etc. That will result in a learning agent that you are reasonably confident *can* learn policies with the state representation and general behaviour of your target system. You then train "for real" again in the physical system.
  
  You can combine these ideas, creating a best-guess controller from simulation, then refining it in a real environment by continuing the training on a real system.
  
  > **Comments:**
  > **username_0**: Thanks. Since I am new to machine learning, I don't know much about it. In ordinary control systems that I am familiar with, we use PID to tune the controller.  I've seen some videos of an inverted pendulum using reinforcement learning.  How does reinforcement learning work? Does it learn to tune the values of PID automatically? Or is it a different algorithm altogether?  
  > **username_1**: @blazingcannon: I cannot answer that in comments with any level of detail, and suggest you ask a new question, but take care not to make it too broad. Reinforcement learning basically uses trial and error to tune a *policy* which maps observed state to a controler output (an "action"). Although it uses different terminology to classic control, a lot of the core ideas are the same. RL can solve wider class of problems than PID, but at the expense of being less efficient where PID works well.  
  > **username_0**: Thanks, I will consider asking another question. But since you said PID is more efficient than reinforcement learning, I am prompted to ask whether PID can outperform RL in situations where the environment keeps changing?  
  > **username_1**: @blazingcannon: That will all depend on details, such as the nature of the change and the orginal control task. You won't get any PID to play a game of Go at all, whilst the best RL agent (Alpha Zero) should cope just fine if the opponent it faced changed every turn  
  
  ---
  
  