Answers.Body: <p>I think the comments are basically on the right track.</p>
  
  <p>PID controllers are useful for finding optimal policies in continuous dynamical systems, and often these domains are also used as benchmarks for RL, precisely because there is an easily derived optimal policy. However, in practice, you'd obviously prefer a PID controller for any domain in which you can easily design one: the controller's behaviors are well understood, while RL solutions are often difficult to interpret. </p>
  
  <p>Where RL shines is in tasks where we know what good behaviour looks like (i.e., we know the reward function), and we know what sensor inputs look like (i.e. we can completely and accurately describe a given state numerically), but we have little or no idea what we actually want the agent to do to achieve those rewards.</p>
  
  <p>Here's a good example:</p>
  
  <ul>
  <li><p>If I wanted to make an agent to <strong>maneuver a plane</strong> from in front of an enemy plane with known movement patterns to behind it, using the least amount of fuel, I'd much prefer to <strong>use a PID controller</strong>.</p></li>
  <li><p>If I wanted to make an agent to <strong>control a plane</strong> and shoot down an enemy plane with enough fuel left to land, but without a formal description of how the enemy plane might attack (perhaps a human expert will pilot it in simulations against our agent), I'd much <strong>prefer RL</strong>. </p></li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-21T03:07:10.173Z
Answers.Text: Nice example. Can u also include MPC in this example ?
Answers.UserDisplayName: Pe Dro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-18T21:36:05.657Z
Answers.Text: Personal example: in one of the labs I used to work in, years of halfhearted work with PID control had failed to achieve anything impressive to electrically stimulate human arms to go from a start position to a target position. Reinforcement learning quickly discovered much better control, since we just weren't good at applying control theory well enough. Might be called brutish, but it worked.
Answers.UserDisplayName: David Cian
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-22T23:56:44.683Z
Answers.LastActivityDate: 2019-05-22T23:56:44.683Z
Answers.UserDisplayName: John Doucette
Body: <p>When designing solutions to problems such as the Lunar Lander on <a href="https://gym.openai.com/envs/LunarLander-v2/" rel="noreferrer">OpenAIGym</a>, Reinforcement Learning is a tempting means of giving the agent adequate action control so as to successfully land.  </p>
  
  <p>But what are the instances in which control system algorithms, such as <a href="https://en.wikipedia.org/wiki/PID_controller" rel="noreferrer">PID controllers</a>, would do just an adequate job as, if not better than, Reinforcement Learning?</p>
  
  <p>Questions such as <a href="https://ai.stackexchange.com/questions/11375/what-is-the-difference-between-reinforcement-learning-and-optimal-control">this one</a> do a great job at addressing the theory of this question, but do little to address the practical component.</p>
  
  <p>As an Artificial Intelligence engineer, what elements of a problem domain should suggest to me that a PID controller is insufficient to solve a problem, and a Reinforcement Learning algorithm should instead be used (or vice versa)?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-05-22T18:27:41.923Z
Comments.Text: actually it's not too much work, its pretty standard in industry, using modern system design tools like MATLAB you can tune PID or any other controller relatively easy to satisfy your needs. Reinforcement learning is not applied in practice since it needs abundance of data and there are no theoretical garanties like there is for classic control theory. By the way, contoller design does not involve working directly with integrals/differentials, for linear systems all the work is done in Laplace domain which involves simple algebraic manipulations
Comments.UserDisplayName: Brale
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-05-22T22:39:18.387Z
Comments.Text: As a good rule of thumb that helped me in past projects, if you cant explain explain the optimal policy (PID, RL, or otherwise) in a few sentences, PIDs will be really really hard. What's the optimal policy for Pacman?
Comments.UserDisplayName: Jaden Travnik
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-08-21T03:06:25.697Z
Comments.Text: Wouldn't it be great to lay a comparison between MPC (Model Predictive Control) and RL too ?
Comments.UserDisplayName: Pe Dro
Tags: <reinforcement-learning><ai-design><control-theory>
Title: When should I use Reinforcement Learning vs PID Control?
UserDisplayName: SeeDerekEngineer
ThreadText: # When should I use Reinforcement Learning vs PID Control?
  
  (Asked by: username_0 on 2019-05-22)
  
  When designing solutions to problems such as the Lunar Lander on [OpenAIGym](https://gym.openai.com/envs/LunarLander-v2/), Reinforcement Learning is a tempting means of giving the agent adequate action control so as to successfully land.
  
  But what are the instances in which control system algorithms, such as [PID controllers](https://en.wikipedia.org/wiki/PID_controller), would do just an adequate job as, if not better than, Reinforcement Learning?
  
  Questions such as [this one](https://ai.stackexchange.com/questions/11375/what-is-the-difference-between-reinforcement-learning-and-optimal-control) do a great job at addressing the theory of this question, but do little to address the practical component.
  
  As an Artificial Intelligence engineer, what elements of a problem domain should suggest to me that a PID controller is insufficient to solve a problem, and a Reinforcement Learning algorithm should instead be used (or vice versa)?
  
  **Tags:** reinforcement-learning, ai-design, control-theory
  
  > **Comments:**
  > **username_1**: actually it's not too much work, its pretty standard in industry, using modern system design tools like MATLAB you can tune PID or any other controller relatively easy to satisfy your needs. Reinforcement learning is not applied in practice since it needs abundance of data and there are no theoretical garanties like there is for classic control theory. By the way, contoller design does not involve working directly with integrals/differentials, for linear systems all the work is done in Laplace domain which involves simple algebraic manipulations  
  > **username_2**: As a good rule of thumb that helped me in past projects, if you cant explain explain the optimal policy (PID, RL, or otherwise) in a few sentences, PIDs will be really really hard. What's the optimal policy for Pacman?  
  > **username_3**: Wouldn't it be great to lay a comparison between MPC (Model Predictive Control) and RL too ?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_4 on 2019-05-22. Score: 10 | Accepted answer
  
  I think the comments are basically on the right track.
  
  PID controllers are useful for finding optimal policies in continuous dynamical systems, and often these domains are also used as benchmarks for RL, precisely because there is an easily derived optimal policy. However, in practice, you'd obviously prefer a PID controller for any domain in which you can easily design one: the controller's behaviors are well understood, while RL solutions are often difficult to interpret.
  
  Where RL shines is in tasks where we know what good behaviour looks like (i.e., we know the reward function), and we know what sensor inputs look like (i.e. we can completely and accurately describe a given state numerically), but we have little or no idea what we actually want the agent to do to achieve those rewards.
  
  Here's a good example:
  
  * If I wanted to make an agent to **maneuver a plane** from in front of an enemy plane with known movement patterns to behind it, using the least amount of fuel, I'd much prefer to **use a PID controller**.
  * If I wanted to make an agent to **control a plane** and shoot down an enemy plane with enough fuel left to land, but without a formal description of how the enemy plane might attack (perhaps a human expert will pilot it in simulations against our agent), I'd much **prefer RL**.
  
  > **Comments:**
  > **username_3**: Nice example. Can u also include MPC in this example ?  
  > **username_5**: Personal example: in one of the labs I used to work in, years of halfhearted work with PID control had failed to achieve anything impressive to electrically stimulate human arms to go from a start position to a target position. Reinforcement learning quickly discovered much better control, since we just weren't good at applying control theory well enough. Might be called brutish, but it worked.  
  
  ---
  
  