Answers.Body: <p>As a supplement to nbro's nice answer, I think a major difference between RL and optimal control lies in the motivation behind the problem you're solving. As has been pointed out by comments and answers here (as well as the OP), the line between RL and optimal control can be quite blurry.</p>
  <p>Consider the <a href="https://en.wikipedia.org/wiki/Linear%E2%80%93quadratic%E2%80%93Gaussian_control" rel="noreferrer">Linear-Quadratic-Gaussian (LQG)</a> algorithm, which is generally considered to be an optimal control method. Here a controller is computed given a stochastic model of the environment and a cost function.</p>
  <p>Now, consider AlphaZero, which is obviously thought of as an RL algorithm. AlphaZero learns a value function (and thus a policy/controller) in a perfect information setting with a known deterministic model.</p>
  <p>So, it's not the stochasticity that separates RL from optimal control, as some people believe. It's also not the presence of a known model. I argue that the difference between RL and optimal control comes from the <strong>generality</strong> of the algorithms.</p>
  <p>For instance, generally, when applying LQG and other optimal control algorithms, you have a specific environment in mind and the big challenge is modeling the environment and the reward function to achieve the desired behavior. In RL, on the other hand, the environment is generally thought of as a sort of black box. While in the case of AlphaZero the model of the environment is known, the reward function itself was not designed specifically for the game of chess (for instance, it's +1 for a win and -1 for a loss, regardless of chess, go, etc.). Furthermore, the neat thing with AlphaZero is that we can use it to train agents in virtually any perfect information game without changing the algorithm at all. Another difference with RL here is that the agent iteratively improves itself, while optimal control algorithms learn controllers offline and then stay fixed.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-05T21:34:04.540Z
Answers.Text: Rather than saying "the environment is generally thought of as a sort of black box", I think you want to say that you don't have the model of environment's dynamics (i.e. the transition function/probability distribution). So, it may be better to be more specific. You may also want to mention "planning" and "dynamic programming" algorithms and how they relate to your description of "optimal control".
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-05T22:11:30.013Z
Answers.Text: That's actually not what I'm saying though -- in AlphaZero you *do* have a dynamics model. My point was that in optimal control your task depends on how you model things, while in RL you have no control over it.
Answers.UserDisplayName: harwiltz
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-05T22:19:49.277Z
Answers.Text: I am not sure if this is what you also mean, but in RL you typically have control over the reward function, so, in this sense, you can control the task.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-28T09:51:05.323Z
Answers.Text: Great answer, thank you. I would like to point out something about your last sentence: even in optimal control there are some kinds of models where the controllers can change and adapt in time, i.e. adaptive control and dual control.
Answers.UserDisplayName: A. Pesare
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-26T21:11:09.817Z
Answers.Text: I disagree that you have a complete dynamics model in the case of Alpha Go. You have the rules of the game, but you have no model of the opponent a priori. Treating the game as an MDP, the transition is the composition of the board rules and the opponent's policy, and that needs to be discovered throughout the learning process.
Answers.UserDisplayName: bayer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-05T15:25:36.050Z
Answers.LastActivityDate: 2021-02-12T20:00:39.700Z
Answers.LastEditDate: 2021-02-12T20:00:39.700Z
Answers.UserDisplayName: harwiltz
Answers.Body: <p>The same book <a href="http://incompleteideas.net/book/RLbook2020.pdf#page=36" rel="noreferrer">Reinforcement learning: an introduction</a> (2nd edition, 2018) by Sutton and Barto has a section, <strong>1.7 Early History of Reinforcement Learning</strong>, that describes what <strong>optimal control</strong> is and how it is related to reinforcement learning. I will quote the most relevant part to answer your question, but you should read all that section to have a full understanding of the relationship between optimal control and reinforcement learning.</p>
  <blockquote>
  <p>The term &quot;optimal control&quot; came into use in the late 1950s to describe the <strong>problem of designing a controller to minimize or maximize a measure of a dynamical system's behavior over time</strong>. One of the <strong>approaches</strong> to this problem was developed in the mid-1950s by <strong>Richard Bellman</strong> and others through extending a nineteenth-century theory of Hamilton and Jacobi. This approach uses the concepts of a dynamical system's <strong>state</strong> and of a <strong>value function</strong>, or &quot;optimal return function&quot;, to define a functional equation, now often called the <strong>Bellman equation</strong>. The <strong>class of methods</strong> for solving optimal control problems by solving this equation came to be known as <strong>dynamic programming</strong> (Bellman, 1957a). Bellman (1957b) also introduced the <strong>discrete stochastic version of the optimal control problem</strong> known as <strong>Markov decision processes (MDPs)</strong>. Ronald Howard (1960) devised the <strong>policy iteration</strong> method for MDPs. <strong>All of these are essential elements underlying the theory and algorithms of modern reinforcement learning</strong>.</p>
  </blockquote>
  <p>To answer your specific questions.</p>
  <blockquote>
  <p>In optimal control we have, controllers, sensors, actuators, plants, etc, as elements. Are these different names for similar elements in deep RL? For example, would an optimal control plant be called an environment in deep RL?</p>
  </blockquote>
  <p>Yes. In reinforcement learning (see the first footnote of the cited book on page 48), the term <a href="https://ai.stackexchange.com/q/13119/2444"><em>control</em></a> is often used as a synonym for <em>action</em>. Similarly, the term <em>controller</em> (or <a href="https://tor-lattimore.com/downloads/book/book.pdf#page=539" rel="noreferrer"><em>decision maker</em></a>) is used as a synonym for <em>agent</em> (and <a href="https://ai.stackexchange.com/a/26296/2444">sometimes also a synonym for <em>policy</em></a>, given that the policy usually defines and <em>controls</em> the agent, although the concept of the agent is more abstract and we could associate more than one policy with the same agent). The term <em>environment</em> is also used as a synonym for <em>controlled system</em> (or <em>plant</em>).</p>
  <p>See also <a href="https://tor-lattimore.com/downloads/book/book.pdf#page=539" rel="noreferrer">section <strong>38.8 Notes</strong> (page 530)</a> of the book <a href="https://tor-lattimore.com/downloads/book/book.pdf#page=539" rel="noreferrer">Bandit Algorithms</a> by Csaba Szepesvari and Tor Lattimore.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-04T22:24:23.227Z
Answers.Text: I think the controller is synonym for agent, at least Sutton & Barto agrees with this "We use the terms agent, environment, and action instead of the engineers’ terms controller, controlled system (or plant), and control signal because they are meaningful to a wider audience." (from http://incompleteideas.net/book/RLbook2020.pdf footnote p48)
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-04T22:37:08.960Z
Answers.Text: @NeilSlater Thanks for noticing this. I was actually looking for that footnote (that I knew existed because I had answered another question based on it), but, for some reason, this time I didn't find it. Anyway, many times, a policy defines and _controls_ the agent, so I would say that a policy is a controller, although an agent is a more abstract concept and we could indeed associate more than one policy with the same agent.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-06T20:01:37.603Z
Answers.Text: For posterity, the relevant part is point 6 in Section 38.8. Thanks for the link!
Answers.UserDisplayName: anaik
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-04T21:22:03.667Z
Answers.LastActivityDate: 2021-02-11T19:18:00.817Z
Answers.LastEditDate: 2021-02-11T19:18:00.817Z
Answers.UserDisplayName: nbro
Body: <p>Coming from a process (optimal) control background, I have begun studying the field of deep reinforcement learning.</p>
  <p>Sutton &amp; Barto (2015) state that</p>
  <blockquote>
  <p>particularly important (to the writing of the text) have been the contributions establishing and developing the relationships to the theory of optimal control and dynamic programming</p>
  </blockquote>
  <p>With an emphasis on the elements of reinforcement learning - that is, policy, agent, environment, etc., what are the key differences between (deep) RL and optimal control theory?</p>
  <p>In optimal control we have, controllers, sensors, actuators, plants, etc, as elements. Are these different names for similar elements in deep RL? For example, would an optimal control plant be called an environment in deep RL?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-09-04T22:22:32.063Z
Comments.Text: Most optimal control books focus on deterministic systems. RL is generally applied to systems which are probabilistic.
Comments.UserDisplayName: FourierFlux
Tags: <reinforcement-learning><comparison><terminology><sutton-barto><control-theory>
Title: What is the difference between reinforcement learning and optimal control?
UserDisplayName: Bionic Buffulo
ThreadText: # What is the difference between reinforcement learning and optimal control?
  
  (Asked by: username_0 on 2019-03-22)
  
  Coming from a process (optimal) control background, I have begun studying the field of deep reinforcement learning.
  
  Sutton & Barto (2015) state that
  
  > particularly important (to the writing of the text) have been the contributions establishing and developing the relationships to the theory of optimal control and dynamic programming
  
  With an emphasis on the elements of reinforcement learning - that is, policy, agent, environment, etc., what are the key differences between (deep) RL and optimal control theory?
  
  In optimal control we have, controllers, sensors, actuators, plants, etc, as elements. Are these different names for similar elements in deep RL? For example, would an optimal control plant be called an environment in deep RL?
  
  **Tags:** reinforcement-learning, comparison, terminology, sutton-barto, control-theory
  > **Comments:**
  > **username_1**: Most optimal control books focus on deterministic systems. RL is generally applied to systems which are probabilistic.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2020-09-04. Score: 10 | Accepted answer
  
  The same book [Reinforcement learning: an introduction](http://incompleteideas.net/book/RLbook2020.pdf#page=36) (2nd edition, 2018) by Sutton and Barto has a section, **1.7 Early History of Reinforcement Learning**, that describes what **optimal control** is and how it is related to reinforcement learning. I will quote the most relevant part to answer your question, but you should read all that section to have a full understanding of the relationship between optimal control and reinforcement learning.
  
  > The term "optimal control" came into use in the late 1950s to describe the **problem of designing a controller to minimize or maximize a measure of a dynamical system's behavior over time**. One of the **approaches** to this problem was developed in the mid-1950s by **Richard Bellman** and others through extending a nineteenth-century theory of Hamilton and Jacobi. This approach uses the concepts of a dynamical system's **state** and of a **value function**, or "optimal return function", to define a functional equation, now often called the **Bellman equation**. The **class of methods** for solving optimal control problems by solving this equation came to be known as **dynamic programming** (Bellman, 1957a). Bellman (1957b) also introduced the **discrete stochastic version of the optimal control problem** known as **Markov decision processes (MDPs)**. Ronald Howard (1960) devised the **policy iteration** method for MDPs. **All of these are essential elements underlying the theory and algorithms of modern reinforcement learning**.
  
  To answer your specific questions.
  
  > In optimal control we have, controllers, sensors, actuators, plants, etc, as elements. Are these different names for similar elements in deep RL? For example, would an optimal control plant be called an environment in deep RL?
  
  Yes. In reinforcement learning (see the first footnote of the cited book on page 48), the term [*control*](https://ai.stackexchange.com/q/13119/2444) is often used as a synonym for *action*. Similarly, the term *controller* (or [*decision maker*](https://tor-lattimore.com/downloads/book/book.pdf#page=539)) is used as a synonym for *agent* (and [sometimes also a synonym for *policy*](https://ai.stackexchange.com/a/26296/2444), given that the policy usually defines and *controls* the agent, although the concept of the agent is more abstract and we could associate more than one policy with the same agent). The term *environment* is also used as a synonym for *controlled system* (or *plant*).
  
  See also [section **38.8 Notes** (page 530)](https://tor-lattimore.com/downloads/book/book.pdf#page=539) of the book [Bandit Algorithms](https://tor-lattimore.com/downloads/book/book.pdf#page=539) by Csaba Szepesvari and Tor Lattimore.
  
  > **Comments:**
  > **username_6**: I think the controller is synonym for agent, at least Sutton & Barto agrees with this "We use the terms agent, environment, and action instead of the engineers’ terms controller, controlled system (or plant), and control signal because they are meaningful to a wider audience." (from http://incompleteideas.net/book/RLbook2020.pdf footnote p48)  
  > **username_3**: @NeilSlater Thanks for noticing this. I was actually looking for that footnote (that I knew existed because I had answered another question based on it), but, for some reason, this time I didn't find it. Anyway, many times, a policy defines and _controls_ the agent, so I would say that a policy is a controller, although an agent is a more abstract concept and we could indeed associate more than one policy with the same agent.  
  > **username_7**: For posterity, the relevant part is point 6 in Section 38.8. Thanks for the link!  
  
  ---
  
  ### Answer by username_2 on 2020-10-05. Score: 9
  
  As a supplement to nbro's nice answer, I think a major difference between RL and optimal control lies in the motivation behind the problem you're solving. As has been pointed out by comments and answers here (as well as the OP), the line between RL and optimal control can be quite blurry.
  
  Consider the [Linear-Quadratic-Gaussian (LQG)](https://en.wikipedia.org/wiki/Linear%E2%80%93quadratic%E2%80%93Gaussian_control) algorithm, which is generally considered to be an optimal control method. Here a controller is computed given a stochastic model of the environment and a cost function.
  
  Now, consider AlphaZero, which is obviously thought of as an RL algorithm. AlphaZero learns a value function (and thus a policy/controller) in a perfect information setting with a known deterministic model.
  
  So, it's not the stochasticity that separates RL from optimal control, as some people believe. It's also not the presence of a known model. I argue that the difference between RL and optimal control comes from the **generality** of the algorithms.
  
  For instance, generally, when applying LQG and other optimal control algorithms, you have a specific environment in mind and the big challenge is modeling the environment and the reward function to achieve the desired behavior. In RL, on the other hand, the environment is generally thought of as a sort of black box. While in the case of AlphaZero the model of the environment is known, the reward function itself was not designed specifically for the game of chess (for instance, it's +1 for a win and -1 for a loss, regardless of chess, go, etc.). Furthermore, the neat thing with AlphaZero is that we can use it to train agents in virtually any perfect information game without changing the algorithm at all. Another difference with RL here is that the agent iteratively improves itself, while optimal control algorithms learn controllers offline and then stay fixed.
  
  > **Comments:**
  > **username_3**: Rather than saying "the environment is generally thought of as a sort of black box", I think you want to say that you don't have the model of environment's dynamics (i.e. the transition function/probability distribution). So, it may be better to be more specific. You may also want to mention "planning" and "dynamic programming" algorithms and how they relate to your description of "optimal control".  
  > **username_2**: That's actually not what I'm saying though -- in AlphaZero you *do* have a dynamics model. My point was that in optimal control your task depends on how you model things, while in RL you have no control over it.  
  > **username_3**: I am not sure if this is what you also mean, but in RL you typically have control over the reward function, so, in this sense, you can control the task.  
  > **username_4**: Great answer, thank you. I would like to point out something about your last sentence: even in optimal control there are some kinds of models where the controllers can change and adapt in time, i.e. adaptive control and dual control.  
  > **username_5**: I disagree that you have a complete dynamics model in the case of Alpha Go. You have the rules of the game, but you have no model of the opponent a priori. Treating the game as an MDP, the transition is the composition of the board rules and the opponent's policy, and that needs to be discovered throughout the learning process.  
  
  ---
  
  