Answers.Body: <p>In optimal control field to minimize certain well-defined costs especially in process industries, continuous state space model-based planning methods such as <a href="https://en.wikipedia.org/wiki/Model_predictive_control" rel="nofollow noreferrer">model predictive control (MPC)</a> is a common decision time planning/control method which can handle both linear and nonlinear models, and unlike training data intensive model-free RL methods it's much more data efficient, faster, online, and good (though may not optimal):</p>
  <blockquote>
  <p>MPC is based on iterative, finite-horizon optimization of a plant model. At time
  <span class="math-container">$t$</span> the current plant state is sampled and a cost minimizing control strategy is computed (via a numerical minimization algorithm) for a relatively short time horizon in the future:<span class="math-container">$[t,t+T]$</span>. Specifically, an online or on-the-fly calculation is used to explore state trajectories that emanate from the current state and find (via the solution of Euler–Lagrange equations) a cost-minimizing control strategy until time <span class="math-container">$t+T$</span>. Although this approach is not optimal, in practice it has given very good results. Much academic research has been done to find fast methods of solution of Euler–Lagrange type equations, to understand the global stability properties of MPC's local optimization, and in general to improve the MPC method.</p>
  </blockquote>
  <p>If your model is linear (most mechanical systems) and cost is quadratic, <a href="https://en.wikipedia.org/wiki/Linear%E2%80%93quadratic_regulator" rel="nofollow noreferrer">LQR</a> is also a common optimal feedback control method.</p>
  <blockquote>
  <p>The main differences between MPC and LQR are that LQR optimizes across the entire time window (horizon) whereas MPC optimizes in a receding time window,[4] and that with MPC a new solution is computed often whereas LQR uses the same single (optimal) solution for the whole time horizon.</p>
  </blockquote>
  <p>Finally similar to Dyna-Q learning architecture with background learning &amp; planning introduced in Sutton and Barto's <em>Reinforcement Learning, An Introduction</em>, especially if your initial known sample model is not accurate or keeps evolving, one can combine model-based with model-free RL methods, you just replace tabular Q-planning there with your linear or nonlinear function approximations planning version such as described in Jong and Stone's paper <a href="https://link.springer.com/chapter/10.1007/978-3-540-73580-9_21" rel="nofollow noreferrer"><em>Model-Based Exploration in Continuous State Spaces</em></a>:</p>
  <blockquote>
  <p>This paper develops a method
  for approximating continuous models by fitting data to a finite sample of states, leading to finite representations compatible with existing
  model-based exploration mechanisms. Experiments with the resulting
  family of fitted-model reinforcement learning algorithms reveals the critical importance of how the continuous model is generalized from finite
  data. This paper demonstrates instantiations of fitted-model algorithms
  that lead to faster learning on benchmark problems than contemporary
  model-free RL algorithms that only apply generalization in estimating
  action values. Finally, the paper concludes that in continuous problems,
  the exploration-exploitation tradeoff is better construed as a balance between exploration and generalization.</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-17T21:47:12.387Z
Answers.LastActivityDate: 2023-04-17T21:52:22.553Z
Answers.LastEditDate: 2023-04-17T21:52:22.553Z
Answers.UserDisplayName: cinch
Body: <p>At the beginning, if I have a complete model <span class="math-container">$p(s' \mid s, a)$</span> (an assumed true model that describes the environment well enough) and the reward function <span class="math-container">$r(s,a,s')$</span>. How can I exploit the model and learn a good policy in this situation? Assume that the state space is continuous, and the action space is finite.</p>
  <p>Traditional dynamic programming/planning methods like <a href="http://incompleteideas.net/book/ebook/node44.html" rel="nofollow noreferrer">policy iteration or value iteration</a> cannot be directly applied since there are infinitely many states. If I try to get samples from the model and apply an algorithm like DQN or any non-linear function approximation, it looks like that I use the model-free approach and do not get the full advantages of the model.</p>
  <p>May I ask if there are any reinforcement learning/planning methods that use the explicit model given at the beginning to solve the MDP? (in case of continuous state space and finite action space)</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-17T12:11:15.007Z
Comments.Text: Are you asking which model-based RL algorithms exist (assuming that you don't consider value/policy iteration an RL algorithm but a planning algorithm)?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-17T12:23:06.190Z
Comments.Text: @nbro Yes, kind of. I understand that policy/value iteration are planning algorithms. But in [Sutton's book](http://incompleteideas.net/book/ebook/node44.html) it's only for finite state space. Assuming that the state space are continuous, and the transition function $p$ and reward function $r$ are explicitly given to the agent at the beginning, is there any method/algorithm to use the model to find a good or optimal policy?
Comments.UserDisplayName: k2pctdn
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-17T12:24:04.300Z
Comments.Text: Please, edit your post to include this info and change the title to be more specific and have your question
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-17T15:11:35.430Z
Comments.Text: @nbro thank you for your editing, I've also added some more information to the post.
Comments.UserDisplayName: k2pctdn
Tags: <reinforcement-learning><reference-request><model-based-methods><continuous-state-spaces><discrete-action-spaces>
Title: Model-based RL algorithms for continuous state space and finite action space
UserDisplayName: k2pctdn
ThreadText: # Model-based RL algorithms for continuous state space and finite action space
  
  (Asked by: username_0 on 2023-04-11)
  
  At the beginning, if I have a complete model $p(s' \mid s, a)$ (an assumed true model that describes the environment well enough) and the reward function $r(s,a,s')$. How can I exploit the model and learn a good policy in this situation? Assume that the state space is continuous, and the action space is finite.
  
  Traditional dynamic programming/planning methods like [policy iteration or value iteration](http://incompleteideas.net/book/ebook/node44.html) cannot be directly applied since there are infinitely many states. If I try to get samples from the model and apply an algorithm like DQN or any non-linear function approximation, it looks like that I use the model-free approach and do not get the full advantages of the model.
  
  May I ask if there are any reinforcement learning/planning methods that use the explicit model given at the beginning to solve the MDP? (in case of continuous state space and finite action space)
  
  **Tags:** reinforcement-learning, reference-request, model-based-methods, continuous-state-spaces, discrete-action-spaces
  > **Comments:**
  > **username_1**: Are you asking which model-based RL algorithms exist (assuming that you don't consider value/policy iteration an RL algorithm but a planning algorithm)?  
  > **username_0**: @nbro Yes, kind of. I understand that policy/value iteration are planning algorithms. But in [Sutton's book](http://incompleteideas.net/book/ebook/node44.html) it's only for finite state space. Assuming that the state space are continuous, and the transition function $p$ and reward function $r$ are explicitly given to the agent at the beginning, is there any method/algorithm to use the model to find a good or optimal policy?  
  > **username_1**: Please, edit your post to include this info and change the title to be more specific and have your question  
  > **username_0**: @nbro thank you for your editing, I've also added some more information to the post.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-04-17. Score: 1 | Accepted answer
  
  In optimal control field to minimize certain well-defined costs especially in process industries, continuous state space model-based planning methods such as [model predictive control (MPC)](https://en.wikipedia.org/wiki/Model_predictive_control) is a common decision time planning/control method which can handle both linear and nonlinear models, and unlike training data intensive model-free RL methods it's much more data efficient, faster, online, and good (though may not optimal):
  
  > MPC is based on iterative, finite-horizon optimization of a plant model. At time
  > $t$ the current plant state is sampled and a cost minimizing control strategy is computed (via a numerical minimization algorithm) for a relatively short time horizon in the future:$[t,t+T]$. Specifically, an online or on-the-fly calculation is used to explore state trajectories that emanate from the current state and find (via the solution of Euler–Lagrange equations) a cost-minimizing control strategy until time $t+T$. Although this approach is not optimal, in practice it has given very good results. Much academic research has been done to find fast methods of solution of Euler–Lagrange type equations, to understand the global stability properties of MPC's local optimization, and in general to improve the MPC method.
  
  If your model is linear (most mechanical systems) and cost is quadratic, [LQR](https://en.wikipedia.org/wiki/Linear%E2%80%93quadratic_regulator) is also a common optimal feedback control method.
  
  > The main differences between MPC and LQR are that LQR optimizes across the entire time window (horizon) whereas MPC optimizes in a receding time window,[4] and that with MPC a new solution is computed often whereas LQR uses the same single (optimal) solution for the whole time horizon.
  
  Finally similar to Dyna-Q learning architecture with background learning & planning introduced in Sutton and Barto's *Reinforcement Learning, An Introduction*, especially if your initial known sample model is not accurate or keeps evolving, one can combine model-based with model-free RL methods, you just replace tabular Q-planning there with your linear or nonlinear function approximations planning version such as described in Jong and Stone's paper [*Model-Based Exploration in Continuous State Spaces*](https://link.springer.com/chapter/10.1007/978-3-540-73580-9_21):
  
  > This paper develops a method
  > for approximating continuous models by fitting data to a finite sample of states, leading to finite representations compatible with existing
  > model-based exploration mechanisms. Experiments with the resulting
  > family of fitted-model reinforcement learning algorithms reveals the critical importance of how the continuous model is generalized from finite
  > data. This paper demonstrates instantiations of fitted-model algorithms
  > that lead to faster learning on benchmark problems than contemporary
  > model-free RL algorithms that only apply generalization in estimating
  > action values. Finally, the paper concludes that in continuous problems,
  > the exploration-exploitation tradeoff is better construed as a balance between exploration and generalization.
  
  
  ---
  
  