Answers.Body: <p>As illustrated under your above flowchart figure 2.4.2, base heuristic/policy is just a simple heuristic that is used to help make a quick decision about what action to take in a given state <span class="math-container">$x_k$</span> via generating a heuristic Q-factor for every state-control pair (<span class="math-container">$x_k, u_k$</span>) while satisfying the control system equations shown on your first screenshot. Then the on-line rollout algorithm selects the control with minimal Q-factor which is a specific <em>heuristic search</em> algorithm using the base heuristic (policy) to further simulate the behavior of the control system over a finite time horizon under the (suboptimal) rollout policy.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-16T04:51:14.530Z
Answers.LastActivityDate: 2023-03-16T04:51:14.530Z
Answers.UserDisplayName: cinch
Body: <p>In the following pages (p.84-85) of RL and Optimal Control book by  D.Bertsekas,
  he is talking about  base heuristic&quot; and a &quot;rollout algorithm&quot; based on this base heuristic.
  However, I am very confused because based on Figure 2.4.2, it seems
  that rollout algorithm and base heuristics are the same things.</p>
  <p><a href="https://i.sstatic.net/3SN8A.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/3SN8A.png" alt="enter image description here" /></a></p>
  <p><a href="https://i.sstatic.net/XDCin.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/XDCin.png" alt="enter image description here" /></a></p>
  
Tags: <reinforcement-learning>
Title: What is the difference between "base heuristic" and a "rollout algorithm" based on this base heuristic?
UserDisplayName: DSPinfinity
ThreadText: # What is the difference between "base heuristic" and a "rollout algorithm" based on this base heuristic?
  
  (Asked by: username_0 on 2023-03-15)
  
  In the following pages (p.84-85) of RL and Optimal Control book by D.Bertsekas,
  he is talking about base heuristic" and a "rollout algorithm" based on this base heuristic.
  However, I am very confused because based on Figure 2.4.2, it seems
  that rollout algorithm and base heuristics are the same things.
  
  [![enter image description here](https://i.sstatic.net/3SN8A.png)](https://i.sstatic.net/3SN8A.png)
  
  [![enter image description here](https://i.sstatic.net/XDCin.png)](https://i.sstatic.net/XDCin.png)
  
  **Tags:** reinforcement-learning
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-03-16. Score: 1 | Accepted answer
  
  As illustrated under your above flowchart figure 2.4.2, base heuristic/policy is just a simple heuristic that is used to help make a quick decision about what action to take in a given state $x\_k$ via generating a heuristic Q-factor for every state-control pair ($x\_k, u\_k$) while satisfying the control system equations shown on your first screenshot. Then the on-line rollout algorithm selects the control with minimal Q-factor which is a specific *heuristic search* algorithm using the base heuristic (policy) to further simulate the behavior of the control system over a finite time horizon under the (suboptimal) rollout policy.
  
  
  ---
  
  