Answers.Body: <p><span class="math-container">$P$</span> tells you how &quot;off&quot; the evaluation for <span class="math-container">$Q(S, A)$</span> is. If the difference between <span class="math-container">$Q(S,A)$</span> (current best guess) and <span class="math-container">$R + \gamma\max_a Q(S', a)$</span> (update-value for <span class="math-container">$Q(S, A)$</span> since you received <span class="math-container">$R$</span> and were moved to <span class="math-container">$S'$</span> when you did <span class="math-container">$A$</span>) is large, then it is an indication that this is a state-action-pair you should consider to learn more about, and <span class="math-container">$(S, A)$</span> will be close to the front of the queue. On the other hand, if <span class="math-container">$P$</span> is small, there is &quot;nothing new to learn&quot;, and no need to prioritize this state-action-pair.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-23T20:19:07.567Z
Answers.LastActivityDate: 2022-03-23T20:19:07.567Z
Answers.UserDisplayName: Balle
Body: <p>Following is the &quot;Prioritized Sweeping&quot; algorithm in Sutton-Barto's RL book (page 170).  What is the meaning of step (e) in the prioritized sweeping algorithm? More importantly, why is P calculated like that?</p>
  <p><a href="https://i.sstatic.net/PKFgd.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/PKFgd.png" alt="enter image description here" /></a></p>
  
Tags: <reinforcement-learning><prioritized-sweeping>
Title: What is the meaning of step (e) in the prioritized sweeping algorithm? Why is P calculated like that?
UserDisplayName: DSPinfinity
ThreadText: # What is the meaning of step (e) in the prioritized sweeping algorithm? Why is P calculated like that?
  
  (Asked by: username_0 on 2022-03-22)
  
  Following is the "Prioritized Sweeping" algorithm in Sutton-Barto's RL book (page 170). What is the meaning of step (e) in the prioritized sweeping algorithm? More importantly, why is P calculated like that?
  
  [![enter image description here](https://i.sstatic.net/PKFgd.png)](https://i.sstatic.net/PKFgd.png)
  
  **Tags:** reinforcement-learning, prioritized-sweeping
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-03-23. Score: 2 | Accepted answer
  
  $P$ tells you how "off" the evaluation for $Q(S, A)$ is. If the difference between $Q(S,A)$ (current best guess) and $R + \gamma\max\_a Q(S', a)$ (update-value for $Q(S, A)$ since you received $R$ and were moved to $S'$ when you did $A$) is large, then it is an indication that this is a state-action-pair you should consider to learn more about, and $(S, A)$ will be close to the front of the queue. On the other hand, if $P$ is small, there is "nothing new to learn", and no need to prioritize this state-action-pair.
  
  
  ---
  
  