Body: <p>Sutton and Barto, in their book (Reinforcement Learning 2nd Edition) begin the discussion of policy improvement by comparing the action value <span class="math-container">$q_\pi(s, \pi'(s))$</span> to the state value <span class="math-container">$v_\pi(s)$</span>.</p>
  <p>What is the intuition behind this comparison?</p>
  <p>It seems more natural to me to compare <span class="math-container">$q_\pi(s, \pi'(s))$</span> and <span class="math-container">$q_\pi(s, \pi(s))$</span>. I understand that for deterministic policies <span class="math-container">$q_\pi(s, \pi(s))$</span> is the same as <span class="math-container">$v_\pi(s)$</span> so mathematically it makes no difference but perhaps conceptually it does?</p>
  
Tags: <reinforcement-learning><value-functions><policy-improvement-theorem>
Title: What is the intuition behind comparing action values to state values in the policy improvement theorem?
UserDisplayName: bonzo_pippinpaddle
ThreadText: # What is the intuition behind comparing action values to state values in the policy improvement theorem?
  
  (Asked by: username_0 on 2021-08-22)
  
  Sutton and Barto, in their book (Reinforcement Learning 2nd Edition) begin the discussion of policy improvement by comparing the action value $q\_\pi(s, \pi'(s))$ to the state value $v\_\pi(s)$.
  
  What is the intuition behind this comparison?
  
  It seems more natural to me to compare $q\_\pi(s, \pi'(s))$ and $q\_\pi(s, \pi(s))$. I understand that for deterministic policies $q\_\pi(s, \pi(s))$ is the same as $v\_\pi(s)$ so mathematically it makes no difference but perhaps conceptually it does?
  
  **Tags:** reinforcement-learning, value-functions, policy-improvement-theorem
  
  ---
  
  ## 0 answers
  
  