<p>Sutton and Barto, in their book (Reinforcement Learning 2nd Edition) begin the discussion of policy improvement by comparing the action value <span class="math-container">$q_\pi(s, \pi'(s))$</span> to the state value <span class="math-container">$v_\pi(s)$</span>.</p>
<p>What is the intuition behind this comparison?</p>
<p>It seems more natural to me to compare <span class="math-container">$q_\pi(s, \pi'(s))$</span> and <span class="math-container">$q_\pi(s, \pi(s))$</span>. I understand that for deterministic policies <span class="math-container">$q_\pi(s, \pi(s))$</span> is the same as <span class="math-container">$v_\pi(s)$</span> so mathematically it makes no difference but perhaps conceptually it does?</p>

<reinforcement-learning><value-functions><policy-improvement-theorem>
What is the intuition behind comparing action values to state values in the policy improvement theorem?
bonzo_pippinpaddle
# What is the intuition behind comparing action values to state values in the policy improvement theorem?

(Asked by: username_0 on 2021-08-22)

Sutton and Barto, in their book (Reinforcement Learning 2nd Edition) begin the discussion of policy improvement by comparing the action value $q\_\pi(s, \pi'(s))$ to the state value $v\_\pi(s)$.

What is the intuition behind this comparison?

It seems more natural to me to compare $q\_\pi(s, \pi'(s))$ and $q\_\pi(s, \pi(s))$. I understand that for deterministic policies $q\_\pi(s, \pi(s))$ is the same as $v\_\pi(s)$ so mathematically it makes no difference but perhaps conceptually it does?

**Tags:** reinforcement-learning, value-functions, policy-improvement-theorem


---

## 0 answers

