Answers.Body: <p>What you suggest will work, the main restriction is needing to know <span class="math-container">$\pi$</span> fully in order to perform the conversion.</p>
  <p>If you know that you are going to be estimating <span class="math-container">$V_{\pi}$</span> from the start, and have a fixed policy, then you could use basic TD learning instead of SARSA, where the update rule is:</p>
  <p><span class="math-container">$$V(s) \leftarrow V(s) + \alpha(r + \gamma V(s') - V(s))$$</span></p>
  <p>Doing this would allow you to estimate <span class="math-container">$V_{\pi}$</span> from observations <em>without</em> knowing <span class="math-container">$\pi$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-20T06:21:50.747Z
Answers.LastActivityDate: 2022-05-20T14:13:20.887Z
Answers.LastEditDate: 2022-05-20T14:13:20.887Z
Answers.UserDisplayName: Neil Slater
Body: <p>For SARSA, I know we can estimate the action value <span class="math-container">$Q(s,a)$</span>, and the relationship between <span class="math-container">$V(s)$</span> and <span class="math-container">$Q(s,a)$</span> is <span class="math-container">$V_{\pi}(s) = \sum_{a \in \mathcal{A}} \pi(a|s)Q_{\pi}
  (s,a)$</span>.</p>
  <p>So my question is, can we simply estimate <span class="math-container">$V_{\pi}$</span> by applying the above equation to the <span class="math-container">$Q_{\pi}$</span> that we derived from SARSA? Will there be any restrictions to prevent estimating <span class="math-container">$V_{\pi}$</span> through SARSA?</p>
  
Tags: <reinforcement-learning><value-functions><bellman-equations><sarsa>
Title: Can we also estimate $V_{\pi}$ with SARSA?
UserDisplayName: Dingzhi Hu
ThreadText: # Can we also estimate $V_{\pi}$ with SARSA?
  
  (Asked by: username_0 on 2022-05-20)
  
  For SARSA, I know we can estimate the action value $Q(s,a)$, and the relationship between $V(s)$ and $Q(s,a)$ is $V\_{\pi}(s) = \sum\_{a \in \mathcal{A}} \pi(a|s)Q\_{\pi}
  (s,a)$.
  
  So my question is, can we simply estimate $V\_{\pi}$ by applying the above equation to the $Q\_{\pi}$ that we derived from SARSA? Will there be any restrictions to prevent estimating $V\_{\pi}$ through SARSA?
  
  **Tags:** reinforcement-learning, value-functions, bellman-equations, sarsa
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-05-20. Score: 3
  
  What you suggest will work, the main restriction is needing to know $\pi$ fully in order to perform the conversion.
  
  If you know that you are going to be estimating $V\_{\pi}$ from the start, and have a fixed policy, then you could use basic TD learning instead of SARSA, where the update rule is:
  
  $$V(s) \leftarrow V(s) + \alpha(r + \gamma V(s') - V(s))$$
  
  Doing this would allow you to estimate $V\_{\pi}$ from observations *without* knowing $\pi$.
  
  
  ---
  
  