Answers.Body: <p>You can obtain the optimal policy from the optimal state value function if you also have the state transition and reward model for the environment <span class="math-container">$p(s',r|s,a)$</span> - the probability of receiving reward <span class="math-container">$r$</span> and arriving in state <span class="math-container">$s'$</span> when starting in state <span class="math-container">$s$</span> and taking action <span class="math-container">$a$</span>.</p>
  <p>This looks like:</p>
  <p><span class="math-container">$$\pi^*(s) = \text{argmax}_a [\sum_{s',r} p(s',r|s,a)(r + \gamma v^*(s'))]$$</span></p>
  <p>There are variations of this function, depending on how you represent knowledge of the environment. For instance, you don't actually need the full distribution model for reward, an expected reward function and separate distribution model for state transition rules would also work.</p>
  <p>Without at least an approximate model of the environment, you cannot derive a policy from state values. If all you have is state values, then to pick an optimal action, you absolutely need the ability to look ahead a time step at what the next state might be for each action choice.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-31T00:08:00.700Z
Answers.Text: How would this extend to two player (and more than three or more player) games, say just adversarial zero-sum ones, assuming reward and transition probability is known? Specifically, how does the argmax work, since the next state corresponds to the next player? Is the state input to the value function ordered to be from the perspective of the current player?
Answers.UserDisplayName: thinkski
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-07T13:09:38.117Z
Answers.LastActivityDate: 2020-11-07T13:09:38.117Z
Answers.UserDisplayName: Neil Slater
Body: <p>One can easily retrieve the optimal policy from the action value function but how about obtaining it from the state value function?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-07T14:31:20.733Z
Comments.Text: This seems to be a duplicate of [this](https://ai.stackexchange.com/q/10584/2444) and [this](https://ai.stackexchange.com/q/22907/2444).
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><dynamic-programming>
Title: Is it possible to retrieve the optimal policy from the state value function?
UserDisplayName: Mika
ThreadText: # Is it possible to retrieve the optimal policy from the state value function?
  
  (Asked by: username_0 on 2020-11-07)
  
  One can easily retrieve the optimal policy from the action value function but how about obtaining it from the state value function?
  
  **Tags:** reinforcement-learning, dynamic-programming
  
  > **Comments:**
  > **username_1**: This seems to be a duplicate of [this](https://ai.stackexchange.com/q/10584/2444) and [this](https://ai.stackexchange.com/q/22907/2444).  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-11-07. Score: 2 | Accepted answer
  
  You can obtain the optimal policy from the optimal state value function if you also have the state transition and reward model for the environment $p(s',r|s,a)$ - the probability of receiving reward $r$ and arriving in state $s'$ when starting in state $s$ and taking action $a$.
  
  This looks like:
  
  $$\pi^\*(s) = \text{argmax}\_a [\sum\_{s',r} p(s',r|s,a)(r + \gamma v^\*(s'))]$$
  
  There are variations of this function, depending on how you represent knowledge of the environment. For instance, you don't actually need the full distribution model for reward, an expected reward function and separate distribution model for state transition rules would also work.
  
  Without at least an approximate model of the environment, you cannot derive a policy from state values. If all you have is state values, then to pick an optimal action, you absolutely need the ability to look ahead a time step at what the next state might be for each action choice.
  
  > **Comments:**
  > **username_3**: How would this extend to two player (and more than three or more player) games, say just adversarial zero-sum ones, assuming reward and transition probability is known? Specifically, how does the argmax work, since the next state corresponds to the next player? Is the state input to the value function ordered to be from the perspective of the current player?  
  
  ---
  
  