Answers.Body: <p>We are ultimately interested in getting an optimal policy, that is the optimal sequence of actions to reach the final goal. State values on its own don't provide that, they tell you expected return from specific state onward but they don't tell you which action to take. In order to derive an optimal action in a specific state you would have to simulate all possible actions one step ahead and then pick the action that leads you to the state with highest state value. That is often inconvenient or impossible. State action values connect the expected return with actions, not states, so you don't need to simulate all actions one step ahead and see where you end up, you only need to pick an action that has the highest value because you know that is the best action to take in that state.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-28T14:59:52.810Z
Answers.Text: what did you mean by `simulate all possible actions one step ahead`?
Answers.UserDisplayName: gfdsal
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-14T14:20:48.503Z
Answers.Text: It may be also worth emphasizing that, if you're using a table, the state-action value function will be larger than the state value function, so it will require more memory to be stored.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-14T13:14:46.893Z
Answers.LastActivityDate: 2020-02-14T13:14:46.893Z
Answers.UserDisplayName: Brale
Body: <p>In reinforcement learning, the state-action value function seems to be used more than the state value function. Why is it so?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-02-14T07:31:55.903Z
Comments.Text: What do you mean by "more"? Btw, if I understand your question correctly, it is because in a non-deterministic setting, you won't know which action to select simply from the state values.
Comments.UserDisplayName: Hai Nguyen
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-02-19T16:37:41.770Z
Comments.Text: @HaiNguyen thanks. What happens in the case of deterministic setting
Comments.UserDisplayName: Bhuwan Bhatt
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-02-20T11:51:51.957Z
Comments.Text: If you know what state you will end up after taking an action (for example if you walk in a maze you always know your next position after you take a step), you can use the value function of possible next states and pick the action that leads to the highest value.
Comments.UserDisplayName: Hai Nguyen
Tags: <reinforcement-learning><comparison><value-functions>
Title: Why is the state-action value function used more than the state value function?
UserDisplayName: Bhuwan Bhatt
ThreadText: # Why is the state-action value function used more than the state value function?
  
  (Asked by: username_0 on 2020-02-14)
  
  In reinforcement learning, the state-action value function seems to be used more than the state value function. Why is it so?
  
  **Tags:** reinforcement-learning, comparison, value-functions
  
  > **Comments:**
  > **username_1**: What do you mean by "more"? Btw, if I understand your question correctly, it is because in a non-deterministic setting, you won't know which action to select simply from the state values.  
  > **username_0**: @username_1 thanks. What happens in the case of deterministic setting  
  > **username_1**: If you know what state you will end up after taking an action (for example if you walk in a maze you always know your next position after you take a step), you can use the value function of possible next states and pick the action that leads to the highest value.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-02-14. Score: 7 | Accepted answer
  
  We are ultimately interested in getting an optimal policy, that is the optimal sequence of actions to reach the final goal. State values on its own don't provide that, they tell you expected return from specific state onward but they don't tell you which action to take. In order to derive an optimal action in a specific state you would have to simulate all possible actions one step ahead and then pick the action that leads you to the state with highest state value. That is often inconvenient or impossible. State action values connect the expected return with actions, not states, so you don't need to simulate all actions one step ahead and see where you end up, you only need to pick an action that has the highest value because you know that is the best action to take in that state.
  
  > **Comments:**
  > **username_4**: It may be also worth emphasizing that, if you're using a table, the state-action value function will be larger than the state value function, so it will require more memory to be stored.  
  > **username_3**: what did you mean by `simulate all possible actions one step ahead`?  
  
  ---
  
  