Answers.Body: <blockquote>
    <p>I guess I'm having difficulty grasping the concept that the goodness of a state changes depending on how an agent got there</p>
  </blockquote>
  
  <p>It doesn't.</p>
  
  <p>The value of a state changes depending <em>on what the agent will do next</em>. That is where the dependency on the policy comes in, not in past behaviour, but expectations of future behaviour. The future behaviour depends on the state transitions and rewards presented by the environment, <em>plus</em> it depends on the distribution of actions chosen by the policy. </p>
  
  <p>More formally, the value function of a state is not just a relative and arbitrary scoring system, but equals the expected (discounted) sum of rewards, assuming the MDP follows the given dynamics, including action selection:</p>
  
  <p><span class="math-container">$$v_{\pi}(s) = \mathbb{E}_{A \sim \pi}[\sum_{k=0}^\infty \gamma^k R_{t+k+1} | S_t = s]$$</span></p>
  
  <p>Without identifying a policy, it is not possible to assess a value function. In value-based <em>control</em> methods, the policy to evaluate can be implied, somewhat self-referentially, as the policy that acts greedily (or maybe <span class="math-container">$\epsilon$</span>-greedily) according to the current estimates of the value function.</p>
  
  <blockquote>
    <p>If there can be a concrete example (perhaps on a GridWorld or on a chess board), that might make it clear why that might be the case</p>
  </blockquote>
  
  <p>A very simple deterministic MDP with a start state and two terminal states illustrates this:</p>
  
  <p><a href="https://i.sstatic.net/L0CeJ.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/L0CeJ.png" alt="Very basic MDP"></a></p>
  
  <p>Start in state <strong>B</strong>. Taking the <em>left</em> action is followed by a transition (with <span class="math-container">$p=1$</span>) to terminal state <strong>A</strong>, and a reward of <span class="math-container">$0$</span>. Taking the <em>right</em> action is followed by a transition (with <span class="math-container">$p=1$</span>) to terminal state <strong>C</strong>, and a reward of <span class="math-container">$3$</span>.</p>
  
  <p>What is the value of state <strong>B</strong>? It depends on what the policy chooses. A deterministic <em>left</em> policy <span class="math-container">$\pi_1$</span> has <span class="math-container">$v_{\pi_1}(B) = 0$</span>, a random policy <span class="math-container">$\pi_2$</span> choosing <em>left</em> and <em>right</em> with <span class="math-container">$p=0.5$</span> has <span class="math-container">$v_{\pi_2}(B) = 1.5$</span>. The optimal policy chooses action <em>right</em> always and has <span class="math-container">$v_{\pi_3}(B) = v^*(B) = 3.0$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-24T01:35:48.290Z
Answers.Text: I guess this does answer my question of why does a value function explicitly depend on a policy. There's no way of knowing how an agent will behave once it leaves the current state unless it has a policy defining its behaviour.
Answers.UserDisplayName: user3079474
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-24T01:38:32.690Z
Answers.Text: Just out of curiosity though, why do states having an absolute value (independent of policy) does not make sense? Why is the measure of how good a state is dependent upon how an agent will act once it is in the state?
Answers.UserDisplayName: user3079474
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-24T01:56:23.680Z
Answers.Text: Is it because we define how good a state is in terms of expected future rewards which implies that by definition a value function needs a policy with it?
Answers.UserDisplayName: user3079474
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-24T07:09:29.550Z
Answers.Text: @user3079474: Yes, the value function in RL is defined that way, so that's precisely what it means. You could invent some function that allocated an absolute "value" to each state, but that would be a different function. Possibly the closest in RL, would be if immediate reward is allocated according only to state (which is quite common situation - e.g. think of winning any board game, this is often purely a function of state) - that would not be the value function though, but how the reward function was set up.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-24T07:11:15.187Z
Answers.Text: Some heuristic systems may also allocate values to states, and use them to guide search for game and problem-solving tree searches (like A*). There is a little bit of overlap here, as you can use a learned RL value function in place of a heuristic - that is essentially what AlphaGo and its variants do.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-23T15:32:50.000Z
Answers.LastActivityDate: 2018-09-23T16:20:12.523Z
Answers.LastEditDate: 2018-09-23T16:20:12.523Z
Answers.UserDisplayName: Neil Slater
Body: <p>From what I understand, the value function estimates how 'good' it is for an agent to be in a state, and a policy is a mapping of actions to state.</p>
  <p>If I have understood these concepts correctly, why does the value of a state change with the policy with which an agent gets there?</p>
  <p>I guess I'm having difficulty grasping the concept that the goodness of a state changes depending on how an agent got there (different policies may have different ways, and hence different values, for getting to a particular state).</p>
  <p>If there can be a concrete example (perhaps on a grid world or on a chessboard), that might make it clear why that might be the case.</p>
  
Tags: <reinforcement-learning><policies><value-functions>
Title: Why does the value of state change depending on the policy used to get to that state?
UserDisplayName: user3079474
ThreadText: # Why does the value of state change depending on the policy used to get to that state?
  
  (Asked by: username_0 on 2018-09-23)
  
  From what I understand, the value function estimates how 'good' it is for an agent to be in a state, and a policy is a mapping of actions to state.
  
  If I have understood these concepts correctly, why does the value of a state change with the policy with which an agent gets there?
  
  I guess I'm having difficulty grasping the concept that the goodness of a state changes depending on how an agent got there (different policies may have different ways, and hence different values, for getting to a particular state).
  
  If there can be a concrete example (perhaps on a grid world or on a chessboard), that might make it clear why that might be the case.
  
  **Tags:** reinforcement-learning, policies, value-functions
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-09-23. Score: 2 | Accepted answer
  
  > I guess I'm having difficulty grasping the concept that the goodness of a state changes depending on how an agent got there
  
  It doesn't.
  
  The value of a state changes depending *on what the agent will do next*. That is where the dependency on the policy comes in, not in past behaviour, but expectations of future behaviour. The future behaviour depends on the state transitions and rewards presented by the environment, *plus* it depends on the distribution of actions chosen by the policy.
  
  More formally, the value function of a state is not just a relative and arbitrary scoring system, but equals the expected (discounted) sum of rewards, assuming the MDP follows the given dynamics, including action selection:
  
  $$v\_{\pi}(s) = \mathbb{E}\_{A \sim \pi}[\sum\_{k=0}^\infty \gamma^k R\_{t+k+1} | S\_t = s]$$
  
  Without identifying a policy, it is not possible to assess a value function. In value-based *control* methods, the policy to evaluate can be implied, somewhat self-referentially, as the policy that acts greedily (or maybe $\epsilon$-greedily) according to the current estimates of the value function.
  
  > If there can be a concrete example (perhaps on a GridWorld or on a chess board), that might make it clear why that might be the case
  
  A very simple deterministic MDP with a start state and two terminal states illustrates this:
  
  [![Very basic MDP](https://i.sstatic.net/L0CeJ.png)](https://i.sstatic.net/L0CeJ.png)
  
  Start in state **B**. Taking the *left* action is followed by a transition (with $p=1$) to terminal state **A**, and a reward of $0$. Taking the *right* action is followed by a transition (with $p=1$) to terminal state **C**, and a reward of $3$.
  
  What is the value of state **B**? It depends on what the policy chooses. A deterministic *left* policy $\pi\_1$ has $v\_{\pi\_1}(B) = 0$, a random policy $\pi\_2$ choosing *left* and *right* with $p=0.5$ has $v\_{\pi\_2}(B) = 1.5$. The optimal policy chooses action *right* always and has $v\_{\pi\_3}(B) = v^\*(B) = 3.0$
  
  > **Comments:**
  > **username_0**: I guess this does answer my question of why does a value function explicitly depend on a policy. There's no way of knowing how an agent will behave once it leaves the current state unless it has a policy defining its behaviour.  
  > **username_0**: Just out of curiosity though, why do states having an absolute value (independent of policy) does not make sense? Why is the measure of how good a state is dependent upon how an agent will act once it is in the state?  
  > **username_0**: Is it because we define how good a state is in terms of expected future rewards which implies that by definition a value function needs a policy with it?  
  > **username_1**: @user3079474: Yes, the value function in RL is defined that way, so that's precisely what it means. You could invent some function that allocated an absolute "value" to each state, but that would be a different function. Possibly the closest in RL, would be if immediate reward is allocated according only to state (which is quite common situation - e.g. think of winning any board game, this is often purely a function of state) - that would not be the value function though, but how the reward function was set up.  
  > **username_1**: Some heuristic systems may also allocate values to states, and use them to guide search for game and problem-solving tree searches (like A*). There is a little bit of overlap here, as you can use a learned RL value function in place of a heuristic - that is essentially what AlphaGo and its variants do.  
  
  ---
  
  