Answers.Body: <blockquote>
    <p>How does policy evaluation work for continuous state space model-free approaches? ... Let's say you use a DQN to find another policy, how does model-free policy evaluation work then? </p>
  </blockquote>
  
  <p>Policy evaluation is the process of determining state-value <span class="math-container">$v_{\pi}(s)$</span> or action-value <span class="math-container">$q_{\pi}(s, a)$</span> functions for the current policy. In the context of continuous state and action spaces without a model of the environment, <strong>policy evaluation must incorporate the agent's past experience instead of the model dynamics</strong> and will generally use a function approximator such as a neural network to estimate the action-values. Many popular approaches apply online updates to the function approximator; e.g., DQN combines Temporal-Difference targets and gradient descent to change the weights of the neural network and the resultant action-value estimates. Since </p>
  
  <ul>
  <li>we gradually change the weights of the neural network at each gradient descent step,</li>
  <li>the estimated action-values are solely dependent on the weights of the neural network,</li>
  <li>the current policy is solely dependent on the estimated action-values (e.g. DQN takes the action with largest action-value),</li>
  </ul>
  
  <p>then <strong>policy evaluation</strong> (updating the estimated action-value function to better match the true action-value function under the current policy) <strong>and policy improvement</strong> (greedily changing the current policy based on the new estimated action-value function) <strong>occur simultaneously at each gradient descent step</strong>. In DQN, a gradient descent step occurs at each time step.</p>
  
  <blockquote>
    <p>I am thinking of Monte Carlo simulation, but that would require many many episodes. </p>
  </blockquote>
  
  <p>After changing the action-value function, we may get a new policy. We assume that the <strong>action-value function of the old policy is similar to that of the new policy</strong> (although it is not guaranteed) because the change in weights of the neural network was <em>small</em>. Therefore, we use the estimated action-value function of the old policy as an initial estimate of the action-value function of the new policy. Specifically, <strong>we use the same neural network with the same weights as the initial approximation</strong>. This is computationally convenient, as it <strong>prevents the need to start the next policy evaluation update from scratch</strong> (e.g. with a Monte Carlo simulation over a painfully large number of episodes). </p>
  
  <blockquote>
    <p>Theoretically, a model-based approach for the discrete state and action space can be computed via dynamic programming and solving the Bellman equation.</p>
  </blockquote>
  
  <p>This same technique of using the estimated action-values of the old policy as the initial estimates for the action-values of the new policy is employed by some Dynamic Programming methods such as <em>value iteration</em>, albeit with known dynamics. <em>Generalized Policy Iteration</em> (GPI) is the notion of letting policy evaluation and policy iteration interact on whatever granularity deemed necessary for the problem at hand. A consequence of adopting the GPI paradigm is the choice to halt policy evaluation before convergence of the action-value function. Many deep reinforcement learning algorithms take this to an extreme and perform policy evaluation and policy improvement simultaneously during a single gradient descent step. For reference, Chapter 4 of <a href="http://incompleteideas.net/book/the-book-2nd.html" rel="nofollow noreferrer">Sutton and Barto</a> provides a brief summary of these ideas.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-11T07:06:38.647Z
Answers.Text: I think OP has a fixed policy to evaluate (indirectly asked here by calling out evaluation only, and implied from other questions by same OP), so the parts of the answer referring to deriving policy from Q or GPI need to be adjusted - either separated out to make it clearer which part is which, or could even be removed entirely.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-11T14:56:34.467Z
Answers.Text: Hi @NeilSlater, and thank you very much for the good advice. It looks like I answered a different question than OP meant to ask - OP was wondering about agent performance (like you pointed out), while I provided an answer about policy eval in the context of GPI for continuous, model-free DRL. Since my answer was accepted, how should I proceed? Should I edit my response to provide more context for my particular answer? I'm very new here and just need some guidance.
Answers.UserDisplayName: DeepQZero
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-11T15:13:27.110Z
Answers.Text: If it has been accepted, probably better to leave as-is. The OP is clearly able to extract the information that they want from the answer, and there's a risk you remove something important to them if you make a major edit as I suggested.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-11T02:18:21.050Z
Answers.LastActivityDate: 2020-06-11T02:18:21.050Z
Answers.UserDisplayName: DeepQZero
Body: <p>How does policy evaluation work for continuous state space model-free approaches?</p>
  
  <p>Theoretically, a model-based approach for the discrete state and action space can be computed via dynamic programming and solving the Bellman equation. </p>
  
  <p>Let's say you use a DQN to find another policy, how does model-free policy evaluation work then? I am thinking of Monte Carlo simulation, but that would require many many episodes. </p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-11T14:18:53.587Z
Comments.Text: Hi calveeen. Great question! To clarify - by policy evaluation, do you mean finding the true action-value function $q_{\pi}(s, a)$ of the current policy $\pi$ during training? Do you mean gauging the performance of the agent after constructing a final, trained policy? Or something else?
Comments.UserDisplayName: DeepQZero
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-11T14:30:49.067Z
Comments.Text: Hi @DeepQZero, thank you for replying. This question seems rather silly as policy evaluation for DQN can be simply done by observing average reward returns. In my case I was not able to work with running the policy in real life. Hence, the way I found to estimate the value of a learnt policy would be through off policy value evaluation, which is the importance sampling and it's variant.
Comments.UserDisplayName: calveeen
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-11T14:45:51.200Z
Comments.Text: Ok I see. For future reference, policy evaluation has a very specific definition in the context of reinforcement learning. It is the process of determining the true state-value $v_{\pi}(s)$ function or action-value function $q_{\pi}(s,a)$ under the current policy $\pi$ *for every state and action*. It seems that you are interested in gauging the agent's performance in the environment - that is equivalent to determining the state-value of *only the start state(s)* and separate from the policy evaluation problem. Both are good questions though.
Comments.UserDisplayName: DeepQZero
Tags: <reinforcement-learning><deep-rl><monte-carlo-methods><model-free-methods><policy-evaluation>
Title: How does policy evaluation work for continuous state space model-free approaches?
UserDisplayName: calveeen
ThreadText: # How does policy evaluation work for continuous state space model-free approaches?
  
  (Asked by: username_0 on 2020-02-19)
  
  How does policy evaluation work for continuous state space model-free approaches?
  
  Theoretically, a model-based approach for the discrete state and action space can be computed via dynamic programming and solving the Bellman equation.
  
  Let's say you use a DQN to find another policy, how does model-free policy evaluation work then? I am thinking of Monte Carlo simulation, but that would require many many episodes.
  
  **Tags:** reinforcement-learning, deep-rl, monte-carlo-methods, model-free-methods, policy-evaluation
  > **Comments:**
  > **username_1**: Hi calveeen. Great question! To clarify - by policy evaluation, do you mean finding the true action-value function $q_{\pi}(s, a)$ of the current policy $\pi$ during training? Do you mean gauging the performance of the agent after constructing a final, trained policy? Or something else?  
  > **username_0**: Hi @DeepQZero, thank you for replying. This question seems rather silly as policy evaluation for DQN can be simply done by observing average reward returns. In my case I was not able to work with running the policy in real life. Hence, the way I found to estimate the value of a learnt policy would be through off policy value evaluation, which is the importance sampling and it's variant.  
  > **username_1**: Ok I see. For future reference, policy evaluation has a very specific definition in the context of reinforcement learning. It is the process of determining the true state-value $v_{\pi}(s)$ function or action-value function $q_{\pi}(s,a)$ under the current policy $\pi$ *for every state and action*. It seems that you are interested in gauging the agent's performance in the environment - that is equivalent to determining the state-value of *only the start state(s)* and separate from the policy evaluation problem. Both are good questions though.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-06-11. Score: 1 | Accepted answer
  
  > How does policy evaluation work for continuous state space model-free approaches? ... Let's say you use a DQN to find another policy, how does model-free policy evaluation work then?
  
  Policy evaluation is the process of determining state-value $v\_{\pi}(s)$ or action-value $q\_{\pi}(s, a)$ functions for the current policy. In the context of continuous state and action spaces without a model of the environment, **policy evaluation must incorporate the agent's past experience instead of the model dynamics** and will generally use a function approximator such as a neural network to estimate the action-values. Many popular approaches apply online updates to the function approximator; e.g., DQN combines Temporal-Difference targets and gradient descent to change the weights of the neural network and the resultant action-value estimates. Since
  
  * we gradually change the weights of the neural network at each gradient descent step,
  * the estimated action-values are solely dependent on the weights of the neural network,
  * the current policy is solely dependent on the estimated action-values (e.g. DQN takes the action with largest action-value),
  
  then **policy evaluation** (updating the estimated action-value function to better match the true action-value function under the current policy) **and policy improvement** (greedily changing the current policy based on the new estimated action-value function) **occur simultaneously at each gradient descent step**. In DQN, a gradient descent step occurs at each time step.
  
  > I am thinking of Monte Carlo simulation, but that would require many many episodes.
  
  After changing the action-value function, we may get a new policy. We assume that the **action-value function of the old policy is similar to that of the new policy** (although it is not guaranteed) because the change in weights of the neural network was *small*. Therefore, we use the estimated action-value function of the old policy as an initial estimate of the action-value function of the new policy. Specifically, **we use the same neural network with the same weights as the initial approximation**. This is computationally convenient, as it **prevents the need to start the next policy evaluation update from scratch** (e.g. with a Monte Carlo simulation over a painfully large number of episodes).
  
  > Theoretically, a model-based approach for the discrete state and action space can be computed via dynamic programming and solving the Bellman equation.
  
  This same technique of using the estimated action-values of the old policy as the initial estimates for the action-values of the new policy is employed by some Dynamic Programming methods such as *value iteration*, albeit with known dynamics. *Generalized Policy Iteration* (GPI) is the notion of letting policy evaluation and policy iteration interact on whatever granularity deemed necessary for the problem at hand. A consequence of adopting the GPI paradigm is the choice to halt policy evaluation before convergence of the action-value function. Many deep reinforcement learning algorithms take this to an extreme and perform policy evaluation and policy improvement simultaneously during a single gradient descent step. For reference, Chapter 4 of [Sutton and Barto](http://incompleteideas.net/book/the-book-2nd.html) provides a brief summary of these ideas.
  
  > **Comments:**
  > **username_2**: I think OP has a fixed policy to evaluate (indirectly asked here by calling out evaluation only, and implied from other questions by same OP), so the parts of the answer referring to deriving policy from Q or GPI need to be adjusted - either separated out to make it clearer which part is which, or could even be removed entirely.  
  > **username_1**: Hi @NeilSlater, and thank you very much for the good advice. It looks like I answered a different question than OP meant to ask - OP was wondering about agent performance (like you pointed out), while I provided an answer about policy eval in the context of GPI for continuous, model-free DRL. Since my answer was accepted, how should I proceed? Should I edit my response to provide more context for my particular answer? I'm very new here and just need some guidance.  
  > **username_2**: If it has been accepted, probably better to leave as-is. The OP is clearly able to extract the information that they want from the answer, and there's a risk you remove something important to them if you make a major edit as I suggested.  
  
  ---
  
  