Answers.Body: <p>For off-policy learning you must have two policies - a behaviour policy and a target policy. If the two policies are the same, then you end up with SARSA, not Q learning.</p>
  <p>You cannot use Q learning directly for evaluating a fixed target policy, because it directly learns optimal value function as the target policy, regardless of the behaviour policy. Instead you must use another variant of off-policy learning that can evaluate an arbitrary target policy.</p>
  <p>Your suggested algorithm is:</p>
  <blockquote>
  <ol>
  <li>Have the policy to be evaluated as the behaviour policy.</li>
  <li>Update the Q value conventionally (i.e. updating <span class="math-container">$Q(s,a)$</span> using the action <span class="math-container">$a'$</span> giving highest <span class="math-container">$Q(s',a')$</span> value)</li>
  <li>The final <span class="math-container">$Q(s,a)$</span> values will reflect the values for the policy being evaluated.</li>
  </ol>
  </blockquote>
  <p>This will not work for evaluating the behaviour policy. If the behaviour policy was stochastic and covered all possible state/action choices, then it will still be Q learning and converge on the optimal value function - maybe very slowly if the behaviour policy did not get to important states very often.</p>
  <p>The &quot;trick&quot; to off-policy is that the environment interaction part uses the behaviour policy to collect data, and the update step uses the target policy to calculate estimated returns. In general for off-policy updates, there can be corrections required to re-weight the estimated returns. However, one nice thing about single-step TD methods is that there are no such additional corrections needed.</p>
  <p>So this gives a way to do off-policy TD learning, using an approach called <a href="https://www.geeksforgeeks.org/expected-sarsa-in-reinforcement-learning/" rel="nofollow noreferrer">Expected SARSA</a>. To use Expected SARSA, you will need to know the distribution of action choices i.e. know <span class="math-container">$\pi(a|s)$</span> for the target policy.</p>
  <p>This is the variant of your description that will work to evaluate your target policy <span class="math-container">$\pi(a|s)$</span>:</p>
  <ol>
  <li>Have any stochastic policy that &quot;covers&quot; the target policy as the behaviour policy.</li>
  <li>Update the Q value using Expected SARSA <span class="math-container">$Q(s,a) = Q(s,a) + \alpha(r + \gamma [\sum_{a'} \pi(a'|s')Q(s',a')]  - Q(s,a))$</span></li>
  <li>The final <span class="math-container">$Q(s,a)$</span> values will reflect the values for the policy being evaluated.</li>
  </ol>
  <p>Worth noting that Expected SARSA with a target policy of <span class="math-container">$\pi(s) = \text{argmax}_a Q(s,a)$</span> is exactly Q learning. Expected SARSA is a strict generalisation of Q learning that allows for learning the value function of <em>any</em> target policy. You may not see it used as much as Q learning, because the goal of learning an optimal value function is more common in practice.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-15T23:24:22.330Z
Answers.Text: Although I don't know the details yet, I think you may also want to take a look at [**off-policy (policy) evaluation**](https://papers.nips.cc/paper/2019/file/4ffb0d2ba92f664c2281970110a2e071-Paper.pdf) (if you are not already aware of these concepts). Can you explain a little why expected SARSA would make this an off-policy evaluation method?
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-16T07:06:08.097Z
Answers.Text: @Neil, Thanks for the detailed answer. I'm unclear why ESARSA would work though. I feel that it would only work if the target policy being learnt is the same as the behaviour policy. i.e. we have an on-policy setting.
Answers.UserDisplayName: Dhruv Mullick
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-16T07:46:27.010Z
Answers.Text: @DhruvMullick: On-policy vs off-policy is a separate issue from evaluation vs control. The key for off-policy evaluation is that you set the target policy to be the policy that you want to evaluate. Nothing more is required. Can you explain more why you feel evaluation needs to be on-policy? Then I may be able to address that.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-16T17:45:47.960Z
Answers.Text: @NeilSlater, thank you, I understand now. I was earlier under the wrong impression that for Policy evaluation, I would have to set the behaviour policy as the policy to be evaluated.
Answers.UserDisplayName: Dhruv Mullick
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-16T00:57:30.747Z
Answers.Text: @NeilSlater Thanks for the answer, this and many of your answers have been very helpful. Could you please help with a follow up question here? I am wondering, since we are evaluating value of the fixed given policy here, is it best to use a fully random behavior policy so that value is trained over more diverse states than we would see with the more commonly used eps-greedy behavior policy?
Answers.UserDisplayName: ModCon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-16T07:04:49.563Z
Answers.Text: @ModCon Generally no, for evaluating a fixed policy where you are in control of the behaviour policy, then you should just use an on-policy method. That's because off-policy methods add variance to measurement. However you are not always in control of behaviour policy - e.g. you might have historical observations where it isn't even clear what the behaviour policy was.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-16T17:03:48.233Z
Answers.Text: @NeilSlater Thanks so much for replying! If I understand correctly, even if I follow fixed policy to choose every behaviour action, given that state is sampled randomly at t=0, we should get experiences for different initial states, proving enough for learning state value function. In my case, I am learning a state action value function, can I keep the action sampled t=0 randomly chosen and only the next actions as per fixed policy (instead of choosing even t=0 action as per fixed policy) so that there’s experiences starting from all possible actions?
Answers.UserDisplayName: ModCon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-17T07:08:41.820Z
Answers.Text: @ModCon I'm not clear why you would want to do that. I don't think the comments section here will be enough to cover your question. Please consider asking a new question on the site
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-15T19:43:24.120Z
Answers.LastActivityDate: 2020-11-15T23:31:42.603Z
Answers.LastEditDate: 2020-11-15T23:31:42.603Z
Answers.UserDisplayName: Neil Slater
Body: <p>For policy evaluation purposes, can we use the Q-learning algorithm even though, technically, it is meant for control?</p>
  <p>Maybe like this:</p>
  <ol>
  <li>Have the policy to be evaluated as the behaviour policy.</li>
  <li>Update the Q value conventionally (i.e. updating <span class="math-container">$Q(s,a)$</span> using the action <span class="math-container">$a'$</span> giving highest <span class="math-container">$Q(s',a')$</span> value)</li>
  <li>The final <span class="math-container">$Q(s,a)$</span> values will reflect the values for the policy being evaluated.</li>
  </ol>
  <p>Am I missing something here, given that I have not seen Q-learning being used anywhere for evaluation purposes?</p>
  
Tags: <reinforcement-learning><q-learning><policy-evaluation>
Title: Can we use Q-learning update for policy evaluation (not control)?
UserDisplayName: Dhruv Mullick
ThreadText: # Can we use Q-learning update for policy evaluation (not control)?
  
  (Asked by: username_0 on 2020-11-15)
  
  For policy evaluation purposes, can we use the Q-learning algorithm even though, technically, it is meant for control?
  
  Maybe like this:
  
  1. Have the policy to be evaluated as the behaviour policy.
  2. Update the Q value conventionally (i.e. updating $Q(s,a)$ using the action $a'$ giving highest $Q(s',a')$ value)
  3. The final $Q(s,a)$ values will reflect the values for the policy being evaluated.
  
  Am I missing something here, given that I have not seen Q-learning being used anywhere for evaluation purposes?
  
  **Tags:** reinforcement-learning, q-learning, policy-evaluation
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-11-15. Score: 1 | Accepted answer
  
  For off-policy learning you must have two policies - a behaviour policy and a target policy. If the two policies are the same, then you end up with SARSA, not Q learning.
  
  You cannot use Q learning directly for evaluating a fixed target policy, because it directly learns optimal value function as the target policy, regardless of the behaviour policy. Instead you must use another variant of off-policy learning that can evaluate an arbitrary target policy.
  
  Your suggested algorithm is:
  
  > 1. Have the policy to be evaluated as the behaviour policy.
  > 2. Update the Q value conventionally (i.e. updating $Q(s,a)$ using the action $a'$ giving highest $Q(s',a')$ value)
  > 3. The final $Q(s,a)$ values will reflect the values for the policy being evaluated.
  
  This will not work for evaluating the behaviour policy. If the behaviour policy was stochastic and covered all possible state/action choices, then it will still be Q learning and converge on the optimal value function - maybe very slowly if the behaviour policy did not get to important states very often.
  
  The "trick" to off-policy is that the environment interaction part uses the behaviour policy to collect data, and the update step uses the target policy to calculate estimated returns. In general for off-policy updates, there can be corrections required to re-weight the estimated returns. However, one nice thing about single-step TD methods is that there are no such additional corrections needed.
  
  So this gives a way to do off-policy TD learning, using an approach called [Expected SARSA](https://www.geeksforgeeks.org/expected-sarsa-in-reinforcement-learning/). To use Expected SARSA, you will need to know the distribution of action choices i.e. know $\pi(a|s)$ for the target policy.
  
  This is the variant of your description that will work to evaluate your target policy $\pi(a|s)$:
  
  1. Have any stochastic policy that "covers" the target policy as the behaviour policy.
  2. Update the Q value using Expected SARSA $Q(s,a) = Q(s,a) + \alpha(r + \gamma [\sum\_{a'} \pi(a'|s')Q(s',a')] - Q(s,a))$
  3. The final $Q(s,a)$ values will reflect the values for the policy being evaluated.
  
  Worth noting that Expected SARSA with a target policy of $\pi(s) = \text{argmax}\_a Q(s,a)$ is exactly Q learning. Expected SARSA is a strict generalisation of Q learning that allows for learning the value function of *any* target policy. You may not see it used as much as Q learning, because the goal of learning an optimal value function is more common in practice.
  
  > **Comments:**
  > **username_2**: Although I don't know the details yet, I think you may also want to take a look at [**off-policy (policy) evaluation**](https://papers.nips.cc/paper/2019/file/4ffb0d2ba92f664c2281970110a2e071-Paper.pdf) (if you are not already aware of these concepts). Can you explain a little why expected SARSA would make this an off-policy evaluation method?  
  > **username_0**: @Neil, Thanks for the detailed answer. I'm unclear why ESARSA would work though. I feel that it would only work if the target policy being learnt is the same as the behaviour policy. i.e. we have an on-policy setting.  
  > **username_1**: @DhruvMullick: On-policy vs off-policy is a separate issue from evaluation vs control. The key for off-policy evaluation is that you set the target policy to be the policy that you want to evaluate. Nothing more is required. Can you explain more why you feel evaluation needs to be on-policy? Then I may be able to address that.  
  > **username_0**: @NeilSlater, thank you, I understand now. I was earlier under the wrong impression that for Policy evaluation, I would have to set the behaviour policy as the policy to be evaluated.  
  > **username_3**: @NeilSlater Thanks for the answer, this and many of your answers have been very helpful. Could you please help with a follow up question here? I am wondering, since we are evaluating value of the fixed given policy here, is it best to use a fully random behavior policy so that value is trained over more diverse states than we would see with the more commonly used eps-greedy behavior policy?  
  > **username_1**: @ModCon Generally no, for evaluating a fixed policy where you are in control of the behaviour policy, then you should just use an on-policy method. That's because off-policy methods add variance to measurement. However you are not always in control of behaviour policy - e.g. you might have historical observations where it isn't even clear what the behaviour policy was.  
  > **username_3**: @NeilSlater Thanks so much for replying! If I understand correctly, even if I follow fixed policy to choose every behaviour action, given that state is sampled randomly at t=0, we should get experiences for different initial states, proving enough for learning state value function. In my case, I am learning a state action value function, can I keep the action sampled t=0 randomly chosen and only the next actions as per fixed policy (instead of choosing even t=0 action as per fixed policy) so that there’s experiences starting from all possible actions?  
  > **username_1**: @ModCon I'm not clear why you would want to do that. I don't think the comments section here will be enough to cover your question. Please consider asking a new question on the site  
  
  ---
  
  