Answers.Body: <blockquote>
  <p>But, in a real application under a given deterministic policy <span class="math-container">$\pi$</span>, how can you choose the initial action <span class="math-container">$a$</span> arbitrarily at state <span class="math-container">$s$</span> because it is already fixed by the policy <span class="math-container">$\pi$</span>: <span class="math-container">$a=\pi(s)$</span>?</p>
  </blockquote>
  <p>You ignore <span class="math-container">$\pi$</span> for the action selection of the action you need to evaluate. This <em>is</em> well-defined, it is the expected future return for taking action <span class="math-container">$a$</span> in state <span class="math-container">$s$</span> and thereafter following the policy <span class="math-container">$\pi$</span>.</p>
  <p>So you may have a problem estimating that value, unless you also force that action choice during training in order to observe what transitions and rewards follow (that you would never see following the deterministic policy). However, it is well-defined conceptually.</p>
  <p>A very simple way to force the assessment of all state/action pairs is <em>exploring starts</em>: Pick an arbitrary state/action pair to evaluate, then follow the transition and policy rules from that point on until the espidode end. This will give you a Monte Carlo sample of the value for the starting point, that you can use to update the estimate.</p>
  <p>If you are not able to use exploring starts, or otherwise take actions different from the supplied deterministic policy, then you may be stuck. You would only have data of certain <span class="math-container">$s,a$</span> pairs. You could estimate <span class="math-container">$Q(s,a)$</span> for those pairs, and not for others.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-29T22:03:43.813Z
Answers.Text: @ Neil Slater In a real application, how can you pick an arbitrary state/action pair to evaluate? Maybe, the  state/action pair you chose does not exist in episodes.
Answers.UserDisplayName: user3489173
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-29T22:09:33.457Z
Answers.Text: The answer includes this - *start* your episode from the arbitrary state/action pair. That is called *exploring starts*. If you only have historical data, or are not able to set up the environment in particular states, then your options are limited. Probably you will not be able to complete the Q function, given your other constraint of a fixed deterministic policy. But maybe that's not important. You could still create a state value function for all reachable states for instance.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-29T22:12:44.817Z
Answers.Text: @user3489173 You don't say in the question what your goal is. If you have a specific goal and constraints you must meet, I may be able to explain whether it is logically possible.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-29T22:19:16.310Z
Answers.Text: @ Neil Slater. Assume that our application is a real-time application and the historical data of this application contains n number of episodes where n is a large number. I wonder how  you can  estimate action value function using monte carlo methods.
Answers.UserDisplayName: user3489173
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-29T22:21:01.623Z
Answers.Text: @user3489173 If you have historical data using a fixed policy, you can only estimate action values of the actions that were taken. If the policy was deterministic, that will mean you know *nothing* about any alternative actions. You can fill in $Q(s,a)$ estimates for the actions that were observed, but not anything else.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-29T22:22:58.727Z
Answers.Text: @ Neil Slater. Perfect explanation! You must write a reinforcement learning book!
Answers.UserDisplayName: user3489173
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-29T21:36:13.463Z
Answers.LastActivityDate: 2022-01-29T23:30:54.777Z
Answers.LastEditDate: 2022-01-29T23:30:54.777Z
Answers.UserDisplayName: Neil Slater
Body: <p>In Monte Carlo-based action value estimation  problem for a deterministic policy (estimation of <span class="math-container">$q_{\pi}(s,a)$</span>),the estimation problem seems not to be well-defined because  <span class="math-container">$q_{\pi}(s,a)$</span> by definition means the value of an arbitrary action <span class="math-container">$a$</span> at a given state <span class="math-container">$s$</span>
  when initial action <span class="math-container">$a$</span> is applied at that state and then following actions from policy <span class="math-container">$\pi$</span> at
  the next states. But, in a real application under a given deterministic policy <span class="math-container">$\pi$</span>, how can you choose the initial
  action <span class="math-container">$a$</span> arbitrarily at state <span class="math-container">$s$</span> because it is already fixed by the policy <span class="math-container">$\pi$</span>: <span class="math-container">$a=\pi(s)$</span>?</p>
  
Tags: <reinforcement-learning><monte-carlo-methods>
Title: Action-value estimation of deterministic policies with Monte Carlo method
UserDisplayName: user3489173
ThreadText: # Action-value estimation of deterministic policies with Monte Carlo method
  
  (Asked by: username_0 on 2022-01-29)
  
  In Monte Carlo-based action value estimation problem for a deterministic policy (estimation of $q\_{\pi}(s,a)$),the estimation problem seems not to be well-defined because $q\_{\pi}(s,a)$ by definition means the value of an arbitrary action $a$ at a given state $s$
  when initial action $a$ is applied at that state and then following actions from policy $\pi$ at
  the next states. But, in a real application under a given deterministic policy $\pi$, how can you choose the initial
  action $a$ arbitrarily at state $s$ because it is already fixed by the policy $\pi$: $a=\pi(s)$?
  
  **Tags:** reinforcement-learning, monte-carlo-methods
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-01-29. Score: 4 | Accepted answer
  
  > But, in a real application under a given deterministic policy $\pi$, how can you choose the initial action $a$ arbitrarily at state $s$ because it is already fixed by the policy $\pi$: $a=\pi(s)$?
  
  You ignore $\pi$ for the action selection of the action you need to evaluate. This *is* well-defined, it is the expected future return for taking action $a$ in state $s$ and thereafter following the policy $\pi$.
  
  So you may have a problem estimating that value, unless you also force that action choice during training in order to observe what transitions and rewards follow (that you would never see following the deterministic policy). However, it is well-defined conceptually.
  
  A very simple way to force the assessment of all state/action pairs is *exploring starts*: Pick an arbitrary state/action pair to evaluate, then follow the transition and policy rules from that point on until the espidode end. This will give you a Monte Carlo sample of the value for the starting point, that you can use to update the estimate.
  
  If you are not able to use exploring starts, or otherwise take actions different from the supplied deterministic policy, then you may be stuck. You would only have data of certain $s,a$ pairs. You could estimate $Q(s,a)$ for those pairs, and not for others.
  
  > **Comments:**
  > **username_0**: @ Neil Slater In a real application, how can you pick an arbitrary state/action pair to evaluate? Maybe, the  state/action pair you chose does not exist in episodes.  
  > **username_1**: The answer includes this - *start* your episode from the arbitrary state/action pair. That is called *exploring starts*. If you only have historical data, or are not able to set up the environment in particular states, then your options are limited. Probably you will not be able to complete the Q function, given your other constraint of a fixed deterministic policy. But maybe that's not important. You could still create a state value function for all reachable states for instance.  
  > **username_1**: @user3489173 You don't say in the question what your goal is. If you have a specific goal and constraints you must meet, I may be able to explain whether it is logically possible.  
  > **username_0**: @ Neil Slater. Assume that our application is a real-time application and the historical data of this application contains n number of episodes where n is a large number. I wonder how  you can  estimate action value function using monte carlo methods.  
  > **username_1**: @user3489173 If you have historical data using a fixed policy, you can only estimate action values of the actions that were taken. If the policy was deterministic, that will mean you know *nothing* about any alternative actions. You can fill in $Q(s,a)$ estimates for the actions that were observed, but not anything else.  
  > **username_0**: @ Neil Slater. Perfect explanation! You must write a reinforcement learning book!  
  
  ---
  
  