Answers.Body: <p>We are choosing actions randomly with probabilities given by the policy <span class="math-container">$\pi$</span>. For example, one policy might make two actions A and B equally likely, another might choose A 90% of the time.</p>
  <p>What we're actually calculating is not the value of a particular path, but the expected value of following the policy (sort of the average value).</p>
  <p>To calculate the expected value, we sum (over all possible actions) the probability of each action (as given by the policy) times the return from that action.</p>
  <p><span class="math-container">$V^\pi(s) = \sum_a \pi(a | s) [R(s,a) + \gamma  V^\pi(s')]$</span></p>
  <p>If the transitions are also stochastic, we do a similar sum over next states:</p>
  <p><span class="math-container">$V^\pi(s) = \sum_a \pi(a | s) \sum_{s'} P(s' | a,s) [R(s,a,s') + \gamma  V^\pi(s')]$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-15T02:02:45.590Z
Answers.LastActivityDate: 2022-06-15T02:08:31.380Z
Answers.LastEditDate: 2022-06-15T02:08:31.380Z
Answers.UserDisplayName: Lee Reeves
Answers.Body: <p>Deterministic policies are just a special case of stochastic policies. The probability distribution of a deterministic policy has a single spike of value 1 for the deterministic action.</p>
  <p>You are right that the value function keeps changing based on the random action chosen but all of these changes cause the value function to converge to a fixed point which is the value of that policy. This follows directly from the law of large numbers.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-08T18:28:28.357Z
Answers.LastActivityDate: 2023-09-08T18:28:28.357Z
Answers.UserDisplayName: user75923
Body: <p>I'm learning about reinforcement learning, particularly policy gradient methods and actor-critic methods. I've noticed that many algortihms use stochastic policies during training (i.e. they select the actions from a probability distribution).</p>
  <p>I don't understand how the value function for stochastic policies is estimated accurately. The value function <span class="math-container">$V^{\pi}(s)$</span> is the expected return when starting at state <span class="math-container">$s$</span> and following policy <span class="math-container">$\pi$</span>, but if we are choosing actions randomly then we are not really following any particular policy.</p>
  <p>How are we getting an accurate estimate of the value of policy <span class="math-container">$\pi$</span> if we are not following a deterministic policy <span class="math-container">$\pi$</span> during training?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-14T13:29:31.093Z
Comments.Text: The value function is the expected (discounted) future rewards from the current state. The expectation is taking with respect to the policy, so if the policy is stochastic then it is measuring the returns under the policy in expectation, I.e. what the policy does on average.
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-14T14:31:49.260Z
Comments.Text: "I've noticed that many algortihms use stochastic policies during training", I know that you're talking about policy gradients and actor-critic algorithms, but I recommend that you provide 1-2 examples of a specific algorithm that does this so that there's no ambiguity and people can answer your question more concretely.
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><policy-gradients><actor-critic-methods><value-functions><stochastic-policy>
Title: How do we estimate the value of a stochastic policy?
UserDisplayName: mac_or_cheese
ThreadText: # How do we estimate the value of a stochastic policy?
  
  (Asked by: username_0 on 2022-06-14)
  
  I'm learning about reinforcement learning, particularly policy gradient methods and actor-critic methods. I've noticed that many algortihms use stochastic policies during training (i.e. they select the actions from a probability distribution).
  
  I don't understand how the value function for stochastic policies is estimated accurately. The value function $V^{\pi}(s)$ is the expected return when starting at state $s$ and following policy $\pi$, but if we are choosing actions randomly then we are not really following any particular policy.
  
  How are we getting an accurate estimate of the value of policy $\pi$ if we are not following a deterministic policy $\pi$ during training?
  
  **Tags:** reinforcement-learning, policy-gradients, actor-critic-methods, value-functions, stochastic-policy
  
  > **Comments:**
  > **username_1**: The value function is the expected (discounted) future rewards from the current state. The expectation is taking with respect to the policy, so if the policy is stochastic then it is measuring the returns under the policy in expectation, I.e. what the policy does on average.  
  > **username_2**: "I've noticed that many algortihms use stochastic policies during training", I know that you're talking about policy gradients and actor-critic algorithms, but I recommend that you provide 1-2 examples of a specific algorithm that does this so that there's no ambiguity and people can answer your question more concretely.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2022-06-15. Score: 1
  
  We are choosing actions randomly with probabilities given by the policy $\pi$. For example, one policy might make two actions A and B equally likely, another might choose A 90% of the time.
  
  What we're actually calculating is not the value of a particular path, but the expected value of following the policy (sort of the average value).
  
  To calculate the expected value, we sum (over all possible actions) the probability of each action (as given by the policy) times the return from that action.
  
  $V^\pi(s) = \sum\_a \pi(a | s) [R(s,a) + \gamma V^\pi(s')]$
  
  If the transitions are also stochastic, we do a similar sum over next states:
  
  $V^\pi(s) = \sum\_a \pi(a | s) \sum\_{s'} P(s' | a,s) [R(s,a,s') + \gamma V^\pi(s')]$
  
  
  ---
  
  ### Answer by username_4 on 2023-09-08. Score: 0
  
  Deterministic policies are just a special case of stochastic policies. The probability distribution of a deterministic policy has a single spike of value 1 for the deterministic action.
  
  You are right that the value function keeps changing based on the random action chosen but all of these changes cause the value function to converge to a fixed point which is the value of that policy. This follows directly from the law of large numbers.
  
  
  ---
  
  