Answers.Body: <p>(This does not fully answer your question. Actually, it just attempts to make sure you understood certain things).</p>
  <p>The distribution that you learn should put more weight/mass/density on the best actions. So, if you sample during inference, you should get the best actions more frequently than other actions.</p>
  <p>You would use the same action every time if you are certain that the optional policy is deterministic, which is the case of finite MDPs.</p>
  <p>There are cases where the best policy is really stochastic (e.g. rock-paper-scissors, which is naturally represented as Markov Game, so not a finite MDP), so choosing always the same action is likely not a good idea, so the best approach during inference would actually be to sample.</p>
  <p>So, you would need to pick the same action every time if you have a finite MDP (finite action and state spaces).</p>
  <p>See also <a href="https://ai.stackexchange.com/a/16853/2444">this answer</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-23T10:26:36.717Z
Answers.Text: Maybe I should convert this to 1 or more comments, if this doesn't even partially answer your questions.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-23T13:06:49.537Z
Answers.Text: Keep as an answer it's interesting. I'm aware of cases like rock-paper-scissor in which the best policy is always stochastic to avoid the opponent to exploit the determinism. But excluding that (I may clarify this in the question), I'd like to understand what is the best to do at inference time with the policy. Actually, could you elaborate a bit more about the difference of *finite MDPs* and *continuous MDPs*, in particular why in the continuous case one don't want to have deterministic actions?
Answers.UserDisplayName: Luca Anzalone
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-23T17:52:01.857Z
Answers.Text: @LucaAnzalone That's actually a good question. I shouldn't have used "only if" but "if" in my last sentence. In other words, I just wanted to point out that, if you have a finite MDP, you're sure that the optimal policy is deterministic. In other cases, it might not be. So, I did _not_ want to say that in continuous actions MDPs the optimal policy is always stochastic, because I don't know if that's the case (I think not - one can probably think of a very simple environment with 2 continuous actions where the optimal policy is deterministic, but I didn't think much about this).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-28T17:49:02.197Z
Answers.Text: Well, I can think of an env in which there are three paths that lead to a terminal state, the paths on the left and right are highly rewarding while the middle path is poor. Suppose the actions are steer left/right (from -1 to 1), the "mode" policy will always pick something close to either -1 or 1, while the "mean" policy just average to 0 (or so) leading to the non-rewarding path. In this case the best to do would either to be deterministic or taking one or the modes randomly at each trial.
Answers.UserDisplayName: Luca Anzalone
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-23T10:20:53.143Z
Answers.LastActivityDate: 2023-04-24T21:44:20.943Z
Answers.LastEditDate: 2023-04-24T21:44:20.943Z
Answers.UserDisplayName: nbro
Body: <p>Policy gradient agents like A2C, PPO, etc learn a distribution over the action space that is parametrized by a neural net. For continuous actions the distribution is usually a Gaussian, while for discrete ones it can be a Categorical (or a relaxation).</p>
  <p><strong>The question is:</strong> during training we sample from such distribution and then compute the log-probability, but during inference (also agent evaluation/test) what is the best to do? Usually, during evaluation we want to <em>exploit</em> so we should't sample the action (unless is recommended for a given environment setup, to avoid an opponent to exploit a deterministic policy - <em>but I'd like to exclude this case</em>), and so we want to take the action that exploits the more.</p>
  <p>I'm puzzled about whether to take the <code>mean</code> or <code>mode</code> of the action distribution during inference and evaluation (I assume the agent training has ended): also, is the <em>most likely action</em> the action that exploits?</p>
  <p>In pseudo-code you can view the problem as follows:</p>
  <pre><code>function act(state):
     action_distribution = policy(state)
     
     if evaluation:
        # here is the problem: mean or mode?
        actions = action_distribution.mean()  # or mode()
     else:
        # training phase
        actions = action_distribution.sample()
  
    return actions
  </code></pre>
  <p><strong>Clarifications:</strong> I assume a generic action distribution in which its mean can be different from its mode, like a Beta distribution for example. What I also ask is:</p>
  <ul>
  <li>Considering a given state, does taking the mean of the distribution corresponds to an action that is associated to the <em>highest expected value</em>?</li>
  <li>Similarly, considering the mode should correspond to the <em>most likely action</em> (to simplify I assume the mode to be unique) which should be associated to the <em>highest state-action</em> (or Q) <em>value</em>. Is that so?</li>
  </ul>
  <p>Additionally, I'm aware of <a href="https://ai.stackexchange.com/questions/16848/what-is-the-effect-of-picking-action-deterministicly-at-inference-with-policy-gr/16853#16853">this answer</a>: it's related but does not answer my question.</p>
  <p>What I'd like to know is: at evaluation/inference time, assuming for a given environment is fine to have a deterministic policy, how to best implement/obtain that from a learned action distribution?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-23T22:30:53.033Z
Comments.Text: How could you choose the mean action from a Bernoulli distn? The mean is $p$, whereas the mode is 0 or 1. for a discrete distn, you would usually want to choose the modal action. In a continuous distribution, the mode might be complex to calculate so I’d recommend using the mean.
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-24T08:37:13.130Z
Comments.Text: Sure, the Bernoulli is a bad example (I'll edit that) so assume you have a Beta instead (indeed, the action space is continuous), what you would do in that case? still choosing the modal action? If so, can you motivate further
Comments.UserDisplayName: Luca Anzalone
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-24T17:12:48.683Z
Comments.Text: I think my last comment is still valid. Finding the mode could be non-trivial (though I suppose in a Beta it is will known). So, in this case, the advice would simply to be to use whichever gives the best performance.
Comments.UserDisplayName: David
Tags: <reinforcement-learning><deep-rl><policy-gradients>
Title: Mean or Mode of Action Distribution when Evaluating Policy Gradient Agents
UserDisplayName: Luca Anzalone
ThreadText: # Mean or Mode of Action Distribution when Evaluating Policy Gradient Agents
  
  (Asked by: username_0 on 2023-04-21)
  
  Policy gradient agents like A2C, PPO, etc learn a distribution over the action space that is parametrized by a neural net. For continuous actions the distribution is usually a Gaussian, while for discrete ones it can be a Categorical (or a relaxation).
  
  **The question is:** during training we sample from such distribution and then compute the log-probability, but during inference (also agent evaluation/test) what is the best to do? Usually, during evaluation we want to *exploit* so we should't sample the action (unless is recommended for a given environment setup, to avoid an opponent to exploit a deterministic policy - *but I'd like to exclude this case*), and so we want to take the action that exploits the more.
  
  I'm puzzled about whether to take the `mean` or `mode` of the action distribution during inference and evaluation (I assume the agent training has ended): also, is the *most likely action* the action that exploits?
  
  In pseudo-code you can view the problem as follows:
  
  ```
  function act(state):
     action_distribution = policy(state)
     
     if evaluation:
        # here is the problem: mean or mode?
        actions = action_distribution.mean()  # or mode()
     else:
        # training phase
        actions = action_distribution.sample()
  
    return actions
  
  ```
  
  **Clarifications:** I assume a generic action distribution in which its mean can be different from its mode, like a Beta distribution for example. What I also ask is:
  
  * Considering a given state, does taking the mean of the distribution corresponds to an action that is associated to the *highest expected value*?
  * Similarly, considering the mode should correspond to the *most likely action* (to simplify I assume the mode to be unique) which should be associated to the *highest state-action* (or Q) *value*. Is that so?
  
  Additionally, I'm aware of [this answer](https://ai.stackexchange.com/questions/16848/what-is-the-effect-of-picking-action-deterministicly-at-inference-with-policy-gr/16853#16853): it's related but does not answer my question.
  
  What I'd like to know is: at evaluation/inference time, assuming for a given environment is fine to have a deterministic policy, how to best implement/obtain that from a learned action distribution?
  
  **Tags:** reinforcement-learning, deep-rl, policy-gradients
  
  > **Comments:**
  > **username_1**: How could you choose the mean action from a Bernoulli distn? The mean is $p$, whereas the mode is 0 or 1. for a discrete distn, you would usually want to choose the modal action. In a continuous distribution, the mode might be complex to calculate so I’d recommend using the mean.  
  > **username_0**: Sure, the Bernoulli is a bad example (I'll edit that) so assume you have a Beta instead (indeed, the action space is continuous), what you would do in that case? still choosing the modal action? If so, can you motivate further  
  > **username_1**: I think my last comment is still valid. Finding the mode could be non-trivial (though I suppose in a Beta it is will known). So, in this case, the advice would simply to be to use whichever gives the best performance.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-04-23. Score: 3 | Accepted answer
  
  (This does not fully answer your question. Actually, it just attempts to make sure you understood certain things).
  
  The distribution that you learn should put more weight/mass/density on the best actions. So, if you sample during inference, you should get the best actions more frequently than other actions.
  
  You would use the same action every time if you are certain that the optional policy is deterministic, which is the case of finite MDPs.
  
  There are cases where the best policy is really stochastic (e.g. rock-paper-scissors, which is naturally represented as Markov Game, so not a finite MDP), so choosing always the same action is likely not a good idea, so the best approach during inference would actually be to sample.
  
  So, you would need to pick the same action every time if you have a finite MDP (finite action and state spaces).
  
  See also [this answer](https://ai.stackexchange.com/a/16853/2444)
  
  > **Comments:**
  > **username_2**: Maybe I should convert this to 1 or more comments, if this doesn't even partially answer your questions.  
  > **username_0**: Keep as an answer it's interesting. I'm aware of cases like rock-paper-scissor in which the best policy is always stochastic to avoid the opponent to exploit the determinism. But excluding that (I may clarify this in the question), I'd like to understand what is the best to do at inference time with the policy. Actually, could you elaborate a bit more about the difference of *finite MDPs* and *continuous MDPs*, in particular why in the continuous case one don't want to have deterministic actions?  
  > **username_2**: @username_0 That's actually a good question. I shouldn't have used "only if" but "if" in my last sentence. In other words, I just wanted to point out that, if you have a finite MDP, you're sure that the optimal policy is deterministic. In other cases, it might not be. So, I did _not_ want to say that in continuous actions MDPs the optimal policy is always stochastic, because I don't know if that's the case (I think not - one can probably think of a very simple environment with 2 continuous actions where the optimal policy is deterministic, but I didn't think much about this).  
  > **username_0**: Well, I can think of an env in which there are three paths that lead to a terminal state, the paths on the left and right are highly rewarding while the middle path is poor. Suppose the actions are steer left/right (from -1 to 1), the "mode" policy will always pick something close to either -1 or 1, while the "mean" policy just average to 0 (or so) leading to the non-rewarding path. In this case the best to do would either to be deterministic or taking one or the modes randomly at each trial.  
  
  ---
  
  