Answers.Body: <p>In those policy gradient algorithms, we typically have:</p>
  
  <ul>
  <li>Input features <span class="math-container">$x(s)$</span> for any state <span class="math-container">$s$</span></li>
  <li>A fixed action set <span class="math-container">$\mathcal{A}$</span> of a fixed size (in reinforcement learning (RL) it is often assumed that all actions are legal in all states. In some situations, like board games, we'll filter out illegal actions manually in a postprocessing step).</li>
  <li>A function approximator <span class="math-container">$\pi(x)$</span> (often a neural network), which takes as input the features <span class="math-container">$x(s)$</span> of some state <span class="math-container">$s$</span>, and produces as output <strong>a vector</strong> of probabilities, <strong>where the vector has a length equal to the size <span class="math-container">$\vert \mathcal{A} \vert$</span> of the action space</strong>.</li>
  </ul>
  
  <p>Those bolded parts in the last point there are critical for your question. We do not normally have a function that "spits out some number" as you described, we typically have one that "spits out" multiple numbers; one for every action. Often, these numbers can also be directly interpreted as probabilities, because they are forced to all lie in <span class="math-container">$[0, 1]$</span> and sum up to <span class="math-container">$1$</span> (this is often done using a softmax layer at the end of a neural network for example).</p>
  
  <p>Once you have such a vector of "probabilities", one for every action, it is relatively straightforward to sample from that. Just give every action a probability of being selected equal to its corresponding output.</p>
  
  <p>Note that in some cases, people describe their function approximator as producing "logits" rather than "probabilities" as outputs. These are real-valued outputs which do not necessarily sum up to one, typically produced by a linear layer at the end of a neural network rather than a softmax layer (see e.g. AlphaGo Zero). It is then still implicitly assumed that we convert them into numbers that can be interpreted as probabilities through the application of a softmax, before sampling from it.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-02T11:17:53.907Z
Answers.LastActivityDate: 2019-06-10T18:09:28.137Z
Answers.LastEditDate: 2019-06-10T18:09:28.137Z
Answers.UserDisplayName: Dennis Soemers
Body: <p>Recently I started to look at policy gradient methods and policies are represented as functions with features for larger problems with many states. Many articles and pseudocodes of algorithms mention sampling an action from the policy, but it is unclear to me how.</p>
  
  <p>Actions are something we do in the environment, like going left, right, etc... And functions take some feature values and parameters, make calculations, and 'spit out' some number. So how do we actually map that number to a certain action, and how do we know what action to take?</p>
  
Tags: <reinforcement-learning>
Title: How do we actually sample an action from a policy in policy gradient methods?
UserDisplayName: Brale
ThreadText: # How do we actually sample an action from a policy in policy gradient methods?
  
  (Asked by: username_0 on 2018-12-01)
  
  Recently I started to look at policy gradient methods and policies are represented as functions with features for larger problems with many states. Many articles and pseudocodes of algorithms mention sampling an action from the policy, but it is unclear to me how.
  
  Actions are something we do in the environment, like going left, right, etc... And functions take some feature values and parameters, make calculations, and 'spit out' some number. So how do we actually map that number to a certain action, and how do we know what action to take?
  
  **Tags:** reinforcement-learning
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-12-02. Score: 2 | Accepted answer
  
  In those policy gradient algorithms, we typically have:
  
  * Input features $x(s)$ for any state $s$
  * A fixed action set $\mathcal{A}$ of a fixed size (in reinforcement learning (RL) it is often assumed that all actions are legal in all states. In some situations, like board games, we'll filter out illegal actions manually in a postprocessing step).
  * A function approximator $\pi(x)$ (often a neural network), which takes as input the features $x(s)$ of some state $s$, and produces as output **a vector** of probabilities, **where the vector has a length equal to the size $\vert \mathcal{A} \vert$ of the action space**.
  
  Those bolded parts in the last point there are critical for your question. We do not normally have a function that "spits out some number" as you described, we typically have one that "spits out" multiple numbers; one for every action. Often, these numbers can also be directly interpreted as probabilities, because they are forced to all lie in $[0, 1]$ and sum up to $1$ (this is often done using a softmax layer at the end of a neural network for example).
  
  Once you have such a vector of "probabilities", one for every action, it is relatively straightforward to sample from that. Just give every action a probability of being selected equal to its corresponding output.
  
  Note that in some cases, people describe their function approximator as producing "logits" rather than "probabilities" as outputs. These are real-valued outputs which do not necessarily sum up to one, typically produced by a linear layer at the end of a neural network rather than a softmax layer (see e.g. AlphaGo Zero). It is then still implicitly assumed that we convert them into numbers that can be interpreted as probabilities through the application of a softmax, before sampling from it.
  
  
  ---
  
  