Answers.Body: <p>You sample according to the probability distribution <span class="math-container">$\pi(a \mid s, \theta)$</span>, so you do <strong>not</strong> always take the action with the highest probability (otherwise there would be no <em>exploration</em> but just <em>exploitation</em>), but the most probable action should be sampled the most. However, keep in mind that the policy, <span class="math-container">$\theta$</span>, changes, so also the probability distribution. <a href="https://github.com/pytorch/examples/blob/151944ecaf9ba2c8288ee550143ae7ffdaa90a80/reinforcement_learning/reinforce.py#L53" rel="nofollow noreferrer">This implementation</a> could be useful, as it shows exactly what I've just said.</p>
  <p>I've also seen another implementation that applied some kind of <span class="math-container">$\epsilon$</span>-greedy policy to sample from <span class="math-container">$\pi$</span> (i.e. with probability <span class="math-container">$\epsilon$</span> you could choose the greedy action and with probability <span class="math-container">$1 - \epsilon$</span> any other action), but I am not sure how common or useful that could be in the basic REINFORCE algorithm. <a href="https://ai.stackexchange.com/q/21699/2444">This question</a> seems to be asking about additional exploration strategies in policy gradients, in case you're interested.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-10T18:34:56.453Z
Answers.LastActivityDate: 2022-01-10T18:34:56.453Z
Answers.UserDisplayName: nbro
Body: <p>Consider the following algorithm from the textbook titled <strong>Reinforcement
  Learning: An Introduction (second edition)</strong> by <em>Richard S. Sutton and Andrew G. Bart</em></p>
  <p><a href="https://i.sstatic.net/jFXnI.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/jFXnI.png" alt="enter image description here" /></a></p>
  <p>While playing the game for the generation of an episode trajectory, how is the action selected by the agent? I mean, how does the agent selection action <span class="math-container">$A_i$</span> from state <span class="math-container">$S_i$</span> for <span class="math-container">$0 \le i \le T-1$</span>. I am getting this doubt as the policy is stochastic in nature and doesn't give a single action as output.</p>
  <p>But the algorithm says to generate the episode following the stochastic policy function.</p>
  <p>Is it always the action that has the high probability in <span class="math-container">$\pi(a|s,\theta)$</span>?</p>
  <p><strong>Note</strong>: The trajectory here is <span class="math-container">$(S_0,A_0,R_1 \cdots S_{T-1},A_{T-1},R_{T})$</span>,</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-10T16:28:03.000Z
Comments.Text: you just sample an action from your current policy. the actions taken will depend on your current version of the policy but actions will be sampled with probability (proportional to, in the case of a continuous action space) given by the policy. this implies that it is more likely that actions with higher probability are more likely to be taken.
Comments.UserDisplayName: David
Tags: <reinforcement-learning><deep-rl><policy-gradients><reinforce><exploration-strategies>
Title: Is it the high probability action that is always selected by the agent in REINFORCE algorithm?
UserDisplayName: hanugm
ThreadText: # Is it the high probability action that is always selected by the agent in REINFORCE algorithm?
  
  (Asked by: username_0 on 2022-01-07)
  
  Consider the following algorithm from the textbook titled **Reinforcement
  Learning: An Introduction (second edition)** by *Richard S. Sutton and Andrew G. Bart*
  
  [![enter image description here](https://i.sstatic.net/jFXnI.png)](https://i.sstatic.net/jFXnI.png)
  
  While playing the game for the generation of an episode trajectory, how is the action selected by the agent? I mean, how does the agent selection action $A\_i$ from state $S\_i$ for $0 \le i \le T-1$. I am getting this doubt as the policy is stochastic in nature and doesn't give a single action as output.
  
  But the algorithm says to generate the episode following the stochastic policy function.
  
  Is it always the action that has the high probability in $\pi(a|s,\theta)$?
  
  **Note**: The trajectory here is $(S\_0,A\_0,R\_1 \cdots S\_{T-1},A\_{T-1},R\_{T})$,
  
  **Tags:** reinforcement-learning, deep-rl, policy-gradients, reinforce, exploration-strategies
  > **Comments:**
  > **username_1**: you just sample an action from your current policy. the actions taken will depend on your current version of the policy but actions will be sampled with probability (proportional to, in the case of a continuous action space) given by the policy. this implies that it is more likely that actions with higher probability are more likely to be taken.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-01-10. Score: 3 | Accepted answer
  
  You sample according to the probability distribution $\pi(a \mid s, \theta)$, so you do **not** always take the action with the highest probability (otherwise there would be no *exploration* but just *exploitation*), but the most probable action should be sampled the most. However, keep in mind that the policy, $\theta$, changes, so also the probability distribution. [This implementation](https://github.com/pytorch/examples/blob/151944ecaf9ba2c8288ee550143ae7ffdaa90a80/reinforcement_learning/reinforce.py#L53) could be useful, as it shows exactly what I've just said.
  
  I've also seen another implementation that applied some kind of $\epsilon$-greedy policy to sample from $\pi$ (i.e. with probability $\epsilon$ you could choose the greedy action and with probability $1 - \epsilon$ any other action), but I am not sure how common or useful that could be in the basic REINFORCE algorithm. [This question](https://ai.stackexchange.com/q/21699/2444) seems to be asking about additional exploration strategies in policy gradients, in case you're interested.
  
  
  ---
  
  