Answers.Body: <p>The fundamental idea behind policy gradient is just to maximise the return averaged across all probably trajectories, i.e</p>
  <p><span class="math-container">$$\begin{align} J(\theta) &amp;= E[\sum\limits_{t=1}^{\tau}r(s_t,a_t)]\\
  &amp;=E_{\tau\sim p(\tau)}[R(\tau)]
  \end{align}$$</span></p>
  <p>Where <span class="math-container">$\tau$</span> represents the probability of selecting a particular trajectory, if the trajectories all have fixed length then <span class="math-container">$\tau$</span> only has non-zero probability for trajectories of the specified length which in this case is 1.</p>
  <p>The REINFORCE algorithm takes this expression and with some simplifications (causality to improve variance) and manipulation (log trick) obtains the gradient, pretty much as simple as that.</p>
  <hr />
  <p><strong>Intuition of algorithm in paper</strong></p>
  <p>In the algorithm they denote the <span class="math-container">$\pi_\theta$</span> (which is typically reserved for describing the policy) as the probability of the selection vector. By considering instead that the function <span class="math-container">$h_{\theta}$</span> as the policy instead I think it can be seen that they are actually averaging over multiple trajectories.</p>
  <p>So we instead think of the data points as state-action pairs that we pass into <span class="math-container">$h_\theta$</span> to get the probability of selecting said action for a given state. These probabilities then dictates whether we choose the action or not. An alternative way to interpret this as if we imagine the action space for each state as binary then we can think of the &quot;other action&quot; as not impacting the predictor.</p>
  <p>The gradient used for updating the parameters associated with the policy uses the log probability of <span class="math-container">$\pi_\theta$</span> which if we expand it, expressing it using <span class="math-container">$h_\theta$</span>, (as done on page 4)  we can see it's the sum of the log probabilities of selecting (or not selecting) each data point.</p>
  <p>By considering the return for each data point constant (each data point has the same loss incurred by predictor model on the validation set) and absorbing the average over batch size into the step size <span class="math-container">$\beta$</span> it could be interpreted as an average</p>
  <hr />
  <p><strong>Stability issues</strong></p>
  <p>RL is plagued with stability issues, be it selection of hyper parameters, random seeding etc. It's hard to often pinpoint why results aren't exact but as long as you get something in a similar ball park i'd say thats pretty good going</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-18T14:18:46.023Z
Answers.LastActivityDate: 2021-03-18T14:18:46.023Z
Answers.UserDisplayName: quest ions
Body: <p>I have come across a Google <a href="https://arxiv.org/abs/1909.11671" rel="nofollow noreferrer">paper</a> that uses the <code>REINFORCE</code> algorithm (a Policy Gradient Method) for a case where the trajectory of the episodes it proposes would be only one step.</p>
  <p>When trying to replicate the experiments they propose I found that there are some problems with the stability of the method (maybe that's why it is not accepted by peer review).</p>
  <p>Researching on my own I have found something I suspected, and that is that the problem they present could be solved as a multiarmed bandit problem in <a href="https://ai.stackexchange.com/questions/22812/is-it-possible-to-have-a-fixed-trajectory-size-in-the-vanilla-policy-gradient-al">THIS</a> link. But because of this event, I have the doubt if using methods based on trajectories (such as Policy Gradient Methods) has some mathematical problem in situations where the trajectory is a single step.</p>
  <p>PS: I think the problem of this paper may be also that they average only after one execution of a trajectory and not over <code>k</code> trajectories as it is necessary for a Policy Gradient Method, so I would also like to know the opinion of more people about this issue.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-03-17T17:23:56.613Z
Comments.Text: You say that the paper is not accepted by peer review but I see that it's accepted to "Proceedings of the 37th International Conference on Machine Learning". There is nothing wrong with using PG methods when there is only 1 step, the validity of approach does not depend on the length of the episode, whether it is useful or not is another question.
Comments.UserDisplayName: Brale
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-03-17T19:06:17.137Z
Comments.Text: True, I was mistaken about that. What do you think about averaging k trajectories?Recoding the problem, I face the issue that as I improve the episodes of the process, every data value converges to a value of 0.5 and I cannot figure out why.
Comments.UserDisplayName: Angelo
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-03-17T19:32:30.883Z
Comments.Text: yes, you can take a batch of samples, average policy gradients and then apply the result, seems reasonable to me
Comments.UserDisplayName: Brale
Tags: <reinforcement-learning><policy-gradients><reinforce>
Title: It is mathematically correct to use a Policy Gradient method for 1-step trajectories?
UserDisplayName: Angelo
ThreadText: # It is mathematically correct to use a Policy Gradient method for 1-step trajectories?
  
  (Asked by: username_0 on 2021-03-14)
  
  I have come across a Google [paper](https://arxiv.org/abs/1909.11671) that uses the `REINFORCE` algorithm (a Policy Gradient Method) for a case where the trajectory of the episodes it proposes would be only one step.
  
  When trying to replicate the experiments they propose I found that there are some problems with the stability of the method (maybe that's why it is not accepted by peer review).
  
  Researching on my own I have found something I suspected, and that is that the problem they present could be solved as a multiarmed bandit problem in [THIS](https://ai.stackexchange.com/questions/22812/is-it-possible-to-have-a-fixed-trajectory-size-in-the-vanilla-policy-gradient-al) link. But because of this event, I have the doubt if using methods based on trajectories (such as Policy Gradient Methods) has some mathematical problem in situations where the trajectory is a single step.
  
  PS: I think the problem of this paper may be also that they average only after one execution of a trajectory and not over `k` trajectories as it is necessary for a Policy Gradient Method, so I would also like to know the opinion of more people about this issue.
  
  **Tags:** reinforcement-learning, policy-gradients, reinforce
  > **Comments:**
  > **username_1**: You say that the paper is not accepted by peer review but I see that it's accepted to "Proceedings of the 37th International Conference on Machine Learning". There is nothing wrong with using PG methods when there is only 1 step, the validity of approach does not depend on the length of the episode, whether it is useful or not is another question.  
  > **username_0**: True, I was mistaken about that. What do you think about averaging k trajectories?Recoding the problem, I face the issue that as I improve the episodes of the process, every data value converges to a value of 0.5 and I cannot figure out why.  
  > **username_1**: yes, you can take a batch of samples, average policy gradients and then apply the result, seems reasonable to me  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-03-18. Score: 2 | Accepted answer
  
  The fundamental idea behind policy gradient is just to maximise the return averaged across all probably trajectories, i.e
  
  $$\begin{align} J(\theta) &= E[\sum\limits\_{t=1}^{\tau}r(s\_t,a\_t)]\\
  &=E\_{\tau\sim p(\tau)}[R(\tau)]
  \end{align}$$
  
  Where $\tau$ represents the probability of selecting a particular trajectory, if the trajectories all have fixed length then $\tau$ only has non-zero probability for trajectories of the specified length which in this case is 1.
  
  The REINFORCE algorithm takes this expression and with some simplifications (causality to improve variance) and manipulation (log trick) obtains the gradient, pretty much as simple as that.
  
  ---
  
  **Intuition of algorithm in paper**
  
  In the algorithm they denote the $\pi\_\theta$ (which is typically reserved for describing the policy) as the probability of the selection vector. By considering instead that the function $h\_{\theta}$ as the policy instead I think it can be seen that they are actually averaging over multiple trajectories.
  
  So we instead think of the data points as state-action pairs that we pass into $h\_\theta$ to get the probability of selecting said action for a given state. These probabilities then dictates whether we choose the action or not. An alternative way to interpret this as if we imagine the action space for each state as binary then we can think of the "other action" as not impacting the predictor.
  
  The gradient used for updating the parameters associated with the policy uses the log probability of $\pi\_\theta$ which if we expand it, expressing it using $h\_\theta$, (as done on page 4) we can see it's the sum of the log probabilities of selecting (or not selecting) each data point.
  
  By considering the return for each data point constant (each data point has the same loss incurred by predictor model on the validation set) and absorbing the average over batch size into the step size $\beta$ it could be interpreted as an average
  
  ---
  
  **Stability issues**
  
  RL is plagued with stability issues, be it selection of hyper parameters, random seeding etc. It's hard to often pinpoint why results aren't exact but as long as you get something in a similar ball park i'd say thats pretty good going
  
  
  ---
  
  