Answers.Body: <p>Think of a surface where z-axis is <span class="math-container">$U$</span>, x and y axis are <span class="math-container">$\theta_{1}$</span> and <span class="math-container">$\theta_{2}$</span> accordingly. Since you are following the gradient direction with respect to <span class="math-container">$\theta$</span> vector, that means you are moving at the direction that increases the <span class="math-container">$U$</span>. If <span class="math-container">$R(\tau)$</span> is positive, you are moving towards the uphill direction and vice-versa. More formally you will say the following:   </p>
  
  <p>In the policy gradient algorithm, our update step is:</p>
  
  <p><span class="math-container">$ \theta_{new} = \theta_{old} + \alpha \nabla_{\theta}U(\theta) $</span></p>
  
  <p>So, if we select a very bad trajectory the sum of all rewards (<span class="math-container">$R(\tau)$</span>) will be negative and the following update will shift the <span class="math-container">$\theta_{new}$</span> vector away from <span class="math-container">$\nabla_{\theta}U(\theta)$</span> vector.</p>
  
  <p>If we select a good trajectory means <span class="math-container">$R(\tau)$</span> is positive, the update will shift <span class="math-container">$\theta_{new}$</span> vector towards the <span class="math-container">$\nabla_{\theta}U(\theta)$</span> vector. So, it will increase the probability of selecting path with positive <span class="math-container">$R$</span> </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-12T10:59:02.433Z
Answers.Text: Hey thanks for your answer. The reward, $R(\tau)$ is a scalar quantity and the grad log probability of a trajectory is negative and the grad log probability of the trajectory returns the gradient of the weights in the neural network rite  ?
Answers.UserDisplayName: calveeen
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-12T11:02:33.463Z
Answers.Text: yes, $R(\tau)$ is a scalar quantity, but the later part you say can not able to understand what you mean by that.
Answers.UserDisplayName: Swakshar Deb
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-12T11:05:42.690Z
Answers.Text: the grad log probability of a trajectory i meant refers to the term $\nabla_{\theta}log P(\tau^{(i)}, \theta)$. so the scalar quantity of the reward is multiplied by this vector to give the gradient of the Utility function w.r.t $\theta$ rite ?
Answers.UserDisplayName: calveeen
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-12T11:10:11.657Z
Answers.Text: yes. reward$(R(\tau)$ is scalar and you are multiplying the scalar$(R)$ with the vector $\nabla_{\theta}logP(\tau^{i},\theta)$.
Answers.UserDisplayName: Swakshar Deb
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-12T11:11:59.287Z
Answers.Text: how would a negative reward shift $\theta_{new}$ to not favour that bad reward trajectory ? because whether or not a positive scalar or negative scalar is multiplied by the weights, $\theta$ will be updated to move in the direction of the gradient, which means $\theta$ the new $\theta$ will overall increase the utility received but the equation does not show how it affects the log probability of a good or bad trajectory ?
Answers.UserDisplayName: calveeen
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-12T11:14:28.407Z
Answers.Text: based on the overall increase in utility we can make the conclusion that the probabilities of trajectories with higher rewards are increased ?
Answers.UserDisplayName: calveeen
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-12T11:19:26.493Z
Answers.Text: When the agent gets neagative $R(\tau)$, it will just decrease the probability of selecting those actions by updating $\theta$.
Answers.UserDisplayName: Swakshar Deb
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-12T11:22:55.490Z
Answers.Text: and that is based on the update rule $\theta_{new} = \theta_{old} + \alpha \nabla_{\theta}U(\theta)$ ?
Answers.UserDisplayName: calveeen
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-12T11:26:33.403Z
Answers.Text: yes, it's based on this update.
Answers.UserDisplayName: Swakshar Deb
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-12T10:14:59.933Z
Answers.LastActivityDate: 2020-05-12T10:22:37.547Z
Answers.LastEditDate: 2020-05-12T10:22:37.547Z
Answers.UserDisplayName: Swakshar Deb
Answers.Body: <p>The grad log probability of the trajectory parameterised by <span class="math-container">$\theta$</span> tells us the direction <span class="math-container">$\theta$</span> should move to increase the probability of that trajectory <span class="math-container">$P(\tau;\theta)$</span> the most. </p>
  
  <p>If the reward is positive, <span class="math-container">$\nabla U(\theta)$</span> tells us how much we want to increase/decrease the probability of that path <span class="math-container">$\tau$</span>. The scalar quantity <span class="math-container">$R(\tau)$</span> determines the magnitude and direction of shift. If <span class="math-container">$R(\tau)$</span> is positive, and <span class="math-container">$\theta$</span> is updated based on equation <span class="math-container">$\theta_{new}$</span> = <span class="math-container">$\theta_{old} + \alpha\nabla_{\theta}U(\theta)$</span>, then <span class="math-container">$\theta$</span> will move in the direction of it's steepest increase, leading to an increase in probability of <span class="math-container">$\tau$</span>. If <span class="math-container">$R(\tau)$</span> is negative, then <span class="math-container">$\theta$</span> moves in the direction of steepest decrease, leading to a decrease in probability of <span class="math-container">$\tau$</span>. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-13T03:06:06.997Z
Answers.LastActivityDate: 2020-05-13T03:06:06.997Z
Answers.UserDisplayName: calveeen
Body: <p>Pieter Abbeel in his <a href="https://www.youtube.com/watch?v=S_gwYj1Q-44&amp;list=PLAdk-EyP1ND8MqJEJnSvaoUShrAWYe51U&amp;index=4" rel="nofollow noreferrer">deep rl bootcamp policy gradient lecture</a> derived the gradient of the utility function with respect to <span class="math-container">$\theta$</span> as <span class="math-container">$\nabla U(\theta) \approx \hat{g} = 1/m\sum_{i=1}^m  \nabla_\theta logP(\tau^{(i)}; \theta)R(\tau^{(i)})$</span>, where <span class="math-container">$m$</span> is the number of rollouts, and <span class="math-container">$\tau$</span> represents the trajectory of <span class="math-container">$s_0,u_0, ..., s_H, u_H$</span> state action sequences.</p>
  
  <p>He also explains that the gradient increases the log probabilities of trajectories that have positive reward and decreases the log probabilities of trajectories with negative reward, as seen in the picture. From the equation, however, I don't see how the gradient tries to increase the probabilities of the path with positive R?</p>
  
  <p>From the equation, what I understand is that we would want to update <span class="math-container">$\theta$</span> in a way that moves in the direction of <span class="math-container">$\nabla U(\theta)$</span> so that the overall utility is maximised, and this entails computing the gradient log probability of a trajectory. </p>
  
  <p>Also, why is <span class="math-container">$\theta$</span> omitted in <span class="math-container">$R(\tau^{(i)})$</span>, since <span class="math-container">$\tau$</span> depends on the policy which is dependent on <span class="math-container">$\theta$</span> ?</p>
  
  <p><a href="https://i.sstatic.net/Yfwre.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Yfwre.png" alt="enter image description here"></a></p>
  
Tags: <reinforcement-learning><policy-gradients>
Title: How does the gradient increase the probabilities of the path with a positive reward in policy gradient?
UserDisplayName: calveeen
ThreadText: # How does the gradient increase the probabilities of the path with a positive reward in policy gradient?
  
  (Asked by: username_0 on 2020-05-12)
  
  Pieter Abbeel in his [deep rl bootcamp policy gradient lecture](https://www.youtube.com/watch?v=S_gwYj1Q-44&list=PLAdk-EyP1ND8MqJEJnSvaoUShrAWYe51U&index=4) derived the gradient of the utility function with respect to $\theta$ as $\nabla U(\theta) \approx \hat{g} = 1/m\sum\_{i=1}^m \nabla\_\theta logP(\tau^{(i)}; \theta)R(\tau^{(i)})$, where $m$ is the number of rollouts, and $\tau$ represents the trajectory of $s\_0,u\_0, ..., s\_H, u\_H$ state action sequences.
  
  He also explains that the gradient increases the log probabilities of trajectories that have positive reward and decreases the log probabilities of trajectories with negative reward, as seen in the picture. From the equation, however, I don't see how the gradient tries to increase the probabilities of the path with positive R?
  
  From the equation, what I understand is that we would want to update $\theta$ in a way that moves in the direction of $\nabla U(\theta)$ so that the overall utility is maximised, and this entails computing the gradient log probability of a trajectory.
  
  Also, why is $\theta$ omitted in $R(\tau^{(i)})$, since $\tau$ depends on the policy which is dependent on $\theta$ ?
  
  [![enter image description here](https://i.sstatic.net/Yfwre.png)](https://i.sstatic.net/Yfwre.png)
  
  **Tags:** reinforcement-learning, policy-gradients
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2020-05-12. Score: 0 | Accepted answer
  
  Think of a surface where z-axis is $U$, x and y axis are $\theta\_{1}$ and $\theta\_{2}$ accordingly. Since you are following the gradient direction with respect to $\theta$ vector, that means you are moving at the direction that increases the $U$. If $R(\tau)$ is positive, you are moving towards the uphill direction and vice-versa. More formally you will say the following:
  
  In the policy gradient algorithm, our update step is:
  
  $ \theta\_{new} = \theta\_{old} + \alpha \nabla\_{\theta}U(\theta) $
  
  So, if we select a very bad trajectory the sum of all rewards ($R(\tau)$) will be negative and the following update will shift the $\theta\_{new}$ vector away from $\nabla\_{\theta}U(\theta)$ vector.
  
  If we select a good trajectory means $R(\tau)$ is positive, the update will shift $\theta\_{new}$ vector towards the $\nabla\_{\theta}U(\theta)$ vector. So, it will increase the probability of selecting path with positive $R$
  
  > **Comments:**
  > **username_0**: Hey thanks for your answer. The reward, $R(\tau)$ is a scalar quantity and the grad log probability of a trajectory is negative and the grad log probability of the trajectory returns the gradient of the weights in the neural network rite  ?  
  > **username_1**: yes, $R(\tau)$ is a scalar quantity, but the later part you say can not able to understand what you mean by that.  
  > **username_0**: the grad log probability of a trajectory i meant refers to the term $\nabla_{\theta}log P(\tau^{(i)}, \theta)$. so the scalar quantity of the reward is multiplied by this vector to give the gradient of the Utility function w.r.t $\theta$ rite ?  
  > **username_1**: yes. reward$(R(\tau)$ is scalar and you are multiplying the scalar$(R)$ with the vector $\nabla_{\theta}logP(\tau^{i},\theta)$.  
  > **username_0**: how would a negative reward shift $\theta_{new}$ to not favour that bad reward trajectory ? because whether or not a positive scalar or negative scalar is multiplied by the weights, $\theta$ will be updated to move in the direction of the gradient, which means $\theta$ the new $\theta$ will overall increase the utility received but the equation does not show how it affects the log probability of a good or bad trajectory ?  
  > **username_0**: based on the overall increase in utility we can make the conclusion that the probabilities of trajectories with higher rewards are increased ?  
  > **username_1**: When the agent gets neagative $R(\tau)$, it will just decrease the probability of selecting those actions by updating $\theta$.  
  > **username_0**: and that is based on the update rule $\theta_{new} = \theta_{old} + \alpha \nabla_{\theta}U(\theta)$ ?  
  > **username_1**: yes, it's based on this update.  
  
  ---
  
  ### Answer by username_0 on 2020-05-13. Score: 0
  
  The grad log probability of the trajectory parameterised by $\theta$ tells us the direction $\theta$ should move to increase the probability of that trajectory $P(\tau;\theta)$ the most.
  
  If the reward is positive, $\nabla U(\theta)$ tells us how much we want to increase/decrease the probability of that path $\tau$. The scalar quantity $R(\tau)$ determines the magnitude and direction of shift. If $R(\tau)$ is positive, and $\theta$ is updated based on equation $\theta\_{new}$ = $\theta\_{old} + \alpha\nabla\_{\theta}U(\theta)$, then $\theta$ will move in the direction of it's steepest increase, leading to an increase in probability of $\tau$. If $R(\tau)$ is negative, then $\theta$ moves in the direction of steepest decrease, leading to a decrease in probability of $\tau$.
  
  
  ---
  
  