Answers.Body: <p>Bear in mind <span class="math-container">$\nabla_{\theta} J$</span> is used to update parameters <span class="math-container">$\theta$</span> in each step and <span class="math-container">$r(\tau)$</span> is the total reward of a sampling trajectory <span class="math-container">$\tau$</span> where we can rewrite the policy gradient equation as <span class="math-container">$$\nabla_{\theta} J(\theta) \approx r(\tau) \sum_{t\ge 0} \nabla_{\theta} \log \pi_{\theta}\left(a_{t} \mid s_{t}\right)$$</span> If <span class="math-container">$r(\tau)$</span> is positively great then the above policy gradient ascent algorithm intuitively intends to update the parameters in a direction so that each term <span class="math-container">$\log \pi_{\theta}\left(a_{t} \mid s_{t}\right)$</span> is pushed up. And since the <span class="math-container">$\log$</span> function is monotone, each term <span class="math-container">$\pi_{\theta}\left(a_{t} \mid s_{t}\right)$</span> is pushed up too which is nothing but exploitation viewed from the thesis of exploration vs exploitation of RL. This is why your reference claims it might seem simplistic to say that if a trajectory is good then all its actions were good.</p>
  <p>If this cannot convince you, let's use a simple grid world example where an agent starts at the left corner and aims to reach a goal at the right corner. Suppose there are only two possible actions in any state <span class="math-container">$s$</span>: <span class="math-container">$UP$</span> and <span class="math-container">$RIGHT$</span>, and initial probabilities of taking these actions under the policy <span class="math-container">$\pi_{\theta}$</span> are <span class="math-container">$\pi_{\theta}(UP∣s)=0.5,\pi_{\theta}(RIGHT∣s)=0.5$</span>. The reward <span class="math-container">$r(τ)$</span> for a trajectory <span class="math-container">$τ$</span> is <span class="math-container">$+10$</span> if it reaches the goal using the action <span class="math-container">$RIGHT$</span>. For simplicity, let's say the softmax parameterization for actins in any state <span class="math-container">$s$</span> is: <span class="math-container">$π_θ(UP∣s)=\frac{e^{θ_1}}{e^{θ_1}+e^{θ_2}}, π_θ(RIGHT∣s)=\frac{e^{θ_2}}{e^{θ_1}+e^{θ_2}}$</span>. Apparently, only by choosing action <span class="math-container">$RIGHT$</span> at each state <span class="math-container">$s$</span> the parameters <span class="math-container">$\theta$</span> will be pushed up in the direction of <span class="math-container">$θ_2$</span> per above policy gradient algorithm and thus <span class="math-container">$π_θ(RIGHT∣s)=\frac{e^{θ_2}}{e^{θ_1}+e^{θ_2}}$</span> is pushed up as well.
  ​</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-01T08:49:59.263Z
Answers.LastActivityDate: 2024-09-02T00:10:47.767Z
Answers.LastEditDate: 2024-09-02T00:10:47.767Z
Answers.UserDisplayName: cinch
Body: <p>Please see slide 78 in <a href="https://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture14.pdf" rel="nofollow noreferrer">https://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture14.pdf</a></p>
  <blockquote>
  <p><span class="math-container">$$\nabla_{\theta} J\left(\theta\right) \approx\sum_{t\ge 0}r(\tau) \nabla_{\theta} \log \pi_{\theta}\left(a_{t} \mid s_{t}\right)$$</span></p>
  </blockquote>
  <blockquote>
  <p>Interpretation:</p>
  </blockquote>
  <blockquote>
  <p>-If <span class="math-container">$r(\tau)$</span> is high, push up the probabilities of the actions seen.</p>
  </blockquote>
  <blockquote>
  <p>-If <span class="math-container">$r(\tau)$</span> is low, push down the probabilities of the actions seen.</p>
  </blockquote>
  <p>Question: It is not clear to me how this interpretation is done based on the mathematical expression. I will be happy for a detailed explanation.</p>
  
Tags: <reinforcement-learning><policy-gradients>
Title: Interpretation of changing action probability based on policy gradient expression
UserDisplayName: DSPinfinity
ThreadText: # Interpretation of changing action probability based on policy gradient expression
  
  (Asked by: username_0 on 2024-09-01)
  
  Please see slide 78 in <https://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture14.pdf>
  
  > $$\nabla\_{\theta} J\left(\theta\right) \approx\sum\_{t\ge 0}r(\tau) \nabla\_{\theta} \log \pi\_{\theta}\left(a\_{t} \mid s\_{t}\right)$$
  
  > Interpretation:
  
  > -If $r(\tau)$ is high, push up the probabilities of the actions seen.
  
  > -If $r(\tau)$ is low, push down the probabilities of the actions seen.
  
  Question: It is not clear to me how this interpretation is done based on the mathematical expression. I will be happy for a detailed explanation.
  
  **Tags:** reinforcement-learning, policy-gradients
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-09-01. Score: -1
  
  Bear in mind $\nabla\_{\theta} J$ is used to update parameters $\theta$ in each step and $r(\tau)$ is the total reward of a sampling trajectory $\tau$ where we can rewrite the policy gradient equation as $$\nabla\_{\theta} J(\theta) \approx r(\tau) \sum\_{t\ge 0} \nabla\_{\theta} \log \pi\_{\theta}\left(a\_{t} \mid s\_{t}\right)$$ If $r(\tau)$ is positively great then the above policy gradient ascent algorithm intuitively intends to update the parameters in a direction so that each term $\log \pi\_{\theta}\left(a\_{t} \mid s\_{t}\right)$ is pushed up. And since the $\log$ function is monotone, each term $\pi\_{\theta}\left(a\_{t} \mid s\_{t}\right)$ is pushed up too which is nothing but exploitation viewed from the thesis of exploration vs exploitation of RL. This is why your reference claims it might seem simplistic to say that if a trajectory is good then all its actions were good.
  
  If this cannot convince you, let's use a simple grid world example where an agent starts at the left corner and aims to reach a goal at the right corner. Suppose there are only two possible actions in any state $s$: $UP$ and $RIGHT$, and initial probabilities of taking these actions under the policy $\pi\_{\theta}$ are $\pi\_{\theta}(UP∣s)=0.5,\pi\_{\theta}(RIGHT∣s)=0.5$. The reward $r(τ)$ for a trajectory $τ$ is $+10$ if it reaches the goal using the action $RIGHT$. For simplicity, let's say the softmax parameterization for actins in any state $s$ is: $π\_θ(UP∣s)=\frac{e^{θ\_1}}{e^{θ\_1}+e^{θ\_2}}, π\_θ(RIGHT∣s)=\frac{e^{θ\_2}}{e^{θ\_1}+e^{θ\_2}}$. Apparently, only by choosing action $RIGHT$ at each state $s$ the parameters $\theta$ will be pushed up in the direction of $θ\_2$ per above policy gradient algorithm and thus $π\_θ(RIGHT∣s)=\frac{e^{θ\_2}}{e^{θ\_1}+e^{θ\_2}}$ is pushed up as well.
  ​
  
  
  ---
  
  