Answers.Body: <p>It's not advisable to remove the log term simply because it's monotone since intuitively as a score function the log term in policy gradient theorem (PGT) is not arbitrary but rather a necessary step to ensure proper scaling and direction of the gradients especially for actions with small probabilities. We can have a simple 1-d parametere space example to show this.</p>
  <p>Let's say we have two possible actions <span class="math-container">$a_1,a_2$</span> of a simple bandit problem with the parameterized policy for both actions defined as <span class="math-container">$\pi_{\theta}(a_1)=\sigma(\theta), \pi_{\theta}(a_2)=1-\sigma(\theta)$</span>, where <span class="math-container">$\sigma$</span> is the standard sigmoid function. Also suppose the rewards for the actions are <span class="math-container">$r(a_1)=1, r(a_2)=0$</span>, since it's a bandit without state the return <span class="math-container">$G_t$</span> is simply <span class="math-container">$r(a)$</span> for each action <span class="math-container">$a$</span>. Then it's straightforward to compute gradient as follows: <span class="math-container">$$\nabla_{\theta} J(\theta)=\mathbb{E}_{\pi_{\theta}}[\nabla_{\theta}\log\pi_{\theta}(a)G_t]=\pi_{\theta}(a_1)(1-\sigma(\theta))r(a_1)+\pi_{\theta}(a_2)(-\sigma(\theta))r(a_2)=\sigma(\theta)(1-\sigma(\theta))$$</span> Similarly you can get the gradient for the case of removing the log as <span class="math-container">$$\nabla_{\theta} J'(\theta)=\sigma(\theta)^2(1-\sigma(\theta))$$</span></p>
  <p>Therefore clearly even in 1-d bandit case the gradient without log will become extremely small if <span class="math-container">$\sigma(\theta)$</span>, i.e., <span class="math-container">$\pi_{\theta}(a_1)$</span> is very small, which means the initially very unlikely actions in the to-be-optimized policy cannot get updated efficiently consistent with above intuition due to losing the proper scaling effect of the log. And in the usual multi-dimensional policy for MDP cases both the magnitude and direction of the gradient will be similarly negatively impacted.</p>
  <p>Finally with the common log gradient trick the existing form in PGT can be simply transformed to a quotient where the numerator is just your gradient of the policy without log and the denominator is the same policy. From the balance need of exploitation-exploration, this denominator is required for exploration since if the probability of taking certain action in a state is small, then the algorithm would update the parameters so that the probability of taking that action can increase. The other term <span class="math-container">$G_t \approx Q_t(s_t,a_t)$</span> reflects that if an action value is great then the algorithm intends to update the parameters so that the probability of taking that action is enhanced which is exploitation. Therefore if you remove the denominator policy then equivalently you're <em>not exploring</em> enough.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-29T00:14:04.710Z
Answers.LastActivityDate: 2024-07-29T03:35:59.497Z
Answers.LastEditDate: 2024-07-29T03:35:59.497Z
Answers.UserDisplayName: cinch
Body: <p>If I understand correctly, the goal of vanilla policy gradients is maximizing <span class="math-container">$E[r(s_t,a_t);\pi_\theta]$</span>; in deriving the gradient of this function as a clearer function on <span class="math-container">$\theta$</span>, we get <span class="math-container">$\sum_{t=0}^{T-1}\nabla_\theta \log\pi_\theta(a_t|s_t)G_t$</span>: a sort of weighted sum of the gradients of log probabilities; we treat <span class="math-container">$\log\pi_\theta(a_t|s_t)G_t$</span> as basically equivalent to <span class="math-container">$E[r(s_t,a_t);\pi_\theta]$</span> as their gradients are essentially equal: maximizing one maximizes the other. In PPO, we arbitrarily replace this with <span class="math-container">$E[\frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{prev}}(a_t|s_t)}A_t]$</span> + clipping. It seems that as long as higher rewards are associated with higher probabilities, anything in the form <span class="math-container">$E[\pi_\theta A_t]$</span> is fine to maximize and will also maximize <span class="math-container">$E[r(s_t,a_t);\pi_\theta]$</span>. Is this correct? Can I simply remove the log from the equation, for example?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-07-28T21:14:24.937Z
Comments.Text: since $\log$ is a strictly monotonic function, you _probably_ can remove the this, but 1) it is not necessarily theoretically motivated, 2) it will almost certainly lead to instabilities. That is why we generally talk about log-likelihoods in stats/ML, as it is much easier to optimiser from a numerical stability point of view.
Comments.UserDisplayName: David
Tags: <reinforcement-learning><policy-gradients>
Title: Can we simply remove the log term for loss in policy gradient methods?
UserDisplayName: User
ThreadText: # Can we simply remove the log term for loss in policy gradient methods?
  
  (Asked by: username_0 on 2024-07-28)
  
  If I understand correctly, the goal of vanilla policy gradients is maximizing $E[r(s\_t,a\_t);\pi\_\theta]$; in deriving the gradient of this function as a clearer function on $\theta$, we get $\sum\_{t=0}^{T-1}\nabla\_\theta \log\pi\_\theta(a\_t|s\_t)G\_t$: a sort of weighted sum of the gradients of log probabilities; we treat $\log\pi\_\theta(a\_t|s\_t)G\_t$ as basically equivalent to $E[r(s\_t,a\_t);\pi\_\theta]$ as their gradients are essentially equal: maximizing one maximizes the other. In PPO, we arbitrarily replace this with $E[\frac{\pi\_\theta(a\_t|s\_t)}{\pi\_{\theta\_{prev}}(a\_t|s\_t)}A\_t]$ + clipping. It seems that as long as higher rewards are associated with higher probabilities, anything in the form $E[\pi\_\theta A\_t]$ is fine to maximize and will also maximize $E[r(s\_t,a\_t);\pi\_\theta]$. Is this correct? Can I simply remove the log from the equation, for example?
  
  **Tags:** reinforcement-learning, policy-gradients
  
  > **Comments:**
  > **username_1**: since $\log$ is a strictly monotonic function, you _probably_ can remove the this, but 1) it is not necessarily theoretically motivated, 2) it will almost certainly lead to instabilities. That is why we generally talk about log-likelihoods in stats/ML, as it is much easier to optimiser from a numerical stability point of view.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2024-07-29. Score: 7
  
  It's not advisable to remove the log term simply because it's monotone since intuitively as a score function the log term in policy gradient theorem (PGT) is not arbitrary but rather a necessary step to ensure proper scaling and direction of the gradients especially for actions with small probabilities. We can have a simple 1-d parametere space example to show this.
  
  Let's say we have two possible actions $a\_1,a\_2$ of a simple bandit problem with the parameterized policy for both actions defined as $\pi\_{\theta}(a\_1)=\sigma(\theta), \pi\_{\theta}(a\_2)=1-\sigma(\theta)$, where $\sigma$ is the standard sigmoid function. Also suppose the rewards for the actions are $r(a\_1)=1, r(a\_2)=0$, since it's a bandit without state the return $G\_t$ is simply $r(a)$ for each action $a$. Then it's straightforward to compute gradient as follows: $$\nabla\_{\theta} J(\theta)=\mathbb{E}\_{\pi\_{\theta}}[\nabla\_{\theta}\log\pi\_{\theta}(a)G\_t]=\pi\_{\theta}(a\_1)(1-\sigma(\theta))r(a\_1)+\pi\_{\theta}(a\_2)(-\sigma(\theta))r(a\_2)=\sigma(\theta)(1-\sigma(\theta))$$ Similarly you can get the gradient for the case of removing the log as $$\nabla\_{\theta} J'(\theta)=\sigma(\theta)^2(1-\sigma(\theta))$$
  
  Therefore clearly even in 1-d bandit case the gradient without log will become extremely small if $\sigma(\theta)$, i.e., $\pi\_{\theta}(a\_1)$ is very small, which means the initially very unlikely actions in the to-be-optimized policy cannot get updated efficiently consistent with above intuition due to losing the proper scaling effect of the log. And in the usual multi-dimensional policy for MDP cases both the magnitude and direction of the gradient will be similarly negatively impacted.
  
  Finally with the common log gradient trick the existing form in PGT can be simply transformed to a quotient where the numerator is just your gradient of the policy without log and the denominator is the same policy. From the balance need of exploitation-exploration, this denominator is required for exploration since if the probability of taking certain action in a state is small, then the algorithm would update the parameters so that the probability of taking that action can increase. The other term $G\_t \approx Q\_t(s\_t,a\_t)$ reflects that if an action value is great then the algorithm intends to update the parameters so that the probability of taking that action is enhanced which is exploitation. Therefore if you remove the denominator policy then equivalently you're *not exploring* enough.
  
  
  ---
  
  