Answers.Body: <p><em>&quot;So the loss is always 0&quot;</em> doesn't mean that there is no learning... indeed, it's called &quot;policy gradient&quot;, not &quot;policy loss&quot;, thus you have to consider the gradient of the logarithm evaluated at 1 (since <span class="math-container">$\pi(a|s) = 1)$</span>)</p>
  <p>And indeed, <span class="math-container">$\nabla \ln \pi(a|s)$</span> when <span class="math-container">$\pi(a|s) = 1$</span> is <span class="math-container">$1$</span>, thus learning can definitely still happen (though it might be slow)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-14T13:15:14.190Z
Answers.LastActivityDate: 2024-01-14T13:15:14.190Z
Answers.UserDisplayName: Alberto
Body: <p>The loss of policy gradient:</p>
  <p><span class="math-container">$$
  \nabla_{\theta}J(\theta)=\mathbb{E}_{\tau\sim\pi\theta}\left[\sum_{t=0}^{T}(R_t-b_t)\nabla_{\theta}\log\pi_\theta(a_t|s_t)\right]
  $$</span></p>
  <p>where <span class="math-container">$\pi_\theta(a_t|s_t)$</span> is the probability of the model <span class="math-container">$\theta$</span> predicting the action <span class="math-container">$a_t$</span> in the state <span class="math-container">$s_t$</span>.</p>
  <p>Howerver, if the model always predicts a certain incorrect action with a probability of 1, then it necessarily follows that <span class="math-container">$\log\pi_\theta(a_t|s_t)=0$</span>. So the loss is alwasy 0.</p>
  <p>I discovered this issue because I found that my model always overfits by outputting any action with a probability of 1, and there's a high chance it's an incorrect action. At this point, the loss is 0, and convergence stops. What could be the problem?</p>
  
Tags: <reinforcement-learning><policy-gradients>
Title: In policy gradient algorithms, if the model always predicts any action with a probability of 1, will the gradient always be 0?
UserDisplayName: Devymex
ThreadText: # In policy gradient algorithms, if the model always predicts any action with a probability of 1, will the gradient always be 0?
  
  (Asked by: username_0 on 2024-01-12)
  
  The loss of policy gradient:
  
  $$
  \nabla\_{\theta}J(\theta)=\mathbb{E}\_{\tau\sim\pi\theta}\left[\sum\_{t=0}^{T}(R\_t-b\_t)\nabla\_{\theta}\log\pi\_\theta(a\_t|s\_t)\right]
  $$
  
  where $\pi\_\theta(a\_t|s\_t)$ is the probability of the model $\theta$ predicting the action $a\_t$ in the state $s\_t$.
  
  Howerver, if the model always predicts a certain incorrect action with a probability of 1, then it necessarily follows that $\log\pi\_\theta(a\_t|s\_t)=0$. So the loss is alwasy 0.
  
  I discovered this issue because I found that my model always overfits by outputting any action with a probability of 1, and there's a high chance it's an incorrect action. At this point, the loss is 0, and convergence stops. What could be the problem?
  
  **Tags:** reinforcement-learning, policy-gradients
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-01-14. Score: 2 | Accepted answer
  
  *"So the loss is always 0"* doesn't mean that there is no learning... indeed, it's called "policy gradient", not "policy loss", thus you have to consider the gradient of the logarithm evaluated at 1 (since $\pi(a|s) = 1)$)
  
  And indeed, $\nabla \ln \pi(a|s)$ when $\pi(a|s) = 1$ is $1$, thus learning can definitely still happen (though it might be slow)
  
  
  ---
  
  