Answers.Body: <p>You might want to try substituting the exponentiation with a piecewise-defined function that uses a numerical approximation that is more numerically stable for low values of the exponent, such as using <span class="math-container">$e^x \approx 1 + x$</span> (First-order Maclaurin series) when x is under a certain threshold</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-02T11:05:10.763Z
Answers.Text: That's actually a very nice trick. I'll accept the answer after some testing.
Answers.UserDisplayName: Antonis Karvelas
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-02T10:59:58.357Z
Answers.LastActivityDate: 2022-06-02T10:59:58.357Z
Answers.UserDisplayName: kmf
Body: <p>I'm implement PPO for a very specific problem, and it seems to be working somewhat, but after a few epochs, I always get something like this:</p>
  <pre><code>/home/antoni4040/anaconda3/lib/python3.9/site-packages/torch/autograd/__init__.py:173: UserWarning: Error detected in MulBackward0. Traceback of forward call that caused the error:
    File &quot;/home/antoni4040/Documents/Toulouse/Toulouse-Optimizer/toulousePPO.py&quot;, line 439, in &lt;module&gt;
      agent.train(120)
    File &quot;/home/antoni4040/Documents/Toulouse/Toulouse-Optimizer/toulousePPO.py&quot;, line 298, in train
      self.learn(ep=epoch, lr=learning_rate, clip=clip)
    File &quot;/home/antoni4040/Documents/Toulouse/Toulouse-Optimizer/toulousePPO.py&quot;, line 169, in learn
      clipped_ratio * sampled_normalized_advantage )
   (Triggered internally at  /opt/conda/conda-bld/pytorch_1646756402876/work/torch/csrc/autograd/python_anomaly_mode.cpp:104.)
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
  Traceback (most recent call last):
    File &quot;/home/antoni4040/Documents/Toulouse/Toulouse-Optimizer/toulousePPO.py&quot;, line 439, in &lt;module&gt;
      agent.train(120)
    File &quot;/home/antoni4040/Documents/Toulouse/Toulouse-Optimizer/toulousePPO.py&quot;, line 298, in train
      self.learn(ep=epoch, lr=learning_rate, clip=clip) 
    File &quot;/home/antoni4040/Documents/Toulouse/Toulouse-Optimizer/toulousePPO.py&quot;, line 197, in learn
      total_loss.backward()
    File &quot;/home/antoni4040/anaconda3/lib/python3.9/site-packages/torch/_tensor.py&quot;, line 363, in backward
      torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
    File &quot;/home/antoni4040/anaconda3/lib/python3.9/site-packages/torch/autograd/__init__.py&quot;, line 173, in backward
      Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
  RuntimeError: Function 'MulBackward0' returned nan values in its 0th output.
  </code></pre>
  <p>I started printing the losses and it turns out that the policy loss becomes nan.
  I do something like this for the policy loss:</p>
  <pre><code>dist, value = self.model(batchStates)
  new_probs = dist.log_prob(batchActions)
  ratio =  torch.exp(new_probs - batchOldProbs)
  
  clipped_ratio = ratio.clamp(min=1.0 - clip, max=1.0 + clip)
  policy_loss = torch.min(ratio * sampled_normalized_advantage,
                     clipped_ratio * sampled_normalized_advantage )
  policy_loss = policy_loss.mean()
  </code></pre>
  <p>Should I add a small value to the ratio (1.0e-6 or something), or is it something else?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-02T10:31:13.310Z
Comments.Text: Have you considered taking an approach that does not require exponentiation?  Without seeing numbers, that is most likely what is vanishing (as you seem to realize from your question).  Could you simply continue to operate on the log values, which will give you a great deal more "room"?
Comments.UserDisplayName: David Hoelzer
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-02T10:45:20.697Z
Comments.Text: That's how PPO works tho. That's the math behind it.
Comments.UserDisplayName: Antonis Karvelas
Tags: <proximal-policy-optimization><probability-distribution><loss>
Title: PPO: policy loss becomes nan
UserDisplayName: Antonis Karvelas
ThreadText: # PPO: policy loss becomes nan
  
  (Asked by: username_0 on 2022-06-02)
  
  I'm implement PPO for a very specific problem, and it seems to be working somewhat, but after a few epochs, I always get something like this:
  /home/antoni4040/anaconda3/lib/python3.9/site-packages/torch/autograd/__init__.py:173: UserWarning: Error detected in MulBackward0. Traceback of forward call that caused the error:
    File "/home/antoni4040/Documents/Toulouse/Toulouse-Optimizer/toulousePPO.py", line 439, in 
      agent.train(120)
    File "/home/antoni4040/Documents/Toulouse/Toulouse-Optimizer/toulousePPO.py", line 298, in train
      self.learn(ep=epoch, lr=learning_rate, clip=clip)
    File "/home/antoni4040/Documents/Toulouse/Toulouse-Optimizer/toulousePPO.py", line 169, in learn
      clipped_ratio * sampled_normalized_advantage )
   (Triggered internally at  /opt/conda/conda-bld/pytorch_1646756402876/work/torch/csrc/autograd/python_anomaly_mode.cpp:104.)
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
  Traceback (most recent call last):
    File "/home/antoni4040/Documents/Toulouse/Toulouse-Optimizer/toulousePPO.py", line 439, in 
      agent.train(120)
    File "/home/antoni4040/Documents/Toulouse/Toulouse-Optimizer/toulousePPO.py", line 298, in train
      self.learn(ep=epoch, lr=learning_rate, clip=clip) 
    File "/home/antoni4040/Documents/Toulouse/Toulouse-Optimizer/toulousePPO.py", line 197, in learn
      total_loss.backward()
    File "/home/antoni4040/anaconda3/lib/python3.9/site-packages/torch/_tensor.py", line 363, in backward
      torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
    File "/home/antoni4040/anaconda3/lib/python3.9/site-packages/torch/autograd/__init__.py", line 173, in backward
      Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
  RuntimeError: Function 'MulBackward0' returned nan values in its 0th output.
  
  I started printing the losses and it turns out that the policy loss becomes nan.
  I do something like this for the policy loss:
  dist, value = self.model(batchStates)
  new_probs = dist.log_prob(batchActions)
  ratio =  torch.exp(new_probs - batchOldProbs)
  
  clipped_ratio = ratio.clamp(min=1.0 - clip, max=1.0 + clip)
  policy_loss = torch.min(ratio * sampled_normalized_advantage,
                     clipped_ratio * sampled_normalized_advantage )
  policy_loss = policy_loss.mean()
  
  Should I add a small value to the ratio (1.0e-6 or something), or is it something else?
  
  **Tags:** proximal-policy-optimization, probability-distribution, loss
  > **Comments:**
  > **username_1**: Have you considered taking an approach that does not require exponentiation?  Without seeing numbers, that is most likely what is vanishing (as you seem to realize from your question).  Could you simply continue to operate on the log values, which will give you a great deal more "room"?  
  > **username_0**: That's how PPO works tho. That's the math behind it.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-06-02. Score: 3 | Accepted answer
  
  You might want to try substituting the exponentiation with a piecewise-defined function that uses a numerical approximation that is more numerically stable for low values of the exponent, such as using $e^x \approx 1 + x$ (First-order Maclaurin series) when x is under a certain threshold
  
  > **Comments:**
  > **username_0**: That's actually a very nice trick. I'll accept the answer after some testing.  
  
  ---
  
  