Hi Steve!



 stevethesteve:

This changes things a lot: If setting the breakpoint too low, the error in the gradient can be 7 orders of magnitude bigger than the error in the log-softplus output.


This may be related to some weirdness in pytorch’s built-in
torch.nn.functional.softplus().  Here is the relevant thread:

  
    
    
    Gradients for torch.nn.functional.softplus() are numerically unstable 
  
  
    Hello Forum! 
It appears that backward() for softplus() has a numerical issue 
(that does not appear to be caused directly by a related forward() 
issue) for large negative arguments. 
A straightforward (but not completely naive) implementation of 
softplus() – that has no explicit backward() –  seems to work 
fine with autograd. 
Here is an illustrative test script: 
import torch
print (torch.__version__)

def test_softplus (x):   # use log-sum-exp trick and loq1p
    return torch.where (x > 0,…
  


Best.
K. Frank