Solved by KFrank in post #2 
              
              
                Hi Dmitriy! 

You are perfectly correct that the derivative of softplus 
is sigmoid, and that calling backward() should give 
you this derivative as the gradient with respect to x. 
Your problem is that, for comparison, you are not calculating 
sigmoid (x) (but, rather, sigmoid (out)). 
Here is a (p…