Hello Zach!



 zacmon:

Do you think autodiff would outperform hard-coding a function for, say, a 60th derivative which uses high-order chain rule?
…
But is there someone familiar with higher-order chain rule and pytorch?


I don’t really understand what your use case is.
However, you should understand that pytorch is a neural-network
framework, and not a general-purpose automatic-differentiation
package.
Pytorch’s purpose in life is to support building and training neural
networks.  A neural network is trained by (partially) minimizing a
loss function with respect to the network’s parameters.  For a
variety of reasons, this minimization is performed using various
gradient-descent algorithms.  To this end pytorch’s autograd
framework calculates the gradient (first-derivative) of the loss
with respect to the network’s parameters.
autograd is a very useful specialized tool for pytorch’s use case,
but it is not particularly general-purpose.
There is no direct support in autograd for higher-order derivative.
There is no support in autograd for finding the derivative of a function
for whose pieces you haven’t already provided derivatives (as
backward() methods).
Note, to clarify the comment in my previous post about “analytic”
differentiation:
Think of a neural network as a bunch of “simple” functions that
are composed with one another – that is, chained together.
autograd differentiates (finds the gradient of) such a function.
To do so, it relies on the authors of the simple functions to
provide their derivatives, packaged in the backward() method.
One imagines that those authors calculated those derivatives
analytically, but they didn’t have to.
autograd then implements the chain rule numerically to differentiate
the big, composite function, using the derivative of its pieces obtained
from their backward() methods.
If you think this through, you will see that autograd is great for pytorch’s
use case, but probably not very useful for what I think you might want
to do.
Good luck!
K. Frank