Exact numeric nth derivatives
I think it's worth noting that the problem with numerical differentiation, fundamentally, is that differentiation is an unbounded operator. In finite-differences, (the more obvious approach), you assume that your data are samples of some, general, function.<p>The problem then, is that that general functions have no (essential) bandlimit [1]. Remember that differentiation acts as a multiplication by a monomial, in the frequency domain [2]. Non-constant polynomials always eventually blow up away from 0, so in differentiating, you're multiplying a function by something that blows up, in the frequency domain. This means that, in the result, higher frequencies are going to dominate over lower frequencies, at a polynomial rate.<p>Let me be clear, the problem with numerical differentiation is not just that rounding errors accumulate, it's that differentiation is fundamentally unstable, and not something you want to apply to real-world data.<p>It depends very much on what your application is, however, I think generally a better approach to AD is to redefine your differentiation, by composing it with a low-pass filter. If designed properly, your low-pass filter will 'go to zero' faster (in the frequency-domain) than any polynomial, thus making this new operator bounded, and hence numerically more stable. It's not a panacea, but it begins to address the fundamental problem.<p>One example of such a filter is Gamma(n+1, n x^2)/Factorial[n], where Gamma is the incomplete gamma function [3].<p>In Python:<p>scipy.special.gammaincc(n+1,n<i>x</i><i>2)
or
mpmath.gammainc(n+1,n</i>x<i></i>2, regularized=True)<p>To see why this is a nice choice, notice item 2 in [4]. This filter is simply the product of exp(- x^2) (the Gaussian) multiplied by the first n-terms of the Taylor series of exp(+ x^2),  (1/ the-Gaussian).  Since this series converges unconditionally everywhere, as n-> +infinity, this filter converges to 1 for  a fixed x (as you increase n), however, since it's still a gaussian times  a polynomial, it always converges to 0 as you increase x, but fix n.<p>This is my area of research, so if anyone's interested I can give more details.<p>[1] <a href="https://en.wikipedia.org/wiki/Band-limit" rel="nofollow">https://en.wikipedia.org/wiki/Band-limit</a>
[2] <a href="https://en.wikipedia.org/wiki/Fourier_transform#Analysis_of_differential_equations" rel="nofollow">https://en.wikipedia.org/wiki/Fourier_transform#Analysis_of_...</a>
[3] <a href="https://en.wikipedia.org/wiki/Incomplete_gamma_function" rel="nofollow">https://en.wikipedia.org/wiki/Incomplete_gamma_function</a>
[4] <a href="https://en.wikipedia.org/wiki/Incomplete_gamma_function#Special_values" rel="nofollow">https://en.wikipedia.org/wiki/Incomplete_gamma_function#Spec...</a>