Hi Zach!



 zacmon:

does pytorch enable the use of arbitrary floating point numbers in its automatic differentiation?


No, pytorch’s autograd framework uses pre-evaluated analytic*
differentiation, not numerical differentiation.
If you can figure out how to write your likelihood function using
pytorch tensor operations, then autograd will differentiate your
function without additional work on your part.
autograd uses subclasses of torch.autograd.Function 1.  These
have a forward() method that calculates the function itself,
and a backward() method that calculates the gradient.  So,
when the pytorch implementers wrote torch.sin(), they called
some math library to compute sin(), they analytically differentiated
sin() to get cos(), and then they called some math library to compute
cos().  (Of course, the math library computes cos() numerically,
but the fact that cos() is the derivative of sin() was determined
analytically.)
If you can’t figure out how to write your likelihood function using
pytorch tensor functions, you will have to write your own backward()
function for it.  (This is perfectly doable, but you’re better off using
the existing pytorch tensor functions, if you can.)
*)  Nothing prevents an implementation from using numerical
differentiation in a backward() function – all backward() needs
to do is return the correct gradient.  The autograd framework neither
knows nor cares how the gradient was computed.  I’m just not aware
of any built-in pytorch function that uses numerical differentiation for
its backward().  So you could write a backward() function that uses
numpy or mpmath or whatever to perform its numerical differentiation,
provided that you repackage the final result you return as a pytorch
tensor.
(Note, pytorch offers torch.autograd.gradcheck() 1 for checking
gradients numerically.)
Best.
K. Frank