If your backward function does not have the once_differentiable decorator and does not create a proper graph in the backward function, then all higher order derivatives will be wrong. You have helper functions here 9 and here 12 if you want to check your first and second order derivatives implementation with finite differences.