Hello,
I’m using PyTorch as an audodiff tool to compute the first and second derivatives of a cost function to be used in a (non-deep-learning) optimization tool (ipopt). The cost function depends about 10 parameters. I am using the new torch.autograd.functional.jacobian and torch.autograd.functional.hessian added to PyTorch 1.5. The Jacobian and Hessian get called several times (about 100) with different input parameters until the function is minimized.
To be more specific, I am computing the Jacobians and Hessians as:
def jac(f, x):
    x_tensor = torch.tensor(x, requires_grad=True, dtype=torch.float64)   # Convert input variable to torch tensor
    jac = torch.autograd.functional.jacobian(f, x_tensor)
    return jac.numpy()

def hess(f, x):
    x_tensor = torch.tensor(x, requires_grad=True, dtype=torch.float64)   # Convert input variable to torch tensor
    hess = torch.autograd.functional.hessian(f, x_tensor)
    return hess.numpy()

Since the functions to compute the derivatives are always the same, I was wondering if there was a way (or if it makes sense) to save the computational graph to avoid computing many times the same quantity, and thus speed up the calculation.
Thanks,
Daniel