Solution: I need to wrap my custom function within a nn.Module or a function to make it callable, for example,
def custom_det(A):
  return Determinant.apply(A)

then,
hessian = torch.autograd.functional.hessian(custom_det, A)