Solved by AlphaBetaGamma96 in post #2 
              
              
                Solution: I need to wrap my custom function within a nn.Module or a function to make it callable, for example, 
def custom_det(A):
  return Determinant.apply(A)

then, 
hessian = torch.autograd.functional.hessian(custom_det, A)