Interesting! Thanks for the detailed explanation. The last line(second derivative) reminds me the Hessian-vector product that we use to calculate a row of hessian. If considered from that perspective it looks like calculating a row of the hessian and then getting the right element, so that overall it is equivalent to calculating the whole hessian and getting the diagonal.
I am not sure there is a space for optimization since, it looks like we need to set the $$a$$ to calculate the second derivative and this would require N hessian-vector product, compare to 1 hessian-vector product for getting a row or column.