The derivation of the BFGS is more intuitive when one considers (strictly) convex cost functionals:
However, some background information is necessary: Assume, one wants to minimize a convex functional $$ f(x) \to \min_{x\in \mathbb R^n}. $$ Say there is an approximate solution $x_k$. Then, one approximates the minimum of $f$ by the minimum of the truncated <PERSON> expansion $$ f(x_k+p) \approx f(x_k) +\nabla f(x_k)^Tp + \frac{1}{2}p^T H(x_k)p. \quad() $$ That is, one looks for $p$ such that $()$ is minimal and sets $x_{k+1} := x_k + p$. Computing the gradient of $(*)$ -- "with respect to $p$" -- and setting it to zero gives the relation $$ H(x_k)[x_{k+1} - x_k] = \nabla f(x_{k+1}) - \nabla f(x_k), $$ where $H$ is the 'Jacobian of the gradient' or the Hessian matrix.
Since the computation and inversion of the Hessian is expensive...
...a short answer
(cf. 768
<PERSON>'s update) might be that the BFGS update $H_{k+1}^{-1}$ minimizes $$ \|H_k^{-1} - H^{-1}\|_W $$ in a smartly chosen weighted Frobenius norm, subject to
1. $H[x_{k+1} - x_k] = \nabla f(x_{k+1}) - \nabla f(x_k)$ -- this is what one is out for -- and
2. $H^T = H$, because the Hessian is symmetric.
Then the choice of the weight $W$ in $\|H\|_W := \|W^{1/2}HW^{1/2}\|_F$ as the inverse of the averaged Hessian $G:=\int_0^1 H(x_k + \tau p) d\tau$, cf. here for the statement but without proof, gives the BFGS update formula (with $\alpha_k = 1$).
The major points are:
* One tries to approximate the solution for the actual costs by the solution for a quadratic approximation
* Computation of the Hessian, and its inverse, is expensive. One prefers simple updates.
* The update is chosen optimal for the inverse rather than the actual Hessian.
* That it is a rank-2 update is a consequence of the particular choice of the weights in the <PERSON> norm.
A longer answer, should include how to choose the weights, how to make this work for nonconvex problems (where a curvature condition appears that requires a scaling of the search direction $p$), and how to derive actual the formula for the update. A reference is here (in German). 768