With regard to original comments, it would probably be useful if you can provide some code that reproduces the trouble you’re having. I tried your example problem and it converged fine, but perhaps for some initial conditions or learning rates it will not.
There is no Hessian in L-BFGS. The idea of BFGS is to approximate the Hessian, and the idea of Limited-memory BFGS is to approximate the Hessian (which requires O(n^2) storage) with O(n) storage. In fact, the approximate Hessian is never materialized, only the approximate Hessian-Gradient product is (i.e. the descent direction). In Nocedal and Wright, they advise to initialize the approximate Hessian as a diagonal matrix, so you need only compute a scalar, which is what you are referring to.
FWIW, I checked the implementation against the text and the implementation in Scala’s Breeze optimization library and they look the same, apart from choice of step size. The choice of a constant step size was by design though, as noted in the post by ProGamerGov. I can’t comment on the memory issues, and haven’t been able to reproduce any failures for the toy scenario mentioned.