Can you share the stack trace? The error  is raised at line X.grad = -f(X) right?
If so, then I would add prints here.
Keep in mind that LBFGS (contrary to all other optimizers) actually evaluate the gradients multiple times after updating the value of your X.