I’ve recently released a modular implementation of L-BFGS that is compatible with many recent algorithmic advancements for improving and stabilizing stochastic quasi-Newton methods and addresses many of the deficiencies with the existing PyTorch L-BFGS implementation. It is designed to provide maximal flexibility to researchers and practitioners in the design and implementation of stochastic quasi-Newton methods for training neural networks.
Link to the Github repository is here:
https://github.com/hjmshi/PyTorch-LBFGS 58
Since the implementation is quite immature, I’m desperately looking for feedback and wanted to provide an open forum for practitioners to share with us their experiences with the code. If you had the opportunity to try it out, please let us know how it’s performing - it will be beneficial for us in improving the implementation and potentially even in posing future research questions on stochastic quasi-Newton methods, particularly for training neural networks.
Thanks for your attention!