There is a paper by Schaul et. al “Adaptive learning rates and…” https://arxiv.org/abs/1301.3764 15 (improvement upon “No More Pesky Learning Rates” https://arxiv.org/abs/1206.1106 1 Schaul et. al )
and python implemetation of the algorithm by the author:
https://github.com/schaul/py-optim/blob/master/PyOptim/algorithms/vsgd.py 13
The paper claims it is optimization algorithm with “linear complexity and is hyper-parameter free”. There is no need to set and therefore decay/adapt learning rate at all with it, sounds really good.
But no DL library seem to have it implemented, is there any reason for this?
Does it require too much memory or computation?