I wouldn’t spend my time on anything that only shows MNIST results.. I wouldn’t spend my time on anything that only shows results on small datasets. It’s really hard to say whether it generalizes to anything in practice or not.
For anyone else visiting this thread, this optimizer is equivalent while reading the code intuitively, I see it roughly doing something like: running several parallel optimizers on the model, each with a different learning rate, and then averaging the weights out.
Edited because the comment oversimplified and incorrectly represented the original paper. See comments below.