Found this old thread for a random chance.
I am the author of that paper, I am a bit surprised by your comments:

the experiments are not on MNIST only
the optimizer is obviously not equivalent to running different learning rate and averaging

Did you actually read the paper?