The authors compare SM3 to Adafactor in their article. I’ll quote the relevant paragraph as I don’t think I can say it better.

Comparison with Adafactor: Adafactor is a very effective method for space-efficient adaptive optimization. SM3 and Adafactor differ in a number of important ways. First, Adafactor is only defined for matrix-shaped parameters while SM3 applies to tensors of arbitrary dimensions, and even more generally, to any predefined cover of the parameters. Second, Adafactor is in essence a fixed learning-rate algorithm, being a memory-constrained variation of Adam, and often requires a manually devised learning-rate schedule to ensure convergence. In contrast, SM3 adapts its learning rates in an adaptive, data-driven manner similar to Adagrad. Finally, SM3 comes with rigorous convergence guarantees in stochastic convex optimization settings.

And yeah, it seems that SM3 offers amazing flexibility in choice of cover. They end up choosing similar to Adafactor despite this flexibility as their experiments showed that rows/columns had correlated activations.