Looks nice! I’m curious how performance compares against the AdaFactor optimizer – https://github.com/pytorch/fairseq/blob/master/fairseq/optim/adafactor.py 24 . AdaFactor was used to train the 1.5B GPT-2 model. SM3 seems to give extra extra flexibility in choosing factors, whereas AdaFactor is limited to output dimension,input dimension split.