The slow for loop approach allows for “hackable” optimizers, so that researchers can easily change the optimizer and run new experiments.
@crcrpar is working on speeding up the step by using the internal for_each methods (previously used in apex as multi_tensor_apply).