There is a rearranged version of the Adam in the paper.
It is mentioned in the last paragraph of section 2 Algorithm.
“Note that the efficiency of algorithm 1 can, at the expense of clarity, be improved upon by changing the order of computation, e.g. by replacing the last three lines in the loop with the following lines…”
The same is implemented in Torch/PyTorch.
–