@falmasri @DoubtWang
If you use Adam optimizer with indentical hyperparameters and calling the optimization step at the same time you will not see any numerical difference.
Adam optimizations step is done per parameter (moving averages are per parameter/independent).
For each parameter we have one state.