Machine Learning, Neural Networks, Statistics, Data Science, AI.
recently shown to not be the case for adaptive gradient algorithms, such as Adam. Why divide by m? So, we are still left with the question of the division by m. After all, proper hyper-parameter optimization should also be able to handle changes in scale (at least in theory). Option 1: A