dashesy:

I am trying to write a SGD that would be more similar to other frameworks .
This is what I currently have:


According to the only difference:

Screen Shot 2019-02-07 at 6.41.10 PM.png721×268 10.9 KB

Isn’t it a bit overkill to rewrite the whole SGD? Wouldn’t it be sufficient to scale the momentum term by the learning rate to get the equivalent when needed sometimes?