I can confirm it was the choice of optimizer which caused the behavior, both adam and sgd with momentum would behave as I previously described. Sgd with momentum=0 does what I want it to do.
Thank you very much for the help!