Could you try to lower the learning rate or momentum step by step and see if and when the loss starts to decrease?
If it just stalls, we would have to try to debug your code I guess.
However, if the loss starts to decrease, I’m wondering if your issue might be related to the problem of accumulating momentum for distributed workers. I’ll try to dig up the paper and see, if it might be related.