Ok, thanks for the update!
Could you post the code calculating the loss, the optimizer and if possible the model?
Maybe your gradients are really high somehow.
Usually a momentum of 0.9 should work fine.
Yes, weight decay penalizes the weight magnitude, forcing it to get lower values.