Feel the same of AladdinPerzon said.
I would lower the lr to approx. 1e-5 to 1e-4 to see what happened.
And another thing I will do is to disable the weight decay since it may cause the training harder.