Can you try with linear weight decaying instead of fixed value i.e. 1st epoch - 0.001, 2nd epoch - 0.0008, 3rd epoch - 0.0004 etc?  I guess for lr = 0.003, having a constant weight decay of 0.001 is reducing lr faster than with lr = 0.03.