Thank you.
I have read the blog.
I also checked the SGD implementation in pytorch. The weight decay part is accumulated in the momentum term. This is the same as the L2 regularization implementation in TF.
So, for momentum optimzation method in pytroch, weight decay is the same as L2 regularization.
Personally, I think the only difference is that,  in pytorch the weight decay term in the grad is x. In TF, I have not find out the source code on l2_loss(), but basically I think according to the static graph, the grad on L2 loss equals to  0.5x + 0.5x,When the weight is small enough, 0.5x causes underflow and becomes to 0 in float32. But such small weight x barely affects the whole grad. So the optimization method may be not the culprit.