The part that I circled doesn’t seem right to me:

39a9f417cb836820174d71e9a9318d5ccc6191f3.png1379×734 65.7 KB

In L2 regularization, you modify the cost as follows

text_Regularized.png2262×325 21.5 KB

The weight update should be then

text_updatedWeig.png2954×154 25.4 KB

The way PyTorch applied the weight decay seems correct to me (you can drop the factor 2)