So let me summarize.
From experiments and from F.sgd 6 code we may observe that weight_decay is in fact add $\ambda w_i$
So:


Original modified code
"print(w_numpy - lr * (w_grad+0.9*w_numpy))"
produce the same result as
"print(w_torch.data.numpy())"
Error in the question was in ignoring the weight_decay parameter.


"The weight_decay parameter adds an L2 penalty to the cost". Such a response is too vague, such a thing as the L2 penalty does not exist in mathematics.


Please don’t blame me, but you use several words from math:

L2 is a name from math that stands for the norm. And it’s not true that you use the L2 norm at all. What you use is the L2 norm square of all vectors of parameters. It’s not L2 norm.
You use the word “Penalty” penalty can be as a constraint or extra term in objective. If it is an extra term then different technics for scalarization are existed.

Next to such responses like “L2 penalty” lead to real problems.

First of all - you multiply it by “\lambda/2”, not by “\lambda”
It’s why such responses add “L2” penalty is too vague, and it only brings too much confusion. It’s better to say everything exactly.


The formula with “2 \lambda” is incorrect as well. And bring only confusion.

So what is implemented in PyTorch is weight_decay corresponds to implicit additive term in the objective function with form $(\lambda/2) |parameters|_2^2$.