Thank you! Your github comments really helps me understand the “hidden” thoughts, though I scratched my head for hours before I realize the thoughts. To verify the implementation,  line 12 could be further expanded as
theta_t <- theta_{t-1} - eta_{t} * apha \frac{.}{.} -  eta_t * lambda * theta_{t-1}
group[“lr”] = eta_{t} * apha because group[“lr”] might be lr scheduler, and thus  group[“weight_decay”] = lambda / apha, when multiplied, the alphas get cancelled.
(Sorry for the ugly plain-text equations).
The actual weight decay parameter is scaled by initial learning rate (1/alpha). And we should take that in to consideration when tuning models.
Correct me if I’m wrong.