It is more obvious in older (1.6) sources, weight_decay only affects one line:

d_p = d_p.add(p, alpha=weight_decay)

i.e. gradient = gradient + param_value * weight_decay
so, as param_value deviates from zero, taken steps increase