Thank you.
How does the weight decay still affect parameters of a layer, which no inputs are forwarded through it as @shivammehta007 said?
It seems that the gradients of this layer will remain None after the backward process, so these parameters will no be appended to params_with_grad in the step function. Then, it seems that these parameters will not be iterated over in F.adam 1 and weight decay will not be applied on these parameters as @pascal_notsawo said.
By the way, according to zero_grad, it indeed does not zero parameters, which are already set to None, so set_to_none indeed does not change the situation.