I have a doubt here. In pytorch the weigh decay,  is it only applied to the weighs or to all the parameters that requires gradient? I mean for instance if I use this piece of code:
optimizer = torch.optim.Adam(model.parameters(), lr=args.lr, betas=args.betas, weight_decay=args.wd)
Will be the weight decay applied to all the parameters of the model including bias and batchnorm parameters?
Thanks