Should I use weight_decay or weight_decay_rate?
Thanks for the awesome work.

Just as line [simple_lm_finetuning.py#L540](https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/examples/lm_finetuning/simple_lm_finetuning.py#L540), When I use bert for downstream tasks, should I use `weight_decay` or `weight_decay_rate` when I add a decay operation to the training parameters?

What if I use apex for mixed precision training?
According to the instructions [module-torch.optim](https://pytorch.org/docs/stable/optim.html?highlight=torch%20optim#module-torch.optim) from PyTorch API and [fused_adam.py](https://github.com/NVIDIA/apex/blob/master/apex/optimizers/fused_adam.py) from apex repo, I think `weight_decay` and `weight_decay_rate` are unified and unified into `weight_decay`, is it correct to understand?<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
