```
Optimizers: BertAdam & OpenAIAdam are now AdamW, schedules are standard PyTorch schedules
The two optimizers previously included, BertAdam and OpenAIAdam, have been replaced by a single AdamW optimizer which has a few differences:
it only implements weights decay correction,
schedules are now externals (see below),
gradient clipping is now also external (see below).
The new optimizer AdamW matches PyTorch Adam optimizer API and let you use standard PyTorch or apex methods for the schedule and clipping.
The schedules are now standard PyTorch learning rate schedulers and not part of the optimizer anymore.
Here is a conversion examples from BertAdam with a linear warmup and decay schedule to AdamW and the same schedule:
```python
Parameters:
lr = 1e-3
max_grad_norm = 1.0
num_training_steps = 1000
