Problem with optimizers after migration
## 📚 Migration

<!-- Important information -->

Model I am using (Bert, XLNet....): Bert

Language I am using the model on (English, Chinese....): Russian

The problem arise when using:
The optimizers. I tried the default parameters from the example with max_grad_norm = 1.0 and lr = 2e-5.

```
warmup_proportion = float(num_warmup_steps) / float(num_total_steps)  # 0.1

### Previously BertAdam optimizer was instantiated like this:
optimizer = BertAdam(model.parameters(), lr=lr, schedule='warmup_linear', warmup=warmup_proportion, t_total=num_total_steps)
### and used like this:
for batch in train_data:
    loss = model(batch)
    loss.backward()
    optimizer.step()

### In PyTorch-Transformers, optimizer and schedules are splitted and instantiated like this:
optimizer = AdamW(model.parameters(), lr=lr, correct_bias=False)  # To reproduce BertAdam specific behavior set correct_bias=False
scheduler = WarmupLinearSchedule(optimizer, warmup_steps=num_warmup_steps, t_total=num_total_steps)  # PyTorch scheduler
### and used like this:
for batch in train_data:
    loss = model(batch)
    loss.backward()
    torch.nn.utils.clip_grad_norm_(model.parameters(), max_grad_norm)  # Gradient clipping is not in AdamW anymore (so you can use amp without issue)
    scheduler.step()
    optimizer.step()
```


The tasks I am working on is:
* [ ] my own task or dataset: private dataset on comments from client support

Details of the issue:

After migration I see that the model converges more slowly and failed to obtain such an accuracy as it was before. In my multi-class classification dataset I get about 0.59 accuracy while the previous version resulted in about 0.63 accuracy after convergence. Do the optimizers in both versions are equivalent? If not, how can I make them absolutely the same?

## Environment

* OS: Ubuntu
* Python version: 3.6
* PyTorch version: 1.0
* PyTorch Transformers version (or branch): 1.0
* Using GPU ? Yes
* Distributed of parallel setup ? No
You should get the same behavior than `BertAdam` by setting `correct_bias=False` in `AdamW` and using the `WarmupLinearSchedule` together with it.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
