Cannot reproduce roberta-large on SQuAD
## Information
Using the same hyper-paramters as [paper](https://arxiv.org/abs/1907.11692), I fine-tund roberta-large on SQuAD1.1 resulting disappointing results as below. I am wondering that the reason of it might be the gradient noralization is different from the [official implementation](), although it hasn't been released yet.

```
{'exact': 0.21759697256385999, 
'f1': 7.113439302309792, '
total': 10570, 
'HasAns_exact': 0.21759697256385999, 
'HasAns_f1': 7.113439302309792, 
'HasAns_total': 10570, 
'best_exact': 0.21759697256385999, 
'best_exact_thresh': 0.0, 
'best_f1': 7.113439302309792, 
'best_f1_thresh': 0.0}
```

## To reproduce
```
python3 -m torch.distributed.launch --nproc_per_node=4 ./examples/question-answering/run_squad.py \
    --model_type roberta \
    --model_name_or_path roberta-large \
    --do_train \
    --do_eval \
    --train_file $SQUAD_DIR/train-v1.1.json \
    --predict_file $SQUAD_DIR/dev-v1.1.json \
    --learning_rate 1.5e-5 \
    --weight_decay 0.01 \
    --max_grad_norm 0.0 \
    --num_train_epochs 2 \
    --warmup_steps 222 \
    --adam_betas '(0.9, 0.98)' \
    --adam_epsilon 1e-6 \
    --max_seq_length 512 \
    --doc_stride 128 \
    --output_dir ./examples/models/finetuned_squad1.1/ \
    --per_gpu_eval_batch_size=8   \
    --per_gpu_train_batch_size=2   \
    --gradient_accumulation_steps=6 \
    --threads 8 \
    --overwrite_cache \
```
And also, it seems that the glue results showcased in https://huggingface.co/roberta-large refers to the paper instead of fine-tuning from your own script.<|||||>@ZheyuYe

Is this hyperparameter  `   --adam_betas '(0.9, 0.98)' \` available in a modified Transformers v3.0.2 `run_squad.py`, or earlier version?
It is not appearing as an available hyperparameter in the current master version.

I was able to fine-tune RoBERTa large on Squad 2.0 as late as 29June20:  ["ahotrod/roberta_large_squad2"](https://huggingface.co/ahotrod/roberta_large_squad2) with satisfactory results.<|||||>@ahotrod 

Since the `betas` is available in current Adamw, I added this flag `adam_betas`  to match the optimizer hyper-parameters as https://github.com/ZheyuYe/transformers/blob/efc022060195dca384a95546c6134667696f957f/examples/question-answering/run_squad.py#L98-L101

 Thanks for providing these useful  hyperparameter, I am going to re-fune-tune  `roberta-large` on Squad 2.0. I was noticed that `warmup_steps = 1642` was selected with total `Total optimization steps = 8211`, so that is `warmup_ratio = 0.2`?  The other thing that confuses me is why would you choose `do_lower_case` when RoBERTa was pretrained with cased setting https://github.com/pytorch/fairseq/issues/1429.<|||||>@ZheyuYe 

I'm guessing the `do_lower_case` was overlooked when I started with script from another model fine-tuning.  However, I believe I've read that `do_lower_case` has no effect with newer models running the latest `run_squad.py`.  If I was fine-tuning RoBERTa_large again I'd leave it out.

I have fine-tuned RoBERTa_large a bunch of times, varying the hyperparameters:  #epochs, learning rate, warmup ratio, etc.  Plus I switched mid-stream from using 2x NVIDIA 1080Ti GPUs to a single 24GB NVIDIA RTX Titan.  Compared to the original RoBERTa paper's **Table 10 Hyperparameters**, for this particular run I bumped epochs from 2 to 3, and increased warmup ratio to 0.2 with good success.  It may be a result of RoBERTa not being that dependent on warmup ratio used, I don't know for sure.  With my configuration, this fine-tuning script produced the best results.  Check-out the tensorboard loss & learning rate graphs, script, etc. at [https://huggingface.co/ahotrod/roberta_large_squad2#list-files](https://huggingface.co/ahotrod/roberta_large_squad2#list-files)

As quoted many times, "Your mileage may vary" ;-]
Have fun with it!  Hope you beat my results.<|||||>I am closing this issue since I have got the competitive results although there is still a gap from the paper's