Model card reports wrong evaluation metrics when load_best_model_at_end=True
## Environment info

- `transformers` version: 4.11.3
- Platform: Linux-5.4.0-1060-aws-x86_64-with-debian-buster-sid
- Python version: 3.7.12
- PyTorch version (GPU?): 1.10.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help

- Trainer: @sgugger

## Information

Model I am using: distilroberta-base

The problem arises when using:
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Train a model using `Trainer()`, set `load_best_model_at_end=True` and add `EarlyStoppingCallback`
2. Use `trainer.push_to_hub()` to publish the model

Here's the full training config I was using:

```
def simple_accuracy(preds, labels):
    return (preds == labels).mean()

def compute_metrics(p):
    preds = np.argmax(p.predictions, axis=1)
    return {"acc": simple_accuracy(preds, p.label_ids)}

training_args = TrainingArguments(
    output_dir=output_dir,
    do_train=True,
    do_eval=True,
    fp16=True,
    weight_decay=0.01,
    num_train_epochs=20,
    per_device_train_batch_size=32,
    per_device_eval_batch_size=32,
    gradient_accumulation_steps=1,
    learning_rate=5e-5,
    warmup_steps=16,
    logging_steps=16,
    evaluation_strategy='epoch',
    save_strategy='epoch',
    save_total_limit=patience,
    load_best_model_at_end=True,
    metric_for_best_model='acc',
    greater_is_better=True,
    seed=seed,
    report_to='none'    
)

trainer = Trainer(
    model=clf,
    args=training_args,
    train_dataset=ds_train,
    eval_dataset=ds_valid,
    compute_metrics=compute_metrics,
)
patience = 5
trainer.add_callback(EarlyStoppingCallback(patience, 1e-3))
```

Training progress:

![image](https://user-images.githubusercontent.com/1490131/152837112-eb12a817-a6e4-453b-ab7a-6a597dc165b8.png)

However in the model card, the reported accuracy is the last checkpoint rather than the best checkpoint:


> This model is a fine-tuned version of [distilroberta-base](https://huggingface.co/distilroberta-base)[](https://huggingface.co/valurank/distilroberta-offensive#model-description) on an unknown dataset. It achieves the following results on the evaluation set:
>
> Loss: 0.4526
> Acc: 0.8975


The model card always reports the last metrics, there is no easy way to change that. In general, you should always do a last manual evaluation with `trainer.evaluate()` to have the correct info being populated in the model card, like we do in all our example scripts.<|||||>@sgugger thanks for the quick response. Good tip, I'll make sure to add a call to `evaluate`.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.