early_stopping_patience_counter increasing + 2 per epoch in distributed mode
## Environment info
- `transformers` version: 4.4.0
- Platform: Linux-5.4.0-1047-azure-x86_64-with-glibc2.27
- Python version: 3.8.0
- PyTorch version (GPU?): 1.7.1+cu101 (True)
- Tensorflow version (GPU?): 2.2.1 (True)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: yes


### Who can help

- trainer: @sgugger

## Information

Model I am using (Bert, XLNet ...): LayoutLMForTokenClassification

The problem arises when using:
* [ ] my own modified scripts
I am training a LayoutLmForTokenClassification model in distributed mode, and added logging for the EarlyStoppingCallback.early_stopping_patience_counter to gain more insights. I noticed that the count augments with 2 at each epoch, instead of with one (it is called on_epoch_end), I guess from each parallel computation?


## To reproduce

Steps to reproduce the behavior:

1. Train LayoutLMForTokenClassification with trainer class with evaluation_strategy=IntervalStrategy.EPOCH and using EarlyStoppingCallback 


## Expected behavior

increase of patience counter + 1 at each epoch

In distributed training, each process will have their own `EarlyStoppingCallback` that are independent and incremented separately, so that's not the issue here.
I tested locally and I see it increases one by one normally even in distributed training.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.