Push to hub fails to "update ref" for GLUE example script
The `--push-to-hub` is throwing an error while running the `run_glue.py` script for FNet model. The checkpoints and runs are getting uploaded [here](https://huggingface.co/gchhablani/fnet-base-finetuned-mrpc/tree/main) but ideally, no errors should be observed.

## Environment info

- `transformers` version: 4.11.0.dev0
- Platform: Linux-5.8.0-1038-gcp-x86_64-with-glibc2.17
- Python version: 3.8.11
- PyTorch version (GPU?): 1.9.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help

Library:
- trainer: @sgugger

## Information

Model I am using (FNet): FNetForSequenceClassification (To Be Merged) #13045 

The problem arises when using:
* [x] the official example scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: MRPC

## To reproduce

Steps to reproduce the behavior:

1. Prepare a shell script:
```bash
#!/usr/bin/bash

python ../run_glue.py \
  --model_name_or_path google/fnet-base \
  --task_name mrpc \
  --do_train \
  --do_eval \
  --max_seq_length 512 \
  --per_device_train_batch_size 16 \
  --learning_rate 2e-5 \
  --num_train_epochs 3 \
  --output_dir fnet-base-finetuned-mrpc \
  --push_to_hub \
  --hub_strategy all_checkpoints \
  --logging_steps 50 \
  --logging_strategy steps \
  --save_steps 50 \
  --save_strategy steps \
  --eval_steps 50 \
  --evaluation_strategy steps \
```
2. Install from the FNet branch:
```bash
pip install git+https://github.com/gchhablani/transformers.git@add_fnet
```
3. Run the shell script:
```bash
bash mrpc.sh
```
Error thrown:
```python
Traceback (most recent call last):trick-general-gpu/events.out.tf
  File "/home/gunjan/anaconda3/envs/fnet_env/lib/python3.8/site-packages/huggingface_hub/repository.py", line 967, in git_push [00
    raise subprocess.CalledProcessError(
subprocess.CalledProcessError: Command '['git', 'push', '--set-upstream', 'origin', 'main']' returned non-zero exit status 1.
Upload file checkpoint-550/rng_state.pth: 100%|█| 14.2k/14.2k [00
During handling of the above exception, another exception occurred:load file checkpoint-600/rng_state.pth: 100%|█| 14.2k/14.2k [00

Traceback (most recent call last):trick-general-gpu/events.out.tf
  File "../run_glue.py", line 566, in <module>
    main()e checkpoint-550/scheduler.pt: 100%|█| 623/623 [00:21<?
  File "../run_glue.py", line 557, in main
    trainer.push_to_hub(**kwargs)ler.pt: 100%|█| 623/623 [00:21<?
  File "/home/gunjan/anaconda3/envs/fnet_env/lib/python3.8/site-packages/transformers/trainer.py", line 2636, in push_to_hub0:21<?
    git_head_commit_url = self.repo.push_to_hub(commit_message=commit_message, blocking=blocking)
  File "/home/gunjan/anaconda3/envs/fnet_env/lib/python3.8/site-packages/huggingface_hub/repository.py", line 1052, in push_to_hub
    return self.git_push(
  File "/home/gunjan/anaconda3/envs/fnet_env/lib/python3.8/site-packages/huggingface_hub/repository.py", line 972, in git_push
    raise EnvironmentError(exc.stderr)
OSError: remote: error: cannot lock ref 'refs/heads/main': is at 7e59d1c65a4cf5010f2445a0bad6199ea4dea34f but expected 31f6a4614c202fa2af7d677bca993b0bf4c1cab9        
To https://huggingface.co/gchhablani/fnet-base-finetuned-mrpc
 ! [remote rejected] main -> main (failed to update ref)
error: failed to push some refs to 'https://user:xxxx@huggingface.co/gchhablani/fnet-base-finetuned-mrpc'
```
I tried deleting the repo from my user and running the script again, still the same issue happens. Is this because of background processes still trying to push during the next run?

## Expected behavior

The checkpoints and logs get pushed to hub without any issues/failures.

**Side Note**: I believe the content after `user:` is the access token? Should it be shown while throwing the error?

CC @patrickvonplaten 
Thanks for flagging this. The content after `user:` is indeed your access token so I edited your message to remove it (opening an issue in parallel on huggingface_hub to make sure we remove those when displaying error).

It is possible that you got an error once and have some processes still trying to push in the background. If you start from scratch, does the issue persist?

cc @LysandreJik <|||||>@sgugger I believe that is what is happening.
I am not getting an error after waiting for a while and cleaning both local and remote copies.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.