Can't push a model to hub
## Environment info
- `transformers` version: 4.16.1
- Platform: Linux-5.4.0-96-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyTorch version (GPU?): 1.10.0+cu113 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help
@patrickvonplaten, @anton-l

## Information

Model I am using: facebook/wav2vec2-xls-r-300m

The problem arises when using:
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: speech-recognition-community-v2

## To reproduce

Steps to reproduce the behavior:

1. I use a modified jupiter notebook from this site: https://huggingface.co/blog/fine-tune-wav2vec2-english
2. Firstly a train was running with TrainingArguments(push_to_hub=True). It  worked, you can see results  here: https://huggingface.co/AigizK/wav2vec2-large-xls-r-300m-bashkir-cv7_opt/commits/main until the commit "Training in progress, step 94900"
3. Then my notebook has crashed
4. I ran all cells until trainer.train()
5. sent the current state to the repository without thinking(commits 293563110d31ba9610e4ad7bffd8b377beaf4cf5 and 3f6e999d7d2fdb4640fd1523bfca744681fcc2fc)
6. Then I run train from last checkpoint
7. When try to send result to hub but get error
8. 
![train-push-to-hub-bug](https://user-images.githubusercontent.com/1628540/151814762-4fb4eecb-1def-4ed8-9c49-cd1c2287f054.png)
9. I can resolve this issue only using this command
- repo = Repository(local_dir="xls-r-300m-ba_v2", clone_from="AigizK/wav2vec2-large-xls-r-300m-bashkir-cv7_opt")
- copy last model.bin to this folder
- repo.push_to_hub(commit_message="50 epochs")


When I asked in  the Discord from @patrickvonplaten  how i can resolve he wrote:
`Could you open an issue on Transformers for this? This looks like a library mismatch between transformers and huggingface_hub`
https://discord.com/channels/879548962464493619/930507803339137024/937665851878952990

Gently pinging @sgugger here. Any idea what could be the problem here? I cannot really reproduce this error<|||||>Are you sure that in your second run you still had `push_to_hub=True` in your `TrainingArguments`? Because the `repo` arg not defined can only come from there I think.<|||||>@sgugger From Step3 i used only `push_to_hub=False`<|||||>So that's why you get an error. We will make it clearer, but you can't use `Trainer.push_to_hub` if `push_to_hub=False`.<|||||>Thank you. I thought that `push_to_hub` only affects during training.