Temporary files from an interrupted download litter the disk
## Environment info
- `transformers` version: 4.5.1
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.9.4
- PyTorch version (GPU?): 1.8.1+cpu (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

## Information

Model I am using (Bert, XLNet ...): EleutherAI/gpt-neo-125M

The problem arises when using:
* the official example scripts

## To reproduce

Steps to reproduce the behavior:

1. run the transformers library code snippet from https://huggingface.co/EleutherAI/gpt-neo-2.7B:
```
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("EleutherAI/gpt-neo-2.7B")
model = AutoModelForCausalLM.from_pretrained("EleutherAI/gpt-neo-2.7B")
```
2. abort as it's downloading
3. rerun: it start's anew
4. behold a tmp file remaining until you remove it manually

## Expected behavior

The download should resume where it left off, or the temp file should be deleted when the process is aborted, or at least when it is rerun. (It would also be nice if the download location was allocated on the same disk partition as the calling .py - I put my pycharm project on a partition with enough space expecting the model file to also go there; instead the remaining space on my primary partition was eaten.) (It would also be nice if the files ended up with the same name as the model.)
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.