Checkpoints are NOT save in GCS (Google Cloud Storage)
### System Info

```shell
- `transformers` version: 4.18.0
- Platform: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.13
- Huggingface_hub version: 0.5.1
- PyTorch version (GPU?): 1.10.0+cu111 (True)
- Tensorflow version (GPU?): 2.8.0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>
```


### Who can help?

@patrickvonplaten @patil-suraj @sgugger

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Steps to reproduce the behavior:

I am working with BART model in Google Colab:
```
from google.colab import auth

auth.authenticate_user()

project_id = "my_project_id"
!gcloud config set project {project_id}

tokenizer = BartTokenizer.from_pretrained('facebook/bart-base')
model = BartForConditionalGeneration.from_pretrained('facebook/bart-base')
....
training_args = Seq2SeqTrainingArguments(output_dir='gs://my_bucket/models/') 

```





### Expected behavior

```shell
It is supposed to save the checkpoints in GCS (Google Cloud Storage). However, it is saving the checkpoints in my local.

In fact, I can see my files if I type:


!gsutil ls gs://my_bucket/
```

But I can not save my checkpoints.

Do you have any idea how can I fix it?

Thank you in advance.
```

Hi @JessicaLopezEspejel 👋  Most file writing functions, like `torch.save`, don't support writing to/reading from GCS out of the box. There are a few workarounds:
- [edit] See @mikcnt comment [here](https://github.com/huggingface/transformers/issues/16858#issuecomment-1114924716) 
- Use TensorFlow, whose native read/write functions support GCS when properly implemented (see [this](https://www.tensorflow.org/api_docs/python/tf/io/gfile/GFile)) ;)
- Wrap your existing code so as to call read from/write to GCS externally;
- After running your code, manually copy files around with `gsutil -m cp -r ...` ([docs](https://cloud.google.com/storage/docs/gsutil/commands/cp))

Also see this [similar StackOverflow question](https://stackoverflow.com/questions/57898998/is-it-possible-to-load-a-pretrained-pytorch-model-from-a-gcs-bucket-url-without).<|||||>Hello @gante

I see, thank you very much. I will try one the workarounds you propose. :)
<|||||>Hello! I just wanted to clarify a little detail about what @gante said: `torch.save` actually supports writing to GCS! You just need to change the way you use it a bit :) I gave a complete explanation on how to do that [here](https://stackoverflow.com/questions/69100496/not-able-to-save-model-to-gs-bucket-using-torch-save).<|||||>Hello @mikcnt, thank you so much! I will try it. <|||||>@mikcnt thanks for the clarification 👍 (gonna post-edit my comment, for future reference)