AutoTokenizer can't find model/tokenizer config.json 
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.4.0
- Platform: Linux-4.19.0-8-amd64-x86_64-with-debian-10.3
- Python version: 3.7.3
- PyTorch version (GPU?): 1.7.0+cu101 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: No


### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people. -->
 
@LysandreJik  @mfuntowicz


## Information

Model I am using (Bert, XLNet ...): XLM-Roberta, but I've noticed this with other models as well

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1.run `tokenizer = AutoTokenizer.from_pretrained(REF_MODEL)`
2. restart the notebook, for example
3.run `tokenizer = AutoTokenizer.from_pretrained(REF_MODEL)` again

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->
The following error occurs:

```
file xlm-roberta-large/config.json not found
---------------------------------------------------
OSError           Traceback (most recent call last)
~/.cache/pypoetry/virtualenvs/masters-utTTC0p8-py3.7/lib/python3.7/site-packages/transformers/configuration_utils.py in get_config_dict(cls, pretrained_model_name_or_path, **kwargs)
    387                 resume_download=resume_download,
--> 388                 local_files_only=local_files_only,
    389             )

~/.cache/pypoetry/virtualenvs/masters-utTTC0p8-py3.7/lib/python3.7/site-packages/transformers/file_utils.py in cached_path(url_or_filename, cache_dir, force_download, proxies, resume_download, user_agent, extract_compressed_file, force_extract, local_files_only)
    961         # File, but it doesn't exist.
--> 962         raise EnvironmentError("file {} not found".format(url_or_filename))
    963     else:

OSError: file xlm-roberta-large/config.json not found

During handling of the above exception, another exception occurred:

OSError           Traceback (most recent call last)
<ipython-input-11-b51d77705f76> in <module>
----> 1 tokenizer = AutoTokenizer.from_pretrained(f'{REF_MODEL}')

~/.cache/pypoetry/virtualenvs/masters-utTTC0p8-py3.7/lib/python3.7/site-packages/transformers/tokenization_auto.py in from_pretrained(cls, pretrained_model_name_or_path, *inputs, **kwargs)
    304         config = kwargs.pop("config", None)
    305         if not isinstance(config, PretrainedConfig):
--> 306             config = AutoConfig.from_pretrained(pretrained_model_name_or_path, **kwargs)
    307 
    308         if "bert-base-japanese" in str(pretrained_model_name_or_path):

~/.cache/pypoetry/virtualenvs/masters-utTTC0p8-py3.7/lib/python3.7/site-packages/transformers/configuration_auto.py in from_pretrained(cls, pretrained_model_name_or_path, **kwargs)
    331             {'foo': False}
    332         """
--> 333         config_dict, _ = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)
    334 
    335         if "model_type" in config_dict:

~/.cache/pypoetry/virtualenvs/masters-utTTC0p8-py3.7/lib/python3.7/site-packages/transformers/configuration_utils.py in get_config_dict(cls, pretrained_model_name_or_path, **kwargs)
    398                 f"- or '{pretrained_model_name_or_path}' is the correct path to a directory containing a {CONFIG_NAME} file\n\n"
    399             )
--> 400             raise EnvironmentError(msg)
    401 
    402         except json.JSONDecodeError:

OSError: Can't load config for 'xlm-roberta-large'. Make sure that:

- 'xlm-roberta-large' is a correct model identifier listed on 'https://huggingface.co/models'

- or 'xlm-roberta-large' is the correct path to a directory containing a config.json file
```
## Expected behavior

I think that it should load it smoothly

Hello! Do you have a notebook handy so that we can see and try to reproduce the error?<|||||>Thanks for the quick reply! 

I'm doing it in a private repo, I'll try to reproduce it and export it to a public repo asap :)<|||||>I just checked it again and it seems to work smoothly now 🤔 

I'm closing this and if this happens again in the future, I'll open it again :)

My bad.
