wrong cache_dir is used when tokenizer is trying to infer config_tokenizer_class
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.10.3
- Platform: Linux-5.4.0-88-generic-x86_64-with-glibc2.10
- Python version: 3.8.8
- PyTorch version (GPU?): 1.9.0a0+df837d0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


### Who can help
@LysandreJik

## Information

Model I am using (Bert, XLNet ...): FlaubertTokenizer

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [X] my own modified scripts: (give details below)

I am only loading the tokenizer, and not even using it afterwards. 

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

Two different cache directories are used, when calling 
```
FlaubertTokenizer.from_pretrained("flaubert/flaubert_base_uncased", cache_dir="my_cache_dir")
```

Most of the tokenizer files are loaded to `my_cache_dir`, as expected. However, there is one more model config file, which is downloaded to `/.cache/`, even though an explicit `cache_dir` is passed to `from_pretrained`. In my docker setup, I don't have permissions to write to `/.cache`, so this rightfully results in the following warning:

```
The tokenizer class you load from this checkpoint is not the same type as the class this function is called from. It may result in unexpected tokenization. 
The tokenizer class you load from this checkpoint is 'BertTokenizer'. 
The class this function is called from is 'FlaubertTokenizer'.
```

I believe, this happens due to the following piece of code in [tokenization_utils](https://huggingface.co/transformers/_modules/transformers/tokenization_utils_base.html). In particular, the `from_pretrained()` call doesn't receive the `cache_dir`.
```
        if config_tokenizer_class is None:
            from .models.auto.configuration_auto import AutoConfig  # tests_ignore

            # Second attempt. If we have not yet found tokenizer_class, let's try to use the config.
            try:
                config = AutoConfig.from_pretrained(pretrained_model_name_or_path, use_auth_token=use_auth_token)
                config_tokenizer_class = config.tokenizer_class
```

## To reproduce
```
from transformers import FlaubertTokenizer
FlaubertTokenizer.from_pretrained("flaubert/flaubert_base_uncased", cache_dir="./my_cache_dir")
```

Then check, that some files are downloaded to `./my_cache_dir`, and some other cache files are downloaded to another directory, which I believe, can be found by `pip cache dir` shell command (not sure here).

## Expected behavior

I expect all downloaded files to be placed to one rovided `cache_dir`.

There are a couple of workarounds for my case. The warning doesn't seem to be crucial, so I can just ignore it (which I don't want). Also, I can set the `TRANSFORMERS_CACHE` environmental variable. 

But still I doubt that the current behaviour is an expected one.


Thanks for opening an issue, this seems correct indeed! Would you like to open a PR to fix this?<|||||>Thanks for opening an issue and solving the problem, @vmaryasin!