DeBerta unable to load from cache
### System Info

```shell
Transformers=4.18.0
Pytorch=1.3.0a0+ee77ccb
Ipython=6.2.1
Python=3.6.9
the transformers-cli is unfortunately not accurate--I am on a networked file system with a base conda environment containing torch on other large libraries installed on every machine and transformers installed in a virtualenv in the filesystem.
```


### Who can help?

@LysandreJik 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
In [1]: import transformers
In [2]: from transformers import AutoConfig, AutoModel
In [3]: config = AutoConfig.from_pretrained('microsoft/deberta-base', cache_dir='path-to-uploaded-transformers-cache')
In [6]: model = AutoModel.from_pretrained('microsoft/deberta-base', config=config, cache_dir='path-to-uploaded-transformers-cache')
ValueError                                Traceback (most recent call last)
/state/partition1/llgrid/pkg/anaconda/anaconda3-2020a/lib/python3.6/tarfile.py in nti(s)
    188             s = nts(s, "ascii", "strict")
--> 189             n = int(s.strip() or "0", 8)
    190         except ValueError:

ValueError: invalid literal for int() with base 8: 'v2\nq\x02((X'
```

### Expected behavior
I have a shared filesystem where I would like to use the Deberta model
The filesystem does not allow file lock because of how the network is setup, so I download all models locally and upload them. 
I have downloaded the Deberta model to a cache dir. Output looks like this:

```shell
me@node:~$ ls transformers_cache/
05056f257c8d2b63ad16fd26f847c9ab9ee34e33cdfad926e132be824b237869.5d12962c5ee615a4c803841266e9c3be9a691a924f72d395d3a6c6c81157788b
05056f257c8d2b63ad16fd26f847c9ab9ee34e33cdfad926e132be824b237869.5d12962c5ee615a4c803841266e9c3be9a691a924f72d395d3a6c6c81157788b.json
05056f257c8d2b63ad16fd26f847c9ab9ee34e33cdfad926e132be824b237869.5d12962c5ee615a4c803841266e9c3be9a691a924f72d395d3a6c6c81157788b.lock
c2bc27a1c7529c177696ff76b1e74cba8667be14e202359f20f9114e407f43e2.a39abb1c6179fb264c2db685f9a056b7cb8d4bc48d729888d292a2280debf8e2
c2bc27a1c7529c177696ff76b1e74cba8667be14e202359f20f9114e407f43e2.a39abb1c6179fb264c2db685f9a056b7cb8d4bc48d729888d292a2280debf8e2.json
c2bc27a1c7529c177696ff76b1e74cba8667be14e202359f20f9114e407f43e2.a39abb1c6179fb264c2db685f9a056b7cb8d4bc48d729888d292a2280debf8e2.lock
ce0ac094af27cf80bbf403595a6d47f1fc632981bf1d4c5bf69968568cbea410.e8ad27cc324bb0dc448d4d95f63e48f72688fb318a4c4c3f623485621b0b515c
ce0ac094af27cf80bbf403595a6d47f1fc632981bf1d4c5bf69968568cbea410.e8ad27cc324bb0dc448d4d95f63e48f72688fb318a4c4c3f623485621b0b515c.json
ce0ac094af27cf80bbf403595a6d47f1fc632981bf1d4c5bf69968568cbea410.e8ad27cc324bb0dc448d4d95f63e48f72688fb318a4c4c3f623485621b0b515c.lock
dataset-metadata.json
dde0725208c11536042f6f416c538792d44a2d57d1ae399bbd1bc5867e02c465.0a3ec262cb3d4f634c72ce55f2766bb88771e6499b2512830e2e63bf19dbf97a
dde0725208c11536042f6f416c538792d44a2d57d1ae399bbd1bc5867e02c465.0a3ec262cb3d4f634c72ce55f2766bb88771e6499b2512830e2e63bf19dbf97a.json
dde0725208c11536042f6f416c538792d44a2d57d1ae399bbd1bc5867e02c465.0a3ec262cb3d4f634c72ce55f2766bb88771e6499b2512830e2e63bf19dbf97a.lock
e313266bff73867debdfa78c78a9a4966d5e78281ac4ed7048c178b16a37eba7.fb501413b9cef9cef6babdc543bb4153cbec58d52bce077647efba3e3f14ccf3
e313266bff73867debdfa78c78a9a4966d5e78281ac4ed7048c178b16a37eba7.fb501413b9cef9cef6babdc543bb4153cbec58d52bce077647efba3e3f14ccf3.json
e313266bff73867debdfa78c78a9a4966d5e78281ac4ed7048c178b16a37eba7.fb501413b9cef9cef6babdc543bb4153cbec58d52bce077647efba3e3f14ccf3.lock

```
However when I try to load from the cache, I get the header error listed above.
This error is not reproduced on my local machine (MacOS)
```

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.