Why it always raise error like this?
### System Info

I could promise that there is not a network connection problem at all,but it true raised like this:
```bash
Traceback (most recent call last):
  File "test.py", line 3, in <module>
    model = AutoModelForCausalLM.from_pretrained("nomic-ai/gpt4all-j", revision="v1.2-jazzy")
  File "/home/elin/anaconda3/envs/nemo/lib/python3.8/site-packages/transformers/models/auto/auto_factory.py", line 444, in from_pretrained
    config, kwargs = AutoConfig.from_pretrained(
  File "/home/elin/anaconda3/envs/nemo/lib/python3.8/site-packages/transformers/models/auto/configuration_auto.py", line 928, in from_pretrained
    config_dict, unused_kwargs = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)
  File "/home/elin/anaconda3/envs/nemo/lib/python3.8/site-packages/transformers/configuration_utils.py", line 574, in get_config_dict
    config_dict, kwargs = cls._get_config_dict(pretrained_model_name_or_path, **kwargs)
  File "/home/elin/anaconda3/envs/nemo/lib/python3.8/site-packages/transformers/configuration_utils.py", line 629, in _get_config_dict
    resolved_config_file = cached_file(
  File "/home/elin/anaconda3/envs/nemo/lib/python3.8/site-packages/transformers/utils/hub.py", line 452, in cached_file
    raise EnvironmentError(
OSError: We couldn't connect to 'https://huggingface.co' to load this file, couldn't find it in the cached files and it looks like nomic-ai/gpt4all-j is not the path to a directory containing a file named config.json.
Checkout your internet connection or see how to run the library in offline mode at 'https://huggingface.co/docs/transformers/installation#offline-mode'.
```

And here is the SDKs version below:
```
Python 3.8.10
transformers 4.29.2
```

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Not only gpt4-j,but also fallcon by using the code they provided:
```python
from transformers import AutoTokenizer, AutoModelForCausalLM
import transformers
import torch

model = "tiiuae/falcon-7b"

tokenizer = AutoTokenizer.from_pretrained(model)
pipeline = transformers.pipeline(
    "text-generation",
    model=model,
    tokenizer=tokenizer,
    torch_dtype=torch.bfloat16,
    trust_remote_code=True,
    device_map="auto",
)
sequences = pipeline(
   "Girafatron is obsessed with giraffes, the most glorious animal on the face of this Earth. Giraftron believes all other animals are irrelevant when compared to the glorious majesty of the giraffe.\nDaniel: Hello, Girafatron!\nGirafatron:",
    max_length=200,
    do_sample=True,
    top_k=10,
    num_return_sequences=1,
    eos_token_id=tokenizer.eos_token_id,
)
for seq in sequences:
    print(f"Result: {seq['generated_text']}")


```

### Expected behavior

Fix it!
Hi @RosterMouch, thanks for raising this issue. 

A few questions on our side to try and help dig into the issue: 
* Could you share which verion of `huggingface_hub` is beign run in your environment? 
* When you say "I could promise that there is not a network connection problem at all", could you share how this was tested?
* Is this an error the consistently happens or sporadically? 
* Is this issue only ever seen with this checkpoint or with other checkpoints too? 