`pipeline` does not load from local folder, instead, it always downloads models from the internet.
### System Info

I create `pipeline` and called `save_pretrained(...)` to save to some local directory. However, when I load it back using `pipeline(model="local_folder")`, it either load from cache or try to start downloading from the internet.

However, if I do the following, it works. I am using the latest `transformers`. am I misused it or misunderstood something?
```
mypipeline.save_pretrained(save_directory=model_path)
mypipeline.model.config.use_pretrained_backbone = False
mypipeline.model.config.save_pretrained(save_directory=model_path)
```

### Who can help?

_No response_

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. run the code below
```
from transformers import pipeline

vision_classifier = pipeline(task="image-classification", model="google/vit-base-patch16-224")
vision_classifier.save_pretrained("./huggingface")
```

2. now delete the cache
3. load the model now
```
vision_classifier = pipeline("./huggingface")

```
and it will start download the pretrained model again.

### Expected behavior

I expect it loads the model from the local folder 
cc @Narsil <|||||>Seems to be working fine on my end, but there needs to be a few modifications (I'm super suprises it *can* download anything, your included code just crashes normally).

```python
from transformers import pipeline

pipe = pipeline(task="image-classification", model="google/vit-base-patch16-224")
pipe.save_pretrained("./local_vit")

pipe = pipeline(task="image-classification", model="./local_vit")
```

You need to specify the `task` for local, since that information is contained in the HUB, not in the config directly. So if you specify it, it works with everything locally.

On what version of transformers are you on ?<|||||>Hi, sorry I specified the task for local, and it still does not work. I will check the version. <|||||>the version is 4.24.0<|||||>I cannot reproduce even on 4.24.0 Can you include the full script + error ? <|||||>sure. I am still seeing the error. I will post the code later.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.