Instantiating a model from `pipeline()` ignores `model_kwargs` parameter
## Environment info

- `transformers` version: 4.8.1
- Platform: Linux-3.10.0-1160.31.1.el7.x86_64-x86_64-with-centos-7.9.2009-Core
- Python version: 3.7.9
- PyTorch version (GPU?): 1.8.1+cu102 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help

This should be a one-line fix, so I will be submitting a PR shortly.

## Information

Model I am using: `gpt2` (not model-specific issue, though)

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Ensure the model cache is already populated with the correct model by running the following code:

```python
from transformers import AutoModelForCausalLM

_ = AutoModelForCausalLM.from_pretrained("gpt2", cache_dir="model_cache")
```

2. Put the following code in `test.py`:

```python
from transformers import pipeline

_ = pipeline("text-generation", model="gpt2", model_kwargs={"cache_dir": "model_cache"})
```

3. Run `time TRANSFORMERS_OFFLINE=1 python test.py` to force the cache to be hit
4. See that the following exception is returned:

```console
Cannot find the requested files in the cached path and outgoing traffic has been disabled. To enable model look-ups and downloads online, set 'local_files_only' to False.
Traceback (most recent call last):
  File "test.py", line 3, in <module>
    _ = pipeline("text-generation", model="gpt2", model_kwargs={"cache_dir": "model_cache"})
  File "venv/lib/python3.7/site-packages/transformers/pipelines/__init__.py", line 409, in pipeline
    model, model_classes=model_classes, config=config, framework=framework, revision=revision, task=task
  File "venv/lib/python3.7/site-packages/transformers/pipelines/base.py", line 136, in infer_framework_load_model
    model = model_class.from_pretrained(model, **kwargs)
  File "venv/lib/python3.7/site-packages/transformers/utils/dummy_tf_objects.py", line 991, in from_pretrained
    requires_backends(cls, ["tf"])
  File "venv/lib/python3.7/site-packages/transformers/file_utils.py", line 612, in requires_backends
    raise ImportError("".join([BACKENDS_MAPPING[backend][1].format(name) for backend in backends]))
ImportError:
TFGPT2LMHeadModel requires the TensorFlow library but it was not found in your environment. Checkout the instructions on the
installation page: https://www.tensorflow.org/install and follow the ones that match your environment.
```

(I edited the stack trace to remove the parts of the path outside the virtual environment.)

## Expected behavior

There should be no output because the model should be loaded from the cache without issues.
You could do it like so:

```py
from transformers import AutoModelForCausalLM, pipeline, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("gpt2", cache_dir="model_cache")
tokenizer = AutoTokenizer.from_pretrained("gpt2", cache_dir="model_cache")

_ = pipeline("text-generation", model=model, tokenizer=tokenizer)
```<|||||>While that's a good temporary workaround (I'm currently using a different one), I was hoping for a longer term solution so [`pipeline()`](https://huggingface.co/transformers/v4.8.0/main_classes/pipelines.html#transformers.pipeline) works as the docs say:

> **model_kwargs** – Additional dictionary of keyword arguments passed along to the model’s `from_pretrained(..., **model_kwargs)` function.

`model_kwags` actually used to work properly, at least when the `framework` parameter was set, but #12025 broke it. #12449 should fix it, although it doesn't address the issue that #12025 broke this behavior without any tests failing.<|||||>Great, thank you for bringing me up to speed! Indeed, I hadn't realized this was a regression - your PR looks great, let's discuss over there.