[pipeline] Explicitly set framework is ignored
### System Info

- `transformers` version: 4.30.2

Omitting the rest because they aren't really relevant. Can submit later if needed.

### Who can help?

@Narsil

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
from transformers import pipeline
from transformers import WhisperProcessor, WhisperForConditionalGeneration

processor = WhisperProcessor.from_pretrained("openai/whisper-tiny")
model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-tiny")

# Using this just to make sure an error if thrown if pipeline tries to check using module instead of specified framework.
class FakeWhisper:
    def __getattr__(self, item):
        return model.__getattr__(item)

pipe = pipeline(
    "automatic-speech-recognition",
    model=FakeWhisper(),
    tokenizer=processor.tokenizer,
    feature_extractor=processor.feature_extractor,
    chunk_length_s=30,
    device=model.device,
    framework="pt",
)
```

The above code raises this error:
```
TypeError: Could not infer framework from class <class '__main__.FakeWhisper'>.
```

### Expected behavior

When specifying the framework explicitely, there is no need to infer it from the module of the model class, as mentioned here:
https://github.com/huggingface/transformers/blob/main/src/transformers/pipelines/__init__.py#L784-L796

But then, inside `infer_framework_load_model`, `infer_framework` is called regardless of the value of the `framework` parameter:
https://github.com/huggingface/transformers/blob/main/src/transformers/pipelines/base.py#L280