Fix ASR pipelines from local directories with wav2vec models that have language models attached
# What does this PR do?

Fixes #15589.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case: https://discord.com/channels/879548962464493619/930507803339137024/939885949633064960
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@patrickvonplaten, @Narsil 

_The documentation is not available anymore as the PR was closed or merged._<|||||>Sure thing, I'll add the test tomorrow 👍🏼 <|||||>@versae you need to protect your test with @require_torch  or @require_pyctcdecode (not sure what was failing)<|||||>@Narsil, thanks! Sorry for all the mess with the commits. I'm trying to figure out what's going on. You guys have a pretty serious CI setup and I don't even have a local GPU. I'll mark it as WIP until I think is ready.<|||||>@versae No worries, it is indeed a pretty involved test suite.

If you want to run it, you don't need to run it entirely though:

```
RUN_PIPELINE_TESTS=1 pytest -sv tests/test_pipelines_automatic_speech_recognition.py::AutomaticSpeechRecognitionPipelineTests::test_with_local_lm_fast
```
or
```
RUN_PIPELINE_TESTS=1  pytest -sv tests/test_pipelines_automatic_speech_recognition.py
```

These only run on CPU for instance.

Should run only the test you are creating, making it much easier to debug. And don't hesitate to ask for help if you don't understand what's happening the `@require` are easy to miss, and hard to understand once they occur for instance.

Thanks for the PR! <|||||>Alright, I think it's ready now. Thanks y'all for all the awesome libraries you open source!