wav2vec with LM leads to CPU OOM
## Environment info
I used the code described in the blog post for [wav2vec with LMs](https://huggingface.co/blog/wav2vec2-with-ngram). Also posted this [on the forum](https://discuss.huggingface.co/t/wav2vec-with-new-lm-causing-cpu-oom/14069) and will update both.

### Who can help
@patrickvonplaten, @anton-l and whoever can help.

Models:
Wav2Vec2ProcessorWithLM with pyctcdecode

## To reproduce

Steps to reproduce the CPU OOM via GPU (maybe increase range):

```python
from datasets import load_dataset
from transformers import Wav2Vec2ProcessorWithLM, Wav2Vec2ForCTC
import torch

dataset = load_dataset("hf-internal-testing/librispeech_asr_demo", "clean", split="validation")
processor = Wav2Vec2ProcessorWithLM.from_pretrained("patrickvonplaten/wav2vec2-base-100h-with-lm")
model = Wav2Vec2ForCTC.from_pretrained("patrickvonplaten/wav2vec2-base-100h-with-lm").to("cuda")
print("* * * * * * loaded models")

for i in range(200):
    audio_sample = dataset[i]
    print(" * * * * Sample: ", i)
    inputs = processor(audio_sample["audio"]["array"], sampling_rate=audio_sample["audio"]["sampling_rate"], return_tensors="pt").to("cuda")
    with torch.no_grad():
      logits = model(**inputs).logits
    transcription = processor.batch_decode(logits.cpu().numpy()).text
    print(transcription[0].lower())
```

and the same for just CPU:

```python
from datasets import load_dataset
from transformers import Wav2Vec2ProcessorWithLM, Wav2Vec2ForCTC
import torch

dataset = load_dataset("hf-internal-testing/librispeech_asr_demo", "clean", split="validation")
processor = Wav2Vec2ProcessorWithLM.from_pretrained("patrickvonplaten/wav2vec2-base-100h-with-lm")
model = Wav2Vec2ForCTC.from_pretrained("patrickvonplaten/wav2vec2-base-100h-with-lm")
print("* * * * * * loaded models")

for i in range(200):
    audio_sample = dataset[i]
    print(" * * * * Sample: ", i)
    inputs = processor(audio_sample["audio"]["array"], sampling_rate=audio_sample["audio"]["sampling_rate"], return_tensors="pt")
    with torch.no_grad():
      logits = model(**inputs).logits
    transcription = processor.batch_decode(logits.numpy()).text
    print(transcription[0].lower())
```

No problems with the classic processor. Just the new processor seems to leak some memory ...
https://github.com/huggingface/transformers/blob/v4.15.0/src/transformers/models/wav2vec2_with_lm/processing_wav2vec2_with_lm.py#L307

I believe closing the pool process here should solve the issue !
(`pool.close()`).

For maintainer info, I also ran into this issue and it seems a few memory leaks were reported in ctcdecode/kenlm on this type of problem.<|||||>Kudos to you, Manuel. Closing the pool solves the problem, feel free to submit a PR or @patrickvonplaten?<|||||>I believe the issue should still be open until the problem is fixed but will let @patrickvonplaten  decide whether it's the cleanest way around the problem<|||||>Good point, will leave this open<|||||>Turns out issue was solved 6 days ago actually (on the master branch). Until the patch is issued, the library can thus be built from source.<|||||>You are right, I did clone the master branch, but 7 days ago ... should have updated. Closing again before I forget to do so.<|||||>Ah yeah the pool close() one. We indeed need to install transformers from scratch for this one!