Decoding with language model issue 
### System Info

- `transformers` version: 4.26.1
- Platform: Windows-10-10.0.22621-SP0
- Python version: 3.10.0
- Huggingface_hub version: 0.12.1
- PyTorch version (GPU?): 1.11.0+cpu (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
from datasets import load_dataset, Dataset,Audio
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
from pyctcdecode import build_ctcdecoder
from transformers import Wav2Vec2ProcessorWithLM,AutoProcessor
import torch

def process_audio(filepath,model_id):

    print(filepath)

    model = Wav2Vec2ForCTC.from_pretrained("media/model/transformer/")
    processor = Wav2Vec2Processor.from_pretrained("media/model/transformer/")

    
    audio_file_path = [filepath]
    audio_data = Dataset.from_dict({"audio": audio_file_path}).cast_column("audio", Audio())
    audio_data = audio_data.cast_column("audio", Audio(sampling_rate=16_000))

    def prepare_dataset(batch):
        audio = batch["audio"]
        batch["input_values"] = processor(audio["array"], sampling_rate=audio["sampling_rate"]).input_values[0]
        batch["input_length"] = len(batch["input_values"])
        return batch

    test_dataset = audio_data.map(prepare_dataset)

    input_dict = processor(test_dataset[0]["input_values"], return_tensors="pt", padding=True)
    logits = model(input_dict.input_values).logits

    if (int(model_id)==1):
        pred_ids = torch.argmax(logits, dim=-1)[0]
        return processor.decode(pred_ids)
    
    elif (int(model_id)==2):
        processorLM = AutoProcessor.from_pretrained("media/model/transformer/")
        vocab_dict = processorLM.tokenizer.get_vocab()
        sorted_vocab_dict = {k.lower(): v for k, v in sorted(vocab_dict.items(), key=lambda item: item[1])}
        
       
        decoder = build_ctcdecoder(labels=list(sorted_vocab_dict.keys()), kenlm_model_path="media/model/transformer/5gram_correct.arpa",)
        processor_with_lm = Wav2Vec2ProcessorWithLM(feature_extractor=processor.feature_extractor,tokenizer=processorLM.tokenizer,decoder=decoder)

       
        return processor_with_lm.batch_decode(logits.detach().numpy()).text
```

### Expected behavior

I have been using the below code to decode my wav2vac2 Automatic Speech Recognition Model and it works fine on the google Collab but when I shift to Django to deploy in website the build_ctcdecoder give UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 174: character maps to <undefined> error

processorLM = AutoProcessor.from_pretrained("media/model/transformer/")
        vocab_dict = processorLM.tokenizer.get_vocab()
        sorted_vocab_dict = {k.lower(): v for k, v in sorted(vocab_dict.items(), key=lambda item: item[1])}
        decoder = build_ctcdecoder(labels=list(sorted_vocab_dict.keys()), kenlm_model_path="media/model/transformer/5gram_correct.arpa",)
        processor_with_lm = Wav2Vec2ProcessorWithLM(feature_extractor=processor.feature_extractor,tokenizer=processorLM.tokenizer,decoder=decoder)
        return processor_with_lm.batch_decode(logits.detach().numpy()).text
        
        Note: I have using Dzongkha Language which is based on utf-8
cc @sanchit-gandhi <|||||>Hey @ngawang88, sorry for the late reply, but unfortunately the code snippet is currently un-reproducible (it leverages a local model which I cannot load). Without the full stack trace I can't see precisely where the code is failing. If you're able to share a reproducible code snippet (e.g. one that loads a model from the HF Hub) and share the full stack trace, I'd be able to have a deeper look. Thanks<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.