Logits size does not match vocabulary size when fine-tuning Hubert large with pyctcdecode
### System Info

```shell
- `transformers` version: 4.18.0
- Platform: Linux-5.13.0-44-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.5.1
- PyTorch version (GPU?): 1.11.0+cu113 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No
```


### Who can help?

@patrickvonplaten

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Hello, I have a very similar issue [like this one](https://github.com/huggingface/transformers/issues/15392) for Hubert large.  I got this logits size value error when fine-tuning Hubert model with pyctcdecode

Tried with the [previous comment on this issue](https://github.com/huggingface/transformers/issues/15392#issuecomment-1024905216). Setting both `eos_token` and `bos_token` did not work and returned the same error.

Here is the code snippet I used for a single audio file processing and debugging

```python
processor = Wav2Vec2Processor.from_pretrained(
      "facebook/hubert-large-ls960-ft",
      eos_token=None, bos_token=None)
  tokenizer_vocab_dict = processor.tokenizer.get_vocab()
  tokenizer_vocab_lowercase = {k.lower(): v for k,v in tokenizer_vocab_dict.items()}
  with open("../vocab/vocab.json", "w", encoding="utf-8") as f: 
      f.write(json.dumps(tokenizer_vocab_lowercase, ensure_ascii=False))
  processor.tokenizer = Wav2Vec2CTCTokenizer("../vocab/vocab.json")
  processor.save_pretrained("../processor-lm")

ngram = 3
from pyctcdecode import build_ctcdecoder
from transformers import Wav2Vec2FeatureExtractor, Wav2Vec2CTCTokenizer, Wav2Vec2ProcessorWithLM
cus_processor = Wav2Vec2Processor.from_pretrained("facebook/hubert-large-ls960-ft")
vocab_dict = cus_processor.tokenizer.get_vocab()
sorted_vocab_dict = {
    k.lower(): v for k, v in sorted(vocab_dict.items(), key=lambda item: item[1])}
decoder = build_ctcdecoder(
    labels=list(sorted_vocab_dict.keys()),
    kenlm_model_path=f"../{ngram}gram_correct.arpa",
)
processor = Wav2Vec2ProcessorWithLM(
    feature_extractor=cus_processor.feature_extractor,
    tokenizer=cus_processor.tokenizer,
    decoder=decoder
)

model = AutoModelForCTC.from_pretrained(
    "facebook/hubert-large-ls960-ft", 
    ctc_loss_reduction="mean", 
    pad_token_id=processor.tokenizer.pad_token_id)
model.freeze_feature_encoder()

# test run
s = "sentence from audio file"
audio_input, sample_rate = sf.read("audio_loc")
inputs = processor(
    audio_input, sampling_rate=sample_rate, return_tensors="pt")
with processor.as_target_processor():
    labels = np.asarray(processor(s, padding=True).input_ids)
print(f"target processor logit shape: {labels.shape}")
with torch.no_grad():
    logits = model(**inputs).logits
print(f"logit shape returned by the model: {logits.shape}")
transcription = processor.batch_decode(logits.numpy()).text[0]
text = processor.decode(labels)
```

### Expected behavior

```python
with processor.as_target_processor():
    labels = np.asarray(processor(s, padding=True).input_ids)
```

`labels` should be a vector with size of 32, then it can be sent to the `map_to_result` and `compute_metrics' functions mentioned in [this blog](https://huggingface.co/blog/fine-tune-wav2vec2-english) with some changes listed below. 

```python
def map_to_result(batch):
    """
    batchfy and map the hidden states into transcript

    :param batch: _description_
    :type batch: _type_
    """
    #model.to("cuda")
    inputs = processor(
      batch["speech"],
      sampling_rate=batch["sampling_rate"],
      return_tensors="pt")
    with torch.no_grad():
        logits = model(**inputs).logits
    batch["pred_str"] = processor.batch_decode(logits.numpy()).text[0]
    batch["text"] = processor.decode(batch["labels"])
    return batch


def compute_metrics(pred):
    """
    batchfy and compute the WER metrics

    :param pred: _description_
    :type pred: _type_
    :return: _description_
    :rtype: _type_
    """
    wer_metric = load_metric("wer")
    transcription = processor.batch_decode(pred.predictions.numpy())
    pred_str = transcription.text[0]
    # we do not want to group tokens when computing the metrics
    label_str = processor.decode(pred.label_ids.numpy())
    wer = wer_metric.compute(predictions=pred_str, references=label_str)
    return {"wer": wer}
```
But it returned a vector with size of 45, thus `text = processor.decode(np.asarray(labels))` got the ValueError for logits size unmatched. As results, `map_to_result` and `compute_metrics` also cannot be run during the fine-tuning process. I was wondering if it needs a similar fix as mentioned in [this issue](https://github.com/huggingface/transformers/issues/15392). If not, do you have any suggestions or commends on solving this issue? Thanks in advance.


Hey @changyeli,

I sadly cannot rerun your code to reproduce the error. Could you try to send a **minimal**, **fully reproducible** code snippet?
E.g. I don't have access to `f"../{ngram}gram_correct.arpa"`<|||||>Hey @patrickvonplaten unfortunately it's a protected corpus so I can't upload the full file here. Will random/first 20 lines from this file work in this case?<|||||>Please don't upload the whole `corpus` - I can try to help, if it's just a some dummy examples. It would be amazing if you could try to make the reproducible code snippet to run as fast as possible and to be as short as possible<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.