Hubert-base Model with new tokenizer is not converging 
## Environment info

- transformers version: 4.12.2
- Platform: Mac
- Python version: 3.7
- PyTorch version (GPU?): 1.9
- Tensorflow version (GPU?): No
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

I just run simple code to load Hubert pretrained base model

```

vocab_dict = dict()
vocab_dict['[PAD]'] =  0
vocab_dict['[UNK]'] =  1
vocab_dict['|'] =  2
vocab_dict['a'] =  len(vocab_dict)
vocab_dict['b'] =  len(vocab_dict)
vocab_dict['c'] =  len(vocab_dict)
vocab_dict['d'] =  len(vocab_dict)
vocab_dict['e'] =  len(vocab_dict)
vocab_dict['f'] =  len(vocab_dict)
vocab_dict['g'] =  len(vocab_dict)
vocab_dict['h'] =  len(vocab_dict)
vocab_dict['i'] =  len(vocab_dict)
vocab_dict['j'] =  len(vocab_dict)
vocab_dict['k'] =  len(vocab_dict)
vocab_dict['l'] =  len(vocab_dict)
vocab_dict['m'] =  len(vocab_dict)
vocab_dict['n'] =  len(vocab_dict)
vocab_dict['o'] =  len(vocab_dict)
vocab_dict['p'] =  len(vocab_dict)
vocab_dict['q'] =  len(vocab_dict)
vocab_dict['r'] =  len(vocab_dict)
vocab_dict['s'] =  len(vocab_dict)
vocab_dict['t'] =  len(vocab_dict)
vocab_dict['u'] =  len(vocab_dict)
vocab_dict['v'] =  len(vocab_dict)
vocab_dict['w'] =  len(vocab_dict)
vocab_dict['x'] =  len(vocab_dict)
vocab_dict['y'] =  len(vocab_dict)
vocab_dict['z'] =  len(vocab_dict)
print("Vocab size:" + str(len(vocab_dict)))

import json
with open('vocab.json', 'w') as vocab_file:
    json.dump(vocab_dict, vocab_file)

tokenizer = Wav2Vec2CTCTokenizer("./vocab.json", unk_token="[UNK]", pad_token="[PAD]", word_delimiter_token="|")
feature_extractor = Wav2Vec2FeatureExtractor(feature_size=1, sampling_rate=16000, padding_value=0.0, do_normalize=True, return_attention_mask=False)
PROCESSOR = Wav2Vec2Processor(feature_extractor=feature_extractor, tokenizer=tokenizer)
model = HubertForCTC.from_pretrained('facebook/hubert-base-ls960', ctc_loss_reduction="mean",pad_token_id=PROCESSOR.tokenizer.pad_token_id,vocab_size=len(PROCESSOR.tokenizer))

model.freeze_feature_extractor()
#model.gradient_checkpointing_enable()

```
This model is not able to coverage, I have been training for a day, but the loss is not decreasing + prediction is always giving '0' 

Must be some silly mistake, as the same code run if I use `hubert-large-ls960-ft`, where I don't touch the tokenizer. I suspect something is wrong with my tokenizer initialization as the rest of the code works for the fine-tuned model where we have a pretrained tokenizer.


@patrickvonplaten, @anton-l: Do you have any idea


I have checked the labels which i feed in and seems to be ok

```
ADAM ANDREASSON
[3, 6, 3, 15, 2, 3, 16, 6, 20, 7, 3, 21, 21, 17, 16]
```<|||||>After 20,000 iterations, it only learns char '3', which must be quite unusual or wrong configuration somewhere. 

My training and test data is same for simplicity.

```
***** Running Evaluation *****
  Num examples = 2
  Batch size = 16
                                                                                                                                                                                  
[[3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3                                                                                   | 0/1 [00:00<?, ?it/s]
  3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
  3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
  3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
  3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3]
 [3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
  3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
  3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
  3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
  3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3]]
['A', 'A']
['ADAM ANDREASSON', 'ADAM ANDREASSON']
1.0
{'eval_loss': 4.131505489349365, 'eval_score': 1.0, 'eval_runtime': 0.442, 'eval_samples_per_second': 4.524, 'eval_steps_per_second': 2.262, 'epoch': 24400.0}
 24%|███████████████████████████████▍                                                                                                 | 24400/100000 [7:02:28<21:13:01,  1.01s/it]
Saving model checkpoint to ./output/checkpoint-24400

```<|||||>Hey @harrypotter90 - could you upload your model somewhere on the hub so that I can compare the model's config and the model's tokenizer file? 

I think that the tokenizer config is wrong<|||||>Thanks @patrickvonplaten : I have uploaded it here : https://huggingface.co/HarryPotter09/hubert-base-tokenizer<|||||>I have compared hubert-base new tokenizer with the hubert-large pretrained tokenizer (which is working) and feature extractors, couldn't find anything wrong.<|||||>More information, something wrong with the `hubert-base` model, as soon as I just change the model to `"facebook/hubert-large-ll60k"` same code works.

Thoughts ?<|||||>@harrypotter90 - I would need your model in the same repo id as well to be able to compare the two. Could you please upload your model as well? (`pytorch_model.bin` + `config.json`)<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>Hi there, I'm having a similar problem that the hubert-base model training does not work for ASR, but the large model works. @harrypotter90 did you solve the problem in the end? Also, I notice that for base model extractor the `return_attention_mask` is set to False (it is true for large and xlarge models). Is there a reason for that?
(Setting that to true still does not solve the problem though.)