TFWav2Vec2ForCTC breaks when not run eagerly
### System Info

`transformers` version: 4.21.0.dev0
- Platform: Linux-5.13.0-48-generic-x86_64-with-glibc2.31
- Python version: 3.7.13
- PyTorch version (GPU?): 1.11.0 (False)
- Tensorflow version (GPU?): 2.9.1 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@Rocketknight1 @gante

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Colab link to reproduce: https://colab.research.google.com/drive/1GxQtnDLaDFooG8t2uAIR-2GnS2RFod_j?usp=sharing

When model is compiled with:
model.compile(optimizer=optimizer ,run_eagerly = False, metrics =[compute_wer])

the code errors out with:
```
ValueError: Label values must be <= vocab_size: 30
```
However, it works perfectly when `run_eagerly = True`.


### Expected behavior

Training should happen fine even when not run eagerly.
Hi @Sreyan88 👋  

That is a big reproduction script 😅 To ensure we can provide quality support, a short reproduction script goes a long way.

I haven't run the code, but my suspicion goes to the line where the model is defined (`model = TFWav2Vec2ForCTC.from_pretrained(MODEL_CHECKPOINT,vocab_size=len(processor.tokenizer), pad_token_id=processor.tokenizer.pad_token_id,apply_spec_augment=False, from_pt = True)`). Try removing the `pad_token_id` keyword argument here -- the data pipeline knows which token is the padding token from the tokenizer. It might be creating a new token with the argument, which causes the `vocab_size` error.

Let me know if it works! If it does, it probably means that our `pad_token_id` is not working properly -- I've been seeing similar errors lately.<|||||>Hi @gante ,

The error persists even after removing it!

I would please request you to run the script twice by toggling run_eagerly boolean in this line:

model.compile(optimizer=optimizer ,run_eagerly = False, metrics =[compute_wer])

which is in the Building and Compiling the Model section. When `run_eagerly = True`, the training does not throw any error!

Apologies for the script but I am about to push it to Keras examples soon so it's indeed a detailed one and I thought of explaining every step because I was not sure about the error. The script takes about 2 mins to run on colab!<|||||>Hi @Sreyan88 it is not about the run time, but about being able to pin the issue. We don't have the bandwidth to help the community with all requests and bugs, so we request some help from the community to create short scripts for bug reproducibility.

Without it, I'm afraid this issue will not jump high on my priority list :)<|||||>No problem! I have modified the script and deleted comments/explanations. Currently, all code blocks are just the ones absolutely necessary. Hope you find some time to look at it! and please also suggest if I should clean more!

Thank You!<|||||>Hi @gante ,

Good day! Any leads to this? I tried but couldn't figure out the exact issue. :( <|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.<|||||>Hi @gante ,

Good day! Could you please re-open this as it wasn't solved? Thank You!