Tutorial on token classification throws casting error in Tensorflow 2.11
### System Info

- `transformers` version: 4.25.1
- Platform: Linux-5.15.0-56-generic-x86_64-with-glibc2.35
- Python version: 3.9.0
- Huggingface_hub version: 0.11.1
- PyTorch version (GPU?): not installed (NA)
- Tensorflow version (GPU?): 2.11.0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@ArthurZucker @younesbelkada, the tutorial at `https://huggingface.co/docs/transformers/tasks/token_classification` throws the following  error in Tensorflow 2.11 but not in Tensorflow 2.9: 

`(0) UNIMPLEMENTED:  Cast string to float is not supported
	 [[{{node Cast_1}}]]
  (1) CANCELLED:  Function was cancelled before it was started
0 successful operations.
`

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

The tutorial at `https://huggingface.co/docs/transformers/tasks/token_classification` for Tensorflow

### Expected behavior

Training should start, but it does not.
Can you please specify where is the error happen? Which step?<|||||>The error is thrown after `model.fit`<|||||>Okay I'll have a look<|||||>Also cc @Rocketknight1 <|||||>Yeah, I should probably take this one. Investigating!<|||||>Managed to reproduce it.<|||||>This is actually a problem with our `AdamWeightDecay`, likely caused by the change in Keras optimizers in 2.11. Figuring out a fix now.