TF Camembert not improving over epochs
# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...):
`jplu/tf-camembert-base`

Language I am using the model on (English, Chinese ...):
French

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1.  Get a custom multi-class dataset with imbalanced data
2. Train TFCamembertForSequenceClassification on this dataset
3. Try with and without `class_weight` or under-sample biggest classes (accuracy and loss change but still don't improve over epochs)

```python
import tensorflow as tf
from transformers import TFCamembertForSequenceClassification, CamembertTokenizer

model = TFCamembertForSequenceClassification.from_pretrained("jplu/tf-camembert-base", num_labels=len(labels))
tokenizer = CamembertTokenizer.from_pretrained("jplu/tf-camembert-base")

model.compile(
    optimizer=tf.keras.optimizers.Adam(learning_rate=3e-5, epsilon=1e-08, clipnorm=1.0),
    loss=tf.keras.losses.CategoricalCrossentropy(from_logits=True),
    metrics=["accuracy"],
)

model.fit(
    custom_generator(), # generator yield encoded sample (by tokenizer) and encoded label (by OneHotEncoder)
    epochs=10,
    max_queue_size=2,
    steps_per_epoch=25,
    #class_weight=class_weights,
    validation_data=custom_generator(),
    validation_steps=4
)
```

## Expected behavior

The classifier should improve over each epochs. In this case it stay at the same accuracy and loss, it just vary with more or less 5% accuracy.
To compare, I tried to run the same code but with `TFFlaubertForSequenceClassification.from_pretrained("jplu/tf-flaubert-base-cased")` and it worked as expected.

## Environment info
     
- `transformers` version: 2.5.1
- Platform: Linux-4.9.0-12-amd64-x86_64-with-debian-9.12 (Google AI Platform)
- Python version: 3.7.6
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): 2.1.0 (True)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

For information, I already posted this problem on [Stack Overflow](https://stackoverflow.com/questions/60761761/hugging-face-transformer-classifier-fail-on-imbalance-dataset) which lead me here.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>@bourrel 
Why do you use `jplu/tf-flaubert-base-cased`? (https://huggingface.co/jplu/tf-flaubert-base-cased)
Any particular reason not to use `flaubert/flaubert_base_cased`? (https://huggingface.co/flaubert/flaubert_base_cased)<|||||>It was 2 years ago, I don't remember sorry 😅 