loss is nan, for training on MNLI dataset
## 🐛 Bug

Model I am using (Bert, XLNet....):
** Bert **
bert_model = TFBertForSequenceClassification.from_pretrained("bert-base-cased")

Language I am using the model on (English, Chinese....):
** English **
 
The problem arise when using:
https://medium.com/tensorflow/using-tensorflow-2-for-state-of-the-art-natural-language-processing-102445cda54a
for MNLI

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: ** MNLI **

## To Reproduce

Steps to reproduce the behavior:

1. create  bert_validation_matched_dataset
example = list(bert_validation_matched_dataset.__iter__())[0]
example
({'attention_mask': <tf.Tensor: id=5022668, shape=(64, 128), dtype=int32, numpy=
  array([[1, 1, 1, ..., 0, 0, 0],
         [1, 1, 1, ..., 0, 0, 0],
         [1, 1, 1, ..., 0, 0, 0],
         ...,
         [1, 1, 1, ..., 0, 0, 0],
         [1, 1, 1, ..., 0, 0, 0],
         [1, 1, 1, ..., 0, 0, 0]], dtype=int32)>,
  'input_ids': <tf.Tensor: id=5022669, shape=(64, 128), dtype=int32, numpy=
  array([[  101,  8147,  1218, ...,     0,     0,     0],
         [  101, 21637,  1103, ...,     0,     0,     0],
         [  101,  1109,  2570, ...,     0,     0,     0],
         ...,
         [  101,  1109,  7271, ...,     0,     0,     0],
         [  101,  7947,  1685, ...,     0,     0,     0],
         [  101,  1130,  1103, ...,     0,     0,     0]], dtype=int32)>,
  'token_type_ids': <tf.Tensor: id=5022670, shape=(64, 128), dtype=int32, numpy=
  array([[0, 0, 0, ..., 0, 0, 0],
         [0, 0, 0, ..., 0, 0, 0],
         [0, 0, 0, ..., 0, 0, 0],
         ...,
         [0, 0, 0, ..., 0, 0, 0],
         [0, 0, 0, ..., 0, 0, 0],
         [0, 0, 0, ..., 0, 0, 0]], dtype=int32)>},
 <tf.Tensor: id=5022671, shape=(64,), dtype=int64, numpy=
 array([2, 1, 0, 1, 1, 1, 1, 0, 0, 0, 1, 1, 0, 0, 0, 1, 0, 2, 0, 2, 0, 1,
        2, 1, 1, 2, 2, 1, 0, 0, 2, 0, 0, 0, 2, 0, 0, 0, 0, 1, 0, 0, 2, 0,
        2, 2, 1, 2, 2, 2, 2, 2, 2, 0, 2, 0, 0, 2, 1, 2, 2, 1, 2, 0])>)

2. create bert_train_dataset such that
example = list(bert_train_dataset.__iter__())[0]
example
({'attention_mask': <tf.Tensor: id=5023289, shape=(32, 128), dtype=int32, numpy=
  array([[1, 1, 1, ..., 0, 0, 0],
         [1, 1, 1, ..., 0, 0, 0],
         [1, 1, 1, ..., 0, 0, 0],
         ...,
         [1, 1, 1, ..., 0, 0, 0],
         [1, 1, 1, ..., 0, 0, 0],
         [1, 1, 1, ..., 0, 0, 0]], dtype=int32)>,
  'input_ids': <tf.Tensor: id=5023290, shape=(32, 128), dtype=int32, numpy=
  array([[  101,   144,  2312, ...,     0,     0,     0],
         [  101,   107,  1192, ...,     0,     0,     0],
         [  101,   149, 10844, ...,     0,     0,     0],
         ...,
         [  101,  1109,  2350, ...,     0,     0,     0],
         [  101,  1262,  1173, ...,     0,     0,     0],
         [  101,  1105,  1128, ...,     0,     0,     0]], dtype=int32)>,
  'token_type_ids': <tf.Tensor: id=5023291, shape=(32, 128), dtype=int32, numpy=
  array([[0, 0, 0, ..., 0, 0, 0],
         [0, 0, 0, ..., 0, 0, 0],
         [0, 0, 0, ..., 0, 0, 0],
         ...,
         [0, 0, 0, ..., 0, 0, 0],
         [0, 0, 0, ..., 0, 0, 0],
         [0, 0, 0, ..., 0, 0, 0]], dtype=int32)>},
 <tf.Tensor: id=5023292, shape=(32,), dtype=int64, numpy=
 array([2, 1, 0, 0, 1, 2, 1, 1, 0, 2, 0, 1, 1, 2, 2, 2, 0, 2, 1, 1, 0, 1,
        1, 2, 0, 1, 1, 0, 2, 2, 2, 1])>)

3. Fit the model
bert_history = bert_model.fit(bert_train_dataset, epochs=1, validation_data=bert_validation_matched_dataset)

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior
The expected behavior is that loss should not be nan.
Fine-tuning BERT on MNLI
      3/Unknown - 4s 1s/step - loss: nan - accuracy: 0.2031

## Environment
colab

Hi, there seems to be a problem indeed. Do you mind sharing a link to a notebook so that I can see what's wrong?<|||||>Is it related to the issue you opened 3 days ago #1704?<|||||>Yes, its is the same, but I though that was my mistake and closed it. I will provide a link to a notebook to see the code. Here it is: https://colab.research.google.com/drive/1WlhgpgERppzZ5HK6E_iappn_Kap9EKOA<|||||>Hi, is there any progress on that?<|||||>Hi, I believe the issue stems from the fact that MNLI has three labels, whereas MRPC only has two. Your model must therefore be instantiated differently:

```py
from transformers import TFBertForSequenceClassification, BertConfig

config = BertConfig.from_pretrained("bert-base-cased", num_labels=3)
bert_model = TFBertForSequenceClassification.from_pretrained("bert-base-cased", config=config)
```
You can see how the configurations and models interact together in the [quickstart part of our documentation](https://huggingface.co/transformers/quickstart.html).

Your model should have a correct loss after specifying the number of labels. We've fixed an issue with the way the `tensorflow_datasets` datasets were handled recently, so I recommend installing directly from source:

```
!pip install git+https://github.com/huggingface/transformers
```

You won't need to specify the `label_list` like you have done in your notebook if you install from the master branch.

If you want a fully functional script that works will all glue tasks, I recommend taking a look at [examples/run_tf_glue.py](https://github.com/huggingface/transformers/blob/master/examples/run_tf_glue.py)
<|||||>Thank you very much! This is very helpful. I see that in the script is an if ``` if TASK == "mrpc": ``` (only for mrpc). Also Is there a similar script for pytorch as well? Seems that this issue is ready for closing.<|||||>Sure, no problem! That flag is because we're then loading the saved model in pytorch to show how easy it is to switch between pytorch/tensorflow. We're showcasing the predictions made by the PyTorch model that was trained in TensorFlow, but we're only showcasing it for MRPC.

We'll update this script sometimes in the next few weeks to evaluate all the tasks.<|||||>Following your recommendations, now the loss takes value. 

For your info, there is an issue with roberta. 
But for that could be opened a new issue, if it is not a fault from my side.
```
Fine-tuning RoBERTa on MNLI
WARNING:tensorflow:Gradients do not exist for variables ['tf_roberta_for_sequence_classification/roberta/pooler/dense/kernel:0', 'tf_roberta_for_sequence_classification/roberta/pooler/dense/bias:0'] when minimizing the loss.

WARNING:tensorflow:Gradients do not exist for variables ['tf_roberta_for_sequence_classification/roberta/pooler/dense/kernel:0', 'tf_roberta_for_sequence_classification/roberta/pooler/dense/bias:0'] when minimizing the loss
<|||||>Thanks for letting us know, we'll look into that!<|||||>Any updates on this one? Still getting the same error when running the `2.2.2` release.

```
WARNING:tensorflow:Gradients do not exist for variables ['tf_roberta_for_sequence_classification/roberta/pooler/dense/kernel:0', 'tf_roberta_for_sequence_classification/roberta/pooler/dense/bias:0'] when minimizing the loss
````<|||||>It's not an error, it just means it's not updating those variables. Those variables (pooler) are not used when doing sequence classification.<|||||>Those weights are part of `trainable_weights` which makes little sense if they are not used when doing sequence classification. Maybe just remove them from trainable weights and avoid any confusion in the future?<|||||>> It's not an error, it just means it's not updating those variables. Those variables (pooler) are not used when doing sequence classification.



> Those weights are part of `trainable_weights` which makes little sense if they are not used when doing sequence classification. Maybe just remove them from trainable weights and avoid any confusion in the future?

We are getting a similar issue here

https://github.com/huggingface/transformers/issues/2256

We are using just the Bert base model to get a vector representation from a piece of text .

Since

>        **pooler_output**: ``tf.Tensor`` of shape ``(batch_size, hidden_size)``
            Last layer hidden-state of the first token of the sequence (classification token)
            further processed by a Linear layer and a Tanh activation function. The Linear
            layer weights are trained from the next sentence prediction (classification)
            objective during Bert pretraining. This output is usually *not* a good summary
            of the semantic content of the input, you're often better with averaging or pooling
            the sequence of hidden-states for the whole input sequence.

That would mean that the warning I am getting is just because we are not using the pooler output?<|||||>I'm getting similar warning on roberta large architecture training on TPU for sequence classification for num_labal = 4: 

It is mostly the same code  as  except for data preparation phase and num of labels and the model is roberta-large:
https://colab.research.google.com/drive/1yWaLpCWImXZE2fPV0ZYDdWWI8f52__9A#scrollTo=G_chN1Sy3IXn

I defined my model as : 

`assert len(label_list) == 4`

`def model_fn():`
`    return TFRobertaForSequenceClassification.from_pretrained("roberta-large",config =RobertaConfig.from_pretrained("roberta-large",num_labels=len(label_list) ))`

The warning i get are : 

WARNING:tensorflow:Gradients do not exist for variables ['tf_roberta_for_sequence_classification_2/roberta/pooler/dense/kernel:0', 'tf_roberta_for_sequence_classification_2/roberta/pooler/dense/bias:0'] when minimizing the loss.
WARNING:tensorflow:Gradients do not exist for variables ['tf_roberta_for_sequence_classification_2/roberta/pooler/dense/kernel:0', 'tf_roberta_for_sequence_classification_2/roberta/pooler/dense/bias:0'] when minimizing the loss.
WARNING:tensorflow:Gradients do not exist for variables ['tf_roberta_for_sequence_classification_2/roberta/pooler/dense/kernel:0', 'tf_roberta_for_sequence_classification_2/roberta/pooler/dense/bias:0'] when minimizing the loss.
WARNING:tensorflow:Gradients do not exist for variables ['tf_roberta_for_sequence_classification_2/roberta/pooler/dense/kernel:0', 'tf_roberta_for_sequence_classification_2/roberta/pooler/dense/bias:0'] when minimizing the loss.

I using : 

-  Tensorflow 2.1.0rc1

-  Tranformers  but with checkout : git checkout f3386 -b tanda-sequential-finetuning

I don't find any diference on models between this checkout and the tranfermers version that exist right now.

Altough the training is decreasing , for exemple:
Iteration 20:

Training: 20it [02:28,  1.46it/s]
Training step 20 Accuracy: 0.901562511920929, Training loss: 0.053004395216703415 

Iteration 2040:

Training: 2040it [20:16,  1.89it/s]
Training step 2040 Accuracy: 0.953576922416687, Training loss: 0.02367759868502617<|||||>i have the same issue. But since i'm using custom loss and need to pass mask from inputs into the loss, i'm not using eager execution. The problem in this case is that model does not train and throw an error 

`ValueError: Variable <tf.Variable 'tf_bert_model/bert/pooler/dense/kernel:0' shape=(768, 768) dtype=float32> has `None` for gradient. Please make sure that all of your ops have a gradient defined (i.e. are differentiable). Common ops without gradient: K.argmax, K.round, K.eval.`. 

If i use simple standard loss with eager execution, i get only warnings and training is possible. I'm training the model using `model.fit()`<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
