Untrainable dense layer in TFBert. "WARNING:tensorflow:Gradients do not exist for variables ['tf_bert_model/bert/pooler/dense/kernel:0', 'tf_bert_model/bert/pooler/dense/bias:0'] when minimizing the loss."
## 🐛 Bug

<!-- Important information -->

I am getting 

> WARNING:tensorflow:Gradients do not exist for variables ['tf_bert_model/bert/pooler/dense/kernel:0', 'tf_bert_model/bert/pooler/dense/bias:0'] when minimizing the loss.
> WARNING:tensorflow:Gradients do not exist for variables ['tf_bert_model/bert/pooler/dense/kernel:0', 'tf_bert_model/bert/pooler/dense/bias:0'] when minimizing the loss.
> WARNING:tensorflow:Gradients do not exist for variables ['tf_bert_model/bert/pooler/dense/kernel:0', 'tf_bert_model/bert/pooler/dense/bias:0'] when minimizing the loss.
> WARNING:tensorflow:Gradients do not exist for variables ['tf_bert_model/bert/pooler/dense/kernel:0', 'tf_bert_model/bert/pooler/dense/bias:0'] when minimizing the loss.

Errors when using the Tensorflow Bert. 

For convenience, here is a colab notebook that reproduces the error
https://colab.research.google.com/drive/1fh8l43Mm7g4-yjlun7nTilmuP0SOMOx-

It looks like there's an operation in the TF Bert that does not allow gradients to flow, judging from goolging this issue

https://github.com/tensorflow/probability/issues/467

https://github.com/tensorflow/tensorflow/issues/27949

https://stackoverflow.com/questions/55434653/batch-normalization-doesnt-have-gradient-in-tensorflow-2-0

https://stackoverflow.com/questions/57144586/tensorflow-gradienttape-gradients-does-not-exist-for-variables-intermittently

Model I am using (Bert, XLNet....):

TFBertModel

Language I am using the model on (English, Chinese....):

English

The problem arise when using:
* [ ] the official example scripts: (give details)
* [X ] my own modified scripts: (give details)

```
!pip install transformers --quiet
%tensorflow_version 2.x

try:
  %tensorflow_version 2.x
except Exception:
  pass
import tensorflow as tf


from tensorflow.keras.datasets import imdb
from tensorflow.keras.preprocessing import sequence
import os
from tensorflow import keras 
from tensorflow.keras.layers import Lambda
from tensorflow.keras import backend as K
from keras.preprocessing.sequence import pad_sequences
from transformers import TFBertModel, BertTokenizer, BertConfig
import numpy as np
from glob import glob
from tqdm import tqdm_notebook

print('TensorFlow:', tf.__version__)

posCites = tf.random.uniform(shape=(3, 4, 768), minval=-1, maxval=1, dtype=tf.dtypes.float32)
negCites = tf.random.uniform(shape=(3, 16, 768), minval=-1, maxval=1, dtype=tf.dtypes.float32)
textInputsIds = tf.random.uniform(shape=(3, 8), minval=0, maxval=200, dtype=tf.dtypes.int32)

dataset = (textInputsIds, posCites, negCites)
batch_size = 3
post_size = 4
neg_size = 16

posLabels = keras.backend.ones(batch_size*post_size)
negLabels = keras.backend.zeros(batch_size*neg_size)

totalLabels = keras.backend.concatenate((posLabels, negLabels), axis=-1)
totalLabels = tf.convert_to_tensor([[totalLabels] * 3])
totalLabels = tf.squeeze(totalLabels)

model = TFBertModel.from_pretrained('bert-base-uncased')
model.compile(
      optimizer=tf.keras.optimizers.Adam(learning_rate=0.01),
      loss='binary_crossentropy',
      metrics=['acc'])
labels = tf.constant(np.array([1,0,1]))
model(textInputsIds)[0]

def create_model():
    textInputs = tf.keras.Input(shape=(8,), dtype=tf.int32)
    
    bert_model = TFBertModel.from_pretrained('bert-base-uncased')
    
    textOut = bert_model(textInputs)
    textOutMean = tf.reduce_mean(textOut[0], axis=1)
    
    logits = tf.reduce_sum(textOutMean, axis=-1)
    return tf.keras.Model(inputs=[textInputs], outputs=[logits])

model = create_model()
# model = TFBertModel.from_pretrained('bert-base-uncased')
model.compile(
      optimizer=tf.keras.optimizers.Adam(learning_rate=0.01),
      loss='binary_crossentropy',
      metrics=['acc'])
labels = tf.constant(np.array([1,0,1]))
model.fit(textInputsIds, labels, epochs=100)

```

WARNING:tensorflow:Gradients do not exist for variables ['tf_bert_model_1/bert/pooler/dense/kernel:0', 'tf_bert_model_1/bert/pooler/dense/bias:0'] when minimizing the loss.
WARNING:tensorflow:Gradients do not exist for variables ['tf_bert_model_1/bert/pooler/dense/kernel:0', 'tf_bert_model_1/bert/pooler/dense/bias:0'] when minimizing the loss.


The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ X] my own task or dataset: (give details)

Using Bert as a text encoder, and matching output embeddings with a target. 

## To Reproduce

Steps to reproduce the behavior:

Here is a ColabNotebook which contains the code posted above, to recreate the error
https://colab.research.google.com/drive/1fh8l43Mm7g4-yjlun7nTilmuP0SOMOx-

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

Gradients should be calculated for all variables in tfBert. 

## Environment

* OS: Linux
* Python version: 3+
* PyTorch version: 1.2+
* PyTorch Transformers version (or branch): same as Pip install
* Using GPU ? colab gpu 
* Distributed of parallel setup ? no
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->

Possibly related issues

https://github.com/huggingface/transformers/issues/1727


After looking at 

https://github.com/huggingface/transformers/issues/1727

I figured out we're getting the warning because we're not using the pooler, therefore it gets no updates. <|||||>@Santosh-Gupta  can you please update your colab notebook with correct version. i am unbale to get this resolved<|||||>Hi, I have exactly the same issue with TFXLMRobertaForSequenceClassification. How did you solve the issue ?<|||||>> Hi, I have exactly the same issue with TFXLMRobertaForSequenceClassification. How did you solve the issue ?

From what understand, this is not a bug. You won't get gradients calculated for variables (kernel and bias) of the layer tf_bert_model/bert/pooler/dense if you don't use it. As such, if you indeed don't use the pooler, you can simply ignore this warning. <|||||>> @Santosh-Gupta can you please update your colab notebook with correct version. i am unbale to get this resolved

It's not a bug, the forward pass doesn't go through the pooler, so the backwards pass doesn't go through it either. <|||||>I'm  having the same issue, although I am using the pooling layer. 
my model is like this one 
```
class TFAlbertForNaturalQuestionAnswering(TFAlbertPreTrainedModel):
    def __init__(self, config, *inputs, **kwargs):
        super().__init__(config, *inputs, **kwargs)
  
        self.albert = TFAlbertMainLayer(config)

        self.initializer = get_initializer(config.initializer_range)
        self.start = tf.keras.layers.Dense(1,
            kernel_initializer=self.initializer, name='start')
        self.end = tf.keras.layers.Dense(1,
            kernel_initializer=self.initializer, name='end')
        self.long_outputs = tf.keras.layers.Dense(1, kernel_initializer=self.initializer,
            name='long')

        self.answerable = tf.keras.layers.Dense(1, kernel_initializer=self.initializer,
            name='answerable', activation = "sigmoid")
    def call(self, inputs, **kwargs):
        outputs = self.albert(inputs, **kwargs)
        sequence_output = outputs[0]
 
        # tf.print(outputs[0].shape) (batch, len->0, hidden) 1->0
        # tf.print(outputs[1].shape) (batch, hidden_size)

        start_logits =  tf.squeeze(self.start(sequence_output), -1)
        end_logits =  tf.squeeze(self.end(sequence_output), -1)
        long_logits = tf.squeeze(self.long_outputs(sequence_output), -1)

        answerable = tf.squeeze(self.answerable(outputs[1]), -1)
```<|||||>Hey ! i'm getting the same issue with attention model and embedding layer, the weights of both layers are not updating .

```

embd=Embedding(input_dim=len(vocab),output_dim=100,name="embd")
lstm1=Bidirectional(LSTM(units=100,return_sequences=True,name="lstm1"),name="bd1")
lstm2=Bidirectional(LSTM(units=100,return_sequences=True,name="lstm2"),name="bd2")
attention_layer=Attention_Model(21,200)
dense1=Dense(units=80,name="dense1",kernel_regularizer="l2")
dropout1=Dropout(0.5)
act1=Activation('sigmoid')

dense2=Dense(units=50,name="dense2",kernel_regularizer="l2")
dropout2=Dropout(0.4)
act2=Activation('sigmoid')

dense3=Dense(units=30,name="dense3",kernel_regularizer="l2")
dropout3=Dropout(0.3)
act3=Activation('sigmoid')

dense4=Dense(units=len(classes),name="dense4")
dropout4=Dropout(0.2)
output=Activation('softmax')

```
Forward Pass : 

```
def forward_pass(X):
  t=embd(X)
 
  t=lstm1(t)
  
  t=lstm2(t)
  

 
  t=attention_layer(t)
  
  
  t=dense1(t)
  t=dropout1(t)
  t=act1(t)

  t=dense2(t)
  t=dropout2(t)
  t=act2(t)

  t=dense3(t)
  t=dropout3(t)
  t=act3(t)
  
  t=dense4(t)
  t=dropout4(t)
  t=output(t)

  return t


```

Attention Model : 

```

class Attention_Model():
  def __init__(self,seq_length,units):
    self.seq_length=seq_length
    self.units=units
    self.lstm=LSTM(units=units,return_sequences=True,return_state=True)
    

  def get_lstm_s(self,seq_no):
    input_lstm=tf.expand_dims(tf.reduce_sum(self.X*(self.alphas[:,:,seq_no:seq_no+1]),axis=1),axis=1)
    a,b,c=self.lstm(input_lstm)
    self.output[:,seq_no,:]=a[:,0,:]

    return b

  def __call__(self,X):
    self.X=X
    self.output=np.zeros(shape=(self.X.shape[0],self.seq_length,self.units))
    self.dense=Dense(units=self.seq_length)
    self.softmax=Softmax(axis=1)
    

    for i in range(self.seq_length+1):
      if i==0 :
        s=np.zeros(shape=(self.X.shape[0],self.units))
      else :
        s=self.get_lstm_s(i-1)
      if(i==self.seq_length):
        break 
      
      s=RepeatVector(self.X.shape[1])(s)
      concate_X=np.concatenate([self.X,s],axis=-1)
      
      self.alphas=self.softmax(self.dense(concate_X))

    return self.output
      
```


is anything wrong with implementation or something else ?<|||||>@MarioBonse , your forward pass isn't going through the pooling layer

'''sequence_output = outputs[0]'''

@gajeshladhar 

Is that code from the hf library? Where are the classes defined? 
<|||||>pooler_output of transformers TFRobertaModel have tf_roberta_model/roberta/pooler/dense/kernel:0. if you have not use pooler_output,tf_roberta_model/roberta/pooler/dense/kernel:0 do not update Gradients