BERT encoding layer produces same output for all inputs during evaluation 
I am having issues with differences between the output of the BERT layer during training and evaluation time. I am fine-tuning BertForSequenceClassification, but have traced the problem to the pretrained BertModel. During training, the sequence_output within BertModel.forward() produces sensible output, for example :

[tensor([[[-0.0474, -0.3332, -0.2803, ..., -0.2278, 0.3694, 0.0433],
[ 0.1383, -0.2213, 0.1137, ..., 0.0103, 0.6756, 0.0800],
[ 0.0701, -0.4075, -0.4439, ..., 0.1196, 0.5344, 0.1538],
...,
[ 0.1345, -0.3650, -0.1050, ..., 0.0817, 0.3069, 0.2953],
[ 0.1033, -0.2574, -0.0028, ..., -0.1782, 0.4725, 0.0200],
[ 0.3067, -0.3785, -0.0043, ..., -0.1458, 0.6485, -0.0157]],

During evaluation time, however, it produces the same output for every input within a batch:

tensor([[[-0.2306, -0.4857, -0.8340, ..., -0.9609, 0.4153, -0.3388],
[-0.2306, -0.4857, -0.8340, ..., -0.9609, 0.4153, -0.3388],
[-0.2306, -0.4857, -0.8340, ..., -0.9609, 0.4153, -0.3388],
...,
[-0.2306, -0.4857, -0.8340, ..., -0.9609, 0.4153, -0.3388],
[-0.2306, -0.4857, -0.8340, ..., -0.9609, 0.4153, -0.3388],
[-0.2306, -0.4857, -0.8340, ..., -0.9609, 0.4153, -0.3388]],

My evaluation code is below.

`def evaluate(self,dataset):

    self.state.model.eval()  # turn on evaluation mode

    with torch.no_grad():

        for x, y in dataset:

      
            # Shape of x is (Batch_size, sequence_length)

            preds = torch.sigmoid(self.state.model(x, token_type_ids = torch.zeros_like(x),labels = None)).numpy()`   # I have tried this line with and without .detach() and it makes no difference
Because of the uniform output from BertLayer, I also get identical output within preds.
Unlike #695 and others regarding non-determinism, I am calling model.eval() <|||||>Can you share your model initialization code as well?<|||||>My model is just a slight modification of BertForSequenceClassification for multilabel.

class BertForMultiLabelSequenceClassification(BertForSequenceClassification):
    """BERT model for classification.
    This module is composed of the BERT model with a linear layer on top of
    the pooled output.
    """

    def forward(self, input_ids, token_type_ids=None, attention_mask=None, labels=None):


        _, pooled_output = self.bert(input_ids, token_type_ids, attention_mask, output_all_encoded_layers=False)  
        # In evaluation mode, _ and pooled_output are already wrong by this point.

        pooled_output = self.dropout(pooled_output)
        logits = self.classifier(pooled_output)

        if labels is not None: # Supervised training mode
            loss_fct = BCEWithLogitsLoss()
            loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1, self.num_labels))
            return loss
        else:

            return logits  #Evaluation mode`



This is the training loop where I am able to successfully get BERT embeddings. the call to self.evaluate() is the function in my original comment.

 def train(self):

        opt = optim.Adam(self.state.model.parameters(), lr=1e-3)


        max_epochs = 10
   

        nn.init.xavier_normal_(self.state.model.classifier.weight)

        print("Initial weights",self.state.model.classifier.weight)

        for epoch in range(1, max_epochs + 1): # main training loop

            running_loss = 0.0

            self.state.model.train()  # turn on training mode

            progress = tqdm.tqdm(total = len(self.state.train))

            for x, y in self.state.train:  # thanks to our wrapper, we can intuitively iterate over our data


                segments = torch.zeros_like(x)

                #print("X: ",x.shape)
                #print("Y: ",y.shape)
                opt.zero_grad()
                loss = self.state.model(x,segments,labels = y)

                loss.backward()  #compute gradients and backpropagate
                running_loss += loss.item()
                opt.step()
                epoch_loss = running_loss / len(self.state.train)
                batch_size = x.shape[0]
                progress.update(batch_size) #increment progress bar

            # calculate the validation loss for this epoch
            val_loss = 0.0

            roc_auc,f1 = self.evaluate(self.state.dev)

            score = print('Epoch: {}, ROC-AUC Score: {:.4f}, F1 Score: {:.4f}'.format(epoch, roc_auc,f1))
            #print(roc_auc)
            progress.close()
<|||||>@josephvalencia 

I'm nowhere near a pro, but I've been playing with `BERTForSequenceClassification` since a week or so; I want to share my experience.

a) I feel that you're not applying the pre-trained weights to your BERT model, I've seen quite a few adaptations of `BertForSequenceClassification` actually implement `BertPreTrainedModel` and apply `pre-trained` weights released from google.

I don't quite see that happening in your code, may be you haven't posted it here yet. I also had a similar issue ( `poor accuracy and heavy loss `) because I failed to use pre-trained weights properly.

b) You're better of using `BertAdam` as an optimizer along any decent `learning rate scheduler` rather than AdamOptimizer.

And you're running this for `10 epochs` !!? Running for 3 epochs on a 64 GB RAM with multi cores itself is taking me about 3 hours to train :D, may be you're using some sort of magic parallelization technique to speed up your training, I could use that info if that's the case. <|||||>@amit8121 Thanks for the tips. I am calling BertForMultilabelSequenceClassification.from_pretrained() elsewhere. I don't actually plan to train for 10 epochs, I will probably implement early stopping once I have the semantics correct.<|||||>Can I see your from pretrained call? The "embeddings" that you're seeing in training stage might just be due to dropout.<|||||>`     model = BertForMultiLabelSequenceClassification.from_pretrained('bert-base-uncased', 
      num_labels=num_classes)

    state = TrainingState(model,test_dataset,dev_dataset,train_dataset)

    trainer = Trainer(state)

    trainer.train()
`<|||||>Anyone have any ideas? I'm about to give up on this use case<|||||>Hi @josephvalencia, I don't have any hint, unfortunately. If the model works well during training, I can't really understand why it would produce always the same output during evaluation.
Do you think you can post a full and self-contained example which exhibit the behavior?<|||||>I have determined that it was an error in my token indexing that happened earlier in my data pipeline / improper use of attention masking<|||||>@josephvalencia What was the solution here? I'm facing the same problem...<|||||>Hi, please open a new issue with a sample from your code and a detailed error log.<|||||>@thomwolf New issue has been opened here - https://github.com/huggingface/transformers/issues/1465<|||||>@thomwolf Is this problem solved<|||||>@thomwolf Is this problem solved<|||||>How the problem is solved?
<|||||>i encountered this issue, 
when i change learning rate value 3e-5 to 5e-5. it worked. 
when i use learning rate 3e-5, I think it gone local minima. <|||||>Observing same behaviour<|||||>Try experimenting with learning rate and optimizer. Adam with lr=5e-5 worked for me (batch size 64).<|||||>Have the same problem. How is it  solved?
<|||||>Try with pytorch optimizer not AdamW in transformer.<|||||>Changing my learning rate from 0.01 to 5e-5 worked for me.<|||||>@abdulsalam-bande I had the same output logits for a fine-tuned model, and your solution worked for me (even a slight decrease in the learning rate). Any idea why having a too large learning rate gives this result?

@Jayaos did not work (adamw_torch vs adamw_hf). Were you suggesting to use an other optimizer?

Specifically, had the same logits for

```
python examples/pytorch/text-classification/run_glue.py   --model_name_or_path roberta-large \
--task_name cola   --do_train   --do_eval   \
--max_seq_length 128   --per_device_train_batch_size 8   --learning_rate 5e-5  \
--num_train_epochs 1 --output_dir run_glue_res_lr_5e5 \
```

but not for

```
python examples/pytorch/text-classification/run_glue.py   --model_name_or_path roberta-large \
 --task_name cola   --do_train   --do_eval  \
--max_seq_length 128   --per_device_train_batch_size 8   --learning_rate 2e-5  \
--num_train_epochs 1   --output_dir run_glue_res_lr_2e5
```

Think it may be related to learning rate / batch size ratio.