Finetuned BERT model does not seem to predict right labels/work properly?
# ❓ Questions & Help
I am trying out a finetuned BERT model for token classification (--> https://huggingface.co/bert-base-cased-finetuned-conll03-english), but when I observe the model output (i.e. the logits after applying the softmax) and compare it with the true label_ids, they are totally uncorrelated (see pictures).

https://i.stack.imgur.com/gVyMn.png
https://i.stack.imgur.com/qS62L.png

## Details

I assume that the finetuned model (bert-base-cased-finetuned-conll03-english) is correctly pretrained, but I don't seem to understand why its predictions are off. I think one issue is that the pretrained model has another labelling scheme than I made myself during data prep (so that the tag2name dict is different), but I don't know how I can find out what label-index map the model uses for its predictions. Even then it is not the case that the model consistently makes the same mistakes, it is outputting things quite randomly.

Any idea what the issue could be?

``
Please post your code using [code blocks](https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks). Don't post screenshots.<|||||>FYI: 
photo of **format input data**;  https://i.stack.imgur.com/t472b.png
photo of **tag2name** ; https://i.stack.imgur.com/RO7dp.png


Assuming the data goes in in the right format, here is the model initialization and evaluation loop.
```
tokenizer = AutoTokenizer.from_pretrained("bert-base-cased-finetuned-conll03-english")
model = BertForTokenClassification.from_pretrained('bert-base-cased-finetuned-conll03-english')


#eval LOOP

model.eval();
eval_loss, eval_accuracy = 0, 0
nb_eval_steps, nb_eval_examples = 0, 0
y_true = []
y_pred = []
valdataset = []

print("***** Running evaluation *****")
print("  Num examples ={}".format(len(val_inputs)))
print("  Batch size = {}".format(batch_num))

for step, batch in enumerate(valid_dataloader):
    batch = tuple(t.to(device) for t in batch) # set every example of batch to device
    input_ids, input_mask, label_ids = batch #same as we did in training loop but only 1 epoch now
    
    
    with torch.no_grad(): #means we don't care about gradients and updating tensors 
        outputs = model(input_ids, token_type_ids=None,
        attention_mask=input_mask)
        # For eval mode, the first result of outputs is logits (for training mode this was loss)
        logits = outputs[0] #  In context of deep learning the logits layer means the layer that feeds in to softmax (or other such normalization).
    
    # Get NER predict result
    logits = torch.argmax(F.log_softmax(logits,dim=2),dim=2)#feed logits into softmax and take the prediction that is maximal 
    logits = logits.detach().cpu().numpy()
    
    if step==1:
        print(logits[0][0:15])
        print(logits[1][0:15])
        print(logits[3][0:15])
        print(logits[4][0:15])
        print(logits[5][0:15])

        print(label_ids[0][0:15])
        print(label_ids[1][0:15])
        print(label_ids[2][0:15])
        print(label_ids[3][0:15])
        
    
    # Get NER true result
    label_ids = label_ids.to('cpu').numpy()
    
    
    # Only predict the real word, mark=0, will not calculate
    input_mask = input_mask.to('cpu').numpy()
    
    # Compare the valuable predict result
    for i,mask in enumerate(input_mask):
        # Real one
        temp_1 = []
        # Predicted one
        temp_2 = []
        
        valtemp = []

        for j, m in enumerate(mask):
            # Mark=0, meaning its a pad word, dont compare
            if m:
                if tag2name[label_ids[i][j]] != "X" and tag2name[label_ids[i][j]] != "[CLS]" and tag2name[label_ids[i][j]] != "[SEP]" : # Exclude the X label
                    temp_1.append(tag2name[label_ids[i][j]])
                    temp_2.append(tag2name[logits[i][j]])
                
                if tag2name[label_ids[i][j]] != "[CLS]" and tag2name[label_ids[i][j]] != "[SEP]" :
                  valtemp.append(input_ids[i][j].item())
                    
            else:
                break
        
        #here are the two lists that contain true and pred labels.    
        y_true.append(temp_1)
        y_pred.append(temp_2)

 
        valdataset.append(valtemp)

tokenized_text_con = [tokenizer.decode(val) for val in valdataset]



 ```
print output: https://i.stack.imgur.com/qS62L.png

<|||||>Hi! From my experience using the community-contributed `dbmdz/bert-large-cased-finetuned-conll03-english` (which is the same checkpoint as) `bert-large-cased-finetuned-conll03-english`, using the `bert-base-cased` tokenizer instead of the tokenizer loaded from that checkpoint works better.

You can see an example of this in the [usage](https://huggingface.co/transformers/usage.html#named-entity-recognition), let me know if it helps.

I suspect the difference between the tokenizers is due to a lowercasing of all inputs. I'm looking into it now.

PS: the file `bert-large-cased-finetuned-conll03-english` is deprecated in favor of the aforementionned `dbmdz/bert-large-cased-finetuned-conll03-english` as they are duplicates. @julien-c is currently deleting it from the S3, please use the `dbmdz` file/folder.<|||||>Also cc'ing @stefan-it for information :)<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
