Customized BertForTokenClassification Model
I try to customize BertForTokenClassification model by myself to perform sequence tagging and strictly follow the [original implementation](https://huggingface.co/pytorch-transformers/_modules/pytorch_transformers/modeling_bert.html#BertForTokenClassification). However, I cannot obtain the same results (lower scores) with those produced by BertForTokenClassification when I simply set the top-most tagging component/model as an Linear layer (i.e., the current model is identical to BertForTokenClassification). My code is below:

```python
class BertTagger(BertPreTrainedModel):

    def __init__(self, bert_config):
        super(BertTagger, self).__init__(bert_config)
        self.num_labels = bert_config.num_labels
        #self.tagger_config = TaggerConfig()
        self.bert = BertModel(bert_config)
        self.bert_dropout = nn.Dropout(bert_config.hidden_dropout_prob)
        self.classifier = nn.Linear(bert_config.hidden_size, bert_config.num_labels)
        self.apply(self.init_weights)

    def forward(self, input_ids, token_type_ids=None, attention_mask=None, labels=None,
                position_ids=None, head_mask=None):
        outputs = self.bert(input_ids, position_ids=position_ids, token_type_ids=token_type_ids,
                            attention_mask=attention_mask, head_mask=head_mask)
        # the hidden states of the last Bert Layer, shape: (bsz, seq_len, hsz)
        tagger_input = outputs[0]
        tagger_input = self.bert_dropout(tagger_input)
        logits = self.classifier(tagger_input)
        outputs = (logits,) + outputs[2:]
        if labels is not None:
            loss_fct = CrossEntropyLoss()
            # Only keep active parts of the loss
            if attention_mask is not None:
                active_loss = attention_mask.view(-1) == 1
                active_logits = logits.view(-1, self.num_labels)[active_loss]
                active_labels = labels.view(-1)[active_loss]
                loss = loss_fct(active_logits, active_labels)
            else:
                loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))
            outputs = (loss,) + outputs
        return outputs
```

Has anyone encountered the same issue?
@lixin4ever  I have the same question. Do you solve it?<|||||>I have solve it. Thank you!<|||||>@searchlink How do you solve this problem?<|||||>For reference, the updated resource link mentioned in the original post can be now found [here](https://huggingface.co/transformers/_modules/transformers/modeling_bert.html#BertForTokenClassification); It was affected by the renaming into `transformers`, too.<|||||>But I can't see any difference between "pytorch-transformers" and "transformers" except the line initializing BERT parameters.  Have you met the same problem? @dennlinger <|||||>I was just pointing to the up-to-date reference. I'm currently looking into token classification using BERT (or in my case, I would prefer RoBERTa or other iterations of BERT, but unfortunately they seem not available yet).<|||||>So Token classification using BERT does not work?
<|||||>As you can see below, `BertForTokenClassification` works as expected with **PyTorch 1.3.1** and **Transformers 2.2.2** installed with `pip install transformers`.
```
Python 3.6.9 |Anaconda, Inc.| (default, Jul 30 2019, 19:07:31) 
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import transformers
>>> from transformers import BertForTokenClassification
>>> from transformers import BertTokenizer
>>> tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
>>> model = BertForTokenClassification.from_pretrained('bert-base-uncased')
>>> text='Hello, my dog is cute'
>>> import torch
>>> input_ids = torch.tensor(tokenizer.encode(text, add_special_tokens=True)).unsqueeze(0)  # Batch size 1
>>> input_ids
tensor([[  101,  7592,  1010,  2026,  3899,  2003, 10140,   102]])
>>> labels = torch.tensor([1] * input_ids.size(1)).unsqueeze(0)  # Batch size 1
>>> labels
tensor([[1, 1, 1, 1, 1, 1, 1, 1]])
>>> outputs = model(input_ids, labels=labels)
>>> outputs
(tensor(0.7529, grad_fn=<NllLossBackward>), tensor([[[ 0.5078,  0.1628],
         [-0.0593,  0.0163],
         [ 0.0308, -0.2312],
         [ 0.0863, -0.1000],
         [-0.2833, -0.2656],
         [-0.2014, -0.5225],
         [-0.2912, -0.1220],
         [-0.2781, -0.2919]]], grad_fn=<AddBackward0>))
>>> len(outputs)
2
>>> loss=outputs[0]
>>> scores=outputs[1]
>>> loss
tensor(0.7529, grad_fn=<NllLossBackward>)
>>> scores
tensor([[[ 0.5078,  0.1628],
         [-0.0593,  0.0163],
         [ 0.0308, -0.2312],
         [ 0.0863, -0.1000],
         [-0.2833, -0.2656],
         [-0.2014, -0.5225],
         [-0.2912, -0.1220],
         [-0.2781, -0.2919]]], grad_fn=<AddBackward0>)
>>>
```

It's working with **TensorFlow 2.0.0** and **Transformers 2.2.2** too!
```
Python 3.6.9 |Anaconda, Inc.| (default, Jul 30 2019, 19:07:31) 
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
/home/vidiemme/anaconda3/envs/huggingface/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/vidiemme/anaconda3/envs/huggingface/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/vidiemme/anaconda3/envs/huggingface/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/vidiemme/anaconda3/envs/huggingface/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/vidiemme/anaconda3/envs/huggingface/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/vidiemme/anaconda3/envs/huggingface/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
>>> import transformers
>>> from transformers import BertTokenizer
>>> from transformers import TFBertForTokenClassification
>>> tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
>>> model = TFBertForTokenClassification.from_pretrained('bert-base-uncased')
2019-12-17 12:54:37.120123: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-12-17 12:54:37.320081: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3408000000 Hz
2019-12-17 12:54:37.320815: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55a42296edd0 executing computations on platform Host. Devices:
2019-12-17 12:54:37.320841: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Host, Default Version
>>> text='Hello, my dog is cute'
>>> input_ids = tf.constant(tokenizer.encode(text))[None, :]  # Batch size 1
>>> input_ids
<tf.Tensor: id=6056, shape=(1, 8), dtype=int32, numpy=
array([[  101,  7592,  1010,  2026,  3899,  2003, 10140,   102]],
      dtype=int32)>
>>> outputs = model(input_ids)
>>> len(outputs)
1
>>> outputs
(<tf.Tensor: id=7961, shape=(1, 8, 2), dtype=float32, numpy=
array([[[ 0.09657212, -0.51087016],
        [ 0.28020248, -0.25160134],
        [-0.09995201,  0.0843759 ],
        [-0.12110823,  0.20886022],
        [-0.03617962,  0.00401567],
        [-0.03330922,  0.01042   ],
        [-0.21674895, -0.1601235 ],
        [ 0.1076538 ,  0.19144017]]], dtype=float32)>,)
>>> scores = outputs[0]
>>> scores
<tf.Tensor: id=7961, shape=(1, 8, 2), dtype=float32, numpy=
array([[[ 0.09657212, -0.51087016],
        [ 0.28020248, -0.25160134],
        [-0.09995201,  0.0843759 ],
        [-0.12110823,  0.20886022],
        [-0.03617962,  0.00401567],
        [-0.03330922,  0.01042   ],
        [-0.21674895, -0.1601235 ],
        [ 0.1076538 ,  0.19144017]]], dtype=float32)>
```

> So Token classification using BERT does not work?<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
