Importing TF checkpoint as BertForTokenClassificiation
Hello Everyone,

I've been stuck with trying to load TensorFlow checkpoints to be used by `pytorch-pretrained-bert` as `BertForTokenClassification`.

**pytorch-pretrained-BERT Version:** Installed from latest master branch.

**What works:**
```python
config = BertConfig.from_json_file(CONFIG_FILE)
model = BertForPreTraining(config)
model = load_tf_weights_in_bert(model, "/home/bert/pt_baseuncased/model.ckpt-98000")
```
**What I want to do:**
```python
config = BertConfig.from_json_file(CONFIG_FILE)
model = BertForTokenClassification(config, num_labels=num_labels)
# the difference is BertForTokenClassification instead of BertForPreTraining
model = load_tf_weights_in_bert(model, "/home/bert/pt_baseuncased/model.ckpt-98000")
```
**When I try to do this it gives me:**
```python
AttributeError: 'BertForTokenClassification' object has no attribute 'bias'
```
**Full Traceback:**
```
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-5-a8225a5966f7> in <module>
      1 config = BertConfig.from_json_file(CONFIG_FILE)
      2 model = BertForTokenClassification(config, num_labels=10)
----> 3 model = load_tf_weights_in_bert(model, "/home/gzhenfun/bert/pt_baseuncased/model.ckpt-98000")

~/pytorch-pretrained-BERT/pytorch_pretrained_bert/modeling.py in load_tf_weights_in_bert(model, tf_checkpoint_path)
     88                 pointer = getattr(pointer, 'weight')
     89             elif l[0] == 'output_bias' or l[0] == 'beta':
---> 90                 pointer = getattr(pointer, 'bias')
     91             elif l[0] == 'output_weights':
     92                 pointer = getattr(pointer, 'weight')

~/anaconda3/envs/bert/lib/python3.6/site-packages/torch/nn/modules/module.py in __getattr__(self, name)
    533                 return modules[name]
    534         raise AttributeError("'{}' object has no attribute '{}'".format(
--> 535             type(self).__name__, name))
    536 
    537     def __setattr__(self, name, value):

AttributeError: 'BertForTokenClassification' object has no attribute 'bias'
```
**To try to resolve this:**
I followed https://github.com/huggingface/pytorch-pretrained-BERT/issues/580#issuecomment-489519231 from #580, and changed my `modeling.py` to this:
```python
        pointer = model
        for m_name in name:
            if re.fullmatch(r'[A-Za-z]+_\d+', m_name):
                l = re.split(r'_(\d+)', m_name)
            else:
                l = [m_name]
            if l[0] == 'kernel' or l[0] == 'gamma':
                pointer = getattr(pointer, 'weight')
            elif l[0] == 'output_bias' or l[0] == 'beta':
                pointer = getattr(pointer, 'cls')
                # added the line above
                pointer = getattr(pointer, 'bias')
            elif l[0] == 'output_weights':
                pointer = getattr(pointer, 'cls')
                # added the line above
                pointer = getattr(pointer, 'weight')
            elif l[0] == 'squad':
                pointer = getattr(pointer, 'classifier')
            else:
                try:
                    pointer = getattr(pointer, l[0])
                except AttributeError:
                    print("Skipping {}".format("/".join(name)))
                    continue
            if len(l) >= 2:
                num = int(l[1])
                pointer = pointer[num]
```

However, that gives me:
```python
AttributeError: 'FusedLayerNorm' object has no attribute 'cls'
```

Anybody here knows how I can fix this and properly import a TF checkpoint as `BertForTokenClassification`?

Will appreciate any help. Thank you!
Hello everyone,

I have temporarily come up with a workaround for this. Not sure if it's the best solution but it works. What I did was I essentially merged what `load_tf_weights_in_bert()` and what part of `BertPreTrainedModel.from_pretrained()` was doing. `BertPreTrainedModel` is the parent class of `BertForTokenClassification`, so if you are trying to do something similar for `BertFor{TaskName}`, it should work too.

```python
def load_BFTC_from_TF_ckpt(bert_config, ckpt_path, num_labels):
    config = BertConfig.from_json_file(bert_config)
    model = BertForPreTraining(config)
    load_tf_weights_in_bert(model, ckpt_path)
    state_dict=model.state_dict()
    model = BertForTokenClassification(config, num_labels=num_labels)

    # Load from a PyTorch state_dict
    old_keys = []
    new_keys = []
    for key in state_dict.keys():
        new_key = None
        if 'gamma' in key:
            new_key = key.replace('gamma', 'weight')
        if 'beta' in key:
            new_key = key.replace('beta', 'bias')
        if new_key:
            old_keys.append(key)
            new_keys.append(new_key)
    for old_key, new_key in zip(old_keys, new_keys):
        state_dict[new_key] = state_dict.pop(old_key)

    missing_keys = []
    unexpected_keys = []
    error_msgs = []
    # copy state_dict so _load_from_state_dict can modify it
    metadata = getattr(state_dict, '_metadata', None)
    state_dict = state_dict.copy()
    if metadata is not None:
        state_dict._metadata = metadata

    def load(module, prefix=''):
        local_metadata = {} if metadata is None else metadata.get(prefix[:-1], {})
        module._load_from_state_dict(
            state_dict, prefix, local_metadata, True, missing_keys, unexpected_keys, error_msgs)
        for name, child in module._modules.items():
            if child is not None:
                load(child, prefix + name + '.')
    start_prefix = ''
    if not hasattr(model, 'bert') and any(s.startswith('bert.') for s in state_dict.keys()):
        start_prefix = 'bert.'
    load(model, prefix=start_prefix)
    if len(missing_keys) > 0:
        logger.info("Weights of {} not initialized from pretrained model: {}".format(
            model.__class__.__name__, missing_keys))
    if len(unexpected_keys) > 0:
        logger.info("Weights from pretrained model not used in {}: {}".format(
            model.__class__.__name__, unexpected_keys))
    if len(error_msgs) > 0:
        raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
                           model.__class__.__name__, "\n\t".join(error_msgs)))
    return model

model = load_BFTC_from_TF_ckpt(CONFIG_FILE, "/home/bert/pt_baseuncased/model.ckpt-98000", num_labels)
```<|||||>Hi chrisgzf, thank you for the quick fix. I saw it's been integrated in the latest version.

I am trying to do quite the same using **BertForSequenceClassification** but the "**AttributeError**: 'BertForTokenClassification' object has no attribute 'bias'" still shows up.

Any ideas how I could use your fix for BertForSequenceClassification too ?

Thank you<|||||>Hi @stormskidd, 

in my code snippet here (https://github.com/huggingface/pytorch-pretrained-BERT/issues/676#issuecomment-501526327), just change

`model = BertForTokenClassification(config, num_labels=num_labels)`
to
`model = BertForSequenceClassification(config, num_labels=num_labels)`

and it _should_ work.<|||||>Thank you for the quick response.
So I tried this as you said :

```
def load_BFTC_from_TF_ckpt(bert_config, ckpt_path, num_labels):
    config = BertConfig.from_json_file(bert_config)
    model = BertForSequenceClassification(config, num_labels=num_labels)
    load_tf_weights_in_bert(model, ckpt_path)
    state_dict=model.state_dict()
    model = **BertForSequenceClassification**(config, num_labels=num_labels)

    # Load from a PyTorch state_dict
    old_keys = []
    new_keys = []
    for key in state_dict.keys():
        new_key = None
        if 'gamma' in key:
            new_key = key.replace('gamma', 'weight')
        if 'beta' in key:
            new_key = key.replace('beta', 'bias')
        if new_key:
            old_keys.append(key)
            new_keys.append(new_key)
    for old_key, new_key in zip(old_keys, new_keys):
        state_dict[new_key] = state_dict.pop(old_key)

    missing_keys = []
    unexpected_keys = []
    error_msgs = []
    # copy state_dict so _load_from_state_dict can modify it
    metadata = getattr(state_dict, '_metadata', None)
    state_dict = state_dict.copy()
    if metadata is not None:
        state_dict._metadata = metadata

    def load(module, prefix=''):
        local_metadata = {} if metadata is None else metadata.get(prefix[:-1], {})
        module._load_from_state_dict(
            state_dict, prefix, local_metadata, True, missing_keys, unexpected_keys, error_msgs)
        for name, child in module._modules.items():
            if child is not None:
                load(child, prefix + name + '.')
    start_prefix = ''
    if not hasattr(model, 'bert') and any(s.startswith('bert.') for s in state_dict.keys()):
        start_prefix = 'bert.'
    load(model, prefix=start_prefix)
    if len(missing_keys) > 0:
        logger.info("Weights of {} not initialized from pretrained model: {}".format(
            model.__class__.__name__, missing_keys))
    if len(unexpected_keys) > 0:
        logger.info("Weights from pretrained model not used in {}: {}".format(
            model.__class__.__name__, unexpected_keys))
    if len(error_msgs) > 0:
        raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
                           model.__class__.__name__, "\n\t".join(error_msgs)))
    return model
```

Then:

```
CONFIG_FILE = "Bert/multi_cased_L-12_H-768_A-12/bert_config.json"
model = load_BFTC_from_TF_ckpt(CONFIG_FILE, "model.ckpt-6032", num_labels = 2)
```


But still got the error:
`
<ipython-input-2-8b2ad52bd838> in load_BFTC_from_TF_ckpt(bert_config, ckpt_path, num_labels)
      2     config = BertConfig.from_json_file(bert_config)
      3     model = BertForSequenceClassification(config, num_labels=num_labels)
----> 4     load_tf_weights_in_bert(model, ckpt_path)
      5     state_dict=model.state_dict()
      6     model = BertForSequenceClassification(config, num_labels=num_labels)

C:\ProgramData\Anaconda3\Lib\site-packages\pytorch_pretrained_bert\modeling.py in load_tf_weights_in_bert(model, tf_checkpoint_path)
     89 
     90             elif l[0] == 'output_bias' or l[0] == 'beta':
---> 91                 pointer = getattr(pointer, 'bias')
     92 
     93             elif l[0] == 'output_weights':

C:\ProgramData\Anaconda3\Lib\site-packages\torch\nn\modules\module.py in __getattr__(self, name)
    537                 return modules[name]
    538         raise AttributeError("'{}' object has no attribute '{}'".format(
--> 539             type(self).__name__, name))
    540 
    541     def __setattr__(self, name, value):

AttributeError: 'BertForSequenceClassification' object has no attribute 'bias''`




Also, I wandered in the source code where the error occurs :
```

     90             elif l[0] == 'output_bias' or l[0] == 'beta':
---> 91                 pointer = getattr(pointer, 'bias')
```


I tried to change getattr(pointer, 'bias') to getattr(pointer, 'beta') but then I got a slightly different error:

```
C:\ProgramData\Anaconda3\Lib\site-packages\pytorch_pretrained_bert\modeling.py in load_tf_weights_in_bert(model, tf_checkpoint_path)
     89 
     90             elif l[0] == 'output_bias' or l[0] == 'beta':
---> 91                 pointer = getattr(pointer, 'beta')
     92 
     93             elif l[0] == 'output_weights':

C:\ProgramData\Anaconda3\Lib\site-packages\torch\nn\modules\module.py in __getattr__(self, name)
    537                 return modules[name]
    538         raise AttributeError("'{}' object has no attribute '{}'".format(
--> 539             type(self).__name__, name))
    540 
    541     def __setattr__(self, name, value):

AttributeError: 'BertLayerNorm' object has no attribute 'beta'
```

Hope it helps. Please let me know you think of any workaround for this !
By the way, I'm on windows using Anaconda with Python 3.7.1

Greetings,
Maxime<|||||>Hi Maxime (@stormskidd),

Do read my comment carefully. Change

`model = BertForTokenClassification(config, num_labels=num_labels)`
to
`model = BertForSequenceClassification(config, num_labels=num_labels)`

Please leave `model = BertForPreTraining(config)` (line 3) as is, and do not change it.

Edit: you might want to check out #685 as well. I submitted a PR to make it easier to do something we are trying to do. Maybe the code examples will make it clearer to you.

In line 3 of the function body, the model has to be an instance of `BertForPreTraining` because that's what `load_tf_weights_in_bert()` is  designed to work with. I'm not sure if what I'm doing is the proper way or just a jank way, but I'm just trying to copy out the state_dict after the weights are imported as a `BertForPreTraining` instances, then creating a brand new `BertFor[TaskName]` instance, and loading the state_dict into it.

Hope this clears things up.

Cheers.<|||||>Oh, you're right I posted the wrong version of the many things I tried :(

Anyway, I'm afraid the error still shows up with the following:

```
def load_BFTC_from_TF_ckpt(bert_config, ckpt_path, num_labels):
    config = BertConfig.from_json_file(bert_config)
    model = BertForPreTraining(config)
    load_tf_weights_in_bert(model, ckpt_path)
    state_dict=model.state_dict()
    model = BertForSequenceClassification(config, num_labels=num_labels)

    # Load from a PyTorch state_dict
    old_keys = []
    new_keys = []
    for key in state_dict.keys():
        new_key = None
        if 'gamma' in key:
            new_key = key.replace('gamma', 'weight')
        if 'beta' in key:
            new_key = key.replace('beta', 'bias')
        if new_key:
            old_keys.append(key)
            new_keys.append(new_key)
    for old_key, new_key in zip(old_keys, new_keys):
        state_dict[new_key] = state_dict.pop(old_key)

    missing_keys = []
    unexpected_keys = []
    error_msgs = []
    # copy state_dict so _load_from_state_dict can modify it
    metadata = getattr(state_dict, '_metadata', None)
    state_dict = state_dict.copy()
    if metadata is not None:
        state_dict._metadata = metadata

    def load(module, prefix=''):
        local_metadata = {} if metadata is None else metadata.get(prefix[:-1], {})
        module._load_from_state_dict(
            state_dict, prefix, local_metadata, True, missing_keys, unexpected_keys, error_msgs)
        for name, child in module._modules.items():
            if child is not None:
                load(child, prefix + name + '.')
    start_prefix = ''
    if not hasattr(model, 'bert') and any(s.startswith('bert.') for s in state_dict.keys()):
        start_prefix = 'bert.'
    load(model, prefix=start_prefix)
    if len(missing_keys) > 0:
        logger.info("Weights of {} not initialized from pretrained model: {}".format(
            model.__class__.__name__, missing_keys))
    if len(unexpected_keys) > 0:
        logger.info("Weights from pretrained model not used in {}: {}".format(
            model.__class__.__name__, unexpected_keys))
    if len(error_msgs) > 0:
        raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
                           model.__class__.__name__, "\n\t".join(error_msgs)))
    return model
```

Then:


```
CONFIG_FILE = "/Bert/multi_cased_L-12_H-768_A-12/bert_config.json"
model = load_BFTC_from_TF_ckpt(CONFIG_FILE, "model.ckpt-6032", num_labels = 2)
```


And :
```
C:\ProgramData\Anaconda3\Lib\site-packages\pytorch_pretrained_bert\modeling.py in load_tf_weights_in_bert(model, tf_checkpoint_path)
     89 
     90             elif l[0] == 'output_bias' or l[0] == 'beta':
---> 91                 pointer = getattr(pointer, 'bias')
     92 
     93             elif l[0] == 'output_weights':

C:\ProgramData\Anaconda3\Lib\site-packages\torch\nn\modules\module.py in __getattr__(self, name)
    537                 return modules[name]
    538         raise AttributeError("'{}' object has no attribute '{}'".format(
--> 539             type(self).__name__, name))
    540 
    541     def __setattr__(self, name, value):

AttributeError: 'BertForPreTraining' object has no attribute 'bias'
```

Greetings,
Max<|||||>@stormskidd,

this is odd.... I actually specifically tested my code above on `BertForSequenceClassification` as well and I was able to successfully import TF weights. The code snippet looks like it would work...

Just want to check:
- are you running the latest pytorch-pretrained-BERT from upstream master?
- did you make any other changes to the source?
- have you tried https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/pytorch_pretrained_bert/convert_tf_checkpoint_to_pytorch.py and successfully converted your TF ckpt to a pytorch dump? if my code snippet above doesn't work for you, this should at least work for you.
- not sure if this matters, but are you using CUDA? and do you have apex installed?

Do let me know if you face any errors. I'm curious about this issue. Btw, check out my edit in the previous comment.

Chris<|||||>Hi, not sure I fully understand the issue here. What is the kind of tensorflow checkpoint you are trying to convert? Is it a pretrained model like the original Bert checkpoints or is it a fine-tuned model with additional elements (like a classification layer on top)?<|||||>Hi @thomwolf,

I am trying to convert a pretrained model like the original Bert checkpoints, except that I did additional pretraining on top of the released models with `run_pretraining.py` from the BERT repo. I then wish to fine-tune these pretrained models in pytorch, which is why I had to do this conversion. I am not importing any fine-tuned TF models.<|||||>Ok so you can just convert your Tensorflow model using the command line script (see [here in the readme](https://github.com/huggingface/pytorch-pretrained-BERT#Command-line-interface)) store the converted pytorch model in a folder with the configuration file and then load it in a `BertForTokenClassification` model as follow:
`BertForTokenClassification.from_pretrained('PATH_TO_YOUR_CONVERTED_MODEL_FOLDER')`

For the tokenizer, you can use the one associated to the original TensorFlow model from which you did the fine-tuning since you probably didn't change the vocabulary itself.<|||||>Hi guys,
I saw your discussion and it gave me the idea to try the following:
It seems like I can use both the function 'load_BFTC_from_TF_ckpt' and the script 'convert_tf_checkpoint_to_pytorch.py' to load a pretrained Google model : multi_cased_L-12_H-768_A-12\bert_model.ckpt.

However the error occurs when I tried to do the same with a fined-tuned model (from Bert script run_classifier.py ran on GluON/MRPC -like data).
Is there any ways I can load a TF checkpoint fine-tuned model directly in Pytorch ? Or do I have to re-finetune it with the pytorch_pretrained_bert library ?

Thank you !
Max<|||||>> Oh, you're right I posted the wrong version of the many things I tried :(
> 
> Anyway, I'm afraid the error still shows up with the following:
> 
> ```
> def load_BFTC_from_TF_ckpt(bert_config, ckpt_path, num_labels):
>     config = BertConfig.from_json_file(bert_config)
>     model = BertForPreTraining(config)
>     load_tf_weights_in_bert(model, ckpt_path)
>     state_dict=model.state_dict()
>     model = BertForSequenceClassification(config, num_labels=num_labels)
> 
>     # Load from a PyTorch state_dict
>     old_keys = []
>     new_keys = []
>     for key in state_dict.keys():
>         new_key = None
>         if 'gamma' in key:
>             new_key = key.replace('gamma', 'weight')
>         if 'beta' in key:
>             new_key = key.replace('beta', 'bias')
>         if new_key:
>             old_keys.append(key)
>             new_keys.append(new_key)
>     for old_key, new_key in zip(old_keys, new_keys):
>         state_dict[new_key] = state_dict.pop(old_key)
> 
>     missing_keys = []
>     unexpected_keys = []
>     error_msgs = []
>     # copy state_dict so _load_from_state_dict can modify it
>     metadata = getattr(state_dict, '_metadata', None)
>     state_dict = state_dict.copy()
>     if metadata is not None:
>         state_dict._metadata = metadata
> 
>     def load(module, prefix=''):
>         local_metadata = {} if metadata is None else metadata.get(prefix[:-1], {})
>         module._load_from_state_dict(
>             state_dict, prefix, local_metadata, True, missing_keys, unexpected_keys, error_msgs)
>         for name, child in module._modules.items():
>             if child is not None:
>                 load(child, prefix + name + '.')
>     start_prefix = ''
>     if not hasattr(model, 'bert') and any(s.startswith('bert.') for s in state_dict.keys()):
>         start_prefix = 'bert.'
>     load(model, prefix=start_prefix)
>     if len(missing_keys) > 0:
>         logger.info("Weights of {} not initialized from pretrained model: {}".format(
>             model.__class__.__name__, missing_keys))
>     if len(unexpected_keys) > 0:
>         logger.info("Weights from pretrained model not used in {}: {}".format(
>             model.__class__.__name__, unexpected_keys))
>     if len(error_msgs) > 0:
>         raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
>                            model.__class__.__name__, "\n\t".join(error_msgs)))
>     return model
> ```
> 
> Then:
> 
> ```
> CONFIG_FILE = "/Bert/multi_cased_L-12_H-768_A-12/bert_config.json"
> model = load_BFTC_from_TF_ckpt(CONFIG_FILE, "model.ckpt-6032", num_labels = 2)
> ```
> 
> And :
> 
> ```
> C:\ProgramData\Anaconda3\Lib\site-packages\pytorch_pretrained_bert\modeling.py in load_tf_weights_in_bert(model, tf_checkpoint_path)
>      89 
>      90             elif l[0] == 'output_bias' or l[0] == 'beta':
> ---> 91                 pointer = getattr(pointer, 'bias')
>      92 
>      93             elif l[0] == 'output_weights':
> 
> C:\ProgramData\Anaconda3\Lib\site-packages\torch\nn\modules\module.py in __getattr__(self, name)
>     537                 return modules[name]
>     538         raise AttributeError("'{}' object has no attribute '{}'".format(
> --> 539             type(self).__name__, name))
>     540 
>     541     def __setattr__(self, name, value):
> 
> AttributeError: 'BertForPreTraining' object has no attribute 'bias'
> ```
> 
> Greetings,
> Max

you can try:
model = BertForPreTraining.from_pretrained(BERT_DIR, from_tf=True)<|||||>Hello guys,
here is something I found that seem to do the trick for now:

in `modeling.py:`

In class `class BertForSequenceClassification(BertPreTrainedModel):`

Add :
```
        self.weight = Variable(torch.ones(2, config.hidden_size), requires_grad=True) 
        self.bias = Variable(torch.ones(2), requires_grad=True)
```
to the attributes.

Obviously juste change the right class regarding your needs. I'll let you know if it provokes another error, but for now I can load in memory the trained model I couldn't load before.

Greetings,
Max
<|||||>if use BertFor* class, will not initiate classifier layer.
when i load tf model for  predict/evaluate modify **load_tf_weights_in_bert**


```
        if re.fullmatch(r'[A-Za-z]+_\d+', m_name):
                l = re.split(r'_(\d+)', m_name)
            else:
                l = [m_name]
            if l[0] == 'kernel' or l[0] == 'gamma':
                pointer = getattr(pointer, 'weight')
            elif l[0] == 'output_bias' or l[0] == 'beta':
                if pointer == model:
                    pointer = getattr(pointer, 'classifier')
                pointer = getattr(pointer, 'bias')
            elif l[0] == 'output_weights':
                if pointer == model:
                    pointer = getattr(pointer, 'classifier')
                pointer = getattr(pointer, 'weight')
            elif l[0] == 'squad':
                pointer = getattr(pointer, 'classifier')
```
<|||||>> 
> 
> Ok so you can just convert your Tensorflow model using the command line script (see [here in the readme](https://github.com/huggingface/pytorch-pretrained-BERT#Command-line-interface)) store the converted pytorch model in a folder with the configuration file and then load it in a `BertForTokenClassification` model as follow:
> `BertForTokenClassification.from_pretrained('PATH_TO_YOUR_CONVERTED_MODEL_FOLDER')`
> 
> For the tokenizer, you can use the one associated to the original TensorFlow model from which you did the fine-tuning since you probably didn't change the vocabulary itself.

Hello @thomwolf,

Yes, I am aware of the conversion script and `from_pretrained()` being able to load full models (PyTorch dumps) from converted TF checkpoints. However, in my use case, I did pre-training with BERT using the script `run_pretraining.py` from the BERT repo, and I wanted to do fine-tuning on the many checkpoint steps that I have saved, so it would make more sense for me to load the checkpoints directly.

However, I am aware that my use case is a very niche one, and the others here are talking about a different use case (loading TF finetuned models). Since my issue has been resolved, I will close this issue.