models and tokenizers trained with pytorch_pretrained_bert are not compatible with transformers
# 📚 Migration

## Information

The models and tokenizers in transformers 2.3.0 are backward incompatible with pytorch_pretrained_bert 0.6.2.

## Details

```
>>> import transformers
>>> transformers.__version__
'2.3.0'
>>> import pytorch_pretrained_bert
>>> pytorch_pretrained_bert.__version__
'0.6.2'
>>> from pytorch_pretrained_bert import OpenAIGPTTokenizer
>>> tokenizer = OpenAIGPTTokenizer.from_pretrained("runs/mymodel")
ftfy or spacy is not installed using BERT BasicTokenizer instead of SpaCy & ftfy.
>>> len(tokenizer)
40483
>>> from transformers import OpenAIGPTTokenizer
>>> tokenizer = OpenAIGPTTokenizer.from_pretrained("runs/mymodel")
ftfy or spacy is not installed using BERT BasicTokenizer instead of SpaCy & ftfy.
>>> len(tokenizer)
40478
>>> 
```

`runs/mymodel` contains a model that was trained using pytorch_pretrained_bert 0.6.2, specifically with the `transfer-learning-conv-ai` repo.

Expected behavior from transformers 2.3.0: `len(tokenizer)` must be 40483, like with pytorch_pretrained_bert 0.6.2.

## Environment

* OS: Amazon Linux
* Python version: 3.6
* PyTorch version:
* `pytorch-transformers` or `pytorch-pretrained-bert` version (or branch): 0.6.2
* `transformers` version (or branch): 2.3.0
* Using GPU? Yes
* Distributed or parallel setup? N/A

## Checklist

- [Y] I have read the migration guide in the readme.
 ([pytorch-transformers](https://github.com/huggingface/transformers#migrating-from-pytorch-transformers-to-transformers);
  [pytorch-pretrained-bert](https://github.com/huggingface/transformers#migrating-from-pytorch-pretrained-bert-to-transformers))
- [Y] I checked if a related official extension example runs on my machine.



-------

UPDATE: this issue is also the case for models, see below:

```
>>> from pytorch_pretrained_bert import OpenAIGPTLMHeadModel
>>> model = OpenAIGPTLMHeadModel.from_pretrained("runs/mymodel")
>>> from transformers import OpenAIGPTLMHeadModel
>>> model = OpenAIGPTLMHeadModel.from_pretrained("runs/mymodel")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/transformers/modeling_utils.py", line 486, in from_pretrained
    model.__class__.__name__, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for OpenAIGPTLMHeadModel:
	size mismatch for transformer.tokens_embed.weight: copying a param with shape torch.Size([40483, 768]) from checkpoint, the shape in current model is torch.Size([40478, 768]).
```

Can you share your vocabulary so we can have a look at the differences?<|||||>@thomwolf I took the `vocab.json` located inside `runs/mymodel` and the `openai-gpt-vocab.json` hosted in the Hugging Face [S3](https://s3.amazonaws.com/models.huggingface.co/bert/openai-gpt-vocab.json) bucket and compared the two as follows:

```
>>> with open("openai-gpt-vocab.json", "r") as f:
...     a = json.load(f)
... 
>>> with open("vocab.json", "r") as f:
...     b = json.load(f)
... 
>>> a == b
True
```

So the `vocab.json` inside `runs/mymodel` is exactly the same as that hosted in the S3 bucket.

------

Some of the other files located in `runs/mymodel` include `config.json`, `merges.txt` and `special_tokens.txt`. If you're interested in the contents of the last file, it is the following:

```
<bos>
<eos>
<speaker1>
<speaker2>
<pad>
```<|||||>From looking at the different releases, I would assume that in the current master branch, more tokens are added (don't have the time to dig through to see where they come from). It happens in `__len__`:

https://github.com/huggingface/transformers/blob/5a6b138b00eef2506e0fc2c6088fb81c064161bf/src/transformers/tokenization_utils.py#L535-L537

That being said, to only get the size of the base vocabulary, there is another method called `vocab_size` which is implemented in the OpenAI tokenizer like so:

https://github.com/huggingface/transformers/blob/5a6b138b00eef2506e0fc2c6088fb81c064161bf/src/transformers/tokenization_openai.py#L115-L117

In the 0.6.2 release, the length is the encoder + the special tokens.

https://github.com/huggingface/transformers/blob/b832d5bb8a6dfc5965015b828e577677eace601e/pytorch_pretrained_bert/tokenization_openai.py#L157-L158

So it seems that the added special tokens are different in the releases, that is `self.added_tokens_encoder` (current master) vs `self.special_tokens` (0.6.2).

Perhaps when I have more time I can look more closely into this.<|||||>@BramVanroy yes indeed, but this is a more serious, systemic issue since it occurs not just for tokenizers but for models as well, as I noted in the original post.

Basically anyone who trained models with pytorch_pretrained_bert 0.6.2 and later upgraded their entire code-base (both training and inference) to transformers 2.3.0 will have to discard those models and train fresh ones.<|||||>Can we get an update on this @thomwolf and @LysandreJik ? Is this expected behavior? Should we have to re-train all models?<|||||>Hi, I'm having a hard time replicating this on my end. Could you try specifying explicitly to the model that you're trying to load a state dict of a certain size, by using the configuration? If I understand correctly, you have a directory `runs/mymodel `, which must contain a `config.json` file and a `pytorch_model.bin` file.

Loading the model as such fails:

```py
from transformers import OpenAIGPTModel

model = OpenAIGPTModel.from_pretrained("runs/mymodel")
```

with the error you mentioned above. Can you try by loading the configuration separately, and then instantiating the model with it, as follows:

```py
from transformers import OpenAIGPTModel, OpenAIGPTConfig

config = OpenAIGPTConfig.from_pretrained("runs/mymodel", vocab_size=40483)
model = OpenAIGPTModel.from_pretrained("runs/mymodel", config=config)
```<|||||>@LysandreJik when I try loading the model like you mentioned above, I don't get any errors. However, `vocab_size=40483` is something that `from_pretrained()` is supposed to figure out from the contents of `runs/mymodel`, right?

And yes, `runs/mymodel` contains all of the following:
```
model_training_args.bin
config.json
vocab.json
special_tokens.txt
merges.txt
checkpoint_mymodel_1.pth
checkpoint_mymodel_2.pth
pytorch_model.bin
```

I trained a separate model with transformers 2.3.0 and its trained model directory within `runs/` didn't have `special_tokens.txt`, but instead I see `special_tokens_map.json` and `added_tokens.json`. These files didn't exist when training models with pytorch_pretrained_bert 0.6.2, understandably because you guys changed the special tokens from a list to a dictionary when you upgraded from 0.6.2.

However, this upgrade should not mean that old models are no longer supported.

Also, while what you are suggesting works for loading the model, can you tell me what would ensure that both tokenizers return the exact same length, i.e., 40483?

Basically what the `OpenAIGPTTokenizer` needs to ensure is that it not only reads the contents of `vocab.json` from `runs/mymodel`, but also the contents of `special_tokens.txt`. This was happening with pytorch_pretrained_bert 0.6.2, and it should continue to happen in transformers 2.3.0. I understand that you guys wanted to change the special tokens from a list to a dictionary for 2.3.0, but that change should be backward compatible.

Look for a `special_tokens.txt` if it exists, and use it if it does.

See how this was being done in 0.6.2 (I'm only pasting the example of the tokenizer, but the same is applicable to the model as well): https://github.com/huggingface/transformers/blob/v0.6.2/pytorch_pretrained_bert/tokenization_openai.py#L128L132
<|||||>@thomwolf @LysandreJik Any updates?

It would be great if y'all could let everyone know if you're working to fix this.

Alternately, please make a recommendation to the community on how to handle this scenario. Should we revert back to pytorch_pretrained_bert 0.6.2 for our old models? Should we just re-train all old models?<|||||>Hi @g-karthik we **don't** plan to assure backward compatibility between `pytorch-pretrained-bert` and `transformers`'s tokenizers.

There were deep changes in the way we handle added tokens from `pytorch-pretrained-bert` (in which it was basic, specific to BERT and broken on some hedge cases) to `transformers` (in which it is more reliable and unified across models).

So in your case, my recommendation is thus to stick with `pytorch_pretrained_bert` indeed.<|||||>@thomwolf Thanks for letting us know!

Just to clarify, it's not just the tokenizers, it's also the models trained with `pytorch_pretrained_bert` 0.6.2 (which includes more than just BERT, btw) that will not be compatible with `transformers` 2.3.0 at run-time/inference-time.