How to use 16 token types in pretrained Albert/BERT?
I have a dialogue task and I use token type to distinguish the diffenrent state of the different speeches, but all the pretrained models I can find are of type_vocab_size=2. To accomplish my goal, I have to rewrite many codes in a dirty way. So I want to ask is there an elegant way to restore the pretrained weights and ignore the token type embeddings at the same time? Roughly modifying the type_vocab_size in the given config.json will certainly raise an error
Maybe I don't get it, but can't you simply do:

```
from transformers.modeling_bert import BertConfig, BertModel

bconfig = BertConfig.from_pretrained('bert-base-uncased')
bconfig.type_vocab_size = 16
model = BertModel(bconfig)
model.parameters
# ...
#    (token_type_embeddings): Embedding(16, 768)
# ...
```<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>> Maybe I don't get it, but can't you simply do:
> 
> ```
> from transformers.modeling_bert import BertConfig, BertModel
> 
> bconfig = BertConfig.from_pretrained('bert-base-uncased')
> bconfig.type_vocab_size = 16
> model = BertModel(bconfig)
> model.parameters
> # ...
> #    (token_type_embeddings): Embedding(16, 768)
> # ...
> ```

Yes. What I can do is to reassign the token type embeddings after init, and thing is if there is any risk to do this. But I don't continue on this because my dialogue task is too difficult for almost language model even with BERT hahhhhh<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
