[ Structure of LM vocab trained from scratch ]
## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

Hi,
I am trying to create a BERT LM trained from scratch and I have a question about the tokenizer.
I have a big text corpus and I trained a tokenizer with SentencePiece with 32K as dimension of the vocabulary. Then I applied a transformation from SentencePiece notation to WordPiece notation in order to be compatible with BERT.
The result is that my dictionary has this structure:

----------------------------------------

MY DICTIONARY
```
0 [PAD]
1 [UNK]
2 [CLS]
3 [SEP]
4 [MASK]
5 ,
6 .
7 '
... 
.... [ ALL USED TOKENS ]
....
100 ##ndo
...
.... [ ALL USED TOKENS ]
....
31741 [ disgustat ]
31742 [ UNUSED TOKEN]
.......
.......  [ ALL UNUSED TOKENS ]
.......
31999 [ UNUSED TOKEN]
```
And all the unused tokens are at the end of the vocab, in my case from 31742 to 31999.

---------------------------

And this is the STANDARD VOCAB for BERT:


BERT cased_L-12_H-768_A-12 VOCABULARY

```
0  [PAD]
1 [unused1]
2  ........ 
3  ........ [ ALL UNUSED TOKENS ]
4  .......
...........
100 [UNK]
101 [CLS]
102 [SEP]
103 [MASK]
[unused100]
[unused101]
!
"
..... [ ALL USED TOKENS ]
```
----------------------------

My question is: the fact that the SPECIAL TOKENS in MY DICTIONARY are in different positions than the Standard BERT VOCABULARY , can be a problem? Do you think I should keep the same positions of BERT vocab also for my dictionary? Also for unused token I have the same doubts. 

( I saw that in the sentence piece training it's possibile to specify the exact positions of special tokens, but my question is if the position of special tokens will affect the LM training in some way )

Thank you





I don't think it is a problem. 
Your model will learn the embeddings of the words in your own dictionary.

Actually Nothing will be unchanged if you changed dictionary position as well as you keeped the embedding weight just the same order with your dictionary. <|||||>Thanks @karajan1001. 
I am not sure if I understood well the last sentence. 
You meant that I must not change the dictionary after I train the LM based on that? 
After that I train the SentencePiece tokenizer and the relative vocabulary is given, than the vocab is an input of Language Model training and it cannot be changed anymore.<|||||>> After that I train the SentencePiece tokenizer and the relative vocabulary is given, than the vocab is an input of Language Model training and it cannot be changed anymore.

I think so. The vocabulary tells the model which array to get with the input tokens.
