SPM Tokenizer confusion with fairseq Roberta
Hi,

I have pre-trained a custom Roberta Model from scratch with a unigram sentencepiece model (also trained from scratch). I have converted the model from fairseq to huggingface with this [script](https://github.com/huggingface/transformers/blob/master/src/transformers/convert_roberta_original_pytorch_checkpoint_to_pytorch.py) successfully.

I have tried loading the model in huggingface, which was successful, but the issue lies with the tokenizer, I tried using the Roberta tokenizer but it screamed at me because it was looking for merges and vocab. I then loaded my spm model with AlbertTokenizer, but when I try to test it out with a simple fill_mask, the answer tokens are incorrect. How do I correctly use my SPM model with Roberta? I also have dict.txt from fairseq.

Any help would be appreciated!
Hi @hichiaty ,

in that case I would try the CamemBERT Tokenizer or the one from XLM-RoBERTa (last one contains some hacky workaround to align fairseq vocab with SPM...) :)<|||||>hey @stefan-it  thanks for the advice! I am still slightly confused though, I loaded my spm model with CamemBERT but for some reason it still doesn't match the tokens from fairseq's roberta.encode.

Using Fairseq's encode I get:

```
roberta.encode('HELLO')
>tensor([0, 7, 4, 6, 2]) 
```
With CamemBERT I get:

```
tokenizer('HELLO')
>{'input_ids': [5, 45, 36, 3863, 3595, 3595, 19, 5], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1]}
```
I'm just trying to figure out how fairseq works at this point because I don't even pass the spm model to it but a dict.txt file<|||||>Solved, created a custom tokenizer (based on camembert) that uses spm to encode as pieces, then fairseq's dict.txt to get ids.