Maybe it is a bug for Roberta vocab file.
# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Roberta

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [1 ] the official example scripts: (give details below)
* [1] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [1 ] my own task or dataset: (give details below)

Hi Huggingface team,

When I use Roberta, I found that there are so many words start with "Ġ" in the vocab file you provided. I just want to make sure whether it is a bug or not. Around 40,771 words start with "Ġ".

Roberta-base vocab file: https://s3.amazonaws.com/models.huggingface.co/bert/roberta-base-vocab.json

Roberta-large vocab file:
https://s3.amazonaws.com/models.huggingface.co/bert/roberta-large-vocab.json

Best Regards,
Chen





    

Hi @czheng17,

that's because RoBERTa uses a byte-level BPE (like GPT-2), see section 4.4 in the original paper.

A good explanation can be found here:

https://github.com/pytorch/fairseq/issues/1716#issuecomment-588750983

:)<|||||>Hi @stefan-it ,

Interesting! Thank you for your help!

Best,