Why does RoBERTa not label custom tokens as special tokens?
# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     -->

## Details
<!-- Description of your issue -->
Recently I’ve been working on training from scratch (with no pre-training whatsoever) a RoBERTa model starting from the code in [this tutorial][1].

I am working with a specific corpus I prepared according to my own format

    <s> ID 10 <i> COUNTRY USA <i> CAPITAL Washington DC </s>

Here, `<s>` is supposed to be the `<bos>` token, `<i>` is my own `<sep>` token and `</s>` is the `<eos>` token.

I noticed that one of the parameters that can be passed to the `tokenizer.encoder_plus` function is `add_special_tokens`.

If `add_special_tokens=True`, the encoding of the sentence

    <s> COUNTRY USA <i> CAPITAL Washington DC </s>

becomes

    <s> <s> COUNTRY USA <i> CAPITAL Washington DC </s> </s>

and the `special_tokens_mask` is `1 0 0 0 0 0 0 0 0 1` (1, 8 zeros, 1).

When I tried `add_special_tokens=False` on the same sentence `<s> COUNTRY USA <i> CAPITAL Washington DC </s>` the result of the encoding was correct:
`<s> COUNTRY USA <i> CAPITAL Washington DC </s>` 
However, the `special_tokens_mask` is `0 0 0 0 0 0 0 0` (8 zeros).


I therefore suspected that (for some reason) my token `<i>` were not recognized as a special token, so this time I tried to encode a sentence setting `add_special_tokens=False` and `True`, using only `<s>` and `</s>`. 

Token 4 is `<s>`, which is both the `<cls>` token and the `<bos>` token, while token 6 is `</s>`, which acts as `sep_token` and `eos_token`.

The first case has `add_special_tokens=False` and its special token mask is full of 0’s, the first case has `add_special_tokens=True` and as expected the `<bos>` and `<eos>` tokens were added by the algorithm. The special tokens mask only shows the first and last tokens as “1”, while all the other 4’s and 6’s are missing.

    # original sentence 
    <s> ID 10 </s><s> NAME Trevor </s> <s> COUNTRY USA </s><s> CAPITAL Washington DC </s>
    # encoding with add_special_tokens=False
    [4, 0, 232, 28, 27, 6, 4, 1, 232, 63, 93, 80, 97, 90, 93, 6, 4, 2, 232, 64, 62, 44, 6, 4, 3, 232, 66, 76, 94, 83, 84, 89, 82, 95, 90, 89, 232, 47, 46, 6]
    # encoding with add_special_tokens=True
    [4, 4, 0, 232, 28, 27, 6, 4, 1, 232, 63, 93, 80, 97, 90, 93, 6, 4, 2, 232, 64, 62, 44, 6, 4, 3, 232, 66, 76, 94, 83, 84, 89, 82, 95, 90, 89, 232, 47, 46, 6, 6]
    # special_tokens_mask with add_special_tokens=True
    [1, 0, 0,   0,  0,  0, 0, 0, 0,   0,  0,  0,  0,  0,  0,  0, 0, 0, 0,   0,  0,  0,  0, 0, 0, 0,   0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,   0,  0,  0, 0, 1]


After testing both versions, the result I got with `add_special_tokens=True` was very good, while the second with `add_special_tokens=False` failed.

This raises a few issues that I wasn’t able to solve on my own:

 - How can I access the `special_tokens_mask` to correct it to what it should be?
   
 -  Where does RoBERTa make use of that mask, if it does?
   
 -  Is there a method for setting the mask to something I want? e.g. the mask for `<s> ID 10 <i> COUNTRY USA </s>` should be `1 0 0 1 0 0 1`  if `<s>`, `</s>` and `<i>` are special tokens.
   
 -  If RoBERTa is not the correct model to do this, what model should I go for?

Thanks a lot!



  [1]: https://huggingface.co/blog/how-to-train




<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:
https://stackoverflow.com/questions/63917142/why-does-roberta-not-label-custom-tokens-as-special-tokens
https://discuss.huggingface.co/t/why-does-roberta-behave-differently-if-i-provide-a-corpus-that-contains-special-tokens/1083/2
Hi! That's a very detailed issue, very cool! Thanks for providing so much information. I believe you have a misconception about how to use `add_special_tokens` and how the special token mask is created.

## Adding special tokens

Firstly, if you use the `add_special_tokens=True` flag, that means you expect the tokenizer to handle the special tokens on its own: you should not provide your special tokens. As you've shown in the example, if you encode a sequence with this flag, you'll get the encoded sequence surrounded by the `<bos>` and `<eos>` tokens:

```py
>>> from transformers import RobertaTokenizer
>>> tokenizer = RobertaTokenizer.from_pretrained("roberta-base")
>>> input_ids_single_sequence = tokenizer.encode("This is a sequence", add_special_tokens=True)
>>> tokenizer.decode(input_ids_single_sequence)
"<s>This is a sequence</s>"
```

In your case, you're actually using a `<sep>` token as you're encoding two sequences. You can use the tokenizer to do the same, by passing two sequences to it:

```py
>>> input_ids_sequence_pair = tokenizer.encode("This is a sequence", "This is another", add_special_tokens=True)
>>> tokenizer.decode(input_ids_sequence_pair)
'<s>This is a sequence</s></s>This is another</s>'
```

⚠️ : Please note that the RoBERTa tokenizer is built using only `<s>` (the BOS token) and `</s>` (the SEP token), with two `</s></s>` as the separator.

## Special token mask

If you try generating the special token mask here, you'll see that it has no issue generating it, __as long as you use the `already_has_special_tokens=True` flag__. You've already built the sequences with special tokens, so you should set this flag to `True`.

```py
>>> tokenizer.get_special_tokens_mask(input_ids_single_sequence, already_has_special_tokens=True)
[1, 0, 0, 0, 0, 1]
>>> tokenizer.get_special_tokens_mask(input_ids_sequence_pair, already_has_special_tokens=True)
[1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 1]
```

## Custom special tokens

In your case you want to use different special tokens than what is done with the original RoBERTa implementation. That's okay, but then you should specify it to your tokenizer:

```py
>>> tokenizer.add_special_tokens({'sep_token': '<i>'})
1  # Returns the number of added tokens
```

When calling the `encode` method, you'll now correctly have the specified separator token where it should be:

```py
>>> input_ids_sequence_pair = tokenizer.encode("This is a sequence", "This is another", add_special_tokens=True)
>>> tokenizer.decode(input_ids_sequence_pair)
'<s>This is a sequence <i> <i> This is another <i>'
```

## Setting a custom encoding behavior

In your use-case you show different behavior than the original RoBERTa tokenizer implementation, where you want your sequences encoded as:

`<s> [SEQ_A] <i> [SEQ_B] </s>`

If you still want to rely on the `add_special_tokens=True` flag in `encode` and the accurate special tokens mask even without respecting the original behavior, you can do so by subclassing the `RobertaTokenizer` (or the fast version) and overriding the following methods `build_inputs_with_special_tokens`, `get_special_tokens_mask`, `create_token_type_ids_from_sequences`:

```py
class MyRobertaTokenizer(RobertaTokenizer):
    def build_inputs_with_special_tokens(
        self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None
    ) -> List[int]:
        if token_ids_1 is None:
            return [self.cls_token_id] + token_ids_0 + [self.eos_token_id]
        cls = [self.cls_token_id]
        sep = [self.sep_token_id]
        eos = [self.eos_token_id]
        return cls + token_ids_0 + sep + token_ids_1 + eos

    def get_special_tokens_mask(
        self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None, already_has_special_tokens: bool = False
    ) -> List[int]:
        if already_has_special_tokens:
            if token_ids_1 is not None:
                raise ValueError(
                    "You should not supply a second sequence if the provided sequence of "
                    "ids is already formatted with special tokens for the model."
                )
            return list(map(lambda x: 1 if x in [self.sep_token_id, self.cls_token_id] else 0, token_ids_0))

        if token_ids_1 is None:
            return [1] + ([0] * len(token_ids_0)) + [1]
        return [1] + ([0] * len(token_ids_0)) + [1] + ([0] * len(token_ids_1)) + [1]

    def create_token_type_ids_from_sequences(
        self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None
    ) -> List[int]:
        sep = [self.sep_token_id]
        cls = [self.cls_token_id]
        eos = [self.eos_token_id]

        if token_ids_1 is None:
            return len(cls + token_ids_0 + eos) * [0]
        return len(cls + token_ids_0 + sep + token_ids_1 + eos) * [0]
```

This will then correctly encode your methods (given you initialize the correct special token as is done in the following snippet):

```py
tokenizer = MyRobertaTokenizer.from_pretrained("roberta-base")
tokenizer.add_special_tokens({'cls_token': '<s>', 'sep_token': '<i>', 'eos_token': '</s>'})

print(tokenizer.decode(tokenizer.encode("This is a sequence", add_special_tokens=True)))
# <s>This is a sequence</s>

print(tokenizer.decode(tokenizer.encode("This is a sequence", "This is another", add_special_tokens=True)))
# <s>This is a sequence <i> This is another</s>
```

## Conclusion

I hope this helps you! If you need a better understanding of what's happening, I encourage you to read the following documentation pages:
- [RobertaTokenizer](https://huggingface.co/transformers/model_doc/roberta.html#transformers.RobertaTokenizer)
- [Tokenizer main class](https://huggingface.co/transformers/main_classes/tokenizer.html)<|||||>Thanks a lot for the extremely detailed answer! That cleared up a lot of doubts. I worked with some of the code you provided and got to the conclusion that RoBERTa as it is right now is probably not the best fit for my problem, so I'll have to work around that. <|||||>What a wonderful question and answer!