”never_split“ not working on BertTokenizer
### System Info

transformers 4.28.1
python 3.8.13

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

- I load BertTokenizer using my own vocab.txt, and add _'[outline]'_ into _never_split_, which is included in my vocab.txt. However,  _'[outline]'_ got splitted. Following is my code:

`tokenizer = BertTokenizer.from_pretrained(pretrained_path,never_split=['[outline]'])   
input = "。[outline]"  
print(tokenizer.tokenize(input))  # ['。', '[', 'out', '##line', ']']
`

- I also do:
`print(tokenizer.basic_tokenizer.tokenize(input))  #['。', '[', 'outline', ']']`


### Expected behavior

When I do:
`tokenizer.tokenize("。[outline]")`
Get the result as `['。', '[outline]']`, the tokens in never_split don't be splited.

cc @ArthurZucker @younesbelkada <|||||>The '[' or ']' in BertTokenizer is punctuation, it will be split at first. And the `outline` or `[outline]` is not in vocab, its will be set UNK. It doesn't seem to make sense anymore.
Look the code: https://github.com/huggingface/transformers/blob/main/src/transformers/models/bert/tokenization_bert.py#L446
![image](https://github.com/huggingface/transformers/assets/26846598/8460c8fb-e8d2-42e8-873b-5d0e182d88c3)
<|||||>> The '[' or ']' in BertTokenizer is punctuation, it will be split at first. And the `outline` or `[outline]` is not in vocab, its will be set UNK. It doesn't seem to make sense anymore. Look the code: https://github.com/huggingface/transformers/blob/main/src/transformers/models/bert/tokenization_bert.py#L446 ![image](https://user-images.githubusercontent.com/26846598/240332356-8460c8fb-e8d2-42e8-873b-5d0e182d88c3.png)

Thanks for replying. As stated before, I am using my own vocab, and ’[outline]‘ is in it,

tokenizer = BertTokenizer.from_pretrained(my_vocab_path, never_split='[outline]')
print(tokenizer.convert_tokens_to_ids('[outline]'))
print(tokenizer.convert_tokens_to_ids('。'))
print(tokenizer.tokenize('。[outline]'))

![1684871975903](https://github.com/huggingface/transformers/assets/48902561/420a2dc6-ed88-4005-add0-e75e5c21bad9)
<|||||>Hey, reading the doc for the `BertTokenizer`, you should be using the `do_basic_tokenize=True` argument, as mentioned [here](https://github.com/ArthurZucker/transformers/blob/f732a643ab47a324405dc583532bbbfc45e2d8dc/src/transformers/models/bert/tokenization_bert.py#L153). <|||||>> Hey, reading the doc for the `BertTokenizer`, you should be using the `do_basic_tokenize=True` argument, as mentioned [here](https://github.com/ArthurZucker/transformers/blob/f732a643ab47a324405dc583532bbbfc45e2d8dc/src/transformers/models/bert/tokenization_bert.py#L153).

Your link is broken, it says '404 - page not found'?
Plus, `do_basic_tokenize=True` is default setting. Even if I add it intentionally, the result stays the same.

tokenizer = BertTokenizer.from_pretrained(my_vocab_path, never_split=['[outline]'], do_basic_tokenize=True)
print(tokenizer.tokenize('。[outline]')) # ['。', '[', 'out', '##line', ']']

Correct me if I do anything wrong.<|||||>Sorry, anyway the argument was set to `True` by default so that's not the problem. 
Let's me investigate, in the mean time doing `tokenizer.add_token("[outline]", special_token = True)`" should (I think) prevent it from being split<|||||>( the doc mentions :            
```python
 never_split (`List[str]`, *optional*)
                Kept for backward compatibility purposes. Now implemented directly at the base class level (see
                [`PreTrainedTokenizer.tokenize`]) List of token not to split.
```<|||||>The best solution is to add the token to the list of special tokens using the `add_token` method<|||||>Yeah, add it as special_token does take care of the splitting problem. But in the latter process, I will decode with argument `skip_special_tokens=True`. Then the token will be skipped, while I don't want it be. For now, I add it to the special token list, but I still suggest fixing the `never_split` argument.
<|||||>Then that means that the token that you want to add is not `special`. I think that if you add it without the `special_token` set to `True` it should not be spilt no? <|||||>Without `special_token` set to True, it will be splitted.<|||||>No it won't : 
```python 
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased", use_fast=False)
tokenizer.add_tokens("[outline]")
tokenizer.added_tokens_encoder
>>>  {'[outline]': 30522}

tokenizer.encode("[outline]")
>>> [101, 30522, 102]
tokenizer.decode(tokenizer.encode("[outline]"))
>>> '[CLS] [outline] [SEP]'
print(tokenizer.tokenize(". [outline]"))
>>> ['.', '[outline]']

tokenizer.decode(tokenizer.encode(". [outline]"), skip_special_tokens=True)
>>> '. [outline]'<|||||>In your case, it won't. But I am using a different vocab.txt, it splits.<|||||>Seems like `'[outline]'` will not be added anyway, since it's already in the vocab.<|||||>I don't understand. You have a very specific usage, where you don't want to split `[outline]` that is already in your vocab. 
The basic tokenizer works as expected: `tokenizer.basic_tokenizer.tokenize("[outline]")` will not split it. 
When you are calling `tokenize` on the `BertTokenizerClass` the `_tokenize` function is then called, which relies on the `all_special_ids`. That means that the token should be added to both lists. 
```python 
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased", use_fast=False, never_split= ["outline"])
tokenizer.add_tokens("[outline]")
```
I am guessing that this should work
<|||||>Edit: I manually added "[outline]" to my vocab and it worked for both the solution I gave you<|||||>Unfortunately, it still doesn't work on my vocab. I think it strictly related to the vocab. So far, only adding it to the special tokens works for me. 
Also, shat I posted before is basic tokenizer split it, `tokenizer.basic_tokenizer.tokenize("[outline]")` splits it into` '[',  'outline', ']'`. The tokenizer then send the split tokens to do Wordpiece instead of fix it to the origin `'[outline]'`. I think that may be the reason. <|||||>[vocab.txt](https://github.com/huggingface/transformers/files/11564244/vocab.txt)
Here is my vocab, you can try on it.<|||||>I tried loading a tokenizer using your vocabulary and I cannot reproduce your issue.
Try downloading the latests `transformer` version!<|||||>Why......
I've updated transformers to 4.29.2, still the same result....
here is my code

```python
tokenizer = BertTokenizer.from_pretrained('../base_model/vocab.txt', never_split= ["[outline]"])
tokenizer.add_tokens("[outline]")
print(tokenizer.tokenize("。[outline]"))
# ['。', '[', 'out', '##line', ']']
``` 
<|||||>Can you try `tokenizer = BertTokenizer.from_pretrained('../base_model', never_split= ["[outline]"])`
Also I would suggest you create a colab , this will make sure that your cache is not messing with this. <|||||>Here is the Colab result:
![1685020473827](https://github.com/huggingface/transformers/assets/48902561/c8701208-1152-4073-b22f-b337676dc138)
<|||||>can you share a link to the colab, I'll try to reproduce and modify a copy 😉 
<|||||>Also you did not add the token using `add_token(..., special_token = False)`<|||||>Another solution is to initialise the tokenizer using `...from_pretrained( path, additional_special_tokens = ["[outline]"]) ` <|||||>https://colab.research.google.com/drive/1EStD5K_lQM0-PgMUQ8z273TzAUcgY2IV?usp=sharing
You need to roll down to the bottom to see the code, `add_token(...)` already added.

`additional_special_tokens` add `[outline]` into special tokens too, so it works fine. But it still meets the `skip_special_token` problem. Anyway, this issue is about 'never_split' argument not working, so let's focus on this.<|||||>Thanks a lot. 
Indeed the token is not added, because in the `_add_token` a check prevent it to be added if it is already in the vocab. 
Workaround: 
```python 
tokenizer.added_tokens_encoder.update({"[outline]":85})
tokenizer.added_tokens_decoder.update({85:"[outline]"})
tokenizer.unique_no_split_tokens = sorted(set(tokenizer.unique_no_split_tokens).union({"[outline]"})) 
tokenizer._create_trie(tokenizer.unique_no_split_tokens)
```
Its is not really elegant indeed. Also adding a token means that whether or not it is in the vocab, we want it to be in the added tokens, so I think it makes sense to add it, even if it exists. WDYT @Narsil 
edit: I think it comes down to a choice, and both could have pos and cons.<|||||>About never split, the last commit is 4 years old, it has never been touch, and I'd rather we find a way to work around your problem using new code rather than changing legacy code! <|||||>Glad we are on the same page in the end. <|||||>I am not entirely sure yet whether or not we will support this as the fast ones don't, and my few tests appear to show that it might not be optimal<|||||>For now closing as `wontfix`, if more people require such usage, will make it available. 
TLDR: adding a tokens as `AddedToken` when it is already in the vocab.
Will not fix because:
- fast does not support this and we want to keep fast stable 
- it's a breaking change 
- seems to be a specific usage, not the one intended for `add_tokens`