[CLS] & [SEP] tokens missing in documentation
https://github.com/huggingface/transformers/blob/fd97761c5a977fd22df789d2851cf57c7c9c0930/transformers/modeling_bert.py#L1017-L1023
In this example of bert for token classification input sentence is encoded, but [CLS] & [SEP] tokens are not added. Is this intentional or just a typo?
Do I need to add [CLS] & [SEP] tokens when I fine tune base bert for sequence classification or token classification?
@hawkeoni  [CLS] and [SEP] tokens are added automatically as long as you use the tokenizer,  ```BertTokenizer```<|||||>@enzoampil It doesn't seem to work.
The following code
```python
from transformers import BertTokenizer
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
sentence = "Hello there, General Kenobi!"
print(tokenizer.encode(sentence))
print(tokenizer.cls_token_id, tokenizer.sep_token_id)
```
produces the next output: 

[7592, 2045, 1010, 2236, 6358, 16429, 2072, 999]
101 102

As you can see, cls and sep tokens are not in the list.<|||||>@hawkeoni please try ```print(tokenizer.encode(sentence, add_special_tokens=True))```<|||||>@enzoampil I'm sorry, but you're missing the point that the documentation is plainly wrong and misleading. That's why I asked whether the tokens should be added at all.<|||||>@hawkeoni Apologies, yes I did miss your point.

Is this intentional or just a typo? **Looks like a typo since special tokens weren't added. Setting ```add_special_tokens=True``` should make this correct (will add this in).**

Do I need to add [CLS] & [SEP] tokens when I fine tune base bert for sequence classification or token classification? **Yes, I believe this is currently handled by ```load_and_cache_examples``` in the sample training scripts (e.g. ```run_ner.py```)**<|||||>@enzoampil Thanks for your answer! If you plan on fixing this typo, please, fix it everywhere, so this issue never occurs again.
You can find it with 
```bash
grep -iR "input_ids = torch.tensor(tokenizer.encode(" .
```
<|||||>@hawkeoni thanks for the bash script reco. Ended up using it :)<|||||>Thank you both for that.
Please note that the special tokens will be added by default from now on (already on master and in the coming release).