Different output encode and encode_plus
Hi everyone,

I'm struggling with the following scenario:

I have the following input:

```
sentence = "Sheldon : If a photon is directed through a plane with two slits in it and either is observed . Sheldon : . it will not go through both . If unobserved , it will . UNKNAME : If it 's observed after it left the plane , before it hits its target . Sheldon : . it will not have gone through both slits . Agreed . Leonard : What 's your point ? Sheldon : There 's no point , I just think it 's a good idea for a T shirt . UNKNAME : Excuse me . Hang on . Leonard : One across is Aegean , eight down is Nabokov . Leonard : Twenty six across is MCM . Leonard : Fourteen down is . Leonard : Move your finger . UNKNAME : . phylum , which makes 14 across Port Au Prince . Leonard : See , Papa Doc 's capital idea , that 's Port Au Prince . Leonard : Haiti . UNKNAME : Can I help you ? Yes . UNKNAME : Um , is this the high IQ sperm bank ?"
```

I am using bert-base-uncased and I get different length of tokens depending if I use `tokenizer.encode` or `tokenizer.encode_plus`.

Below is an example:

```
test = tokenizer.encode_plus(sentence, add_special_tokens=True, max_length=512)["input_ids"]
print(test)
214
```
```
test2 = test2 = tokenizer.encode_plus(sentence, add_special_tokens=True, max_length=512)["input_ids"]
print(test2)
189
```

In above scenario I expect the amount of tokens to be the same length. I looked at the documentation but I cannot find an explanation for the difference. It is problematic for me because I need to use tokenizer.batch_encode_plus but for my model I expect and need the length of 189 instead of 214.

Can someone please explain why the output is different and how to make encode.plus output the same as encode?

Thanks in advance ;)







I made a mistake in my code. The output is the same.