Why isn't BERT doing wordpiece tokenization?
My code is
```
from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
text = '''why isn't my card working'''
encoded = tokenizer.encode(text, add_special_tokens=False)
text_tokenized = tokenizer.decode(encoded, clean_up_tokenization_spaces=False)
print(text_tokenized)
```
and the output result (the tokenization) is
```
why isn ' t my card working
```

But this isn't the wordpiece nokenization BERT should be using. E.g., `working` should be tokenized as `work ##ing`. Is there anything wrong with my code? And will the fact we are not using wordpiece tokenization decrease BERT's performance?

Even if I do `add_special_tokens=True` when encoding, I get

```
[CLS] why isn ' t my card working [SEP]
```

which is still not wordpiece tokenization.<|||||>When using `encode` and `decode` you're performing the full tokenization steps each time:

encode: tokenizing -> convert tokens to ids
decode: convert tokens to ids -> detokenizing
 
If you want to see the middle step, you can use the `tokenize` method:

```py
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
text = '''why isn't my card working'''
print(tokenizer.tokenize(text))  # ['why', 'isn', "'", 't', 'my', 'card', 'working']
```

All the words are in the vocabulary, but if you use more complex words:

```py
tokenizer.tokenize("Why isn't my text tokenizing")  # ['why', 'isn', "'", 't', 'my', 'text', 'token', '##izing']
```

You'll see the `##ing` you were looking for.

<|||||>Looks great! Thank you!