Tokenizer is adding ## to every word from the second.
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.4.2
- Platform: Linux-5.8.0-44-generic-x86_64-with-Ubuntu-20.04-focal
- Python version: 3.6.13
- PyTorch version (GPU?): 1.8.1+cu102 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help

 tokenizers: @LysandreJik



## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [ O ] the official example scripts: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. The tokenizer is adding ## to every words from the second.

For example, the code is:
text = 'テレビでサッカーの試合を見る。'
tokenized_text = tokenizer.tokenize(text)

Output is expected to be:['テレビ', 'で', 'サッカー', 'の', '試合', 'を', '見る', '。']
But I get  ['テレビ', '##で', '##サッカー', '##の', '##試', '##合', '##を', '##見', '##る', '。']

I dont know why it add ## at the start of the words...



```
import torch
from transformers import BertJapaneseTokenizer, BertForMaskedLM


# Model path
def sel_model(pre_model='32d'):
    if pre_model == '32d':
        sel = '/home/Xu_Zhenyu/cl-tohoku/BERT-base_mecab-ipadic-bpe-32k_do-whole-word-mask/'
    elif pre_model == '4d':
        sel = '/home/Xu_Zhenyu/cl-tohoku/BERT-base_mecab-ipadic-char-4k_do-whole-word-mask/'
    elif pre_model == '32n':
        sel = '/home/Xu_Zhenyu/cl-tohoku/BERT-base_mecab-ipadic-bpe-32k_no-whole-word-mask/'
    elif pre_model == '4n':
        sel = '/home/Xu_Zhenyu/cl-tohoku/BERT-base_mecab-ipadic-char-4k_no-whole-word-mask/'
    else:
        sel = '/home/Xu_Zhenyu/cl-tohoku/BERT-base_mecab-ipadic-bpe-32k_do-whole-word-mask/'
    return sel


# Load pre-trained tokenizer
tokenizer = BertJapaneseTokenizer.from_pretrained(sel_model())

# Tokenize input
text = 'テレビでサッカーの試合を見る。'
tokenized_text = tokenizer.tokenize(text)
# ['テレビ', 'で', 'サッカー', 'の', '試合', 'を', '見る', '。']

# Mask a token that we will try to predict back with `BertForMaskedLM`
masked_index = 2
tokenized_text[masked_index] = '[MASK]'
print(tokenized_text)
# ['テレビ', 'で', '[MASK]', 'の', '試合', 'を', '見る', '。']

# Convert token to vocabulary indices
indexed_tokens = tokenizer.convert_tokens_to_ids(tokenized_text)
# [571, 12, 4, 5, 608, 11, 2867, 8]

# Convert inputs to PyTorch tensors
tokens_tensor = torch.tensor([indexed_tokens])
# tensor([[ 571,   12,    4,    5,  608,   11, 2867,    8]])

# Load pre-trained model
model = BertForMaskedLM.from_pretrained(sel_model())
model.eval()

# Predict
with torch.no_grad():
    outputs = model(tokens_tensor)
    predictions = outputs[0][0, masked_index].topk(5)  # 予測結果の上位5件を抽出

# Show results
for i, index_t in enumerate(predictions.indices):
    index = index_t.item()
    token = tokenizer.convert_ids_to_tokens([index])[0]
    print(i, token)


```



Maybe @polm or @singletongue have an idea!<|||||>I didn't implement BertTokenizer so I'm a little out of my depth here, but the code below in a clean environment worked fine for me with no weird hashes.

```
from transformers import BertJapaneseTokenizer

name = "cl-tohoku/bert-base-japanese-whole-word-masking"
name = "cl-tohoku/bert-base-japanese"
tokenizer = BertJapaneseTokenizer.from_pretrained(name)

text = "テレビでサッカーの試合を見る。"

out = tokenizer.tokenize(text)
print(out)
```

I will note it is especially weird that the last word in your list (`。`) doesn't have the hashes.<|||||>Thank you for you reply.
Here is the result.
['テレビ', '##で', '##サッカー', '##の', '##試', '##合', '##を', '##見', '##る', '。']
<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>@leoxu1007
Could it be possible that you set `word_tokenizer_type` to `basic` ?
I reproduced the same result by this configuration.
I mean I got ['テレビ', '##で', '##サッカー', '##の', '##試', '##合', '##を', '##見', '##る', '。'].

Now, `BertJapaneseTokenizer` pretrained tokenizer's default configuration is  `word_tokenizer_type='mecab'`.
So we don't usually get this unexpected result.
I tried the example with `mecab` I got ['テレビ', 'で', 'サッカー', 'の', '試合', 'を', '見る', '。'].
