How to predict masked whole word which was tokenized as sub-words for bert-base-multilingual-cased
## ❓ Questions & Help
Hello,
I have started working with pytorch-transformers and want to use it to predict masked words in polish. I use ' bert-base-multilingual-cased' pre-trained model and want to predict masked words which very often are tokenized into sub-word.
My question is how can I predict the whole word?
When I predict each token separately the results are poor. especially when I try to concatenate those predicted tokens

Here is sample code showing the problem

```python
import torch
from pytorch_transformers import BertTokenizer, BertModel, BertForMaskedLM
import logging
logging.basicConfig(level=logging.INFO)

USE_GPU = 1
# Device configuration
device = torch.device('cuda' if (torch.cuda.is_available() and USE_GPU) else 'cpu')

# Load pre-trained model tokenizer (vocabulary)
pretrained_model = 'bert-base-multilingual-cased'
tokenizer = BertTokenizer.from_pretrained(pretrained_model)
text = "[CLS] Who was Jim Henson ? [SEP] Jim Henson was a puppeteer [SEP]"
tokenized_text = tokenizer.tokenize(text)

# Mask a token that we will try to predict back with `BertForMaskedLM`
mask1 = 13
mask2 = 14
mask3 = 15
tokenized_text[mask1] = '[MASK]'
tokenized_text[mask2] = '[MASK]'
tokenized_text[mask3] = '[MASK]'
assert tokenized_text == ['[CLS]', 'Who', 'was', 'Jim', 'Hen', '##son', '?', '[SEP]', 'Jim', 'Hen', '##son', 'was', 'a', '[MASK]', '[MASK]', '[MASK]', '[SEP]']

# Convert token to vocabulary indices
indexed_tokens = tokenizer.convert_tokens_to_ids(tokenized_text)
# Define sentence A and B indices associated to 1st and 2nd sentences (see paper)
segments_ids = [0, 0, 0, 0, 0, 0, 0,0, 1, 1, 1, 1, 1, 1, 1,1,1]

# Convert inputs to PyTorch tensors
tokens_tensor = torch.tensor([indexed_tokens])
segments_tensors = torch.tensor([segments_ids])


# Load pre-trained model (weights)
model = BertForMaskedLM.from_pretrained(pretrained_model)
model.eval()

# If you have a GPU, put everything on cuda
tokens_tensor = tokens_tensor.to(device)
segments_tensors = segments_tensors.to(device)
model.to(device)

# Predict all tokens
with torch.no_grad():
    outputs = model(tokens_tensor, token_type_ids=segments_tensors)
    predictions = outputs[0]

# get predicted tokens

#prediction for mask1
predicted_index = torch.argmax(predictions[0, mask1]).item()
predicted_token = tokenizer.convert_ids_to_tokens([predicted_index])[0]
print(predicted_token) #returns "baseball"


#prediction for mask2
predicted_index = torch.argmax(predictions[0, mask2]).item()
predicted_token = tokenizer.convert_ids_to_tokens([predicted_index])[0]
print(predicted_token) #returns "actor"


#prediction for mask3
predicted_index = torch.argmax(predictions[0, mask3]).item()
predicted_token = tokenizer.convert_ids_to_tokens([predicted_index])[0]
print(predicted_token) # returns "."
```



Hi I don't have any good solution for your use-case, unfortunately.

There are two "Whole-Word_masking" models for Bert (see the [list here](https://huggingface.co/pytorch-transformers/pretrained_models.html)) that would be better at guessing full words but they are only in English unfortunately.

SpanBert (whose open-sourcing we are still waiting) may also be better but I think they also only trained an English model...<|||||>@thomwolf thanks for the reply. 
can you specify what exactly the problem lies in? I know the model is not capable of properly tokenizing polish. 

Assuming I have pre-trained model for Polish, or just working with English text how can I predict a sequence of masked two or three tokens side by side<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>if i want to fine-tune with my dataset,what should i do?<|||||>> if i want to fine-tune with my dataset,what should i do?

Hi tom1125, to fine-tune you can run this script:

export TRAIN_FILE=/path/to/dataset/wiki.train.raw
export TEST_FILE=/path/to/dataset/wiki.test.raw

python run_language_modeling.py \
    --output_dir=output \
    --model_type=roberta \
    --model_name_or_path=roberta-base \
    --do_train \
    --train_data_file=$TRAIN_FILE \
    --do_eval \
    --eval_data_file=$TEST_FILE \
    --mlm

Explained here: https://huggingface.co/transformers/examples.html#roberta-bert-and-masked-language-modeling