FillMaskPipeline return special tokens i.e. <mask> as prediction
# ❓ FillMaskPipeline return special tokens i.e. \<mask\> as prediction


## Details
Im training a new language model from scratch using ByteLevelBPETokenizer for tokenizer and RobertaForMaskedLM for self-supervised language model. The config is as follow:

Tokenizer:

```
tokenizer = ByteLevelBPETokenizer()

tokenizer.train(files=paths, vocab_size=100000, min_frequency=2, special_tokens=[
    "<s>",
    "<pad>",
    "</s>",
    "<unk>",
    "<mask>",
])
```

After the tokenizer was trained, it was loaded using

`tokenizer = RobertaTokenizerFast.from_pretrained("./BERTmese/Tokenizer", max_length=512)`

And the RobertaForMaskedLM  is configed as follow:

```
config = RobertaConfig(
    vocab_size=100000,
    max_position_embeddings=514,
    num_attention_heads=12,
    num_hidden_layers=6,
)
model = RobertaForMaskedLM(config=config)
```

Then I train the RobertaForMaskedLM  model using custom dataset and Adam optimizer as follow: 
```
class CustomTextDataset(Dataset):
    def __init__(self, tokenizer, dataset, max_length):
        self.tokenizer = tokenizer
        self.dataset = dataset
        self.max_length = max_length

    def __len__(self):
        return len(self.dataset)

    def __getitem__(self, i) -> torch.Tensor:
        encoding = self.tokenizer(self.dataset[i]['text'],
                                add_special_tokens=True, 
                                max_length=self.max_length, 
                                padding='max_length',
                                truncation=True)
        return torch.tensor(encoding["input_ids"], dtype=torch.long)
```
```
for step in data_loader:
    input_ids = next(iter(train_loader))
    input_ids = input_ids.to(device)

    outputs = model(input_ids, labels=input_ids)
    loss, _ = outputs[:2]
    loss.backward()
    ...
```
For every 100 steps, I test the trained model using this code snippet:
```
model.eval()
# test on a sample text

fill_mask = FillMaskPipeline(
    model=model,
    tokenizer=tokenizer,
    topk=1
)

mask = fill_mask("ယခုလတွင်ပျားရည်နှင့်ပျားဖယောင်းများကိုစုဆောင်း<mask>သည်ဟုခန့်မှန်းနိုင်သည်။")[0] 

print(mask['token'], decoder.decode([mask['token_str']]), mask['score'])

# set model back to train mode

model.train()
```
In the first hundred steps, the predict token is fined (it predict some Burmese word). However, after that, the model start to predict the '\<mask\>' token as the result. Im not sure if predicting \<mask\> token for masked language is normal?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
