Training RoBerta using transformers on masked language task giving weird results
# ❓ Questions & Help

## Details
I trained a RoBERTa model following this colab - https://colab.research.google.com/github/huggingface/blog/blob/master/notebooks/01_how_to_train.ipynb#scrollTo=XaFAsB_fnU3K

Here is how my data looked:
```
Merkel bemoans lack of rain as Germany fears for its forests .\n
Germany’s forests, covering a third of its territory and as much a part of its cultural landscape as its physical one, are in danger.\n
An aerial view shows a forest near Gummersbach, Germany, April 24, 2020, following an unusually warm, dry winter after a summer of record temperatures leaving forests dried out.\n
Picture taken with a drone.\n
The last two exceptionally hot and dry summers have weakened millions of trees, undermining their defences against the bark beetle, which can be fatal to ancient woodlands.\n
And after an exceptionally dry April, with summer still two months away, a forest fire has already had to be put out near the town of Gummersbach in western Germany this week.\n
“We’re already noticing these days that it’s not raining enough in many areas. 
```
After training the model I used pipeline from the transforms library for the fill_mask task
```
from transformers import pipeline

fill_mask = pipeline(
    "fill-mask",
    model="./output",
    tokenizer="./output"
fill_mask("Merkel bemoans lack of rain as <mask> fears for its forests")
)
```
These are the results:
```
[{'sequence': '<s> Merkel bemoans lack of rain as. fears for its forests</s>',
  'score': 0.040456026792526245,
  'token': 18},
 {'sequence': '<s> Merkel bemoans lack of rain as, fears for its forests</s>',
  'score': 0.03502459451556206,
  'token': 16},
 {'sequence': '<s> Merkel bemoans lack of rain as the fears for its forests</s>',
  'score': 0.03497963398694992,
  'token': 269},
 {'sequence': '<s> Merkel bemoans lack of rain as\n fears for its forests</s>',
  'score': 0.03180328756570816,
  'token': 203},
 {'sequence': '<s> Merkel bemoans lack of rain as to fears for its forests</s>',
  'score': 0.020796578377485275,
  'token': 288}]
```
As you can see there is no meaningful word(s) returned only punctuations and one other word (to) which doesn't make sense. What am i doing wrong here? Do I have to remove all punctuations?
<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->

**A link to original question on Stack Overflow**:
https://stackoverflow.com/questions/62276011/training-roberta-using-transformers-on-masked-language-task-giving-weird-results

Still facing this problem. Has anyone else encountered something similar? <|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>I also face this problem. How to solve it?