I found that if I use tokenizer for BERT,and decode some sentence with [MASK] before ‘.’,it would gain an additional idx 13
like this:
tokenizer = AlbertTokenizer.from_pretrained('albert-base-v2') inputs = tokenizer("The capital of France is [MASK].", return_tensors="pt")
the ‘input_ids’ would be:
tensor([[   2,   14, 1057,   16,  714,   25,    4,   13,    9,    3]])
4 refers to [MASK], 9 refers to ‘.’, but 13 refers to None
this would be happen when [MASK] is before ‘.’
is there something wrong?