This behavior is also present in other models, e.g. in xlm-roberta-base where a token with id 6 is injected. The token is empty string '', which can be quite confusing. This bug should be addressed or at least it should be indicated that placing mask tokens is not recommended.