hi @ayalaall

I was wondering if there is a function that generates “[MASK]” on 15% of the tokens

DataCollatorForLanguageModeling does the masking as described, you can find it here

  
      github.com
  
  
    huggingface/transformers/blob/master/src/transformers/data/data_collator.py#L285 11

        return self.tokenizer.pad(            features,            padding=self.padding,            max_length=self.max_length,            pad_to_multiple_of=self.pad_to_multiple_of,            return_tensors="pt",        )@dataclassclass DataCollatorForLanguageModeling:    """    Data collator used for language modeling. Inputs are dynamically padded to the maximum length of a batch if they    are not all of the same length.    Args:        tokenizer (:class:`~transformers.PreTrainedTokenizer` or :class:`~transformers.PreTrainedTokenizerFast`):            The tokenizer used for encoding the data.        mlm (:obj:`bool`, `optional`, defaults to :obj:`True`):            Whether or not to use masked language modeling. If set to :obj:`False`, the labels are the same as the            inputs with the padding tokens ignored (by setting them to -100). Otherwise, the labels are -100 for