Hi @Anthony,
Thank you for your prompt reply!
The approach you proposed, if I’m not mistaken, would be helpful in permanently masking words when reading the dataset.
Instead, I am interested in dynamically masking words at batch time.
To be more clear, I would like to implement (efficiently) a mask_tokens() 15 function (as the one defined in the DataCollatorForLanguageModeling class) which masks only IDs corresponding to words provided in a specific list.
These IDs would not be masked during each batch but following some stochastic strategy, unlike the approach which would mask them when the dataset is read at the beginning.
I am wondering whether there is an efficient and neat way to do so, exploiting the Huggingface functions and taking also into account that some words get split in multiple IDs depending on the tokenizer at hand.