Why is masking ID optional?
I see many models have masking ID as an optional input. To my understanding, the models need to know which tokens are padded to avoid doing attention with those tokens. How can the models figure this out if we do not provide masking ID? Do these models consider tokens with id 0 to be the masked tokens?
In my past experience, I noticed that masked tokens would get very low attention scores(that's what self attention's supposed to do) and would not contribute to the model's accuracy. I wonder how would it be in a transformer architecture.<|||||>Not all models requires this feature, mainly Roberta so it includes XLM-R, Camembert, etc.<|||||>I think your question is why are the `attention_mask` model inputs optional?
This is because you need attention masks only when padding. If you're not batching your inputs, you do not need to pad and therefore do not need to specify an attention mask.<|||||>@LysandreJik I wonder if there is padding and the user does not provide `attention_mask`, does the model think there is no padding or there is a mechanism to figure it out uder the hood (maybe treat token ID 0 as padded token). I am just trying to fully understand the model's behavior. Thanks<|||||>No, if you don't provide any `attention_mask` then the model will attend to every token the same way. You absolutely should use the `attention_mask` as soon as you're padding, or the model will output flawed results!<|||||>Got it, thanks a lot.