Whole word mask in run_mlm_wwm.py
I find that `run_mlm_wwm.py` uses the whole word mask class `DataCollatorForWholeWordMask`.
But in this class `_whole_word_mask` function, we recognize if a token is the beginning of a word by this way:

```
cand_indexes = []
for (i, token) in enumerate(input_tokens):
    if token == "[CLS]" or token == "[SEP]":
        continue

    if len(cand_indexes) >= 1 and token.startswith("##"):
        cand_indexes[-1].append(i)
    else:
        cand_indexes.append([i])
```

I also notice that `run_mlm_wwm.py` is also used for Roberta per-training in the examples. However, the tokenizer for Roberta doesn't contain tokens like `[CLS]` and `[SEP]`. Also, the subwords do not start with `##`.

How can this code handle language models use Roberta-like tokenizer?

Thanks!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>I noticed this myself, `DataCollatorForWholeWordMask` seems to be very specific to the BERT tokenizer. It seems that it should be using the special tokens mask, and the word_ids() from the tokenizer rather than rely on [CLS],[SEP] tokens and subwords starting with ## (so it fails with metaspace for example).

Edit: It also calls `self._tensorize_batch` which as far as I can see isn't implemented, so I assume this class isn't maintained?<|||||>It's a bit old but I wanted to share my quick fix for Roberta-like tokenizers (I think this can be more general propose, but I just needed this for Herbert tokenizer):
```
def _whole_word_mask(self, input_tokens: List[str], max_predictions=512):
        """
        Get 0/1 labels for masked tokens with whole word mask proxy
        """
        if not isinstance(self.tokenizer, (BertTokenizer, BertTokenizerFast,
                                           RobertaTokenizer, RobertaTokenizerFast,
                                           XLMRobertaTokenizer, XLMRobertaTokenizerFast,
                                           HerbertTokenizer, HerbertTokenizerFast,
                                           XLMTokenizer)):
            warnings.warn(
                "DataCollatorForWholeWordMask is only suitable for BertTokenizer or RobertaTokenizer-like tokenizers. "
                "Please refer to the documentation for more information."
            )

        cand_indexes = []
        special_tokens = [val for key, val in self.tokenizer.special_tokens_map.items()
                          if key not in ['unk_token', 'mask_token']]
        is_bert_tokenizer = isinstance(self.tokenizer, (BertTokenizer, BertTokenizerFast))
        for (i, token) in enumerate(input_tokens):
            if token in special_tokens:
                continue

            if is_bert_tokenizer:
                if len(cand_indexes) >= 1 and token.startswith("##"):
                    cand_indexes[-1].append(i)
                else:
                    cand_indexes.append([i])
            else:  # Roberta-like tokenizers have </w> token at the end to indicate end of word
                # edge case for chinese (##) are added in DataCollatorForWholeWordMask
                if token.startswith("##"):
                    token = token[2:]
                    if token.endswith("</w>"):
                        token = token[:-4]
                if len(cand_indexes) == 0:
                    cand_indexes.append([i])
                else:
                    cand_indexes[-1].append(i)

                if token.endswith("</w>"):
                    cand_indexes.append([])

        if len(cand_indexes[-1]) == 0:
            cand_indexes = cand_indexes[:-1]

        random.shuffle(cand_indexes)
        num_to_predict = min(max_predictions, max(1, int(round(len(input_tokens) * self.mlm_probability))))
        masked_lms = []
        covered_indexes = set()
        for index_set in cand_indexes:
            if len(masked_lms) >= num_to_predict:
                break
            # If adding a whole-word mask would exceed the maximum number of
            # predictions, then just skip this candidate.
            if len(masked_lms) + len(index_set) > num_to_predict:
                continue
            is_any_index_covered = False
            for index in index_set:
                if index in covered_indexes:
                    is_any_index_covered = True
                    break
            if is_any_index_covered:
                continue
            for index in index_set:
                covered_indexes.add(index)
                masked_lms.append(index)

        if len(covered_indexes) != len(masked_lms):
            raise ValueError("Length of covered_indexes is not equal to length of masked_lms.")
        mask_labels = [1 if i in covered_indexes else 0 for i in range(len(input_tokens))]
        return mask_labels
```<|||||>If anybody else has this issue. I fixed it for RoBERTa by adding a few lines that deal with the way RoBERTa tokenizes. Note that it's not a general purpose solution for other LMs. The previous comment did not work for me. See here:

https://github.com/RikVN/transformers/blob/main/src/transformers/data/data_collator.py#L948