How to enable sampling when using unigram tokenizers?
# ❓ Questions & Help

## Details
It is mentioned on [Tokenizer summary][1] that when using unigram tokenizers, "you could sample one of the tokenization according to their probabilities". It however did not detail how to do so.

Looking at the source-code, all of `AlbertTokenizer`, `T5Tokenizer` and `XLNetTokenizer` have an argument "sample" in their `_tokenize()` method, which in turns call the encoding method from sentencepiece with sampling enabled. 
```
    def _tokenize(self, text, sample=False):
        """ Tokenize a string. """
        text = self.preprocess_text(text)

        if not sample:
            pieces = self.sp_model.EncodeAsPieces(text)
        else:
            pieces = self.sp_model.SampleEncodeAsPieces(text, 64, 0.1)
        ...
```
But when I check the usages of _tokenize, it seems to be only called without a "sample" argument, in `PreTrainedTokenizer`. 
```
        def split_on_tokens(tok_list, text):
            if not text.strip():
                return []
            if not tok_list:
                return self._tokenize(text)
```

I tried calling the `encode()` method of the tokenizer class with `sample=True`, but the keyword was not recognized. 
```
albert_tokenizer = AlbertTokenizer('m.model')
albert_tokenizer.encode(msg, sample=True)

>> Keyword arguments {'sample': True} not recognized.
```

How should I enable sampling when using unigram tokenizers?


  [1]: https://huggingface.co/transformers/tokenizer_summary.html#unigram
**A link to original question on the forum/Stack Overflow**:
[https://stackoverflow.com/questions/63436152/how-to-enable-sampling-when-using-unigram-tokenizers-in-huggingface](url)
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
