How to get a spiece.model from customize chinese vocab.txt in Albert xlnet ?
## ❓ Questions & Help
How to get a spiece.model from customize chinese vocab.txt in Albert xlnet ?
Have you taken a look at [sentencepiece](https://github.com/google/sentencepiece)?<|||||>请问这个问题您解决了吗<|||||>I have the same problem. Have you solved it?<|||||>> Have you taken a look at [sentencepiece](https://github.com/google/sentencepiece)?

I have taken a look at sentencepiece documents, but found nothing to build a spiece.model from customized chinese vocab.txt in ALBERT. Do you have any solution to solve this problem?<|||||>I think that the chinese version of albert uses wordpiece model instead of sentencepiece model.
[https://github.com/google-research/ALBERT/issues/58](url)

> For Chinese models, we use word piece model provided by Jacob as sentence piece get worse performance on reading comprehension tasks for Chinese.

https://github.com/google-research/ALBERT/blob/master/tokenization.py
```python
class FullTokenizer(object):
  """Runs end-to-end tokenziation."""

  def __init__(self, vocab_file, do_lower_case=True, spm_model_file=None):
    self.vocab = None
    self.sp_model = None
    if spm_model_file:
      self.sp_model = spm.SentencePieceProcessor()
      tf.logging.info("loading sentence piece model")
      self.sp_model.Load(spm_model_file)
      # Note(mingdachen): For the purpose of consisent API, we are
      # generating a vocabulary for the sentence piece tokenizer.
      self.vocab = {self.sp_model.IdToPiece(i): i for i
                    in range(self.sp_model.GetPieceSize())}
    else:
      self.vocab = load_vocab(vocab_file)
      self.basic_tokenizer = BasicTokenizer(do_lower_case=do_lower_case)
      self.wordpiece_tokenizer = WordpieceTokenizer(vocab=self.vocab)
    self.inv_vocab = {v: k for k, v in self.vocab.items()}
```
When the sentencepiece model is None, the full tokenizer is initialized with a basic tokenizer and a workpiece tokenizer.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
