How can I generate sentencepiece file or vocabulary from tokenizers?
After I make custom tokenizer using Tokenizers library, I could load it into XLNetTokenizerFast using
```
tokenizer = Tokenizer.from_file("unigram.json")
tok = XLNetTokenizerFast(tokenizer_object=tokenizer)
```

After I called
```
tok .save_vocabulary("ss")
```

it throws an error since I didnt load XLNetTokenizerFast using spm file. I believe save_vocabulary is looking for vocab_file parameter.

Is there any way to save_vocab after loading it from XLNetTokenizer?


Could you try `save_pretrained` instead?<|||||>It throws
```
Traceback (most recent call last):
  File "~/pretrain.py", line 26, in <module>
    tokenizer.save_pretrained('./test_Spm')
  File ~/anaconda3/envs/deltalake/lib/python3.9/site-packages/transformers/tokenization_utils_base.py", line 1958, in save_pretrained
    save_files = self._save_pretrained(
  File "~/anaconda3/envs/deltalake/lib/python3.9/site-packages/transformers/tokenization_utils_fast.py", line 555, in _save_pretrained
    vocab_files = self.save_vocabulary(save_directory, filename_prefix=filename_prefix)
  File "~/anaconda3/envs/deltalake/lib/python3.9/site-packages/transformers/models/xlnet/tokenization_xlnet_fast.py", line 232, in save_vocabulary
    if os.path.abspath(self.vocab_file) != os.path.abspath(out_vocab_file):
  File "~/anaconda3/envs/deltalake/lib/python3.9/posixpath.py", line 374, in abspath
    path = os.fspath(path)
TypeError: expected str, bytes or os.PathLike object, not NoneType
```

It looked for vocab file (spm), but since I initiated it with tokenizer_object, there is no vocab_file<|||||>Could you share your `unigram.json` file or mention how you obtained it so that I can reproduce the issue? Thank you!<|||||>Here it is

[unigram.json.zip](https://github.com/huggingface/transformers/files/6839324/unigram.json.zip)


Edit:

```
from transformers import AutoTokenizer, XLNetTokenizerFast

tokenizer = AutoTokenizer.from_pretrained('xlnet-base-cased')
print(tokenizer.save_pretrained("./dump"))
tokenizer = XLNetTokenizerFast(tokenizer_file='./dump/tokenizer.json')
print(tokenizer.save_pretrained("./dump"))
```
Above Throws an error

```

from transformers import AutoTokenizer, XLNetTokenizerFast, BertTokenizerFast

tokenizer = AutoTokenizer.from_pretrained('bert-base-cased')
print(tokenizer.save_pretrained("./dump"))
tokenizer = BertTokenizerFast(tokenizer_file='./dump/tokenizer.json')
print(tokenizer.save_pretrained("./dump"))

```
Above does not throw an error<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>Sorry, is there any progress on this?<|||||>@SaulLu can you give this one a look?<|||||>Thank you very much for reporting this problem @darwinharianto . 

I have the impression that it linked to the problem reported in [this issue](https://github.com/huggingface/transformers/issues/12762) and that I had started to work on in this [PR](https://github.com/huggingface/transformers/pull/12806). As the cleanest looking fix requires quite a bit of work, I had put it on hold. I'll try to work on it again at the beginning of the week.<|||||>@darwinharianto, to answer the question in the issue title, at the moment it is not possible to transform a fast tokenizer initialized only with a `tokenizer_object` into an spm file compatible with the `SentencePiece` library. The conversion is only supported in the opposite direction.

Why did you want the vocabulary? Because the vocabulary can be found in the `tokenizer.json` file or by doing: 
```
tok.get_vocab()
```
<|||||>Since I tried to make my own XLNet tokenizer, i wanted to check if the saved format for vocabulary is the same as the published models.
I thought the fastest way would be comparing saved vocab file from huggingface model hub with mine.
Just for sanity check.<|||||>Duly noted! If it's to do a sanity check, would it be ok to compare the files of the fast version of the tokenizers (in particular the `tokenizer.json` file)? 

To retrieve these files for the new tokenizer `tok` you made, the following command should work: 
```
tok.save_pretrained("./dump", legacy_format=False)
```

For information, the vocabulary will be visible in the `tokenizer.json` file.<|||||>Thanks! it works.

One more question, I can see this
```
                "type": "Precompiled",
                "precompiled_charsmap": 
```
under normalizers, when I tried to save a pretrained tokenizer.

My custom tokenizer doesn't have this attribute. Is this normal?<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>I'm sorry, I realize that I never answered your last question. 

This type of `Precompiled` normalizer is only used to recover the normalization operation which would be contained in a file generated by the sentencepiece library. If you have ever created your tokenizer with the tokenizers library it is perfectly normal that you do not have this type of normalization. Nevertheless, if you want to have an equivalent normalization for your tokenizer, it is generally possible to build it with the tokenizers library but it requires to know exactly which normalizations you want to apply. :slightly_smiling_face: <|||||>> @darwinharianto, to answer the question in the issue title, at the moment it is not possible to transform a fast tokenizer initialized only with a `tokenizer_object` into an spm file compatible with the `SentencePiece` library. The conversion is only supported in the opposite direction.
> 
> Why did you want the vocabulary? Because the vocabulary can be found in the `tokenizer.json` file or by doing:
> 
> ```
> tok.get_vocab()
> ```

Hi @SaulLu,

Sorry to open this old thread, I noticed that you mentioned transferring from spm tokenizer to a huggingface one is easy, but I could not find any function which does that for me. I would be grateful if you could share any piece of code to help me with that.
(Just to give you a quick background, I trained a SPM tokenizer and would like to use it in huggingface, but I have .vocab and .model for it, and huggingface expect a .json file) 

Thank you,
Soheila