Saving Pretrained Tokenizer
I created a custom `tokenizers.Tokenizer` and saved it as follows

```
tokenizer.model.save("./tokenizer")
tokenizer.save("./tokenizer.json") 
```

This produces 3 files, merges.txt, vocab.json and tokenizer.json

Then I created a `transformers.RobertaTokenizerFast` and saved it to the same folder

```
tokenizer = RobertaTokenizerFast.from_pretrained("./tokenizer")  
tokenizer.save_pretrained("./tokenizer") 
```

This adds `special_tokens_map.json` and `tokenizer_config.json`

I then saved it to another folder to simulate what happens when I train my model

```
tokenizer.save_pretrained("./model") 
tokenizer = RobertaTokenizerFast.from_pretrained("./model")  
```

What I noticed was `tokenizer_config.json` contains a key `name_or_path` which still points to `./tokenizer`, so what seems to be happening is `RobertaTokenizerFast.from_pretrained("./model") ` is loading files from two places (./model and ./tokenizer)

Not sure if this is expected, it seems that the tokenizer_config.json should be updated in save_pretrained, and tokenizer.json should be saved with it?

Or perhaps this is just an issue because I'm training my tokenizer in a subdirectory of the model folder?

I realised that

`tokenizer.model.save("./tokenizer")`

Is unnecessary. I've started saving only the `tokenizer.json` since this contains not only the merges and vocab but also the pipeline.

And I noticed that `tokenizer.save_pretrained()` has a parameter  `legacy_format` which defaults to True. When I set it to false it properly round trips (i.e. saves out the unified tokenizer.json rather than the model files (merges.txt and vocab.json)

The only issue now is if I create a trainer

```
trainer = Trainer(
    model=model,
    tokenizer=tokenizer
    args=training_args,
    train_dataset=tokenized_datasets["train"],
    eval_dataset=tokenized_datasets["validation"] ,
    data_collator=data_collator,
)

logger.info("*** Train ***")
trainer.train(model_path=model_path)
trainer.save_model()
```

The last line doesn't accept kwargs so it saves the tokenizer in the legacy format.

A workaround is to do it manually ie.

```
trainer = Trainer(
    model=model,
    #tokenizer=tokenizer
    args=training_args,
    train_dataset=tokenized_datasets["train"],
    eval_dataset=tokenized_datasets["validation"] ,
    data_collator=data_collator,
)

logger.info("*** Train ***")
trainer.train(model_path=model_path)
trainer.save_model()
tokenizer.save_pretrained(output_dir, legacy_format=False)
```

Only issue is I don't see a workaround to ensure the checkpoints are created using `legacy_format=False`



<|||||>This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.