Why don't you include the tokenizer ?
#2
by
Jeronymous
- opened
Why did you not include tokenizer.json and tokenizer_config.json files in this repo, so that the model is complete ?
(or maybe I missed something)
The tokenizer would be unchanged meaning that you should still be able to use tokenizer = AutoTokenizer.from_pretrained("OpenLLM-France/Lucie-7B") and get it works.
@nifleisch
might be able to add the original tokenizer as well ;)
Good point @Jeronymous ! I just added the original "OpenLLM-France/Lucie-7B"-Tokenizer to the repository. 鈽猴笍
I see it was added. That should unlock the use with some tools where the user can only configure one HF repo.
Thanks !
Jeronymous
changed discussion status to
closed