tokenizer.json + tokenizer_config.json missing?
#25
by
am17an
- opened
I'm trying to create a GGUF for this model but I don't see these files (tokenizer.json + tokenizer_config.json or related tokenizer files). Will they be added?
Hey
@am17an
, can you please also open an issue on Transformers?
The Transformers integration was done using mistral-common
, but I actually have commits converting the tokenizer and using Transformers code only, so I'm happy to add this codepath if needed! π€