Update tokenization_arcade100k.py
#1
by
mmnga
- opened
When converting to gguf in llama.cpp, special_tokens is required because it is converted using the qwen like approach, so you need to add this.
When converting to gguf in llama.cpp, special_tokens is required because it is converted using the qwen like approach, so you need to add this.