metadata
license: llama2
This is a copy of the llama2 tokenizer for use as a fallback tokenizer for KoboldAI, optimized with defaults for text completion.
We aim to keep this copy functional / identical to the upstream llama2 tokenizer with minor differences in its defaults. In case of differences a more functional copy is chosen.