File size: 333 Bytes
3cc6215 aba9319 2216092 ee3fc8f 2216092 |
1 2 3 4 5 6 7 8 |
---
license: llama2
---
This is a copy of the llama2 tokenizer for use as a fallback tokenizer for KoboldAI, optimized with defaults for text completion.
We aim to keep this copy functional / identical to the upstream llama2 tokenizer with minor differences in its defaults.
In case of differences a more functional copy is chosen. |