BOS off by default
Browse files- README.md +3 -3
- tokenizer_config.json +1 -1
README.md
CHANGED
|
@@ -2,7 +2,7 @@
|
|
| 2 |
license: llama2
|
| 3 |
---
|
| 4 |
|
| 5 |
-
This is a copy of the llama2 tokenizer for use as a fallback tokenizer for KoboldAI.
|
| 6 |
|
| 7 |
-
We aim to keep this copy functional / identical to the upstream llama2 tokenizer.
|
| 8 |
-
In case of differences a functional copy is chosen.
|
|
|
|
| 2 |
license: llama2
|
| 3 |
---
|
| 4 |
|
| 5 |
+
This is a copy of the llama2 tokenizer for use as a fallback tokenizer for KoboldAI, optimized with defaults for text completion.
|
| 6 |
|
| 7 |
+
We aim to keep this copy functional / identical to the upstream llama2 tokenizer with minor differences in its defaults.
|
| 8 |
+
In case of differences a more functional copy is chosen.
|
tokenizer_config.json
CHANGED
|
@@ -1,5 +1,5 @@
|
|
| 1 |
{
|
| 2 |
-
"add_bos_token":
|
| 3 |
"add_eos_token": false,
|
| 4 |
"bos_token": {
|
| 5 |
"__type": "AddedToken",
|
|
|
|
| 1 |
{
|
| 2 |
+
"add_bos_token": false,
|
| 3 |
"add_eos_token": false,
|
| 4 |
"bos_token": {
|
| 5 |
"__type": "AddedToken",
|