Update README.md (#966)
Browse files
README.md
CHANGED
|
@@ -981,7 +981,7 @@ wandb_log_model:
|
|
| 981 |
|
| 982 |
##### Special Tokens
|
| 983 |
|
| 984 |
-
It is important to have special tokens like delimiters, end-of-sequence, beginning-of-sequence in your tokenizer's
|
| 985 |
|
| 986 |
```yml
|
| 987 |
special_tokens:
|
|
|
|
| 981 |
|
| 982 |
##### Special Tokens
|
| 983 |
|
| 984 |
+
It is important to have special tokens like delimiters, end-of-sequence, beginning-of-sequence in your tokenizer's vocabulary. This will help you avoid tokenization issues and help your model train better. You can do this in axolotl like this:
|
| 985 |
|
| 986 |
```yml
|
| 987 |
special_tokens:
|