chore(readme): update instruction to set config to load from cache (#1030)
Browse files
README.md
CHANGED
|
@@ -935,8 +935,9 @@ accelerate launch -m axolotl.cli.train your_config.yml
|
|
| 935 |
You can optionally pre-tokenize dataset with the following before finetuning.
|
| 936 |
This is recommended for large datasets.
|
| 937 |
|
| 938 |
-
- Set `
|
| 939 |
-
-
|
|
|
|
| 940 |
|
| 941 |
```bash
|
| 942 |
python -m axolotl.cli.preprocess your_config.yml
|
|
|
|
| 935 |
You can optionally pre-tokenize dataset with the following before finetuning.
|
| 936 |
This is recommended for large datasets.
|
| 937 |
|
| 938 |
+
- Set `dataset_prepared_path:` to a local folder for saving and loading pre-tokenized dataset.
|
| 939 |
+
- (Optional): Set `push_dataset_to_hub: hf_user/repo` to push it to Huggingface.
|
| 940 |
+
- (Optional): Use `--debug` to see preprocessed examples.
|
| 941 |
|
| 942 |
```bash
|
| 943 |
python -m axolotl.cli.preprocess your_config.yml
|