Spaces:
Running
Running
burtenshaw
commited on
Commit
·
b5eec3d
1
Parent(s):
8b0cd75
use lighteval for evaluation
Browse files
docs.md
CHANGED
@@ -36,16 +36,14 @@ Your trained model will be available at `your-username/your-model-name`. For det
|
|
36 |
Now, we will need to evaluate the model. We will use `hf jobs` to evaluate the model as well and combine it with `openbench`. We will push the evaluation results to a dataset on the hub.
|
37 |
|
38 |
```sh
|
39 |
-
|
40 |
-
|
41 |
-
|
42 |
-
|
43 |
-
|
44 |
-
bench eval mmlu --model vllm/<your-username>/your-model-name \ # define model repo
|
45 |
-
--hub-repo <your-username>/<your-model-name>-chapter-1 # define output dataset
|
46 |
```
|
47 |
|
48 |
-
This command will evaluate the model using `
|
49 |
|
50 |
<Tip>
|
51 |
|
|
|
36 |
Now, we will need to evaluate the model. We will use `hf jobs` to evaluate the model as well and combine it with `openbench`. We will push the evaluation results to a dataset on the hub.
|
37 |
|
38 |
```sh
|
39 |
+
hf jobs uv run \ # run a hf jobs job with uv
|
40 |
+
--flavor a10g-large \ # select the machine size
|
41 |
+
--with "lighteval[vllm]" \ # install lighteval with vllm dependencies
|
42 |
+
s HF_TOKEN \ # share the huggingface write token
|
43 |
+
lighteval vllm "model_name=<your-username>/<your-model-name>" "lighteval|gsm8k|0|0" --push-to-hub --results-org <your-username>
|
|
|
|
|
44 |
```
|
45 |
|
46 |
+
This command will evaluate the model using `lighteval` and `vllm` and save the results to the Hugging Face Hub in the dataset repo that you defined.
|
47 |
|
48 |
<Tip>
|
49 |
|