burtenshaw commited on
Commit
b5eec3d
·
1 Parent(s): 8b0cd75

use lighteval for evaluation

Browse files
Files changed (1) hide show
  1. docs.md +6 -8
docs.md CHANGED
@@ -36,16 +36,14 @@ Your trained model will be available at `your-username/your-model-name`. For det
36
  Now, we will need to evaluate the model. We will use `hf jobs` to evaluate the model as well and combine it with `openbench`. We will push the evaluation results to a dataset on the hub.
37
 
38
  ```sh
39
- export HF_TOKEN=<your-huggingface-token>
40
-
41
- hf jobs uv run \
42
- --flavor a100-large \ # define machine size
43
- --with openbench --with vllm --with 'transformers<4.54.0' \ # install dependencies
44
- bench eval mmlu --model vllm/<your-username>/your-model-name \ # define model repo
45
- --hub-repo <your-username>/<your-model-name>-chapter-1 # define output dataset
46
  ```
47
 
48
- This command will evaluate the model using `openbench` and `vllm` and save the results to the Hugging Face Hub in the dataset repo that you defined.
49
 
50
  <Tip>
51
 
 
36
  Now, we will need to evaluate the model. We will use `hf jobs` to evaluate the model as well and combine it with `openbench`. We will push the evaluation results to a dataset on the hub.
37
 
38
  ```sh
39
+ hf jobs uv run \ # run a hf jobs job with uv
40
+ --flavor a10g-large \ # select the machine size
41
+ --with "lighteval[vllm]" \ # install lighteval with vllm dependencies
42
+ s HF_TOKEN \ # share the huggingface write token
43
+ lighteval vllm "model_name=<your-username>/<your-model-name>" "lighteval|gsm8k|0|0" --push-to-hub --results-org <your-username>
 
 
44
  ```
45
 
46
+ This command will evaluate the model using `lighteval` and `vllm` and save the results to the Hugging Face Hub in the dataset repo that you defined.
47
 
48
  <Tip>
49