Spaces:
				
			
			
	
			
			
					
		Running
		
	
	
	
			
			
	
	
	
	
		
		
					
		Running
		
	
		burtenshaw
		
	commited on
		
		
					Commit 
							
							·
						
						b5eec3d
	
1
								Parent(s):
							
							8b0cd75
								
use lighteval for evaluation
Browse files
    	
        docs.md
    CHANGED
    
    | @@ -36,16 +36,14 @@ Your trained model will be available at `your-username/your-model-name`. For det | |
| 36 | 
             
            Now, we will need to evaluate the model. We will use `hf jobs` to evaluate the model as well and combine it with `openbench`. We will push the evaluation results to a dataset on the hub. 
         | 
| 37 |  | 
| 38 | 
             
            ```sh
         | 
| 39 | 
            -
             | 
| 40 | 
            -
             | 
| 41 | 
            -
             | 
| 42 | 
            -
             | 
| 43 | 
            -
             | 
| 44 | 
            -
            bench eval mmlu --model vllm/<your-username>/your-model-name \ # define model repo
         | 
| 45 | 
            -
            --hub-repo <your-username>/<your-model-name>-chapter-1 # define output dataset
         | 
| 46 | 
             
            ```
         | 
| 47 |  | 
| 48 | 
            -
            This command will evaluate the model using ` | 
| 49 |  | 
| 50 | 
             
            <Tip>
         | 
| 51 |  | 
|  | |
| 36 | 
             
            Now, we will need to evaluate the model. We will use `hf jobs` to evaluate the model as well and combine it with `openbench`. We will push the evaluation results to a dataset on the hub. 
         | 
| 37 |  | 
| 38 | 
             
            ```sh
         | 
| 39 | 
            +
            hf jobs uv run \ # run a hf jobs job with uv
         | 
| 40 | 
            +
            --flavor a10g-large \ # select the machine size
         | 
| 41 | 
            +
            --with "lighteval[vllm]" \ # install lighteval with vllm dependencies
         | 
| 42 | 
            +
            s HF_TOKEN \ # share the huggingface write token
         | 
| 43 | 
            +
            lighteval vllm "model_name=<your-username>/<your-model-name>" "lighteval|gsm8k|0|0" --push-to-hub --results-org <your-username>
         | 
|  | |
|  | |
| 44 | 
             
            ```
         | 
| 45 |  | 
| 46 | 
            +
            This command will evaluate the model using `lighteval` and `vllm` and save the results to the Hugging Face Hub in the dataset repo that you defined.
         | 
| 47 |  | 
| 48 | 
             
            <Tip>
         | 
| 49 |  | 
