Model Evaluation and Leaderboard

1) Model Evaluation
Before integrating a model into the leaderboard, it must first be evaluated using the lm-eval-harness library in both zero-shot and 5-shot configurations.

This can be done with the following command:

lm_eval --model hf --model_args pretrained=google/gemma-3-12b-it \
  --tasks evalita-mp --device cuda:0 --batch_size 1 --trust_remote_code \
  --output_path model_output --num_fewshot 5 --

The output generated by the library will include the model's accuracy scores on the benchmark tasks.
This output is written to the standard output and should be saved in a txt file (e.g., slurm-8368.out), which needs to be placed in the
 evalita_llm_models_output directory for further processing.

2) Extracting Model Metadata
To display model details on the leaderboard (e.g., organization/group, model name, and parameter count), metadata must be retrieved from Hugging Face.

This can be done by running:

python get_model_info.py

This script processes the evaluation files from Step 1 and saves each model's metadata in a JSON file within the evalita_llm_requests directory.

3) Generating Leaderboard Submission File
The leaderboard requires a structured file containing each model’s metadata along with its benchmark accuracy scores.

To generate this file, run:

python preprocess_model_output.

This script combines the accuracy results from Step 1 with the metadata from Step 2 and outputs a JSON file in the evalita_llm_results directory.

4) Updating the Hugging Face Repository
The evalita_llm_results repository on HuggingFace must be updated with the newly generated files from Step 3.

5) Running the Leaderboard Application
Finally, execute the leaderboard application by running:

python app.py