Update README_EVALUATION.md

README_EVALUATION.md

@@ -28,10 +28,11 @@ Math: see data/*
 For model generation on single seed, please use the following command:
 
 ```
-bash generate_livecodebench.sh ${model_path} ${seed} ${output_path}
-bash generate_aime.sh ${model_path} ${seed} aime24 ${output_path}
-bash generate_aime.sh ${model_path} ${seed} aime25 ${output_path}
+bash generate_livecodebench.sh ${model_path} ${seed} ${output_path} ${model_type}
+bash generate_aime.sh ${model_path} ${seed} aime24 ${output_path} ${model_type}
+bash generate_aime.sh ${model_path} ${seed} aime25 ${output_path} ${model_type}
 ```
+Please specify model_type as r1 for AceReason-Nemotron-1.0 models, and qwen for AceReason-Nemotron-1.1 models.
 
 Or you can use our configured seeds to reproduce our results on AIME 24/25 (avg@64) and LiveCodeBench v5/v6 (avg@8) as follows:
 
@@ -52,7 +53,7 @@ python evaluate_aime.py --modelfolder ${output_path} --test_data data/aime25.jso
 We also left our generations into cache.tar.gz as references.
 
 ```
-LiveCodeBench AceReason-Nemotron-7B (Avg@8)
+LiveCodeBench AceReason-Nemotron-1.0-7B (Avg@8)
 =================================================================
 Months          Corrects        Total           Accuracy
 2023-05         180             272             66.17647058823529
@@ -84,7 +85,7 @@ Months          Corrects        Total           Accuracy
 v5              1142            2232            51.16487455197132
 v6              621             1400            44.357142857142854
 
-LiveCodeBench AceReason-Nemotron-14B (Avg@8)
+LiveCodeBench AceReason-Nemotron-1.0-14B (Avg@8)
 =================================================================
 Months          Corrects        Total           Accuracy
 2023-05         211             272             77.57352941176471
@@ -116,6 +117,38 @@ Months          Corrects        Total           Accuracy
 v5              1350            2232            60.483870967741936
 v6              775             1400            55.357142857142854
 
+LiveCodeBench AceReason-Nemotron-1.1-7B (Avg@8)
+=================================================================
+Months          Corrects        Total           Accuracy
+2023-05         205             272             75.36764705882354
+2023-06         255             312             81.73076923076923
+2023-07         356             432             82.4074074074074
+2023-08         208             288             72.22222222222223
+2023-09         287             352             81.5340909090909
+2023-10         278             352             78.97727272727273
+2023-11         234             280             83.57142857142857
+2023-12         263             320             82.1875
+2024-01         215             288             74.65277777777777
+2024-02         182             256             71.09375
+2024-03         270             360             75.0
+2024-04         254             296             85.8108108108108
+2024-05         221             288             76.73611111111111
+05/23-05/24     3228            4096            78.80859375
+2024-06         309             368             83.96739130434783
+2024-07         235             344             68.31395348837209
+2024-08         292             528             55.303030303030305
+2024-09         211             376             56.11702127659574
+2024-10         254             424             59.905660377358494
+2024-11         269             456             58.99122807017544
+2024-12         239             392             60.96938775510204
+2025-01         194             408             47.549019607843135
+06/24-01/25     2003            3296            60.77063106796116
+2025-02         203             408             49.754901960784316
+2025-03         306             544             56.25
+2025-04         41              96              42.708333333333336
+v5              1283            2232            57.482078853046595
+v6              726             1400            51.857142857142854
+
 AceReason-Nemotron-7B
 ====================================
 AIME2024 (Avg@64) 68.64583333333334
@@ -125,4 +158,9 @@ AceReason-Nemotron-14B
 ====================================
 AIME2024 (Avg@64) 78.43749999999997
 AIME2025 (Avg@64) 67.65625
-```
+
+AceReason-Nemotron-1.1-7B
+====================================
+AIME2024 (Avg@64) 72.60416666666667
+AIME2025 (Avg@64) 64.84375
+```