Update README.md

README.md

@@ -5,8 +5,58 @@ tags: []
 
 # Model Card for Model ID
 
-<!-- Provide a quick summary of what the model is/does. -->
-
+GaMS-9B after the second round of training - high quality Slovene corpora. The training was done using NeMo 2.0. To fill the batches, the EOS token was added.
+
+Starting model: GaMS-Beta/GaMS-Parallel-2.0
+
+## Data
+
+| Corpus | Language | # Tokens | Percentage |
+| :----- | :------- | :------: | :--------: |
+| KAS	 | Slovene  | 2.77 B   | 20.34 %    |
+| Metafida	| Slovene | 4.66 B | 34.18 %    |
+| Wikipedia-En | English | 5.45 B |	39.99 % |
+| Wikipedia-Sl | Slovene | 0.16 B |	1.19 %  |
+| Wikipedia-Hr | Croatian | 0.15 B | 1.13 % |
+| Wikipedia-Bs | Bosnian | 0.07 B | 0.50 %  |
+| Wikipedia-Sr-Latin | Serbian | 0.36 B | 2.68 % |
+| Total | | 13.62 B | |
+
+## Slovenian-LLM-Eval results
+
+![image/png](https://cdn-uploads.huggingface.co/production/uploads/652d40a78fa1fbb0aae165bb/YszAknPaoxiBR0c_7Gu3U.png)
+
+## Slobench Results
+
+The reported results were obtained using guided decoding.
+
+### 0-shot results
+
+|   | Model                                  | BoolQ_accuracy    | MultiRC_exact_match   | MultiRC_per_question_f1   | MultiRC_f1_over_all_answers   | WSC_accuracy      | COPA_accuracy     | RTE_accuracy      | CB_accuracy       | CB_f1             | NLI_accuracy      | NLI_precision_entailment   | NLI_recall_entailment   | NLI_f1_entailment   | NLI_precision_neutral   | NLI_recall_neutral   | NLI_f1_neutral    | NLI_precision_contradiction   | NLI_recall_contradiction   | NLI_f1_contradiction   |
+|--:|:---------------------------------------|:------------------|:----------------------|:--------------------------|:------------------------------|:------------------|:------------------|:------------------|:------------------|:------------------|:------------------|:---------------------------|:------------------------|:--------------------|:------------------------|:---------------------|:------------------|:------------------------------|:---------------------------|:-----------------------|
+| 0 | /models/hf_models/GaMS-9B-Parallel-2.0 | 0.76 [0.74, 0.77] | 0.21 [0.19, 0.24]     | 0.56 [0.54, 0.58]         | 0.55 [0.53, 0.57]             | 0.38 [0.28, 0.47] | 0.6 [0.5, 0.7]    | 0.6 [0.54, 0.66]  | 0.68 [0.55, 0.8]  | 0.59 [0.39, 0.76] | 0.35 [0.31, 0.39] | 0.5 [0.23, 0.77]           | 0.04 [0.02, 0.07]       | 0.08 [0.03, 0.13]   | 0.33 [0.29, 0.37]       | 0.85 [0.79, 0.9]     | 0.47 [0.42, 0.52] | 0.41 [0.31, 0.52]             | 0.19 [0.13, 0.24]          | 0.26 [0.18, 0.32]      |
+| 1 | google/gemma-2-9b                      | 0.78 [0.77, 0.8]  | 0.2 [0.18, 0.23]      | 0.48 [0.45, 0.5]          | 0.49 [0.48, 0.51]             | 0.63 [0.54, 0.73] | 0.68 [0.59, 0.77] | 0.69 [0.63, 0.74] | 0.55 [0.42, 0.69] | 0.31 [0.22, 0.41] | 0.33 [0.29, 0.37] | 0.55 [0.25, 0.83]          | 0.03 [0.01, 0.06]       | 0.06 [0.02, 0.11]   | 0.32 [0.28, 0.36]       | 0.98 [0.95, 0.99]    | 0.48 [0.44, 0.52] | 0.57 [0.14, 1.0]              | 0.02 [0.01, 0.04]          | 0.04 [0.01, 0.08]      |
+| 2 | google/gemma-2-9b-it                   | 0.83 [0.82, 0.84] | 0.18 [0.16, 0.2]      | 0.6 [0.58, 0.62]          | 0.5 [0.49, 0.52]              | 0.62 [0.52, 0.71] | 0.86 [0.79, 0.93] | 0.8 [0.75, 0.85]  | 0.82 [0.72, 0.92] | 0.73 [0.58, 0.85] | 0.47 [0.43, 0.51] | 0.48 [0.28, 0.68]          | 0.06 [0.03, 0.1]        | 0.11 [0.05, 0.17]   | 0.4 [0.35, 0.46]        | 0.66 [0.59, 0.74]    | 0.5 [0.44, 0.56]  | 0.55 [0.49, 0.62]             | 0.72 [0.66, 0.78]          | 0.62 [0.57, 0.68]      |
+| 3 | zlatorog/Zlatorog_SFT_v2               | 0.82 [0.81, 0.84] | 0.35 [0.32, 0.38]     | 0.72 [0.7, 0.74]          | 0.71 [0.69, 0.72]             | 0.65 [0.56, 0.75] | 0.79 [0.71, 0.87] | 0.79 [0.74, 0.84] | 0.77 [0.65, 0.88] | 0.63 [0.46, 0.82] | 0.63 [0.59, 0.67] | 0.54 [0.48, 0.59]          | 0.93 [0.9, 0.96]        | 0.68 [0.63, 0.73]   | 0.8 [0.25, 1.0]         | 0.02 [0.01, 0.05]    | 0.04 [0.01, 0.1]  | 0.79 [0.73, 0.84]             | 0.9 [0.86, 0.94]           | 0.84 [0.8, 0.88]       |
+| 4 | cjvt/GaMS-1B                           | 0.52 [0.5, 0.54]  | 0.01 [0.0, 0.01]      | 0.05 [0.04, 0.05]         | 0.04 [0.04, 0.05]             | 0.61 [0.51, 0.7]  | 0.55 [0.45, 0.65] | 0.48 [0.42, 0.54] | 0.21 [0.1, 0.33]  | 0.19 [0.09, 0.27] | 0.33 [0.29, 0.36] | 0.34 [0.28, 0.4]           | 0.43 [0.36, 0.5]        | 0.38 [0.32, 0.44]   | 0.33 [0.25, 0.41]       | 0.26 [0.2, 0.33]     | 0.29 [0.22, 0.35] | 0.3 [0.23, 0.37]              | 0.28 [0.21, 0.34]          | 0.29 [0.22, 0.35]      |
+| 5 | cjvt/GaMS-1B-Chat                      | 0.62 [0.61, 0.64] | 0.03 [0.02, 0.04]     | 0.13 [0.12, 0.14]         | 0.07 [0.07, 0.08]             | 0.5 [0.4, 0.6]    | 0.55 [0.45, 0.65] | 0.47 [0.41, 0.53] | 0.5 [0.36, 0.64]  | 0.22 [0.18, 0.26] | 0.35 [0.31, 0.39] | 0.35 [0.31, 0.39]          | 0.99 [0.98, 1.0]        | 0.52 [0.47, 0.56]   | 0.0 [0.0, 0.0]          | 0.0 [0.0, 0.0]       | 0.0 [0.0, 0.0]    | 0.0 [0.0, 0.0]                | 0.0 [0.0, 0.0]             | 0.0 [0.0, 0.0]         |
+| 6 | utter-project/EuroLLM-9B               | 0.77 [0.76, 0.79] | 0.12 [0.1, 0.14]      | 0.53 [0.52, 0.55]         | 0.53 [0.52, 0.55]             | 0.55 [0.45, 0.65] | 0.78 [0.7, 0.86]  | 0.73 [0.68, 0.79] | 0.79 [0.67, 0.9]  | 0.55 [0.48, 0.62] | 0.39 [0.34, 0.43] | 0.64 [0.33, 1.0]           | 0.04 [0.01, 0.07]       | 0.07 [0.02, 0.12]   | 0.34 [0.3, 0.38]        | 0.95 [0.92, 0.98]    | 0.5 [0.45, 0.54]  | 0.87 [0.76, 0.96]             | 0.22 [0.15, 0.28]          | 0.35 [0.26, 0.42]      |
+| 7 | utter-project/EuroLLM-9B-Instruct      | 0.81 [0.79, 0.82] | 0.18 [0.15, 0.2]      | 0.64 [0.62, 0.65]         | 0.64 [0.62, 0.66]             | 0.6 [0.5, 0.69]   | 0.58 [0.48, 0.68] | 0.82 [0.77, 0.87] | 0.77 [0.65, 0.88] | 0.63 [0.46, 0.82] | 0.38 [0.34, 0.42] | 0.46 [0.41, 0.52]          | 0.76 [0.69, 0.82]       | 0.57 [0.52, 0.62]   | 0.24 [0.18, 0.3]        | 0.21 [0.15, 0.26]    | 0.22 [0.16, 0.28] | 0.31 [0.21, 0.42]             | 0.14 [0.09, 0.2]           | 0.2 [0.13, 0.26]       |
+| 8 | /models/hf_models/GaMS-9B-SecondRound  | 0.78 [0.76, 0.79] | 0.23 [0.21, 0.26]     | 0.63 [0.61, 0.65]         | 0.53 [0.52, 0.55]             | 0.59 [0.49, 0.68] | 0.65 [0.55, 0.75] | 0.77 [0.72, 0.82] | 0.68 [0.55, 0.8]  | 0.64 [0.52, 0.76] | 0.46 [0.42, 0.5]  | 0.49 [0.42, 0.57]          | 0.45 [0.38, 0.52]       | 0.47 [0.41, 0.53]   | 0.33 [0.25, 0.4]        | 0.29 [0.23, 0.36]    | 0.31 [0.24, 0.37] | 0.53 [0.46, 0.6]              | 0.62 [0.56, 0.7]           | 0.57 [0.51, 0.63]      |
+
+### 3-shot results
+
+|   | Model                                  | BoolQ_accuracy    | MultiRC_exact_match   | MultiRC_per_question_f1   | MultiRC_f1_over_all_answers   | WSC_accuracy      | COPA_accuracy     | RTE_accuracy      | CB_accuracy       | CB_f1             | NLI_accuracy      | NLI_precision_entailment   | NLI_recall_entailment   | NLI_f1_entailment   | NLI_precision_neutral   | NLI_recall_neutral   | NLI_f1_neutral    | NLI_precision_contradiction   | NLI_recall_contradiction   | NLI_f1_contradiction   |
+|--:|:---------------------------------------|:------------------|:----------------------|:--------------------------|:------------------------------|:------------------|:------------------|:------------------|:------------------|:------------------|:------------------|:---------------------------|:------------------------|:--------------------|:------------------------|:---------------------|:------------------|:------------------------------|:---------------------------|:-----------------------|
+| 0 | /models/hf_models/GaMS-9B-Parallel-2.0 | 0.83 [0.81, 0.84] | 0.36 [0.33, 0.39]     | 0.74 [0.72, 0.75]         | 0.74 [0.73, 0.76]             | 0.64 [0.55, 0.74] | 0.87 [0.8, 0.94]  | 0.78 [0.73, 0.83] | 0.84 [0.74, 0.94] | 0.59 [0.53, 0.64] | 0.48 [0.43, 0.52] | 0.38 [0.21, 0.56]          | 0.07 [0.03, 0.11]       | 0.11 [0.06, 0.18]   | 0.37 [0.32, 0.42]       | 0.67 [0.6, 0.74]     | 0.48 [0.42, 0.53] | 0.66 [0.59, 0.73]             | 0.72 [0.66, 0.78]          | 0.69 [0.63, 0.75]      |
+| 1 | google/gemma-2-9b                      | 0.82 [0.81, 0.83] | 0.37 [0.34, 0.4]      | 0.75 [0.73, 0.77]         | 0.75 [0.74, 0.77]             | 0.66 [0.57, 0.76] | 0.88 [0.82, 0.94] | 0.79 [0.74, 0.84] | 0.88 [0.79, 0.96] | 0.61 [0.56, 0.65] | 0.48 [0.44, 0.52] | 0.52 [0.37, 0.65]          | 0.15 [0.1, 0.19]        | 0.23 [0.15, 0.29]   | 0.37 [0.32, 0.42]       | 0.77 [0.71, 0.83]    | 0.5 [0.45, 0.55]  | 0.75 [0.68, 0.82]             | 0.56 [0.48, 0.63]          | 0.64 [0.57, 0.71]      |
+| 2 | google/gemma-2-9b-it                   | 0.84 [0.83, 0.85] | 0.15 [0.13, 0.18]     | 0.66 [0.64, 0.67]         | 0.65 [0.64, 0.67]             | 0.7 [0.61, 0.79]  | 0.89 [0.83, 0.95] | 0.82 [0.78, 0.87] | 0.84 [0.74, 0.94] | 0.75 [0.59, 0.88] | 0.65 [0.61, 0.69] | 0.66 [0.6, 0.72]           | 0.77 [0.71, 0.83]       | 0.71 [0.66, 0.76]   | 0.56 [0.46, 0.68]       | 0.28 [0.22, 0.35]    | 0.37 [0.3, 0.45]  | 0.68 [0.61, 0.74]             | 0.88 [0.83, 0.92]          | 0.76 [0.72, 0.81]      |
+| 3 | zlatorog/Zlatorog_SFT_v2               | 0.83 [0.82, 0.84] | 0.04 [0.02, 0.05]     | 0.54 [0.52, 0.55]         | 0.55 [0.53, 0.56]             | 0.51 [0.41, 0.61] | 0.66 [0.57, 0.75] | 0.73 [0.67, 0.78] | 0.7 [0.57, 0.82]  | 0.48 [0.4, 0.56]  | 0.56 [0.52, 0.6]  | 0.55 [0.49, 0.62]          | 0.62 [0.55, 0.69]       | 0.58 [0.53, 0.64]   | 0.39 [0.31, 0.49]       | 0.24 [0.19, 0.31]    | 0.3 [0.23, 0.37]  | 0.64 [0.58, 0.71]             | 0.79 [0.73, 0.85]          | 0.71 [0.66, 0.76]      |
+| 4 | cjvt/GaMS-1B                           | 0.49 [0.47, 0.5]  | 0.07 [0.06, 0.09]     | 0.41 [0.39, 0.43]         | 0.37 [0.35, 0.39]             | 0.58 [0.48, 0.67] | 0.49 [0.39, 0.59] | 0.47 [0.41, 0.53] | 0.43 [0.29, 0.56] | 0.21 [0.17, 0.25] | 0.32 [0.28, 0.36] | 0.36 [0.27, 0.44]          | 0.22 [0.16, 0.29]       | 0.27 [0.21, 0.34]   | 0.31 [0.27, 0.36]       | 0.77 [0.7, 0.83]     | 0.44 [0.39, 0.49] | 0.0 [0.0, 0.0]                | 0.0 [0.0, 0.0]             | 0.0 [0.0, 0.0]         |
+| 5 | cjvt/GaMS-1B-Chat                      | 0.59 [0.57, 0.61] | 0.04 [0.02, 0.05]     | 0.34 [0.32, 0.36]         | 0.16 [0.15, 0.16]             | 0.63 [0.54, 0.73] | 0.55 [0.45, 0.65] | 0.47 [0.41, 0.53] | 0.43 [0.29, 0.56] | 0.2 [0.16, 0.24]  | 0.36 [0.32, 0.4]  | 0.36 [0.32, 0.4]           | 0.99 [0.97, 1.0]        | 0.53 [0.48, 0.57]   | 0.47 [0.2, 0.73]        | 0.04 [0.01, 0.07]    | 0.07 [0.02, 0.13] | 0.0 [0.0, 0.0]                | 0.0 [0.0, 0.0]             | 0.0 [0.0, 0.0]         |
+| 6 | utter-project/EuroLLM-9B               | 0.81 [0.79, 0.82] | 0.17 [0.15, 0.2]      | 0.61 [0.59, 0.63]         | 0.58 [0.57, 0.6]              | 0.65 [0.56, 0.75] | 0.63 [0.53, 0.73] | 0.73 [0.67, 0.78] | 0.73 [0.61, 0.85] | 0.59 [0.47, 0.71] | 0.45 [0.41, 0.49] | 1.0 [0.0, 1.0]             | 0.02 [0.0, 0.04]        | 0.03 [0.0, 0.07]    | 0.38 [0.32, 0.45]       | 0.53 [0.46, 0.61]    | 0.44 [0.38, 0.51] | 0.49 [0.44, 0.55]             | 0.83 [0.78, 0.88]          | 0.62 [0.57, 0.67]      |
+| 7 | utter-project/EuroLLM-9B-Instruct      | 0.81 [0.79, 0.82] | 0.07 [0.05, 0.08]     | 0.52 [0.5, 0.53]          | 0.53 [0.51, 0.55]             | 0.65 [0.56, 0.75] | 0.69 [0.6, 0.78]  | 0.79 [0.74, 0.84] | 0.79 [0.67, 0.9]  | 0.71 [0.56, 0.84] | 0.42 [0.38, 0.47] | 0.61 [0.53, 0.69]          | 0.48 [0.41, 0.55]       | 0.53 [0.47, 0.6]    | 0.31 [0.26, 0.36]       | 0.57 [0.5, 0.64]     | 0.4 [0.35, 0.45]  | 0.54 [0.43, 0.65]             | 0.23 [0.17, 0.29]          | 0.32 [0.25, 0.39]      |
+| 8 | /models/hf_models/GaMS-9B-SecondRound  | 0.8 [0.78, 0.81]  | 0.15 [0.13, 0.17]     | 0.64 [0.62, 0.65]         | 0.58 [0.57, 0.6]              | 0.66 [0.57, 0.76] | 0.9 [0.84, 0.96]  | 0.82 [0.77, 0.87] | 0.84 [0.74, 0.94] | 0.73 [0.56, 0.87] | 0.56 [0.52, 0.6]  | 0.58 [0.5, 0.66]           | 0.38 [0.31, 0.46]       | 0.46 [0.39, 0.53]   | 0.42 [0.35, 0.5]        | 0.44 [0.37, 0.51]    | 0.43 [0.37, 0.49] | 0.65 [0.59, 0.71]             | 0.86 [0.8, 0.91]           | 0.74 [0.69, 0.79]      |
 
 
 ## Model Details