Update README.md
Browse files
README.md
CHANGED
@@ -24,7 +24,36 @@ The Turkish-Gemma-9b-v0.1 is designed for Turkish text generation tasks, providi
|
|
24 |
You can easily demo the model here: https://cosmos.yildiz.edu.tr/cosmosgemma
|
25 |
|
26 |
The results of a voting conducted by 18 judges on 1,450 questions are as follows:
|
27 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
28 |
|
29 |
#### Transformers pipeline
|
30 |
|
|
|
24 |
You can easily demo the model here: https://cosmos.yildiz.edu.tr/cosmosgemma
|
25 |
|
26 |
The results of a voting conducted by 18 judges on 1,450 questions are as follows:
|
27 |
+
|
28 |
+
### 🏆 Model Comparison: Win Rates
|
29 |
+
|
30 |
+
| Model Name | Win Rate |
|
31 |
+
|---------------------------------------------|-----------------|
|
32 |
+
| Qwen/Qwen3-30B-A3B | 62.39% |
|
33 |
+
| gpt-4o-mini | 62.12% |
|
34 |
+
| google/gemma-3-12b-it | 61.61% |
|
35 |
+
| google/gemma-2-27b-it | 57.91% |
|
36 |
+
| **ytu-ce-cosmos/Turkish-Gemma-9b-v0.1** | **57.30%** |
|
37 |
+
| google/gemma-2-9b-it | 54.13% |
|
38 |
+
| ytu-ce-cosmos/Turkish-Llama-8b-DPO-v0.1 | 36.89% |
|
39 |
+
|
40 |
+
### 📊 Turkish Evaluation Benchmark Results (via `malhajar17/lm-evaluation-harness_turkish`)
|
41 |
+
|
42 |
+
| Model Name | Average | MMLU | Truthful_QA | ARC | Hellaswag | Gsm8K | Winogrande |
|
43 |
+
|---------------------------------------------|---------|-------|--------------|-------|-----------|-------|------------|
|
44 |
+
| Qwen/Qwen2.5-72B-Instruct | 67.69 | 77.28 | 59.86 | 61.52 | 61.98 | 83.6 | 61.92 |
|
45 |
+
| google/gemma-3-27b-it | 67.36 | 70.2 | 57.06 | 66.98 | 66.58 | 77.52 | 65.8 |
|
46 |
+
| google/gemma-2-27b-it | 65.57 | 66.49 | 57.45 | 63.65 | 63.86 | 76.54 | 65.4 |
|
47 |
+
| meta-llama/Llama-3-1-70B-Instruct | 63.92 | 74.00 | 51.41 | 59.64 | 64.31 | 66.13 | 66.90 |
|
48 |
+
| Qwen/Qwen2.5-32B-Instruct | 63.74 | 70.93 | 57.87 | 57.00 | 57.04 | 77.83 | 61.77 |
|
49 |
+
| **ytu-ce-cosmos/Turkish-Gemma-9b-v0.1** | **63.31** | **63.85** | **54.21** | **59.64** | **64.19** | **73.42** | **64.53** |
|
50 |
+
| google/gemma-3-12b-it | 62.94 | 63.92 | 57.16 | 60.67 | 62.00 | 72.06 | 61.77 |
|
51 |
+
| Qwen/Qwen2.5-14B-it | 60.34 | 65.28 | 59.00 | 50.00 | 52.22 | 76.77 | 58.77 |
|
52 |
+
| google/gemma-2-9b-it | 59.14 | 61.07 | 55.77 | 56.31 | 56.48 | 63.10 | 62.09 |
|
53 |
+
| ytu-ce-cosmos/Turkish-Llama-8b-DPO-v0.1 | 55.03 | 51.97 | 57.56 | 51.02 | 52.96 | 59.87 | 57.77 |
|
54 |
+
| Qwen/Qwen2.5-7B-Instruct | 53.42 | 56.31 | 55.99 | 42.06 | 44.71 | 64.16 | 59.66 |
|
55 |
+
|
56 |
+
|
57 |
|
58 |
#### Transformers pipeline
|
59 |
|