huseyinemreseyrek commited on
Commit
0cc1428
·
verified ·
1 Parent(s): 2ed5a49

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +30 -1
README.md CHANGED
@@ -24,7 +24,36 @@ The Turkish-Gemma-9b-v0.1 is designed for Turkish text generation tasks, providi
24
  You can easily demo the model here: https://cosmos.yildiz.edu.tr/cosmosgemma
25
 
26
  The results of a voting conducted by 18 judges on 1,450 questions are as follows:
27
- <img src="./oylama.jpeg"/>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
28
 
29
  #### Transformers pipeline
30
 
 
24
  You can easily demo the model here: https://cosmos.yildiz.edu.tr/cosmosgemma
25
 
26
  The results of a voting conducted by 18 judges on 1,450 questions are as follows:
27
+
28
+ ### 🏆 Model Comparison: Win Rates
29
+
30
+ | Model Name | Win Rate |
31
+ |---------------------------------------------|-----------------|
32
+ | Qwen/Qwen3-30B-A3B | 62.39% |
33
+ | gpt-4o-mini | 62.12% |
34
+ | google/gemma-3-12b-it | 61.61% |
35
+ | google/gemma-2-27b-it | 57.91% |
36
+ | **ytu-ce-cosmos/Turkish-Gemma-9b-v0.1** | **57.30%** |
37
+ | google/gemma-2-9b-it | 54.13% |
38
+ | ytu-ce-cosmos/Turkish-Llama-8b-DPO-v0.1 | 36.89% |
39
+
40
+ ### 📊 Turkish Evaluation Benchmark Results (via `malhajar17/lm-evaluation-harness_turkish`)
41
+
42
+ | Model Name | Average | MMLU | Truthful_QA | ARC | Hellaswag | Gsm8K | Winogrande |
43
+ |---------------------------------------------|---------|-------|--------------|-------|-----------|-------|------------|
44
+ | Qwen/Qwen2.5-72B-Instruct | 67.69 | 77.28 | 59.86 | 61.52 | 61.98 | 83.6 | 61.92 |
45
+ | google/gemma-3-27b-it | 67.36 | 70.2 | 57.06 | 66.98 | 66.58 | 77.52 | 65.8 |
46
+ | google/gemma-2-27b-it | 65.57 | 66.49 | 57.45 | 63.65 | 63.86 | 76.54 | 65.4 |
47
+ | meta-llama/Llama-3-1-70B-Instruct | 63.92 | 74.00 | 51.41 | 59.64 | 64.31 | 66.13 | 66.90 |
48
+ | Qwen/Qwen2.5-32B-Instruct | 63.74 | 70.93 | 57.87 | 57.00 | 57.04 | 77.83 | 61.77 |
49
+ | **ytu-ce-cosmos/Turkish-Gemma-9b-v0.1** | **63.31** | **63.85** | **54.21** | **59.64** | **64.19** | **73.42** | **64.53** |
50
+ | google/gemma-3-12b-it | 62.94 | 63.92 | 57.16 | 60.67 | 62.00 | 72.06 | 61.77 |
51
+ | Qwen/Qwen2.5-14B-it | 60.34 | 65.28 | 59.00 | 50.00 | 52.22 | 76.77 | 58.77 |
52
+ | google/gemma-2-9b-it | 59.14 | 61.07 | 55.77 | 56.31 | 56.48 | 63.10 | 62.09 |
53
+ | ytu-ce-cosmos/Turkish-Llama-8b-DPO-v0.1 | 55.03 | 51.97 | 57.56 | 51.02 | 52.96 | 59.87 | 57.77 |
54
+ | Qwen/Qwen2.5-7B-Instruct | 53.42 | 56.31 | 55.99 | 42.06 | 44.71 | 64.16 | 59.66 |
55
+
56
+
57
 
58
  #### Transformers pipeline
59