---
library_name: transformers
license: mit
datasets:
- ethicalabs/Kurtis-E1-SFT
language:
- en
base_model:
- Qwen/Qwen2.5-3B-Instruct
pipeline_tag: text-generation
---

# Model Card for Kurtis-E1.1-Qwen2.5-3B-Instruct

Kurtis E1.1 fine-tuned with [flower](https://flower.ai/)

## Eval Results

Evaluation tasks were performed with the [LM Evaluation Harness](https://github.com/EleutherAI/lm-evaluation-harness) on an NVIDIA A40.


### hellaswag

```
lm_eval --model hf --model_args pretrained=ethicalabs/Kurtis-E1.1-Qwen2.5-3B-Instruct  --tasks hellaswag --device cuda:0 --batch_size 8
```

|  Tasks  |Version|Filter|n-shot| Metric |   |Value |   |Stderr|
|---------|------:|------|-----:|--------|---|-----:|---|-----:|
|hellaswag|      1|none  |     0|acc     |↑  |0.5555|±  |0.0050|
|         |       |none  |     0|acc_norm|↑  |0.7412|±  |0.0044|

### arc_easy

```
lm_eval --model hf --model_args pretrained=ethicalabs/Kurtis-E1.1-Qwen2.5-3B-Instruct  --tasks arc_easy --device cuda:0 --batch_size 8
```

| Tasks  |Version|Filter|n-shot| Metric |   |Value |   |Stderr|
|--------|------:|------|-----:|--------|---|-----:|---|-----:|
|arc_easy|      1|none  |     0|acc     |↑  |0.7710|±  |0.0086|
|        |       |none  |     0|acc_norm|↑  |0.6789|±  |0.0096|


### arc_challenge

```
lm_eval --model hf --model_args pretrained=ethicalabs/Kurtis-E1.1-Qwen2.5-3B-Instruct  --tasks arc_challenge --device cuda:0 --batch_size 8
```

|    Tasks    |Version|Filter|n-shot| Metric |   |Value|   |Stderr|
|-------------|------:|------|-----:|--------|---|----:|---|-----:|
|arc_challenge|      1|none  |     0|acc     |↑  |0.436|±  |0.0145|
|             |       |none  |     0|acc_norm|↑  |0.448|±  |0.0145|

### mmlu

```
lm_eval --model hf --model_args pretrained=ethicalabs/Kurtis-E1.1-Qwen2.5-3B-Instruct  --tasks mmlu --device cuda:0 --batch_size 8
```

|                 Tasks                 |Version|Filter|n-shot|Metric|   |Value |   |Stderr|
|---------------------------------------|------:|------|-----:|------|---|-----:|---|-----:|
|mmlu                                   |      2|none  |      |acc   |↑  |0.6522|±  |0.0038|
| - humanities                          |      2|none  |      |acc   |↑  |0.5734|±  |0.0066|
|  - formal_logic                       |      1|none  |     0|acc   |↑  |0.4603|±  |0.0446|
|  - high_school_european_history       |      1|none  |     0|acc   |↑  |0.7939|±  |0.0316|
|  - high_school_us_history             |      1|none  |     0|acc   |↑  |0.8333|±  |0.0262|
|  - high_school_world_history          |      1|none  |     0|acc   |↑  |0.8397|±  |0.0239|
|  - international_law                  |      1|none  |     0|acc   |↑  |0.7769|±  |0.0380|
|  - jurisprudence                      |      1|none  |     0|acc   |↑  |0.7963|±  |0.0389|
|  - logical_fallacies                  |      1|none  |     0|acc   |↑  |0.7975|±  |0.0316|
|  - moral_disputes                     |      1|none  |     0|acc   |↑  |0.6850|±  |0.0250|
|  - moral_scenarios                    |      1|none  |     0|acc   |↑  |0.2905|±  |0.0152|
|  - philosophy                         |      1|none  |     0|acc   |↑  |0.7106|±  |0.0258|
|  - prehistory                         |      1|none  |     0|acc   |↑  |0.7438|±  |0.0243|
|  - professional_law                   |      1|none  |     0|acc   |↑  |0.4759|±  |0.0128|
|  - world_religions                    |      1|none  |     0|acc   |↑  |0.8246|±  |0.0292|
| - other                               |      2|none  |      |acc   |↑  |0.7087|±  |0.0079|
|  - business_ethics                    |      1|none  |     0|acc   |↑  |0.7300|±  |0.0446|
|  - clinical_knowledge                 |      1|none  |     0|acc   |↑  |0.7321|±  |0.0273|
|  - college_medicine                   |      1|none  |     0|acc   |↑  |0.6705|±  |0.0358|
|  - global_facts                       |      1|none  |     0|acc   |↑  |0.3900|±  |0.0490|
|  - human_aging                        |      1|none  |     0|acc   |↑  |0.7130|±  |0.0304|
|  - management                         |      1|none  |     0|acc   |↑  |0.7961|±  |0.0399|
|  - marketing                          |      1|none  |     0|acc   |↑  |0.8803|±  |0.0213|
|  - medical_genetics                   |      1|none  |     0|acc   |↑  |0.7600|±  |0.0429|
|  - miscellaneous                      |      1|none  |     0|acc   |↑  |0.7957|±  |0.0144|
|  - nutrition                          |      1|none  |     0|acc   |↑  |0.7353|±  |0.0253|
|  - professional_accounting            |      1|none  |     0|acc   |↑  |0.5426|±  |0.0297|
|  - professional_medicine              |      1|none  |     0|acc   |↑  |0.6434|±  |0.0291|
|  - virology                           |      1|none  |     0|acc   |↑  |0.4880|±  |0.0389|
| - social sciences                     |      2|none  |      |acc   |↑  |0.7618|±  |0.0076|
|  - econometrics                       |      1|none  |     0|acc   |↑  |0.5439|±  |0.0469|
|  - high_school_geography              |      1|none  |     0|acc   |↑  |0.7677|±  |0.0301|
|  - high_school_government_and_politics|      1|none  |     0|acc   |↑  |0.8860|±  |0.0229|
|  - high_school_macroeconomics         |      1|none  |     0|acc   |↑  |0.6949|±  |0.0233|
|  - high_school_microeconomics         |      1|none  |     0|acc   |↑  |0.7773|±  |0.0270|
|  - high_school_psychology             |      1|none  |     0|acc   |↑  |0.8477|±  |0.0154|
|  - human_sexuality                    |      1|none  |     0|acc   |↑  |0.7786|±  |0.0364|
|  - professional_psychology            |      1|none  |     0|acc   |↑  |0.7075|±  |0.0184|
|  - public_relations                   |      1|none  |     0|acc   |↑  |0.6818|±  |0.0446|
|  - security_studies                   |      1|none  |     0|acc   |↑  |0.7224|±  |0.0287|
|  - sociology                          |      1|none  |     0|acc   |↑  |0.8458|±  |0.0255|
|  - us_foreign_policy                  |      1|none  |     0|acc   |↑  |0.8400|±  |0.0368|
| - stem                                |      2|none  |      |acc   |↑  |0.6070|±  |0.0085|
|  - abstract_algebra                   |      1|none  |     0|acc   |↑  |0.4700|±  |0.0502|
|  - anatomy                            |      1|none  |     0|acc   |↑  |0.6667|±  |0.0407|
|  - astronomy                          |      1|none  |     0|acc   |↑  |0.6776|±  |0.0380|
|  - college_biology                    |      1|none  |     0|acc   |↑  |0.7222|±  |0.0375|
|  - college_chemistry                  |      1|none  |     0|acc   |↑  |0.5000|±  |0.0503|
|  - college_computer_science           |      1|none  |     0|acc   |↑  |0.6000|±  |0.0492|
|  - college_mathematics                |      1|none  |     0|acc   |↑  |0.3400|±  |0.0476|
|  - college_physics                    |      1|none  |     0|acc   |↑  |0.4902|±  |0.0497|
|  - computer_security                  |      1|none  |     0|acc   |↑  |0.7000|±  |0.0461|
|  - conceptual_physics                 |      1|none  |     0|acc   |↑  |0.6468|±  |0.0312|
|  - electrical_engineering             |      1|none  |     0|acc   |↑  |0.6690|±  |0.0392|
|  - elementary_mathematics             |      1|none  |     0|acc   |↑  |0.5979|±  |0.0253|
|  - high_school_biology                |      1|none  |     0|acc   |↑  |0.8129|±  |0.0222|
|  - high_school_chemistry              |      1|none  |     0|acc   |↑  |0.5813|±  |0.0347|
|  - high_school_computer_science       |      1|none  |     0|acc   |↑  |0.7800|±  |0.0416|
|  - high_school_mathematics            |      1|none  |     0|acc   |↑  |0.5037|±  |0.0305|
|  - high_school_physics                |      1|none  |     0|acc   |↑  |0.4437|±  |0.0406|
|  - high_school_statistics             |      1|none  |     0|acc   |↑  |0.5972|±  |0.0334|
|  - machine_learning                   |      1|none  |     0|acc   |↑  |0.4554|±  |0.0473|

|      Groups      |Version|Filter|n-shot|Metric|   |Value |   |Stderr|
|------------------|------:|------|------|------|---|-----:|---|-----:|
|mmlu              |      2|none  |      |acc   |↑  |0.6522|±  |0.0038|
| - humanities     |      2|none  |      |acc   |↑  |0.5734|±  |0.0066|
| - other          |      2|none  |      |acc   |↑  |0.7087|±  |0.0079|
| - social sciences|      2|none  |      |acc   |↑  |0.7618|±  |0.0076|
| - stem           |      2|none  |      |acc   |↑  |0.6070|±  |0.0085|

### mmlu (5-shot)

```
lm_eval   --model hf   --model_args pretrained=ethicalabs/Kurtis-E1.1-Qwen2.5-3B-Instruct   --tasks mmlu   --device cuda:0   --batch_size 8 --num_fewshot 5
```

|                 Tasks                 |Version|Filter|n-shot|Metric|   |Value |   |Stderr|
|---------------------------------------|------:|------|-----:|------|---|-----:|---|-----:|
|mmlu                                   |      2|none  |      |acc   |↑  |0.6629|±  |0.0038|
| - humanities                          |      2|none  |      |acc   |↑  |0.5862|±  |0.0067|
|  - formal_logic                       |      1|none  |     5|acc   |↑  |0.4683|±  |0.0446|
|  - high_school_european_history       |      1|none  |     5|acc   |↑  |0.7818|±  |0.0323|
|  - high_school_us_history             |      1|none  |     5|acc   |↑  |0.8284|±  |0.0265|
|  - high_school_world_history          |      1|none  |     5|acc   |↑  |0.8692|±  |0.0219|
|  - international_law                  |      1|none  |     5|acc   |↑  |0.7769|±  |0.0380|
|  - jurisprudence                      |      1|none  |     5|acc   |↑  |0.7963|±  |0.0389|
|  - logical_fallacies                  |      1|none  |     5|acc   |↑  |0.8098|±  |0.0308|
|  - moral_disputes                     |      1|none  |     5|acc   |↑  |0.7110|±  |0.0244|
|  - moral_scenarios                    |      1|none  |     5|acc   |↑  |0.3464|±  |0.0159|
|  - philosophy                         |      1|none  |     5|acc   |↑  |0.7042|±  |0.0259|
|  - prehistory                         |      1|none  |     5|acc   |↑  |0.7284|±  |0.0247|
|  - professional_law                   |      1|none  |     5|acc   |↑  |0.4759|±  |0.0128|
|  - world_religions                    |      1|none  |     5|acc   |↑  |0.8304|±  |0.0288|
| - other                               |      2|none  |      |acc   |↑  |0.7171|±  |0.0078|
|  - business_ethics                    |      1|none  |     5|acc   |↑  |0.7400|±  |0.0441|
|  - clinical_knowledge                 |      1|none  |     5|acc   |↑  |0.7321|±  |0.0273|
|  - college_medicine                   |      1|none  |     5|acc   |↑  |0.6647|±  |0.0360|
|  - global_facts                       |      1|none  |     5|acc   |↑  |0.4100|±  |0.0494|
|  - human_aging                        |      1|none  |     5|acc   |↑  |0.7220|±  |0.0301|
|  - management                         |      1|none  |     5|acc   |↑  |0.7864|±  |0.0406|
|  - marketing                          |      1|none  |     5|acc   |↑  |0.8889|±  |0.0206|
|  - medical_genetics                   |      1|none  |     5|acc   |↑  |0.7900|±  |0.0409|
|  - miscellaneous                      |      1|none  |     5|acc   |↑  |0.7957|±  |0.0144|
|  - nutrition                          |      1|none  |     5|acc   |↑  |0.7680|±  |0.0242|
|  - professional_accounting            |      1|none  |     5|acc   |↑  |0.5532|±  |0.0297|
|  - professional_medicine              |      1|none  |     5|acc   |↑  |0.6471|±  |0.0290|
|  - virology                           |      1|none  |     5|acc   |↑  |0.5120|±  |0.0389|
| - social sciences                     |      2|none  |      |acc   |↑  |0.7735|±  |0.0075|
|  - econometrics                       |      1|none  |     5|acc   |↑  |0.5877|±  |0.0463|
|  - high_school_geography              |      1|none  |     5|acc   |↑  |0.7828|±  |0.0294|
|  - high_school_government_and_politics|      1|none  |     5|acc   |↑  |0.8756|±  |0.0238|
|  - high_school_macroeconomics         |      1|none  |     5|acc   |↑  |0.7051|±  |0.0231|
|  - high_school_microeconomics         |      1|none  |     5|acc   |↑  |0.7773|±  |0.0270|
|  - high_school_psychology             |      1|none  |     5|acc   |↑  |0.8550|±  |0.0151|
|  - human_sexuality                    |      1|none  |     5|acc   |↑  |0.8092|±  |0.0345|
|  - professional_psychology            |      1|none  |     5|acc   |↑  |0.7288|±  |0.0180|
|  - public_relations                   |      1|none  |     5|acc   |↑  |0.6909|±  |0.0443|
|  - security_studies                   |      1|none  |     5|acc   |↑  |0.7551|±  |0.0275|
|  - sociology                          |      1|none  |     5|acc   |↑  |0.8308|±  |0.0265|
|  - us_foreign_policy                  |      1|none  |     5|acc   |↑  |0.8300|±  |0.0378|
| - stem                                |      2|none  |      |acc   |↑  |0.6159|±  |0.0084|
|  - abstract_algebra                   |      1|none  |     5|acc   |↑  |0.5000|±  |0.0503|
|  - anatomy                            |      1|none  |     5|acc   |↑  |0.6222|±  |0.0419|
|  - astronomy                          |      1|none  |     5|acc   |↑  |0.7500|±  |0.0352|
|  - college_biology                    |      1|none  |     5|acc   |↑  |0.7083|±  |0.0380|
|  - college_chemistry                  |      1|none  |     5|acc   |↑  |0.4700|±  |0.0502|
|  - college_computer_science           |      1|none  |     5|acc   |↑  |0.6200|±  |0.0488|
|  - college_mathematics                |      1|none  |     5|acc   |↑  |0.4000|±  |0.0492|
|  - college_physics                    |      1|none  |     5|acc   |↑  |0.4902|±  |0.0497|
|  - computer_security                  |      1|none  |     5|acc   |↑  |0.8200|±  |0.0386|
|  - conceptual_physics                 |      1|none  |     5|acc   |↑  |0.6383|±  |0.0314|
|  - electrical_engineering             |      1|none  |     5|acc   |↑  |0.6483|±  |0.0398|
|  - elementary_mathematics             |      1|none  |     5|acc   |↑  |0.5820|±  |0.0254|
|  - high_school_biology                |      1|none  |     5|acc   |↑  |0.8161|±  |0.0220|
|  - high_school_chemistry              |      1|none  |     5|acc   |↑  |0.6059|±  |0.0344|
|  - high_school_computer_science       |      1|none  |     5|acc   |↑  |0.7500|±  |0.0435|
|  - high_school_mathematics            |      1|none  |     5|acc   |↑  |0.4926|±  |0.0305|
|  - high_school_physics                |      1|none  |     5|acc   |↑  |0.4702|±  |0.0408|
|  - high_school_statistics             |      1|none  |     5|acc   |↑  |0.6343|±  |0.0328|
|  - machine_learning                   |      1|none  |     5|acc   |↑  |0.4911|±  |0.0475|

|      Groups      |Version|Filter|n-shot|Metric|   |Value |   |Stderr|
|------------------|------:|------|------|------|---|-----:|---|-----:|
|mmlu              |      2|none  |      |acc   |↑  |0.6629|±  |0.0038|
| - humanities     |      2|none  |      |acc   |↑  |0.5862|±  |0.0067|
| - other          |      2|none  |      |acc   |↑  |0.7171|±  |0.0078|
| - social sciences|      2|none  |      |acc   |↑  |0.7735|±  |0.0075|
| - stem           |      2|none  |      |acc   |↑  |0.6159|±  |0.0084|