metadata
library_name: transformers
license: mit
datasets:
- ethicalabs/Kurtis-E1-SFT
language:
- en
base_model:
- Qwen/Qwen2.5-3B-Instruct
pipeline_tag: text-generation
Model Card for Kurtis-E1.1-Qwen2.5-3B-Instruct
Kurtis E1.1 fine-tuned with flower
Eval Results
Evaluation tasks were performed with the LM Evaluation Harness on an NVIDIA A40.
hellaswag
lm_eval --model hf --model_args pretrained=ethicalabs/Kurtis-E1.1-Qwen2.5-3B-Instruct --tasks hellaswag --device cuda:0 --batch_size 8
| Tasks | Version | Filter | n-shot | Metric | Value | Stderr | ||
|---|---|---|---|---|---|---|---|---|
| hellaswag | 1 | none | 0 | acc | ↑ | 0.5555 | ± | 0.0050 |
| none | 0 | acc_norm | ↑ | 0.7412 | ± | 0.0044 |
arc_easy
lm_eval --model hf --model_args pretrained=ethicalabs/Kurtis-E1.1-Qwen2.5-3B-Instruct --tasks arc_easy --device cuda:0 --batch_size 8
| Tasks | Version | Filter | n-shot | Metric | Value | Stderr | ||
|---|---|---|---|---|---|---|---|---|
| arc_easy | 1 | none | 0 | acc | ↑ | 0.7710 | ± | 0.0086 |
| none | 0 | acc_norm | ↑ | 0.6789 | ± | 0.0096 |
arc_challenge
lm_eval --model hf --model_args pretrained=ethicalabs/Kurtis-E1.1-Qwen2.5-3B-Instruct --tasks arc_challenge --device cuda:0 --batch_size 8
| Tasks | Version | Filter | n-shot | Metric | Value | Stderr | ||
|---|---|---|---|---|---|---|---|---|
| arc_challenge | 1 | none | 0 | acc | ↑ | 0.436 | ± | 0.0145 |
| none | 0 | acc_norm | ↑ | 0.448 | ± | 0.0145 |
mmlu
lm_eval --model hf --model_args pretrained=ethicalabs/Kurtis-E1.1-Qwen2.5-3B-Instruct --tasks mmlu --device cuda:0 --batch_size 8
| Tasks | Version | Filter | n-shot | Metric | Value | Stderr | ||
|---|---|---|---|---|---|---|---|---|
| mmlu | 2 | none | acc | ↑ | 0.6522 | ± | 0.0038 | |
| - humanities | 2 | none | acc | ↑ | 0.5734 | ± | 0.0066 | |
| - formal_logic | 1 | none | 0 | acc | ↑ | 0.4603 | ± | 0.0446 |
| - high_school_european_history | 1 | none | 0 | acc | ↑ | 0.7939 | ± | 0.0316 |
| - high_school_us_history | 1 | none | 0 | acc | ↑ | 0.8333 | ± | 0.0262 |
| - high_school_world_history | 1 | none | 0 | acc | ↑ | 0.8397 | ± | 0.0239 |
| - international_law | 1 | none | 0 | acc | ↑ | 0.7769 | ± | 0.0380 |
| - jurisprudence | 1 | none | 0 | acc | ↑ | 0.7963 | ± | 0.0389 |
| - logical_fallacies | 1 | none | 0 | acc | ↑ | 0.7975 | ± | 0.0316 |
| - moral_disputes | 1 | none | 0 | acc | ↑ | 0.6850 | ± | 0.0250 |
| - moral_scenarios | 1 | none | 0 | acc | ↑ | 0.2905 | ± | 0.0152 |
| - philosophy | 1 | none | 0 | acc | ↑ | 0.7106 | ± | 0.0258 |
| - prehistory | 1 | none | 0 | acc | ↑ | 0.7438 | ± | 0.0243 |
| - professional_law | 1 | none | 0 | acc | ↑ | 0.4759 | ± | 0.0128 |
| - world_religions | 1 | none | 0 | acc | ↑ | 0.8246 | ± | 0.0292 |
| - other | 2 | none | acc | ↑ | 0.7087 | ± | 0.0079 | |
| - business_ethics | 1 | none | 0 | acc | ↑ | 0.7300 | ± | 0.0446 |
| - clinical_knowledge | 1 | none | 0 | acc | ↑ | 0.7321 | ± | 0.0273 |
| - college_medicine | 1 | none | 0 | acc | ↑ | 0.6705 | ± | 0.0358 |
| - global_facts | 1 | none | 0 | acc | ↑ | 0.3900 | ± | 0.0490 |
| - human_aging | 1 | none | 0 | acc | ↑ | 0.7130 | ± | 0.0304 |
| - management | 1 | none | 0 | acc | ↑ | 0.7961 | ± | 0.0399 |
| - marketing | 1 | none | 0 | acc | ↑ | 0.8803 | ± | 0.0213 |
| - medical_genetics | 1 | none | 0 | acc | ↑ | 0.7600 | ± | 0.0429 |
| - miscellaneous | 1 | none | 0 | acc | ↑ | 0.7957 | ± | 0.0144 |
| - nutrition | 1 | none | 0 | acc | ↑ | 0.7353 | ± | 0.0253 |
| - professional_accounting | 1 | none | 0 | acc | ↑ | 0.5426 | ± | 0.0297 |
| - professional_medicine | 1 | none | 0 | acc | ↑ | 0.6434 | ± | 0.0291 |
| - virology | 1 | none | 0 | acc | ↑ | 0.4880 | ± | 0.0389 |
| - social sciences | 2 | none | acc | ↑ | 0.7618 | ± | 0.0076 | |
| - econometrics | 1 | none | 0 | acc | ↑ | 0.5439 | ± | 0.0469 |
| - high_school_geography | 1 | none | 0 | acc | ↑ | 0.7677 | ± | 0.0301 |
| - high_school_government_and_politics | 1 | none | 0 | acc | ↑ | 0.8860 | ± | 0.0229 |
| - high_school_macroeconomics | 1 | none | 0 | acc | ↑ | 0.6949 | ± | 0.0233 |
| - high_school_microeconomics | 1 | none | 0 | acc | ↑ | 0.7773 | ± | 0.0270 |
| - high_school_psychology | 1 | none | 0 | acc | ↑ | 0.8477 | ± | 0.0154 |
| - human_sexuality | 1 | none | 0 | acc | ↑ | 0.7786 | ± | 0.0364 |
| - professional_psychology | 1 | none | 0 | acc | ↑ | 0.7075 | ± | 0.0184 |
| - public_relations | 1 | none | 0 | acc | ↑ | 0.6818 | ± | 0.0446 |
| - security_studies | 1 | none | 0 | acc | ↑ | 0.7224 | ± | 0.0287 |
| - sociology | 1 | none | 0 | acc | ↑ | 0.8458 | ± | 0.0255 |
| - us_foreign_policy | 1 | none | 0 | acc | ↑ | 0.8400 | ± | 0.0368 |
| - stem | 2 | none | acc | ↑ | 0.6070 | ± | 0.0085 | |
| - abstract_algebra | 1 | none | 0 | acc | ↑ | 0.4700 | ± | 0.0502 |
| - anatomy | 1 | none | 0 | acc | ↑ | 0.6667 | ± | 0.0407 |
| - astronomy | 1 | none | 0 | acc | ↑ | 0.6776 | ± | 0.0380 |
| - college_biology | 1 | none | 0 | acc | ↑ | 0.7222 | ± | 0.0375 |
| - college_chemistry | 1 | none | 0 | acc | ↑ | 0.5000 | ± | 0.0503 |
| - college_computer_science | 1 | none | 0 | acc | ↑ | 0.6000 | ± | 0.0492 |
| - college_mathematics | 1 | none | 0 | acc | ↑ | 0.3400 | ± | 0.0476 |
| - college_physics | 1 | none | 0 | acc | ↑ | 0.4902 | ± | 0.0497 |
| - computer_security | 1 | none | 0 | acc | ↑ | 0.7000 | ± | 0.0461 |
| - conceptual_physics | 1 | none | 0 | acc | ↑ | 0.6468 | ± | 0.0312 |
| - electrical_engineering | 1 | none | 0 | acc | ↑ | 0.6690 | ± | 0.0392 |
| - elementary_mathematics | 1 | none | 0 | acc | ↑ | 0.5979 | ± | 0.0253 |
| - high_school_biology | 1 | none | 0 | acc | ↑ | 0.8129 | ± | 0.0222 |
| - high_school_chemistry | 1 | none | 0 | acc | ↑ | 0.5813 | ± | 0.0347 |
| - high_school_computer_science | 1 | none | 0 | acc | ↑ | 0.7800 | ± | 0.0416 |
| - high_school_mathematics | 1 | none | 0 | acc | ↑ | 0.5037 | ± | 0.0305 |
| - high_school_physics | 1 | none | 0 | acc | ↑ | 0.4437 | ± | 0.0406 |
| - high_school_statistics | 1 | none | 0 | acc | ↑ | 0.5972 | ± | 0.0334 |
| - machine_learning | 1 | none | 0 | acc | ↑ | 0.4554 | ± | 0.0473 |
| Groups | Version | Filter | n-shot | Metric | Value | Stderr | ||
|---|---|---|---|---|---|---|---|---|
| mmlu | 2 | none | acc | ↑ | 0.6522 | ± | 0.0038 | |
| - humanities | 2 | none | acc | ↑ | 0.5734 | ± | 0.0066 | |
| - other | 2 | none | acc | ↑ | 0.7087 | ± | 0.0079 | |
| - social sciences | 2 | none | acc | ↑ | 0.7618 | ± | 0.0076 | |
| - stem | 2 | none | acc | ↑ | 0.6070 | ± | 0.0085 |
mmlu (5-shot)
lm_eval --model hf --model_args pretrained=ethicalabs/Kurtis-E1.1-Qwen2.5-3B-Instruct --tasks mmlu --device cuda:0 --batch_size 8 --num_fewshot 5
| Tasks | Version | Filter | n-shot | Metric | Value | Stderr | ||
|---|---|---|---|---|---|---|---|---|
| mmlu | 2 | none | acc | ↑ | 0.6629 | ± | 0.0038 | |
| - humanities | 2 | none | acc | ↑ | 0.5862 | ± | 0.0067 | |
| - formal_logic | 1 | none | 5 | acc | ↑ | 0.4683 | ± | 0.0446 |
| - high_school_european_history | 1 | none | 5 | acc | ↑ | 0.7818 | ± | 0.0323 |
| - high_school_us_history | 1 | none | 5 | acc | ↑ | 0.8284 | ± | 0.0265 |
| - high_school_world_history | 1 | none | 5 | acc | ↑ | 0.8692 | ± | 0.0219 |
| - international_law | 1 | none | 5 | acc | ↑ | 0.7769 | ± | 0.0380 |
| - jurisprudence | 1 | none | 5 | acc | ↑ | 0.7963 | ± | 0.0389 |
| - logical_fallacies | 1 | none | 5 | acc | ↑ | 0.8098 | ± | 0.0308 |
| - moral_disputes | 1 | none | 5 | acc | ↑ | 0.7110 | ± | 0.0244 |
| - moral_scenarios | 1 | none | 5 | acc | ↑ | 0.3464 | ± | 0.0159 |
| - philosophy | 1 | none | 5 | acc | ↑ | 0.7042 | ± | 0.0259 |
| - prehistory | 1 | none | 5 | acc | ↑ | 0.7284 | ± | 0.0247 |
| - professional_law | 1 | none | 5 | acc | ↑ | 0.4759 | ± | 0.0128 |
| - world_religions | 1 | none | 5 | acc | ↑ | 0.8304 | ± | 0.0288 |
| - other | 2 | none | acc | ↑ | 0.7171 | ± | 0.0078 | |
| - business_ethics | 1 | none | 5 | acc | ↑ | 0.7400 | ± | 0.0441 |
| - clinical_knowledge | 1 | none | 5 | acc | ↑ | 0.7321 | ± | 0.0273 |
| - college_medicine | 1 | none | 5 | acc | ↑ | 0.6647 | ± | 0.0360 |
| - global_facts | 1 | none | 5 | acc | ↑ | 0.4100 | ± | 0.0494 |
| - human_aging | 1 | none | 5 | acc | ↑ | 0.7220 | ± | 0.0301 |
| - management | 1 | none | 5 | acc | ↑ | 0.7864 | ± | 0.0406 |
| - marketing | 1 | none | 5 | acc | ↑ | 0.8889 | ± | 0.0206 |
| - medical_genetics | 1 | none | 5 | acc | ↑ | 0.7900 | ± | 0.0409 |
| - miscellaneous | 1 | none | 5 | acc | ↑ | 0.7957 | ± | 0.0144 |
| - nutrition | 1 | none | 5 | acc | ↑ | 0.7680 | ± | 0.0242 |
| - professional_accounting | 1 | none | 5 | acc | ↑ | 0.5532 | ± | 0.0297 |
| - professional_medicine | 1 | none | 5 | acc | ↑ | 0.6471 | ± | 0.0290 |
| - virology | 1 | none | 5 | acc | ↑ | 0.5120 | ± | 0.0389 |
| - social sciences | 2 | none | acc | ↑ | 0.7735 | ± | 0.0075 | |
| - econometrics | 1 | none | 5 | acc | ↑ | 0.5877 | ± | 0.0463 |
| - high_school_geography | 1 | none | 5 | acc | ↑ | 0.7828 | ± | 0.0294 |
| - high_school_government_and_politics | 1 | none | 5 | acc | ↑ | 0.8756 | ± | 0.0238 |
| - high_school_macroeconomics | 1 | none | 5 | acc | ↑ | 0.7051 | ± | 0.0231 |
| - high_school_microeconomics | 1 | none | 5 | acc | ↑ | 0.7773 | ± | 0.0270 |
| - high_school_psychology | 1 | none | 5 | acc | ↑ | 0.8550 | ± | 0.0151 |
| - human_sexuality | 1 | none | 5 | acc | ↑ | 0.8092 | ± | 0.0345 |
| - professional_psychology | 1 | none | 5 | acc | ↑ | 0.7288 | ± | 0.0180 |
| - public_relations | 1 | none | 5 | acc | ↑ | 0.6909 | ± | 0.0443 |
| - security_studies | 1 | none | 5 | acc | ↑ | 0.7551 | ± | 0.0275 |
| - sociology | 1 | none | 5 | acc | ↑ | 0.8308 | ± | 0.0265 |
| - us_foreign_policy | 1 | none | 5 | acc | ↑ | 0.8300 | ± | 0.0378 |
| - stem | 2 | none | acc | ↑ | 0.6159 | ± | 0.0084 | |
| - abstract_algebra | 1 | none | 5 | acc | ↑ | 0.5000 | ± | 0.0503 |
| - anatomy | 1 | none | 5 | acc | ↑ | 0.6222 | ± | 0.0419 |
| - astronomy | 1 | none | 5 | acc | ↑ | 0.7500 | ± | 0.0352 |
| - college_biology | 1 | none | 5 | acc | ↑ | 0.7083 | ± | 0.0380 |
| - college_chemistry | 1 | none | 5 | acc | ↑ | 0.4700 | ± | 0.0502 |
| - college_computer_science | 1 | none | 5 | acc | ↑ | 0.6200 | ± | 0.0488 |
| - college_mathematics | 1 | none | 5 | acc | ↑ | 0.4000 | ± | 0.0492 |
| - college_physics | 1 | none | 5 | acc | ↑ | 0.4902 | ± | 0.0497 |
| - computer_security | 1 | none | 5 | acc | ↑ | 0.8200 | ± | 0.0386 |
| - conceptual_physics | 1 | none | 5 | acc | ↑ | 0.6383 | ± | 0.0314 |
| - electrical_engineering | 1 | none | 5 | acc | ↑ | 0.6483 | ± | 0.0398 |
| - elementary_mathematics | 1 | none | 5 | acc | ↑ | 0.5820 | ± | 0.0254 |
| - high_school_biology | 1 | none | 5 | acc | ↑ | 0.8161 | ± | 0.0220 |
| - high_school_chemistry | 1 | none | 5 | acc | ↑ | 0.6059 | ± | 0.0344 |
| - high_school_computer_science | 1 | none | 5 | acc | ↑ | 0.7500 | ± | 0.0435 |
| - high_school_mathematics | 1 | none | 5 | acc | ↑ | 0.4926 | ± | 0.0305 |
| - high_school_physics | 1 | none | 5 | acc | ↑ | 0.4702 | ± | 0.0408 |
| - high_school_statistics | 1 | none | 5 | acc | ↑ | 0.6343 | ± | 0.0328 |
| - machine_learning | 1 | none | 5 | acc | ↑ | 0.4911 | ± | 0.0475 |
| Groups | Version | Filter | n-shot | Metric | Value | Stderr | ||
|---|---|---|---|---|---|---|---|---|
| mmlu | 2 | none | acc | ↑ | 0.6629 | ± | 0.0038 | |
| - humanities | 2 | none | acc | ↑ | 0.5862 | ± | 0.0067 | |
| - other | 2 | none | acc | ↑ | 0.7171 | ± | 0.0078 | |
| - social sciences | 2 | none | acc | ↑ | 0.7735 | ± | 0.0075 | |
| - stem | 2 | none | acc | ↑ | 0.6159 | ± | 0.0084 |