|
|
--- |
|
|
library_name: transformers |
|
|
license: mit |
|
|
datasets: |
|
|
- ethicalabs/Kurtis-E1-SFT |
|
|
language: |
|
|
- en |
|
|
base_model: |
|
|
- Qwen/Qwen3-4B |
|
|
pipeline_tag: text-generation |
|
|
tags: |
|
|
- text-generation-inference |
|
|
--- |
|
|
|
|
|
# Model Card for ethicalabs/Kurtis-E1.1-Qwen3-4B |
|
|
|
|
|
|
|
|
Kurtis E1.1 fine-tuned with [flower](https://flower.ai/) |
|
|
|
|
|
## Eval Results |
|
|
|
|
|
Evaluation tasks were performed with the [LM Evaluation Harness](https://github.com/EleutherAI/lm-evaluation-harness) on a Mac Mini M4 Pro. |
|
|
|
|
|
### mmlu |
|
|
|
|
|
``` |
|
|
lm_eval --model hf --model_args pretrained=ethicalabs/Kurtis-E1.1-Qwen3-4B --tasks mmlu --device mps --batch_size 4 |
|
|
``` |
|
|
|
|
|
| Tasks |Version|Filter|n-shot|Metric| |Value | |Stderr| |
|
|
|---------------------------------------|------:|------|-----:|------|---|-----:|---|-----:| |
|
|
|mmlu | 2|none | |acc |↑ |0.6849|± |0.0037| |
|
|
| - humanities | 2|none | |acc |↑ |0.5951|± |0.0067| |
|
|
| - formal_logic | 1|none | 0|acc |↑ |0.5952|± |0.0439| |
|
|
| - high_school_european_history | 1|none | 0|acc |↑ |0.7879|± |0.0319| |
|
|
| - high_school_us_history | 1|none | 0|acc |↑ |0.8333|± |0.0262| |
|
|
| - high_school_world_history | 1|none | 0|acc |↑ |0.8439|± |0.0236| |
|
|
| - international_law | 1|none | 0|acc |↑ |0.7686|± |0.0385| |
|
|
| - jurisprudence | 1|none | 0|acc |↑ |0.7685|± |0.0408| |
|
|
| - logical_fallacies | 1|none | 0|acc |↑ |0.8037|± |0.0312| |
|
|
| - moral_disputes | 1|none | 0|acc |↑ |0.7081|± |0.0245| |
|
|
| - moral_scenarios | 1|none | 0|acc |↑ |0.3754|± |0.0162| |
|
|
| - philosophy | 1|none | 0|acc |↑ |0.7170|± |0.0256| |
|
|
| - prehistory | 1|none | 0|acc |↑ |0.7346|± |0.0246| |
|
|
| - professional_law | 1|none | 0|acc |↑ |0.4844|± |0.0128| |
|
|
| - world_religions | 1|none | 0|acc |↑ |0.7778|± |0.0319| |
|
|
| - other | 2|none | |acc |↑ |0.7161|± |0.0078| |
|
|
| - business_ethics | 1|none | 0|acc |↑ |0.7300|± |0.0446| |
|
|
| - clinical_knowledge | 1|none | 0|acc |↑ |0.7396|± |0.0270| |
|
|
| - college_medicine | 1|none | 0|acc |↑ |0.7168|± |0.0344| |
|
|
| - global_facts | 1|none | 0|acc |↑ |0.3300|± |0.0473| |
|
|
| - human_aging | 1|none | 0|acc |↑ |0.6771|± |0.0314| |
|
|
| - management | 1|none | 0|acc |↑ |0.8155|± |0.0384| |
|
|
| - marketing | 1|none | 0|acc |↑ |0.8675|± |0.0222| |
|
|
| - medical_genetics | 1|none | 0|acc |↑ |0.7600|± |0.0429| |
|
|
| - miscellaneous | 1|none | 0|acc |↑ |0.8008|± |0.0143| |
|
|
| - nutrition | 1|none | 0|acc |↑ |0.7255|± |0.0256| |
|
|
| - professional_accounting | 1|none | 0|acc |↑ |0.5390|± |0.0297| |
|
|
| - professional_medicine | 1|none | 0|acc |↑ |0.7390|± |0.0267| |
|
|
| - virology | 1|none | 0|acc |↑ |0.5000|± |0.0389| |
|
|
| - social sciences | 2|none | |acc |↑ |0.7813|± |0.0074| |
|
|
| - econometrics | 1|none | 0|acc |↑ |0.6228|± |0.0456| |
|
|
| - high_school_geography | 1|none | 0|acc |↑ |0.8283|± |0.0269| |
|
|
| - high_school_government_and_politics| 1|none | 0|acc |↑ |0.8756|± |0.0238| |
|
|
| - high_school_macroeconomics | 1|none | 0|acc |↑ |0.7590|± |0.0217| |
|
|
| - high_school_microeconomics | 1|none | 0|acc |↑ |0.8151|± |0.0252| |
|
|
| - high_school_psychology | 1|none | 0|acc |↑ |0.8679|± |0.0145| |
|
|
| - human_sexuality | 1|none | 0|acc |↑ |0.7405|± |0.0384| |
|
|
| - professional_psychology | 1|none | 0|acc |↑ |0.7173|± |0.0182| |
|
|
| - public_relations | 1|none | 0|acc |↑ |0.6818|± |0.0446| |
|
|
| - security_studies | 1|none | 0|acc |↑ |0.7265|± |0.0285| |
|
|
| - sociology | 1|none | 0|acc |↑ |0.8308|± |0.0265| |
|
|
| - us_foreign_policy | 1|none | 0|acc |↑ |0.8100|± |0.0394| |
|
|
| - stem | 2|none | |acc |↑ |0.6943|± |0.0079| |
|
|
| - abstract_algebra | 1|none | 0|acc |↑ |0.5700|± |0.0498| |
|
|
| - anatomy | 1|none | 0|acc |↑ |0.6370|± |0.0415| |
|
|
| - astronomy | 1|none | 0|acc |↑ |0.8092|± |0.0320| |
|
|
| - college_biology | 1|none | 0|acc |↑ |0.8333|± |0.0312| |
|
|
| - college_chemistry | 1|none | 0|acc |↑ |0.5400|± |0.0501| |
|
|
| - college_computer_science | 1|none | 0|acc |↑ |0.6600|± |0.0476| |
|
|
| - college_mathematics | 1|none | 0|acc |↑ |0.5700|± |0.0498| |
|
|
| - college_physics | 1|none | 0|acc |↑ |0.5784|± |0.0491| |
|
|
| - computer_security | 1|none | 0|acc |↑ |0.7800|± |0.0416| |
|
|
| - conceptual_physics | 1|none | 0|acc |↑ |0.7787|± |0.0271| |
|
|
| - electrical_engineering | 1|none | 0|acc |↑ |0.7586|± |0.0357| |
|
|
| - elementary_mathematics | 1|none | 0|acc |↑ |0.6878|± |0.0239| |
|
|
| - high_school_biology | 1|none | 0|acc |↑ |0.8742|± |0.0189| |
|
|
| - high_school_chemistry | 1|none | 0|acc |↑ |0.7192|± |0.0316| |
|
|
| - high_school_computer_science | 1|none | 0|acc |↑ |0.8500|± |0.0359| |
|
|
| - high_school_mathematics | 1|none | 0|acc |↑ |0.4741|± |0.0304| |
|
|
| - high_school_physics | 1|none | 0|acc |↑ |0.6225|± |0.0396| |
|
|
| - high_school_statistics | 1|none | 0|acc |↑ |0.7083|± |0.0310| |
|
|
| - machine_learning | 1|none | 0|acc |↑ |0.5268|± |0.0474| |
|
|
|
|
|
| Groups |Version|Filter|n-shot|Metric| |Value | |Stderr| |
|
|
|------------------|------:|------|------|------|---|-----:|---|-----:| |
|
|
|mmlu | 2|none | |acc |↑ |0.6849|± |0.0037| |
|
|
| - humanities | 2|none | |acc |↑ |0.5951|± |0.0067| |
|
|
| - other | 2|none | |acc |↑ |0.7161|± |0.0078| |
|
|
| - social sciences| 2|none | |acc |↑ |0.7813|± |0.0074| |
|
|
| - stem | 2|none | |acc |↑ |0.6943|± |0.0079| |
