transformers pytorch llama text-generation medical en dataset:xzuyn/chatdoctor-200k-stripped dataset:Technoculture/riddle_sense dataset:axiong/pmc_llama_instructions dataset:Open-Orca/SlimOrca-Dedup autotrain_compatible text-generation-inference endpoints_compatible region:us
---
datasets:
- xzuyn/chatdoctor-200k-stripped
- Technoculture/riddle_sense
- axiong/pmc_llama_instructions
- Open-Orca/SlimOrca-Dedup
language:
- en
tags:
- medical
---


![image/png](https://cdn-uploads.huggingface.co/production/uploads/63486df1f8f01fcc4b23e97d/nMuS3Qnb5m0dENIixWv0q.png)

[Technoculture/MT7Bi-alpha](https://huggingface.co/Technoculture/MT7Bi-alpha) adapter merged with its Base Model (Meditron 7B)

# Evaluations

## Open LLM Leaderboard
|                       Model                       | ARC |HellaSwag|TruthfulQA|Winogrande|GSM8K|
|---------------------------------------------------|----:|--------:|---------:|---------:|----:|
|[MT7Bi-sft (epoch 4)](https://huggingface.co/Technoculture/MT7Bi-sft)|54.1|    75.11| 43.08|     72.14|15.54|
|[MT7Bi-sft (epoch 1)](https://huggingface.co/Technoculture/MT7Bi)|50.94|    73.24| 43.04|     72.06|22.52|

### Model Evaluation Benchmark

|          |        |      |      |      |      |      |      |       |
| -------- | ------ |----- |----- |----- |----- |----- |----- |------ |
|Category  | MT7Bi  | meditron-70b | llama-2-70b | med42-70b* | meditron-7b | llama-2-7b | PMC-llama-7b |
|Health    |  |      81.8    |    69.1     |    83.6    | 27.3        |   16.4     |      3.6     |
|Nutrition |  |      77.9    |    68.8     |    62.5    | 31.1        |   12.5     |      6.3     |
|Psychology|  |      47.4    |    36.8     |    52.6    | 21.1        |   10.5     |      0.0     |
|Science   |  |      77.8    |    44.4     |    33.3    | 33.3        |   11.1     |      0.0     |
|Avg       |  |      71.2    |    54.8     |    58.0    | 28.3        |   12.6     |      2.5     |
|          |  |              |             |            |             |            |              |


|     |        |        |      |      |      |      |
| --- | ------ | ------ |----- |----- |----- |----- |
|Dataset| MT7Bi |     meditron-70b | llama-2-70b | med42-70b* | clinical-camel-70b* |
|MMLU-Medical | 46.9 | 77.6       |    77.9     | 74.5      | 65.7               |
|PubMedQA     | 65.2 | 81.6       |    80.0     | 61.2      | 67.0               |
|MedMCQA      | 42.7 | 66.0       |    62.6     | 59.2      | 46.7               |
|MedQA        |  | 64.4       |    61.5     | 59.1      | 50.8               |
|MedQA-4-Option| 44.3 | 70.2      |    63.8     | 63.9      | 56.8               |
|Avg          |  | 72.0       |    69.2     | 63.6      | 57.4               |
|     |        |  |      |      |      |      |

|     |        |      |      |      |      |       |
| --- | ------ |----- |----- |----- |----- |------ |
|Dataset       | meditron-7b | llama-2-7b  | pmc-llama-7b | Zephyr-7B-beta* | Mistral-7B-instruct* | MT7Bi |
|MMLU-Medical  | 54.2        |    53.7     |    56.4      |      63.3       |        60.0          | 46.9  |
|PubMedQA      | 74.4        |    61.8     |    59.2      |      46.0       |        17.8          | 65.2  |
|MedMCQA       | 59.2        |    54.4     |    57.6      |      43.0       |        40.2          | 42.7  |
|MedQA         | 47.9        |    44.0     |    42.4      |      42.8       |        32.4          |  |
|MedQA-4-Option| 52.0        |    49.6     |    49.2      |      48.5       |        41.1          | 44.3 |
|Avg           | 57.5        |    52.7     |    53.0      |      48.7       |        38.3          |  |
|              |             |             |              |                 |                      |  |


| Model Name         | ARC      | HellaSwag | MMLU | TruthfulQA | Winogrande | GSM8K    |
| ------------------ | -------- | --------- | ---- | ---------- | ---------- | -------- |
| Orca-2-7b          | **78.4** | 76.1      | 53.7 | **52.4**   | **74.2**   | **47.2** |
| LLAMA-2-7b         | 43.2     | **77.1**  | 44.4 | 38.7       | 69.5       | 16       |
| MT7Bi-sft    | 54.1    | 75.11     | -    | 43.08      | 72.14      | 15.54    |

### ARC: 54.1%
|    Task     |Version|       Metric       |    Value    |   |Stderr|
|-------------|------:|--------------------|-------------|---|------|
|arc_challenge|      1|acc,none            |         0.51|   |      |
|             |       |acc_stderr,none     |         0.01|   |      |
|             |       |acc_norm,none       |         0.54|   |      |
|             |       |acc_norm_stderr,none|         0.01|   |      |
|             |       |alias               |arc_challenge|   |      |

### HellaSwag: 75.11%
|  Task   |Version|       Metric       |  Value  |   |Stderr|
|---------|------:|--------------------|---------|---|------|
|hellaswag|      1|acc,none            |     0.57|   |      |
|         |       |acc_stderr,none     |        0|   |      |
|         |       |acc_norm,none       |     0.75|   |      |
|         |       |acc_norm_stderr,none|        0|   |      |
|         |       |alias               |hellaswag|   |      |

### TruthfulQA: 43.08%
|     Task     |Version|        Metric         |      Value      |   |Stderr|
|--------------|-------|-----------------------|-----------------|---|------|
|truthfulqa    |N/A    |bleu_max,none          |            18.31|   |      |
|              |       |bleu_max_stderr,none   |             0.46|   |      |
|              |       |bleu_acc,none          |             0.39|   |      |
|              |       |bleu_acc_stderr,none   |                0|   |      |
|              |       |bleu_diff,none         |            -1.63|   |      |
|              |       |bleu_diff_stderr,none  |             0.39|   |      |
|              |       |rouge1_max,none        |            41.99|   |      |
|              |       |rouge1_max_stderr,none |             0.71|   |      |
|              |       |rouge1_acc,none        |             0.39|   |      |
|              |       |rouge1_acc_stderr,none |                0|   |      |
|              |       |rouge1_diff,none       |            -2.88|   |      |
|              |       |rouge1_diff_stderr,none|             0.66|   |      |
|              |       |rouge2_max,none        |            27.42|   |      |
|              |       |rouge2_max_stderr,none |             0.80|   |      |
|              |       |rouge2_acc,none        |             0.32|   |      |
|              |       |rouge2_acc_stderr,none |                0|   |      |
|              |       |rouge2_diff,none       |            -3.11|   |      |
|              |       |rouge2_diff_stderr,none|             0.78|   |      |
|              |       |rougeL_max,none        |            38.81|   |      |
|              |       |rougeL_max_stderr,none |             0.71|   |      |
|              |       |rougeL_acc,none        |             0.38|   |      |
|              |       |rougeL_acc_stderr,none |                0|   |      |
|              |       |rougeL_diff,none       |            -3.01|   |      |
|              |       |rougeL_diff_stderr,none|             0.66|   |      |
|              |       |acc,none               |             0.33|   |      |
|              |       |acc_stderr,none        |             0.05|   |      |
|              |       |alias                  |truthfulqa       |   |      |
|truthfulqa_gen|      3|bleu_max,none          |            18.31|   |      |
|              |       |bleu_max_stderr,none   |             0.68|   |      |
|              |       |bleu_acc,none          |             0.39|   |      |
|              |       |bleu_acc_stderr,none   |             0.02|   |      |
|              |       |bleu_diff,none         |            -1.63|   |      |
|              |       |bleu_diff_stderr,none  |             0.62|   |      |
|              |       |rouge1_max,none        |            41.99|   |      |
|              |       |rouge1_max_stderr,none |             0.84|   |      |
|              |       |rouge1_acc,none        |             0.39|   |      |
|              |       |rouge1_acc_stderr,none |             0.02|   |      |
|              |       |rouge1_diff,none       |            -2.88|   |      |
|              |       |rouge1_diff_stderr,none|             0.81|   |      |
|              |       |rouge2_max,none        |            27.42|   |      |
|              |       |rouge2_max_stderr,none |             0.89|   |      |
|              |       |rouge2_acc,none        |             0.32|   |      |
|              |       |rouge2_acc_stderr,none |             0.02|   |      |
|              |       |rouge2_diff,none       |            -3.11|   |      |
|              |       |rouge2_diff_stderr,none|             0.88|   |      |
|              |       |rougeL_max,none        |            38.81|   |      |
|              |       |rougeL_max_stderr,none |             0.84|   |      |
|              |       |rougeL_acc,none        |             0.38|   |      |
|              |       |rougeL_acc_stderr,none |             0.02|   |      |
|              |       |rougeL_diff,none       |            -3.01|   |      |
|              |       |rougeL_diff_stderr,none|             0.82|   |      |
|              |       |alias                  | - truthfulqa_gen|   |      |
|truthfulqa_mc1|      2|acc,none               |             0.28|   |      |
|              |       |acc_stderr,none        |             0.02|   |      |
|              |       |alias                  | - truthfulqa_mc1|   |      |
|truthfulqa_mc2|      2|acc,none               |             0.43|   |      |
|              |       |acc_stderr,none        |             0.01|   |      |
|              |       |alias                  | - truthfulqa_mc2|   |      |

### Winogrande: 72.14%
|   Task   |Version|    Metric     |  Value   |   |Stderr|
|----------|------:|---------------|----------|---|------|
|winogrande|      1|acc,none       |      0.72|   |      |
|          |       |acc_stderr,none|      0.01|   |      |
|          |       |alias          |winogrande|   |      |

### GSM8K: 15.54%
|Task |Version|           Metric            |Value|   |Stderr|
|-----|------:|-----------------------------|-----|---|------|
|gsm8k|      2|exact_match,get-answer       | 0.16|   |      |
|     |       |exact_match_stderr,get-answer| 0.01|   |      |
|     |       |alias                        |gsm8k|   |      |

Elapsed time: 04:06:36