transformers pytorch llama text-generation en license:other autotrain_compatible text-generation-inference endpoints_compatible region:us
---
language:
- en
library_name: transformers
license: other
metrics:
- accuracy 85%
---

# Asclepius
#### Version: Initial (v1)
***

## Overview
Asclepius is a state-of-the-art Language Model developed with a special focus on healthcare queries. 
It is as LLama-2-70b based model fine-tuned on a broad range of publicly available healthcare data. It can render timely and consistent answers to health-related queries with good accuracy.

## Setup Guide & Usage
The model is available on this huggingface repository. It supports up to 4-bit quantization with sustained performance.
To use the model you can use the usual HuggingFace Transformers API, like this:
```python
from transformers import AutoTokenizer
import transformers
import torch

model = "elbonaparte/asclepius_v1"

tokenizer = AutoTokenizer.from_pretrained(model)
pipeline = transformers.pipeline(
    "text-generation",
    model=model,
    torch_dtype=torch.float16,
    device_map="auto",
)

sequences = pipeline(
    'Question: What is the most common cause of chest pain in men < 50? Answer:\n',
    do_sample=True,
    top_k=10,
    num_return_sequences=1,
    eos_token_id=tokenizer.eos_token_id,
    max_length=200,
)
for seq in sequences:
    print(f"Result: {seq['generated_text']}")
```
The output would be:
```
The most common causes of chest pain in men < 50 are:
1. Gastroesophageal reflux disease (GERD): This occurs when stomach acid flows back into the esophagus, causing a burning sensation in the chest (heartburn).

2. Musculoskeletal issues: Strained muscles or inflammation in the chest wall can cause chest pain, especially during physical activity or after an injury.

3. Anxiety and stress-related chest pain: Anxiety or panic attacks can cause chest tightness and discomfort, mimicking heart-related pain.

4. Costochondritis: Inflammation of the cartilage that connects the ribs to the breastbone can cause sharp chest pain, often aggravated by physical activity or deep breathing.
```
Although this is plain, non-structured text, several prompt strategies can be used to obtain structured text as lists or JSON format from the LLM.

In the `asclepius_api_helpers.py` we provide several prompt-optimized functions for returning structured responses for the most common use cases: medications list, most/least common findings, diagnostic hypothesis, etc.

These functions can be used to create a server tasked with querying the model (API gateway) or integrated within the application (example provided on https://asclepiusv1.streamlit.app/).

#### Inference server
Alternatively, a RunPod image template is available pre-configured for text-generation-inference allowing you to get the model up and running in a few clicks following these steps:

1. Choose an instance to deploy. This model needs a GPU with at least 48gb of VRAM.
![alt text for screen readers](assets/rpstep1.png "Text to show on mouseover")

2. Choose the provided "Asclepius Template" image template from the dropdown list. It should be the first. Runpod will provision the VM and setup is done automatically.
It takes around 15 to 20 minutes to download the model.
![alt text for screen readers](assets/rpstep2.png "Text to show on mouseover")

3. The model server will be available 15 to 20 minutes after that. The URL will look like this: `[POD ID]-80.proxy.runpod.net`.
The POD ID is the string of text under the Pod Name in the top left.
![alt text for screen readers](assets/rpstep4.png "Text to show on mouseover")

4. A easy-to-use UI for interacting with Asclepius is available over https://asclepiusv1.streamlit.app. Change the Pod ID to the runnning POD ID and it will return the answers from that model.
![alt text for screen readers](assets/rpstep5.png "Text to show on mouseover")

The UI code is also accessible in this repository. It can be utilized with the installation of [streamlit.io](https://docs.streamlit.io/library/get-started/installation) and by running this command in the terminal inside the repo folder: `streamlit run app.py`.

The example app and use cases interact optimally with the model using input prompts engineered within the functions found in the `asclepius_api_helpers.py` file.
## Training Data
An extensive exploration of publicly available healthcare information datasets was conducted and these were considered as potential inputs:

| Dataset Name                               | URL                                            | Description                                                    | Availability                                   | Type             | Data Quality | Data Preparation Effort |
| :----------------------------------------- | :--------------------------------------------- | :------------------------------------------------------------- | :--------------------------------------------- | :--------------- | :----------- | :---------------------- |
| icliniq-10k                                | icliniq/medical_dialog                         | conversation between patient and doctors                       | Easily accessible                              | Real world data  | 7            | 6                       |
| HealthCareMagic-100k                       | HMC/patient_doctor_convo                       | conversation between patient and doctors                       | Easily Accessible                              | Real world data  | 7            | 6                       |
| Medical Dialog                             | hf/medical_dialog                              | conversation between patient and doctors                       | Easily accessible                              | Real world data  | 8            | 5                       |
| Medical Notes 40                           | rungalileo/medical_transcription_4             | Hospitalist Notes (PreOp, procedure, discharge summaries, etc) | Easily accessible                              | Real World Data  | 9            | 5                       |
| (MIMIC) Indiana University Medical Reports | Indiana_University_Medical_reports_original    | Radiologic Reports and Clinical notes                          | Easily accessible                              | Real World Data  | 9            | 5                       |
| Medical Domain                             | argilla/medical-domain                         | Clinical Notes                                                 | Easily accessible                              | Real World Data  | 8            | 5                       |
| Medical Keyword                            | argilla/medical-keywords                       | Clinical Notes                                                 | Easily accessible                              | Real World Data  | 8            | 5                       |
| Medical QA                                 | eswardivi/medical_qa                           | Question answering to patient doubts                           | Easily accessible                              | Real World Data  | 9            | 4                       |
| Medical Transcriptions                     | tchebonenko/MedicalTranscriptions              | Clinical Notes                                                 | Easily accessible                              | Real World Data  | 7            | 6                       |
| Synthea                                    | synthea.mitre.org                              | Synthetic health data generator                                | Easily accessible, graph-based logic necessary | Synthetic Data   | 8            | 10                      |
| MIMIC IV 2.0                               |                                                | Real world medical texts                                       | Accessible after 14-modules online training    | Real World Data  | 9            | 7                       |
|                                            |                                                |                                                                |                                                |                  |              |                         |
| Medline Plus                               | medlineplus.gov                                | Curated Medical Information                                    | Needs web scrapping                            | Referential Data | 10           | 9                       |
| CDC                                        | cdc.gov                                        | Curated Medical Information                                    | Needs web scrapping                            | Referential Data | 10           | 9                       |
| National Institutes of Health              | nih.gov                                        | Curated Medical Information                                    | Needs web scrapping                            | Referential Data | 10           | 9                       |
| WHO                                        | who.int                                        | Curated Medical Information                                    | Needs web scrapping                            | Referential Data | 10           | 9                       |
| Mayo Clinic                                | mayoclinic.org                                 | Curated Medical Information                                    | Needs web scrapping                            | Referential Data | 10           | 9                       |
| Merck Manual                               | merckmanuals.com/professional                  | Curated Medical Information                                    | Web scrapping unavailable                      | Referential Data | 10           | 9                       |
| Open Medical Terms                         | gamino/wiki_medical_terms                      | Explanation of medical terms                                   | Easily accessible                              | Referential Data | 8            | 4                       |
| MedQA                                      | medalpaca/medical_meadow_medqa                 | Medical question answering (USMLE)                             | Easily accessible                              | Referential Data | 10           | 4                       |
| USMLE SA                                   | medalpaca/medical_meadow_usmle_self_assessment | USMLE self-assesment questions and answers                     | Easily accessible                              | Referential Data | 10           | 5                       |
| PubMed Health Advice                       | medalpaca/medical_meadow_health_advice         | Extracted data from Pubmed articles                            | Easily accessible                              | Referential Data | 8            | 6                       |
| Wikidoc explanations                       | medalpaca/medical_meadow_wikidoc               | Explanation of medical conditions and procedures               | Easily accesible                               | Referential Data | 10           | 6                       |
| Medical Flashcard                          | medalpaca/medical_meadow_medical_flashcards    | General Healthcare questions and answers                       | Easily accessible                              | Referential Data | 8            | 6                       |
| Pubmed Causal                              | medalpaca/medical_meadow_pubmed_causal         | Causality between health events                                | Easily accessible                              | Referential Data | 9            | 7                       |
| Medical Questions DB                       | fhirfly/medicalquestions                       | A dataset containing general health questions without answers  | Easily accessible                              | Referential Data | 8            | 9                       |


## Evaluation
The model evaluation is based on its consistency and accuracy in responding to healthcare inquiries.

#### - Custom QA dataset (20 questions)
(1-shot)
| Model Name            | Correct Answers  (%) | Total Questions |
| :-------------------- | :------------------- | :-------------- | 
| GPT-4                 |      18 (90%)        |     20          |
| *Asclepius* | 16 (80%) | 20 |
| Llama-2 | 15 (70%) | 20 |
| Falcon-40b | 11 (55%) | 20 |
| Others (GPT-J, T5, Graph-based) | < 8 | 20 |

#### - MMLU (Professional Medicine + Clinical Knowledge + College Medicine)
(5-shot)
| Model Name            | Score | 
| :-------------------- | :------------------- | 
| GPT-4                 |      88%        |
| MedPalm-2                |      88%        |
| *Asclepius* | 67.2% |
| GPT-3.5 | 67.2% |
| Llama-2 | 66.8% |

#### - MedQA-USMLE (1200 USMLE-style questions)
(5-shot)
| Model Name            | Score | 
| :-------------------- | :------------------- | 
| GPT-4                 |      86.1%        |
| MedPalm-2       |      79.7%        |
| *Asclepius* | 60.1% |
| Llama-2 | 58.9% |
| GPT-3.5 | 53.5% |


## Limitations
Despite its impressive capabilities in answering medical questions, Asclepius occasionally errs particularly in the section of clinical cases. Consequently, output from the model should not be used without the supervision of a professional healthcare practitioner. 

As a tool for physicians, it has immense potential in suggesting diagnoses and prescriptions, in addition to reviewing notes. Nevertheless, potential harm to patients due to erroneous output needs consideration and studies on outcomes should be undertaken before utilization.

This is in agreement with the FDA's recommendation on the usage of AI, ML and Software as a Medical Device (https://www.fda.gov/medical-devices/software-medical-device-samd/artificial-intelligence-and-machine-learning-software-medical-device)

## License
Asclepius has been developed by Leonardo Canela Almeida, an independent contractor at Phire Health LLC. All rights are reserved by Phire Health LLC.