transformers pytorch pegasus text2text-generation generated_from_trainer autotrain_compatible endpoints_compatible region:us
---
metrics:
- rouge
tags:
- generated_from_trainer
widget:
- text: 'PREOPERATIVE DIAGNOSIS: Chronic obstructive pulmonary disease (COPD).POSTOPERATIVE
    DIAGNOSIS: COPD.PROCEDURE: Bilateral video-assisted thoracoscopic lung volume
    reduction surgery (LVRS).ANESTHESIA: General anesthesia with single-lumen endotracheal
    tube.INDICATIONS FOR PROCEDURE: This 65-year-old female patient presented with
    severe COPD symptoms, including dyspnea and decreased exercise tolerance. After
    thorough evaluation and discussions of available treatment options, the decision
    for bilateral LVRS was made in order to improve lung function and quality of life.PROCEDURE
    IN DETAIL: Informed consent was obtained after explaining the risks and benefits
    of the procedure. The patient was placed in a lateral decubitus position, and
    general anesthesia was induced. Bilateral LVRS was performed using video-assisted
    thoracoscopic techniques. Intraoperatively, attention was given to minimize bleeding
    and ensure proper lung tissue removal. The patient tolerated the procedure well,
    and postoperative care instructions were provided.'
  example_title: Example 1
- text: 'PREOPERATIVE DIAGNOSIS: Coronary artery disease.POSTOPERATIVE DIAGNOSIS:
    Coronary artery disease.PROCEDURE: Coronary artery bypass grafting (CABG) surgery.ANESTHESIA:
    General anesthesia with cardiopulmonary bypass.INDICATIONS FOR PROCEDURE: This
    60-year-old male patient presented with significant coronary artery disease, with
    multiple vessels showing significant stenosis on angiography. After a thorough
    evaluation of his condition and considering the extent of the disease, the decision
    was made to proceed with CABG surgery to improve blood flow to the heart muscle.PROCEDURE
    IN DETAIL: After obtaining informed consent and ensuring adequate preoperative
    preparations, the patient was brought to the operating room. General anesthesia
    was induced, and cardiopulmonary bypass was established. The bypass grafts were
    harvested, and the stenotic coronary arteries were bypassed using appropriate
    grafts. Hemostasis was ensured, and the patient was weaned off cardiopulmonary
    bypass. The patient was transferred to the intensive care unit for postoperative
    monitoring and recovery. Postoperative care instructions were provided to the
    patient and family members.'
  example_title: Example 2
- text: 'PREOPERATIVE DIAGNOSIS: Lumbar disc herniation.POSTOPERATIVE DIAGNOSIS: Lumbar
    disc herniation.PROCEDURE: Minimally invasive lumbar microdiscectomy.ANESTHESIA:
    General anesthesia with endotracheal intubation.INDICATIONS FOR PROCEDURE: This
    42-year-old male patient presented with radiating low back pain and leg numbness,
    along with positive imaging findings of a lumbar disc herniation. After conservative
    treatment failed to provide relief, the decision was made to proceed with a minimally
    invasive microdiscectomy to alleviate the symptoms.PROCEDURE IN DETAIL: The patient
    was positioned prone on the operating table, and general anesthesia was administered.
    A small incision was made, and using fluoroscopic guidance, the herniated disc
    material was carefully removed. The surgical site was inspected for any bleeding
    or complications before closure. The patient was awakened from anesthesia without
    any immediate postoperative complications. Postoperative instructions were given
    regarding activity restrictions and pain management.'
  example_title: Example 3
model-index:
- name: PegasusMedicalSummary
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# PegasusMedicalSummary

### Authors
This model was created by [mereshd](https://huggingface.co/mereshd), [renegarza](https://huggingface.co/renegarza) and [jasmeeetsingh](https://huggingface.co/jasmeeetsingh).

This model is a fine-tuned version of [google/pegasus-xsum](https://huggingface.co/google/pegasus-xsum) on the [MTSamples](https://mtsamples.com/) dataset.
It achieves the following results on the evaluation set:
- Loss: 0.1438
- Rouge1: 0.4318
- Rouge2: 0.2525
- Rougel: 0.3524
- Rougelsum: 0.3525
- Gen Len: 55.882

### Project Purpose

Our goal is to deliver an effective summarization solution aimed at making doctor discharge notes more structured and comprehensive.
A physician's job goes far beyond saving lives, doctors are also responsible for providing a comforting environment for their patients.
With that in mind, while accommodating in a high-stress environment it is difficult to follow a structure and formulate notes with universal interpretability in mind.
This leads to long and convoluted discharge documentation that becomes very tedious to leverage and understand. Our model is a product that will alleviate 
a significant amount of discomfort when creating and utilizing physician notes, which ultimately will lead to more fluid workflows and increased convenience for healthcare providers.

### Intended Use
#### Model
We leveraged Google's Pegasus abstractive text summarization to generate summaries of the discharged transcriptions included in the MTSamples dataset. 
This was later utilized to prompt the Transformer's Masked Language Modeling(MLM) functionality to train the model to generate meaningful text with better structure and organization than the original.

#### Use Cases
This model allows for the efficient summarization of complexly documented doctor notes. It provides instant access to insight with proper semantic cues in place. Additionally, Data Engineers that work with patient electronic records consistently spend an excessive amount of time parsing through the unstructured discharge notes format to accomplish their tasks.
The solution will be instrumental for agents who are not directly facing patients but hold back-end roles that are also of immense importance.

#### Limitations & Future Aspirations
With an increased amount of data, more deliberate results might be achieved through more training. Synthetic transcriptions could be created with GPT models to in turn train on. Also, further improvements on the model's summarization capabilities have been considered.
One of which is implementing summarization based on clustered titles within the discharge notes. The feature would allow for easier traversal through partitioned summarization and result in better structure.

#### Training and evaluation data
The generated summaries were assigned to the original transcription and after splitting the data into the train and test sets, the table was converted into a json file.
The structure allowed us to effectively train the model on the premise of transcription to summarization prompts. After all the metrics were evaluated, a number of medical transcriptions were generated through
generative transformers to summarize and upon testing the model performed well.  

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 1
- eval_batch_size: 1
- seed: 42
- gradient_accumulation_steps: 4
- total_train_batch_size: 4
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 4
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch | Step | Validation Loss | Rouge1 | Rouge2 | Rougel | Rougelsum | Gen Len |
|:-------------:|:-----:|:----:|:---------------:|:------:|:------:|:------:|:---------:|:-------:|
| 6.5172        | 1.0   | 999  | 0.1784          | 0.4161 | 0.2373 | 0.3388 | 0.3384    | 52.102  |
| 0.3174        | 2.0   | 1999 | 0.1550          | 0.4236 | 0.2434 | 0.343  | 0.3428    | 54.458  |
| 0.2632        | 3.0   | 2999 | 0.1462          | 0.4269 | 0.2467 | 0.3465 | 0.3464    | 55.503  |
| 0.2477        | 4.0   | 3996 | 0.1438          | 0.4318 | 0.2525 | 0.3524 | 0.3525    | 55.882  |


### Framework versions

- Transformers 4.28.1
- Pytorch 2.0.0+cu117
- Datasets 2.11.0
- Tokenizers 0.13.3
