keras gpt2 medical biology text-generation en license:apache-2.0 region:us
---
language:
- en
library_name: keras
license: apache-2.0
pipeline_tag: text-generation
tags:
- medical
- biology
---


## GPT-2-LoRA-HealthCare

**Original Author**: [BastienHot](https://github.com/BastienHot) [ZhanPascal](https://github.com/ZhanPascal)<br>
**Date created**: 2024/02/25 <br>
**Dataset**: 
<ul>
    <li><a href="https://www.kaggle.com/datasets/jpmiller/layoutlm">MedQuAD from Kaggle</a></li>
    <li><a href="https://pubmedqa.github.io/">PubMedQA</a></li>
    <li><a href="http://participants-area.bioasq.org/datasets/">BioASQ – Task B</a></li>
    <li><a href="https://quac.ai/">QuAC</a></li>
    <li><a href="https://rajpurkar.github.io/SQuAD-explorer/">SQuAD</a></li>
</ul>

## [Spaces Demo](https://huggingface.co/spaces/FFatih/SAE-GPT2-PROD)

## Description : GPT-2-LoRA-HealthCare

The GPT-2-LoRA-HealthCare model was developed as part of a student project during the Bachelor of Technology (BUT) in Computer Science at IUT Villetaneuse. It is based on the pre-trained model from keras_nlp (gpt2-large-en), with the incorporation of the LoRA technique to enhance training efficiency. The model is specifically designed for Q&A interactions in a healthcare context, with a patient asking a question and the model responding with an appropriate answer.

LoRA is a training technique used for large language models (LLM) to train them more efficiently and with a less time consuming approach. How it works is the following:

<ul>
  <li>Look at the pretrained model's weights</li>
  <li>Determine the linearly independent and dependent columns of the matrix</li>
  <li>Create the new LoRA layer with only the linearly independent matrix columns</li>
  <li>Freeze the pretrained model's weights</li>
  <li>Train the LoRA layer weights</li>
  <li>Merge the weights of the pretrained model and the LoRA layer</li>
</ul>