PubMedBERT-MNLI-MedNLI

This model is a fine-tuned version of PubMedBERT on the MNLI dataset first and then on the MedNLI dataset. It achieves the following results on the evaluation set:

• Loss: 0.9501
• Accuracy: 0.8667

Model description

More information needed

Intended uses & limitations

The model can be used for NLI tasks related to biomedical data and even be adapted to fact-checking tasks. It can be used from the Huggingface pipeline method as follows:

from transformers import TextClassificationPipeline, AutoModel, AutoTokenizer, AutoModelForSequenceClassification
model = AutoModelForSequenceClassification.from_pretrained("pritamdeka/PubMedBERT-MNLI-MedNLI", num_labels=3, id2label = {1: 'entailment', 0: 'contradiction',2:'neutral'})
tokenizer = AutoTokenizer.from_pretrained("pritamdeka/PubMedBERT-MNLI-MedNLI")
pipe = TextClassificationPipeline(model=model, tokenizer=tokenizer, return_all_scores=True, device=0, batch_size=128)

pipe(['ALDH1 expression is associated with better breast cancer outcomes',
      'In a series of 577 breast carcinomas, expression of ALDH1 detected by immunostaining correlated with poor prognosis.'])

The output for the above will be:

[[{'label': 'contradiction', 'score': 0.10193759202957153},
  {'label': 'entailment', 'score': 0.2933262586593628},
  {'label': 'neutral', 'score': 0.6047361493110657}],
 [{'label': 'contradiction', 'score': 0.21726925671100616},
  {'label': 'entailment', 'score': 0.24485822021961212},
  {'label': 'neutral', 'score': 0.5378724932670593}]]

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 32
• eval_batch_size: 32
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 20.0

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
0.5673	1.42	500	0.4358	0.8437
0.2898	2.85	1000	0.4845	0.8523
0.1669	4.27	1500	0.6233	0.8573
0.1087	5.7	2000	0.7263	0.8573
0.0728	7.12	2500	0.8841	0.8638
0.0512	8.55	3000	0.9501	0.8667
0.0372	9.97	3500	1.0440	0.8566
0.0262	11.4	4000	1.0770	0.8609
0.0243	12.82	4500	1.0931	0.8616
0.023	14.25	5000	1.1088	0.8631
0.0163	15.67	5500	1.1264	0.8581
0.0111	17.09	6000	1.1541	0.8616
0.0098	18.52	6500	1.1542	0.8631
0.0074	19.94	7000	1.1653	0.8638

Framework versions

• Transformers 4.22.0.dev0
• Pytorch 1.12.1+cu113
• Datasets 2.4.0
• Tokenizers 0.12.1

Citing & Authors

If you use the model kindly cite the following work

@inproceedings{deka-etal-2023-multiple,
    title = "Multiple Evidence Combination for Fact-Checking of Health-Related Information",
    author = "Deka, Pritam  and
      Jurek-Loughrey, Anna  and
      P, Deepak",
    booktitle = "The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.bionlp-1.20",
    pages = "237--247",
    abstract = "Fact-checking of health-related claims has become necessary in this digital age, where any information posted online is easily available to everyone. The most effective way to verify such claims is by using evidences obtained from reliable sources of medical knowledge, such as PubMed. Recent advances in the field of NLP have helped automate such fact-checking tasks. In this work, we propose a domain-specific BERT-based model using a transfer learning approach for the task of predicting the veracity of claim-evidence pairs for the verification of health-related facts. We also improvise on a method to combine multiple evidences retrieved for a single claim, taking into consideration conflicting evidences as well. We also show how our model can be exploited when labelled data is available and how back-translation can be used to augment data when there is data scarcity.",
}