README.md

---
tags:
- autotrain
- text-classification
- protein-classification
- protbert
- antiobiotic-resistance
widget:
- text: I love AutoTrain
- text: M T L A L V G E K I D R N R F T G E K V E N S T F F N C D F S G A D L S G T E F I G C Q F Y D R E S Q K G C N F S R A N L K D A I F K S C D L S M A D F R N I N A L G I E I R H C R A Q G S D F R G A S F M N M I T T R T W F C S A Y I T N T N L S Y A N F S K V V L E K C E L W E N R W M G T Q V L G A T F S G S D L S G G E F S S F D W R A A N V T H C D L T N S E L G D L D I R G V D L Q G V K L D S Y Q A S L L L E R L G I A V M G
datasets:
- as-cle-bert/AMR-Gene-Families
pipeline_tag: text-classification
---

   <table>
   <tr>
   <td>
   <img src="https://img.shields.io/github/languages/top/AstraBert/resistML" alt="GitHub top language">
   </td>
   <td>
   <img src="https://img.shields.io/github/commit-activity/t/AstraBert/resistML" alt="GitHub commit activity">
   </td>
   <td>
   <img src="https://img.shields.io/badge/resistML-stable-green" alt="Static Badge">
   </td>
   <td>
   <img src="https://img.shields.io/badge/resistBERT-unstable-orange" alt="Static Badge">
   </td>
   <td>
   <img src="https://img.shields.io/badge/Release-v0.0.0-blue" alt="Static Badge">
   </td>
   </tr>
   </table>


# resistML

A tool for AMR gene family prediction, simple and ML-based. Please refer to [this GitHub repository](https://github.com/AstraBert/resistML).

## Training


### Data collection for training

Latest reference sequences release (Feb 2024) were downloaded from **CARD** (*The Comprehensive Antibiotic Resistance Database*). If you want to automatically download them too, use `this link <https://card.mcmaster.ca/latest/data>`_.

Protein sequences were mapped with their ARO indices to the corrresponding AMR gene families (see [this file](https://github.com/AstraBert/resistML/tree/main/data/aro_categories_index.tsv) for reference) and the 12 most common families were chosen to train resistML and resistBERT.

### Training procedures

#### resistML (stable)

resistML was trained starting from all the protein sequences retrieved beforehands, extracting their features in a [csv file](https://github.com/AstraBert/resistML/tree/main/data/proteinstats.tsv). 

Features were extracted through biopython `Bio.SeqUtils.ProtParam --> ProteinAnalysis` subclass, and they are (maiusc is for the header you can find in the csv):

- HIDROPHOBICITY score
- ISOELECTRIC point
- AROMATICity
- INSTABility
- MW (molar weight)
- HELIX,TURN,SHEET (percentage of these three secondary strcutures)
- MOL_EXT_RED,MOL_EXT_OX (molar extinction reduced and oxidized)

Dataset building occured [here](https://github.com/AstraBert/resistML/tree/main/scripts/build_base_dataset.py)

The base model itself is a simple Voting Classifier based on a DecisionTreeClassifier, ExtraTreesClassifier and HistGradientBoostingClassifier, all provided by scikit-learn library.

During validation, it yielded 100% accuracy on predicting training data.

#### resistBERT (unstable)


resistBERT is a BERT model for text classification, finetuned from [prot_bert](https://huggingface.co/Rostlab/prot_bert) by RosettaLab.

Data using from finetuning were a selection of 1496 sequences out of the total 1836 ones. 80% were used for training, 20% were used for validations.

Sequences were preprocessed and labelled [here](https://github.com/AstraBert/resistML/tree/main/scripts/build_base_dataset.py), then the complete jsonl file was reduced [here](https://github.com/AstraBert/resistML/tree/main/scripts/reduce_dataset.py) and uploaded to Huggingface under the identifier `as-cle-bert/AMR-Gene-Families` through [this script](https://github.com/AstraBert/resistML/tree/main/scripts/jsonl2hfdataset.py).

Finetuning occurred from the HF dataset thanks to AutoTrain: during validation, the model yielded the following stats:

- loss: 0.08235077559947968

- f1_macro: 0.986759581881533

- f1_micro: 0.99

- f1_weighted: 0.9899790940766551

- precision_macro: 0.9871615312791784

- precision_micro: 0.99

- precision_weighted: 0.9901213818860879

- recall_macro: 0.986574074074074

- recall_micro: 0.99

- recall_weighted: 0.99

- accuracy: 0.99

The model is now available on Huggingface under the identifier `as-cle-bert/resistBERT`. There is also a widget through which you can make inferences thanks to HF `Inference API`. Keep in mind that Inference API *can* be unstable, so downloading the model and using it from a local machine/cloud service would be preferable. 

## Testing


### Data retrieval for tests

Data were downloaded from **CARD** (*The Comprehensive Antibiotic Resistance Database*), as the annotations for the family names used to label training sequences were the same. 

For families "PDC beta-lactamase", "CTX-M beta-lactamase", "SHV beta-lactamase", "CMY beta-lactamase", sequences were downloaded after having searched the exact AMR gene family as in the labels used for training, through `Download sequences` method. In the downloading customization page, filters were set to `is_a` and `Protein`.

For all the other families, procedure was the same but customization filters were set to `is_a`, `structurally_homologous_to`, `evolutionary_variant_of` and `Protein` to increase the number of retrieved sequences.

### Test building


Test were built thanks to [this script](https://github.com/AstraBert/resistML/tree/main/scripts/build_tests.py). 

These are the test metadata:

**Metadata for test 0:**

- Protein statistics for resistML were saved in test/testfiles/test_0.csv
- Sequences and labels for resistBERT were saved in test/testfiles/test_0.jsonl
- 12 protein sequences were taken into account for 2 families
- Families taken into account were: quinolone resistance protein (qnr), CMY beta-lactamase

**Metadata for test 1:**

- Protein statistics for resistML were saved in test/testfiles/test_1.csv
- Sequences and labels for resistBERT were saved in test/testfiles/test_1.jsonl
- 11 protein sequences were taken into account for 2 families
- Families taken into account were: VIM beta-lactamase,IMP beta-lactamase

**Metadata for test 2:**

- Protein statistics for resistML were saved in test/testfiles/test_2.csv
- Sequences and labels for resistBERT were saved in test/testfiles/test_2.jsonl
- 13 protein sequences were taken into account for 2 families
- Families taken into account were: quinolone resistance protein (qnr),SHV beta-lactamase

**Metadata for test 3:**

- Protein statistics for resistML were saved in test/testfiles/test_3.csv
- Sequences and labels for resistBERT were saved in test/testfiles/test_3.jsonl
- 10 protein sequences were taken into account for 3 families
- Families taken into account were: quinolone resistance protein (qnr),VIM beta-lactamase,CMY beta-lactamase

**Metadata for test 4:**

- Protein statistics for resistML were saved in test/testfiles/test_4.csv
- Sequences and labels for resistBERT were saved in test/testfiles/test_4.jsonl
- 12 protein sequences were taken into account for 2 families
- Families taken into account were: CMY beta-lactamase,IMP beta-lactamase

**Metadata for test 5:**

- Protein statistics for resistML were saved in test/testfiles/test_5.csv
- Sequences and labels for resistBERT were saved in test/testfiles/test_5.jsonl
- 12 protein sequences were taken into account for 2 families
- Families taken into account were: VIM beta-lactamase,SHV beta-lactamase

**Metadata for test 6:**

- Protein statistics for resistML were saved in test/testfiles/test_6.csv
- Sequences and labels for resistBERT were saved in test/testfiles/test_6.jsonl
- 11 protein sequences were taken into account for 3 families
- Families taken into account were: PDC beta-lactamase,MCR phosphoethanolamine transferase,ACT beta-lactamase

**Metadata for test 7:**

- Protein statistics for resistML were saved in test/testfiles/test_7.csv
- Sequences and labels for resistBERT were saved in test/testfiles/test_7.jsonl
- 10 protein sequences were taken into account for 3 families
- Families taken into account were: MCR phosphoethanolamine transferase,CTX-M beta-lactamase,PDC beta-lactamase

**Metadata for test 8:**

- Protein statistics for resistML were saved in test/testfiles/test_8.csv
- Sequences and labels for resistBERT were saved in test/testfiles/test_8.jsonl
- 12 protein sequences were taken into account for 2 families
- Families taken into account were: ACT beta-lactamase,CMY beta-lactamase

**Metadata for test 9:**
- Protein statistics for resistML were saved in test/testfiles/test_9.csv
- Sequences and labels for resistBERT were saved in test/testfiles/test_9.jsonl
- 15 protein sequences were taken into account for 3 families
- Families taken into account were: quinolone resistance protein (qnr),SHV beta-lactamase,KPC beta-lactamase

All data can be found [here](http://github.com/AstraBert/resistML/tree/main/test), along with the seqences used to generate them.

### Test results

**resistML** yielded 100% accuracy, f1 score, recall score and precision score in all 10 tests.

**resistBERT** was more unstable:

- On test_0, test_2, test_4, test_6, test_7, test_8 and test_9 yielded 100% accuracy, f1 score, recall score and precision score
- On test_1 it yielded:
  1. Accuracy: 50%
  2. f1 score: 33%
  3. Precision: 25%
  4. Recall: 50%
- On test_3 it yielded 66.7% accuracy, f1 score, recall score and precision score
- On test_5 it yielded 50% accuracy, f1 score, recall score and precision score


All results for resistBERT can be found [in the dedicated notebook](http://github.com/AstraBert/resistML/scripts/test_resistBERT.ipynb) . 

## License and rights of usage 


The[ GitHub repository](http://github.com/AstraBert/resistML) is provided under MIT license (more at [LICENSE](https://github.com/AstraBert/resistML/tree/main/LICENSE)`).

If you use this work for your projects, please consider citing the author [Astra Bertelli](http://astrabert.vercel.app).

## References


1. **CARD - The Comprehensive Antibiotic Resistance Database**

2. **Biopython**

3. **Scikit-learn** 

4. **Hugging Face's prot_bert Model**

5. **Hugging Face's AutoTrain**

If you feel that your work was relevant in building resistML and you weren't referenced in this section, feel free to flag an issue on GitHub or to contact the author.