# A novel Sentence Transformer for DNA tasks

This is a [sentence-transformers](https://www.SBERT.net) model trained. It maps sentences & paragraphs to a 512-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

## Model Details

### Model Description
- **Model Type:** Sentence Transformer
<!-- - **Base model:** [Unknown](https://huggingface.co/unknown) -->
- **Maximum Sequence Length:** 512 tokens
- **Output Dimensionality:** 512 tokens
- **Similarity Function:** Cosine Similarity
<!-- - **Training Dataset:** Unknown -->
<!-- - **Language:** Unknown -->
<!-- - **License:** Unknown -->

### Model Sources

- **Documentation:** [Sentence Transformers Documentation](https://sbert.net)
- **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers)
- **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers)

### Full Model Architecture

```
SentenceTransformer(
  (0): SentenceTransformer(
    (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: MPNetModel 
    (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  )
  (1): CNN(
    (convs): ModuleList(
      (0): Conv1d(768, 256, kernel_size=(1,), stride=(1,))
      (1): Conv1d(768, 256, kernel_size=(3,), stride=(1,), padding=(1,))
      (2): Conv1d(768, 256, kernel_size=(5,), stride=(1,), padding=(2,))
    )
  )
  (2): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (3): Dense({'in_features': 768, 'out_features': 512, 'bias': True, 'activation_function': 'torch.nn.modules.activation.ReLU'})
  (4): Normalize()
)
```

## Usage

### Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

```bash
pip install -U sentence-transformers
```

Then you can load this model and run inference.
```python
from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("dsfsi/dna-paraphrase-mpnet-base-v2")
# Run on kmer DNA sequences
#
sentences = [
    'ATCCCC ATGGAA ATTTGG',
    'TTGGCC AAAGGG GGGTTT',
    'CCCTTT CTCTGTTT AAATTTTGG',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 512]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]
```
## Downstream Performance 

### Average Train Accuracy

| Dataset              | Infersent1   | all-MiniLM-L6-v2   | all-mpnet-base-v2   | paraphrase-mpnet-base-v2   | Proposed model   |
|----------------------|--------------|---------------------|----------------------|----------------------------|------------------|
| **H3**               | 61±0.014     | 62±0.027           | 63±0.034            | **65±0.021**               | 63±0.017         |
| **H4**               | 63±0.030     | 60±0.014           | 61±0.011            | 64±0.025                   | **66±0.012**     |
| **enhancers**        | 53±0.015     | 77±0.0203          | 79±0.022            | **83±0.020**               | 74±0.022         |
| **promoter_all**     | 54±0.011     | 75±0.007           | 77±0.008            | **83±0.006**               | 76±0.005         |
| **enhancers_types**  | 49±0.004     | 64±0.027           | 66±0.021            | **70±6.435**               | 59±0.013         |
| **promoter_no_tata** | 54±0.007     | 76±0.011           | 77±0.013            | **82±0.011**               | 76±0.011         |
| **promoter_tata**    | 51±0.023     | 74±0.037           | 74±0.039            | **82±0.016**               | 71±0.033         |
| **splice_sites_donors** | 54±0.022 | 55±0.019           | 56±0.012            | **58±0.021**               | **58±0.011**     |

### Test Accuracy

| Dataset               | Infersent1   | all-MiniLM-L6-v2   | all-mpnet-base-v2   | paraphrase-mpnet-base-v2   | Proposed model   |
|-----------------------|--------------|---------------------|----------------------|----------------------------|------------------|
| **H3**                | 61           | 64                 | 65                  | **67**                     | **67**           |
| **H4**                | 63           | 61                 | 60                  | 64                         | **65**           |
| **enhancers**         | 53           | 78                 | 76                  | **87**                     | 67               |
| **promoter_all**      | 54           | 76                 | 78                  | **83**                     | 78               |
| **enhancers_types**   | 46           | 67                 | 68                  | **70**                     | 62               |
| **promoter_no_tata**  | 54           | 76                 | 78                  | **83**                     | 78               |
| **promoter_tata**     | 52           | 76                 | 77                  | **84**                     | 75               |
| **splice_sites_donors** | 58        | 59                 | 60                  | **63**                     | 62               |


### Framework Versions
- Python: 3.8.10
- Sentence Transformers: 3.0.1
- Transformers: 4.43.4
- PyTorch: 2.4.0+cu121
- Accelerate: 
- Datasets: 
- Tokenizers: 0.19.1

## Citation

TBA

### BibTeX

TBA 


## Contributing

Your contributions are welcome! Feel free to improve the model.


## Model Card Authors

Mpho Mokoatle

## Model Card Contact

Email: u19394277@tuks.co.za