README.md

---
base_model:
- meta-llama/Llama-3.1-8B-Instruct
datasets:
- DongkiKim/Mol-LLaMA-Instruct
language:
- en
license: apache-2.0
tags:
- biology
- chemistry
- medical
pipeline_tag: text-generation
library_name: transformers
---

# Mol-Llama-3.1-8B-Instruct
[[Project Page](https://mol-llama.github.io/)] [[Paper](https://arxiv.org/abs/2502.13449)] [[GitHub](https://github.com/DongkiKim95/Mol-LLaMA)]

This repo contains the weights of Mol-LLaMA including the LoRA weights and projectors, based on [meta-llama/Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct).

## Architecture
![image.png](architecture.png)
1) Molecular encoders: Pretrained 2D encoder ([MoleculeSTM](https://huggingface.co/chao1224/MoleculeSTM)) and 3D encoder ([Uni-Mol](https://huggingface.co/dptech/Uni-Mol-Models))
2) Blending Module: Combining complementary information from 2D and 3D encoders via cross-attention
3) Q-Former: Embed molecular representations into query tokens based on [SciBERT](https://huggingface.co/allenai/scibert_scivocab_uncased)
4) LoRA: Adapters for fine-tuning LLMs


## Training Dataset

Mol-LLaMA is trained on [Mol-LLaMA-Instruct](https://huggingface.co/datasets/DongkiKim/Mol-LLaMA-Instruct), to learn the fundamental characteristics of molecules with the reasoning ability and explanbility.

## How to Use

Please check out [the exemplar code for inference](https://github.com/DongkiKim95/Mol-LLaMA/blob/master/playground.py) in the Github repo.

## Citation

If you find our model useful, please consider citing our work.
```
@misc{kim2025molllama,
    title={Mol-LLaMA: Towards General Understanding of Molecules in Large Molecular Language Model},
    author={Dongki Kim and Wonbin Lee and Sung Ju Hwang},
    year={2025},
    eprint={2502.13449},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}
```

## Acknowledgements

We appreciate [LLaMA](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct), [3D-MoLM](https://huggingface.co/Sihangli/3D-MoLM), [MoleculeSTM](https://huggingface.co/chao1224/MoleculeSTM), [Uni-Mol](https://huggingface.co/dptech/Uni-Mol-Models) and [SciBERT](https://huggingface.co/allenai/scibert_scivocab_uncased) for their open-source contributions.