brineylab/CurrAb

CurrAb

CurrAb is an antibody language model that uses an ESM-2 architecture. It was pre-trained on unpaired and paired sequences from the OAS, using the curriculum learning approach described in our preprint on biorxiv. Datasets used for pre-training are avaliable on Zenodo and code is avaliable on GitHub.

Use

Load the model and tokenizer as follows:

from transformers import EsmTokenizer, EsmForMaskedLM

model = EsmForMaskedLM.from_pretrained("sburbach/CurrAb")
tokenizer = EsmTokenizer.from_pretrained("sburbach/CurrAb")

The tokenizer expects inputs in the format: ["VQ..SS<cls>EV..IK"] for paired sequences, ["VQ..SS<cls>"] for unpaired heavy chains and ["<cls>EV..IK"] for unpaired light chains.

The model can be finetuned for classification tasks (such as specificity and pair classification in the paper) by loading the model with a sequence classification head:

from transformers import EsmForSequenceClassification

model = EsmForSequenceClassification.from_pretrained("sburbach/CurrAb")

# freeze the base model weights prior to finetuning
for param in model.base_model.parameters():
    param.requires_grad = False