we may not release the original code.
I am currently working on a replication of the inference code. training would be next.

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer
from peft import PeftModel, PeftConfig

def get_model(peft_model_name):
config = PeftConfig.from_pretrained(peft_model_name)
base_model = AutoModelForSequenceClassification.from_pretrained(config.base_model_name_or_path)
model = PeftModel.from_pretrained(base_model, peft_model_name)
model = model.merge_and_unload()
model.eval()
return model

Load the tokenizer and model

tokenizer = AutoTokenizer.from_pretrained('meta-llama/Llama-2-7b-hf')
model = get_model('castorini/rankllama-v1-7b-lora-passage')

Define a query-passage pair

query = "What is llama?"
title = "Llama"
passage = "The llama is a domesticated South American camelid, widely used as a meat and pack animal by Andean cultures since the pre-Columbian era."

Tokenize the query-passage pair

inputs = tokenizer(f'query: {query}', f'document: {title} {passage}', return_tensors='pt')

Run the model forward

with torch.no_grad():
outputs = model(**inputs)
logits = outputs.logits
score = logits[0][0]
print(score)

Inference code is this ? I try it, but failed.

model = PeftModel.from_pretrained(base_model, 'castorini/rankllama-v1-7b-lora-doc')
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python3.9/dist-packages/peft/peft_model.py", line 161, in from_pretrained
model = set_peft_model_state_dict(model, adapters_weights)
File "/usr/local/lib/python3.9/dist-packages/peft/utils/save_and_load.py", line 74, in set_peft_model_state_dict
model.load_state_dict(peft_model_state_dict, strict=False)
File "/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py", line 2041, in load_state_dict
raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for PeftModelForSequenceClassification:
size mismatch for base_model.model.score.weight: copying a param with shape torch.Size([1, 4096]) from checkpoint, the shape in current model is torch.Size([2, 4096]).

Is peft not OK. what's your peft version.

oops... I had typo in this line

 base_model = AutoModelForSequenceClassification.from_pretrained(config.base_model_name_or_path, num_labels=1)

num_labels=1

thanks, fixed the error by adding num_labels=1.

btw, I have implemented a batch inference code for rankllama, can be found at here

added a replication of RankLLaMA training here https://github.com/texttron/tevatron/tree/main/examples/rankllama#train-rank-llama-from-scratch