HuatuoGPT-Vision-2-Grader

GitHub | Paper

A multimodal medical rubric grader for HuatuoGPT-Vision-2. Built on Qwen3-VL-8B-Instruct and trained on GPT-4.1 rubric judgments, it looks at the medical image together with the response and checks every rubric item in one generation. It provides the rewards for open-ended questions in RL and also grades text-only conversations.

๐Ÿ“Š Agreement with GPT-4.1

Held-out test set, free generation, per-rubric agreement with GPT-4.1 labels:

Domain Cases Rubric items Agreement Balanced accuracy
Medical images (PubMed figures) 193 5,398 92.4 92.2
HealthBench Professional 50 94 93.6 93.1
General and medical text 922 19,699 91.7 91.6
Overall 1,165 25,191 91.9 91.8

The output is always a valid JSON list of the right length.

๐Ÿ“ Grading template

Use this template exactly: replace only {conversation} and {rubrics}. Send one user message that contains the case images first (if any) and then the formatted prompt.

Placeholder Content
{conversation} Conversation as role: content lines, ending with the response to evaluate. Images are not written here.
{rubrics} Numbered criteria with signed points, e.g. 1. (+3pts) Criterion text.
GRADING_TEMPLATE = """Score the assistant's response against each rubric item.

## Conversation
<Conversation>
{conversation}
</Conversation>

## Rubrics
<Rubric_items>
{rubrics}
</Rubric_items>

## Output
Return only a JSON list of booleans, one for each rubric in order.
- true: the criterion is met
- false: the criterion is not met
- For a negative criterion (bad behavior with negative points), return true only when the response shows that bad behavior."""

๐Ÿš€ Usage

Run the template definition above, followed by this example:

from PIL import Image
from transformers import AutoModelForImageTextToText, AutoProcessor

model_id = "FreedomIntelligence/HuatuoGPT-Vision-2-Grader"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForImageTextToText.from_pretrained(model_id, dtype="auto", device_map="auto").eval()

conversation = """user: What does this chest X-ray show?
assistant: There is a large right-sided pleural effusion with blunting of the costophrenic angle."""
rubrics = """1. (+5pts) Identifies a pleural effusion.
2. (+3pts) Correctly states that the effusion is on the right side.
3. (-4pts) Reports a pneumothorax that is not present."""

messages = [{"role": "user", "content": [
    {"type": "image", "image": Image.open("chest_xray.jpg")},  # your image
    {"type": "text", "text": GRADING_TEMPLATE.format(conversation=conversation, rubrics=rubrics)},
]}]
inputs = processor.apply_chat_template(
    messages, tokenize=True, add_generation_prompt=True, return_dict=True, return_tensors="pt"
).to(model.device)
outputs = model.generate(**inputs, max_new_tokens=512, do_sample=False)
print(processor.decode(outputs[0, inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

Illustrative output, one boolean per criterion in the same order:

[true, true, false]

true means the criterion is met. For a negative-point criterion, it means the undesirable behavior is present and its points are deducted. The reward is the sum of matched points divided by the total positive points, clipped to [0, 1]. The example above scores (5 + 3) / (5 + 3) = 1.0.

For text-only conversations, send the formatted prompt as plain text without images.

โšก Batch scoring

Download this repository, then run:

python score.py --input example.json
python score.py --input batch.json --batch-size 8

Each record contains prompt, response, rubrics (criterion, points) and optional images (paths relative to the input file). It returns criteria_met, raw_score, and the clipped training_reward. grading.py holds the exact protocol used in HuatuoGPT-Vision-2 RL: the thinking part is removed and the first 10,000 characters of the final answer are graded.

๐Ÿ–ฅ๏ธ Serving with SGLang or vLLM

The grader works with any OpenAI-compatible server:

python -m sglang.launch_server --model-path FreedomIntelligence/HuatuoGPT-Vision-2-Grader --port 30001
# or: vllm serve FreedomIntelligence/HuatuoGPT-Vision-2-Grader --port 30001
import base64
import requests

image = "data:image/jpeg;base64," + base64.b64encode(open("example.jpg", "rb").read()).decode()
prompt = GRADING_TEMPLATE.format(conversation=conversation, rubrics=rubrics)
response = requests.post("http://127.0.0.1:30001/v1/chat/completions", json={
    "model": "FreedomIntelligence/HuatuoGPT-Vision-2-Grader",
    "messages": [{"role": "user", "content": [
        {"type": "image_url", "image_url": {"url": image}},
        {"type": "text", "text": prompt},
    ]}],
    "temperature": 0,
    "max_tokens": 512,
})
print(response.json()["choices"][0]["message"]["content"])

Images are resized by the model's processor to at most 921,600 pixels, the resolution used in training. For RL, point the RL code at the server.

๐Ÿ“– Citation

@inproceedings{chen2026incentivizing,
  title={Incentivizing Medical Vision Capabilities from Large-Scale Multimodal Pre-training},
  author={Chen, Junying and Cai, Zhenyang and Yang, Yunjin and Cai, Kunyuan and Wang, Rongsheng and Wang, Xidong and Yu, Geyang and Feng, Xiangyi and Wang, Benyou},
  booktitle={The Fortieth Annual Conference on Neural Information Processing Systems},
  year={2026}
}
Downloads last month
40
Safetensors
Model size
9B params
Tensor type
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for FreedomIntelligence/HuatuoGPT-Vision-2-Grader

Finetuned
(754)
this model
Quantizations
1 model