Instructions to use FreedomIntelligence/HuatuoGPT-Vision-2-Grader with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use FreedomIntelligence/HuatuoGPT-Vision-2-Grader with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="FreedomIntelligence/HuatuoGPT-Vision-2-Grader") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("FreedomIntelligence/HuatuoGPT-Vision-2-Grader") model = AutoModelForMultimodalLM.from_pretrained("FreedomIntelligence/HuatuoGPT-Vision-2-Grader", device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use FreedomIntelligence/HuatuoGPT-Vision-2-Grader with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "FreedomIntelligence/HuatuoGPT-Vision-2-Grader" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "FreedomIntelligence/HuatuoGPT-Vision-2-Grader", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/FreedomIntelligence/HuatuoGPT-Vision-2-Grader
- SGLang
How to use FreedomIntelligence/HuatuoGPT-Vision-2-Grader with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "FreedomIntelligence/HuatuoGPT-Vision-2-Grader" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "FreedomIntelligence/HuatuoGPT-Vision-2-Grader", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "FreedomIntelligence/HuatuoGPT-Vision-2-Grader" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "FreedomIntelligence/HuatuoGPT-Vision-2-Grader", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use FreedomIntelligence/HuatuoGPT-Vision-2-Grader with Docker Model Runner:
docker model run hf.co/FreedomIntelligence/HuatuoGPT-Vision-2-Grader
A multimodal medical rubric grader for HuatuoGPT-Vision-2. Built on Qwen3-VL-8B-Instruct and trained on GPT-4.1 rubric judgments, it looks at the medical image together with the response and checks every rubric item in one generation. It provides the rewards for open-ended questions in RL and also grades text-only conversations.
๐ Agreement with GPT-4.1
Held-out test set, free generation, per-rubric agreement with GPT-4.1 labels:
| Domain | Cases | Rubric items | Agreement | Balanced accuracy |
|---|---|---|---|---|
| Medical images (PubMed figures) | 193 | 5,398 | 92.4 | 92.2 |
| HealthBench Professional | 50 | 94 | 93.6 | 93.1 |
| General and medical text | 922 | 19,699 | 91.7 | 91.6 |
| Overall | 1,165 | 25,191 | 91.9 | 91.8 |
The output is always a valid JSON list of the right length.
๐ Grading template
Use this template exactly: replace only {conversation} and {rubrics}. Send one user message that contains the case images first (if any) and then the formatted prompt.
| Placeholder | Content |
|---|---|
{conversation} |
Conversation as role: content lines, ending with the response to evaluate. Images are not written here. |
{rubrics} |
Numbered criteria with signed points, e.g. 1. (+3pts) Criterion text. |
GRADING_TEMPLATE = """Score the assistant's response against each rubric item.
## Conversation
<Conversation>
{conversation}
</Conversation>
## Rubrics
<Rubric_items>
{rubrics}
</Rubric_items>
## Output
Return only a JSON list of booleans, one for each rubric in order.
- true: the criterion is met
- false: the criterion is not met
- For a negative criterion (bad behavior with negative points), return true only when the response shows that bad behavior."""
๐ Usage
Run the template definition above, followed by this example:
from PIL import Image
from transformers import AutoModelForImageTextToText, AutoProcessor
model_id = "FreedomIntelligence/HuatuoGPT-Vision-2-Grader"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForImageTextToText.from_pretrained(model_id, dtype="auto", device_map="auto").eval()
conversation = """user: What does this chest X-ray show?
assistant: There is a large right-sided pleural effusion with blunting of the costophrenic angle."""
rubrics = """1. (+5pts) Identifies a pleural effusion.
2. (+3pts) Correctly states that the effusion is on the right side.
3. (-4pts) Reports a pneumothorax that is not present."""
messages = [{"role": "user", "content": [
{"type": "image", "image": Image.open("chest_xray.jpg")}, # your image
{"type": "text", "text": GRADING_TEMPLATE.format(conversation=conversation, rubrics=rubrics)},
]}]
inputs = processor.apply_chat_template(
messages, tokenize=True, add_generation_prompt=True, return_dict=True, return_tensors="pt"
).to(model.device)
outputs = model.generate(**inputs, max_new_tokens=512, do_sample=False)
print(processor.decode(outputs[0, inputs["input_ids"].shape[-1]:], skip_special_tokens=True))
Illustrative output, one boolean per criterion in the same order:
[true, true, false]
true means the criterion is met. For a negative-point criterion, it means the undesirable behavior is present and its points are deducted. The reward is the sum of matched points divided by the total positive points, clipped to [0, 1]. The example above scores (5 + 3) / (5 + 3) = 1.0.
For text-only conversations, send the formatted prompt as plain text without images.
โก Batch scoring
Download this repository, then run:
python score.py --input example.json
python score.py --input batch.json --batch-size 8
Each record contains prompt, response, rubrics (criterion, points) and optional images (paths relative to the input file). It returns criteria_met, raw_score, and the clipped training_reward. grading.py holds the exact protocol used in HuatuoGPT-Vision-2 RL: the thinking part is removed and the first 10,000 characters of the final answer are graded.
๐ฅ๏ธ Serving with SGLang or vLLM
The grader works with any OpenAI-compatible server:
python -m sglang.launch_server --model-path FreedomIntelligence/HuatuoGPT-Vision-2-Grader --port 30001
# or: vllm serve FreedomIntelligence/HuatuoGPT-Vision-2-Grader --port 30001
import base64
import requests
image = "data:image/jpeg;base64," + base64.b64encode(open("example.jpg", "rb").read()).decode()
prompt = GRADING_TEMPLATE.format(conversation=conversation, rubrics=rubrics)
response = requests.post("http://127.0.0.1:30001/v1/chat/completions", json={
"model": "FreedomIntelligence/HuatuoGPT-Vision-2-Grader",
"messages": [{"role": "user", "content": [
{"type": "image_url", "image_url": {"url": image}},
{"type": "text", "text": prompt},
]}],
"temperature": 0,
"max_tokens": 512,
})
print(response.json()["choices"][0]["message"]["content"])
Images are resized by the model's processor to at most 921,600 pixels, the resolution used in training. For RL, point the RL code at the server.
๐ Citation
@inproceedings{chen2026incentivizing,
title={Incentivizing Medical Vision Capabilities from Large-Scale Multimodal Pre-training},
author={Chen, Junying and Cai, Zhenyang and Yang, Yunjin and Cai, Kunyuan and Wang, Rongsheng and Wang, Xidong and Yu, Geyang and Feng, Xiangyi and Wang, Benyou},
booktitle={The Fortieth Annual Conference on Neural Information Processing Systems},
year={2026}
}
- Downloads last month
- 40