Instructions to use HongxinLi/UIPro-7B_Stage1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use HongxinLi/UIPro-7B_Stage1 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="HongxinLi/UIPro-7B_Stage1")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForImageTextToText

processor = AutoProcessor.from_pretrained("HongxinLi/UIPro-7B_Stage1")
model = AutoModelForImageTextToText.from_pretrained("HongxinLi/UIPro-7B_Stage1")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use HongxinLi/UIPro-7B_Stage1 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "HongxinLi/UIPro-7B_Stage1"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "HongxinLi/UIPro-7B_Stage1",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/HongxinLi/UIPro-7B_Stage1

SGLang

How to use HongxinLi/UIPro-7B_Stage1 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "HongxinLi/UIPro-7B_Stage1" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "HongxinLi/UIPro-7B_Stage1",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "HongxinLi/UIPro-7B_Stage1" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "HongxinLi/UIPro-7B_Stage1",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use HongxinLi/UIPro-7B_Stage1 with Docker Model Runner:
```
docker model run hf.co/HongxinLi/UIPro-7B_Stage1
```

UIPro: Unleashing Superior Interaction Capability For GUI Agents

[💻Code] [🚀Quick Start] [📝Paper]

Model Details

Model Description

Developed by: Brave Group, CASIA
Model type: Vision-Language Model
Language(s) (NLP): English
License: Apache License 2.0
Finetuned from model [optional]: Qwen2-VL-7B-Instruct

Model Sources [optional]

UIPro_1stage-7B is a GUI grounding model finetuned from Qwen2-VL-7B-Instruct.

Repository: https://github.com/ZJULiHongxin/UIPro
Paper [optional]: https://arxiv.org/abs/2509.17328

Uses

Direct Use

First, ensure that the necessary dependencies are installed:

pip install transformers
pip install qwen-vl-utils

Inference code example:

from transformers import Qwen2VLForConditionalGeneration, AutoProcessor
from qwen_vl_utils import process_vision_info

# Default: Load the model on the available device(s)
model = Qwen2VLForConditionalGeneration.from_pretrained(
    "HongxinLi/UIPro_1stage", torch_dtype="auto", device_map="auto"
)
processor = AutoProcessor.from_pretrained("HongxinLi/UIPro_1stage")

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": "./web_6f93090a-81f6-489e-bb35-1a2838b18c01.png",
            },

            # For ScreenSpot-v2, MOTIF, RefExp, and VisualWebBench Action Grounding
            {"type": "text", "text": "I want to {goal_info}. Please locate the target element I should interact with. (Output the center coordinates of the target)"},
            # For AutoGUI
            {"type": "text", "text": "Locate the element according to its detailed functionality description. {goal_info} (Output the center coordinates of the target)"},
            # For VisualWebBench Element Grounding
            {"type": "text", "text": "Locate the text "{goal_info}" (Output the center coordinates of the target)"},
        ],
    }
]

Evaluation

It is recommended to use AutoGUI evaluation suite based on LMMS-EVAL to evaluate it on multiple GUI Grounding benchmarks.

Results

Model	Size	Input Res.	FuncGnd	ScreenSpot	ScreenSpot-v2	MOTIF	RefExp	VWB EG	VWB AG
GPT-4o	-	AnyRes	9.8	17.8	20.4	30.5	21.8	5.6	6.8
Qwen2VL [1]	72B	AnyRes	47.7	71.4	73.2	80.3	77.7	60.5	62.1
Qwen2VL [1]	7B	AnyRes	38.7	66.4	66.9	75.1	64.8	55.9	62.1
CogAgent [2]	18B	1120	29.3	47.4	49.2	46.7	35.0	55.7	59.2
SeeClick [3]	10B	448	19.8	53.4	54.0	11.1	58.1	39.2	27.2
Ferret-UI [4]	8B	AnyRes	1.2	7.1	7.8	15.9	5.5	3.9	1.9
UGround [5]	7B	AnyRes	48.8	74.8	76.5	72.4	73.6	85.2	63.1
OS-ATLAS-Base [6]	7B	AnyRes	52.1	82.5	84.1	78.8	66.5	82.6	69.9
UIPro-Qwen2VL (ours)	7B	AnyRes	58.8	82.5	86.9	80.6	81.9	94.9	70.9
Qwen2-VL [4]	2B	AnyRes	7.1	17.9	18.6	28.8	29.2	17.9	17.5
UIPro-SLiME (ours)	3B	AnyRes	58.3	60.7	61.1	73.3	59.0	60.0	40.8

Comparison on the GUI element grounding benchmarks. UIPro achieves impressive grounding accuracy, especially on FuncPred, RefExp, and VWB EG. AnyRes means using an image division strategy to handle images with variable resolutions.

References: [1] Qwen2VL [2] CogAgent [3] SeeClick [4] Ferret-UI [5] UGround [6] OS-ATLAS-Base

Citation

BibTeX:

@InProceedings{Li_2025_ICCV,
    author    = {Li, Hongxin and Su, Jingran and Chen, Jingfan and Ju, Zheng and Chen, Yuntao and Li, Qing and Zhang, Zhaoxiang},
    title     = {UIPro: Unleashing Superior Interaction Capability For GUI Agents},
    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
    month     = {October},
    year      = {2025},
    pages     = {1613-1623}
}

Framework versions

PEFT 0.11.1

Downloads last month: 3

Safetensors

Model size

8B params

Tensor type

BF16

Model tree for HongxinLi/UIPro-7B_Stage1

Base model

Qwen/Qwen2-VL-7B

Finetuned

Qwen/Qwen2-VL-7B-Instruct

Finetuned

(595)

this model

Paper for HongxinLi/UIPro-7B_Stage1

UIPro: Unleashing Superior Interaction Capability For GUI Agents

Paper • 2509.17328 • Published Sep 22, 2025