2kxx/URM

SkyJM-Gen-9B-pointwise (stage-6 checklist judge)

SkyJM-Gen-9B-pointwise is the pointwise checklist scorer variant of SkyJM-Gen-9B. Given one generated image and the text prompt, it writes its own evaluation checklist and returns a scalar quality score.

Where the base model answers "which of these two images is better", this model answers "how good is this image, and why" — which is what an online RL loop needs (a reward per rollout, not a preference between two).

	base SkyJM-Gen-9B	this model
Task	pairwise preference	pointwise scoring
Input	prompt + 2 images	prompt + 1 image
Output	preference + rubric	eval_points[] (0/1 each) + final_score (0–10)
Dimensions	dynamic rubric	fixed 3 dimensions (see below)

Output format

The model replies with JSON only:

{
  "eval_points": [
    {"question": "Is the bus yellow?", "dimension": "Prompt Alignment", "score": 1}
  ],
  "dimension_summary": {
    "Prompt Alignment": "...", "Aesthetics": "...", "Technical Quality": "..."
  },
  "score_reason": "...",
  "final_score": 3.5
}

• eval_points[].dimension is exactly one of Prompt Alignment, Aesthetics, Technical Quality.
• eval_points[].score is 0 or 1.
• final_score is on the 0–10 scale (decimals allowed); typical checklist is 8–14 points, weighted toward Prompt Alignment by construction.

Reward for RL = final_score / 10 in [0, 1].

The full request/response contract (system prompt, user template, failure modes) is shipped as reward_protocol/ inside this repository, and mirrored in the accompanying code release (Flow-Factory reward adapter for this judge); the system prompt used at training time is reproduced verbatim there.

Quick start (serving)

# ms-swift + vLLM (recommended: this checkpoint uses Qwen3.5's linear-attention
# layers, and ms-swift's deploy path is the one it was validated with)
CUDA_VISIBLE_DEVICES=0,1 python -m swift.cli.deploy \
  --model skylenage-ai/SkyJM-Gen-9B-pointwise \
  --infer_backend vllm --served_model_name stage6-judge \
  --host 0.0.0.0 --port 8000 \
  --vllm_tensor_parallel_size 2 --vllm_gpu_memory_utilization 0.85 \
  --vllm_max_model_len 16384 --vllm_max_num_seqs 64 \
  --vllm_limit_mm_per_prompt '{"image": 1}'

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="EMPTY")
reply = client.chat.completions.create(
    model="stage6-judge",
    messages=[
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": [
            {"type": "image_url", "image_url": {"url": data_url}},
            {"type": "text", "text": USER_TEMPLATE.format(instruction=prompt)},
        ]},
    ],
    temperature=0.0,
    max_tokens=4096,
    # NOTE: do NOT set frequency_penalty / presence_penalty — see Limitations.
)

Quick start (as an RL reward)

This checkpoint was used as the reward model for a GRPO fine-tune of SD3.5-medium on GenEval prompts, through Flow-Factory:

rewards:
  - name: "stage6_judge"
    reward_model: "stage6_judge_t2i"        # reward_protocol/overlay adapter
    api_base_url: "http://127.0.0.1:8000/v1"
    vlm_model: "stage6-judge"
    max_tokens: 4096
    temperature: 0.0
    frequency_penalty: 0.0
    image_max_pixels: 1048576
    fallback_reward: 0.0
    async_reward: true
    num_workers: 4

Training

Starting from the stage-6 SFT checkpoint of the RubricRM pipeline (SkyJM-Gen-9B lineage), fine-tuned to emit the fixed 3-dimension pointwise checklist above:

Item	Value
Backbone	Qwen3.5-9B VL (same as SkyJM-Gen-9B)
Adapter	LoRA, r=32, alpha=64, attention + MLP projections (vision tower frozen)
Training data	prompt + single generated image + checklist labels with 0/1 point scores and a 0–10 final_score
Context	system prompt + user text + image + JSON reply (<= ~2.5k completion tokens)
Released checkpoint	stage-6 SFT checkpoint-250, merged into full weights (best of 250/500/552, see below)

Why checkpoint-250

The three saved checkpoints of the stage-6 run were evaluated with the same protocol (pointwise scoring -> pairwise comparison, temperature=0, tie gap 0.25):

checkpoint	GenAI-Bench (image_generation, 1305 pairs)	MMRB2 (t2i, 1000 pairs)	mean
250	0.6100 (no-tie 0.7032, tie 0.1326)	0.5210 (no-tie 0.6570, tie 0.2070)	0.5655
500	0.5893 (no-tie 0.6947, tie 0.1517)	0.5180 (no-tie 0.6607, tie 0.2160)	0.5536
552	0.5904 (no-tie 0.6971, tie 0.1531)	0.5210 (no-tie 0.6688, tie 0.2210)	0.5557

checkpoint-250 also has the lowest tie rate, and it is the checkpoint the downstream RL recipes and benchmark reruns standardised on.

Reproduce the ranking with sd35_rl/scripts/pick_best_stage6_ckpt.py.

Limitations

• Do not apply a repetition / frequency penalty. With frequency_penalty=1.2 the model penalises JSON syntax tokens and aborts mid-JSON (finish_reason="stop", no final_score). Keep it at 0 and control runaway checklists with a length cap instead.
• It is a pointwise scorer: preference benchmarks must be derived by scoring two images separately and comparing final_score.
• Scores are not calibrated across prompts; compare deltas for the same prompt, not absolute values between different prompts.
• Text rendering in images is only checked when the prompt asks for text.
• The model writes its own checklist per call, so scores have run-to-run variance even at temperature=0 when the checklist changes; pin temperature=0 and average over repeats if you need stability.

Citation

@misc{skyjm_gen_9b_pointwise,
  title = {SkyJM-Gen-9B-pointwise: a pointwise checklist judge for text-to-image generation},
  year  = {2026}
}

See also: SkyJM-Gen-9B, RubricRM-Data.