Pixel-SAIL: Single Transformer For Pixel-Grounded Understanding
Multimodal Large Language Models (MLLMs) achieve remarkable performance for
fine-grained pixel-level understanding tasks. However, all the works rely
heavily on extra components, such as vision encoder (CLIP), segmentation
experts, leading to high system complexity and limiting model scaling. In this
work, our goal is to explore a highly simplified MLLM without introducing extra
components. Our work is motivated by the recent works on Single trAnsformer as
a unified vIsion-Language Model (SAIL) design, where these works jointly learn
vision tokens and text tokens in transformers. We present Pixel-SAIL, a single
transformer for pixel-wise MLLM tasks. In particular, we present three
technical improvements on the plain baseline. First, we design a learnable
upsampling module to refine visual token features. Secondly, we propose a novel
visual prompt injection strategy to enable the single transformer to understand
visual prompt inputs and benefit from the early fusion of visual prompt
embeddings and vision tokens. Thirdly, we introduce a vision expert
distillation strategy to efficiently enhance the single transformer's
fine-grained feature extraction capability. In addition, we have collected a
comprehensive pixel understanding benchmark (PerBench), using a manual check.
It includes three tasks: detailed object description, visual prompt-based
question answering, and visual-text referring segmentation. Extensive
experiments on four referring segmentation benchmarks, one visual prompt
benchmark, and our PerBench show that our Pixel-SAIL achieves comparable or
even better results with a much simpler pipeline. Code and model will be
released at https://github.com/magic-research/Sa2VA.
67ff26c3414c03ebc1d42529
Tao Zhang
admin_assigned
2025-04-17T11:14:46.402Z
660d28db4215cc70372bc432
/avatars/a515303c6e29725ef3698bb695ffa743.svg
Tao Zhang
user
TaoZhang
67ff26c3414c03ebc1d4252a
Xiangtai Li
claimed_verified
2025-04-16T08:42:53.598Z
63958b4414513eaf9029ebf1
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/U1g5H071pWRswGAG9UTpo.png
Xiangtai Li
user
LXT
67ff26c3414c03ebc1d4252b
Zilong Huang
claimed_verified
2025-05-14T07:36:53.396Z
63a430ec84a6a25c65bff20c
/avatars/3ab51214c491f28d8fe6b70c373d72aa.svg
Zilong Huang
user
speedinghzl
67ff26c3414c03ebc1d4252c
Yanwei Li
admin_assigned
2025-04-17T11:15:18.824Z
643ff78cdc984afcbbbc3b1a
/avatars/eec5198ce88aaf8156840bec0d190a7f.svg
Yanwei Li
user
YanweiLi
67ff26c3414c03ebc1d4252d
Weixian Lei
67ff26c3414c03ebc1d4252e
Xueqing Deng
admin_assigned
2025-04-17T11:15:36.341Z
65ca9b1743207e438a95e90c
/avatars/8f7bde1c44d8e665a29ee08ce7fedfa4.svg
Xueqing Deng
user
xdeng77
67ff26c3414c03ebc1d4252f
Shihao Chen
admin_assigned
2025-04-17T11:15:44.009Z
66c7daa4c5c61e4baa771267
/avatars/c4cc5679d5a2225320c551f88e064bd6.svg
Shihao chen
user
sekirochen
67ff26c3414c03ebc1d42530
Shunping Ji
67ff26c3414c03ebc1d42531
Jiashi Feng
admin_assigned
2025-04-17T11:15:58.219Z
67298e44017b96a1d0101dc4
/avatars/1f8ed1a3e911e6a3021087b9371d284c.svg
Jiashi Feng
user
jshfeng
Pixel-SAIL: Single Transformer For Pixel-Grounded Understanding
Multimodal Large Language Models (MLLMs) achieve remarkable performance for
fine-grained pixel-level understanding tasks. However, all the works rely
heavily on extra components, such as vision encoder (CLIP), segmentation
experts, leading to high system complexity and limiting model scaling. In this
work, our goal is to explore a highly simplified MLLM without introducing extra
components. Our work is motivated by the recent works on Single trAnsformer as
a unified vIsion-Language Model (SAIL) design, where these works jointly learn
vision tokens and text tokens in transformers. We present Pixel-SAIL, a single
transformer for pixel-wise MLLM tasks. In particular, we present three
technical improvements on the plain baseline. First, we design a learnable
upsampling module to refine visual token features. Secondly, we propose a novel
visual prompt injection strategy to enable the single transformer to understand
visual prompt inputs and benefit from the early fusion of visual prompt
embeddings and vision tokens. Thirdly, we introduce a vision expert
distillation strategy to efficiently enhance the single transformer's
fine-grained feature extraction capability. In addition, we have collected a
comprehensive pixel understanding benchmark (PerBench), using a manual check.
It includes three tasks: detailed object description, visual prompt-based
question answering, and visual-text referring segmentation. Extensive
experiments on four referring segmentation benchmarks, one visual prompt
benchmark, and our PerBench show that our Pixel-SAIL achieves comparable or
even better results with a much simpler pipeline. Code and model will be
released at https://github.com/magic-research/Sa2VA.
Pixel-SAIL is a single transformer model that performs pixel-level understanding tasks without additional components, achieving comparable results through a simplified pipeline with three technical improvements.
multimodal large language models MLLMs fine-grained pixel-level understanding vision encoder CLIP segmentation experts system complexity SAIL Pixel-SAIL learnable upsampling module visual prompt injection vision expert distillation pixel understanding benchmark PerBench object description visual prompt-based question answering visual-text referring segmentation referring segmentation benchmarks visual prompt benchmark