title: Pixel-SAIL: Single Transformer For Pixel-Grounded Understanding
summary: Multimodal Large Language Models (MLLMs) achieve remarkable performance for
  fine-grained pixel-level understanding tasks. However, all the works rely
  heavily on extra components, such as vision encoder (CLIP), segmentation
  experts, leading to high system complexity and limiting model scaling. In this
  work, our goal is to explore a highly simplified MLLM without introducing extra
  components. Our work is motivated by the recent works on Single trAnsformer as
  a unified vIsion-Language Model (SAIL) design, where these works jointly learn
  vision tokens and text tokens in transformers. We present Pixel-SAIL, a single
  transformer for pixel-wise MLLM tasks. In particular, we present three
  technical improvements on the plain baseline. First, we design a learnable
  upsampling module to refine visual token features. Secondly, we propose a novel
  visual prompt injection strategy to enable the single transformer to understand
  visual prompt inputs and benefit from the early fusion of visual prompt
  embeddings and vision tokens. Thirdly, we introduce a vision expert
  distillation strategy to efficiently enhance the single transformer's
  fine-grained feature extraction capability. In addition, we have collected a
  comprehensive pixel understanding benchmark (PerBench), using a manual check.
  It includes three tasks: detailed object description, visual prompt-based
  question answering, and visual-text referring segmentation. Extensive
  experiments on four referring segmentation benchmarks, one visual prompt
  benchmark, and our PerBench show that our Pixel-SAIL achieves comparable or
  even better results with a much simpler pipeline. Code and model will be
  released at https://github.com/magic-research/Sa2VA.
paper_authors._id: 67ff26c3414c03ebc1d42529
paper_authors.name: Tao Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-17T11:14:46.402Z
paper_authors._id: 660d28db4215cc70372bc432
paper_authors.avatarUrl: /avatars/a515303c6e29725ef3698bb695ffa743.svg
paper_authors.fullname: Tao Zhang
paper_authors.type: user
paper_authors.user: TaoZhang
paper_authors._id: 67ff26c3414c03ebc1d4252a
paper_authors.name: Xiangtai Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-16T08:42:53.598Z
paper_authors._id: 63958b4414513eaf9029ebf1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/U1g5H071pWRswGAG9UTpo.png
paper_authors.fullname: Xiangtai Li
paper_authors.type: user
paper_authors.user: LXT
paper_authors._id: 67ff26c3414c03ebc1d4252b
paper_authors.name: Zilong Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-14T07:36:53.396Z
paper_authors._id: 63a430ec84a6a25c65bff20c
paper_authors.avatarUrl: /avatars/3ab51214c491f28d8fe6b70c373d72aa.svg
paper_authors.fullname: Zilong Huang
paper_authors.type: user
paper_authors.user: speedinghzl
paper_authors._id: 67ff26c3414c03ebc1d4252c
paper_authors.name: Yanwei Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-17T11:15:18.824Z
paper_authors._id: 643ff78cdc984afcbbbc3b1a
paper_authors.avatarUrl: /avatars/eec5198ce88aaf8156840bec0d190a7f.svg
paper_authors.fullname: Yanwei Li
paper_authors.type: user
paper_authors.user: YanweiLi
paper_authors._id: 67ff26c3414c03ebc1d4252d
paper_authors.name: Weixian Lei
paper_authors._id: 67ff26c3414c03ebc1d4252e
paper_authors.name: Xueqing Deng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-17T11:15:36.341Z
paper_authors._id: 65ca9b1743207e438a95e90c
paper_authors.avatarUrl: /avatars/8f7bde1c44d8e665a29ee08ce7fedfa4.svg
paper_authors.fullname: Xueqing Deng
paper_authors.type: user
paper_authors.user: xdeng77
paper_authors._id: 67ff26c3414c03ebc1d4252f
paper_authors.name: Shihao Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-17T11:15:44.009Z
paper_authors._id: 66c7daa4c5c61e4baa771267
paper_authors.avatarUrl: /avatars/c4cc5679d5a2225320c551f88e064bd6.svg
paper_authors.fullname: Shihao chen
paper_authors.type: user
paper_authors.user: sekirochen
paper_authors._id: 67ff26c3414c03ebc1d42530
paper_authors.name: Shunping Ji
paper_authors._id: 67ff26c3414c03ebc1d42531
paper_authors.name: Jiashi Feng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-17T11:15:58.219Z
paper_authors._id: 67298e44017b96a1d0101dc4
paper_authors.avatarUrl: /avatars/1f8ed1a3e911e6a3021087b9371d284c.svg
paper_authors.fullname: Jiashi Feng
paper_authors.type: user
paper_authors.user: jshfeng
paper_title: Pixel-SAIL: Single Transformer For Pixel-Grounded Understanding
paper_summary: Multimodal Large Language Models (MLLMs) achieve remarkable performance for
  fine-grained pixel-level understanding tasks. However, all the works rely
  heavily on extra components, such as vision encoder (CLIP), segmentation
  experts, leading to high system complexity and limiting model scaling. In this
  work, our goal is to explore a highly simplified MLLM without introducing extra
  components. Our work is motivated by the recent works on Single trAnsformer as
  a unified vIsion-Language Model (SAIL) design, where these works jointly learn
  vision tokens and text tokens in transformers. We present Pixel-SAIL, a single
  transformer for pixel-wise MLLM tasks. In particular, we present three
  technical improvements on the plain baseline. First, we design a learnable
  upsampling module to refine visual token features. Secondly, we propose a novel
  visual prompt injection strategy to enable the single transformer to understand
  visual prompt inputs and benefit from the early fusion of visual prompt
  embeddings and vision tokens. Thirdly, we introduce a vision expert
  distillation strategy to efficiently enhance the single transformer's
  fine-grained feature extraction capability. In addition, we have collected a
  comprehensive pixel understanding benchmark (PerBench), using a manual check.
  It includes three tasks: detailed object description, visual prompt-based
  question answering, and visual-text referring segmentation. Extensive
  experiments on four referring segmentation benchmarks, one visual prompt
  benchmark, and our PerBench show that our Pixel-SAIL achieves comparable or
  even better results with a much simpler pipeline. Code and model will be
  released at https://github.com/magic-research/Sa2VA.
paper_ai_summary: Pixel-SAIL is a single transformer model that performs pixel-level understanding tasks without additional components, achieving comparable results through a simplified pipeline with three technical improvements.
paper_ai_keywords: multimodal large language models MLLMs fine-grained pixel-level understanding vision encoder CLIP segmentation experts system complexity SAIL Pixel-SAIL learnable upsampling module visual prompt injection vision expert distillation pixel understanding benchmark PerBench object description visual prompt-based question answering visual-text referring segmentation referring segmentation benchmarks visual prompt benchmark