title: REF-VLM: Triplet-Based Referring Paradigm for Unified Visual Decoding
summary: Multimodal Large Language Models (MLLMs) demonstrate robust zero-shot
  capabilities across diverse vision-language tasks after training on mega-scale
  datasets. However, dense prediction tasks, such as semantic segmentation and
  keypoint detection, pose significant challenges for MLLMs when represented
  solely as text outputs. Simultaneously, current MLLMs utilizing latent
  embeddings for visual task decoding generally demonstrate limited adaptability
  to both multi-task learning and multi-granularity scenarios. In this work, we
  present REF-VLM, an end-to-end framework for unified training of various visual
  decoding tasks. To address complex visual decoding scenarios, we introduce the
  Triplet-Based Referring Paradigm (TRP), which explicitly decouples three
  critical dimensions in visual decoding tasks through a triplet structure:
  concepts, decoding types, and targets. TRP employs symbolic delimiters to
  enforce structured representation learning, enhancing the parsability and
  interpretability of model outputs. Additionally, we construct Visual-Task
  Instruction Following Dataset (VTInstruct), a large-scale multi-task dataset
  containing over 100 million multimodal dialogue samples across 25 task types.
  Beyond text inputs and outputs, VT-Instruct incorporates various visual prompts
  such as point, box, scribble, and mask, and generates outputs composed of text
  and visual units like box, keypoint, depth and mask. The combination of
  different visual prompts and visual units generates a wide variety of task
  types, expanding the applicability of REF-VLM significantly. Both qualitative
  and quantitative experiments demonstrate that our REF-VLM outperforms other
  MLLMs across a variety of standard benchmarks. The code, dataset, and demo
  available at https://github.com/MacavityT/REF-VLM.
paper_authors._id: 67d050c28db3ce9654a56a96
paper_authors.name: Yan Tai
paper_authors._id: 67d050c28db3ce9654a56a97
paper_authors.name: Luhao Zhu
paper_authors._id: 67d050c28db3ce9654a56a98
paper_authors.name: Zhiqiang Chen
paper_authors._id: 67d050c28db3ce9654a56a99
paper_authors.name: Ynan Ding
paper_authors._id: 67d050c28db3ce9654a56a9a
paper_authors.name: Yiying Dong
paper_authors._id: 67d050c28db3ce9654a56a9b
paper_authors.name: Xiaohong Liu
paper_authors._id: 67d050c28db3ce9654a56a9c
paper_authors.name: Guodong Guo
paper_title: REF-VLM: Triplet-Based Referring Paradigm for Unified Visual Decoding
paper_summary: Multimodal Large Language Models (MLLMs) demonstrate robust zero-shot
  capabilities across diverse vision-language tasks after training on mega-scale
  datasets. However, dense prediction tasks, such as semantic segmentation and
  keypoint detection, pose significant challenges for MLLMs when represented
  solely as text outputs. Simultaneously, current MLLMs utilizing latent
  embeddings for visual task decoding generally demonstrate limited adaptability
  to both multi-task learning and multi-granularity scenarios. In this work, we
  present REF-VLM, an end-to-end framework for unified training of various visual
  decoding tasks. To address complex visual decoding scenarios, we introduce the
  Triplet-Based Referring Paradigm (TRP), which explicitly decouples three
  critical dimensions in visual decoding tasks through a triplet structure:
  concepts, decoding types, and targets. TRP employs symbolic delimiters to
  enforce structured representation learning, enhancing the parsability and
  interpretability of model outputs. Additionally, we construct Visual-Task
  Instruction Following Dataset (VTInstruct), a large-scale multi-task dataset
  containing over 100 million multimodal dialogue samples across 25 task types.
  Beyond text inputs and outputs, VT-Instruct incorporates various visual prompts
  such as point, box, scribble, and mask, and generates outputs composed of text
  and visual units like box, keypoint, depth and mask. The combination of
  different visual prompts and visual units generates a wide variety of task
  types, expanding the applicability of REF-VLM significantly. Both qualitative
  and quantitative experiments demonstrate that our REF-VLM outperforms other
  MLLMs across a variety of standard benchmarks. The code, dataset, and demo
  available at https://github.com/MacavityT/REF-VLM.
paper_ai_keywords: multimodal large language models (MLLMs) zero-shot capabilities vision-language tasks dense prediction tasks semantic segmentation keypoint detection latent embeddings multi-task learning multi-granularity scenarios REF-VLM end-to-end framework Triplet-Based Referring Paradigm (TRP) concepts decoding types targets symbolic delimiters structured representation learning parsability interpretability Visual-Task Instruction Following Dataset (VTInstruct) multimodal dialogue samples point box scribble mask visual units depth