title: DeCLIP: Decoupled Learning for Open-Vocabulary Dense Perception
summary: Dense visual prediction tasks have been constrained by their reliance on
  predefined categories, limiting their applicability in real-world scenarios
  where visual concepts are unbounded. While Vision-Language Models (VLMs) like
  CLIP have shown promise in open-vocabulary tasks, their direct application to
  dense prediction often leads to suboptimal performance due to limitations in
  local feature representation. In this work, we present our observation that
  CLIP's image tokens struggle to effectively aggregate information from
  spatially or semantically related regions, resulting in features that lack
  local discriminability and spatial consistency. To address this issue, we
  propose DeCLIP, a novel framework that enhances CLIP by decoupling the
  self-attention module to obtain ``content'' and ``context'' features
  respectively. The ``content'' features are aligned with image crop
  representations to improve local discriminability, while ``context'' features
  learn to retain the spatial correlations under the guidance of vision
  foundation models, such as DINO. Extensive experiments demonstrate that DeCLIP
  significantly outperforms existing methods across multiple open-vocabulary
  dense prediction tasks, including object detection and semantic segmentation.
  Code is available at magenta{https://github.com/xiaomoguhz/DeCLIP}.
paper_authors._id: 681d615fbd89ba9ceb5e94bc
paper_authors.name: Junjie Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-09T07:21:48.630Z
paper_authors._id: 64a385281cbf675203fbb7df
paper_authors.avatarUrl: /avatars/f259d080d3127c45bcf564a8d1fafcc6.svg
paper_authors.fullname: Junjie Wang
paper_authors.type: user
paper_authors.user: xiaomoguhzz
paper_authors._id: 681d615fbd89ba9ceb5e94bd
paper_authors.name: Bin Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-15T13:47:00.478Z
paper_authors._id: 647d70d136e109abce415e0e
paper_authors.avatarUrl: /avatars/c89864c663cbf3256e34785be85561bc.svg
paper_authors.fullname: Bin Chen
paper_authors.type: user
paper_authors.user: BinChen68
paper_authors._id: 681d615fbd89ba9ceb5e94be
paper_authors.name: Yulin Li
paper_authors._id: 681d615fbd89ba9ceb5e94bf
paper_authors.name: Bin Kang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-15T13:47:31.037Z
paper_authors._id: 67069836f95c7ec727df2806
paper_authors.avatarUrl: /avatars/fbe87f41e680362595b5864e690b62b4.svg
paper_authors.fullname: bin kang
paper_authors.type: user
paper_authors.user: tygeer
paper_authors._id: 681d615fbd89ba9ceb5e94c0
paper_authors.name: Yichi Chen
paper_authors._id: 681d615fbd89ba9ceb5e94c1
paper_authors.name: Zhuotao Tian
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-15T13:48:12.084Z
paper_authors._id: 64d104a37a7305c5895bd720
paper_authors.avatarUrl: /avatars/2d9eff3a2dff6d02e45ae8964fb91f27.svg
paper_authors.fullname: zt tian
paper_authors.type: user
paper_authors.user: tianzhuotao
paper_title: DeCLIP: Decoupled Learning for Open-Vocabulary Dense Perception
paper_summary: Dense visual prediction tasks have been constrained by their reliance on
  predefined categories, limiting their applicability in real-world scenarios
  where visual concepts are unbounded. While Vision-Language Models (VLMs) like
  CLIP have shown promise in open-vocabulary tasks, their direct application to
  dense prediction often leads to suboptimal performance due to limitations in
  local feature representation. In this work, we present our observation that
  CLIP's image tokens struggle to effectively aggregate information from
  spatially or semantically related regions, resulting in features that lack
  local discriminability and spatial consistency. To address this issue, we
  propose DeCLIP, a novel framework that enhances CLIP by decoupling the
  self-attention module to obtain ``content'' and ``context'' features
  respectively. The ``content'' features are aligned with image crop
  representations to improve local discriminability, while ``context'' features
  learn to retain the spatial correlations under the guidance of vision
  foundation models, such as DINO. Extensive experiments demonstrate that DeCLIP
  significantly outperforms existing methods across multiple open-vocabulary
  dense prediction tasks, including object detection and semantic segmentation.
  Code is available at magenta{https://github.com/xiaomoguhz/DeCLIP}.
paper_ai_summary: DeCLIP enhances CLIP by separating content and context features, improving performance on open-vocabulary dense prediction tasks like object detection and semantic segmentation.
paper_ai_keywords: Vision-Language Models (VLMs) CLIP image tokens self-attention module content features context features image crop representations DINO spatial correlations object detection semantic segmentation