DOEI: Dual Optimization of Embedding Information for Attention-Enhanced
  Class Activation Maps
Weakly supervised semantic segmentation (WSSS) typically utilizes limited
semantic annotations to obtain initial Class Activation Maps (CAMs). However,
due to the inadequate coupling between class activation responses and semantic
information in high-dimensional space, the CAM is prone to object co-occurrence
or under-activation, resulting in inferior recognition accuracy. To tackle this
issue, we propose DOEI, Dual Optimization of Embedding Information, a novel
approach that reconstructs embedding representations through semantic-aware
attention weight matrices to optimize the expression capability of embedding
information. Specifically, DOEI amplifies tokens with high confidence and
suppresses those with low confidence during the class-to-patch interaction.
This alignment of activation responses with semantic information strengthens
the propagation and decoupling of target features, enabling the generated
embeddings to more accurately represent target features in high-level semantic
space. In addition, we propose a hybrid-feature alignment module in DOEI that
combines RGB values, embedding-guided features, and self-attention weights to
increase the reliability of candidate tokens. Comprehensive experiments show
that DOEI is an effective plug-and-play module that empowers state-of-the-art
visual transformer-based WSSS models to significantly improve the quality of
CAMs and segmentation performance on popular benchmarks, including PASCAL VOC
(+3.6%, +1.5%, +1.2% mIoU) and MS COCO (+1.2%, +1.6% mIoU). Code will be
available at https://github.com/AIGeeksGroup/DOEI.
67c05aeca2a76d8a27d33c8a
Hongjie Zhu
67c05aeca2a76d8a27d33c8b
Zeyu Zhang
claimed_verified
2025-02-27T12:53:51.821Z
64ec877bb93654d4ca5c92e9
https://cdn-avatars.huggingface.co/v1/production/uploads/64ec877bb93654d4ca5c92e9/GvHk_KSdE9Rhnk_o-NaZX.jpeg
Zeyu Zhang
user
SteveZeyuZhang
67c05aeca2a76d8a27d33c8c
Guansong Pang
67c05aeca2a76d8a27d33c8d
Xu Wang
67c05aeca2a76d8a27d33c8e
Shimin Wen
67c05aeca2a76d8a27d33c8f
Yu Bai
67c05aeca2a76d8a27d33c90
Daji Ergu
67c05aeca2a76d8a27d33c91
Ying Cai
67c05aeca2a76d8a27d33c92
Yang Zhao
DOEI: Dual Optimization of Embedding Information for Attention-Enhanced
  Class Activation Maps
Weakly supervised semantic segmentation (WSSS) typically utilizes limited
semantic annotations to obtain initial Class Activation Maps (CAMs). However,
due to the inadequate coupling between class activation responses and semantic
information in high-dimensional space, the CAM is prone to object co-occurrence
or under-activation, resulting in inferior recognition accuracy. To tackle this
issue, we propose DOEI, Dual Optimization of Embedding Information, a novel
approach that reconstructs embedding representations through semantic-aware
attention weight matrices to optimize the expression capability of embedding
information. Specifically, DOEI amplifies tokens with high confidence and
suppresses those with low confidence during the class-to-patch interaction.
This alignment of activation responses with semantic information strengthens
the propagation and decoupling of target features, enabling the generated
embeddings to more accurately represent target features in high-level semantic
space. In addition, we propose a hybrid-feature alignment module in DOEI that
combines RGB values, embedding-guided features, and self-attention weights to
increase the reliability of candidate tokens. Comprehensive experiments show
that DOEI is an effective plug-and-play module that empowers state-of-the-art
visual transformer-based WSSS models to significantly improve the quality of
CAMs and segmentation performance on popular benchmarks, including PASCAL VOC
(+3.6%, +1.5%, +1.2% mIoU) and MS COCO (+1.2%, +1.6% mIoU). Code will be
available at https://github.com/AIGeeksGroup/DOEI.
A novel method, DOEI, enhances WSSS models by optimizing embedding representations and aligning class activation responses with semantic information, improving segmentation accuracy on benchmarks.
Class Activation Maps Dual Optimization of Embedding Information semantic-aware attention weight matrices class-to-patch interaction hybrid-feature alignment module embedding-guided features self-attention weights visual transformer-based WSSS models mIoU PASCAL VOC MS COCO