Weakly Supervised Co-training with Swapping Assignments for Semantic Segmentation
<|reference_start|>Weakly Supervised Co-training with Swapping Assignments for Semantic Segmentation: Class activation maps (CAMs) are commonly employed in weakly supervised semantic segmentation (WSSS) to produce pseudo-labels. Due to incomplete or excessive class activation, existing studies often resort to offline CAM refinement, introducing additional stages or proposing offline modules. This can cause optimization difficulties for single-stage methods and limit generalizability. In this study, we aim to reduce the observed CAM inconsistency and error to mitigate reliance on refinement processes. We propose an end-to-end WSSS model incorporating guided CAMs, wherein our segmentation model is trained while concurrently optimizing CAMs online. Our method, Co-training with Swapping Assignments (CoSA), leverages a dual-stream framework, where one sub-network learns from the swapped assignments generated by the other. We introduce three techniques: i) soft perplexity-based regularization to penalize uncertain regions; ii) a threshold-searching approach to dynamically revise the confidence threshold; and iii) contrastive separation to address the coexistence problem. CoSA demonstrates exceptional performance, achieving mIoU of 76.2\% and 51.0\% on VOC and COCO validation datasets, respectively, surpassing existing baselines by a substantial margin. Notably, CoSA is the first single-stage approach to outperform all existing multi-stage methods including those with additional supervision. Code is avilable at \url{https://github.com/youshyee/CoSA}.<|reference_end|>
@article{yang2024weakly,
    title={Weakly Supervised Co-training with Swapping Assignments for Semantic
  Segmentation},
    author={Xinyu Yang, Hossein Rahmani, Sue Black, Bryan M. Williams},
    journal={arXiv preprint arXiv:2402.17891},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.17891},
    primaryClass={cs.CV}
}