title: UniRef++: Segment Every Reference Object in Spatial and Temporal Spaces
summary: The reference-based object segmentation tasks, namely referring image
  segmentation (RIS), few-shot image segmentation (FSS), referring video object
  segmentation (RVOS), and video object segmentation (VOS), aim to segment a
  specific object by utilizing either language or annotated masks as references.
  Despite significant progress in each respective field, current methods are
  task-specifically designed and developed in different directions, which hinders
  the activation of multi-task capabilities for these tasks. In this work, we end
  the current fragmented situation and propose UniRef++ to unify the four
  reference-based object segmentation tasks with a single architecture. At the
  heart of our approach is the proposed UniFusion module which performs
  multiway-fusion for handling different tasks with respect to their specified
  references. And a unified Transformer architecture is then adopted for
  achieving instance-level segmentation. With the unified designs, UniRef++ can
  be jointly trained on a broad range of benchmarks and can flexibly complete
  multiple tasks at run-time by specifying the corresponding references. We
  evaluate our unified models on various benchmarks. Extensive experimental
  results indicate that our proposed UniRef++ achieves state-of-the-art
  performance on RIS and RVOS, and performs competitively on FSS and VOS with a
  parameter-shared network. Moreover, we showcase that the proposed UniFusion
  module could be easily incorporated into the current advanced foundation model
  SAM and obtain satisfactory results with parameter-efficient finetuning. Codes
  and models are available at https://github.com/FoundationVision/UniRef.
paper_authors._id: 658bc1f96f5f457dea421554
paper_authors.name: Jiannan Wu
paper_authors._id: 658bc1f96f5f457dea421555
paper_authors.name: Yi Jiang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-05T10:01:31.927Z
paper_authors._id: 6344dcb1cd37e44d9ed46508
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6344dcb1cd37e44d9ed46508/J92UKSxKR3iziD2WJfih4.jpeg
paper_authors.fullname: Yi Jiang
paper_authors.type: user
paper_authors.user: JiangYi
paper_authors._id: 658bc1f96f5f457dea421556
paper_authors.name: Bin Yan
paper_authors._id: 658bc1f96f5f457dea421557
paper_authors.name: Huchuan Lu
paper_authors._id: 658bc1f96f5f457dea421558
paper_authors.name: Zehuan Yuan
paper_authors._id: 658bc1f96f5f457dea421559
paper_authors.name: Ping Luo
paper_title: UniRef++: Segment Every Reference Object in Spatial and Temporal Spaces
paper_summary: The reference-based object segmentation tasks, namely referring image
  segmentation (RIS), few-shot image segmentation (FSS), referring video object
  segmentation (RVOS), and video object segmentation (VOS), aim to segment a
  specific object by utilizing either language or annotated masks as references.
  Despite significant progress in each respective field, current methods are
  task-specifically designed and developed in different directions, which hinders
  the activation of multi-task capabilities for these tasks. In this work, we end
  the current fragmented situation and propose UniRef++ to unify the four
  reference-based object segmentation tasks with a single architecture. At the
  heart of our approach is the proposed UniFusion module which performs
  multiway-fusion for handling different tasks with respect to their specified
  references. And a unified Transformer architecture is then adopted for
  achieving instance-level segmentation. With the unified designs, UniRef++ can
  be jointly trained on a broad range of benchmarks and can flexibly complete
  multiple tasks at run-time by specifying the corresponding references. We
  evaluate our unified models on various benchmarks. Extensive experimental
  results indicate that our proposed UniRef++ achieves state-of-the-art
  performance on RIS and RVOS, and performs competitively on FSS and VOS with a
  parameter-shared network. Moreover, we showcase that the proposed UniFusion
  module could be easily incorporated into the current advanced foundation model
  SAM and obtain satisfactory results with parameter-efficient finetuning. Codes
  and models are available at https://github.com/FoundationVision/UniRef.
paper_ai_keywords: UniRef++ UniFusion module multiway-fusion Transformer architecture instance-level segmentation parameter-efficient fine-tuning