MaskRIS: Semantic Distortion-aware Data Augmentation for Referring Image
  Segmentation
Referring Image Segmentation (RIS) is an advanced vision-language task that
involves identifying and segmenting objects within an image as described by
free-form text descriptions. While previous studies focused on aligning visual
and language features, exploring training techniques, such as data
augmentation, remains underexplored. In this work, we explore effective data
augmentation for RIS and propose a novel training framework called Masked
Referring Image Segmentation (MaskRIS). We observe that the conventional image
augmentations fall short of RIS, leading to performance degradation, while
simple random masking significantly enhances the performance of RIS. MaskRIS
uses both image and text masking, followed by Distortion-aware Contextual
Learning (DCL) to fully exploit the benefits of the masking strategy. This
approach can improve the model's robustness to occlusions, incomplete
information, and various linguistic complexities, resulting in a significant
performance improvement. Experiments demonstrate that MaskRIS can easily be
applied to various RIS models, outperforming existing methods in both fully
supervised and weakly supervised settings. Finally, MaskRIS achieves new
state-of-the-art performance on RefCOCO, RefCOCO+, and RefCOCOg datasets. Code
is available at https://github.com/naver-ai/maskris.
674fdf3e6bc2d6af0e165bf0
Minhyun Lee
674fdf3e6bc2d6af0e165bf1
Seungho Lee
674fdf3e6bc2d6af0e165bf2
Song Park
674fdf3e6bc2d6af0e165bf3
Dongyoon Han
claimed_verified
2025-07-11T08:21:17.133Z
660f8cc1a61244f3df3d4426
/avatars/45d59766122bb3482f6dd7f9d98aa87a.svg
Dongyoon Han
user
calintz
674fdf3e6bc2d6af0e165bf4
Byeongho Heo
claimed_verified
2024-12-04T09:44:21.729Z
64b9feed96676e40d0fa89a7
/avatars/2154a6ceb87677ad2c9d9620de5b18ec.svg
Byeongho Heo
user
bhheo
674fdf3e6bc2d6af0e165bf5
Hyunjung Shim
MaskRIS: Semantic Distortion-aware Data Augmentation for Referring Image
  Segmentation
Referring Image Segmentation (RIS) is an advanced vision-language task that
involves identifying and segmenting objects within an image as described by
free-form text descriptions. While previous studies focused on aligning visual
and language features, exploring training techniques, such as data
augmentation, remains underexplored. In this work, we explore effective data
augmentation for RIS and propose a novel training framework called Masked
Referring Image Segmentation (MaskRIS). We observe that the conventional image
augmentations fall short of RIS, leading to performance degradation, while
simple random masking significantly enhances the performance of RIS. MaskRIS
uses both image and text masking, followed by Distortion-aware Contextual
Learning (DCL) to fully exploit the benefits of the masking strategy. This
approach can improve the model's robustness to occlusions, incomplete
information, and various linguistic complexities, resulting in a significant
performance improvement. Experiments demonstrate that MaskRIS can easily be
applied to various RIS models, outperforming existing methods in both fully
supervised and weakly supervised settings. Finally, MaskRIS achieves new
state-of-the-art performance on RefCOCO, RefCOCO+, and RefCOCOg datasets. Code
is available at https://github.com/naver-ai/maskris.
A novel training framework called Masked Referring Image Segmentation (MaskRIS) enhances RIS performance through image and text masking combined with Distortion-aware Contextual Learning.
referring image segmentation Masked Referring Image Segmentation MaskRIS data augmentation image masking text masking Distortion-aware Contextual Learning RIS RefCOCO RefCOCO+ RefCOCOg