SyncMask: Synchronized Attentional Masking for Fashion-centric Vision-Language Pretraining
<|reference_start|>SyncMask: Synchronized Attentional Masking for Fashion-centric Vision-Language Pretraining: Vision-language models (VLMs) have made significant strides in cross-modal understanding through large-scale paired datasets. However, in fashion domain, datasets often exhibit a disparity between the information conveyed in image and text. This issue stems from datasets containing multiple images of a single fashion item all paired with one text, leading to cases where some textual details are not visible in individual images. This mismatch, particularly when non-co-occurring elements are masked, undermines the training of conventional VLM objectives like Masked Language Modeling and Masked Image Modeling, thereby hindering the model's ability to accurately align fine-grained visual and textual features. Addressing this problem, we propose Synchronized attentional Masking (SyncMask), which generate masks that pinpoint the image patches and word tokens where the information co-occur in both image and text. This synchronization is accomplished by harnessing cross-attentional features obtained from a momentum model, ensuring a precise alignment between the two modalities. Additionally, we enhance grouped batch sampling with semi-hard negatives, effectively mitigating false negative issues in Image-Text Matching and Image-Text Contrastive learning objectives within fashion datasets. Our experiments demonstrate the effectiveness of the proposed approach, outperforming existing methods in three downstream tasks.<|reference_end|>
@article{song2024syncmask:,
    title={SyncMask: Synchronized Attentional Masking for Fashion-centric
  Vision-Language Pretraining},
    author={Chull Hwan Song, Taebaek Hwang, Jooyoung Yoon, Shunghyun Choi, Yeong
  Hyeon Gu},
    journal={arXiv preprint arXiv:2404.01156},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.01156},
    primaryClass={cs.CV cs.AI}
}