MDS-ViTNet: Improving saliency prediction for Eye-Tracking with Vision Transformer
<|reference_start|>MDS-ViTNet: Improving saliency prediction for Eye-Tracking with Vision Transformer: In this paper, we present a novel methodology we call MDS-ViTNet (Multi Decoder Saliency by Vision Transformer Network) for enhancing visual saliency prediction or eye-tracking. This approach holds significant potential for diverse fields, including marketing, medicine, robotics, and retail. We propose a network architecture that leverages the Vision Transformer, moving beyond the conventional ImageNet backbone. The framework adopts an encoder-decoder structure, with the encoder utilizing a Swin transformer to efficiently embed most important features. This process involves a Transfer Learning method, wherein layers from the Vision Transformer are converted by the Encoder Transformer and seamlessly integrated into a CNN Decoder. This methodology ensures minimal information loss from the original input image. The decoder employs a multi-decoding technique, utilizing dual decoders to generate two distinct attention maps. These maps are subsequently combined into a singular output via an additional CNN model. Our trained model MDS-ViTNet achieves state-of-the-art results across several benchmarks. Committed to fostering further collaboration, we intend to make our code, models, and datasets accessible to the public.<|reference_end|>
@article{ignat2024mds-vitnet:,
    title={MDS-ViTNet: Improving saliency prediction for Eye-Tracking with Vision
  Transformer},
    author={Polezhaev Ignat, Goncharenko Igor, Iurina Natalya},
    journal={arXiv preprint arXiv:2405.19501},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.19501},
    primaryClass={cs.CV cs.AI}
}