Keypoint Aware Masked Image Modelling
<|reference_start|>Keypoint Aware Masked Image Modelling: SimMIM is a widely used method for pretraining vision transformers using masked image modeling. However, despite its success in fine-tuning performance, it has been shown to perform sub-optimally when used for linear probing. We propose an efficient patch-wise weighting derived from keypoint features which captures the local information and provides better context during SimMIM's reconstruction phase. Our method, KAMIM, improves the top-1 linear probing accuracy from 16.12% to 33.97%, and finetuning accuracy from 76.78% to 77.3% when tested on the ImageNet-1K dataset with a ViT-B when trained for the same number of epochs. We conduct extensive testing on different datasets, keypoint extractors, and model architectures and observe that patch-wise weighting augments linear probing performance for larger pretraining datasets. We also analyze the learned representations of a ViT-B trained using KAMIM and observe that they behave similar to contrastive learning with regard to its behavior, with longer attention distances and homogenous self-attention across layers. Our code is publicly available at https://github.com/madhava20217/KAMIM.<|reference_end|>
@article{krishna2024keypoint,
    title={Keypoint Aware Masked Image Modelling},
    author={Madhava Krishna, A V Subramanyam},
    journal={arXiv preprint arXiv:2407.13873},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.13873},
    primaryClass={cs.CV cs.LG}
}