Self-Guided Masked Autoencoders for Domain-Agnostic Self-Supervised Learning
<|reference_start|>Self-Guided Masked Autoencoders for Domain-Agnostic Self-Supervised Learning: Self-supervised learning excels in learning representations from large amounts of unlabeled data, demonstrating success across multiple data modalities. Yet, extending self-supervised learning to new modalities is non-trivial because the specifics of existing methods are tailored to each domain, such as domain-specific augmentations which reflect the invariances in the target task. While masked modeling is promising as a domain-agnostic framework for self-supervised learning because it does not rely on input augmentations, its mask sampling procedure remains domain-specific. We present Self-guided Masked Autoencoders (SMA), a fully domain-agnostic masked modeling method. SMA trains an attention based model using a masked modeling objective, by learning masks to sample without any domain-specific assumptions. We evaluate SMA on three self-supervised learning benchmarks in protein biology, chemical property prediction, and particle physics. We find SMA is capable of learning representations without domain-specific knowledge and achieves state-of-the-art performance on these three benchmarks.<|reference_end|>
@article{xie2024self-guided,
    title={Self-Guided Masked Autoencoders for Domain-Agnostic Self-Supervised
  Learning},
    author={Johnathan Xie, Yoonho Lee, Annie S. Chen, Chelsea Finn},
    journal={arXiv preprint arXiv:2402.14789},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.14789},
    primaryClass={cs.LG cs.AI}
}