Learning Effective Representations for Retrieval Using Self-Distillation with Adaptive Relevance Margins
<|reference_start|>Learning Effective Representations for Retrieval Using Self-Distillation with Adaptive Relevance Margins: Representation-based retrieval models, so-called biencoders, estimate the relevance of a document to a query by calculating the similarity of their respective embeddings. Current state-of-the-art biencoders are trained using an expensive training regime involving knowledge distillation from a teacher model and batch-sampling. Instead of relying on a teacher model, we contribute a novel parameter-free loss function for self-supervision that exploits the pre-trained language modeling capabilities of the encoder model as a training signal, eliminating the need for batch sampling by performing implicit hard negative mining. We investigate the capabilities of our proposed approach through extensive ablation studies, demonstrating that self-distillation can match the effectiveness of teacher distillation using only 13.5% of the data, while offering a speedup in training time between 3x and 15x compared to parametrized losses. Code and data is made openly available.<|reference_end|>
@article{gienapp2024learning,
    title={Learning Effective Representations for Retrieval Using Self-Distillation
  with Adaptive Relevance Margins},
    author={Lukas Gienapp, Niklas Deckers, Martin Potthast, Harrisen Scells},
    journal={arXiv preprint arXiv:2407.21515},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.21515},
    primaryClass={cs.IR}
}