Is Cosine-Similarity of Embeddings Really About Similarity?
<|reference_start|>Is Cosine-Similarity of Embeddings Really About Similarity?: Cosine-similarity is the cosine of the angle between two vectors, or equivalently the dot product between their normalizations. A popular application is to quantify semantic similarity between high-dimensional objects by applying cosine-similarity to a learned low-dimensional feature embedding. This can work better but sometimes also worse than the unnormalized dot-product between embedded vectors in practice. To gain insight into this empirical observation, we study embeddings derived from regularized linear models, where closed-form solutions facilitate analytical insights. We derive analytically how cosine-similarity can yield arbitrary and therefore meaningless `similarities.' For some linear models the similarities are not even unique, while for others they are implicitly controlled by the regularization. We discuss implications beyond linear models: a combination of different regularizations are employed when learning deep models; these have implicit and unintended effects when taking cosine-similarities of the resulting embeddings, rendering results opaque and possibly arbitrary. Based on these insights, we caution against blindly using cosine-similarity and outline alternatives.<|reference_end|>
@article{steck2024is,
    title={Is Cosine-Similarity of Embeddings Really About Similarity?},
    author={Harald Steck, Chaitanya Ekanadham, Nathan Kallus},
    journal={ACM Web Conference 2024 (WWW 2024 Companion)},
    year={2024},
    doi={10.1145/3589335.3651526},
    archivePrefix={arXiv},
    eprint={2403.05440},
    primaryClass={cs.IR cs.LG}
}