On the rankability of visual embeddings
We study whether visual embedding models capture continuous, ordinal
attributes along linear directions, which we term _rank axes_. We define a
model as _rankable_ for an attribute if projecting embeddings onto such an axis
preserves the attribute's order. Across 7 popular encoders and 9 datasets with
attributes like age, crowd count, head pose, aesthetics, and recency, we find
that many embeddings are inherently rankable. Surprisingly, a small number of
samples, or even just two extreme examples, often suffice to recover meaningful
rank axes, without full-scale supervision. These findings open up new use cases
for image ranking in vector databases and motivate further study into the
structure and learning of rankable embeddings. Our code is available at
https://github.com/aktsonthalia/rankable-vision-embeddings.
686d26c6b720651ca3f52ce4
Ankit Sonthalia
686d26c6b720651ca3f52ce5
Arnas Uselis
claimed_verified
2025-07-09T08:51:28.251Z
6520898f7bf8cc2dd28b7a9c
/avatars/87a29ba95b71ee2dce18e97aa85e17a1.svg
Arnas Uselis
user
Gigglingface
686d26c6b720651ca3f52ce6
Seong Joon Oh
On the rankability of visual embeddings
We study whether visual embedding models capture continuous, ordinal
attributes along linear directions, which we term _rank axes_. We define a
model as _rankable_ for an attribute if projecting embeddings onto such an axis
preserves the attribute's order. Across 7 popular encoders and 9 datasets with
attributes like age, crowd count, head pose, aesthetics, and recency, we find
that many embeddings are inherently rankable. Surprisingly, a small number of
samples, or even just two extreme examples, often suffice to recover meaningful
rank axes, without full-scale supervision. These findings open up new use cases
for image ranking in vector databases and motivate further study into the
structure and learning of rankable embeddings. Our code is available at
https://github.com/aktsonthalia/rankable-vision-embeddings.
Visual embedding models often capture continuous, ordinal attributes along linear directions, enabling image ranking with minimal supervision.
visual embedding models rank axes rankable embeddings attribute order vector databases rankable embeddings