Hypencoder: Hypernetworks for Information Retrieval
The vast majority of retrieval models depend on vector inner products to
produce a relevance score between a query and a document. This naturally limits
the expressiveness of the relevance score that can be employed. We propose a
new paradigm, instead of producing a vector to represent the query we produce a
small neural network which acts as a learned relevance function. This small
neural network takes in a representation of the document, in this paper we use
a single vector, and produces a scalar relevance score. To produce the little
neural network we use a hypernetwork, a network that produce the weights of
other networks, as our query encoder or as we call it a Hypencoder. Experiments
on in-domain search tasks show that Hypencoder is able to significantly
outperform strong dense retrieval models and has higher metrics then reranking
models and models an order of magnitude larger. Hypencoder is also shown to
generalize well to out-of-domain search tasks. To assess the extent of
Hypencoder's capabilities, we evaluate on a set of hard retrieval tasks
including tip-of-the-tongue retrieval and instruction-following retrieval tasks
and find that the performance gap widens substantially compared to standard
retrieval tasks. Furthermore, to demonstrate the practicality of our method we
implement an approximate search algorithm and show that our model is able to
search 8.8M documents in under 60ms.
67ab99c8bb44ec714c6a4a96
Julian Killingback
claimed_verified
2025-02-12T09:16:46.458Z
635582fb6b58fa7cc8701580
/avatars/a00791ba70a2de40dacac4582307c0f2.svg
Julian Killingback
user
jfkback
67ab99c8bb44ec714c6a4a97
Hansi Zeng
67ab99c8bb44ec714c6a4a98
Hamed Zamani
Hypencoder: Hypernetworks for Information Retrieval
The vast majority of retrieval models depend on vector inner products to
produce a relevance score between a query and a document. This naturally limits
the expressiveness of the relevance score that can be employed. We propose a
new paradigm, instead of producing a vector to represent the query we produce a
small neural network which acts as a learned relevance function. This small
neural network takes in a representation of the document, in this paper we use
a single vector, and produces a scalar relevance score. To produce the little
neural network we use a hypernetwork, a network that produce the weights of
other networks, as our query encoder or as we call it a Hypencoder. Experiments
on in-domain search tasks show that Hypencoder is able to significantly
outperform strong dense retrieval models and has higher metrics then reranking
models and models an order of magnitude larger. Hypencoder is also shown to
generalize well to out-of-domain search tasks. To assess the extent of
Hypencoder's capabilities, we evaluate on a set of hard retrieval tasks
including tip-of-the-tongue retrieval and instruction-following retrieval tasks
and find that the performance gap widens substantially compared to standard
retrieval tasks. Furthermore, to demonstrate the practicality of our method we
implement an approximate search algorithm and show that our model is able to
search 8.8M documents in under 60ms.
A hypencoder, a hypernetwork-based query encoder, produces a neural network for relevance scoring, outperforming traditional retrieval models across various tasks.
retrieval models vector inner products relevance score neural network hypernetwork Hypencoder in-domain search tasks reranking models out-of-domain search tasks tip-of-the-tongue retrieval instruction-following retrieval tasks approximate search algorithm