Dense Retrievers Can Fail on Simple Queries: Revealing The Granularity
  Dilemma of Embeddings
This work focuses on an observed limitation of text encoders: embeddings may
not be able to recognize fine-grained entities or events within the semantics,
resulting in failed dense retrieval on even simple cases. To examine such
behaviors, we first introduce a new evaluation dataset in Chinese, named
CapRetrieval, whose passages are image captions, and queries are phrases
inquiring entities or events in various forms. Zero-shot evaluation suggests
that encoders may fail on these fine-grained matching, regardless of training
sources or model sizes. Aiming for enhancement, we proceed to finetune encoders
with our proposed data generation strategies, which obtains the best
performance on CapRetrieval. Within this process, we further identify an issue
of granularity dilemma, a challenge for embeddings to express fine-grained
salience while aligning with overall semantics. Our dataset, code and models in
this work are publicly released at https://github.com/lxucs/CapRetrieval.
684cfefc3b733ba3336873a6
Liyan Xu
claimed_verified
2025-06-16T07:50:16.178Z
650f0fac11f3210cf7a8a849
/avatars/687d56c3a6d4f5cdb34e424cdcff954d.svg
Leon Xu
user
lxucs
684cfefc3b733ba3336873a7
Zhenlin Su
684cfefc3b733ba3336873a8
Mo Yu
684cfefc3b733ba3336873a9
Jiangnan Li
684cfefc3b733ba3336873aa
Fandong Meng
684cfefc3b733ba3336873ab
Jie Zhou
Dense Retrievers Can Fail on Simple Queries: Revealing The Granularity
  Dilemma of Embeddings
This work focuses on an observed limitation of text encoders: embeddings may
not be able to recognize fine-grained entities or events within the semantics,
resulting in failed dense retrieval on even simple cases. To examine such
behaviors, we first introduce a new evaluation dataset in Chinese, named
CapRetrieval, whose passages are image captions, and queries are phrases
inquiring entities or events in various forms. Zero-shot evaluation suggests
that encoders may fail on these fine-grained matching, regardless of training
sources or model sizes. Aiming for enhancement, we proceed to finetune encoders
with our proposed data generation strategies, which obtains the best
performance on CapRetrieval. Within this process, we further identify an issue
of granularity dilemma, a challenge for embeddings to express fine-grained
salience while aligning with overall semantics. Our dataset, code and models in
this work are publicly released at https://github.com/lxucs/CapRetrieval.
A new dataset named CapRetrieval is introduced to evaluate the ability of text encoders to recognize fine-grained entities and events, highlighting challenges in dense retrieval tasks.
text encoders embeddings fine-grained entities events dense retrieval zero-shot evaluation data generation strategies granularity dilemma