DyVo: Dynamic Vocabularies for Learned Sparse Retrieval with Entities
Learned Sparse Retrieval (LSR) models use vocabularies from pre-trained
transformers, which often split entities into nonsensical fragments. Splitting
entities can reduce retrieval accuracy and limits the model's ability to
incorporate up-to-date world knowledge not included in the training data. In
this work, we enhance the LSR vocabulary with Wikipedia concepts and entities,
enabling the model to resolve ambiguities more effectively and stay current
with evolving knowledge. Central to our approach is a Dynamic Vocabulary (DyVo)
head, which leverages existing entity embeddings and an entity retrieval
component that identifies entities relevant to a query or document. We use the
DyVo head to generate entity weights, which are then merged with word piece
weights to create joint representations for efficient indexing and retrieval
using an inverted index. In experiments across three entity-rich document
ranking datasets, the resulting DyVo model substantially outperforms
state-of-the-art baselines.
6710f99feea7a876d49d4088
Thong Nguyen
6710f99feea7a876d49d4089
Shubham Chatterjee
admin_assigned
2024-10-21T12:31:33.851Z
62118220cf354d3c41b3e1b6
/avatars/d360a92001d7a09fcfe8290160e4dab9.svg
Shubham Chatterjee
user
ShubhamC
6710f99feea7a876d49d408a
Sean MacAvaney
admin_assigned
2024-10-21T12:31:40.123Z
61dc410f7813d319d7387635
https://cdn-avatars.huggingface.co/v1/production/uploads/1666975288423-61dc410f7813d319d7387635.jpeg
Sean MacAvaney
user
macavaney
6710f99feea7a876d49d408b
Iain Mackie
6710f99feea7a876d49d408c
Jeff Dalton
admin_assigned
2024-10-21T12:31:48.759Z
650c05f9d4a0852d3c89bfd2
/avatars/a1b3bf4a861a8f0bd5a0ff3a076ed5a2.svg
dalton
user
jeffdalton
6710f99feea7a876d49d408d
Andrew Yates
admin_assigned
2024-10-21T12:31:55.238Z
5efef195ff69163f6f59e647
https://cdn-avatars.huggingface.co/v1/production/uploads/1599574940380-5efef195ff69163f6f59e647.jpeg
Andrew Yates
user
andrewyates
DyVo: Dynamic Vocabularies for Learned Sparse Retrieval with Entities
Learned Sparse Retrieval (LSR) models use vocabularies from pre-trained
transformers, which often split entities into nonsensical fragments. Splitting
entities can reduce retrieval accuracy and limits the model's ability to
incorporate up-to-date world knowledge not included in the training data. In
this work, we enhance the LSR vocabulary with Wikipedia concepts and entities,
enabling the model to resolve ambiguities more effectively and stay current
with evolving knowledge. Central to our approach is a Dynamic Vocabulary (DyVo)
head, which leverages existing entity embeddings and an entity retrieval
component that identifies entities relevant to a query or document. We use the
DyVo head to generate entity weights, which are then merged with word piece
weights to create joint representations for efficient indexing and retrieval
using an inverted index. In experiments across three entity-rich document
ranking datasets, the resulting DyVo model substantially outperforms
state-of-the-art baselines.
The introduction of a Dynamic Vocabulary head enhances Learned Sparse Retrieval models by integrating Wikipedia concepts, improving entity resolution and retrieval accuracy.
Learned Sparse Retrieval LSR transformers vocabulary Wikipedia concepts Dynamic Vocabulary DyVo head entity embeddings entity retrieval joint representations inverted index entity-rich document ranking