Hard Negative Mining for Domain-Specific Retrieval in Enterprise Systems
Enterprise search systems often struggle to retrieve accurate,
domain-specific information due to semantic mismatches and overlapping
terminologies. These issues can degrade the performance of downstream
applications such as knowledge management, customer support, and
retrieval-augmented generation agents. To address this challenge, we propose a
scalable hard-negative mining framework tailored specifically for
domain-specific enterprise data. Our approach dynamically selects semantically
challenging but contextually irrelevant documents to enhance deployed
re-ranking models.
  Our method integrates diverse embedding models, performs dimensionality
reduction, and uniquely selects hard negatives, ensuring computational
efficiency and semantic precision. Evaluation on our proprietary enterprise
corpus (cloud services domain) demonstrates substantial improvements of 15\% in
MRR@3 and 19\% in MRR@10 compared to state-of-the-art baselines and other
negative sampling techniques. Further validation on public domain-specific
datasets (FiQA, Climate Fever, TechQA) confirms our method's generalizability
and readiness for real-world applications.
6838949f702f865b81701ff0
Hansa Meghwani
6838949f702f865b81701ff1
Amit Agarwal
extracted_confirmed
2025-05-30T23:03:01.880Z
6321a50a6b1992383fa00db1
/avatars/92311b4a5e8bcc8f5477747621334fc8.svg
Amit Agarwal
user
amitbcp
6838949f702f865b81701ff2
Priyaranjan Pattnayak
6838949f702f865b81701ff3
Hitesh Laxmichand Patel
6838949f702f865b81701ff4
Srikant Panda
Hard Negative Mining for Domain-Specific Retrieval in Enterprise Systems
Enterprise search systems often struggle to retrieve accurate,
domain-specific information due to semantic mismatches and overlapping
terminologies. These issues can degrade the performance of downstream
applications such as knowledge management, customer support, and
retrieval-augmented generation agents. To address this challenge, we propose a
scalable hard-negative mining framework tailored specifically for
domain-specific enterprise data. Our approach dynamically selects semantically
challenging but contextually irrelevant documents to enhance deployed
re-ranking models.
  Our method integrates diverse embedding models, performs dimensionality
reduction, and uniquely selects hard negatives, ensuring computational
efficiency and semantic precision. Evaluation on our proprietary enterprise
corpus (cloud services domain) demonstrates substantial improvements of 15\% in
MRR@3 and 19\% in MRR@10 compared to state-of-the-art baselines and other
negative sampling techniques. Further validation on public domain-specific
datasets (FiQA, Climate Fever, TechQA) confirms our method's generalizability
and readiness for real-world applications.
A scalable hard-negative mining framework enhances domain-specific enterprise search by dynamically selecting semantically challenging irrelevant documents, improving re-ranking models' performance.
hard-negative mining domain-specific enterprise data semantically challenging documents contextually irrelevant documents re-ranking models embedding models dimensionality reduction MRR@3 MRR@10 FiQA Climate Fever TechQA