Evaluating Deduplication Techniques for Economic Research Paper Titles with a Focus on Semantic Similarity using NLP and LLMs
<|reference_start|>Evaluating Deduplication Techniques for Economic Research Paper Titles with a Focus on Semantic Similarity using NLP and LLMs: This study investigates efficient deduplication techniques for a large NLP dataset of economic research paper titles. We explore various pairing methods alongside established distance measures (Levenshtein distance, cosine similarity) and a sBERT model for semantic evaluation. Our findings suggest a potentially low prevalence of duplicates based on the observed semantic similarity across different methods. Further exploration with a human-annotated ground truth set is completed for a more conclusive assessment. The result supports findings from the NLP, LLM based distance metrics.<|reference_end|>
@article{you2024evaluating,
    title={Evaluating Deduplication Techniques for Economic Research Paper Titles
  with a Focus on Semantic Similarity using NLP and LLMs},
    author={Doohee You, Karim Lasri, Samuel Fraiberger},
    journal={arXiv preprint arXiv:2410.01141},
    year={2024},
    archivePrefix={arXiv},
    eprint={2410.01141},
    primaryClass={cs.CL cs.AI}
}