Evaluating ChatGPT on Nuclear Domain-Specific Data
<|reference_start|>Evaluating ChatGPT on Nuclear Domain-Specific Data: This paper examines the application of ChatGPT, a large language model (LLM), for question-and-answer (Q&A) tasks in the highly specialized field of nuclear data. The primary focus is on evaluating ChatGPT's performance on a curated test dataset, comparing the outcomes of a standalone LLM with those generated through a Retrieval Augmented Generation (RAG) approach. LLMs, despite their recent advancements, are prone to generating incorrect or 'hallucinated' information, which is a significant limitation in applications requiring high accuracy and reliability. This study explores the potential of utilizing RAG in LLMs, a method that integrates external knowledge bases and sophisticated retrieval techniques to enhance the accuracy and relevance of generated outputs. In this context, the paper evaluates ChatGPT's ability to answer domain-specific questions, employing two methodologies: A) direct response from the LLM, and B) response from the LLM within a RAG framework. The effectiveness of these methods is assessed through a dual mechanism of human and LLM evaluation, scoring the responses for correctness and other metrics. The findings underscore the improvement in performance when incorporating a RAG pipeline in an LLM, particularly in generating more accurate and contextually appropriate responses for nuclear domain-specific queries. Additionally, the paper highlights alternative approaches to further refine and improve the quality of answers in such specialized domains.<|reference_end|>
@article{anwar2024evaluating,
    title={Evaluating ChatGPT on Nuclear Domain-Specific Data},
    author={Muhammad Anwar, Mischa de Costa, Issam Hammad, Daniel Lau},
    journal={43rd Annual CNS Conference and the 48th Annual CNS/CNA Student
  Conference Sheraton Cavalier Saskatoon Hotel, Saskatoon, SK, Canada, June
  16-19, 2024},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.00090},
    primaryClass={cs.CL cs.AI}
}