Do LLMs Find Human Answers To Fact-Driven Questions Perplexing? A Case Study on Reddit
<|reference_start|>Do LLMs Find Human Answers To Fact-Driven Questions Perplexing? A Case Study on Reddit: Large language models (LLMs) have been shown to be proficient in correctly answering questions in the context of online discourse. However, the study of using LLMs to model human-like answers to fact-driven social media questions is still under-explored. In this work, we investigate how LLMs model the wide variety of human answers to fact-driven questions posed on several topic-specific Reddit communities, or subreddits. We collect and release a dataset of 409 fact-driven questions and 7,534 diverse, human-rated answers from 15 r/Ask{Topic} communities across 3 categories: profession, social identity, and geographic location. We find that LLMs are considerably better at modeling highly-rated human answers to such questions, as opposed to poorly-rated human answers. We present several directions for future research based on our initial findings.<|reference_end|>
@article{seegmiller2024do,
    title={Do LLMs Find Human Answers To Fact-Driven Questions Perplexing? A Case
  Study on Reddit},
    author={Parker Seegmiller, Joseph Gatto, Omar Sharif, Madhusudan Basak and
  Sarah Masud Preum},
    journal={arXiv preprint arXiv:2404.01147},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.01147},
    primaryClass={cs.CL cs.LG}
}