The LLM Effect: Are Humans Truly Using LLMs, or Are They Being Influenced By Them Instead?
<|reference_start|>The LLM Effect: Are Humans Truly Using LLMs, or Are They Being Influenced By Them Instead?: Large Language Models (LLMs) have shown capabilities close to human performance in various analytical tasks, leading researchers to use them for time and labor-intensive analyses. However, their capability to handle highly specialized and open-ended tasks in domains like policy studies remains in question. This paper investigates the efficiency and accuracy of LLMs in specialized tasks through a structured user study focusing on Human-LLM partnership. The study, conducted in two stages-Topic Discovery and Topic Assignment-integrates LLMs with expert annotators to observe the impact of LLM suggestions on what is usually human-only analysis. Results indicate that LLM-generated topic lists have significant overlap with human generated topic lists, with minor hiccups in missing document-specific topics. However, LLM suggestions may significantly improve task completion speed, but at the same time introduce anchoring bias, potentially affecting the depth and nuance of the analysis, raising a critical question about the trade-off between increased efficiency and the risk of biased analysis.<|reference_end|>
@article{choi2024the,
    title={The LLM Effect: Are Humans Truly Using LLMs, or Are They Being
  Influenced By Them Instead?},
    author={Alexander S. Choi, Syeda Sabrina Akter, JP Singh, Antonios
  Anastasopoulos},
    journal={arXiv preprint arXiv:2410.04699},
    year={2024},
    archivePrefix={arXiv},
    eprint={2410.04699},
    primaryClass={cs.CL cs.HC}
}