People over trust AI-generated medical responses and view them to be as valid as doctors, despite low accuracy
<|reference_start|>People over trust AI-generated medical responses and view them to be as valid as doctors, despite low accuracy: This paper presents a comprehensive analysis of how AI-generated medical responses are perceived and evaluated by non-experts. A total of 300 participants gave evaluations for medical responses that were either written by a medical doctor on an online healthcare platform, or generated by a large language model and labeled by physicians as having high or low accuracy. Results showed that participants could not effectively distinguish between AI-generated and Doctors' responses and demonstrated a preference for AI-generated responses, rating High Accuracy AI-generated responses as significantly more valid, trustworthy, and complete/satisfactory. Low Accuracy AI-generated responses on average performed very similar to Doctors' responses, if not more. Participants not only found these low-accuracy AI-generated responses to be valid, trustworthy, and complete/satisfactory but also indicated a high tendency to follow the potentially harmful medical advice and incorrectly seek unnecessary medical attention as a result of the response provided. This problematic reaction was comparable if not more to the reaction they displayed towards doctors' responses. This increased trust placed on inaccurate or inappropriate AI-generated medical advice can lead to misdiagnosis and harmful consequences for individuals seeking help. Further, participants were more trusting of High Accuracy AI-generated responses when told they were given by a doctor and experts rated AI-generated responses significantly higher when the source of the response was unknown. Both experts and non-experts exhibited bias, finding AI-generated responses to be more thorough and accurate than Doctors' responses but still valuing the involvement of a Doctor in the delivery of their medical advice. Ensuring AI systems are implemented with medical professionals should be the future of using AI for the delivery of medical advice.<|reference_end|>
@article{shekar2024people,
    title={People over trust AI-generated medical responses and view them to be as
  valid as doctors, despite low accuracy},
    author={Shruthi Shekar, Pat Pataranutaporn, Chethan Sarabu, Guillermo A.
  Cecchi, Pattie Maes},
    journal={arXiv preprint arXiv:2408.15266},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.15266},
    primaryClass={cs.HC cs.AI cs.CY}
}