Zahra Azizi|Pouria Alipour|Sofia Gomez|Cassandra Broadwin|Sumaiya Islam|Ashish Sarraju|Albert J. Rogers|Alexander T. Sandhu|Fátima Rodríguez
Evaluating Recommendations About Atrial Fibrillation for Patients and Clinicians Obtained From Chat-Based Artificial Intelligence Algorithms
McGill University|Stanford Medicine|Cleveland Clinic|Cardiovascular Institute of the South|Stanford University|Cardiovascular Institute of the South|Stanford University|Cleveland Clinic|Cardiovascular Institute of the South|Stanford University
homecirculation arrhythmia electrophysiologyvol 16 7evaluating recommendations atrial fibrillation patients clinicians obtained chatbased artificial intelligence algorithms free accessresearch articlepdfepubaboutview pdfview epubsections toolsadd favoritesdownload citationstrack citationspermissions shareshare onfacebooktwitterlinked inmendeleyreddit jump tofree accessresearch articlepdfepubevaluating recommendations atrial fibrillation patients clinicians obtained chatbased artificial intelligence algorithms zahra azizi pouria alipour sofia gomez cassandra broadwin sumaiya islam ashish sarraju aj rogers alexander sandhu fatima rodriguez zahra azizizahra azizi httpsorcidorg0000000278970934 center digital health za cb si pouria alipourpouria alipour httpsorcidorg0000000207512995 department medicine mcgill university montreal qc canada pa sofia gomezsofia gomez department medicine stanford school medicine ca sg cassandra broadwincassandra broadwin center digital health za cb si sumaiya islamsumaiya islam httpsorcidorg0000000169019883 center digital health za cb si ashish sarrajuashish sarraju httpsorcidorg0000000316492110 department cardiovascular medicine cleveland clinic oh aj rogersaj rogers httpsorcidorg000000016585534x stanford university division cardiovascular medicine cardiovascular institute department medicine stanford university ca ajr ats fr alexander sandhualexander sandhu httpsorcidorg0000000332081143 stanford university division cardiovascular medicine cardiovascular institute department medicine stanford university ca ajr ats fr department cardiovascular medicine cleveland clinic oh fatima rodriguezfatima rodriguez correspondence fatima rodriguez md mph division cardiovascular medicine stanford university mail code 5687 453 quarry rd center academic medicine stanford ca 94304 email email address email protected httpsorcidorg0000000252260723 stanford university division cardiovascular medicine cardiovascular institute department medicine stanford university ca ajr ats fr originally published19 jun 2023httpsdoiorg101161circep123012015circulation arrhythmia electrophysiology 202316415417other versions articleyou viewing recent version article previous versions june 19 2023 ahead print atrial fibrillation af common arrhythmia worldwide1 crowdsourced information peertopeer recommendations online resulted dissemination sometimes inaccurate information af prevention management13 increasing availability artificial intelligence ai machine learning technologies consumer level patients likely turn modalities medical guidance13 however appropriateness advice provided af chatbased ai models yet investigated recently sarraju et al4 evaluated appropriateness chatgpts responses various questions preventive cardiologists commonly encounter clinical settings extended work evaluating performance differences patient clinician questions af using 2 popular chatbased language models chatgpt bing aieighteen prompts designed consultation clinicians expertise managing af reflect questions patients commonly ask af using public free version chatgpt v35 another 18 prompts reflected perspectives clinicians using chatgpt bing ai gpt4 question input platform 3 times figure4 patients questions classified 4 major categories overall af treatments medications daily lifestyle clinician questions began prompt physician included phrases based recent guidelines reference elicit higherlevel scientific medical responses ai enginesdownload figuredownload powerpointfigure evaluation atrial fibrillation recommendations patients clinicians online chatbased artificial intelligence algorithms patientfacing clinicianfacing questions input ai tools listed patientfacing questions categorized question topic overall treatment medication lifestyle question asked 3 times ensure consistency responses case inquiries clinicians asked machine provide evidence citations given recommendation output considered appropriate given response 3 instances along evidence case clinicianrelated questions correct based recent evidence guidelines set responses graded 2 clinicians cardiologist electrophysiologist case disagreement results checked third clinician reviewers agreement 89100 cases cohens kappa chatgpt 077 1 chatgpt text1 reference077 bing aitext077 reference077 af indicates atrial fibrillation afl atrial flutter ai artificial intelligence laa left atrial appendageeach set responses reviewed 3 expert clinicians categorized either appropriate inappropriate agreement reviewers determined calculating cohens kappa statistics appropriate response one contained accurate information also employed language deemed clear comprehensible user patientgeneral public clinician given different levels assumed medical literacy clinician question responses labeled inappropriate either incorrect based expert judgement contained inaccurate references language deemed unclear data support findings study available za zazizistanfordedu upon reasonable request za takes responsibility integrity study subject institutional review board approval request exemption human animal subjects utilizedthe overall appropriateness responses displayed figure level agreement reviewers determined kappa statistic 077 1 indicating high agreement overall 833 overall af 833 treatments 100 medications 100 daily lifestyle 714 patientinitiated prompts appropriate responses generated chatgpt however chatgpt yielded inaccurate responses questions particularly pertaining triggers af alcohol coffee figure clinicianrelated questions text accuracy chatgpt bing ai responses 333 666 reference accuracy 555 50 respectively cases appropriate responses cited references included current us european guidelines algorithms also able provide responses using appropriately cited primary literature however 2 instances relating af management chatgpt confabulated references trials existour findings demonstrate chatgpt may appropriately summarize certain general information regarding af patients completely accurate moreover found chatgpt bing ai lacked accuracy provide appropriate scientific basis recommendations aimed health care professionals suggesting tools may unsuitable clinical decisionmaking researchnotable shortcomings chatgpt patient questions included inability provide correct responses af triggers alcohol coffee consumption nuanced topics varying opinions medical literature may contribute disagreements seen among accurate responses provided chatgpt bing ai clinician focused queries certain cited references fictitious incorrectly cited commonly called ai hallucinations5 compared chatgpt bing ai higher response accuracy comparable frequency inaccurate referencesone limitation study use ai chatbots designed general public rather medical applications algorithms trained health care data possible accuracy scientific merit responses clinician queries would improvein conclusion current ai chatbots may useful tool provide cursory overview diseases patients general public however tool showed limited appropriateness clinicianfacing questions study tech rapidly evolving future models may updated real time avoid wrong answers due outdated information finally imperative note tool able replace humans intuition situational awareness importantly clinical judgment much greater accuracy clinical prowess expected clinicians managing diseases atrial fibrillationarticle informationsources fundingdr rodriguez funded grants nih national heart lung blood institute 1k01hl144607 american heart associationharold amos faculty development program doris duke foundation grant 2022051 dr azizi funded american heart association health tech sfrn fellowshipnonstandard abbreviations acronymsafatrial fibrillationaiartificial intelligencedisclosures dr rodriguez reports consulting relationships healthpals novartis amgen novonordisk cec astrazeneca remaining authors nothing disclosefootnotesthis article sent andrew e epstein md guest editor review expert referees editorial decision final dispositionfor sources funding disclosures see page 416correspondence fatima rodriguez md mph division cardiovascular medicine stanford university mail code 5687 453 quarry rd center academic medicine stanford ca 94304 email frodrigustanfordedureferences1 wilson rush kl engaging patients atrial fibrillation selfcare opportunities future research los angeles ca sage publications sage ca 202110911092google scholar2 palm p qvist rasmussen tb christensen sw hkonsen sj risom ss educational interventions improve outcomes patients atrial fibrillationa systematic reviewint j clin pract 2020 74e13629 doi 101111ijcp13629crossrefmedlinegoogle scholar3 mann dl artificial intelligence discusses role artificial intelligence translational medicinejacc basic transl sci 2023 8221223 doi 101016jjacbts202301001crossrefmedlinegoogle scholar4 sarraju bruemmer van iterson e cho l rodriguez f laffin l appropriateness cardiovascular disease prevention recommendations obtained popular online chatbased artificial intelligence modeljama 2023 329842844 doi 101001jama20231044crossrefmedlinegoogle scholar5 alkaissi h mcfarlane si artificial hallucinations chatgpt implications scientific writingcureus 2023 15e35179 doi 107759cureus35179crossrefmedlinegoogle scholar eletters0eletters relate article recently published journal forum providing unpublished data comments reviewed appropriate use tone language comments peerreviewed acceptable comments posted journal website comments published issue indexed pubmed comments longer 500 words posted online references limited 10 authors article cited comment invited reply appropriatecomments feedback ahaasa scientific statements guidelines directed ahaasa manuscript oversight committee via correspondence pagesign submit response article previous back top next figuresreferencesrelateddetailscited hillmann h angelini e karfoul n feickert muellerleisse j duncker 2023 accuracy comprehensibility chatbased artificial intelligence patient information atrial fibrillation cardiac implantable electronic devices europace 101093europaceeuad369 261 online publication date 28dec2023 july 2023vol 16 issue 7 advertisement article informationmetrics 2023 american heart association inchttpsdoiorg101161circep123012015pmid 37334705 originally publishedjune 19 2023 keywordsarrhythmiaatrial fibrillationpatientsphysicianspdf download advertisement subjectsatrial fibrillationmachine learning artificial intelligence