Michael Yurcheshen|Wilfred R. Pigeon|Carolina Z. Marcus|Jonathan A. Marcus|Susan Messing|Kevin Nguyen|Jennifer Marsella
Interrater reliability between in-person and telemedicine evaluations in obstructive sleep apnea
University of Rochester|University of Rochester|University of Rochester|University of Rochester|University of Rochester|Saddleback Memorial Medical Center|University of Rochester
free accessscientific investigationsinterrater reliability inperson telemedicine evaluations obstructive sleep apnea michael e yurcheshen md faasm wilfred pigeon phd carolina z marcus md jonathan marcus md susan messing ms kevin nguyen md jennifer marsella md michael e yurcheshen md faasm address correspondence michael e yurcheshen md faasm ur sleep center 2337 south clinton avenue rochester ny 14610 email email address email protected university rochester school medicine dentistry rochester new york wilfred pigeon phd university rochester school medicine dentistry rochester new york carolina z marcus md university rochester school medicine dentistry rochester new york jonathan marcus md university rochester school medicine dentistry rochester new york susan messing ms university rochester school medicine dentistry rochester new york kevin nguyen md saddleback medical center laguna hills california jennifer marsella md university rochester school medicine dentistry rochester new york published onlinejuly 1 2021httpsdoiorg105664jcsm9220cited by6sectionsabstractepubpdf shareshare onfacebooktwitterlinkedinredditemail toolsadd favoritesdownload citationstrack citations aboutabstractstudy objectiveswe examined telemedicine evaluation compares facetoface evaluation identifying risk sleepdisordered breathingmethodsthis randomized interrater reliability study 90 participants referred university sleep center participants evaluated clinician investigator seeing patient inperson randomized second clinician investigator performed patient evaluation online via audiovideo conferencing primary comparator pretest probability obstructive sleep apnearesultsthe primary outcome comparing pretest probability obstructive sleep apnea showed weighted kappa value 0414 standard error 0090 p 002 suggesting moderate agreement 2 raters kappa values secondary outcomes varied widely kappa values lower physical exam findings compared historical elementsconclusionsevaluation pretest probability obstructive sleep apnea via telemedicine moderate interrater correlation inperson assessment low degree interrater reliability physical exam elements suggests telemedicine assessment obstructive sleep apnea could hampered suboptimal physical exam employing standardized scales obstructive sleep apnea performing telemedicine evaluations may help riskstratification ultimately lead tailored clinical managementcitationyurcheshen pigeon w marcus cz et al interrater reliability inperson telemedicine evaluations obstructive sleep apnea j clin sleep med 202117714351440brief summarycurrent knowledgestudy rationale telemedicine promising technology widely used practice sleep medicine accuracy telemedicine compared inperson assessment evaluating patients obstructive sleep apnea still unknown current study interrater reliability study comparing telemedicine inperson evaluator assessing pretest probability obstructive sleep apnea community populationstudy impact telemedicine inperson investigators moderate agreement evaluating pretest probability mild moderate severe obstructive sleep apnea current study underscores need consider standardized processes optimize telemedicine support online clinicians ability accurately assess sleepdisordered breathingintroductiontelemedicine changing health care delivery clinical medicine increasing demand sleep medicine specialists online consultants potential deliver care despite distance transportation concerns pandemic conditions patients sleep medicine providers insurance carriers industry stake development telemedicine servicesobstructive sleep apnea osa common expensive medical condition estimated afflict 2 20 united states population estimated annual cost nearly 150 billion dollars us economy13 suspicion osa based largely clinical history additional detail gained physical examination survey tools also play role assessing osa47 condition common evaluated history examination suited remote videoaudio communication osa good candidate telemedicine evaluation based promise industry groups looking optimize technology8telemedicine consultation studied various disease processes including parkinson disease emergency ocular disorders acute pediatric illnesses913 type assessment led substantial satisfaction medical care well reduced travel time distance9 regarding telemedicine management osa literature lacks large communitybased outcome studies seem promise technology based number studies 1 study 60 osa patients felt comfortable engaging virtual consultations14 likewise community population showed equal satisfaction inperson telemedicine evaluations sleep condition15 one study showed small decrease continuous positive airway pressure compliance among participants managed teleconsultation16 contrast continuous positive airway pressure participants recruited veteran population similar functional outcomes telemedicine compared inperson management17these studies encouraging limitations lacked control group others limited numbers participants even compelling studies limited generalizability veteran population only17 although sleep field rapidly moving toward adoption telemedicine still gaps understanding care model instance community studies examining accuracy remote assessments compared gold standard inperson evaluation18 many treatment decisions including decision order athome inlab sleep testing based patients pretest probability sleep apnea19 telemedicine useful patient assessments sufficiently accurate drive type clinical decisionmakingthe present randomized clinical trial aimed compare telemedicine inperson assessment sleepdisordered breathing hypothesized would high interrater reliability telemedicine inperson assessors determining pretest probability osa secondary aims included interrater reliability historical physical exam elements suggestive condition well comparison raters interpreting home sleep apnea testingmethodsstudy design randomizationwe conducted randomized blinded interrater reliability study comparing impressions clinician seeing patient inperson clinician seeing patient via telemedicine subjects recruited march 2017 january 2019 study design outlined figure 1 study design based comparing assessments inperson telemedicine physicians rather single telemedicine rater different time points 2 different telemedicine raters design chosen compromise given rapid pace clinical assessmenttesting impracticality purposely delaying repeat assessment would minimize evaluator memory biasfigure 1 study designdownload figurethree american board medical specialties sleepboard eligiblecertified clinicians mey czm jm underwent group training familiarize study protocol outcome assessments training raters reviewed 20 theoretical case histories reach consensus pretest probability osa training included description physical exam include simulated video footage oral cavity elected use 3 different raters study expedite recruitment rater responsible recruiting roughly onethird cohort minimize bias may stemmed using 2 ratersconsenting participants asked complete demographic profile epworth sleepiness scale inperson clinical encounter inperson investigator reviewed patients electronic medical record conducted history physical examination investigator completed clinical impression battery included responses relating primary secondary endpoints participants upon consent randomized 1 2 raters utilizing randomized block design size 4 online clinical encounter conducted audiovisual conferencing application zoom san jose ca telemedicine encounter scheduled within 5 business days teleevaluation included record review history brief noninvasive examination oral cavity using patients webenabled camera incandescent light source telemedicine investigator blinded inperson assessment blinding ensured thirdparty audit sample 5 study participants participant completed repeat epworth sleepiness scale around time telemedicine encounter completed survey questionnaire regarding experience telemedicinefor participants completed home sleep apnea testing hsat type 3 home sleep testing device braebon medibyte kanata canada respironics alice nightone murrysville pa used devices recorded following signals snoring body position heart rate oxygen saturation airflow nasal pressure thoracoabdominal movement studies scored experienced new york state licensed sleep technologist unaware participant randomization respiratory scoring performed accordance published criteria20 study interpreted inperson telemedicine clinicians independently impression mild moderate severe sleep apnea uninterpretable study recordedthe protocol study approved university rochester institutional review board complied ethical principles declaration helsinki accordance international conference harmonization good clinical practice guideline participants provided written informed consent study enrollmentparticipantseligible participants recruited general referrals university rochester sleep center men women 3070 years old referred reason asked participate potential participants needed sufficient computer literacy access highspeed internet minimum 384 kbps computing device appropriate video camera minimum 640 480 resolution 30 framess microphone patients dementia severe psychiatric developmental illness complete hearing visual loss fluent english excluded participants conducted telemedicine evaluation domicile private area participants offered 25 gift card completed phases study participants also provided incandescent pen light evaluator examined oral cavity incandescent light source helped minimize white contrast issues oral cavity seen brighter led lighting usedmeasuresthe studys primary outcome assess pretest probability sleep apnea high moderate low secondary outcome measures included interrater reliability historical physical exam findings including snoring volume none mild moderate severe sure witnessed apneas yes sure degree daytime sleepiness none mild moderate severe modified mallampati class 14 airway crowding overjet distance maxillarymandibular relationship 0 mm 03 mm 3 mm tonsil size 04 severity sleep apnea based hsat inconclusive mild moderate severestatistical methodssample sizea sample size 80 participants selected based 80 power significance level 05 detect kappa 060 test kappa 04 assuming overall clinic population risk profile 30 low 30 moderate 40 high pretest probability osa recruited 90 participants study based assumed 10 dropoutstatistical analysiskappa weighted kappa evaluated inperson vs telemedicine evaluations weighted kappa utilized cicchettiallison weighting scheme study missing data raterswhen score values relatively equally exercised along ordinal scale percent agreement kappa demonstrate concordance categories sparsely populated kappa values drop precipitously evaluate amount agreement followed landis koch suggestions 0 poor agreement 00020 slight agreement 021040 fair agreement 041060 moderate agreement 061 080 substantial agreement 081100 almost perfect agreement21resultsninety 90 participants enrolled study 58 participants completed entire protocol consort consolidated standards reporting trials diagram figure 2 depicts participant progress throughout study based posthoc audit electronic health record none telemedicine investigators accessed evaluation primary investigator embargoed timeframe 2 evaluations one participant discovered hsat results shared telemedicine investigator since telemedicine investigator access inperson investigators assessments decided include participant analysis reason noncompletion losstofollowup participant showing telemedicine appointment 32 participants 37 participants underwent hsat evaluation evaluation stopped technical concernsfigure 2 consort consolidated standards reporting trials diagram hsat home sleep apnea testingdownload figureparticipant characteristics included table 1 58 completers mean age 499 years enrolled participants 11 africanamerican 4 hispanic 1 asian american completing participants 7 africanamerican 4 hispanic 0 asian american baseline epworth sleepiness scale score 94table 1 participant characteristicsall n 90completers n 58 65age mean sd y4884 8044993 810female 533552raceethnicity asian1100 blackafrican american11169 hispanic4434 whitecaucasian833897income 50000200190 50000100000333379 100000367345 preferred answer10086average business days enrollmentna498 298average days enrollmentna648 468na available sd standard deviationof completed appointments 13 participants completed telemedicine appointment later 5businessday target window ranging 624 days delay telemedicine appointment due patient factorspreferences reschedulingstatistics patient evaluator satisfaction process timing duration telemedicine appointment preparation another manuscript reported hereinterrater reliabilitythe results primary secondary outcomes included table 2 including percentage agreement kappa weighted kappa appropriatetable 2 interrater reliability inperson vs telemedicine physician evaluations obstructive sleep apneaevaluation agreementkappa seweighted kappa sep valuepretest probability osa low moderate high53440286 1000414 090002level daytime sleepiness44830225 0920391 083004snoring maximum level63160354 1070353 1090002apneas witnessed third party77430628 0860702 079 0001apneas witnessed third party yn91300825 084 0001modified mallampati score48280179 0930200 091067tonsils yn64290134 105326overjet82460069 0220044 014496after reviewing home sleep apnea test mild moderate severe91890872 0700899 056 0001osa obstructive sleep apnea se standard error yn yesnobased sample size protocol completers quadratic weighted kappa value 0414 standard error se 0090 p 002 determining pretest probability osa based clinical evaluation value consistent moderate agreement evaluators posthoc analysis sample categories compressed high includes high moderaterisk categories low pretest probability linear weighted kappa value calculated 028 se 017 95 confidence interval 0062 case value suggests fair agreement evaluatorsof secondary clinical endpoints historical element witnessed apneas highest weighted kappa 0702 se 0079 p 0001 assessment physical exam finding overjet lowest weighted kappa reach statistical significance 0044 se 0014 p 496regarding home sleep testing based sample size 37 participants hsat completers dual impressions calculated weighted kappa value 0899 se 0056 p 0001 determining severity osa based home sleep testing value consistent almost perfect agreement evaluatorsdiscussionalthough telemedicine promising tool delivery sleep care accuracy compared inperson evaluation uncertain present study first evaluate accuracy telemedicine determining pretest probability obstructive sleep apnea community populationour results show moderate agreement inperson telemedicine evaluator determining pretest probability obstructive sleep apnea much higher level agreement noted secondary endpoint witnessed apneas low elements physical examination agreement raters ultimately determining degree sleep apnea looking hsat results almost perfect according published criteria21telesleepmedicine becoming increasingly visible adoption happening quickly middle coronavirus pandemic middecade review patient attitudes toward sleep telemedicine 63 respondents surveyed stated would comfortable willing try telemedicine visits sleep appointments14 considering shortage sleep medicine providers time expense safety traveling conducting inperson appointments telemedicine evaluation helping improve access important sleep field continue optimize technology clinical standards toolpretest probability sleep apnea selected primary endpoint study major determinant developing evaluation management plans19 uncomplicated patients moderate high pretest probability significant osa may well suited home testing whereas patients low pretest probability may referred testing still patients may significant pretest probability mild sleep apnea inlab testing may appropriate pretest probability drives clinical decisionmakingformulation pretest probability driven 1 history 2 physical exam analysis primary aim suggests reasonable imperfect interrater reliability deciding pretest probability osa par interrater reliability medical conditions less others2223 reliability primary aim stands contrast substantial agreement one historical elements witnessed apneas poor unclear agreement physical exam findings modified mallampati class overjet distance tonsil size results suggest uncertainty introduced physical exam may tempered clinical picture generated historyin study participants home sleep testing excellent agreement determining severity sleep apnea based hsat result supports similar finding veteran population suggests telemedicine provider unlikely miss sleep apnea hsat appropriate patients identified17the results suggest challenges way forward developing evaluationmanagement plans via telemedicine minimize uncertainty introduced telemedicine assessment standardized protocoldriven approach predictive survey tools ie stopbang berlin questionnaire could help stratify patients risk47 although tools employed study value predicting sleep apnea performed remotely might increase accuracy telemedicine evaluations recently structured interviews assess wide variety sleep disorders developed2425 could play role telemedicine assessment although time intensive aforementioned sleepapnea focused questionnairesalso consideration given optimize physical examination portion remote assessment renewed interest role physical exam sleep medicine best unify descriptions airway26 telemedicine additional considerations might include better lighting higher resolution cameras use inperson patient presenters new technologies certainly play role improving remote physical exam27 telemedicine evaluation opens unconventional avenues clinician least include physical examination patients sleep environment evaluator may able get additional information via telemedicine could directly appreciate office ie easy chair bedroom continuous positive airway pressure bedside etc capability directly studied trialthere several limitations study significant absence inpersontoinperson telemedicinetotelemedicine comparison interrater reliability published literature includes limited examples inperson interrater reliability sleep disorders one study using structured interview template showed kappa coefficient 073 obstructive sleep apnea substantially better weighted kappa 041424 another study using different structured interview template assessed kappa secondary outcome measure result closer kappa obstructive sleep apnea 03825 still studies employed designs different present study direct comparison imperfectlikewise interrater reliability study could designed delayed repeated encounters using single rater timely assessments 2 raters assessments employing telemedicine28 given practical concerns scheduling patient 2 different assessments sleep testing study design employed 2 different raters across 2 different settings inperson vs telemedicine employed type mixed methodology used telemedicine interrater reliability studies22232930 imagine introduction extra variable 2 different raters 2 different settings negatively impacted kappa coefficients might expect values would even stronger raters utilized telemedicine platform limitations include individual evaluator practices temporal dispersion 2 evaluations also concern tried minimize encouraging 5businessday timeframe also sample size smaller anticipated due relatively high number patients lost followup shows second visit higher expected despite confirmation adequate technology multiple attempts schedule teleassessment phone email small financial incentive perhaps postpandemictelemedicinefamiliar mindset higher financial incentive could aided retention feel noshow rate necessarily reflective missed telemedicine appointments clinical practice addition although raters received standardized case training protocol training include video review airway shortcoming may impacted raters impressions due study design limitations inperson evaluation always predated telemedicine evaluation potentially introducing patient bias telemedicine assessment also number participants study small determine confidence 2 raters closely aligned 2 pairings last population recruited study mostly white higher socioeconomic status although impact demographic uncertainthe present study demonstrated promising signal determining accuracy telemedicine encounters osa ultimately outcome costanalysis studies needed determine utility promising technologydisclosure statementall authors seen approved manuscript work study performed ur sleep center university rochester rochester ny study funded grant american academy sleep medicine foundation aasm foundation grant 163fp17 drs yurcheshen c marcus j marcus messing received financial support grant dr yurcheshen served clinical trials consultant jazz pharmaceuticals harmony biosciences none consulting activities involve subject matter present study dr pigeon subinvestigator observational trials funded pfizer inc abbvie inc unrelated manuscript employee us department veterans affairs va views opinions expressed herein necessarily represent va us government drs c marcus j marcus marsella messing nguyen report conflicts interestabbreviationshsathome sleep apnea testosaobstructive sleep apneasestandard errorreferences1 young palta dempsey j skatrud j weber badr occurrence sleepdisordered breathing among middleaged adults n engl j med 19933281712301235 httpsdoiorg101056nejm199304293281704 crossrefgoogle scholar2 frost sullivan hidden health crisis costing america billions underdiagnosing undertreating obstructive sleep apnea draining healthcare system darien il american academy sleep medicine 2016 httpsaasmorgadvocacyinitiativeseconomicimpactobstructivesleepapnea accessed april 5 2021 google scholar3 peppard pe young barnet jh et al increased prevalence sleepdisordered breathing adults j epidemiol 2013177910061014 httpsdoiorg101093ajekws342 crossrefgoogle scholar4 vana kd silva ge goldberg r predictive abilities stopbang epworth sleepiness scale identifying sleep clinic patients high risk obstructive sleep apnea res nurs health 20133618494 httpsdoiorg101002nur21512 crossrefgoogle scholar5 netzer nc stoohs ra netzer cm clark k strohl kp using berlin questionnaire identify patients risk sleep apnea syndrome ann intern med 19991317485491 httpsdoiorg10732600034819131719991005000002 crossrefgoogle scholar6 fenton heathcote k bryce r et al utility elbow sign diagnosis osa chest 20141453518524 httpsdoiorg101378chest131046 crossrefgoogle scholar7 prasad kt sehgal agarwal r nath aggarwal behera dhooria assessing likelihood obstructive sleep apnea comparison nine screening questionnaires sleep breath 2017214909917 httpsdoiorg101007s1132501714954 crossrefgoogle scholar8 singh j badr ms diebert w et al american academy sleep medicine aasm position paper use telemedicine diagnosis treatment sleep disorders j clin sleep med 2015111011871198 httpsdoiorg105664jcsm5098 linkgoogle scholar9 bowman rj kennedy c kirwan jf sze p murdoch ie reliability telemedicine diagnosing managing eye problems accident emergency departments eye lond 2003176743746 httpsdoiorg101038sjeye6700489 crossrefgoogle scholar10 siew l hsiao mccarthy p agarwal lee e chen l reliability telemedicine assessment seriously ill children pediatrics 20161373e20150712 httpsdoiorg101542peds20150712 crossrefgoogle scholar11 dorsey er venkataraman v grana mj et al randomized controlled clinical trial virtual house calls parkinson disease jama neurol 2013705565570 httpsdoiorg101001jamaneurol2013123 crossrefgoogle scholar12 venkataraman v donohue sj biglan km wicks p dorsey er virtual visits parkinson disease case series neurol clin pract 201442146152 httpsdoiorg10121201cpj0000437937633475a crossrefgoogle scholar13 schneider r dorsey er biglan k telemedicine care nursing home residents parkinsonism j geriatr soc 2016641218220 httpsdoiorg101111jgs13909 crossrefgoogle scholar14 kelly jm schwamm lh bianchi mt sleep telemedicine survey study patient preferences isrn neurol 20122012135329 httpsdoiorg1054022012135329 crossrefgoogle scholar15 parikh r touvelle mn wang h zallek sn sleep telemedicine patient satisfaction treatment adherence telemed j e health 2011178609614 httpsdoiorg101089tmj20110025 crossrefgoogle scholar16 comadelcorral mj alonsoalvarez ml allende et al reliability telemedicine diagnosis treatment sleep apnea syndrome telemed j e health 2013191712 httpsdoiorg101089tmj20120007 google scholar17 fields bg behari pp mccloskey et al remote ambulatory management veterans obstructive sleep apnea sleep 2016393501509 httpsdoiorg105665sleep5514 crossrefgoogle scholar18 zia fields bg sleep telemedicine emerging fields latest frontier chest 2016149615561565 httpsdoiorg101016jchest201602670 crossrefgoogle scholar19 rosen im kirsch db chervin rd et al american academy sleep medicine board directors clinical use home sleep apnea test american academy sleep medicine position statement j clin sleep med 2017131012051207 httpsdoiorg105664jcsm6774 linkgoogle scholar20 berry rb quan sf abreu ar et al american academy sleep medicine aasm manual scoring sleep associated events rules terminology technical specifications version 26 darien il american academy sleep medicine 2020 google scholar21 landis jr koch gg measurement observer agreement categorical data biometrics 1977331159174 httpsdoiorg1023072529310 crossrefgoogle scholar22 awadallah janssen f krber b breuer l scibor handschu r telemedicine general neurology interrater reliability clinical neurological examination via audiovisual telemedicine eur neurol 201880289294 httpsdoiorg101159000497157 crossrefgoogle scholar23 handschu r littmann r reulbach u et al telemedicine emergency evaluation acute stroke interrater agreement remote video examination novel multimedia system stroke 2003341228422846 httpsdoiorg10116101str000010204370312e9 crossrefgoogle scholar24