Ashrita Saran|Howard White
Evidence and gap maps: a comparison of different approaches
evidence maps relatively new approach systematically identify report range research activity broad topic areas policy domains although go various names common feature adopting systematic approach mapping begins broad question theme issue defines scope evidence map reasons discussed adopt term evidence gap map refer specific approach proposed campbell evidence map refer collectively range approaches adopted describe part 2 evidence mapping various forms around least 15 years range different approaches evidence mapping synthesis developed support evidence informed policy making 2016 miakelye hempel shanman shekelle 2016 conducted systematic review evidence maps health sector authors identified 145 titles 34 publications explicitly presented evidence maps five publications used mapping methodology without presenting map authors conclude diversity definition methods mapping stakeholders cannot necessarily know expect commission evidence map seek identify existing maps addition repository evidence maps maps difficult locate authors less likely build existing approaches one purpose paper propose general principles evidence maps maps often provide foundation focused research synthesis used guide users high quality research inform research priority setting help define focus evidence synthesis systematic reviews katz et al 2003 health sector maps intended provide clinicians researchers clients caregivers tools guidance engage evidencebased decision making maps allow identification research gaps particularly important interventions implemented without insufficient evidence katz et al 2003 popularity mapping grown recent years map maps international development interventions identified 73 maps half completed maps commissioned 2016 phillips et al 2017 paper reviews range mapping methodologies adopted across different sectors motivation paper develop general principles maps particularly inform development standards campbell collaboration basis publication evidence gap maps campbell library approaches discussed include systematic map bates clapton coren 2007 oakley gough oliver thomas 2005 salina bates 2006 peersman 1996 evidence maps liu parker hetrick callahan 2010 evidencebased policing matrix lum koper telep 2011 evidence mapping parkhill et al 2011 bragge et al 2011 scientific uncertainties jacobson 2014 evidence gap maps snilstveit vojtkova bhavsar gaarder 2013 papers mentioned discussed presented overview different types evidence mapping scoping synthesis focus various approaches towards evidence syntheses products whereas focus paper methodological approach organisations working evidence maps purposes paper present brief history evidence mapping turning analysis approach evidence mapping broken stages definition map scope inclusion criteria critical appraisal screening searching coding presentation reporting use maintenance conclude general principles evidence mapping earliest evidence maps identified 2003 year saw publication map yale prevention research centre collaboration complementary alternative medicine cam katz williams girard goodman 2003 see table 1 year evidence policy practice information coordinating centre eppicenter published review personal development planning improving student learning distinguishing systematic map research undertaken systematic synthesis evidence says gough kiwan sutcliffe simpson houghton 2003 two years later first visual map published evidence based policing matrix rosenberg knox 2005 2005 eppicentre published mapping methodology adapted following year social care institute excellence scie response lack empirical data answer specific questions using systematic reviews perceived need methodology describe literature broader field interest typically covered single review 2010 3ie produced first evidence map labelled evidence gap map organisations experience preparing maps range topics international development reviewed snilstveit snilstveit vojtkova bhavsar gaarder 2013 several organizations international development sphere since worked 3ie independently develop evidence maps typically variations design include international rescue committee irc 2016 usaid vinogradova heaner 2016 sightsavers virendrakumar jolley gordon bascaran schmidt 2016 beam exchange market research agencies producing maps different approach defines evidence map differently review definitions see annex 1 full definitions shows following eight components contained definitions see table 21 perhaps missing definitions explicit statement mapping commonly tells us evidence evidence says though maps include information effects point often misunderstood commissioning evidence maps made clear front case definitions suggest following comprehensive definition elements square brackets may considered optional evidence gap map systematic visual presentation availability relevant evidence effects particular policy domain evidence identified search following prespecified published search protocol map may accompanied descriptive report summarize evidence stakeholders researchers research commissioners policy makers practitioners evidence maps summarize evidence evidence says evidence gap maps two sets relatives first relatives evidence synthesis research outputs second different evidence platforms figure 1 shows different evidence synthesis products inverse relationship scope content systematic reviews narrowest scope deep content coding analyzing reporting effect sizes intervention characteristics contextual factors review reviews may broader scope may restricted depth analysis evidence gap maps still broader scope report far limited range information reviews primary studies include megamap map broad scope shows maps reviews primary studies map maps reports egms policy space phillips et al 2017 evidence maps break content spectrum reviews reviews reviews report evidence says whereas many maps report evidence says scope content different evidence synthesis products evidence maps may way curating brokering body evidence make accessible users distinguish three main evidence platforms successively higher levels curation brokering beyond evidence portals evidencebased guidelines checklists contain recommendations directions last three different databases maps databases maps take users back research studies whereas portals etc present evidence inform decisionmaking without user needing consult research directly paper focused evidence maps companion paper discuss evidence portals nearly agencies reviewed evidence maps generally broad thematic scope covering range interventions outcomes example 3ie maps generally cover sector broad subsector forest conservation puri nath bhatia glew 2016 youth transferable skills rankin et al 2015 peacebuilding cameron et al 2015 primary secondary education sbu mapped gaps uncertainties topics attention deficit hyperactivity disorder adhd home care evidencebased nursing treatment people schizophrenia department veteran affairs topics women veteran health research literatureelisheva et al 2017 evidence map mindfulness hempel et al 2014 since 2007 global evidence mapping initiative mapped 1644 neurotrauma publications covering fiftythree high priority questions bragge et al 2011 example evidence map global evidence mapping initiative social communication disorder following brain injury social care institute excellence maps cover areas extent impact parental mental health problems families acceptability salina bates 2006 accessibility effectiveness interventions second examines recovery approach day services adult mental health care sharif brown rutter 2008 january 2018 epistemonikos database contains 115000 documents 22649 systematic reviews 76834 primary studies 17144 structured summaries evidence database available nine different languages arabic german english spanish italian dutch portuguese chinese french contains 7266 official translations 642 collaborative translations 899022 automated translations rada prez capurro 2012 database supports maps evidence found database searches maps show primary studies reported systematic reviews found particular search izas world labour presents reviews issues related labour market policy related economic issues immigrant workers depress wages native workersgiovanni 2014 reviews include actual geographic map evidence comes scope sightsavers evidence map limited eye health published five evidence maps far cataract diabetic retinopathy glaucoma trachoma refractive error beam exchange evidence map focused marketbased approaches promote economic development improve access services reduce poverty robinson rustsmith 2017 irc published five evidence maps covering organizations five main outcomes areas health education economic wellbeing safety power addition three crosscutting maps cash transfer interventions service delivery interventions interventions humanitarian emergencies scope evidence maps collaboration environmental evidence typically broader examples include woodfuel value chain sola et al 2017 farm land abandonment high altitude haddaway styles pullin 2014 exception general broad scope maps evidence based policing matrix one outcome reducing crime disorder rather set outcomes odi marcus cunningham 2016 summarized evidence concerning initiatives outcomes four evidence maps one providing visual overview evidence database one focused initiatives young people acting agents one young people advocates one summarising evidence initiatives involved agents advocates activity unicefinnocenti published evidence gap map egm evidence base adolescent interventions low middleincome countries lmics thematic scope broadly corresponds unicef adolescent wellbeing outcome domains protection participation livelihoods agencies reviewed state prespecified inclusion criteria evidence maps inclusion criteria maps use picos approach population intervention relevant comparison groups effectiveness studies though specifying comparisons usually included map inclusion criteria outcomes eligible study designs practice usually iteration studies identified finalizing intervention outcome categories agencies produce maps including systematic reviews primary studies exceptions evidencebased policing matrix includes primary studies sbu sightsavers maps include systematic reviews 3ie egms maps effectiveness studies primary studies studies effects impact evaluations unlike systematic reviews evidence maps may include ongoing studies especially case ongoing reviews since registering review protocols common practice remains less case primary studies 3ie staff argued term egm restricted sort map snilstveit vojtkova bhavsar gaarder 2013 compare evidence gap maps supported international initiative impact evaluation 3ie evidence mapping synthesis products scoping reviews systematic review rapid evidence assessment propose gap maps focus studies assessing intervention effectiveness well systematic reviews studies mapping scoping approaches may include broader range evidence address questions intervention effectiveness epistemonikos uses systematic reviews fundamental piece database architecture includes articles linked included systematic review ie overviews reviews including evidencebased policy briefs guidelines primary studies included systematic reviews structured summaries evidence agencies restrict included studies specific ways unicefinnocenti exclude transferable skills youth employmentrelated interventions outcomes egms address beam exchange evidence gap includes empirical data studies earlier 2000 original framework document making markets work better poor m4p developed critical appraisal quality studies indicates confidence study findings example 3ie maps systematic reviews colour coded green orange red corresponding high medium low confidence study findings 3ie conduct critical appraisal primary studies grounds would time consuming given time frame maps usually produced 36 months 2017 beam exchange updated original evidence inclusion protocol include additional evidence quality grading system assesses evidence documents either high confidence represented purple bubbles low confidence represented orange bubbles critical appraisal applied using prespecific critical appraisal tool chechklist 3ie unicefinnocenti sightsavers use adapted version sure checklist sbu odi apply amstar checklist epistemonikos presents critical appraisal articles included structured summaries cee critical appraisal formalised assessment reliability risk bias individual studies may performed extent typically restricted assessment study internal validity quality susceptibility bias since external validity generalisability cannot typically assessed broad topics often investigated systematic maps evidence based policing matrix reports primary studies critical appraisal performed using scientific methods scale sms critical appraisal presented scie gemi recommendation critical appraisal included studies desirable performed unless resources allow search comprehensive systematic similar systematic review search strategy sometimes purposive impact evaluations choice source varies among organisations may include considerable overlap search strategy adopted organizations evidence maps uniform adoption comprehensive systematic search involves search across sources found relevant research question however 3ie search ie may purposive key difference source search 3ie involve hand searching journals literatures evidence based policing matrix specifically involves searching website professional government organization epistemonikos articles coming different databases deduplicated using software manual revision borderline cases coding captures key features included studies allowing analysis map data captured dimensions map filters applied restrict map subset evidence userdetermined criteria following domains used different organisations classifying included studies table 3 provides overview organizations use study characteristics addition study publication details course coded users locate evidence underlying map data represented discussed next section reporting presentation typically intervention categories outcome categories study designs used coding methodologies evidence mapping representation outcome categories 3ie gap maps evidence based policing matrix differ evidence based policing matrix key intervention dimensions specificity strategy level proactivity intervention nature target evaluation design outcome direction statistical significance effect used data extraction 3ie studies coded according relevant intervention outcome categories included framework detailed study characteristics included gem maps use codes international classification functioning disability health icf ebpm also exception map report evidence says size direction significance effects epistemonikos document uploaded metadata relational panel allows entering document metadata documents category like primary studies systematic reviews overviews documents relation five approaches used presentation evidence maps different organisations 5d bubble plot 3dmatrix 2dmatrix geographical map descriptive report withwithout visualisation latter two cases additional dimensions may added use different colors symbols size plot studies map approaches described turn 5dbubble format used evidence based synthesis program bubble plot shows five dimensions display information x axes show effect size literature respectively bubble reflects three dimensions separate bubble intervention size bubble representing number reviews intervention color bubble intervention category figure shows example bubble plot evidence map mindfulness evidence based synthesis program source evidence based synthesis program httpswwwncbinlmnihgovpubmed25577939 3dmatrix used evidence based policing matrix display single studies three dimensions intervention categories x axis defined target individual group etc specificity intervention general focused degree proactivity reactive highly proactive study plotted circle triangle color coded show study findings ie effects added fourth dimension map approach shows studies common intervention referred realms effectiveness realms effectiveness said provide insights nature commonalities effective police strategies used police agencies guide various aspects operations however approach formal evidence synthesis encourages votecounting approach assess evidence matrix mapped 97 police intervention studies source evidence based policing matrix httpscebcporgevidencebasedpolicingthematrix 3ie present evidence gap maps 2dinterventionoutcome framework provides links summaries included studies traffic light color coding used indicate quality systematic review evidence gap map users visually assess state evidence field use symbols assessment made primary studies size circle represents number studies adolescent sexual health egm source 3ie adolescent sexual reproductive health evidence gap map httpgapmaps3ieimpactorgevidencemapsadolescentsexualandreproductivehealthevidencegapmap evidence gap map adolescent wellbeing low middleincome countries protection participation financial material wellbeing source unicef innocenti httpswwwunicefircorgevidencegapmap sightsaver egms based matrix framework designed capture evidence specific interventions focus areas focus areas reviews identified found along x axis strength evidence shown along axis bubbles matrix cells denote existence systematic literature review examining relevant focus area clicking bubble open summary page strength evidence presented reviews categorised cataract gap map source sightsaver httpsresearchsightsaversnetgapmapscataractgapmap beam evidence map database compiles presents robust evidence impact effectiveness market system approaches vertical axis type intervention contributed results described document horizontal axis results level evidence document contains beam exchange evidence map source beam exchange httpsbeamexchangeorgresourcesevidencemap matrix evidence table displaying systematic reviews answering question studies included reviews matrix displays systematic reviews yaxis studies included reviews xaxis systematic reviews table address question interest ie melatonin jet lag broader question includes question interest ie melatonin promoting healthy sleep matrix evidence epistemonikos source epistemonikos httpswwwepistemonikosorgendocuments934b7d3da7216f697504b80170b87b58ec04709bmatrix irc evidence maps available online tool available downloadable excel format health map source irc httpswwwrescueorgresourcestrategy2020outcomesandevidenceframeworkevidencemaps cells colorcoded based studys conclusion young people agents advocates development source odi httpswwwodiorgpublications10653youngpeopleagentsandadvocatesdevelopment iza world labor article map indicates countries empirical evidence specific topic exists list key additional references provided article provides basis map empirical evidence exists one particular country country shown different color varying yellow different shades green colour country indicate classified according classifications used global competitiveness report gcr 20142015 schwab 2014 world economic forums wef center global competitiveness performance country categorized gcr colored gray feature seen assisting users assist transferability findings given study context captured gcr classification number shown inside pin indicates many relevant academic studies address policy question immigrant workers depress wages native workers evidence map source iza httpswolizaorgarticlesdoimmigrantworkersdepressthewagesofnativeworkersmap additional feature maps may available online possibility user click map entry access either record giving study information ie publication details summary study summaries typically include information agencies provide visualization present results mapping descriptive report descriptive report describes overall amount evidence characteristics geographical distribution study designs used reports identify knowledge gaps knowledge clusters agencies like sbu present form descriptive report table show evidence shown figure sbu scientific knowledge systematic review high moderate quality demonstrating effectiveness treatment source treatment maternal birth injuries following vaginal birth sbu httpswwwsbuseenpublicationssbukartlaggertreatmentofmaternalbirthinjuriesfollowingvaginalbirth global evidence mapping initiative presents output tables called maps along descriptive report interventions study design output prevention management skeletal muscle spasticity rehabilitation phase tbi source global evidence mapping initiative httpswwwncbinlmnihgovpmcarticlespmc3141802 collaboration environmental evidence use pivot tables pivot charts easily visualize quantity quality assessed evidence across suite metadata variables sometimes present study metadata layer within geographical information system gis may simple world map showing location number included studies descriptive reports outlining methodologies used global evidence mapping initiative social care institute excellence collaboration environmental evidence evidence policy practice information coordinating centre eppicentre organizations 3ie veterans association provide visualizations also provide descriptive report variations whether evidence accessed map interactive nature 3ie maps example means user access database record included study turn link study though may behind paywall link summary findings conclusion provided irc beam exchange odi sightsavers evidence map varied use depending mandate organizations evidence maps used inform research commissioning facilitating use evidence inform decision making 3ie global evidence mapping initiative social care institute excellence collaboration environmental evidence evidence based synthesis program evidence based policing matrix evidence policy practice information coordinating centre use evidence map part initial steps systematic review planning even help make decision whether undertake systematic review evidence maps also used identify evidence gaps 3ie social care institute excellence collaboration environmental evidence evidence based synthesis program example sbu used findings gaps evidence treating adhd basis consultation exercise identify top ten research priorities jacobson et al 2016 used annual grant programme funding decisions swedish agency health technology assessment assessment social services global evidence mapping initiative collaboration environmental evidence 3ie 3ie evidence map evidence based policing matrix used inform policy decision translation tool adoption evidence based research funding particularly due visual representation evidence gaps search evidence map undertaken particular point time map thus becomes date new evidence accumulates maps two three years old less useful guide research priorities though partly mitigated map successfully captured ongoing studies hence systematic reviews stay relevant maps need updated maintained agencies reviewed evidence based policing matrix epistemonikos updated regular basis sbu updates evidence gap database evidence maps like global evidence mapping initiative collaboration environmental evidence policy maps updated mechanism ensuring happens based discussion table 5 summarizes similarities differences evidence gap maps systematic reviews many points similarity evidence gap maps systematic need ex ante protocol explicit search strategy clear inclusion exclusion criteria main difference limited coding studies carried map since reports studies says based discussion propose following principles producing evidence maps updated within three years last completion preferably annually scope broad field changing rapidly evidence gap map definitions per definition used sbu scientific evidence gap exists ircs evidence maps adaptation gap map approach developed international initiative impact evaluation 3ie evidence maps organize visually present best available research evidence every step theory change best determine interventions work outcome area egm simply grid outcomes interest defining columns eg literacy numeracy attendance interventions interest defining rows eg cash transfers early childhood nutrition teacher training