Daniel A. Wagner
Quality of education, comparability, and assessment choice in developing countries
University of Pennsylvania
abstract past decade international development agencies begun emphasize improvement quality rather simply quantity education developing countries new focus paralleled significant increase use educational assessments way measure gains losses quality interest assessment grown lowincome countries begun adopt adapt international assessments variety uses including comparability national quality countries improved ways measuring reading achievement attempts reach marginalized populations within country present paper reviews number international national hybrid assessments considers merits terms learning measured well credibility sampling scaling methodologies new hybrid assessments example provide innovative opportunities early intervention children local languages also put premium local validity international comparability review concludes single assessment dominant scientific superiority strict comparability across populations nations requirement rather different assessments different policy practical purposes used important differing ways improve educational quality educational decision makers working developing countries important assessment needs priorities choose carefully order address keywords assessmentquality educationcomparabilitychoice notes 1 parts paper adapted report initially prepared international institute educational planning iiepunited nations educational scientific cultural organization unesco fast track initiative fti see wagner citation2010 2 present discussion understood education focus almost entirely schooling even though education efa covers broader range educational efforts 3 six goals dakar efa framework action early childhood care compulsory primary school ensuring learning needs adult literacy gender disparities quality measurement learning outcomes see unesco citation2004 28 4 united nations citation2000 5 adapted braun kanjee citation2006 5 6 example unesco citation2004 42 7 blum goldstein gurinpace citation2001 france participated 1995 1998 ials apparently also differences swiss french francophone translations 8 must admitted reading often seen essential schoolbased cognitive learning evidenced inclusion efa mdg goals taken type learning relevance schools noted wide variety skills attitudes values transferredin schooling process clear reading important also important reading skill may easily measured softer metrics say attitudes values 9 course many looked role socioeconomic status ses inschool factors textbooks teacher training management use resources forth explanations educational outcomes see example heyneman loxley citation1983 recent review gamaron long 2006 10 see 2010 gmr report entitled reaching marginalized unesco citation2010 11 destefano elaheebocus citation2009 13 also report students reported missed school previous week reading fluency rates half students said missed school student selfreported attendance explains 35 variation schools average reading fluency 12 see among others muthwii citation2004 kenya uganda also commeyras inyega citation2007 13 one reviewer paper correctly noted curriculumderived tests orginated imperial china yet chinese examinations focused universal public education case postrevolutionary france rather version meritocratic selection public administration 14 others less technocratic political perspective would say main purpose testing legitimation distribution scarce public good education thanks one reviewers pointing 15 fti fast track initiative see httpwwweducationfasttrackorg 16 see chromy citation2002 84 listing major studies also lockheed citation2008 6 17 chromy citation2002 kelleghan greaney citation2001 32 18 lockheed verspoor citation1991 19 unesco citation2000 21 20 braun kanjee citation2006 8 21 greaney kellaghan citation2008 89 22 according survey national policy makers gilmore citation2005 45 world bank funding key determinant decision making lsea adoption low middleincome countries 23 emphasis pirls reading studies reference also made third international math science study timss second international science study siss math achievement studies also undertaken iea 24 kamens mcneely citation2010 point increased globalization one reason dramatic increase number countries participating international testing claim globalization fostered world educational ideology well hegemony science led acceptance educational testing much greater heretofore seen 25 iliunesco citation1998 wagner citation2003 2010 26 instance hybrid refers drawing together elements lseas national curricular assessments tests initially designed cognitive assessments reading basic skills 27 egra see rti citation2009 28 sjoberg citation2007 claimed test items deviated substantially stated pisa goal evaluating competencies workforce howie hugues citation2000 found timss covered small fraction 18 curriculum science grade 7 south africa much 50 grade 8 29 see ross genevois citation2006 sacmeq 30 see hambleton kanjee citation1995 discussion translation issues international assessments 31 emic approaches conciously focused local cultural relevance local words descriptors intelligent person etic approaches define intelligence universal concept try measure individuals across cultures single concept definition also see one way think boundary disciplines anthropology emic versus psychology etic see harris citation1976 32 unescollece citation2008 33 elley citation1992 34 confemen citation2008 35 egra subtests require students read aloud therefore require intervention enumerator reading aloud tasks involve fluency accuracy speed measured mean correct items processed one minute various subtasks typically include engagement relationship print indicate begin reading direction reading within line page b letter name knowledge 1 minute test provide name sometimes sound upper lowercase letters distributed random order c phonemic awareness segment words phonemes pronunciation different phonemes word containing 2 5 phonemes identifying initial sounds different words familiar word reading 1 minute test read simple common oneand twosyllable words e unfamiliar nonword pseudoword reading 1 minute test use graphemephoneme correspondences read simple nonsense words f oral reading fluency orf text reading 1 minute test read short text accuracy g reading comprehension respond correctly different type questions literal inferential text read h listening comprehension respond different type questions similar used assess reading comprehension story told adult enumerator dictation write spell use grammar properly dictation exercise detail see rti toolkit rti citation2009 egra studies used subtests changes subtests development 36 two additional criteria important geographical location school situated status school public school private school religious countries status criteria clear leading various problems comparison 37 second llece assessment known serce evaluated grades 3 6 see unescollece citation2008 38 sacmeq countries students lower grades transition usage local national languages classrooms primary school language transition occurs generally around grade 3 grade 4 assumption national language learned sufficiently students grade 6 see ross genevois citation2006 3941 course assumption quite variable one location another one principal reasons egra assessments local languages proven attractive 39 stratification implemented type school type geographical area rural urban without differentiating geographical area school chosen pasec proceeds pooling fixed number student groups level tested minimum 150 schools required 40 example france certain 15yearold students upper secondary level others lower secondary level many countries especially ldcs results students chosen one grade school example west africa unusual 15yearolds lower grades primary school noted comparisons lseas time across years problematic well since countries participate irregular basis 41 example according greaney kellaghan citation2008 71 various sampling problems timss appeared republic yemen number schools grade 4 classes nomadic children could located 42 united states example 2003 national assessment adult literacy english spanish literacy assessed even though dozens languages used adult learners american adult education classes us department education nces citation2009 43 lesotho school less ten students grade 6 excluded population sample seychelles botswana tanzania schools fewer 20 students excluded uganda students excluded zones civil conflict process see ross et al citation2005 see also sacmeq ii report kenya onsumu nzomo obiero citation2005 44 crouch citation2009 45 upcoming 2011 prepirls study lower benchmarks easier vocabulary shorter passages utilized explanatory statistical power available bottom end scale according mullis et al citation2009 prepirls also gather background information home schools classrooms well opportunity learn 46 rti citation2008 47 stevenson stigler citation1982 48 gilmore citation2005 26 49 see greaney kellaghan citation1996 useful overview issue 50 noted donor agencies often play role decision making supporting certain assessments part package support evaluation capacity building 51 another wellknown hybrid assessment aser citation2009 52 heckman citation2006 53 indepth comparative assessment costs see wagner citation2010 cost per learner currently different international hybrid assessments costs latter dropping research costs reduced total cost carrying international regional assessments much higher typically hybrid focused local assessments 54 basic idea evidencecentered design designers work backwards first determining claims would like users make assessment evidence needed support claims develop exercises items probes performance challenges etc elicit desired learner responses scoring rubrics used transform responses relevant evidence measurement models cumulate summarize evidence braun kanjee citation2006 13 55 see hambleton swaminathan rogers citation1991 56 also disadvantages irt especially ldcs beginning assessment programme administration example printing distribution complex scoring scaling scores analyses involving individual students school data problematic require sophisticated personnel see greaney kellaghan citation2008 42 57 see example goldstein citation2004 goldstein bonnet rocher citation2007 mislevy verhelst citation1990 58 smith mccarthy anderson citation2000 review describes various wellknown tests manipulated shorter forms provides methodological suggestions improve short form versions 59 effort present create benchmark using one egras subtests oral reading fluency orf tentative norms 40 60 correct words per minute may become crossnational comparative indicator time writing considerable debate matter 60 also possible focus generic benchmarks rather summative total scores international test example indicators recently advocated fti suggest schoolbased benchmark proportion students two years schooling demonstrate sufficient reading fluency comprehension read learn one could also use read short text first language international benchmark see httpwwweducationfasttrackorgthemeslearningoutcomes indicators also tied use orf possible benchmark see previous footnote 61 see steinerkhamsi citation2010 recent discussion comparability comparative education