Yvette Graham|Timothy Baldwin|Alistair Moffat|Justin Zobel
Can machine translation systems be evaluated by the crowd alone
University of Melbourne|Trinity College Dublin|University of Melbourne|University of Melbourne|University of Melbourne
abstract crowdsourced assessments machine translation quality allow evaluations carried cheaply large scale essential however crowds work filtered avoid contamination results inclusion false assessments one method filter via agreement experts even amongst experts agreement levels may high paper present new methodology crowdsourcing human assessments translation quality allows individual workers develop individual assessment strategy agreement experts longer required worker deemed reliable consistent relative previous work individual translations assessed isolation others form direct estimates translation quality allows meaningful statistics computed systems enables significance determined smaller sets assessments demonstrate methodologys feasibility largescale human evaluation replication human evaluation component workshop statistical machine translation shared translation task two language pairs spanishtoenglish englishtospanish results measurement based solely crowdsourced assessments show system rankings line original evaluation comparison results produced relative preference approach direct estimate method described demonstrate direct estimate method substantially increased ability identify significant differences translation systems