Rita Borgo|Luana Micallef|Benjamin Bach|Fintan McGee|B. Lee
Information Visualization Evaluation Using Crowdsourcing
King's College Hospital|Aalto University|University of Edinburgh|Luxembourg Institute of Science and Technology|Microsoft (United States)
abstract visualization researchers increasingly leveraging crowdsourcing approaches overcome number limitations controlled laboratory experiments including small participant sample sizes narrow demographic backgrounds study participants however community little understanding researchers use crowdsourcing approaches visualization research paper review use crowdsourcing evaluation visualization research analyzed 190 crowdsourcing experiments reported 82 papers published major visualization conferences journals 2006 2017 tagged experiment along 36 dimensions identified crowdsourcing experiments grouped dimensions six important aspects study design amp procedure task type participants measures amp metrics quality assurance reproducibility report main findings review discuss challenges opportunities improvements conducting crowdsourcing studies visualization research