The Shape of and Solutions to the MTurk Quality Crisis
<|begin_of_text|><|start_header_id|>system<|end_header_id|> You are a research archivist expert in dataset metadata. Answer the question truthfully.  <|eot_id|><|start_header_id|>user<|end_header_id|> Here is the Description of a dataset. Description:'Abstract: Amazon’s Mechanical Turk (MTurk) is widely used for data collection, however, data quality may be declining due to the use of Virtual Private Servers (VPSs) to fraudulently gain access to studies. Unfortunately, we know little about the scale and consequence of this fraud, and tools for social scientists to detect and prevent this fraud are underdeveloped. We first analyze 38 studies and show that this fraud is not new, but has increased recently. We then show that these fraudulent respondents provide particularly low-quality data and can weaken treatment effects. Finally, we provide two solutions: an easy-to-use application for identifying fraud in existing datasets and a method for blocking fraudulent respondents in Qualtrics surveys' . Please return only a json list of at most 3 Keywords corresponding to the Description:.<|eot_id|><|start_header_id|>assistant<|end_header_id|> {"Answer" : ["crowdsourcing","experiments","MTurk","online research","survey research"]} <|eot_id|>
 Here is the Description of a dataset. Description:'Abstract: Amazon’s Mechanical Turk (MTurk) is widely used for data collection, however, data quality may be declining due to the use of Virtual Private Servers (VPSs) to fraudulently gain access to studies. Unfortunately, we know little about the scale and consequence of this fraud, and tools for social scientists to detect and prevent this fraud are underdeveloped. We first analyze 38 studies and show that this fraud is not new, but has increased recently. We then show that these fraudulent respondents provide particularly low-quality data and can weaken treatment effects. Finally, we provide two solutions: an easy-to-use application for identifying fraud in existing datasets and a method for blocking fraudulent respondents in Qualtrics surveys' . Please return only a json list of at most 3 Keywords corresponding to the Description:.
Abstract: Amazon’s Mechanical Turk (MTurk) is widely used for data collection, however, data quality may be declining due to the use of Virtual Private Servers (VPSs) to fraudulently gain access to studies. Unfortunately, we know little about the scale and consequence of this fraud, and tools for social scientists to detect and prevent this fraud are underdeveloped. We first analyze 38 studies and show that this fraud is not new, but has increased recently. We then show that these fraudulent respondents provide particularly low-quality data and can weaken treatment effects. Finally, we provide two solutions: an easy-to-use application for identifying fraud in existing datasets and a method for blocking fraudulent respondents in Qualtrics surveys
crowdsourcing; experiments; MTurk; online research; survey research