Amazon’s Mechanical Turk Has Reinvented Research
Mechanical Turk is great for "open", public research. We used to use them a lot for machine learning tasks (data cleanup, model comparisons, label annotations), along with similar services like CrowdFlower / Figure Eight. We saw two primarily issues when applied to "non-open" (commercial) projects:- business-related data too sensitive to share with strangers (contractual obligations, too much risk)- some tasks required non-trivial subject matter expertise and context to annotate properly (quality control issues)For this reason, we gradually moved to an in-house team of long-term annotators. It's not much more expensive (moms on maternity leave, students…), but infinitely more flexible and safer for our purposes. YMMV.