Published as a conference paper at ICLR 2018 MASTERING THE DUNGEON: GROUNDED LANGUAGE LEARNING BY MECHANICAL TURKER DESCENT
Contrary to most natural language processing research, which makes use of static datasets, humans learn language interactively, grounded in an environment. In this work we propose an interactive learning procedure called Mechanical Turker Descent (MTD) and use it to train agents to execute natural language commands grounded in a fantasy text adventure game. In MTD, Turkers compete to train better agents in the short term, and collaborate by sharing their agents' skills in the long term. This results in a gamified, engaging experience for the Turkers and a better quality teaching signal for the agents compared to static datasets, as the Turkers naturally adapt the training data to the agent's abilities.
TL;DR of paper: Improved human-in-the-loop data collection using crowdsourcing. The basic gist is that on every round, N mechanical turkers will create their own dataset. Each turker gets a copy of a base model which is trained on their own dataset, and each trained model is evaluated on all the other turker datasets. The top-performing models get a cash bonus, incentivizing turkers to provide high quality training data. A new base model is trained on the pooled-together data of all the turkers, and a new round begins. The results indicate an improvement over static data collection.
This idea of HITL dataset creation is interesting, because the competitive aspect incentivizes turkers to produce high quality data. Judging by the feedback given by turkers in the appendix, the workers seem to enjoy the competitive aspect, which would hopefully lead to better data. The results seem to suggest that MTD provides an improvement over non-HITL methods.
The authors repeatedly emphasize the "collaborative" aspect of MTD, saying that the turkers have to collaborate to produce similar dataset distributions, but this is misleading because the turkers don't get to see other datasets. MTD is mostly competitive, and the authors should reduce the emphasis on a stretched definition of collaboration.
One questionable aspect of MTD is that the turkers somehow have to anticipate what are the best examples for the model to train with. That is, the turkers have to essentially perform the example selection process in active learning with relatively little interaction with the training model. While the turkers are provided immediate feedback when the model already correctly classifies the proposed training example, it seems difficult for turkers to anticipate when an example is too hard, because they have no idea about the learning process.
My biggest criticism is that MTD seems more like an NLP paper rather than an ICLR paper. I gave a 7 because I like the idea, but I wouldn't be upset if the AC recommends submitting to an NLP conference instead.