Data Programming: Creating Large Training Sets, Quickly
Large labeled training sets are the critical building blocks of supervised learning methods and are key enablers of deep learning techniques. For some applications, creating labeled training sets is the most time-consuming and expensive part of applying machine learning. We therefore propose a paradigm for the programmatic creation of training sets called data programming in which users express weak supervision strategies or domain heuristics as labeling functions, which are programs that label subsets of the data, but that are noisy and may conflict. We show that by explicitly representing this training set labeling process as a generative model, we can "denoise" the generated training set, and establish theoretically that we can recover the parameters of these generative models in a handful of settings. We then show how to modify a discriminative loss function to make it noise-aware, and demonstrate our method over a range of discriminative models including logistic regression and LSTMs. Experimentally, on the 2014 TAC-KBP Slot Filling challenge, we show that data programming would have led to a new winning score, and also show that applying data programming to an LSTM model leads to a TAC-KBP score almost 6 F1 points over a state-of-the-art LSTM baseline (and into second place in the competition). Additionally, in initial user studies we observed that data programming may be an easier way for non-experts to create machine learning models when training data is limited or unavailable.
The paper talks about a fundamental paradigm shift in building supervised ML systems. Generating ground truth (labeled dataset) to be used as training sets for different ML algorithms and frameworks through crowdsourcing or otherwise is an expensive step since it may not always be available. The data programming paradigm discussed in this paper instead focuses on how to improve existing ML systems by using unlabeled data along with some broad labeling heuristics given by domain experts (which are albeit noisy). Experimental results indicate that data programming can significantly improve supervised ML systems under certain conditions.
This is a interesting work. The author motivates the problem regarding how availability of large labeled training sets may be hindrance to several supervised ML systems and deep learning techniques. And data programming can be an interesting approach here. Also the user study indicating how researchers find it easy to write labeling heuristics instead of generating ground truth through crowdsourcing or otherwise is a good indication of the utility of this technique. The writing is clear and easy to follow, the experiments are thorough. Following are a few pointers/suggestions which I think the authors should follow up on:  1. There was an older NIPS paper by Rich Caruana regarding “Do deep nets really need to be deep” which explores the ideas of model compression. Here, a very small labeled training data is used to build a large model which then produces massive amounts of synthetically labeled dataset from massive unlabeled datasets to train compressed models. I think this should also be cited in the related work as it has been successfully deployed in many practical systems. This brings us to two significant advantages of data programming: In the former work it was mentioned that the synthetically labeled dataset needs to be around 10 times larger than the actual labeled ground truth. There are no such restrictions with respect to data programming. Second, data programming seems to improve performance of supervised ML algorithms without requiring any additional training instances which is counter-intuitive.   2. The results indicate that the coverage of the labeling functions can be as low as 7% which is quite surprising? Is their any further insight into this?  3. My one concern is that the labeling heuristics might need to have very high accuracy (as given by alpha values) which may not always be possible. It will be interesting to see what happens when labeling functions have poor to moderate accuracy even when they are independent? Nevertheless this work provides an interesting direction.