Active Learning from Peers
This paper addresses the challenge of learning from peers in an online multitask setting. Instead of always requesting a label from a human oracle, the proposed method first determines if the learner for each task can acquire that label with sufficient confidence from its peers either as a task-similarity weighted sum, or from the single most similar task. If so, it saves the oracle query for later use in more difficult cases, and if not it queries the human oracle. The paper develops the new algorithm to exhibit this behavior and proves a theoretical mistake bound for the method compared to the best linear predictor in hindsight. Experiments over three multitask learning benchmark datasets show clearly superior performance over baselines such as assuming task independence, learning only from the oracle and not learning from peer tasks.
The authors propose an extension of the active learning framework to address two key issues: active learning of multiple related tasks, and using the 'peers' as an additional oracle to reduce the cost of querying the user all the time. The experimental evaluation of the proposed learning algorithms is performed on several benchmark datasets, and in this context, the authors proved their claims. 
The main novelty lies in the extension of the SHAMPO algorithm so that it can leverage the information from multiple tasks, and also peer-oracles (using the confidence in the predictions), to reduce the annotation efforts by the user. A sound theoretical formulation to incorporate these two is proposed. Experimental results show the validity of the proposed. 
The authors did not comment on the cost of using the peers as oracles, but assumed their availability.  This should be detailed more. Also, it is difficult to assess the model's performance fully compared to the SHAMPO algorithm as its results are missing in Table 1. The accuracy difference between peer and one version of the model is marginal. What is the benefit of the former? 
Overall, this is a nice paper with a good mix of theory and  empirical analysis. 
Lots of typos in the text that should be corrected.