We propose the technique of quasi-multitask learning (Q-MTL), a simple and easy to implement modification of standard multitask learning, in which the tasks to be modeled are identical. We illustrate it through a series of sequence labeling experiments over a diverse set of languages, that applying Q-MTL consistently increases the generalization ability of the applied models. The proposed architecture can be regarded as a new regularization technique encouraging the model to develop an internal representation of the problem at hand that is beneficial to multiple output units of the classifier at the same time. This property hampers the convergence to such internal representations which are highly specific and tailored for a classifier with a particular set of parameters. Our experiments corroborate that by relying on the proposed algorithm, we can approximate the quality of an ensemble of classifiers at a fraction of computational resources required. Additionally, our results suggest that Q-MTL handles the presence of noisy training labels better than ensembles.
This paper considers a regularization technique, derived from multi-task learning, where multiple models with some shared parameters are jointly trained to solve copies of the same task. The technique is well-motivated as an efficient alternative to ensemble learning. The method is validated for a BiLSTM NLP model, which is applied to several POS tagging and named entity recognition tasks. The power of the technique as a regularizer is also demonstrated in the case of highly noisy labels, surprisingly, even outperforming ensemble learning in this setting.
Despite this, my inclination is to reject the paper because of the substantial overlap with previous work and the limited scope of experiments.
My primary concern is that the proposed technique was previously introduced in [1]. This prior work is not acknowledged in the current paper. Perhaps it was overlooked because it is situated tightly in Multi-task Learning, whereas the present work is motivated mainly with respect to Ensembling.
Although the general method was introduced previously, the paper does have some key experimental differences that would be interesting to see explored further.
(1)	The paper uses a hidden layer in the separate classification heads, whereas previous work only used a linear classifier. The intuition that more complex heads will yield more diverse models is clear, but it would be great to see experimental evidence that this complexity helps. The conclusion states that the computational overhead is “infinitesimal”; does increasing the complexity of the classifier trade cost for performance?
(2)	This paper uses Eq. 3 to make predictions, whereas previous work found that this did not improve over simply using the best single prediction model, which makes prediction somewhat more efficient. Is there some experimental evidence that Eq. 3 leads to improvements?
(3)	This paper considers the comparison to ensembling, whereas previous work only considered comparisons to single task and standard multitask learning. Additional experiments showing the advantages over ensembling could make this extension a significant contribution.
(4)	This paper presents novel investigation of the regularization effects of the method, i.e., the resilience to noisy labels and the analysis of learned weight matrices. Is there a real problem where this resilience to noise will improve over ensembles, i.e., without randomly replacing labels? Such an experiment would make this point more compelling. Also, is there some underlying reason why the method outperforms ensembles in this case? Is it simply because the method is less expressive so cannot overfit?
In effect, if the paper could clearly show that (1) or other practical extensions lead to improvements over ensembling in settings where ensembling is commonly used, or enable ensembling in settings where vanilla ensembling fails (i.e., the case of noisy labels), then it could be a substantial contribution. The current scope of the experiments is too limited to conclusively show these points. For example, the technique can be applied to any architecture, but the experiments in the paper are limited to a single architecture; and additional experiments with architectures and tasks that commonly use ensembling would make the experiments more compelling, ideally with comparisons to external results.
Other minor comments:
-	It would be good to see the number of model parameters for easy comparison, especially in table 2 with different value of k.
-	It looks like the x and y axis labels are swapped in Figure 3; from the Figure it looks like STL gets higher accuracies.
-	Figure 2 should say epochs instead of iterations.
[1] Meyerson, E. & Miikkulainen R. “Pseudo-task Augmentation: From Deep Multitask Learning to Intratask Sharing---and Back”, ICML 2018.