MAX-MIG: AN INFORMATION THEORETIC APPROACH FOR JOINT LEARNING FROM CROWDS
Eliciting labels from crowds is a potential way to obtain large labeled data. Despite a variety of methods developed for learning from crowds, a key challenge remains unsolved: learning from crowds without knowing the information structure among the crowds a priori, when some people of the crowds make highly correlated mistakes and some of them label effortlessly (e.g. randomly). We propose an information theoretic approach, Max-MIG, for joint learning from crowds, with a common assumption: the crowdsourced labels and the data are independent conditioning on the ground truth. Max-MIG simultaneously aggregates the crowdsourced labels and learns an accurate data classifier. Furthermore, we devise an accurate data-crowds forecaster that employs both the data and the crowdsourced labels to forecast the ground truth. To the best of our knowledge, this is the first algorithm that solves the aforementioned challenge of learning from crowds. In addition to the theoretical validation, we also empirically show that our algorithm achieves the new state-of-the-art results in most settings, including the real-world data, and is the first algorithm that is robust to various information structures.
Update after feedback: I would like to thank the authors for their detailed answers, it would be great to see some revisions in the paper also though (except new experimental results).
Especially thank you for providing details of a training procedure which I was missing in the initial draft. I hope to see them in the paper (at least some of them).
I have increased the rating to 6. Given new experimental results both on real data and forecaster comparison I would like to increase the rating to 7. However, I am not sure that this is fair to other authors who would might not be physically able to provide new experimental results due to computational constraints, please note that the experiments in this paper are rather 'light' in the standards of modern deep learning experiments and can be done within the rebuttal period.  
====================================================
The paper finds a practical implementation of ideas from Kong & Schoenebeck (2018) for the learning with crowd problem. It proofs the claims from Kong & Schoenebeck (2018) for the specific family of data classifiers and crowd aggregators. From the general perspective, the papers proposes a method for joint training a classifier and a crowd label aggregator with particular consideration of correlated crowd labels. 
The paper is fairly well-written and well-balanced between theoretical and empirical justification of the method. I see 1 major and 1 big issues with the paper.
Major issue: I am missing details of the actual procedure of training the model. Is MIG set as a loss function for the data classifier NN? Is crowd aggregator trained also as an NN with MIG as a loss function? How do the authors find the optimal p? Also, in order all the provided theory to work all the found data classifier NN, the aggregator and p should be exact maximisers of MIG as far as I understand. How do the author ensure that they find the exact maximisers? Also related to understanding how training works: on p.15 the authors claim “Note that our method can handle this simple correlated mistakes case and will give all useless experts weight zero based on Theorem 3.4.” I have trouble understanding why the proposed method should find these zero weights rather than it is just able to find them?
I am willing to change my judgement if the authors provide convincing details on the training procedure.
Big issue: Experimental settings. 
a) Though it is interesting to see the analysis of the method under controlled environments of synthetic crowd labels with different properties that show benefits of the proposed method (such as dealing with correlated crowd labels), it would be also appealing to see the results with real labels, for example, Rodrigues & Pereira (2017) provide Amazon MTurk crowd labels for the LabelMe data
b) Is the proposed data-crowd forecaster the only method that uses crowd labels on the test data? While it can be argued that it is not straightforward in the test regime to include crowd labels into Crowd Layer, for example, without retraining the neural net, AggNet can use crowd labels without retraining the neural net part. In the presented format, it is unfair to compare the forecaster with the other methods because it uses more information, and essentially, the forecaster is not compared with anything (that uses the same information). It can be compared, at least, with pure Majority Voting, or more advanced pure crowdsourcing aggregation methods. Yes, they won’t use image data, but at least they can use the same amount of crowd label information, which would make a nice comparison with the presented related work and proposed NN: this is what you can get using just image data during test (Crowd Layer, Max-MIG, and others from the current paper), this is what you can get using just crowd labels during test (Majority Voting or, preferably, more advanced pure crowdsourcing aggregators), and this is what you can get using both image and crowd labels during test (the proposed forecaster and AggNet, for example)
Questions out of curiosity: 
i). Does Max-MIG handle missing crowd labels for some data points? Did the author use missing labels in the experiments?
ii). Both the Dogs vs. Cats and CIFAR-10 datasets have more or less balanced data, i.e., the number of data points belonging to each ground truth class is similar between classes. Is this true for the LUNA16 dataset? If yes, have the authors tried their method with heavily imbalanced data? In my experience, some crowdsourcing methods may suffer with imbalanced data, for example, Crowd Layer does so on some data. This tendency of Crowd Layer is kind of confirmed on the provided Dogs vs. Cats in the naïve majority case, where based on crowd labels the first class dominates the second.
Other questions/issues/suggestions:
1. Until the formal introduction of the forecaster on page 4, it is not entirely clear what is the difference between the data classifier and data-crowd forecaster. It should be explained more clearly at the beginning that the 3 concepts (data classifier, crowd label aggregator and "data-crowd forecaster") are separated. Also some motivation why we should care about forecaster would be beneficial because one can argue that if we could train a NN that would make good enough predictions why we should waste resources on crowd labels. For example, the provided empirical results can be used as an argument for this.
2. From the introduction it is unclear that there are methods in crowdsourcing that do not rely on the assumption that data and crowd labels are independent given the ground truth labels. As mentioned in related works there are methods dealing with difficulty of data points, where models assume that crowd labels maybe biased on some data points due to their difficulty, e.g., if images are blurred, which violates this assumption.
Also the note that considering image difficulty violates the independence assumption could be added on page 3 around "[we] do not consider the image difficulty"
3. The beginning of page 4. I think it would be more clear to replace "5 experts' labels:" by $y^{[5]}=$
4. I suggest to move the caption of Figure 3 into the main text.  
5. p.3 "However, these works are still not robust to correlated mistakes" - Why? 
6. Data-crowds forecaster equation. It would be good to add some intuition about this choice. The product between the classifier and aggregator predictions seems reasonable, division on p_c is not that obvious. This expression presumably maximises the information gain introduced below. Some link between this equation and the gain introduction would be nice. Also, minor point – it is better to enlarge inner brackets ()_c
7. The formulation “To the best of our knowledge, our approach is a very early algorithm”, and namely “a very early algorithm” is unclear for me
8. Dual usage of “information intersection” as an assumption and as something that Max-MIG finds is confusing
9. Any comments how the learning rates were chosen are always beneficial
10. Proof of Proposition C.3: “Based on the result of Lemma C.2, by assuming that h ∗ ∈ H_{NN} , we can see (h ∗ , g∗ ,p ∗ ) is a maximizer of max_{h∈H_{NN} ,g∈G_{W A},p∈∆_C} MIGf (h, g,p)” – is expectation missing in the max equation? Is this shown below on page 13? If yes, then the authors should paraphrase this sentence as it does not imply that this is actually shown below
11. p.12 (and below) – what is $\mathbf{C}^m$? Is it $\mathbf{W}^m$?
12. p.15 (at the end of proof) $p \log q$ and $p \log p$ are not formally defined
Minor:
1. p.1 "of THE data-driven-based machine learning paradigm"
2. "crowds aggregator" -> "crowd aggregator"?
3. p.2 (and below) "between the data and crowdsourced labels i.e. the ground truth labelS"
4. Rodrigues & Pereira (2017) has a published version (AAAI) of their paper
5. p.2 "that model multiple experts individually and explicitly in A neural network"
6. p.3 "model the crowds by A Gaussian process"
7. p.3 "We model the crowds via confusion matriCES"
8. p.3 "only provide A theoretic framework and assume AN extremely high model complexity"
9. p.4 "forecast" for h and g -> "prediction"?
10. p.6 “between the data and the crowdsourced labelS”?
11. p.6 “However, in practice, with A finite number of datapoints”
12. p.6 “the experiment section will show that our picked H_{NN} and G_{W A} are sufficientLY simple to avoid over-fitting”
13. p.6 “We call them A Bayesian posterior data classifier / crowds aggregator / data-crowds forecaster, RESPECTEVILY”
14. p.6 “Theorem 3.4. With assumptionS 3.1, 3.3”
15. p.7 “DoctOr Net, the method proposed by Guan et al. (2017)”
16. p.7 “including the naive majority case since naive expert is independent with everything” – rephrasing is required, unclear what “independent with everything” means and who is “naïve expert”
17. Please capitalised names of conferences and journals in References
18. p.10 “she labels the image as “dog”/“cat” with THE probability 0.6/0.8 respectively”, “(e.g. B labels the image as “cat” with THE probability 0.5 and “dog” with THE probability 0.5 when the image has cats or dogs)”
19. p.12 “Lemma C.2. (Kong & Schoenebeck, 2018) With assumptionS 3.1, 3.3”, “Proposition C.3. [Independent mistakes] With assumptionS 3.1, 3.3”