The problem of learning from label proportions (LLP) involves training classifiers with weak labels on bags of instances, rather than strong labels on individual instances. The weak labels only contain the label proportion of each bag. The LLP problem is important for many practical applications that only allow label proportions to be collected because of data privacy or annotation cost, and has recently received lots of research attention. Most existing works focus on extending supervised learning models to solve the LLP problem, but the weak learning nature makes it hard to further improve LLP performance with a supervised angle. In this paper, we take a different angle from semi-supervised learning. In particular, we propose a novel model inspired by consistency regularization, a popular concept in semi-supervised learning that encourages the model to produce a decision boundary that better describes the data manifold. With the introduction of consistency regularization, we further extend our study to non-uniform baggeneration and validation-based parameter-selection procedures that better match practical needs. Experiments not only justify that LLP with consistency regularization achieves superior performance, but also demonstrate the practical usability of the proposed procedures.
Summary of the paper:  Learning from label proportions (LLP) is an area in machine learning that tries to learn a classifier that predicts labels of instances, with only bag-level aggregated labels given at the training stage.  Instead of proposing a loss specialized for this problem, this paper proposes a regularization term for the LLP problem. The core contribution of this paper is to use the idea of consistency regularization, which has become very popular in semi-supervised learning in the recent years.  The regularization term takes a perturbation of an input sample, and then force the output of the original and perturbed sample to be similar by minimizing  a KL divergence of the two output distributions.  Experiments show the performance of the proposed method under two bag generation settings.  The paper also finds empirically that the hard L_1 has high correlation with the test error rate, which makes it an ideal candidate when the user splits the validation data from training data (meaning there are no ground truth labels for each instances).
Reasons for the decision of the paper:  The proposed consistency regularization term seems to be borrowed directly from semi-supervised learning (SSL) research, and is not really specialized for the LLP problem.  Although the discovery that adding this regularization term for the LLP problem increases generalization performance is novel, using this term for data without ground truth labels in image datasets such as CIFAR10/100 and SVHN itself has low novelty, since these datasets satisfy the smoothness assumption used by consistency regularization methods.  Another issue is that there are only one trial for every experiment, making it hard for the reader to see if the results are statistically significant or not.  For these two reasons, it was hard to give a high score decision for this paper.  However, the paper’s motivation is interesting, and the direction to explore regularization techniques for the label proportion learning problem is important and seems novel for this area.  If the regularization method can be extended in a way that relates to the LLP problem, it would make the paper much stronger.
Other minor comments:
In the experiments, the average test accuracy of the last 10 epochs are reported.  I was curious if the last epochs have better models than earlier epochs.  In many weakly-supervised areas such as noisy labels, it is common that the accuracy goes up very quickly but then gradually decreases.  Does this also happen for the LLP problem?
In J_cons, does the expectation need to be over p?
***After author response:
Thank you for answering my questions!  Although I have a better understanding of the paper now, I still have the same concerns and would like to keep my score.