In contrast to the standard classification paradigm where the true (or possibly noisy) class is given to each training pattern, complementary-label learning only uses training patterns each equipped with a complementary label. This only specifies one of the classes that the pattern does not belong to. The seminal paper on complementary-label learning proposed an unbiased estimator of the classification risk that can be computed only from complementarily labeled data. However, it required a restrictive condition on the loss functions, making it impossible to use popular losses such as the softmax cross-entropy loss. Recently, another formulation with the softmax cross-entropy loss was proposed with consistency guarantee. However, this formulation does not explicitly involve a risk estimator. Thus model/hyper-parameter selection is not possible by cross-validationwe may need additional ordinarily labeled data for validation purposes, which is not available in the current setup. In this paper, we give a novel general framework of complementary-label learning, and derive an unbiased risk estimator for arbitrary losses and models. We further improve the risk estimator by non-negative correction and demonstrate its superiority through experiments.
This paper proposes an improved approach to the "complementary-label" form of weak supervision, in which a label that is *not* the true label is marked. Specifically, this paper proposes an unbiased estimator that accepts arbitrary loss functions and models. Noting that this proposed estimator can suffer from overfitting due to unbounded negative loss, a lower-bounded estimator is proposed. Experiments are then performed on several image classification datasets.
Pros:
- This paper addresses a creative form of weak supervision, proposed by prior work, in which labels that are *not* the true label are labeled, in a clear fashion.
- The first proposed estimator is unbiased, as shown by a proof, and accepts arbitrary losses, an improvement over prior approaches
- The overall presentation is clear and clean
Cons:
- One of the main claims of the paper is the proposal of an unbiased estimator. However, this estimator then does not seem to work well enough due to degenerate negative loss.  So then a modified version is proposed- which does not appear to be unbiased?  Either way, no assertion or proof of it being unbiased is given.  So then presumably this also reverses the claim of being able to cross-validate?  This seems like a major weakening of the paper's contributions
- Since the unbiased estimator does not appear to work well, two implementations of a corrected one are proposed, using heuristic approaches without explicit theoretical guarantees.  This shifts the burden to the experimental studies.  These are somewhat thorough, but not extremely so: for example, one set of hyperparameters were used for all of the methods?  This seems like it could implicitly handicap / favor some over others?
- The proposed estimator is based on the assumption that the probability of classes in the complement set (the set of labels other than the one marked as incorrect) is uniformly distributed (e.g. see beginning of Proof of thm 1).  However, this seems like a potentially naive assumption. Indeed, in the related work section, it is mentioned that work in 2018 already considered the case where this uniformity assumption does not hold.
- More broadly, but following from the above: The paper does not provide any real world examples, real or hypothetical, to give the reader an idea of whether the above uniformity assumption---or really any of these assumptions---are well-motivated or empirically justified.  At the bottom of page 3 in the related work, a concrete application used in prior work is mentioned---where crowd workers are shown single labels and vote Y/N, leading to a mix of standard (if Y) and complement-labeled (if N) data---however this mixed setting is not considered explicitly in this paper.  So, how is the reader supposed to get any idea of whether the assumed setup is motivated or justified?  The experiments do not provide this, because the complementary labels are synthetically generated according to the model assumed in the paper.  Additionally, it is briefly mentioned that collecting complementary labeled data is faster, but again no concrete examples are given to support this.