Learning with noisy labels has drawn a lot of attention. In this area, most of recent works only consider class-conditional noise, where the label noise is independent of its input features. This noise model may not be faithful to many real-world applications. Instead, few pioneer works have studied instance-dependent noise, but these methods are limited to strong assumptions on noise models. To alleviate this issue, we introduce confidence-scored instance-dependent noise (CSIDN), where each instance-label pair is associated with a confidence score. The confidence scores are sufficient to estimate the noise functions of each instance with minimal assumptions. Moreover, such scores can be easily and cheaply derived during the construction of the dataset through crowdsourcing or automatic annotation. To handle CSIDN, we design a benchmark algorithm termed instance-level forward correction. Empirical results on synthetic and real-world datasets demonstrate the utility of our proposed method.
--- Overall ---
I think the problem of incorporating information about labeling uncertainty (when such information is available) is an interesting and important problem; however, I think this paper contains a crucial misunderstanding of the definition of calibration (as defined in Guo et al. 2017), is missing some important parts of the literature, and does not provide adequate convergence guarantees for the proposed method.
--- Major comments ---
1. In section 3.1, the authors substitute the confidence score r_x for the conditional probability P(Y=y|\bar{Y}=y,X=x); however, even if the confidence scores are perfectly calibrated, these quantities are not necessarily equal (or even particularly close). As defined in Guo et al. (2017), a confidence score is well calibrated if the probability that a prediction is correct given the confidence score is equal to the confidence score. Using their notation: P(Y=\hat{Y}|\hat{P}=p) = p. Importantly, this is not conditioned on \hat{Y} or X. One way to see that these two quantities are not equal is to observe that there are many possible confidence score functions that satisfy the above definition whereas the conditional probability defined above is a unique function. In particular, if the confidence score function is a constant equal to the accuracy of the model, then the confidence score is well calibrated, but clearly not equal to the conditional probability above.
2. The other scenario considered by the authors is when there are multiple annotators; however, there is substantial literature on learning from multiple noisy annotations which the authors do not review. I suggest the authors start with "Modeling annotator expertise: Learning when everybody knows a bit of something" by Yan et al. (2010)  (and the citations therein) which also considers the instance dependent noise case.
3. Under what conditions does the proposed algorithm converge and to what does it converge to?
4. The proposed method relies on several assumptions that are scattered throughout the description of the algorithm. I highly recommend making these assumptions clear near the beginning of the paper. Specifically, my understanding is that the main assumptions are:
i. Confidence scores r_x are available for each instance and r_x \approx P(Y=y|\bar{Y}=y,X=x).
ii. Y _|_ X | Y\neq\bar{Y}, \bar{Y} = y
iii. Anchor points are available on some portion of the data.
Beyond the presentation, I find this to be a fairly strong set of assumptions, particularly the first assumption. 
--- Minor comments ---
1. I really appreciated the synthetic example demonstrating the potential pitfalls of the small loss approach; however, I would spend a bit more time clearly explaining the small loss approach so that readers understand why it fails and how you are solving it's problems.
2. Also in the synthetic example, the authors state that covariate shift leads poor accuracy, however, I think this point would be stronger if demonstrated instead of just asserted.