Discrete Rényi Classifiers
Consider the binary classification problem of predicting a target variable Y from a discrete feature vector X = (X 1 , . . . , X d ). When the probability distribution P(X, Y ) is known, the optimal classifier, leading to the minimum misclassification rate, is given by the Maximum A-posteriori Probability (MAP) decision rule. However, in practice, estimating the complete joint distribution P(X, Y ) is computationally and statistically impossible for large values of d. Therefore, an alternative approach is to first estimate some low order marginals of the joint probability distribution P(X, Y ) and then design the classifier based on the estimated low order marginals. This approach is also helpful when the complete training data instances are not available due to privacy concerns. In this work, we consider the problem of finding the optimum classifier based on some estimated low order marginals of (X, Y ). We prove that for a given set of marginals, the minimum Hirschfeld-Gebelein-Rényi (HGR) correlation principle introduced in [1] leads to a randomized classification rule which is shown to have a misclassification rate no larger than twice the misclassification rate of the optimal classifier. Then, under a separability condition, it is shown that the proposed algorithm is equivalent to a randomized linear regression approach. In addition, this method naturally results in a robust feature selection method selecting a subset of features having the maximum worst case HGR correlation with the target variable. Our theoretical upper-bound is similar to the recent Discrete Chebyshev Classifier (DCC) approach [2], while the proposed algorithm has significant computational advantages since it only requires solving a least square optimization problem. Finally, we numerically compare our proposed algorithm with the DCC classifier and show that the proposed algorithm results in better misclassification rate over various UCI data repository datasets.
The problem of fitting a classifier to some data is investigated in this paper. Since full join distributions are infeasible when the number of features is large, the authors follow a standard approach of using marginal distributions to build a classifier. The extreme example of this approach is a Naive Bayes classifier which assumes that all features are independent given the class, and marginal for individual features only are considered. In this work, having pairwise marginal distributions the authors search for stochastic decision rules which minimise error for the worst case joint probability consistent with marginal distributions; those marginal distributions are computed from the data.The authors could also mention that even if the full joint could be modelled, it is not always wise to do that because it usually leads to over-fitting as, e.g., in unbiased Bayesian learning. But, that's a rather minor thing.The paper is very well written, and it contains several, important, technical contributions. The discussion of related work seems to be sufficient. Especially the connection with existing work is very well explained which adds to the significance of this paper.It would be useful if the authors could add a citation showing another example where harmonic mean was used to obtain a smooth objective function. It is a nice transformation.I am not sure if section 4 should not be restructured. Perhaps it would be better to emphasize the connection with Eq. 5. This could be done if the observation in section 4 was replaced by a theorem. This is an important part, and a clearly explained connection would be nice here.An important question, how does your a 2-factor approximation gap relate to the problem of over-fitting. It looks that over-fitting is somehow ignored in this work until feature selection is considered? Could you please explain?The connection with the group lasso regulariser applied to the standard linear regression problem is very interesting.Practical implications of this work are not clear at this point, but the amount of analytical contribution in this paper makes me believe that this could be a great paper if accepted.