Efficient High-Order Interaction-Aware Feature Selection Based on Conditional Mutual Information
This study introduces a novel feature selection approach CMICOT, which is a further evolution of filter methods with sequential forward selection (SFS) whose scoring functions are based on conditional mutual information (MI). We state and study a novel saddle point (max-min) optimization problem to build a scoring function that is able to identify joint interactions between several features. This method fills the gap of MI-based SFS techniques with high-order dependencies. In this high-dimensional case, the estimation of MI has prohibitively high sample complexity. We mitigate this cost using a greedy approximation and binary representatives what makes our technique able to be effectively used. The superiority of our approach is demonstrated by comparison with recently proposed interactionaware filters and several interaction-agnostic state-of-the-art ones on ten publicly available benchmark datasets.
This paper presents a new feature selection algorithm, named CMICOT, that considers high-order dependences among features (up to order t, with t > 3), and uses Mutual Information (MI) to measure such dependences. Additionally, in order to alleviate the lack of data and the increase on computational cost to properly estimate the mutual information, the CMICOT algorithm is a greedy approach that uses binary representatives of each feature.  The proposed algorithm is tested against some state-of-the-art feature selection algorithms, and interaction-aware SFS-based filters. The presented results on 10 public datasets show that overall the CMICOT algorithm outperforms the other feature selection algorithm in terms of classification accuracy when kNN or AdaBoost are used as classifiers.
The problem of designing a feature selection algorithm capable of efficiently deal with high-order interaction among features is an interesting and open problem in the feature selection area. That is why this paper is appealing. However, there are several issues regarding the computational cost and the experimental setup that need a clarification in order to consider it for acceptance.  • My main concern is about the justification and analysis of the Binary representatives procedure described in Section 3.3, which I think compromises the acceptance of this paper. • On the one and, the theoretical justification of this procedure is not very convincing. It is said (lines 219-222) that “The described technique has been inspired by the intuition that probably two binary representatives of two different features interact on average better than two binary representatives of one feature”; however, no references or examples are provided to support this idea.  On the other hand, when comparing the computational cost between the algorithm with and without binary representations (lines 215-219), the same values for t and s are considered. This is not a fair comparison as both cases are not taking the same level of information. Additionally, I have serious doubts about the reduction in computational cost of the binary representatives. Then, in Algorithm 2, when searching for complementary features and opposing features, I can see a reduction in the cost of computing the mutual information (we are working with binary variables), but the cardinalities of S^bin U B[f] and S^bin are larger than in the non-binary case, as all the set B[f_best] is added to the set S^bin set in line 21. In short, it seems like there is a trade-off between the computational cost to compute the argmax and the mutual information. • Lines 180-182: If an “optimal” interaction-aware MI-based feature selection method  is wanted and not a low dimensional approximation of G, the proposed method as a computational cost O(i^2), which is the same as RelaxMRMR. What are reasonable values for t and s? Please, clarify this point.  • Please, rephrase/clarify lines 183-191. • I don’t understand why the particular case t=s is described in Algorithm 1, and a generic algorithm is not provided. • Proposition 3. First sentence in Appendix A.4 needs clarification and it is fundamental for the correctness of Proposition 3 (“Proof. The calculation of a joint entropy of m variables over N requires takes O(mN) simple operations. Hence, any MI that involve m variables requires O(mN) simple operations as well.”). • Line 249-250: “but estimation of…. BR technique.” Please clarify this sentence. Isn’t it  a problem-dependent factor? • Experimental setup. There is another critical point here. It is not clear whether the curves presented in Figure 1 are obtained over a validation/test set or over the training set. It is said later on that 10-fold cross-validation is applied to estimate the significance of differences in classification quality, but it is not clear whether this procedure was also applied in Figure 1.This must be clarified. • The proposed method does not provide good results for the NBC classifier; though these results are provided in the appendices, they are not mentioned or even discussed in the main paper. • There is relevant information in the appendices regarding the comparison between different feature selection methods. I think that the average results reported in the main manuscript are not enough. • Experiments. How can it be explained that the strongest competitor is CMIM, and not one of the interaction-aware FS methods? • It would be interesting to empirically compare CMICOT with and without binary representatives in terms of classification accuracy and computational cost. Additionally, it is highly advisable to include some comparisons with the other feature selection methods in terms of computational cost. It is especially critical to make it clear the advantage of the proposed method with respect to other interaction-aware SFS-based filters.  • Minor comments: - Line 27: Should it be SFS-based filters instead of SBS-based filters? - Lines 144-145: clarify the sentence in cursive. - Missing axes labels in Figure 1. - Please, indicate the statistical test used in Table 1.