  The paper reports an experimentally identified list of benchmark data sets
that are hard for representative classification and feature selection methods.
This was done after systematically evaluating a total of 48 combinations of
methods, involving eight state-of-the-art classification algorithms and six
commonly used feature selection methods, on 129 data sets from the UCI
repository (some data sets with known high classification accuracy were
excluded). In this paper, a data set for classification is called hard if none
of the 48 combinations can achieve an AUC over 0.8 and none of them can achieve
an F-Measure value over 0.8; it is called easy otherwise. A total of 15 out of
the 129 data sets were found to be hard in that sense. This paper also compares
the performance of different methods, and it produces rankings of
classification methods, separately on the hard data sets and on the easy data
sets. This paper is the first to rank methods separately for hard data sets and
for easy data sets. It turns out that the classifier rankings resulting from
our experiments are somehow different from those in the literature and hence
they offer new insights on method selection. It should be noted that the Random
Forest method remains to be the best in all groups of experiments.

Machine Learning