Online F-Measure Optimization
The F-measure is an important and commonly used performance metric for binary prediction tasks. By combining precision and recall into a single score, it avoids disadvantages of simple metrics like the error rate, especially in cases of imbalanced class distributions. The problem of optimizing the F-measure, that is, of developing learning algorithms that perform optimally in the sense of this measure, has recently been tackled by several authors. In this paper, we study the problem of F-measure maximization in the setting of online learning. We propose an efficient online algorithm and provide a formal analysis of its convergence properties. Moreover, first experimental results are presented, showing that our method performs well in practice.
The problem of learning binary classifiers optimizing non-decomposable performance metrics has recently gained a lot of traction; this paper considers the popular F-measure specifically, and proposes an algorithm for optimizing F-measure in the online learning setting. The authors give a formal consistency analysis for the online algorithm. Experimental results on fairly large real-world datasets are presented.The main idea in the paper -- and the online algorithm itself -- is neat and simple. It is easy to understand and more or less easy to implement (except for the caveat of estimating a model for the class-conditional probabilities in each round, which can be quite tricky). The consistency analysis is also fairly novel. Though, I'm not quite sure if the way the guarantee is presented in Theorem 3 (and 5) is precise. Perhaps it would be clearer to make the regret, which in this case is F((y_1, ... , y_t), (\hat{y}_1,...,\hat{y}_t)) - F(\tau^*)), explicit in the statement (which the authors do mention later in Section 6).Recently, [9] introduced a framework for optimizing & analyzing non-decomposable measures in the online setting. It is not quite convincing from the discussion in Section 6, how exactly the analysis of OFO is stronger than that in [9]. In fact, the framework in [9] based on a certain per-round non-decomposable loss is quite intuitive. In particular, [9] also includes convergence rates for regret. It is also surprising that the method in [9] is not compared to in experiments (at least a justification is needed otherwise).The other concern is with respect to the organization of the paper itself. The main idea/contribution which is Algorithm 1 and its analysis appear much later in the paper. Sections 2 and 3 are redundant for the most part. In particular, I fail to see the point of Section 3 in the context of this paper and its key contribution --- at best, it can be summarized in a Background section that combines the key ideas in Sections 2 and 3. This way, the readers can get to Section 4 (& 5), which is the focus of this paper, without distractions (and possibly confusions). The authors even had to move some useful results on Synthetic data to the Appendix -- I suggest including this in the main paper.Responses are satisfactory. Raising my score to Accept.