Many important classification performance metrics, e.g. the F -measure, are nondifferentiable and non-decomposable, and are thus unfriendly to the gradient descent algorithm. Consequently, despite their popularity as evaluation metrics, these metrics are rarely optimized as training objectives in neural network community. In this paper, we propose an empirical utility maximization scheme with provable learning guarantees to address the non-differentiability of these metrics. We then derive a strongly consistent gradient estimator to handle nondecomposability. These innovations enable end-to-end optimization of these performance metrics with the same computational complexity as optimizing a decomposable and differentiable metric, e.g. the cross-entropy loss.
Update: I still feel that the paper should have either strong theory, strong experiments, or some of each to be accepted, but that both are lacking. The revisions required would be too great for acceptance at this time. 
Original review:
The paper proposes a general method to optimize for performance metrics which can be written in terms of the entries of the confusion matrix. The idea is to approximate the entries of the confusion matrix using their expected values for a randomized classifier, plug these estimates into the formula for the desired metric, and optimize that quantity. This is a compelling idea but it needs more support than the theoretical or experimental sections give. 
The simplicity and generality of the method are appealing. Smooth surrogates derived from randomized classifiers have been considered in the context of accuracy [1] and other performance measures [2, 3] and the paper should include some discussion of this prior work, but to my knowledge the broad applicability to non-decomposable and non-differentiable metrics expressible in terms of the confusion matrix is new. 
The theoretical sections could use some improvement. It is worth mentioning that the loss obtained with the proposed method is nonconvex. The first equation in theorem 1 is described with “... where convergence in probability is entry-wise”, when the equation refers to almost sure convergence for a scalar, not convergence in probability for entries of a matrix. 
No convergence rates are given, only asymptotic almost sure convergence as the size of the dataset or the minibatch goes to infinity. For finite datasets these statements are obvious, and while convergence is reassuring for infinite datasets, I imagine the rates will look very different for the loss (a scalar) and the gradient (which may have millions of coordinates). Theorem 3 considers the generalization of a single classifier which is independent of the empirical sample, which makes it irrelevant to cases where the model is learned. Theorem 4, which seeks to give a uniform bound over the model class, only shows that generalization occurs in the limit of infinitely much data (which is not surprising or particularly interesting).
The experimental section compares the algorithm against a well-known and strong baseline, but without any information about the variance of the results and only for a deep network. Several questions remain: Where the proposed method improves over the baseline, is this improvement due to the new method or the interaction between the method and the model? How would the method perform on e.g. a linear model, which is better understood? How do the results depend on batch size, which affects the bias in the gradients? 
[1] Roux, Nicolas Le. "Tighter bounds lead to improved classifiers." arXiv preprint arXiv:1606.09202 (2016).
[2] Mozer, Michael C., et al. "Prodding the ROC curve: Constrained optimization of classifier performance." Advances in Neural Information Processing Systems. 2002.
[3] Goh, Gabriel, et al. "Satisfying real-world goals with dataset constraints." Advances in Neural Information Processing Systems. 2016.