UniLoss: Unified Surrogate Loss by Adaptive Interpolation
We introduce UniLoss, a unified framework to generate surrogate losses for training deep networks with gradient descent, reducing the amount of manual design of task-specific surrogate losses. Our key observation is that in many cases, evaluating a model with a performance metric on a batch of examples can be re-factored into four steps: from input to real-valued scores, from scores to comparisons of pairs of scores, from comparisons to binary variables, and from binary variables to the final performance metric. Using this re-factorization we generate a unified differentiable approximation of the evaluation computation, through adaptive interpolation at selective configurations of the binary variables. Using UniLoss, we can optimize for different tasks and metrics using one unified framework, achieving comparable performance compared with task-specific losses. We validate the effectiveness of UniLoss on three tasks and four datasets.
This paper presents UniLoss, a general framework for training a neural network model with respect to a non-differentiable evaluation metric as an objective function. The key idea of this paper is to decompose an evaluation metric into a sequence of functions (f, h, g), where: f maps scores computed by a neural network model to comparisons with the ground truth; h maps the comparisons to binary variables; and g computes the performance value from the binary variables. Even with the decomposition, the evaluation metric is still indifferentiable. In order to make the evaluation metric differentiable, this study approximates h with the sigmoid function and g with an interpolation method (inverse distance weighting). The experimental results show that the loss functions obtained by UniLoss can achieve roughly comparable performances to the conventional loss functions on MNIST (binary classification, average precision, cross-entropy loss), CIFAR-10/100 (multi-class classification, accuracy, cross-entropy loss), and MPII (pose estimation, PCKh, MSE with the manually-designed *ground truth* heatmaps). This paper is well structured and easy to read. The idea of decomposing an evaluation metric into functions and considering the approximated versions the functions is interesting. It is great to see that the empirical results demonstrated that the proposed method could achieve the same level of performances with the conventional losses. The impact of this paper would be greater if the proposed method could discover a better approximation of an evaluation metric and achieve a better result. Although the procedure for obtaining a loss function looks systematic, I am wondering of the generality of this framework. For example, I*m not sure whether this method can be applied to other evaluation metrics such as F1 and BLEU scores. Even if this method can support these metrics, I would like to know how much effort we need to craft functions f, h, and g. A discussion about the applicability and limitation of the proposed framework would be useful. I*m also wondering the additional computational cost introduced by a surrogate loss. The binary classification on the MNIST, for example, requires computations for binary variables whereas the original evaluation metric and cross-entropy loss require only . For this reason, I*d like to read a discussion and empirical results about the computational cost. Minor comments: Section 1: *For example, cross-entropy is a popular loss function for training a multi-class classifier, but when it comes to comparing models on a test set, classification error is used instead.* *For example, cross-entropy or hinge loss for classification, both of which have proven to work well in practice.* This discussion is a bit confusing because the former sentence claims that the cross-entropy loss is inappropriate, and the latter suddenly explains that the cross-entropy loss works well in practice. Section 1: *Non-differentiability occurs when the output of the task is discrete (e.g. class labels)* This sentence is a bit misleading because the cross-entropy loss is differentiable and used for multi-class classification (even though labels are discrete). Section 3.1: *Our approach seeks to automatically find a surrogate loss to minimize e* When we call as *performance*, we want to *maximize* (maximize the performance) instead of *minimize* (because we do not want to minimize the performance). We may call as *error* to be consistent.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
UniLoss: Unified Surrogate Loss by Adaptive Interpolation
We introduce UniLoss, a unified framework to generate surrogate losses for training deep networks with gradient descent, reducing the amount of manual design of task-specific surrogate losses. Our key observation is that in many cases, evaluating a model with a performance metric on a batch of examples can be re-factored into four steps: from input to real-valued scores, from scores to comparisons of pairs of scores, from comparisons to binary variables, and from binary variables to the final performance metric. Using this re-factorization we generate a unified differentiable approximation of the evaluation computation, through adaptive interpolation at selective configurations of the binary variables. Using UniLoss, we can optimize for different tasks and metrics using one unified framework, achieving comparable performance compared with task-specific losses. We validate the effectiveness of UniLoss on three tasks and four datasets.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This paper presents UniLoss, a general framework for training a neural network model with respect to a non-differentiable evaluation metric as an objective function. The key idea of this paper is to decompose an evaluation metric into a sequence of functions (f, h, g), where: f maps scores computed by a neural network model to comparisons with the ground truth; h maps the comparisons to binary variables; and g computes the performance value from the binary variables. Even with the decomposition, the evaluation metric is still indifferentiable. In order to make the evaluation metric differentiable, this study approximates h with the sigmoid function and g with an interpolation method (inverse distance weighting). The experimental results show that the loss functions obtained by UniLoss can achieve roughly comparable performances to the conventional loss functions on MNIST (binary classification, average precision, cross-entropy loss), CIFAR-10/100 (multi-class classification, accuracy, cross-entropy loss), and MPII (pose estimation, PCKh, MSE with the manually-designed *ground truth* heatmaps). This paper is well structured and easy to read. The idea of decomposing an evaluation metric into functions and considering the approximated versions the functions is interesting. It is great to see that the empirical results demonstrated that the proposed method could achieve the same level of performances with the conventional losses. The impact of this paper would be greater if the proposed method could discover a better approximation of an evaluation metric and achieve a better result. Although the procedure for obtaining a loss function looks systematic, I am wondering of the generality of this framework. For example, I*m not sure whether this method can be applied to other evaluation metrics such as F1 and BLEU scores. Even if this method can support these metrics, I would like to know how much effort we need to craft functions f, h, and g. A discussion about the applicability and limitation of the proposed framework would be useful. I*m also wondering the additional computational cost introduced by a surrogate loss. The binary classification on the MNIST, for example, requires computations for binary variables whereas the original evaluation metric and cross-entropy loss require only . For this reason, I*d like to read a discussion and empirical results about the computational cost. Minor comments: Section 1: *For example, cross-entropy is a popular loss function for training a multi-class classifier, but when it comes to comparing models on a test set, classification error is used instead.* *For example, cross-entropy or hinge loss for classification, both of which have proven to work well in practice.* This discussion is a bit confusing because the former sentence claims that the cross-entropy loss is inappropriate, and the latter suddenly explains that the cross-entropy loss works well in practice. Section 1: *Non-differentiability occurs when the output of the task is discrete (e.g. class labels)* This sentence is a bit misleading because the cross-entropy loss is differentiable and used for multi-class classification (even though labels are discrete). Section 3.1: *Our approach seeks to automatically find a surrogate loss to minimize e* When we call as *performance*, we want to *maximize* (maximize the performance) instead of *minimize* (because we do not want to minimize the performance). We may call as *error* to be consistent.
assistant
This paper presents UniLoss, a general framework for training a neural network model with respect to a non-differentiable evaluation metric as an objective function. The key idea of this paper is to decompose an evaluation metric into a sequence of functions (f, h, g), where: f maps scores computed by a neural network model to comparisons with the ground truth; h maps the comparisons to binary variables; and g computes the performance value from the binary variables. Even with the decomposition, the evaluation metric is still indifferentiable. In order to make the evaluation metric differentiable, this study approximates h with the sigmoid function and g with an interpolation method (inverse distance weighting). The experimental results show that the loss functions obtained by UniLoss can achieve roughly comparable performances to the conventional loss functions on MNIST (binary classification, average precision, cross-entropy loss), CIFAR-10/100 (multi-class classification, accuracy, cross-entropy loss), and MPII (pose estimation, PCKh, MSE with the manually-designed *ground truth* heatmaps). This paper is well structured and easy to read. The idea of decomposing an evaluation metric into functions and considering the approximated versions the functions is interesting. It is great to see that the empirical results demonstrated that the proposed method could achieve the same level of performances with the conventional losses. The impact of this paper would be greater if the proposed method could discover a better approximation of an evaluation metric and achieve a better result. Although the procedure for obtaining a loss function looks systematic, I am wondering of the generality of this framework. For example, I*m not sure whether this method can be applied to other evaluation metrics such as F1 and BLEU scores. Even if this method can support these metrics, I would like to know how much effort we need to craft functions f, h, and g. A discussion about the applicability and limitation of the proposed framework would be useful. I*m also wondering the additional computational cost introduced by a surrogate loss. The binary classification on the MNIST, for example, requires computations for binary variables whereas the original evaluation metric and cross-entropy loss require only . For this reason, I*d like to read a discussion and empirical results about the computational cost. Minor comments: Section 1: *For example, cross-entropy is a popular loss function for training a multi-class classifier, but when it comes to comparing models on a test set, classification error is used instead. * *For example, cross-entropy or hinge loss for classification, both of which have proven to work well in practice. * This discussion is a bit confusing because the former sentence claims that the cross-entropy loss is inappropriate, and the latter suddenly explains that the cross-entropy loss works well in practice. Section 1: *Non-differentiability occurs when the output of the task is discrete (e.g. class labels)* This sentence is a bit misleading because the cross-entropy loss is differentiable and used for multi-class classification (even though labels are discrete). Section 3.1: *Our approach seeks to automatically find a surrogate loss to minimize e* When we call as *performance*, we want to *maximize* (maximize the performance) instead of *minimize* (because we do not want to minimize the performance). We may call as *error* to be consistent.