On the implicit minimization of alternative loss functions when training deep networks
Understanding the implicit bias of optimization algorithms is important in order to improve generalization of neural networks. One approach to try to exploit such understanding would be to then make the bias explicit in the loss function. Conversely, an interesting approach to gain more insights into the implicit bias could be to study how different loss functions are being implicitly minimized when training the network. In this work, we concentrate our study on the inductive bias occurring when minimizing the cross-entropy loss with different batch sizes and learning rates. We investigate how three loss functions are being implicitly minimized during training. These three loss functions are the Hinge loss with different margins, the cross-entropy loss with different temperatures and a newly introduced Gcdf loss with different standard deviations. This Gcdf loss establishes a connection between a sharpness measure for the 0?1 loss and margin based loss functions. We find that a common behavior is emerging for all the loss functions considered.
This paper makes a step towards understanding of the implicit bias of optimization algorithms in deep learning. The authors consider alternative loss functions for deep networks: (1) the temperature-scaled cross-entropy loss with different values of the temperature; (2) the hinge-loss with different values of the margin parameter; (3) the Gcdf loss with different values of the variance parameter. The paper introduces the Gcdf loss which is derived as a modification of the 0-1 loss under the noise in the parameters of the linear output layer. The authors propose to use the alternative losses as measures of margin and sharpness associated with a solution found by an optimization algorithm. The experiments show how SGD in different learning scenarios (low/high learning rate and small/large batch) performs implicit minimization of the alternative loss functions with different parameters. Specifically, using larger learning rates/smaller batch sizes is shown to implicitly minimize the losses corresponding to higher values of the temperature/margin/variance. The results provide insights about margins and sharpness of solutions found by different modes of SGD. The direction explored in the paper is important for the understanding of the connections between optimization, properties of the loss landscapes (such as sharpness), and generalization. The results reported in the paper are interesting. However, currently I am not convinced that the contributions are sufficient for publication at ICLR as the scope of the performed analysis is limited. In my view, the study is not comprehensive enough and the paper would benefit from incorporating additional results. Detailed comments: 1) My main concern is that currently there is very little explanation provided for the observed experimental findings. The paper would strongly benefit from additional results focused on identification and verification of the mechanisms behind the observed behavior of the optimizer. 2) Many connections mentioned in the paper are left unexplored. It would help to investigate the mentioned connections between the implicit minimization of the considered losses and sharpness, curvature, and generalization. A similar design of the experiment can be used in which the alternative loss values can be tracked alongside with the validation loss (or multiple losses) as well as the measures of sharpness and the characteristics of the Hessian. 3) Another direction for improvement is the extension of the set of analyzed settings (as it was mentioned in the discussion section). This includes performing the analysis for a broader set of architectures (potentially with different normalization schemes), optimizers, and choices of the hyperparameters (momentum, weight decay). These experiments would help to better understand the observed phenomenon and analyze the effect of different settings.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
On the implicit minimization of alternative loss functions when training deep networks
Understanding the implicit bias of optimization algorithms is important in order to improve generalization of neural networks. One approach to try to exploit such understanding would be to then make the bias explicit in the loss function. Conversely, an interesting approach to gain more insights into the implicit bias could be to study how different loss functions are being implicitly minimized when training the network. In this work, we concentrate our study on the inductive bias occurring when minimizing the cross-entropy loss with different batch sizes and learning rates. We investigate how three loss functions are being implicitly minimized during training. These three loss functions are the Hinge loss with different margins, the cross-entropy loss with different temperatures and a newly introduced Gcdf loss with different standard deviations. This Gcdf loss establishes a connection between a sharpness measure for the 0?1 loss and margin based loss functions. We find that a common behavior is emerging for all the loss functions considered.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This paper makes a step towards understanding of the implicit bias of optimization algorithms in deep learning. The authors consider alternative loss functions for deep networks: (1) the temperature-scaled cross-entropy loss with different values of the temperature; (2) the hinge-loss with different values of the margin parameter; (3) the Gcdf loss with different values of the variance parameter. The paper introduces the Gcdf loss which is derived as a modification of the 0-1 loss under the noise in the parameters of the linear output layer. The authors propose to use the alternative losses as measures of margin and sharpness associated with a solution found by an optimization algorithm. The experiments show how SGD in different learning scenarios (low/high learning rate and small/large batch) performs implicit minimization of the alternative loss functions with different parameters. Specifically, using larger learning rates/smaller batch sizes is shown to implicitly minimize the losses corresponding to higher values of the temperature/margin/variance. The results provide insights about margins and sharpness of solutions found by different modes of SGD. The direction explored in the paper is important for the understanding of the connections between optimization, properties of the loss landscapes (such as sharpness), and generalization. The results reported in the paper are interesting. However, currently I am not convinced that the contributions are sufficient for publication at ICLR as the scope of the performed analysis is limited. In my view, the study is not comprehensive enough and the paper would benefit from incorporating additional results. Detailed comments: 1) My main concern is that currently there is very little explanation provided for the observed experimental findings. The paper would strongly benefit from additional results focused on identification and verification of the mechanisms behind the observed behavior of the optimizer. 2) Many connections mentioned in the paper are left unexplored. It would help to investigate the mentioned connections between the implicit minimization of the considered losses and sharpness, curvature, and generalization. A similar design of the experiment can be used in which the alternative loss values can be tracked alongside with the validation loss (or multiple losses) as well as the measures of sharpness and the characteristics of the Hessian. 3) Another direction for improvement is the extension of the set of analyzed settings (as it was mentioned in the discussion section). This includes performing the analysis for a broader set of architectures (potentially with different normalization schemes), optimizers, and choices of the hyperparameters (momentum, weight decay). These experiments would help to better understand the observed phenomenon and analyze the effect of different settings.
assistant
This paper makes a step towards understanding of the implicit bias of optimization algorithms in deep learning. The authors consider alternative loss functions for deep networks: (1) the temperature-scaled cross-entropy loss with different values of the temperature; (2) the hinge-loss with different values of the margin parameter; (3) the Gcdf loss with different values of the variance parameter. The paper introduces the Gcdf loss which is derived as a modification of the 0-1 loss under the noise in the parameters of the linear output layer. The authors propose to use the alternative losses as measures of margin and sharpness associated with a solution found by an optimization algorithm. The experiments show how SGD in different learning scenarios (low/high learning rate and small/large batch) performs implicit minimization of the alternative loss functions with different parameters. Specifically, using larger learning rates/smaller batch sizes is shown to implicitly minimize the losses corresponding to higher values of the temperature/margin/variance. The results provide insights about margins and sharpness of solutions found by different modes of SGD. The direction explored in the paper is important for the understanding of the connections between optimization, properties of the loss landscapes (such as sharpness), and generalization. The results reported in the paper are interesting. However, currently I am not convinced that the contributions are sufficient for publication at ICLR as the scope of the performed analysis is limited. In my view, the study is not comprehensive enough and the paper would benefit from incorporating additional results. Detailed comments: 1) My main concern is that currently there is very little explanation provided for the observed experimental findings. The paper would strongly benefit from additional results focused on identification and verification of the mechanisms behind the observed behavior of the optimizer. 2) Many connections mentioned in the paper are left unexplored. It would help to investigate the mentioned connections between the implicit minimization of the considered losses and sharpness, curvature, and generalization. A similar design of the experiment can be used in which the alternative loss values can be tracked alongside with the validation loss (or multiple losses) as well as the measures of sharpness and the characteristics of the Hessian. 3) Another direction for improvement is the extension of the set of analyzed settings (as it was mentioned in the discussion section). This includes performing the analysis for a broader set of architectures (potentially with different normalization schemes), optimizers, and choices of the hyperparameters (momentum, weight decay). These experiments would help to better understand the observed phenomenon and analyze the effect of different settings.