Understanding the implicit bias of optimization algorithms is important in order to improve generalization of neural networks. One approach to try to exploit such understanding would be to then make the bias explicit in the loss function. Conversely, an interesting approach to gain more insights into the implicit bias could be to study how different loss functions are being implicitly minimized when training the network. In this work, we concentrate our study on the inductive bias occurring when minimizing the cross-entropy loss with different batch sizes and learning rates. We investigate how three loss functions are being implicitly minimized during training. These three loss functions are the Hinge loss with different margins, the cross-entropy loss with different temperatures and a newly introduced Gcdf loss with different standard deviations. This Gcdf loss establishes a connection between a sharpness measure for the 0 − 1 loss and margin based loss functions. We find that a common behavior is emerging for all the loss functions considered.
This paper adds to a large body of research on implicit regularization:
in the context of deep networks, that, when SGD with a certain step
size or batch size is used, from among parameter vectors that fit
the data equally well, some are much more likely to be chosen than
others.  This paper attacks this question through the lens of
loss functions: when SGD is applied to the usual softmax loss
with a given learning rate, how does the choice of learning
rate effect how rapidly other loss functions are reduced?  They
pay special attention to a loss function that they call Gcdf,
which is motivated by "wide minima" considerations.  In particular,
for the Gcdf to be small, not only must training examples be
classified correctly, but randomly perturbing the weights in
the last layer should not change this correct prediction.
I am convinced that Figures 6 and 12 of this paper show
that optimization of the softmax with a larger step size
implicitly optimizes a loss function that rewards robustness
to a greater extent than when a small step size is used.
I find this interesting.
The authors do a nice job of summarizing a lot of related work.
The Gcdf loss is similar to the ramp loss used in [1]
(see Section 3.1) and elsewhere, including to analyze
generalization in deep learning.  It is also like the
potential function optimized by RobustBoost
(see (4) of [2]) -- the RobustBoost loss function does
not scale by the norm of x, but since it is an ensemble
method, the role of x is played by the predictions of
members of the ensemble, which have a fixed scale.
I assume that, when they evaluate the Gcdf loss for a deep
network, they normalize by the norm of the last hidden
layer.  If this is true, it only captures "wide minima"
in the sense of being robust with respect to perturbations
of the output layer.  (They seem to acknowledge this point
in their paper.)
The results in the paper are not described in enough
detail to be reproduced.  For example, I don't see
where they specify the architecture of the network
that they used in Section 4.
The experiments are limited and narrow in scope.
In Figures 2 and 3 I don't see that they have adequately controlled
for the effect of the learning rate on how fast the explicitly
minimized loss is reduced.  Part of the effect observed is simply
that, when a small learning rate is used, after a given number of
epochs, the weights are just not changed much, so that no loss is
reduced much.  In Figure 2, I find it strange that they did not plot
the values for T=1.
Figure 6 is the most interesting to me.  It seems to show that
training the same loss function with a larger learning rate
effectively optimizes the gcdf loss that rewards sacrificing
training error to achieve stronger robustness.  
Small point: the first time I read Appendix A, I thought that the
results were not there.  It was only later that I saw the figures with
the MNIST results.  It would be helpful if the authors wrote
"The results are in Figures 8-12".
While, as I wrote above, I did find Figure 6 interesting, I feel that
the increment of this research over the large body of previous work
on this topic is not enough to justify publication in ICLR.

[1] Bartlett, Peter L., Dylan J. Foster, and Matus
J. Telgarsky. "Spectrally-normalized margin bounds for neural
networks." Advances in Neural Information Processing Systems. 2017.
[2] https://arxiv.org/pdf/0905.2138.pdf