summaries: This paper introduced a simple but effective method to mitigate overfitting, which modifies the existing Flooding scheme. The proposed iFlood can solve the potential problems of the Flooding algorithm and improve the stability. Furthermore, the authors gave some theoretical analyses of the proposed method. The experiments indicated the proposed method is better than the baselines.  This paper proposed a stable and effective regularizer to prevent overfitting. Specifically, this paper proposes individual Flood. Different from Flood which constrains training loss on mini-batch level, iFlood gives instance-level constraints on training loss. This paper also theoretically shows that the design of iFlood can be intrinsically connected with removing the noise or bias in training data.  This paper extends the idea of Flooding to a sample-specific level and call it iFlooding. The extension is intuitively important and the authors also offered several analytical discussions to show its importance beyond the intuition. The empirical results are fairly relevant and strong.  The paper proposes a flooding loss function that encourages the training loss for each example to be a positive bias instead of zero. The paper has found that such training objective stabilizes the training compared to the regular Flooding, which regularizes the average loss instead. The paper also provides reasoning why such a loss function provides more robustness for training with noisy labels and biased label distributions. The empirical benefit on standard datasets seems marginal, but the benefit on noisy labels is significant.
abstractText: Various regularization methods have been designed to prevent overfitting of machine learning models. Among them, a surprisingly simple yet effective one, called Flooding, is proposed recently, which directly constrains the training loss on average to stay at a given level. However, our further studies uncover that the design of the loss function of Flooding can lead to a discrepancy between its objective and implementation, and cause the instability issue. To resolve these issues, in this paper, we propose a new regularizer, called individual Flood (denoted as iFlood). With instance-level constraints on training loss, iFlood encourages the trained models to better fit the under-fitted instances while suppressing the confidence on over-fitted ones. We theoretically show that the design of iFlood can be intrinsically connected with removing the noise or bias in training data, which makes it suitable for a variety of applications to improve the generalization performances of learned models. We also theoretically link iFlood to some other regularizers by comparing the inductive biases they introduce. Our experimental results on both image classification and language understanding tasks confirm that models learned with iFlood can stably converge to solutions with better generalization ability, and behave consistently at instance-level.
authors.name: EFFECTIVE REGULARIZER
authors.name: Yuexiang Xie
authors.name: Zhen Wang
authors.name: Yaliang Li
authors.name: Ce Zhang
authors.name: Jingren Zhou
authors.name: Bolin Ding
references.authors: Himan Abdollahpouri Masoud Mansoury Robin Burke Bamshad Mobasher
references.title: The unfairness of popularity bias in recommendation
references.venue: In the RMSE workshop held in conjunction with the 13th ACM Conference on Recommender Systems,
references.authors: Dana Angluin Philip Laird
references.title: Learning from noisy examples
references.venue: Machine Learning,
references.authors: Devansh Arpit Stanisław Jastrzębski Nicolas Ballas David Krueger Emmanuel Bengio Maxinder S Kanwal Tegan Maharaj Asja Fischer Aaron Courville Yoshua Bengio
references.title: A closer look at memorization in deep networks
references.venue: In International Conference on Machine Learning,
references.authors: Andrei Barbu David Mayo Julian Alverio William Luo Christopher Wang Dan Gutfreund Josh Tenenbaum Boris Katz
references.title: Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models
references.venue: In Advances in neural information processing systems,
references.authors: Mikhail Belkin Daniel Hsu Siyuan Ma Soumik Mandal
references.title: Reconciling modern machine-learning practice and the classical bias–variance trade-off
references.venue: Proceedings of the National Academy of Sciences,
references.authors: Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova
references.title: BERT: Pre-training of deep bidirectional transformers for language understanding
references.venue: In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,
references.authors: Pierre Foret Ariel Kleiner Hossein Mobahi Behnam Neyshabur
references.title: Sharpness-aware minimization for efficiently improving generalization
references.venue: In International Conference on Learning Representations,
references.authors: Aditya Sharad Golatkar Alessandro Achille Stefano Soatto
references.title: Time matters in regularizing deep networks: Weight decay and data augmentation affect early learning dynamics, matter little near convergence
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Stephen José Hanson Lorien Y Pratt
references.title: Comparing biases for minimal network construction with back-propagation
references.venue: In Advances in neural information processing systems,
references.authors: Kaiming He Xiangyu Zhang Shaoqing Ren Jian Sun
references.title: Deep residual learning for image recognition
references.venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,
references.authors: Takashi Ishida Ikko Yamane Tomoya Sakai Gang Niu Masashi Sugiyama
references.title: Do we need zero training loss after achieving zero training error
references.venue: In Proceedings of the 37th International Conference on Machine Learning,
references.authors: Pavel Izmailov Dmitrii Podoprikhin Timur Garipov Dmitry P. Vetrov Andrew Gordon Wilson
references.title: Averaging weights leads to wider optima and better generalization
references.venue: Proceedings of the Thirty-Fourth Conference on Uncertainty in Artificial Intelligence,
references.authors: Diederik P Kingma Jimmy Ba
references.title: Adam: A method for stochastic optimization
references.venue: In Proceedings of the International Conference on Learning Representations,
references.authors: Adam Kortylewski Bernhard Egger Andreas Schneider Thomas Gerig Andreas Morel-Forster Thomas Vetter
references.title: Analyzing and reducing the damage of dataset bias to face recognition with synthetic data
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops,
references.authors: Dominik Kowald Markus Schedl Elisabeth Lex
references.title: The unfairness of popularity bias in music recommendation: A reproducibility study
references.venue: In European Conference on Information Retrieval,
references.authors: Alex Krizhevsky Geoffrey Hinton
references.title: Learning multiple layers of features from tiny images
references.venue: Technical report,
references.authors: Alex Krizhevsky Ilya Sutskever Geoffrey E Hinton
references.title: Imagenet classification with deep convolutional neural networks. Advances in neural information processing
references.authors: Clara Meister Elizabeth Salesky Ryan Cotterell
references.title: Generalized entropy regularization or: There’s nothing special about label smoothing
references.venue: In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics,
references.authors: Mehryar Mohri Afshin Rostamizadeh Ameet Talwalkar
references.title: Foundations of machine learning
references.venue: MIT press,
references.authors: Yuval Netzer Tao Wang Adam Coates Alessandro Bissacco Bo Wu Andrew Y Ng
references.title: Reading digits in natural images with unsupervised feature learning
references.venue: NIPS Workshop on Deep Learning and Unsupervised Feature Learning,
references.authors: Adam Paszke Sam Gross Francisco Massa Adam Lerer James Bradbury Gregory Chanan Trevor Killeen Zeming Lin Natalia Gimelshein Luca Antiga Alban Desmaison Andreas Kopf Edward Yang Zachary DeVito Martin Raison Alykhan Tejani Sasank Chilamkurthy Benoit Steiner Lu Fang Junjie Bai Soumith Chintala
references.title: Pytorch: An imperative style, high-performance deep learning library
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Giorgio Patrini Frank Nielsen Richard Nock Marcello Carioni
references.title: Loss factorization, weakly supervised learning and label noise robustness
references.venue: In International conference on machine learning,
references.authors: Gabriel Pereyra George Tucker Jan Chorowski Łukasz Kaiser Geoffrey Hinton
references.title: Regularizing neural networks by penalizing confident output distributions
references.venue: In 5th International Conference on Learning Representations,
references.authors: Olga Russakovsky Jia Deng Hao Su Jonathan Krause Sanjeev Satheesh Sean Ma Zhiheng Huang Andrej Karpathy Aditya Khosla Michael Bernstein
references.title: Imagenet large scale visual recognition challenge
references.venue: International journal of computer vision,
references.authors: Karen Simonyan Andrew Zisserman
references.title: Very deep convolutional networks for large-scale image recognition
references.venue: In Proceedings of the International Conference on Learning Representations,
references.authors: Nitish Srivastava Geoffrey Hinton Alex Krizhevsky Ilya Sutskever Ruslan Salakhutdinov
references.title: Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning
references.authors: Christian Szegedy Vincent Vanhoucke Sergey Ioffe Jon Shlens Zbigniew Wojna
references.title: Rethinking the inception architecture for computer vision
references.venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,
references.authors: Sunil Thulasidasan Tanmoy Bhattacharya Jeff A. Bilmes Gopinath Chennupati Jamal MohdYusof
references.title: Combating label noise in deep learning using abstention
references.venue: In Proceedings of the 36th International Conference on Machine Learning,
references.authors: Alex Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R Bowman
references.title: Glue: A multi-task benchmark and analysis platform for natural language understanding
references.venue: In Proceedings of the International Conference on Learning Representations,
references.authors: Haohan Wang Xindi Wu Zeyi Huang Eric P Xing
references.title: High-frequency component helps explain the generalization of convolutional neural networks
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Thomas Wolf Lysandre Debut Victor Sanh Julien Chaumond Clement Delangue Anthony Moi Pierric Cistac Tim Rault Rémi Louf Morgan Funtowicz Joe Davison Sam Shleifer Patrick von Platen Clara Ma Yacine Jernite Julien Plu Canwen Xu Teven Le Scao Sylvain Gugger Mariama Drame Quentin Lhoest Alexander M. Rush
references.title: Transformers: State-of-the-art natural language processing
references.venue: In Proceedings of the 2020 Conference on Empirical Methods in Natural
references.authors: Saining Xie Ross Girshick Piotr Dollár Zhuowen Tu Kaiming He
references.title: Aggregated residual transformations for deep neural networks
references.venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,
references.authors: Taojiannan Yang Sijie Zhu Chen Chen
references.title: Gradaug: A new regularization method for deep neural networks
references.venue: Advances in Neural Information Processing Systems,
references.authors: Yuan Yao Lorenzo Rosasco Andrea Caponnetto
references.title: On early stopping in gradient descent learning
references.venue: Constructive Approximation,
references.authors: Chiyuan Zhang Samy Bengio Moritz Hardt Benjamin Recht Oriol Vinyals
references.title: Understanding deep learning requires rethinking generalization
references.venue: In Proceedings of the International Conference on Learning Representations,
references.authors: Hongyi Zhang Moustapha Cisse Yann N Dauphin David Lopez-Paz
references.title: mixup: Beyond empirical risk minimization
references.venue: In Proceedings of the International Conference on Learning Representations,
references.authors: Yaowei Zheng Richong Zhang Yongyi Mao
references.title: Regularizing neural networks via adversarial model perturbation
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
sections.heading: 1 INTRODUCTION
sections.text: Though overparameterized neural networks have achieved success on a wide range of tasks and applications, it is worth noting that their capacities are sufficient to memorize the entire training data (Zhang et al., 2017a; Arpit et al., 2017), which often leads to an intolerable generalization gap, or in other words, overfitting. To prevent overfitting, many methods have been proposed to regularize how machine learning models fit the training data via introducing additional constraints to control the capacity in effect. For example, L1- or L2- regularizer (i.e., Weight Decay) (Hanson & Pratt, 1989), Early Stopping (Yao et al., 2007), Dropout (Srivastava et al., 2014), Label Smoothing (Szegedy et al., 2016), Confident Penalty (Pereyra et al., 2017), etc (Zhang et al., 2017b; Izmailov et al., 2018; Zheng et al., 2021; Foret et al., 2021; Yang et al., 2020).
  Recently, a new method named Flooding (Ishida et al., 2020) is proposed, which controls the extent to which machine learning models fit the training data via directly encouraging the averaged training loss to stay at a given level rather than achieving (near-)zero loss. Formally, Flooding suggests the following loss function LFlooding to be minimized:
  LFlooding = ∣L − b∣ + b, (1)
  where L denotes the averaged training loss that is defined over a total of N training instances as L = 1
  N ∑ N i=1Li, and b ≥ 0 is a hyper-parameter called “the flood level” to control the training loss. Designed in such a way, whenever L is below b, the gradients will be negated to increase it, so that L will stay around b and avoid being (near-)zero. Ishida et al. (2020) suggests to implement Flooding by minimizing LFlooding with mini-batch SGD, where L is estimated over a mini-batch of instances rather than the full data. Although they have pointed out that the objective being optimized by SGD is an upper bound of its desired one, we notice that this gap increases as the batch size decreases, which makes the discrepancy between the objective and implementation of Flooding more serious.
  By investigating how machine learning models learned with Flooding behave on the training instances and generalize on the testing ones, we uncover the instability issue of Flooding, where it can lead
  to different solutions, and the solutions are inconsistent in their generalization abilities and their behaviors over individuals. Further, we point out that the reason for the instability issue is that Flooding can only guarantee “global convergence”— Flooding encourages the averaged training loss to be sufficiently close to b, while having no requirement on the individual losses.
  Both the discrepancy and instability issues can be attributed to the design of its loss function, which motivates us to propose a new regularizer in this paper, called individual Flood (denoted as iFlood). The proposed regularizer iFlood defines a loss function LiFlood as:
  LiFlood = 1
  N
  N ∑ i=1 (∣Li − b∣ + b), (2)
  where b ≥ 0 is the flood level, N is the size of training sample and Li is the loss function defined upon the i-th instance. When b = 0, L, LFlooding, and LiFlood are equivalent. With b > 0, iFlood encourages the model to better fit the under-fitted instances while suppressing the confidence of over-fitted ones.
  Although the modification is simple, the proposed new regularizer has the following merits: (1) The design of the loss function of iFlood ensures that it can be optimized by SGD without discrepancy between its objective and implementation. (2) Compared with Flooding, the models learned with iFlood can achieve “local convergence”, that is, the individual losses (i.e., Lis) rather than the averaged loss (i.e., L) are encouraged to be sufficiently close to the specified level b, which ensures that the learned models behave consistently over individual instances and produce stable generalization performance. (3) Meanwhile, we theoretically show that the design of iFlood can be intrinsically connected with removing the noise or bias in the training data, making iFlood suitable for a variety of applications to improve the generalization abilities. (4) Moreover, we theoretically compare iFlood with some related works (Szegedy et al., 2016; Pereyra et al., 2017), showing that iFlood discounts the over-confident predictions with less inductive bias.
  We conduct extensive experiments on both image classification and language understanding tasks to compare the performance improvements gained by different regularizers, demonstrating the effectiveness of the proposed iFlood. Further, we evaluate the stability1 of iFlood from several measurements, such as total variation distance and gradient norm. All the experimental results show that, with the “local convergence” suggested by iFlood, the learned models stably converge to solutions with better generalization ability.
sections.heading: 2 PRELIMINARY
sections.text: For ease of discussion, we introduce some notations at first. Without loss of generality, we consider a typical classification problem: Given a training dataset D = {(xi, yi)∣x ∈ X , y ∈ Y, i = 1, . . . ,N} where X stands for the instance domain, Y stands for the set of labels, and each instance (xi, yi) is independently drawn from an underlying joint distribution Pr(X,Y ), we aim to learn a function f ∶ X → ∆(Y) (i.e., a mapping from the instance domain to the space of probability distributions over the labels) to minimize the generalization error E(x,y)∼Pr(X,Y )[1y≠argmax
  y′ f(x)y′ ], where f(x)y′
  denotes the probability of taking the class y′.
  The function f is often learned by minimizing certain loss function l(y, f(x)) (e.g., Cross-Entropy loss). For simplicity, we denote the loss over the i-th training instance as Li and that over the whole training dataset as L = 1
  N ∑ N i=1Li. Note that, besides the classification problem, the regularization
  methods discussed in this paper are applicable to other machine learning tasks, e.g., regression.
sections.heading: 3 INDIVIDUAL FLOOD (IFLOOD)
sections.text: In this section, we first compare iFlood with Flooding from various aspects to demonstrate the advantages of the proposed new regularizer iFlood. Then we provide theoretical analysis about the effect of iFlood for removing the noise or bias in the training data, and connect iFlood with existing regularization methods, such as Label Smoothing (Szegedy et al., 2016) and Confident Penalty (Pereyra et al., 2017).
  1In this paper, “stability” has a different meaning from that used in learning theory (Mohri et al., 2018).
sections.heading: 3.1 LOCAL CONVERGENCE
sections.text: Let us revisit how Flooding works first. By applying Flooding, once the original averaged loss L has approached the flood level, it goes below and above b repeatedly until its convergence. Such process is called “flooding” in Ishida et al. (2020). The loss function of Flooding encourages the averaged loss L to approach to b, thus Flooding pursues the “global convergence” (i.e., ∣L − b∣ ≈ 0). To learn the model parameters θ by minimizing LFlooding defined in Eq.(1), the gradient of LFlooding w.r.t. θ is in the same direction as that of L when L ≥ b, and in the opposite direction when L < b. To be specific,
  ∇θLFlooding = ⎧⎪⎪⎪⎪⎪⎪⎪ ⎨ ⎪⎪⎪⎪⎪⎪⎪⎩ ∇θL = 1 N N ∑ i=1 ∇θLi, if L ≥ b, −∇θL = 1 N N ∑ i=1 (−1) ⋅ ∇θLi, if L < b.
  (3)
  From Eq.(3), we know that whether the original gradient of an instance (i.e., ∇θLi) should be negated is determined by the fact that whether the averaged loss L is below or above the flood level b. However, in most practical cases and meanwhile the experiments of Flooding, deep neural networks are trained with mini-batch SGD where L is estimated from a sampled mini-batch of instances at each step. Thus, whether the gradients should be negated is determined by the averaged training loss estimated over a mini-batch of instances rather than the full data. In this way, the objective being actually optimized by mini-batch SGD is an upper bound of LFlooding, as pointed out by Ishida et al. (2020). This leads to a discrepancy between the loss function of Flooding and its implementation, which introduces randomness to the choice of whether to negate the gradients and makes the flooding process become random. In this study, we further notice that, due to the property of absolute operator, such discrepancy increases along with the decreasing of batch size.
  For iFlood, according to Eq.(2), in a training dataset consisting of N instances, the gradient of LiFlood w.r.t. the model parameters θ can be given as ∇θLiFlood = 1N ∑ N i=1∇θ ∣Li − b∣. Therefore, for each individual, it is guaranteed to contribute 1 N ∇θLi to the aggregated gradient, when Li ≥ b; or to contribute the negation of that, when Li < b. By applying iFlood, the model parameters are encouraged to walk along the contour of L = b like what Flooding does, but, in contrast to Flooding, this is achieved by demanding each individual loss Li,∀i ∈ {1, . . . ,N} to be close to b. Intuitively speaking, iFlood encourages the model to better fit the under-fitted instances while suppressing the confidence of over-fitted ones, according to the given flood level b. In a word, iFlood pursues the “local convergence” (i.e., ∣Li − b∣ ≈ 0,∀i ∈ {1, . . . ,N}). We will analyze the advantage of “local convergence” over “global convergence” from the aspect of stability in Section 3.2. For now, the additive property of Eq.(2) ensures the gradients calculated in each SGD an unbiased estimation, which seemingly eliminates the discrepancy between the design and implementation.
  Note that when optimized by SGD with batch size of 1, Flooding coincides with iFlood. However, it is unreasonable to set batch size to 1 for training machine learning models in most cases, which is empirically confirmed in Appendix C. For practical batch sizes, the designs of the loss function of iFlood and Flooding are different, as aforementioned, and we will empirically demonstrate those differences in Section 4.
sections.heading: 3.2 STABILITY
sections.text: To better understand the differences between Flooding and iFlood, we plot the distributions of individual training losses with Flooding and iFlood in Figure 1, which is produced by training a ResNet18 model on CIFAR-10. Compared with the distribution corresponding to Flooding (see Figure 1a), we can observe that the distribution corresponding to iFlood (i.e., Figure 1b) is much more concentrated, and almost all the instances can be regularized by iFlood to achieve “local convergence” (i.e., ∣Li − b∣ ≈ 0,∀i ∈ {1, . . . ,N}).
  Figure 1a confirms that the models learned with Flooding can achieve “global convergence” (i.e., ∣L − b∣ ≈ 0). The inconsistent in behaviors over individuals achieved by Flooding can causes the instability of Flooding, namely that Flooding leads to various solutions with different generalization abilities. Let’s consider the following cases: One learned model achieves Li = b for every training instance, and the other learned model achieves Li = 0 on half of the whole training instances while
  Li = 2b on the rest ones. Obviously, these two learned models have achieved the same “global convergence” that L − b = 0, but they behave inconsistently over the (training) individuals. Moreover, when b is taken to be a level where Li = b corresponds to being correctly classified and Li = 2b corresponds to being mis-classified, the generalization ability of the former surpasses that of the latter in most cases. The analysis of these cases sheds light on the reason for Flooding’s instability. Albeit the learning dynamics would not lead to these extreme cases in practice, we experimentally observe that, over a considerable portion of instances, the individual losses produced by a model learned with Flooding are far away from b, as Figure 1a shows.
  To address this issue, iFlood is introduced to achieve “local convergence”, which ensures the learned models behave more consistently than those models learned with Flooding. In this way, iFlood can achieve a stable boost of generalization ability. We will provide quantitative analysis to demonstrate the advantages of iFlood on stability in Section 4.
sections.heading: 3.3 THEORETICAL ANALYSIS
sections.text: It has been noticed by Ishida et al. (2020) that a larger b is in favor, when the training data contain noisy labels. Intuitively, b controls how confident the model should trust the training data. Inspired by this observation, it is worth discussing (1) what is the meaning of b in iFlood; (2) when and why do the models learned with iFlood benefit from the regularization. In the rest of this section, we look into these questions under two settings—noisy labels and biased sample, which are commonly considered in both academia and industry.
sections.heading: 3.3.1 NOISY LABELS
sections.text: For many real-world applications, the observed labels are polluted due to certain reasons (Angluin & Laird, 1988; Patrini et al., 2016; Thulasidasan et al., 2019), e.g., the mistakes made by annotators, the perturbation of some attackers, etc. Here, we consider a widely adopted setting where a noisy training sample Dnoisy = {(xi, zi)∣xi ∈ X , zi ∈ Y, i = 1, . . . ,N} is given, with each (xi, zi) being independently drawn from a distribution Pr(X,Z) in the following process: first, an instance (x, y) is drawn from the underlying joint distribution Pr(X,Y ); then, y is perturbed to be z, taking an incorrect label y′ with probability α,0 ≤ α ≤ 1, or keeping the correct label y with probability (1 − α), where the incorrect label is drawn from a uniform distribution over Y/{y}. This generation process implies Z ⊥ X ∣Y and a conditional probability distribution Pr(Z ∣Y ) with ∀y ∈ Y,Pr(Z = y∣y) = 1 − α and ∀y′ ≠ y,Pr(Z = y′∣y) = α∣Y ∣−1 .
  Given Dnoisy, suppose the data generation process is known, we can learn a model parameterized with θ by minimizing its negative log-likelihood:
  − logPr(z1∶N ∣x1∶N , θ) = − log N
  ∏ i=1
  Pr(zi∣xi, θ) = − N
  ∑ i=1 log∑ y∈Y Pr(yi, zi∣xi, θ)
  = − N
  ∑ i=1 log∑ y∈Y
  Pr(zi∣yi)Pr(yi∣xi, θ) = − N
  ∑ i=1
  log{(1 − α)Pr(Y = zi∣xi, θ) + α
  ∣Y ∣ − 1 ∑ y≠zi
  Pr(y∣xi, θ)}
  = − N
  ∑ i=1
  log{(1 − α)Pr(Y = zi∣xi, θ) + α
  ∣Y ∣ − 1 (1 −Pr(Y = zi∣xi, θ))},
  (4) where we denote the cardinality of Y as ∣Y ∣. However, when we have no idea about the underlying data generation process, the RHS of Eq.(4) is unknown to us.
  If Dnoisy is directly adopted as training data without denoising, θ will be learned by minimizing the Cross-Entropy loss: L = −1
  N ∑ N i=1∑y∈Y 1zi=y logPr(Y = y∣xi, θ). In this case, the learned model
  tends to generalize poorly, as it has been misled by the noisy sample and learns a distribution different from the actual joint distribution Pr(X,Y ).
  To explore the usage of iFlood in denoising, we first analyze Eq.(4) and present the following proposition about it. Proposition 1. The negative log-likelihood (i.e., Eq.(4)) is upper bounded by:
  N log(∣Y ∣ − 1) − N
  ∑ i=1 {(1 − α) logPr(Y = zi∣xi, θ) + α log(1 −Pr(Y = zi∣xi, θ))}, (5)
  which is minimized when Pr(Y = zi∣xi, θ) = 1 − α, i = 1, . . . ,N .
  Proof. Since − log(⋅) is monotonically decreasing and ∣Y ∣ ≥ 2, Eq.(4) is upper bounded by: −∑ N i=1 log{ 1−α ∣Y ∣−1 Pr(Y = zi∣xi, θ) + α ∣Y ∣−1(1 − Pr(Y = zi∣xi, θ))} within its domain. This function is further upper bounded by Eq.(5) with Jensen’s inequality. By checking the derivatives of Eq.(5), we know that it is minimized when Pr(Y = zi∣xi, θ) = 1 − α, i = 1, . . . ,N .
  When we regularize the Cross-Entropy loss defined over Dnoisy with iFlood, it becomes:
  LiFlood = 1
  N
  N
  ∑ i=1 {∣ − ∑ y∈Y 1zi=y logPr(Y = y∣xi, θ) − b∣ + b}. (6)
  Once we specify b = − log(1 − α), LiFlood encourages every Pr(Y = zi∣xi, θ), i = 1, . . . ,N to take exp(−b) = 1 − α, which minimizes an upper bound (Eq.(5)) of the actual negative log-likelihood (Eq.(4)). In this way, without knowing how the observed sample becomes noisy beforehand, LiFlood can still serve as a surrogate function for the unknown negative log-likelihood (Eq.(4)) of interest, extending the usage of iFlood to denoising.
sections.heading: 3.3.2 BIASED SAMPLE
sections.text: The second setting to explore iFlood is when the collected training data does not follow the probability distribution of interest. Such a data sample is often named biased sample, which is ubiquitously observed in real-world applications (Abdollahpouri et al., 2019; Kowald et al., 2020). For instance, in recommendation systems, there exists popularity bias (Abdollahpouri et al., 2019; Kowald et al., 2020), and in face recognition and object recognition, data samples are biased regarding the backgrounds or the illumination conditions (Kortylewski et al., 2019; Barbu et al., 2019). Again, we represent an observed label by Z and denote such a biased sample as Dbiased = {(xi, zi)∣xi ∈ X , zi ∈ Y, i = 1, . . . ,N}. The biased sample setting can be formally given as: The observed distribution of input, denoted as Pr′(X), is different from the ground-truth one Pr(X), which leads to the joint distribution Pr(X,Z) ≠ Pr(X,Y ) and thus model tends to learn a biased conditional distribution Pr(Z ∣X) rather than the underlying one Pr(Y ∣X).
  Models learned from a biased sample cannot generalize well due to the distribution drift. Luckily, we can apply iFlood to debiase. In most cases, although we have no idea about the analytic form of Pr(Z ∣X,Y ), it is reasonable to assume and validate some properties about Z ∣X that Pr(Z ∣X,Y ) implies. With such properties, iFlood is able to recover the unbiased label. Formally, let’s assume that according to distributions Pr(Z ∣X,Y ) and Pr(X,Y ) which generate Dbiased, the difference between observed labels Z ∣X and underlying labels Y ∣X follows some specific distribution:
  l(Z ∣X,Y ∣X) ∼ Lap(⋅∣µ,λ), (7) where l denotes the adopted loss function, and Lap(⋅∣µ,λ) denotes the Laplace distribution with parameters µ ≥ 0, λ > 0. With such knowledge, the negative log-likelihood of observing the biased sample Dnoisy is as follows:
  − 1
  N
  N
  ∑ i=1
  logPr[l(zi, f(xi))] = − 1
  N
  N
  ∑ i=1
  log{ 1 2λ e ∣− l(zi,f(xi))−µλ ∣}∝ 1 N N ∑ i=1 ∣l(zi, f(xi)) − µ∣. (8)
  This happens to be in the same form as LiFlood. Thus, learning a model from the biased sample Dbiased via iFlood with b = µ leads to the same solution as learning from the corresponding unbiased sample.
sections.heading: 3.4 COMPARING IFLOOD WITH OTHER RELATED WORKS
sections.text: There exist some regularization methods that prevent a model from overfitting by suppressing its over-confident predictions, such as Label Smoothing (denoted as LS) (Szegedy et al., 2016) and Confident Penalty (denoted as CP) (Pereyra et al., 2017). Following Meister et al. (2020), these methods can be summarized as follows:
  LLS = L + βDKL(u∥p), LCP = L + βDKL(p∥u), (9)
  where β is the regularization strength, DKL(⋅∥⋅) represents the KL divergence, p is the predicted probability distribution over labels, and u is a prior of the label distribution. From Eq.(9) we can see that Label Smoothing and Confident Penalty add a regularization term to the loss function L to encourage the distribution of predicted probability to be close to the prior. Without additional domain knowledge, the uniform distribution is often adopted as the prior, which is expected to prevent the peaked distributions (i.e., the over-confident predictions) and lead to a better generalization.
  Compared with Label Smoothing and Confident Penalty, iFlood performs similarly but has less assumption on the prior of the label distribution. iFlood prevents the model from becoming overconfident on the observed training sample via discounting its prediction probability over the observed label, where the extent of discount is controlled by the flood level b. On the other hand, iFlood does not make any assumption on how to distribute the reserved confidence.
  Formally, we compare iFlood with Label Smoothing under the setting where Cross-Entropy loss is considered and a uniform prior is adopted by Label Smoothing. In this case, the predicted probability distribution p (over labels) that minimizes the objective of Label Smoothing (Eq.(9)) is exactly one of the distributions that minimize the loss function of iFlood. As the regularization is posed at instance-level in both methods, we show this relationship by checking the minimizer of the objective of Label Smoothing over any (x, y) ∈ D:
  l(y, f(x)) + βDKL(u∥f(x)) = l(y, p) + βDKL(u∥p) = − log(py) − β
  ∣Y ∣ log(py) −
  β
  ∣Y ∣ ∑ y′≠y
  log(py′)
  (10)
  ≥ −[ ∣Y ∣ + β
  ∣Y ∣ log(py) +
  (∣Y ∣ − 1)β
  ∣Y ∣ log(
  1 − py ∣Y ∣ − 1 )] = −[ ∣Y ∣ + β ∣Y ∣ log(py) + (∣Y ∣ − 1)β ∣Y ∣ log(1 − py) + c],
  (11)
  where p denotes the probability distribution over labels predicted by f , c is a constant, and ∣Y ∣ denotes the cardinality of Y . On one hand, Eq.(11) is a tight lower bound of Eq.(10), where equality is established when p satisfies that ∀y′ ≠ y, py′ = 1−py ∣Y ∣−1 . On another hand, derivative of Eq.(11) shows that it (as well as Eq.(10)) can be minimized at py = ∣Y ∣+β ∣Y ∣(1+β) . Thus, the objective of Label Smoothing encourages the learned function f to predict a distribution p that satisfies both of the above conditions. Meanwhile, when we specify iFlood with b = log( ∣Y ∣(1+β)∣Y ∣+β ), according to Eq.(2), it encourages the learned function f to predict a distribution p that satisfies py = exp(−b) = ∣Y ∣+β ∣Y ∣(1+β) . Based on the above analysis, we see that iFlood discounts the prediction confidence over the observed label to the same level as that of Label Smoothing, without encouraging the reserved confidence to be equally distributed to other labels. When it is hard to obtain some prior knowledge about the label distribution, iFlood can provide a less-biased and more flexible regularization, compared with Label Smoothing. Similar analysis and conclusion can be applied to Confident Penalty.
sections.heading: 4 EXPERIMENTS
sections.text: In this section, we conduct a series of experiments to demonstrate the effectiveness of iFlood, with the aim to answer the following questions: Q1: Does iFlood provide larger boost of generalization ability, compared with existing regularization methods on benchmark datasets? Q2: Compared to Flooding, can iFlood stably converge to solutions with better generalization ability? Q3: Can iFlood address the noisy label issue well in practice?
sections.heading: 4.1 SETTINGS
sections.text: Datasets. We consider both image classification and language understanding tasks. For image classification, we use CIFAR-10, CIFAR-100 (Krizhevsky et al., 2009), SVHN (Netzer et al., 2011), and ImageNet (Russakovsky et al., 2015). For language understanding, we adopt the General Language Understanding Evaluation (GLUE) benchmark (Wang et al., 2019), and report the experimental results on SST-2, QQP, and QNLI. The details of these datasets and more experimental results on GLUE benchmark can be found in Appendix A and D.2 respectively.
  Baselines. We compare iFlood with the following baselines: (1) Flooding (Ishida et al., 2020); (2) Label Smoothing (Szegedy et al., 2016); (3) Confident Penalty (Pereyra et al., 2017); (4) Unregularized, denotes that the models are learned without Flooding, iFlood, Label Smoothing and Confident Penalty. Data augmentation (e.g., random crop (Krizhevsky et al., 2012) and horizontal flip (Simonyan & Zisserman, 2015)) have been necessary plug-ins of the learning procedure on image classification datasets, we do not remove them in our experiments. As the basic regularizers, L2-regularization is used in both image classification and language understanding tasks, and Dropout is adopted in language understanding task.
  Implementation details. On the image classification datasets, we consider training ResNet18 (He et al., 2016) for CIFAR-10, CIFAR-100 and SVHN, and training ResNeXt50 (Xie et al., 2017) for ImageNet, adopting momentum SGD as the learning procedure to be regularized. We train ResNet18 for 300 epochs with 128 as the batch size. The learning rate is initialized as 0.1 and decays (multiplied by 0.2) at the 80-th, 160-th and 200-th epochs. As for ResNeXt50, we train it for 90 epochs with 256 as the batch size. The learning rate is initialized as 0.1 and decays (multiplied by 0.1) at the 30-th, and 60-th. On the language understanding datasets, we consider fine-tuning BERT (Devlin et al., 2019) via Adam (Kingma & Ba, 2015) as the learning procedure to be regularized. We adopt the pre-trained BERT model provided by huggingface (Wolf et al., 2020) and fine-tune it on the target datasets. The number of epochs is tuned among {3,4,5}, the batch size is 16, the learning rate is tuned among {2e− 5, 5e− 5}, and the dropout rate is 0.1. For Flooding and iFlood, the flood level b is tuned in the range of [0.10,0.50] via grid seach with 0.05 as the step size for ImageNet, and tuned in the range of [0.01,0.10] via grid search with 0.01 as the step size for other datasets.
  Hyper-parameter spaces considered for baseline methods can be found in the Appendix B. All models are implemented using PyTorch (Paszke et al., 2019) and trained on NVIDIA GeForce GTX 1080 Ti or Tesla V100 GPUs. For fair comparison, we search for the optimal configuration of hyperparameters for each method. Then we run each method for 5 times using its optimal configuration and report the averaged results, reducing the randomness in the comparison.
sections.heading: 4.2 IMPROVEMENTS ON GENERALIZATION ABILITY (Q1)
sections.text: We compare iFlood with the baseline methods on benchmark datasets to demonstrate its effectiveness. The experimental results are summarized in Table 1. Obviously, with appropriate configurations (i.e., the strength of regularization), all these regularization methods can improve the performance of the learned model compared to unregularized case, on most of the datasets. This phenomenon can be explained by the fact that, in this experiment, the number of parameters to be estimated is larger than the size of training sample, which makes regularization indispensable. We can also observe that, on both the image classification and language understanding datasets, iFlood provides larger improvements of the performance than those of the baseline methods. This confirms the effectiveness of iFlood in improving generalization ability of learned models, which is brought by the design of iFlood, encouraging the learned model to better fit under-fitted instances while suppressing the confidence on overfitted ones. We also conduct experiments on large-scale dataset Criteo and report the results in Appendix D.1, which confirms the effectiveness of iFlood.
  It is worth pointing out that, iFlood can work well with some other regularization methods, such as weight decay. Under the scenario of over-parameterized neural networks, near-zero training loss is achievable regardless of the regularization constraints posed upon the model parameters. Therefore, when cooperated with weight decay, iFlood controls the ultimate extent to which such models fit the training data, while weight decay affects learning dynamics (Golatkar et al., 2019).
sections.heading: 4.3 STABILITY OF IFLOOD (Q2)
sections.text: In Section 3, we provide analysis about the instability of Flooding, which further motivates us to propose iFlood. In this section, we conduct experiments to show that, compared to Flooding, iFlood can stably converge to solutions with better generalization ability.
  Variances. We use the standard deviation (denoted as “std.”) of test accuracy to measure the differences of the generalization abilities among the learned models. Beseides, we adopt the total variation distance (denoted as DTV) to measure the difference between any two learned models, which can be estimated on the training sample as:
  DDTV(f, g) = 1
  2N
  N ∑ i=1 ∑ y∈Y ∣f(xi)y − g(xi)y ∣, (12)
  where f and g denote two learned models. We estimate the distance for every pair of the ten models and report both the maximum and the averaged values of them. The experimental results are summarized in Table 2.
  The results show that the test accuracy of models learned with iFlood vary less than that of models learned with Flooding, with the largest difference between them could be 0.16%. The less variance in generalization ability demonstrates the stability of iFlood. Meanwhile, from Table 2 we can observe that, the variance of iFlood is much smaller than that of Flooding (e.g., nearly 24% - 70% of Flooding from the aspect of avg DTV). Further, the comparison of total variation distance DTV between Flooding and iFlood is consistent with the individual losses shown in Figure 1, which supports our idea: Under the individual-level constraint, iFlood can achieve “local convergence” and converge to solutions with better generalization ability, in a stabler manner than Flooding.
  The norm of gradients. To further compare the stability between Flooding and iFlood, we train a ResNet18 on CIFAR-10 (Figure 2a) and CIFAR-100 (Figure 2b), and monitor the L1 norm of the gradients at each epoch. From the figure, we can observe that, during the training course of Flooding, the norm of gradients is larger than that of iFlood by a noticeable margin, and the gap has not been filled until the end of training. It implies that even at the end of training, the model parameters learned with Flooding are changed more significantly than those of iFlood, which could cause the instability of learned models.
  The effect of b. We further study the effect of flood level b via training a ResNet18 on CIFAR-10, and report the training accuracies of both Flooding and iFlood with varied b in Figure 2c. From the
  figure we can observe that, as the flood level b increases, the training accuracy of Flooding drops drastically, while that of iFlood stays at the same level until b > 0.60. This result implies that iFlood is much more robust w.r.t. the hyper-parameter b compared to Flooding. Since Flooding just encourages the averaged loss to be close enough to b but has no requirement on the individual losses, some instances stay under-fitted regrading the flood level b. When b is taken to be a relatively large value, the training instances whose individual losses are larger than b might become under-fitted or even mis-classified. Such instances might degrade the model’s generalization ability (Belkin et al., 2019). As iFlood encourages the model to fit every individual instance to the same extent (i.e., ∣Li − b∣ ≈ 0), it can overcome such under-fitted instance issues via achieving “local convergence”.
sections.heading: 4.4 EFFECTIVENESS IN DENOISING (Q3)
sections.text: In this section, we instantiate the noise label setting to evaluate the effectiveness of iFlood in denoising. Following the setting discussed in Section. 3.3.1, the polluted version of CIFAR-10 and SST-2 training datasets are constructed. We train ResNet18 model on the polluted CIFAR-10 dataset and train BERT model on the polluted SST-2 dataset, with Flooding or iFlood applied as the regularizer for denoising. More implementation details can be found in the Appendix E.
  Results are plotted in Figure 3. It can be observed that, on both polluted CIFAR-10 and polluted SST-2 datasets, models learned with iFlood generalize much better than those with Flooding and those without any regularization. Further, the advantages of iFlood become more significant as the ratio of noisy instance α increases. These experimental results confirm our theoretical analysis in Section 3.3.1 that iFlood, as a regularizer, has the capability to denoise data, so that the regularized models can achieve better generalization performances, even when they are trained on noisy datasets.
sections.heading: 5 CONCLUSIONS
sections.text: In this paper, we uncover the instability issue of a recently proposed regularization method Flooding. Our analysis on its causes motivates us to propose a new regularizer iFlood, which encourages the individual losses to approach a specified level. Experimental results on a variety of tasks show that the models learned with iFlood produce stabler generalization improvement. We also theoretically show that, the objective of iFlood over a noisy or biased training sample can serve as a surrogate or even match the maximum likelihood estimator applied to a clean sample, which is also confirmed by our experimental results. Further, we theoretically compare iFlood with some other regularization methods and find that iFlood discounts the over-confident predictions with less inductive bias injected.
sections.heading: A DATASETS
sections.text: We conside both image classification and language understanding tasks, and adopt 7 benchmark datasets for evaluation, including CIFAR-102, CIFAR-1003, SVHN4, ImageNet5, SST-26, QQP7 and QNLI8. The statistics of the datasets are summarized in Table 3.
sections.heading: B DETAILS OF HYPERPARAMETER OPTIMIZATION
sections.text: We randomly split the training data into training and validation sets with the proportion of 9:1, and apply grid search on the validation dataset for hyperparameter optimization (HPO). We adopt the optimal configuration provided by HPO to train and evaluate each method 5 times, alleviating the impact of randomness.
  Specifically, for Label Smoothing (Szegedy et al., 2016), we tune the smoothing parameter value ϵ among {0.05,0.1,0.2}, and adopt a uniform distribution as the prior of label distribution for interpolation. For Confident Penalty (Pereyra et al., 2017), following the original paper, the strength of the confidence penalty β is tuned among {0.1,0.5,1.0,2.0}.
sections.heading: C SGD WITH VARIOUS BATCH SIZE
sections.text: When optimized by SGD with batch size of 1, Flooding coincides with iFlood. However, it is unreasonable to set batch size to 1 for training practical machine learning models, according to the literature and our experimental results shown in Figure 4.
  As mentioned in (Ishida et al., 2020), the objective being optimized by SGD is an upper bound of its desired objective, where we notice that the gap between these two objectives increases w.r.t. the
  2https://www.cs.toronto.edu/ kriz/cifar.html 3https://www.cs.toronto.edu/ kriz/cifar.html 4http://ufldl.stanford.edu/housenumbers/ 5https://image-net.org/challenges/LSVRC/2012/ 6https://nlp.stanford.edu/sentiment/index.html 7https://data.quora.com/First-Quora-Dataset-Release-Question-Pairs 8https://rajpurkar.github.io/SQuAD-explorer/
  decreasing of batch size. As for the practical batch sizes, the design of the loss function of iFlood and Flooding are different, and we have shown iFlood outperforms Flooding in Section 4.2.
sections.heading: D EXPERIMENTS ON LARGE-SCALE DATASET AND GLUE BENCHMARK
sections.text: D.1 LARGE-SCALE DATASET CRITEO
  To demonstrate the effectiveness of the proposed iFlood, we train an MLP for click-through rate prediction on Criteo, a real-world advertisement dataset including around 45 million instances. The AUC of “Unregurized” v.s. Flooding v.s. iFlood are 78.08% v.s. 78.41% v.s. 79.14%. According to literature, a 0.001-level improvement in offline AUC evaluation makes a significant difference. Thus these results confirm the effectiveness of iFlood on large-scale real-world datasets.
  D.2 GLUE BENCHMARK
  The experimental results on GLUE benchmark are shown in Table 49, from which we can observe that iFlood outperforms other baseline methods by a noticeable margin. These experimental results are consistency with those reported in Table 1, which confirm the effectiveness of iFlood in improving generalization ability of learned models.
sections.heading: E DATASETS CONSTRUCTION IN NOISY LABEL
sections.text: Following the setting discussed in noisy label (Section 3.3.1), the polluted version of CIFAR-10 and SST-2 datasets are constructed as follow: We randomly choose a proportion of the original training instances according to α(0 < α < 1), and then pollute the label of each chosen instance by uniformly picking one from its corresponding incorrect classes. We enumerated the noise instance ratio α with 0.1, 0.2, 0.3 and 0.4. Note that the test instances are kept clean to reflect the real-world scenarios. We search for the optimal flood level b in the range of [0.10,0.50] via grid search with 0.05 as the step size for both Flooding and iFlood. The experimental results can be found in Section 4.4.
sections.heading: F ADDITIONAL EXPERIMENTS
sections.text: F.1 LOW-FREQUENCY COMPONENT V.S. HIGH-FREQUENCY COMPONENT
  Inspired by previous study (Wang et al., 2020), we conduct experiments to show how model learned with different regularizers reacts to different levels of details of the data (e.g., the low-frequency component and high-frequency component of images). To be specific, for each instance in the training data, we decompose the data into low-frequency component and high-frequency component w.r.t. different radius thresholds r via applying Fourier transform and inverse Fourier transform. Then we train a ResNet-18 on CIFAR-10 using the raw training data, and evaluate the model on both
  9Label Smoothing and Confident Penalty are not suitable to be adopted on STS-B, since the output dimension is 1.
  low-frequency component and high-frequency component. More details can be referred to Section 3.1 in Wang et al. (2020).
  The experimental results are illustrated in Figure 5, where r = 4/8/12/16 low denotes the lowfrequency component and r = 4/8/12/16 high denotes the high-frequency component. From these experimental results we can conclude that model performs better on low-frequency component than high-frequency component when r = 8/12/16, but worse when r = 4, which are consistency with the results in Wang et al. (2020) (note that BatchNorm is adopted). Compared to Flooding and Vanilla, we can observe that model learned with iFlood catches more low-frequency component (e.g., r = 4/8 low) and less high-frequency component (e.g., r = 8/16 high), which confirms the effectiveness of iFlood in improving the generalization ability of model since low-frequency component is much more generalizable than high-frequency component (Wang et al., 2020).
  F.2 THE NORM OF GRADIENTS
  We train a ResNet18 on CIFAR-10 and CIFAR-100, and monitor the L1 norm of the gradients at each epoch. The experimental results shown in Figure 6a and 6b confirm that: (1) The design of the loss function of Flooding leads to the instability issue, which is supported by the phenomenon that the norm of gradients is larger than other methods by a noticeable margin; (2) The design of the loss function of iFlood brings the merit of significantly reducing the gap of the norm of gradients between Flooding and other methods.
  F.3 EFFECTIVENESS IN DENOISING
  We train a ResNet-18 on polluted CIFAR-10 to evaluate the effectiveness of iFlood in denoising. The generation process of polluted CIFAR-10 can be referred to Appendix E, and the experimental results are shown in Figure 6c. From the figure we can observe that, models learned with iFlood outperform those with other regularizers by a noticeable margin, and the advantages of iFlood become more significant as the ratio of noisy instance α increases. These results confirm the effectiveness of iFlood in denoising.
  F.4 CONFIDENCE DISTRIBUTION
  To further confirm that “iFlood encourages the model to better fit the under-fitted instances while suppressing the confidence of over-fitted ones“ from the perspective of model confidence, we demonstrate the distribution of model confidence in Figure 7. We adopt the same experimental settings as those used in Figure 1: a ResNet-18 is trained on CIFAR-10, and the flooding level b is set to 0.03 for both Flooding and iFlood. From the figure we can observe that, compared with Flooding, iFlood encourages the model to continue to fit the under-fitted instances w.r.t. b, (i.e., the instance with confidence less than e−b ≈ 0.97), while suppressing the over-fitted ones (i.e., the instance with confidence large than e−b ≈ 0.97). These results are consistent with those in Figure 1.