summaries: This paper propose a computationally efficient Wasserstein distributional normalization algorithm for accurate classification of noisy labels. An explicit upper bound for the Wasserstein-2 distance is derived and such a bound can be used as an estimator to determine if a network is over-parameterized. Empirical results on CIFAR-10/100 and Clothing1M suggest that the propose algorithm outperforms other SOTA approaches.  The paper is a contribution that aims at solving the label noise problem. In this setting, the labels are possibly corrupted, this yielding a potentially significant underperformance of a (neural network) classifier when minimizing the empirical risk. This problem is ubiquitous and important in real life scenarii. The paper builds on the idea of small loss criteria, which favors learning on certain samples in the beginning of the learning process, and gradually incorporate uncertain samples along iterations. The paper proposes a novel type of distributional normalization based on Wasserstein distance. It projects uncertain samples on a Wasserstein ball defined wrt. the certain samples. This process is done with a particle based stochastic dynamics, based on a Ornstein-Ulenbeck process. A theoretical Analysis is given, along with results on classical datasets in the symmetric noise setting, open noise and a real world dataset (clothing 1M), for which it achieves very good performances compared to state of the art competing methods.  The paper introduces a novel objective function by imposing geometric constraints on the logits of uncertain samples. The authors' approach is to map the distribution logits of uncertain samples onto the 2-Wasserstein ball centered on the measure of certain samples. To overcome the dilemma of selecting the ball radius, the authors propose a surrogate objective, namely Wasserstein Normalization. An SDE grad flow is proposed for solving the Wasserstein normalization. The paper also keeps the Gaussian parameters as moving average during training in light of batch normalization. The paper both theoretically and empirically validate their method.
abstractText: We propose a novel Wasserstein distributional normalization (WDN) algorithm to handle noisy labels for accurate classification. In this paper, we split our data into uncertain and certain samples based on small loss criteria. We investigate the geometric relationship between these two different types of samples and enhance this relation to exploit useful information, even from uncertain samples. To this end, we impose geometric constraints on the uncertain samples by normalizing them into the Wasserstein ball centered on certain samples. Experimental results demonstrate that our WDN outperforms other state-of-the-art methods on the Clothing1M and CIFAR-10/100 datasets, which have diverse noisy labels. The proposed WDN is highly compatible with existing classification methods, meaning it can be easily plugged into various methods to improve their accuracy significantly.
authors.name: NOISY LABELS
references.authors: Luigi Ambrosio Nicola Gigli Giuseppe Savaré
references.title: Bakry–émery curvature-dimension condition and riemannian ricci curvature bounds
references.venue: The Annals of Probability,
references.authors: Eric Arazo Diego Ortego Paul Albert Noel O’Connor Kevin Mcguinness
references.title: Unsupervised label noise modeling and loss correction
references.venue: In International Conference on Machine Learning,
references.authors: Dominique Bakry Ivan Gentil Michel Ledoux
references.title: Analysis and geometry of Markov diffusion operators, volume 348
references.venue: Springer Science & Business Media,
references.authors: François Bolley Ivan Gentil
references.title: Phi-entropy inequalities for diffusion semigroups
references.venue: Journal de mathématiques pures et appliquées,
references.authors: Pengfei Chen Benben Liao Guangyong Chen Shengyu Zhang
references.title: Understanding and utilizing deep neural networks trained with noisy labels
references.authors: Bharath Bhushan Damodaran Kilian Fatras Sylvain Lobry Rémi Flamary Devis Tuia Nicolas Courty
references.title: Wasserstein adversarial regularization (war) on label noise
references.authors: Bo Han Quanming Yao Xingrui Yu Gang Niu Miao Xu Weihua Hu Ivor W. Tsang Masashi Sugiyama
references.title: Co-teaching: Robust training of deep neural networks with extremely noisy labels
references.venue: NeurIPS,
references.authors: Bo Han Gang Niu Xingrui Yu Quanming Yao Miao Xu Ivor W Tsang Masashi Sugiyama
references.title: Sigua: Forgetting may make learning with noisy labels more robust
references.venue: In ICML,
references.authors: D. Hendrycks M. Mazeika D. Wilson K. Gimpel
references.title: Using trusted data to train deep networks on labels corrupted by severe noise
references.venue: NeurIPS,
references.authors: Sergey Ioffe Christian Szegedy
references.title: Batch normalization: Accelerating deep network training by reducing internal covariate shift
references.venue: In ICML,
references.authors: L. Jiang Z. Zhou T. Leung L.-J. Li L. Fei-Fei
references.title: Mentornet: Regularizing very deep neural networks on corrupted labels
references.authors: Oliver Johnson Yurii Suhov
references.title: Entropy and random vectors
references.venue: Journal of Statistical Physics, 104: 145–165,
references.authors: Samuli Laine Timo Aila
references.title: Temporal ensembling for semi-supervised learning
references.venue: arXiv preprint arXiv:1610.02242,
references.authors: Kimin Lee Sukmin Yun Kibok Lee Honglak Lee Bo Li Jinwoo Shin
references.title: Robust inference via generative classifiers for handling noisy labels
references.authors: Junnan Li Richard Socher Steven CH Hoi
references.title: Dividemix: Learning with noisy labels as semisupervised learning
references.venue: In International Conference on Learning Representations,
references.authors: Junnan Li Yongkang Wong Qi Zhao Mohan S Kankanhalli
references.title: Learning to learn from noisy labeled data
references.venue: In CVPR,
references.authors: Michal Lukasik Srinadh Bhojanapalli Aditya Krishna Menon Sanjiv Kumar
references.title: Does label smoothing mitigate label noise
references.venue: arXiv preprint arXiv:2003.02819,
references.authors: Yueming Lyu Ivor W. Tsang
references.title: Curriculum loss: Robust learning and generalization against label corruption
references.venue: In ICLR,
references.authors: Yueming Lyu Ivor W. Tsang
references.title: Curriculum loss: Robust learning and generalization against label corruption
references.venue: In ICLR,
references.authors: Xingjun Ma Yisen Wang Michael E Houle Shuo Zhou Sarah M Erfani Shu-Tao Xia Sudanthi Wijewickrema James Bailey
references.title: Dimensionality-driven learning with noisy labels
references.venue: arXiv preprint arXiv:1806.02612,
references.authors: Luigi Malagò Luigi Montrucchio Giovanni Pistone
references.title: Wasserstein riemannian geometry of gaussian densities
references.venue: Information Geometry,
references.authors: Robert J McCann
references.title: A convexity principle for interacting gases
references.venue: Advances in mathematics,
references.authors: Ivan Nourdin Giovanni Peccati Yvik Swan
references.title: Entropy and the fourth moment phenomenon
references.venue: Journal of Functional Analysis,
references.authors: Felix Otto Cédric Villani
references.title: Generalization of an inequality by talagrand and links with the logarithmic sobolev inequality
references.venue: Journal of Functional Analysis,
references.authors: G. Patrini A. Rozza A.K. Menon R. Nock L. Qu
references.title: Making deep neural networks robust to label noise: A loss correction approach
references.venue: In CVPR,
references.authors: Giorgio Patrini Alessandro Rozza Aditya Krishna Menon Richard Nock Lizhen Qu
references.title: Making deep neural networks robust to label noise: A loss correction approach
references.venue: In CVPR,
references.authors: Geoff Pleiss Tianyi Zhang Ethan R Elenberg Kilian Q Weinberger
references.title: Identifying mislabeled data using the area under the margin ranking
references.venue: arXiv preprint arXiv:2001.10528,
references.authors: M. Ren W. Zeng B. Yang R. Urtasun
references.title: Learning to reweight examples for robust deep learning
references.venue: In ICML,
references.authors: Daiki Tanaka Daiki Ikami Toshihiko Yamasaki Kiyoharu Aizawa
references.title: Joint optimization framework for learning with noisy labels
references.authors: C. Villani
references.title: Optimal Transport: Old and New
references.venue: Grundlehren der mathematischen Wissenschaften. Springer Berlin Heidelberg,
references.authors: Yisen Wang Weiyang Liu Xingjun Ma James Bailey Hongyuan Zha Le Song Shu-Tao Xia
references.title: Iterative learning with open-set noisy labels
references.authors: Yisen Wang Xingjun Ma Zaiyi Chen Yuan Luo Jinfeng Yi James Bailey
references.title: Symmetric cross entropy for robust learning with noisy labels
references.authors: Hongxin Wei Lei Feng Xiangyu Chen Bo An
references.title: Combating noisy labels by agreement: A joint training method with co-regularization
references.authors: Tong Xiao Tian Xia Yi Yang Chang Huang Xiaogang Wang
references.title: Learning from massive noisy labeled data for image classification
references.venue: In CVPR,
references.authors: Yilun Xu Peng Cao Yuqing Kong Yizhou Wang
references.title: L_dmi: A novel information-theoretic loss function for training deep nets robust to label noise
references.venue: NeurIPS,
references.authors: Kun Yi Jianxin Wu
references.title: Probabilistic end-to-end noise correction for learning with noisy labels
references.authors: Xingrui Yu Bo Han Jiangchao Yao Gang Niu Ivor Tsang Masashi Sugiyama
references.title: How does disagreement help generalization against label corruption
references.authors: Xiyu Yu Tongliang Liu Mingming Gong Dacheng Tao
references.title: Learning with biased complementary labels
references.venue: In Proceedings of the European Conference on Computer Vision (ECCV),
references.authors: Hongyi Zhang Moustapha Cisse Yann N Dauphin David Lopez-Paz
references.title: mixup: Beyond empirical risk minimization
references.venue: In International Conference on Learning Representations,
references.authors: Zhilu Zhang Mert Sabuncu
references.title: Generalized cross entropy loss for training deep neural networks with noisy labels
references.venue: In NeurIPS,
references.authors: In Arazo
references.title: beta-mixture model as a label distribution on label space. But due to the non-deterministic type of noisy label distribution, it sometimes fails to train with extremely non-uniform type of noise. For example, Arazo et al. (2019) reported failure case with Clothing1M dataset. It seems that fundamental assumption on noise model of mixup will be improved in future
references.authors: Fs?μ [Proposition 4.4.2 Bakry
references.title: 2013)], we can derive the following probability inequality: Fs?μ [σ(Xs?(w)) ≥ EFs?μ[σ
sections.text: We propose a novel Wasserstein distributional normalization (WDN) algorithm to handle noisy labels for accurate classification. In this paper, we split our data into uncertain and certain samples based on small loss criteria. We investigate the geometric relationship between these two different types of samples and enhance this relation to exploit useful information, even from uncertain samples. To this end, we impose geometric constraints on the uncertain samples by normalizing them into the Wasserstein ball centered on certain samples. Experimental results demonstrate that our WDN outperforms other state-of-the-art methods on the Clothing1M and CIFAR-10/100 datasets, which have diverse noisy labels. The proposed WDN is highly compatible with existing classification methods, meaning it can be easily plugged into various methods to improve their accuracy significantly.
sections.heading: 1 INTRODUCTION
sections.text: The successful results of deep neural networks (DNNs) on supervised classification tasks heavily rely on accurate and high-quality label information. However, annotating large-scale datasets is extremely expensive and a time-consuming task. Because obtaining high-quality datasets is very difficult, in most conventional works, training data have been obtained alternatively using crowd-sourcing platforms Yu et al. (2018) to obtain large-scaled datasets, which leads inevitable noisy labels in the annotated samples.
  While there are numerous methods that can deal with noisy labeled data, recent methods actively adopt the small loss criterion, which enables to construct classification models that are not susceptible to noise corruption. In this learning scheme, a neural network is trained using easy samples first in the early stages of training. Harder samples are then gradually selected to train mature models as training proceeds. Jiang et al. (2018) suggested collaborative learning models, in which a mentor network delivers the data-driven curriculum loss to a student network. Han et al. (2018); Yu et al. (2019) proposed dual networks to generate gradient information jointly using easy samples and employed this information to allow the networks to teach each other. Wei et al. (2020) adopted a disagreement strategy, which determines the gradient information to update based on disagreement values between dual networks. Han et al. (2020) implemented accumulated gradients to escape optimization processes from over-parameterization and to obtain more generalized results. In this paper, we tackle to solve major issues raised from the aforementioned methods based on the small-loss criterion, as follows.
  In comprehensive experiments, the aforementioned methods gain empirical insight regarding network behavior under noisy labels. However, theoretical and quantitative explanation have not been closely investigated. In contrast, we give strong theoretical/empirical explanations to understand the network under noisy labels. In particular, we present an in-depth analysis of small loss criteria in a probabilistic sense. We exploit the stochastic properties of noisy labeled data and develop probabilistic descriptions of data under the small loss criteria, as follows. Let P be a probability measure for the pre-softmax logits of the training samples, l be an objective function for classification, and 1{·} be an indicator function. Then, our central object to deal with is a truncated measure defined as
  X ∼ µ|ζ = 1{X;l(X)>ζ}P P[l(X) > ζ] , Y ∼ ξ|ζ = 1{X;l(Y )≤ζ}P P[l(Y ) ≤ ζ] , (1)
  where X and Y , which are sampled from µ|ζ and ξ|ζ, denote uncertain and certain samples defined in the pre-softmax feature space1 (i.e.,Rd), respectively. In equation 1, µ and ξ denote the probability measures of uncertain and certain samples, respectively, and ζ is a constant. Most previous works have focused on the usage of Y and the sampling strategy of ζ, but poor generalization capabilities based on the abundance of uncertain samples X has not been thoroughly investigated, even though these samples potentially contain important information. To understand the effect of noisy labels on the generalized bounds, we provide the concentration inequality of uncertain measure µ, which renders the probabilistic relation between µ and ξ and learnability of the network under noisy labels.
  While most conventional methods Han et al. (2018); Wei et al. (2020); Li et al. (2019a); Yu et al. (2019) require additional dual networks to guide misinformed noisy samples, the scalability is not guaranteed due to the existence of dual architectures, which have the same number of parameters as the base network. To alleviate this problem, we build a statistical machinery, which should be fully non-parametric, simple to implement, and computationally efficient to reduce the computational complexity of conventional approaches, while maintaining the concept of small-loss criterion. Based on the empirical observation of ill-behaved certain/uncertain samples, we propose the gradient flow in the Wasserstein space, which can be induced by simulating non-parametric stochastic differential equation (SDE) with respect to the Ornstein-Ulenbeck type to control the ill-behaved dynamics. The reason for selecting these dynamics will be thoroughly discussed in the following sections.
  Thus, key contributions of our work are as follows.
  • We theoretically verified that there exists a strong correlation between model confidence and statistical distance between X and Y . We empirically investigate that the classification accuracy worsens when the upper-bound of 2-Wasserstein distance W2(µ, ξ) ≤ ε (i.e., distributional distance between certain and uncertain samples) drastically increase. Due to the empirical nature of upper-bound ε, it can be used as an estimator to determine if a network suffers from over-parameterization.
  • Based on empirical observations, we develop a simple, non-parametric, and computationally efficient stochastic model to control the observed ill-behaved sample dynamics. As a primal object, we propose the stochastic dynamics of gradient flow (i.e.,, Ornstein-Ulenbeck process) to simulate simple/non-parametric stochastic differential equation. Thus, our method do not require any additional learning parameters.
  • We provide important theoretical results. First, the controllable upper-bound ε with the inverse exponential ratio is induced, which indicates that our method can efficiently control the diverging effect of Wasserstein distance. Second, the concentration inequality of transported uncertain measure is presented, which clearly renders the probabilistic relation between µ and ξ.
sections.heading: 2 RELATED WORK
sections.text: Curriculum Learning & Small-loss Criterion. To handle noisy labels, Han et al. (2018); Yu et al. (2019); Jiang et al. (2018); Wei et al. (2020); Lyu & Tsang (2020a); Han et al. (2020) adopted curriculum learning or sample selection frameworks. However, these methods only consider a small number of selected samples, where large portion of samples are excluded at the end of the training. This inevitably leads to poor generalization capabilities. However, this conflicts with sample selection methods because a large portion of training samples are gradually eliminated. By contrast, our method can extract useful information from unselected samples X ∼ µ (i.e., uncertain samples) and enhance these samples (e.g., X ′ ∼ Fµ) for more accurate classification. Chen et al. (2019) iteratively apply cross-validation to randomly partitioned noisy labeled data to identify most samples that have correct labels. To generate such partitions, they adopt small-loss criterion for selecting samples.
  Loss Correction & Label Correction. Patrini et al. (2017a); Hendrycks et al. (2018); Ren et al. (2018) either explicitly or implicitly transformed noisy labels into clean labels by correcting classification losses. Unlike these methods, our method transforms the holistic information from uncertain samples into certain samples, which implicitly reduces the effects of potentially noisy labels. While correction of label noisy by modifying the loss-dynamics do not perform well under extreme noise environments, Arazo et al. (2019) adopt label augmentation method called MixUp Zhang et al. (2018).
  1Due to the technical difficulties, we define our central objects on pre-softmax space rather than label space, i.e., the space of σ(X), σ(Y ), where σ indicates softmax function. Please refer to Appendix for more details.
  Distillation. Li et al. (2019b) updated mean teacher parameters by calculating the exponential moving average of student parameters to mitigate the impact of gradients induced by noisy labels. Lukasik et al. (2020) deeply investigated the effects of label smearing for noisy labels and linked label smoothing to loss correction in a distillation framework. Similar to these methods, our method leverages the useful properties of distillation models. We set ν as a pivot measure, which guides our normalization functional Fµ for uncertain measures. This is similar to self-distillation because uncertain training samples are forced to be normalized to those of past states.
  Other methods. Lee et al. (2019) induced a robust generative classifier based on pre-trained deep models. Similar to our method, Damodaran et al. (2019) designed a constraint on the Wasserstein space and adopted an adversarial framework for classification models of noisy labeled data by implementing semantic Wasserstein distance. Pleiss et al. (2020) identify noisy labeled samples by considering AUM statistics which exploits differences in training dynamics of clean and mislabeled samples. In most recent work, Li et al. (2019a) adopts semi-supervised learning (SSL) methods to deal with noisy labels where the student network utilizes both labeled/unlabeled samples to perform semi-supervised learning guided by the other teacher network.
sections.heading: 3 DISTRIBUTIONAL NORMALIZATION
sections.text: Because our main target object is a probability measure (distribution), we first define an objective function in a distributional sense. Let l be cross entropy and r̂ be a corrupted label random vector for an unknown label transition matrix from a clean label r which is independent of X , with label transition matrix Q. Then, a conventional objective function for classification with noisy labels can be defined as follows:
  min µ J [µ] = min µ EX∼µ,r̂|Q [l(X; r̂)] . (2)
  However, due to the significant changes in label information, the conventional objective function defined in equation 2 cannot be used for accurate classification. Instead of directly using uncertain samples X ∼ µ as in previous works, we normalize µ in the form of a metric ball and present a holistic constraint. For a clear mathematical description, we first introduce the following definition. Definition 1. (Wasserstein ambiguity set) Let P2(Rd) = {µ : Eµd2E(x0, x) <∞,∀x0 ∈ Rd} be a 2-Wasserstein space, where d denotes the number of classes, dE is Euclidean distance defined on Rd. Then, we define a Wasserstein ambiguity set (i.e., metric ball) in this space as follows:
  BW2(ν, ε) = { µ ∈ P2 ( Rd ) :W2(µ, ν) ≤ ε } , (3)
  whereW2 denotes the 2-Wasserstein distance and ν is the pivot measure. Then, we propose a new objective function by imposing geometric constraints on µ as follows:
  min Fµ∈BW2 (ν,ε),ξ J [Fµ] + J [ξ] = min θ EX∼Fµθ,r̂[l(X; r̂)] + EX∼ξθ,r̂[l(Y ; r̂)], (4)
  where F : P2(Rd)→ P2(Rd) is a functional for probability measures, which assures the constraint on Fµ (i.e., Fµ ∈ BW2(ν, ε)) and our main objective. The right-hand side of equation equation 4 is equivalent vectorial form of distributional form in left-hand side. While our main objects are defined on pre-softmax, both probability measures µθ and ξθ is parameterized by neural network with parameters θ. This newly proposed objective function uses the geometrically enhanced version of an uncertain measure Fµ with a certain measure ξ. In equation 4, probability measure ν is defined as follows: ν = arg minJ [ξk? ], where ξk denotes a certain measure at the current k-th iteration and k? ∈ Ik−1 = {1, · · · , k − 1}. In other words, our method finds the best probability measure that represents all certain samples so far at training time, where the uncertain measures are transported to be lying in the Wasserstein ball centered on ν. In equation 4, the Wasserstein constraint on Fµ enforces uncertain measures statistically resemble ν from a geometric perspective (i.e.,W2(ν,Fµ) ≤ ε). Now, an important question naturally stems from the aforementioned analysis: how can we select the optimal radius ε? Clearly, finding an F that induces a small ε ≈ 0 is suboptimal because Fµ ≈ ν and using objective function J [Fµ ≈ ν] can lead to the following critical problem. As the optimization process proceeds, enhanced uncertain samples X ′ ∼ Fµ contribute less and less, because it is statistically identical to ν, meaning our objective in equation 4 would receive little benefits from these transported uncertain samples. By contrast, if we adopt a large radius for ε, enhanced uncertain samples will be statistically and geometrically unrelated to ν, which causes the normalized measure Fµ to yield large losses and violates our objective.
  To overcome two problems above and select the radius, we make a detour, i.e., a Gaussian measure, for cutting the path between ν and Fµ (i.e., ν → N (mν ,Σν)→ Fµ) rather than directly calculating the geodesic between ν and Fµ (i.e., ν → Fµ). Specifically, we decompose the original constraint in equation 4 into two terms using the triangle inequality of the Wasserstein distance:
  W2 (ν,Fµ) ≤ ε =W2 (ν,N (mν ,Σν))︸ ︷︷ ︸ d1: Intrinsic statistics +W2 (N (mν ,Σν),Fµ)︸ ︷︷ ︸ d2: Wasserstein Normalization . (5)
  The first intrinsic statistics term sets a detour point as a Gaussian measure, for which the mean and covariance are the same as those for ν (i.e., mν = EY∼ν [Y ] and Σν = CovY∼ν [Y ]). The Wasserstein upper bound of this term is only dependent on the statistical structure of ν because (mν ,Σν) is dependent on ν. Thus, this term induces a data-dependent, non-zero constant upper bound whenever ν 6= N and can prevent the upper-bound from collapsing to ε → 0, regardless of F . This gives huge advantage when dealing with ε because the first term can be considered a fixed constant during the training. The second normalization term represents our central objective. F facilitates geometric manipulation in the Wasserstein space and prevent uncertain measure µ from diverging, where µ is normalized onto the Wasserstein ambiguity BW2(ν, ε) in Fig1. The theoretical/numerical advantages of setting detour measure as Gaussian is well-explained following section.
sections.heading: 3.1 WASSERSTEIN NORMALIZATION
sections.text: In the previous section, we present a novel objective function that imposes a geometric constraint on µ such that the transformed measure Fµ lies in BW2(ν, ε) for ν. Now, we specify F and relate it to the Gaussian measure (generally Gibbs measure). For simplicity, we denote Nν = N (mν ,Σν). Proposition 1. F : R+×P2 → P2 is a functional on the probability measure such thatF [t, µ] = µt, where dµt = ptdNν , dNν = dqtdx, and µt is a solution to the following continuity equations:
  ∂tµt = ∇ · (µtvt) , (6)
  which is read as ∂tp(t, x) = ∇ · (p(t, x)∇ log q(t, x)) in a distributional sense. Then, a uniquely defined functional Ft[·] = F [t, ·] normalizes µ onto BW2 (Nν , e−tK2 (µ)), where K2(µ) > 0 is a constant that depends on µ.
  It is well known that the solution to equation 6 induces a geodesic in the 2-Wasserstein space (Villani (2008)), which is the shortest path from µ = µt=0 to Nν . The functional Ft generates a path for µt, in which the distance is exponentially decayed according to the auxiliary variable t and constant K2, meaningW2(Nν ,Ftµ) ≤ K2e−t. This theoretical results indicates that the Wasserstein distance of second term in equation 5 can be reduced/controlled with exponential ratio. Thus, by setting a different t, our method can efficiently control the diverging distance in equation 5. Unfortunately, it is typically intractable to compute the partial differential equation (PDE) in equation 6.
  Algorithm 1 Wasserstein Distributional Normalization Require: α ∈ [0, 0.2], % ∈ [0.1, 0.65], T = 64,∆t = 10−4, τ = 0.001,
  for k = 1 to K (i.e., the total number of training iterations) do 1) Select uncertain (1− ρ)N and certain ρN samples from the mini-batch N . {Y nk }{n≤ρN} ∼ ξk, {X n k }{n≤(1−ρ)N} ∼ µk
  2) Update the most certain measure ν. if J [ξk] < J [ν] then ν ← ξk,mν ← E [Yk], and Σν ← Cov [Yk] end if 3) Update the moving geodesic averageN (mα,Σα). Solve the Ricatti equation T ΣνT = Σξk . Σα = ((1− α)Id + αT ) Σν ((1− α)Id + αT ) and mα = (1− α)mν + αmξk 4) Simulate the discrete SDE for T steps. for t = 0 to T − 1 do Xnk,t+1 = −∇φ(Xnk,t;mα)∆t + √ 2τ−1Σαν dW n t s.t. { Xnk,t=0 } ∼ µk, { Xnk,t=T } ∼ FTµk end for 5) Update the network with the objective function. J [Fµk] + J [ξk] = EFT µk [l(Xk,T ; r̂)] + Eξk [l(Yk; r̂)]
  end for
  To solve this problem, we adopt particle-based stochastic dynamics, which enables tractable computation. There exists a unique iterative form corresponding PDE in equation 6 which is called as multi-dimensional Ornstein-Ulenbeck process, which can be approximated using particle-based dynamics. In particular, we draw N(1 − %) uncertain samples from a single batch of N samples using equation 1 for hyper-parameter 0 ≤ % ≤ 1. We then simulate a discrete stochastic differential equation (SDE) for each particle using the Euler-Maruyama scheme as follows:
  Xnt+1 = X n t −∇φ (Xnt ;mν) ∆t + √ 2τ−1∆tΣZ n I , (7)
  where φ (Xt;mν) = τ2d 2 E (Xt,mν), n ∈ {1 · · · , N(1− %)}, dE is a Euclidean distance, and N is a single mini-batch size. We selected OU process as our stochastic dynamic due to the following reasons: First, we want to build computationally efficient, and non-parametric method to estimate/minimize the second term of equation 5. The SDE in equation 7 corresponding OU process have simple form with fixed drift and diffusion terms which is invariant over times which makes us to induce the non-parametric representations of simulation of SDE. While the simulation of equation 7 is just non-parametric for-loops in implementation algorithm, our method is computationally very efficient compared to other baseline methods such as Han et al. (2018). Second, when estimating empirical upper-bound of Wasserstein distance, OU process allows us to use explicit form called Meheler’s formula which can be efficiently estimated (Please refer to Appendix for more details). The overall procedure for our method is summarized in Algorithm 1.
sections.heading: 3.2 WASSERSTEIN MOVING GEODESIC AVERAGE
sections.text: In our experiments, we observe that the best measure ν is not updated for a few epochs after the training begins. This is problematic because ν diverges significantly from the current certain measure ξk, which is equivalent to the normalized measure Fµk diverging from ξk, meaning XT and Y become increasingly statistically inconsistent. To alleviate this statistical distortion, we modify detour measure from Nν to other Gaussian measure, which allows us to capture the statistics of both ξk and ν. Inspired by the moving average of Gaussian parameters in batch normalization Ioffe & Szegedy (2015), we propose the Wasserstein moving geodesic average. Specifically, we replace Gaussian parameters {mν ,Σν} with {mα,Σα} such that mα = (1 − α)mν + αmξk and Σα = ((1− α)Id + αT ) Σν ((1− α)Id + αT ), where T is a solution to the Riccati equation T ΣνT = Σξk . Therefore our final detour Gaussian measure is set to Nαν := N (m(α),Σ(α)), 0 ≤ α ≤ 12.
sections.heading: 4 THEORETICAL ANALYSIS
sections.text: In equation 5, we select the detour point as a Gaussian measure because this measure can provide a statistical structure, which is similar to that of the optimal ν. In addition to this heuristic motivation, setting a detour point as a Gaussian measure (Gibbs measure) also provides theoretical advantages, e.g., the theoretical upper bound of the Wasserstein constraint terms. In this section, we investigate the explicit upper bounds of two terms in equation 5, which are naturally induced by the SDE.
  2Please refer to Appendix C.4 for more details.
  Proposition 2. A scalar 0 < β <∞ exists and depends on ν, resulting in the following inequality: W2(ν,Ftµ) ≤ ε = K1(ν) ∨ [ e−tK2(µ) +K2(ν) ] , (8)
  where λmax(Σν) denotes the maximum eigenvalue of the covariance matrix Σν and for some constant 0 < K1 <∞, we have K1(ν) = √ dβλmax(Σν) + ‖EνY ‖2 which is only dependent on ν.
  Intuitively, K2(µ) can be interpreted as an indicator that tells us how the uncertain measure µ is diffused, whereas the designed term e−tK2(µ) controls the upper bound of the Wasserstein distance using a variable t. The other term K2(ν) does not vanish even with a very large t, which assures a non-collapsing upper-bound ε.
  Proposition 3. (Concentration inequality for the normalized uncertain measure). Assume that there are some constants T ∈ [ 1η ,∞), η ≥ 0 such that the following inequality holds:
  EFTµ[f2]− [EFTµ[f ]]2 ≤ (1 + η)EFTµ[A∇fT∇f ], f ∈ C∞0 (Rd), (9)
  for A ∈ Sym+d and D(A,Σν) ≤ aη for some a > 0 with any metric D defined on Sym + d . In this case, there is a δ such that the following probability inequality for an uncertain measure is induced:
  FTµ ( |σ − Eν [σ]| ≥ δ ) ≤ 6e− √ 2δ 3 2 K2(µ) , (10)
  where σ denotes a soft-max function.
  In equation 10, we show that the label information induced by the normalized uncertain measure is close to that of most certain measure Eν [σ], where the upper bound is exponentially relative to the initial diffuseness of µ (i.e.,K2(µ)). Because the upper bound of the probability inequality does not collapse to zero and FTµ is concentrated around the most certain labels (i.e.,Eν [σ]), the uncertain sample XT ∼ FTµ helps our method avoid over-parameterization.
sections.heading: 4.1 EMPIRICAL UNDERSTANDINGS
sections.text: We investigate the theoretical upper bound of the Wasserstein ambiguity (i.e., radius of the Wasserstein ball) for Fµ and its corresponding probability inequality. To provide more in-depth insights into the proposed method, we approximate the upper bound and demonstrate that our Wasserstein normalization actually makes neural networks more robust to label noise.
  As we verified previously, according to Proposition 2, the following inequality holds:
  W2(Ftµ, ν) ≤ ε = K1(ν) ∨ (K2(ν) +K2(Ftµ)) . (11)
  Because the first term K1(ν) is constant, dependent on ν, and generally small compared to the second term with t ≤ T , we only examine the behavior of the second term K2(ν) +K2(Ftµ), which can be efficiently approximated using a simple form. Because our detour measure is Gaussian, we have the following inequality for any h ∈ C∞0 (Rd)3:
  K̂2(µ) = lim s→0
  1 s EX,Z∼NI
  [ h ( e−sX + √ 1− e−2s(Σ 1 2 ν Z + mν) ) − h(X) ] ≤ K2(µ), (12)
  where this equality holds if h is selected to induce a supremum over the set C∞0 . For approximation, we simply consider h(X) = ‖X‖2 as a test function. In this case, the following inequality naturally holds: ε̂ = K̂2(ν) + K̂2(Fµ) ≤ K2(ν) + K2(Fµ) ≤ K1(ν) ∨ (K2(ν) + K2(Fµ)) = ε. Thus, ε̂ can be considered as an approximation of the theoretical upper bound ε suggested in Proposition 2. Subsequently, we investigate the effects of Wasserstein normalization based on K̂2(µ) in equation 12.
  (1) The proposed WDN ensures that the Wasserstein ambiguity is bounded. We examine the relation between ε̂ and test accuracy in an experiment using the CIFAR-10 dataset with symmetric noise at a ratio of 0.5. Fig.2 presents the landscape for the log10-scaled cumulative average of ε̂ and test accuracy over epochs. The red dotted lines represent the landscape of the vanilla network with cross-entropy loss, where ε̂k = K̂2(νk)+K̂2(Ft=0µk) and k is the epoch index. In this case, the time constant t is set to zero, because Wasserstein normalization is not employed for the vanilla network. The black lines indicate the landscape of the proposed method, where ε̂k = K̂2(νk) + K̂2(Ft=Tµk)
  3Please refer to Appendix C.2 for additional details.
  in this case. It is noteworthy that the test accuracy of the vanilla network begins to decrease after 13- epochs (red-dotted vertical lines in the top-right plot), whereas the Wasserstein ambiguity (i.e., upper bound of the Wasserstein distance) increases quadratically in the top-left plot. These experimental results verify that the distance between uncertain and most certain measure (i.e., ν) becomes large in the 2-Wasserstein space without any constraints in vanilla networks. They also indicate a definite relationship between Wasserstein ambiguity and test accuracy. In the proposed WDN, Wasserstein ambiguity can be efficiently bounded (i.e., lim supk ε̂k ≈ 2.15) as the test accuracy continues to increase, even after 13-epochs. For detailed analysis, we compute the deviation of an empirical upper bound as follows: ∆̂k = ε̂k − ε̂k−1. In the gray regions, the deviation for the vanilla network is grater than 2.5× 10−2, i.e., ∆k > 2.5× 10−2. Then, its test accuracy begins to drop, as shown in Fig.2. In contrast to the vanilla network, the maximum deviation of the proposed WDN is bounded above by a very small value (supk ∆̂k ≤ 8× 10−3). (2) The proposed WDN helps networks to escape from over-parameterization. To analyze the behavior of deep neural networks under over-parameterization with and without the proposed WDN, we design several variants of the WDN, which begin at delayed epochs. The green, orange, and blue curves in the second row of Fig.2 represent the landscapes, when our WDN is applied after kd ∈ {10, 15, 20} epochs, respectively. In this experiment, the upper bound ε̂k is defined as
  ε̂k = { K̂2(νk) + K̂2(Ft=0µk), if k < kd, K̂2(νk) + K̂2(Ft=Tµk), else k ≥ kd.
  (13)
  Consider kd = 20, which is represented by the blue dotted vertical lines. Before our WDN is applied (i.e., k < kd), the network suffers from over-parameterization, which induces a significant performance drop, as indicated by the blue curve in the bottom-right plot. However, the network rapidly recovers to normal accuracy following Wasserstein normalization (i.e., k ≥ kd). Please note that similar behavior can be observed in the green and orange curves. In particular, the orange curve produces less fluctuations than the blue curve in terms of test accuracy. This indicates that the proposed WDN can help a network escape from over-parameterization by imposing geometric constraints on the Wasserstein space with proposed method.
  (3) The proposed WDN can derive data-dependent bounds according to different noise levels. Another interesting point in Fig.2 is that all curves, excluding the red curve, converge to specific numbers 2.15 = ε := lim infk ε̂k ≤ lim supk ε̂k := ε̄ = 2.2. The upper bound ε̄ is neither overly enlarged nor collapsed to zero, while the lower bound ε is fixed for all curves. We argue that this behavior stems from the geometric characteristics of the proposed method, where the first term in equation 5, namelyW2(ν,Nν) ∝ K̂2(ν), is a non-zero data-dependent term that is minimized by the proposed geometric constraint. Therefore, we can derive the following relationship:
  [W2(ν,Fµ) ≤ W2(ν,Nν) +W2(Nν ,Fµ)]⇓ ∝ [K̂2(ν) + K̂2(Fµ) = ε̂]⇓. (14) This empirical observation verifies that a detour point, which is set as a Gaussian measure, can induce the data-dependent bound (ε, ε̄), where our data-dependent bound can vary according to different
  noise levels and efficiently leverage data-dependent statistics. Fig.2 indicates that classification models with more stable data-dependent bounds also induce more stable convergence in test accuracy.
sections.heading: 5 EXPERIMENTS
sections.heading: 5.1 EXPERIMENTS ON THE CIFAR-10/100 DATASET
sections.text: We used settings similar to those proposed by Laine & Aila (2016); Han et al. (2018) for our experiments on the CIFAR10/100 dataset. We used a 9-layered CNN as a baseline architecture with a batch size of 128. We used the Adam optimizer with (β1, β2) = (0.9, 0.99), where the learning rate linearly decreased from 10−3 to 10−5. Synthetic Noise. We injected label noise into clean datasets using a noise transition matrix Qi,j = Pr(r̂ = j|r = i), where a noisy label r̂ is obtained from a true clean label r. We defined Qi,j by following the approach discussed by Han et al. (2018). For symmetric noise, we used the polynomial, % = −1.11r2 + 1.78r + 0.04 for 0.2 ≤ r ≤ 0.65, where r is the noise ratio. For the asymmetric noise, we set % to 0.35. To select the enhanced detour measure, we set α to 0.2 for the Wasserstein moving geodesic average in all experiments. We trained our classification model over 500 epochs because the test accuracy of our method continued increasing, whereas those of the other methods did not. We compared our method with other state-of-the-art methods, including [MentorNet, Jiang et al. (2018)], [Co-teaching, Han et al. (2018)], [Co-teaching+, Yu et al. (2019)], [GCE, Zhang & Sabuncu (2018)], [RoG, Lee et al. (2019)], [JoCoR, Wei et al. (2020)], [NPCL, Lyu & Tsang (2020b)], [SIGUA, Han et al. (2020)], and [DivideMix, Li et al. (2019a)]. As shown in Table 1, the proposed WDN significantly outperformed other baseline methods. Please note that our WDN utilizes a simple Gaussian measure as a target pivot measure. Thus, there are potential risks when handling highly concentrated and non-smooth types of noise (e.g., asymmetric noise). Nevertheless, the proposed WDN still produced accurate results, even with asymmetric noise. In this case, a variant of our WDN (i.e., WDNcot) exhibited the best performance. Open-set Noise. In this experiment, we considered the open-set noisy scenario suggested by Wang et al. (2018), where a large number of training images were sampled from other CIFAR-100 dataset; however, these images were still labeled according to the classes in the CIFAR-10 dataset. We used a 9-layered CNN, which also used in our previous experiment. For hyper-parameters, we set % and α to 0.5 and 0.2, respectively. As shown in Table 2, our method achieved state-of-the-art accuracy. Collaboration with Other Methods. Because our core methodology is based on small loss criteria, our method can collaborate with co-teaching methods. In Han et al. (2018), only certain samples (Y ∼ ξ) were used for updating colleague networks, where the number of uncertain samples gradually decreased until it reached a predetermined portion. To enhance potentially bad statistics for co-teaching, we taught dual networks by considering a set of samples (Y,XT ), where XT ∼ FTµ are uncertain samples enhanced using equation 7.
  Table 1 shows the test accuracy results for the proposed collaboration model with a co-teaching network (WDNcot). This collaboration model achieved the most accurate performance for the CIFAR100 dataset with asymmetric noise, which verifies that our WDN can be integrated into existing methods to improve their performance significantly, particularly when the density of pre-logits is highly-concentrated. Fig.3 reveals that co-teaching quickly falls into over-parameterization and induces drastic drop in accuracy after the 15th-epoch. WDNcot also exhibits a slight accuracy drop. However, it surpassed the baseline co-teaching method by a large margin (+7%) during training. This demonstrates that our enhanced samples XT can alleviate the over-parameterization issues faced by conventional co-teaching models, which helps improve their accuracy significantly.
sections.heading: 5.2 EXPERIMENTS ON A REAL-WORLD DATASET
sections.text: To evaluate our method on real-world datasets, we employed the Clothing1M dataset presented by Xiao et al. (2015), which consists of 1M noisy, labeled, and large-scale cloth images with 14 classes collected from shopping websites. It contains 50K, 10K, and 14K clean images for training, testing, and validation, respectively. We only used a noisy set for training; for testing, we used a clean set. We set α = 0.2 and % = 0.1. For fair comparison, we followed the settings suggested in previous works. We used a pre-trained ResNet50 for a baseline architecture with a batch size of 48. For the pre-processing steps, we applied a random center crop, random flipping, and normalization to 224× 224 pixels. We adopted the Adam optimizer with a learning rate starting at 10−5 that linearly decayed to 5× 10−6 at 24K iterations. Regarding the baseline methods, we compared the proposed method to [GCE, Zhang & Sabuncu (2018)], [D2L, Ma et al. (2018)], [FW, Patrini et al. (2017b)], [WAR, Damodaran et al. (2019)], [SL, Wang et al. (2019)], [JOFL, Tanaka et al. (2018)], [DMI, Xu et al. (2019)], [PENCIL, Yi & Wu (2019)], and [MLNT, Li et al. (2019b)]. Table 3 reveals that our method achieved competitive performance as comparison with other baseline methods.
sections.heading: 5.3 COMPUTATIONAL COST
sections.text: Because Co-teaching, JoCoR, and DivideMix use additional networks, the number of network parameters is twice (8.86M) as many as that of the Vanilla network (4.43M ). In Table 4, we compare the average training time for first 5-epochs over various baseline methods under symmetric noise on the CIFAR-10 dataset. While non-parametric methods such as GCE and WDN require less than 12% additional time, other methods that require additional networks spent more time than non-parametric methods. The averaging time can vary according to different experimental environments. In table 4, we measure the time using publicly available code provided by authors.
sections.heading: 6 CONCLUSION
sections.text: We proposed a novel method called WDN for accurate classification of noisy labels. The proposed method normalizes uncertain measures to data-dependent Gaussian measures by imposing geometric constraints in the 2-Wasserstein space. We simulated discrete SDE using the Euler-Maruyama scheme, which makes our method fast, computationally efficient, and non-parametric. In theoretical analysis, we derived the explicit upper-bound of the proposed Wasserstein normalization and experimentally demonstrated a strong relationship between this upper-bound and the over-parameterization. We conducted experiments both on the CIFAR-10/100 and Clothing1M datasets. The results demonstrated that the proposed WDN significantly outperforms other state-of-the-art methods.
sections.heading: A OPEN-SOURCE DATASET
sections.text: Transition matrix for CIFAR-10/100. For the experiment summarized in Table 1, we implemented open-source code to generate the noise transition matrix discussed by Han et al. (2018), as well as the 9-layered CNN architecture (https://github.com/bhanML/Co-teaching).
  Open-set noise. For the experiment summarized in Table 2, we used the same dataset for open-set noisy labels presented by Lee et al. (2019) (https://github.com/pokaxpoka/ RoGNoisyLabel).
  Clothing1M. For the experiment summarized in Table 3, we used the open-source dataset presented by Xiao et al. (2015) (https://github.com/Cysu/noisy_label).
sections.heading: B COMPARISONS TO RELATED WORKS
sections.text: Methodology Parametric Class-dependency Distillation Sample-weight Sample-selection
  DivideMix 3 7 7 7 3 Co-teaching 3 7 3 7 3
  JoCoR 3 7 3 7 3 MLNT 3 3 3 7 7 Ren et al. (2018) 7 7 7 3 7 NPCL 7 7 7 3 7 GCE 7 7 7 3 7
  WDN 7 7 7 7 7
  Table B indicates that no previous methodologies can conceptually include our method.
  Because the solution to the Fokker-plank equation can be explicitly calculated without any additional parameters, our method is fully non-parametric (in terms of additional parameters beyond those required by the original neural network). By contrast, co-teaching is parametric because it requires a clone network with additional parameters that are copies of those in the original network. Similarly, MLNT requires an additional teacher network for training, which also contains a number of parameters.
  Many method based on small loss criteria select certain samples, whereas our method uses the combination of ρN certain and (1 − ρ)N normalized uncertain samples. Therefore, our method can fully leverage the batches of training datasets, where (1− ρ)N + ρN = N . Additionally, our method does not assume any class-dependent prior knowledge. Rather than considering class-wise prior knowledge, our method uses holistic information from both certain and uncertain samples (i.e., Y and XT ) in the logit space. Other meta-class-based model, such as MLNT, assume class-wise meta prior knowledge from a teacher network.
  In Arazo et al. (2019), they assumed the beta-mixture model as a label distribution on label space. But due to the non-deterministic type of noisy label distribution, it sometimes fails to train with extremely non-uniform type of noise. For example, Arazo et al. (2019) reported failure case with Clothing1M dataset. It seems that fundamental assumption on noise model of mixup will be improved in future work. Similar to this method, our work have trouble when dealing with synthetic asymmetric noise with high ratio where relatively large performance drop is observed in Table 1 (despite our method produces second best performance in the table).
  Most recent work Li et al. (2019a), they also adopt Co-train by implementing additional dual network, but much sophisticated methodology called Co-divide/guessing based on SSL. We predict that the Wasserstein distance between labeled and unlabeled probability measures is well-controlled in their method. We think that applying the OT/Markov theory (as in our paper) to their method will broaden the understanding of LNL problem.
  In contrast to sample weight methods such as GCE and NPCL, which require prior knowledge regarding the cardinality of the training samples to be weighted, our method is free from such assumptions because our Wasserstein normalization is applied in a batch-wise manner.
sections.heading: C TECHNICAL DIFFICULTY FOR APPLYING GENERAL OPTIMAL TRANSPORT/MARKOV THEORY TO LABEL SPACE.
sections.text: LetX,Y be uncertain and certain samples in pre-softmax feature space. And assume that we consider the distributional constraint on label-space (the space of σ(X), σ(Y ), where σ denotes the soft-max function). This space is not proper to define the objective function such as (5). Because, all the samples in this label space is of the form σ(X) = [a1, a2, · · · , an] such that ∑d i=1 ai = 1, thus label-space is d-dimensional affine-simplex Ud which is subset of Euclidean space Ud ⊂ Rd. In this case, the definition of Wasserstein space in equation (4) is unacceptable while dE is not true metric on Ud. The Wasserstein space P2(Ud) is merely investigated in the mathematical literature which makes unable to use all the technical details and assumptions, theories developed in the P2(Rd) which are theoretical ground of our work. But, if we look this problem slightly different point of view, for example, consider pre-softmax Rd,P2(Rd) as our base space. In this case, all the technical issues/problems when we try to use OT tools in P2(Ud) can be overcome/ignored. while softmax is non-parametric one-to-one function connecting pre-softmax feature space Rd to Ud, there exists a unique labels in Ud as a mapped point of the manipulated uncertain samples. Even though our objects are defined on pre-softmax space, the theoretical analysis in Proposition 3 contains softmax function to evaluate the concentration inequality of proposed transformation F affecting in label-space Ud.
sections.heading: D MATHEMATICAL BACKGROUND
sections.text: In this section, we introduce important definitions, notations, and propositions used in our proofs and the main paper.
  D.1 NOTATION
  We denote f#µ as a push-forward of µ through f . C∞0 (Rd) denotes the set of∞-class functions with compact support in Rd. For the Lp-norm of the function f , we denote ‖f‖p,ν = ( ∫ |f |pdν) 1 p . The Hessian matrix of the function f is denoted as Hess[f ] = [∂i∂jf ]di,j . Sym + d denotes the space for semi-definite positive symmetric matrices of size d× d. ‖f‖Lip denotes the Lipschitz norm of the function f . For any matrix A ∈Md, we let ‖A‖op denote the operator norm of A.
  D.2 DIFFUSION-INVARIANCE AND HYPER-CONTRACTIVITY
  Definition 2. The Markov semigroup (Pt)t≥0 in Rd acting on a function f ∈ C∞0 is defined as follows:
  Ptf(x) = ∫ f(x′)pt(x, dx ′), (15)
  where pt(x, dx′) is a transition kernel that is the probability measure for all t ≥ 0. Definition 3. (Diffusion Operator) Given a Markov semi-group Pt at time t, the diffusion operator (i.e., infinitesimal generator) L of Pt is defined as
  Lg(y) = lim t→0
  1 t (Ptg(y)− g(y)) = ∑ i,j ∂2 ∂yi∂yj Bij(y)g(y)− ∑ i Ai(y) ∂ ∂yi g(y), (16)
  where B and A are matrix and vector-valued measurable functions, respectively. Bij denotes the (i, j)-th function of B and Ai denotes the i-th component function of A. Definition 4. (Diffusion-invariant Measure) Given the diffusion operator L, the probability measure µ is considered to be invariant measure to L when EX∼µ[Lf(X)] = 0 for any f ∈ C∞0 . Lemma 1. (Infinitesimal generator for the multivariate Gaussian measure, Bolley & Gentil (2010).) The Gaussian measure Nν := N (mν ,Σν) with a mean mν and covariance Σν is an invariant measure according to the following diffusion-operator L:
  Lf(x) = ΣνHess[f ](x)− (x−mν)T ∇f(x), ∀f ∈ C∞0 (Rd), (17) where Bij(x) := [Σν ]ij is a constant function, and Ai(x) := xi −miν .
  This generator serves as our main tool for the geometric analysis of the upper bound ε. In Section 4.1 in the main paper, we introduced an approximate upper-bound K̂2(µ) without any general description of the inequality involved. We now introduce the underlying mathematics for equation 12. Because our detour measure is Gaussian, there is a unique semi-group Pth called the multidimensional Ornstein-Ulenbeck semi-group that is invariant to Nν . Specifically, Pt is defined as follows:
  Psh(X) = EZ∼NI [ h ( e−sX + √ 1− e−2s(Σ 1 2 ν Z + mν) )] , ∀h ∈ C∞0 . (18)
  The invariance property of Pt relative to our detour measure is naturally induced by the following Proposition: Proposition 4. We define C : Rd → Rd and C(X) = AX + b such that A ∈ Sym+d ,b ∈ Rd, and select an arbitrary smooth h ∈ C∞0 (Rd). We then define the diffusion Markov semi-group Psh as follows:
  Psh(X) = EZ∼N [ h ( e−sX + √ 1− e−2sC(Z) )] . (19)
  Then, N (A2,b) is invariant with respect to Ps, meaning the following equality holds for every h and s ≥ 0: ∫
  Rd [Psh(X)− h(X)]dN (A2,b)(X) = 0. (20)
  Proof. For simplicity, we denote N (A2,b) := NC .∫ Psh(X)dNC(X) = ∫ ∫ h(e−sX + √ 1− e−2sC(Z))dNC(X)dN (Z)
  = ∫ ∫ h ◦ C(e−sZ ′ + √ 1− e−2sZ)dN (Z ′)dN (Z).
  (21)
  The second equality holds because C is linear in Rd. Let e−s = cos θ and e−2s = sin θ for any 0 ≤ θ ≤ 2π. Then, we define φ as φ(Z ′, Z) = e−sZ ′ + √ 1− e−2sZ = cos(θ)Z ′ + sin(θ)Z, and π(Z ′, Z) = Z. Based on the rotation property of the standard Gaussian measure, one can induce the following equality.
  (N ⊗N ) ◦ (C ◦ φ)−1 = ((N ⊗N ) ◦ φ−1) ◦ C−1 = N ◦ C−1. (22) However, we know that dN [C−1(X)] = dNC(X) = ( (2π)d|A2| )− 12 e−0.5(X−b)TA−2(X−b). By combining equation 21 and equation 22, one can derive the following result:∫
  h ◦ C(e−sZ ′ + √ 1− e−2sZ)d[N ⊗N ] = ∫ h(X)d [ (N ⊗N ) ◦ φ−1 ◦ C−1 ] (X)
  = ∫ h(X)d[N ◦ C−1](X) = ∫ h(X)dN [C−1(X)]
  = ∫ h(X)dNC(X).
  (23)
  Proposition 4 demonstrates the invariance property of the defined semi-group. If we setA = Σ 1 2 ν ,b = mν , then we can recover equation 18.
  We are now ready to define the approximation of K2(µ) in terms of semi-group invariance. Specifically, for any real-valued smooth h, we define the following inequality:
  K̂2(µ) = EX∼µ[Lh(X)] = lim s→0
  EX∼µ [ 1
  s (Psh(X)− h(X)) ] = lim s→0 1 s EX,Z∼NI [ h ( e−sX + √ 1− e−2s(Σ 1 2 ν Z + mν) ) − h(X) ] ≤ K2(µ).
  (24)
  This inequality holds if h is selected to induce a supremum over the set C∞0 , where suph K̂2(µ, h) = suph EX∼µ[Lh(X)] = K2(µ). Although a more sophisticated design for the test function h will induce a tighter upper bound for K̂2, we determined that the L2-norm is generally sufficient.
  Definition 5. (Diffuseness of the probability measure) We define the integral operator K2 : W2(Rd)→ R+ as follows:
  K2(µ) = √ sup f∈C∞0 ∫ Rd |Lf(x)| dµ(x). (25)
  According to Definition 4, we know that ∫ Lf(X)dNν(X) = 0 for any f . Based on this observation, it is intuitive that K2 estimates how the probability measure ν is distorted in terms of diffusion invariance. While this measure takes a supremum over the function space C∞0 , it searches for a function that enables the estimation of maximal distortion. Because the value of K2 is entirely dependent on the structure of µ, K2 can be considered as a constant for the sake of simplicity if the uncertain measure µ is fixed over one iteration of training. Definition 6. (Diffusion carré du champ) Let f, g ∈ C∞0 (Rd). Then, we define a bilinear form Γc in C∞0 (Rd)× C∞0 (Rd) as
  Γe(f, g) = 1
  2 [LΓe−1(fg)− Γe−1(fLg)− Γe−1(gLf)], e ≥ 1. (26)
  We also denote Γ(f) ≡ Γ(f, f). The bilinear form Γ can be considered as a generalization of the integration by the parts formula, where ∫ fLg + Γ(f)dµ = 0 for the invariant measure µ of L.
  Definition 7. (Curvature-Dimension condition, Ambrosio et al. (2015)) We can say that the infinitesimal generator L induces the CD(ρ,∞) curvature-dimension condition if it satisfies Γ1(f) ≤ ρΓ2(f) for all f ∈ C∞0 .
  Because our diffusion operator generates a semi-group with respect to the Gibbs measure, the curvature-dimension condition can be calculated explicitly. Through simple calculations, the firstorder (c = 1) diffusion carré du champ can be induced as follows:
  Γ1(f) = ( [∇f ]TΣν∇f )2 . (27)
  Similarly, the second-order (c = 2) diffusion carré du champ is calculated as follows:
  Γ2(f) = 1
  2
  [ L ( Γ1(f 2) ) − 2Γ1 (f,L(f)) ] = Tr ([ Σν∇2f ]2) + ( [∇f ]TΣν∇f )2 = Tr ([ Σν∇2f ]2) + Γ1(f),
  (28)
  for an arbitrary f ∈ C∞0 (Rd). While Tr ([ Σ∇2f ]2)
  is non-negative, we can infer that Γ1 ≤ Γ2. In this case, the diffusion operator L defined in Lemma 1 induces the CD(ρ = 1,∞) curvaturedimension condition. For the other diffusion operators, please refer to Bolley & Gentil (2010). Proposition 5. (Decay of Fisher information along a Markov semigroup, Bakry et al. (2013).) If we assume the curvature-dimension condition CD(ρ,∞), then I(µt|Nν) ≤ e−2ρtI(µ|Nν).
  The exponential decay of the Fisher information in Proposition 5 is a core property of the exponential decay of the Wasserstein distance, which will be used in the proof of Proposition 2.
  D.3 FOKKER-PLANK EQUATION, SDE
  Definition 8. (Over-damped Langevin Dynamics) We have dXt = −∇φ(Xt;mν)dt+ √ 2τ−1ΣνdWt, (29)
  where φ (Xt;mν) = τ2d 2 (Xt,mν), Wt denotes Brownian motion, and d denotes Euclidean distance. The particle Xt is distributed in Xt ∼ pt. The probability density limt→∞ p(x, t) with respect to X∞ converges to the Gaussian density X∞ = √ Σν(Z + mν) ∼ p∞(x) = q(x) ∝ e−d(x,mν) TΣ−1ν d(x,mν).
  In classical SDE literature, it is stated that E [ sup0≤t≤T ∣∣∣X̂t −Xt∣∣∣] ≤ G(N%)− 12 , where G(T ) is some constant that depends only on T and X̂ denotes the true solution of the SDE in equation 29. While the number of uncertain samples is greater than N% > 40, our method exhibits acceptable convergence.
  D.4 GAUSSIAN WASSERSTEIN SUBSPACES
  It is known that the space of non-degenerate Gaussian measures (i.e., covariance matrices are positivedefinite) forms a subspace in the 2-Wasserstein space denoted asW2,g ∼= Sym+d × Rd. Because the 2-Wasserstein space can be considered as a Riemannian manifold equipped with Riemannian metrics Villani (2008), W2,g can be endowed with a Riemannian structure that also induces the Wasserstein metric (McCann (1997)). In the Riemannian sub-manifold of Gaussian measures, the geodesic between two points γ(0) = NA and γ(1) = NB is defined as follows Malagò et al. (2018):
  γ(α) = Nt = N (m(α),Σ(α)), (30)
  where m(α) = (1 − α)mA + αmB and Σ(α) = [(1− α)I + αT ] ΣA [(1− α)I + αT ], where T ΣAT = ΣB . In Section 3.2, we set (mA,ΣA) → (mν ,Σν) and (mB ,ΣB) → (mξk ,Σξk). Regardless of how ν is updated, the statistical information regarding the current certain measure ξk is considered in the detour Gaussian measure, which yields a much smoother geometric constraint on µ.
sections.heading: E PROOFS
sections.text: Proposition 6. Let Γ(µ, ν) be a set of couplings between µ and ν, and assume that the noisy label r̂ is independent of X . For functional J [µ] = Eµ∼X l(X; r̂), we define D(µ, ν) as:
  D(µ, ν) = inf γ∈Γ(µ,ν)
  |J [µ]− J [ν]| , (31)
  where D : P2 ×P2 → R. Then, D is the metric defined on P2, which is weaker than the Wasserstein metric, where D(µ, ν) ≤ αW2(µ, ν) for α = c−10 r̂ + c −1 1 (1− r̂) and some constants c0, c1 > 0.
  Proof.
  |J [ν]− J [µ]| = |Eµ[l(X; r̂)]− Eν [l(Z; r̂)]| = |Eµ⊗ν [r̂ (log σ(X)− log σ(Z))− (1− r̂) (log(1− σ(X))− log(1− σ(Z)))]| ≤ E |r̂Eµ⊗ν [log σ(X)− log σ(Z)]|+ E |(1− r̂)Eµ⊗ν [log(1− σ(X))− log(1− σ(Z))]| ≤ Er̂Eµ⊗ν |log σ(X)− log σ(Z)|+ E(1− r̂)Eµ⊗ν |log(1− σ(X))− log(1− σ(Z))| ≤ c−10 E(r̂)Eµ⊗ν |X − Z|+ c −1 1 E(1− r̂)Eµ⊗ν |Z −X| = E[c−10 r̂ + c −1 1 (1− r̂)]Eµ⊗ν |X − Z| (32)
  By taking the infimum of the aforementioned inequality with set of couplings γ(µ, ν), we obtain the following inequality:
  D(ν, µ) = inf γ(µ,ν)
  |J [ν]− J [µ]| ≤ E[c−10 Y + c −1 1 (1− Y )] inf γ(µ,ν) Eγ |X − Z|
  = E[c−10 Y + c −1 1 (1− Y )]W1(µ, ν) ≤ E[c−10 Y + c −1 1 (1− Y )]W2(µ, ν),
  (33)
  which completes the proof.
  Proposition 6 follows from the Lipschitzness of the functional J , where D searches for the best coupling to derive the minimal loss difference between two probability measures. This proposition indicates that inf |J [ν]− J [Fµ]| is bounded by the Wasserstein distance, which justifies our geometric constraint presented in equation 4. It should be noted that the prior assumption regarding noisy labels is essential for Lipschitzness. Proposition 7. Let F : R+ × P2 be a functional on probability measures such that F [t, µ] = µt, where dµt = ptdNν , dNν = dqtdx, and let µt be a solution of the continuity equation in the 2-Wasserstein space defined as follows:
  ∂tµt = ∇ · (µt∇Φt) , (34)
  which is represented as ∂tp(t, x) = ∇ · (p(t, x)∇ log q(t, x)) in a distributional sense. Then, the functional Ft[·] = F [t, ·] is defined unique and normalizes µ onto BW2 (Nν , e−tK2 (µ)), where K2(µ) ≤ ∞ is an integral operator in Definition 5 with respect to µ.
  Proof. We assume that the probability measure µt is absolutely continuous with respect to the detour Gaussian measure N (mν ,Σν) = Nν , µt Nν . In this case, according to the Radon-Nikodym theorem, there is a corresponding unique probability density q(t, x) = qt(x) ∈ C∞0 such that dµt = qtdNν . Lemma 2. (WI-inequality, Otto & Villani (2000)) If the stationary state of µt with respect to Pt satisfies limt→∞ Eµ[Ptf ] = 0 for any f ∈ C∞0 , then the following inequality holds:
  d dt+ W2(µ, µt) ≤
  √ I(µt|Nν). (35)
  By integrating both sides of the inequality in Lemma 2 with respect to t ∈ (0,∞), the following inequality can be obtained:
  W2(µt,Nν) = ∫ ∞
  0
  d dt+ W2(µt,Nν)dt ≤ ∫ ∞ 0 √ I(µt|Nν)dt. (36)
  In the aforementioned inequality, we replace the Fisher information with the diffusion generator L as follows:
  W2(µ,Nν) ≤ ∫ ∞
  0
  √ I(µt|Nν)dt
  = ∫ ∞ 0 √∫ [Ptq]−1Γ(Ptq)dNνdt = ∫ ∞ 0 √∫ L(− logPtq)dµtdt. (37)
  The second equality above is derived by leveraging the properties of the bilinear operator Γ (Bakry et al. (2013); Villani (2008)) with respect to the diffusion operator L, which is defined as follows:∫
  [Ptq] −1Γ(Ptq)dNν = − ∫ L(logPtq)qtdNν = ∫ L(− logPtq)dµt ≥ 0. (38)
  For simplicity, we denote |g| = g+ for any g ∈ C∞0 . According to Proposition 5, we can relate Ftµ = µt to its initial term µ = µt=0 as follows:∫ ∞
  0
  √∫ L(− logPtq)(X)d[Ftµ](X)dt ≤ ∫ ∞ 0 √ e−2ρt ∫ L (− logPt=0q) (X)dµ(X)dt
  ≤ ∫ ∞
  0
  √ e−2ρt sup
  g∈C∞0
  ∫ L+g(Z)qdNν(Z)dt
  = ∫ ∞ 0 √ e−2ρtdt √ sup g∈C∞0 ∫ L+g(X)dµ(X)
  = ρ−1K2(µ).
  (39)
  The second inequality is naturally induced, because the proposed objective function is defined to select the maximum elements over the set of functions g ∈ C∞0 and Lg ≤ L+g. If the integral interval is set to (0, s), then we can induceW2(µ,Ftµ) ≤ 1ρ (1− e
  −s)K2(µ). Our diffusion-operator induces ρ = 1, which completes the proof.
  Proposition 8. There is a scalar 0 < β < ∞ dependent on ν such that the following inequality holds: W2(ν,Ftµ) ≤ [√ dβλmax(Σν) + ‖EνY ‖2 ] ∨ [ e−tK2(µ) +K2(ν) ] . (40)
  As a motivation for setting a detour measure to Nν , we mentioned the natural property of the non-collapsing Wasserstein distance ofW2(ν,Nν) 6= 0. However, it is unclear from a geometric perspective exactly how the upper bound (i.e.,W2(ν,Nν) ≤ ?) can be induced based on the intrinsic statistics term (i.e., d1 in Fig.1). Specifically, in the situation where the covariance matrices of ν and Nν are identical, it is difficult to determine a theoretical upper bound without additional tools. The first part of this proof focuses on resolving this important issue. The second part of the proof is naturally induced by Proposition 1. Please note that in the following proposition, parameter for Wasserstein moving average is set to α = 0 for clarity.
  Proof. Before proceeding with the first part of the proof, we define a constant β as follows:
  β = sup 1≤j≤d ∫ 1 0 1 s EYsv2s,j(Ys)ds. (41)
  If we assume a mild condition such that mins,j inf1≤j≤dO(vs,j) ≥ O( √ s), then the integral term in β is finite and well-defined. This value will directly yield the upper bound of the Kullback–Leibler (KL) divergence of ν. First, we introduce the following inequality.
  Lemma 3. (de Bruijn’s identity, Johnson & Suhov (2001); Nourdin et al. (2014)) We let Y ∼ ν, Z ∼ N (0, I) denote a standard Gaussian random variable, and let define Ys = √ sY + √ 1− sΣ 1 2 ν Z with the score function defined as vs(x) = ∇ log ps(x) with respect to the random variable Ys. Then, the following equality holds:
  KL(ν|N (0,Σν)) = ∫ 1
  0
  Tr
  ( 1
  2s ΣνEps∼Ys [vs(Ys)vs(Ys)T ]
  ) ds. (42)
  From equation 42, we can derive the relations between KL-divergence and the constant β defined earlier.∫ 1
  0
  1 2s Tr ( ΣνEx[vs(Ys)vs(Ys)T ]) ) ds ≤ ∫ 1 0 1 2s Tr ( ΣνEx[vs,ivs,j ]di,j) ) ds
  ≤ ∫ 1
  0
  1 2 λmax(Σν) d∑ j=1 E
  [ v2s,j(Ys)
  s
  ] ds ≤ 1
  2 λmax ∫ 1 0 d∑ j=1 βds = 1 2 λmax(Σν)dβ.
  (43)
  The second inequality holds based on the following element property of symmetric positive-definite matrices:
  Tr(AB) ≤ ‖A‖opTr(B) = λmax(A)Tr(B), ∀A,B ∈ Sym + d . (44)
  It should be noted that because the distribution of ν is compactly supported (i.e., supp(q) is compact), the maximum eigenvalue of the covariance Σν is finite. The other relations are induced by the aforementioned definition. Next, we relate the KL-divergence and 2-Wasserstein distance naturally.
  Definition 9. (Talagrand inequality for Gaussian measures, Otto & Villani (2000)) For any nondegenerate Gaussian measure N with a mean 0, the following inequality is satisfied:
  W2(ν,N ) ≤ √ 2KL(ν|N ), ∀ν ∈ P2(Rd). (45)
  By combining Definition 9 and equation 43, we can derive the following expression: W2(ν,N (0,Σν)) ≤ √ 2KL(ν|N (0,Σν)) ≤ √ dβλmax(Σν) <∞. (46)
  According to the triangle inequality for the 2-Wasserstein distance, we obtain:
  W2(ν,N (mν ,Σν)) ≤ W2(ν,N (0,Σν)) +W2(N (mν ,Σν),N (0,Σν)) (47)
  In Appendix C.3, we investigated that the geodesic distance between two Gaussian measures having the same covariance is equivalent to the Euclidean distance between two means. Therefore, we can obtain the following equality:
  W2(N (mν ,Σν),N (0,Σν)) =W2(ιmν# [N (0,Σν)],N (0,Σν)) = ‖mν − 0‖2 = ‖EνY ‖2 ,
  (48)
  where ιa(X) = X + a for any vector a ∈ supp(q). Now, by adding the two inequalities defined earlier, we can obtain
  W2(ν,N (mν ,Σν)) ≤ ‖EνY ‖2 + √ dβλmax(Σν), (49)
  where it is easily shown that the upper-bound is only dependent on the statistical structure of ν. Specifically, the term ‖EνY ‖2 represents the center of mass for a density of ν and √ dβλmax(Σν) is related to the covariance structure of ν.
  By applying Proposition 8 to both Ftµ and ν, we can easily recover equation 5 as follows: W2(ν,Ftµ) ≤ ε =W2(ν,N (mν ,Σν)) +W2(N (mν ,Σν),Ftµ)
  ≤ ([ ‖EνY ‖2 + √ dβλmax(Σν) ] ∧K2(ν) ) + e−tK2(µ)
  ≤ [√ dβλmax(Σν) + ‖EνY ‖2 ] ∨ [ e−tK2(µ) +K2(ν) ] .
  (50)
  The second inequality is easily obtained as (a ∧ b) + c ≤ a ∨ (b + c) for any a, b, c ≥ 0, which completes the proof.
  Proposition 9. (Concentration inequality for uncertain measures). Assume that there are some constants s? ∈ [ 1η ,∞), η ≥ 0 such that the following inequality is satisfied:
  EFs?µ[f 2]− [EFs?µ[f ]] 2 ≤ (1 + η)EFs?µ[A∇f T∇f ], (51)
  for A ∈ Sym+d , D(A,Σν) ≤ aη for some a > 0, and for any metric D defined on Sym + d . In this case, there is a δ such that the following probability inequality for an uncertain measure is induced:
  Fs?µ ( |σ − Eν [σ]| ≥ δ ) ≤ 6e− √ 2δ 3 2 K2 , (52)
  where κ denotes the Lipschitz constant of σ.
  Proof. Before proceeding with the main proof, we first prove the existence of s?. The limit of the interval with respect to η converges to a singleton {∞} as I = limη→0[ 1η ,∞). In this case, equation 51 is the same as the Poincaré inequality for a Gaussian measure Nν , which can be written as
  lim η→0
  EFs?µ[f 2]− [EFs?µ[f ]] 2 ≤ lim η→0 (1 + η)EFs?µ[A∇f T∇f ]
  = EFs?µ[Σν∇f T∇f ].
  (53)
  While the Poincaré inequality in equation 53 is uniquely defined, we can find at least one value s? satisfying equation 51. Let X(t, w) = Xt(w) denote the stochastic process with respect to qt(x) defined in the proof of Proposition 2. Additionally, let c = Eν [σ]− EFs?µ[σ]. Then, we can obtain the following inequality:
  c = Eν [σ]− EFs?µ[σ] = κ ( Eν [σ κ ] − EFs?µ [σ κ ]) ≤ κ sup
  g∈Lip1 (Eνg − EFs?µg)
  ≤ κW1(Fs?µ, ν) ≤ κW2(Fs?µ, ν) ≤ κK2(µ)
  1 + η .
  (54)
  The first inequality is induced by the assumption regarding the κ-Lipschitzness of the function σ and the second inequality is induced by the Kantorovich-Rubinstein theorem. The third inequality is natural becauseWa(·, ·) ≤ Wb(·, ·) for any 1 ≤ a ≤ b <∞. because equation 51 is equivalent to the Poincaré inequality for the measure Fs?µ, it satisfies the Bakry-emery curvature-dimension condition CD(1 + η,∞). Thus, as shown in the proof of Proposition 2 (i.e., equation 39), the last inequality is induced. Additionally, based on the concentration inequality of Fs?µ [Proposition 4.4.2 Bakry et al. (2013)], we can derive the following probability inequality:
  Fs?µ [σ(Xs?(w)) ≥ EFs?µ[σ] + δ] ≤ 3e − δ√ 1+ηκ , (55)
  where the Poincaré constant for Fs?µ is naturally 1 + η and ‖σ‖Lip = κ. Next, we will derive the desired form from equation 55. First, we introduce the following inequality.
  σ(Xs?) ≥ EFs?µ[σ] + δ ≥ Eν [σ] + δ − κ
  1 + η K2 (56)
  The last inequality is directly induced by equation 54 because −c ≥ − κ1+ηK2. While η, κ, and K2 are constants with respect to w, the following set inclusion can be obtained naturally:
  S1 = {w : σ(Xs?(w)) ≥ EFs?µ[σ] + δ} ⊇ {w : σ(Xs?(w)) ≥ Eν [σ] + δ − κ
  1 + η K2} = S2.
  (57)
  For the modified version of the original probability inequality, we take probability measure Fs?µ[·] for the sets S1,S2, which is defined as
  3e − δ√
  1+ηκ ≥ Fs?µ ({w : σ(Xs?(w)) ≥ EFs?µ[σ] + δ}) ≥ Fs?µ ( {w : σ(Xs?(w)) ≥ Eν [σ] + δ − κ
  1 + η K2}
  ) .
  (58)
  The concentration inequality around Eν [σ] is obtained by combining the inequalities induced by σ and −σ as follows:
  1 2 Fs?µ  ⋃ h∈{σ,−σ} {w : h(Xs?(w))− Eν [h] ≥ ± ( δ − κ 1 + η K2} ) = Fs?µ ( {w : |σ(Xs?(w))− Eν [σ]| ≥ δ − κ
  1 + η K2}
  ) ≤ 6e− δ√ 1+ηκ .
  (59)
  The inequality in equation 59 is the general form containing the relation between the upper bound of the probability and (η, κ,K2). While this form is quite complicated and highly technical, we choose not to present all the detailed expressions of equation 59 in the main paper. Rather than that, we re-write it in a much simplified form for clarity. Specifically, by setting κK2/(1 + η) = 0.5δ and rescaling δ to 2δ, the aforementioned inequality in equation 59 can be converted into the following simpler form:
  Fs?µ ({w : |σ(Xs?(w))− Eν [l]| ≥ δ) ≤ 6e− √ 2δ 3 2 κK2 . (60)
  Finally, if we set σ = Softmax, then the Lipschitz constant is induced as κ = 1. This proof is completed by setting s? := T .