summaries: In this paper the authors inspect the popular Physics Informed Neural Network (PINN) set of models and characterize their behavior (specifically their failure modes) in the context of two popular PDE systems, namely diffusion and convection. In each case, the authors demonstrate that PINNs learn effective representations in simple regimes while failing to learn representations of the PDE systems in more complex regimes. In addition, authors also demonstrate that it is not the inability of the neural network to fit the functions (due to some potential inability to represent non-trivial functions), but rather the way the PINNs incorporate the PDEs into the learning pipeline that is the reason for the failure. Finally, the authors also propose casting the PINN problem as a sequential prediction task (i.e., predicting all collocation points per time period) as opposed to tasking the neural network with simultaneously learning the full spatial and temporal domain.

summaries: This work studies two specific (yet essential) PDE learning cases in the PINN framework. Notably, the work proposes two main conclusions to explain the difficult training in PINN networks:  - the loss landscape varies greatly with the PDE coefficient values (and works well with only low diffusion/conduction coefficient),  - the loss landscape (at the end of training) varies greatly with the physics informed penalty coefficient.   The main explanation to such difficulties lies in the condition number associated to PINN penalty. The work proposes two solutions: a warm-up, and a sequence to sequence approach.

summaries: The paper studies the behavior of Physics Informed NNs (PINNs) on two common PDEs of physical relevance. It observes that PINNs fail to learn a good solution under standard training regime when the convection coefficient or viscosity coefficient is high, even when the closed-form solution does not depend on the constants.  The authors then seek to explain this behavior from an optimization standpoint. They plot the loss landscape along the two dominant hessian eigenvectors. They hypothesize that the optimization problem is ill conditioned due to the regularization term in the optimization objective that enforces the pde constraint.  They also run experiments to argue that sequence to sequence learning might work better for these ill conditioned problems.

summaries: The authors provide a detailed empirical analysis of the numerical difficulties encountered when training so-called "physics informed neural networks" (PINNs), a recent class of models which incorporate physical (often differential) equations into the loss function of a neural network. The authors perform a systematic analysis on two relatively simple but common use-cases in scientific literature, convection and diffusion equations, and demonstrate the PINNs fail to adequately learn the underlying dynamics of the system even in simple cases. They show that these failures are due to the effect of the convection/diffusion coefficients and forcing terms on the conditioning of the dynamical system, i.e. larger coefficients or forcings tend to make the problem increasingly ill-conditioned, thereby impacting characteristics of the loss function. The authors then propose two possible workarounds for these issues; improved initialization via a warm start or preconditioning and re-framing the problem as a sequence-to-sequence learning task.

abstractText: Recent work in scientific machine learning has developed so-called physicsinformed neural network (PINN) models. The typical approach is to incorporate physical domain knowledge as soft constraints on an empirical loss function and use existing machine learning methodologies to train the model. We demonstrate that, while existing PINN methodologies can learn good models for relatively trivial problems, they can easily fail to learn relevant physical phenomena for even slightly more complex problems. In particular, we analyze several distinct situations of widespread physical interest, including learning differential equations with convection, reaction, and diffusion operators. We provide evidence that the soft regularization in PINNs, which involves PDE-based differential operators, can introduce a number of subtle problems, including making the problem more ill-conditioned. Importantly, we show that these possible failure modes are not due to the lack of expressivity in the NN architecture, but that the PINN’s setup makes the loss landscape very hard to optimize. We then describe two promising solutions to address these failure modes. The first approach is to use curriculum regularization, where the PINN’s loss term starts from a simple PDE regularization, and becomes progressively more complex as the NN gets trained. The second approach is to pose the problem as a sequence-to-sequence learning task, rather than learning to predict the entire space-time at once. Extensive testing shows that we can achieve up to 1-2 orders of magnitude lower error with these methods as compared to regular PINN training.

authors:
Y. Bengio
J. Louradour
R. Collobert
J. Weston

title: Curriculum learning

venue: Proceedings of the 26th annual international conference on machine learning, pages 41–48

authors:
S.L. Brunton
B.R. Noack
P. Koumoutsakos

title: Machine learning for fluid mechanics

venue: Annual Review of Fluid Mechanics, 52:477–508

authors:
Y. Chen
L. Lu
G.E. Karniadakis
L. Dal Negro

title: Physics-informed neural networks for inverse problems in nano-optics and metamaterials

authors:
M. Cranmer
S. Greydanus
S. Hoyer
P. Battaglia
D. Spergel
S. Ho

title: Lagrangian neural networks

venue: arXiv preprint arXiv:2003.04630

authors:
M. Dissanayake
N. Phan-Thien

title: Neural-network-based approximations for solving partial differential equations

venue: communications in Numerical Methods in Engineering, 10(3):195–201

authors:
P.L. Donti
D. Rolnick
J.Z. Kolter

title: Dc3: A learning method for optimization with hard constraints

venue: arXiv preprint arXiv:2104.12225

authors:
V. Dwivedi
N. Parashar
B. Srinivasan

title: Distributed learning machines for solving forward and inverse problems in partial differential equations

venue: Neurocomputing, 420:299–316

authors:
K. Eriksson
D. Estep
P. Hansbo
C. Johnson

title: Computational differential equations

venue: Cambridge University Press

title: A practical guide to pseudospectral methods

venue: Cambridge university press

authors:
O. Fuks
H.A. Tchelepi

title: Limitations of physics informed machine learning for nonlinear twophase transport in porous media

venue: Journal of Machine Learning for Modeling and Computing, 1 (1)

authors:
N. Geneva
N. Zabaras

title: Modeling the dynamics of pde systems with physics-constrained deep auto-regressive networks

venue: Journal of Computational Physics, 403:109056

authors:
S. Greydanus
M. Dzamba
J. Yosinski

title: Hamiltonian neural networks

venue: Advances in Neural Information Processing Systems, 32:15379–15389

authors:
J. Han
A. Jentzen
E. Weinan

title: Solving high-dimensional partial differential equations using deep learning

venue: Proceedings of the National Academy of Sciences, 115(34):8505–8510

authors:
O. Hennigh
S. Narasimhan
M.A. Nabian
A. Subramaniam
K. Tangsali
Z. Fang
M. Rietmann
W. Byeon
S. Choudhry

title: Nvidia simnetTM: An ai-accelerated multi-physics simulation framework

venue: International Conference on Computational Science, pages 447–461. Springer

authors:
W. Ji
W. Qiu
Z. Shi
S. Pan
S. Deng

title: Stiff-pinn: Physics-informed neural network for stiff chemical kinetics

venue: arXiv preprint arXiv:2011.04520

authors:
X. Jin
S. Cai
H. Li
G.E. Karniadakis

title: Nsfnets (navier-stokes flow nets): Physics-informed neural networks for the incompressible navier-stokes equations

venue: Journal of Computational Physics, 426:109951

authors:
G.E. Karniadakis
I.G. Kevrekidis
L. Lu
P. Perdikaris
S. Wang
L. Yang

title: Physicsinformed machine learning

venue: Nature Reviews Physics, 3(6):422–440

authors:
I.E. Lagaris
A. Likas
D.I. Fotiadis

title: Artificial neural networks for solving ordinary and partial differential equations

venue: IEEE transactions on neural networks, 9(5):987–1000

authors:
Z. Long
Y. Lu
X. Ma
B. Dong

title: Pde-net: Learning pdes from data

venue: International Conference on Machine Learning, pages 3208–3216. PMLR

authors:
L. Lu
X. Meng
Z. Mao
G.E. Karniadakis

title: Deepxde: A deep learning library for solving differential equations

authors:
L. Lu
R. Pestourie
W. Yao
Z. Wang
F. Verdugo
S.G. Johnson

title: Physics-informed neural networks with hard constraints for inverse design

venue: arXiv preprint arXiv:2102.04626

authors:
P. Márquez-Neila
M. Salzmann
P. Fua

title: Imposing hard constraints on deep networks: Promises and limitations

venue: arXiv preprint arXiv:1706.02025

title: Fundamentals of engineering numerical analysis

authors:
Y. Nandwani
A. Pathak
P. Singla

title: et al

venue: A primal dual formulation for deep learning with constraints. Advances in Neural Information Processing Systems

authors:
D.R. Parisi
M.C. Mariani
M.A. Laborde

title: Solving differential equations with unsupervised neural networks

venue: Chemical Engineering and Processing: Process Intensification, 42(8-9):715– 721

authors:
C. Rackauckas
Y. Ma
J. Martensen
C. Warner
K. Zubov
R. Supekar
D. Skinner
A. Ramadhan
A. Edelman

title: Universal differential equations for scientific machine learning

venue: arXiv preprint arXiv:2001.04385

authors:
M. Raissi
P. Perdikaris
G.E. Karniadakis

title: Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations

venue: Journal of Computational Physics, 378:686–707

authors:
M. Raissi
A. Yazdani
G.E. Karniadakis

title: Hidden fluid mechanics: Learning velocity and pressure fields from flow visualizations

authors:
F. Sahli Costabal
Y. Yang
P. Perdikaris
D.E. Hurtado
E. Kuhl

title: Physics-informed neural networks for cardiac activation mapping

venue: Frontiers in Physics, 8:42

authors:
J. Sirignano
K. Spiliopoulos

title: Dgm: A deep learning algorithm for solving partial differential equations

venue: Journal of computational physics, 375:1339–1364

authors:
B.P. van Milligen
V. Tribaldos
J. Jiménez

title: Neural network differential equation and plasma equilibrium solver

venue: Physical review letters,

authors:
L. von Rueden
S. Mayer
K. Beckh
B. Georgiev
S. Giesselbach
R. Heese
B. Kirsch
J. Pfrommer
A. Pick
R. Ramamurthy

title: Informed machine learning–a taxonomy and survey of integrating knowledge into learning systems

venue: arXiv preprint arXiv:1903.12394,

authors:
B. Wang
W. Zhang
W. Cai

title: Multi-scale deep neural network (mscalednn) methods for oscillatory stokes flows in complex domains

venue: arXiv preprint arXiv:2009.12729

authors:
S. Wang
Y. Teng
P. Perdikaris

title: Understanding and mitigating gradient pathologies in physics-informed neural networks

venue: arXiv preprint arXiv:2001.04536

authors:
S. Wang
H. Wang
P. Perdikaris

title: On the eigenvector bias of fourier feature networks: From regression to solving multi-scale pdes with physics-informed neural networks

venue: arXiv preprint arXiv:2012.10047

authors:
S. Wang
X. Yu
P. Perdikaris

title: When and why pinns fail to train: A neural tangent kernel perspective

venue: arXiv preprint arXiv:2007.14527

authors:
E. Weinan
J. Han
A. Jentzen

title: Deep learning-based numerical methods for highdimensional parabolic partial differential equations and backward stochastic differential equations

venue: Communications in Mathematics and Statistics, 5(4):349–380

authors:
J. Willard
X. Jia
S. Xu
M. Steinbach
V. Kumar

title: Integrating physics-based modeling with machine learning: A survey

venue: arXiv preprint arXiv:2003.04919

authors:
K. Xu
E. Darve

title: Physics constrained learning for data-driven inverse modeling from sparse observations

venue: arXiv preprint arXiv:2002.10521

authors:
Z. Yao
A. Gholami
Q. Lei
K. Keutzer
M.W. Mahoney

title: Hessian-based analysis of large batch training and robustness to adversaries

venue: Advances in Neural Information Processing Systems

authors:
Z. Yao
A. Gholami
K. Keutzer
M.W. Mahoney

title: Pyhessian: Neural networks through the lens of the hessian

venue: 2020 IEEE International Conference on Big Data (Big Data), pages 581–590. IEEE

authors:
Y. Zhu
N. Zabaras
P.-S. Koutsourelakis
P. Perdikaris

title: Physics-constrained deep learning for high-dimensional surrogate modeling and uncertainty quantification without labeled data

venue: Journal of Computational Physics, 394:56–81

authors:
O.C. Zienkiewicz
R.L. Taylor
P. Nithiarasu
J. Zhu

title: The finite element method

venue: volume 3. McGraw-hill London

heading: 1 Introduction

text:
Partial differential equations (PDEs) are commonly used to describe different phenomena in science and engineering. These PDEs are often derived by starting from governing first principles (e.g., conservation of mass or energy). It is typically not possible to find analytical solutions to these PDEs for many real-world settings. Thus, many different numerical methods (e.g., the finite element method [44], pseudo-spectral methods [9], etc.) have been introduced to approximate their solutions/behavior. However, these PDEs can be quite complex for several settings (e.g., turbulence simulations), and numerical integration techniques, which typically update and improve a candidate solution iteratively until convergence, are often quite computationally expensive. Motivated by this—as well as the increasing quantities of data available in many scientific and engineering applications—there has been recent interest in developing machine learning (ML) approaches to find the solution of the underlying PDEs (and/or work in tandem with numerical solutions). As a result, the area of Scientific Machine Learning (SciML)—which aims to couple traditional scientific mechanistic modeling
∗Equal contribution
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
(typically, differential equations) with data-driven ML methodologies (most recently, neural network training)—has emerged. In this vein, there have been a number of ML approaches to incorporate scientific knowledge into such problems while keeping the automatic, data-driven estimates of the solution [2, 17, 33, 39].
A recent line of work involves Physics-Informed Neural Network (PINN) models, which aim to incorporate physical domain knowledge as soft constraints on an empirical loss function, that is then optimized using existing ML training methodologies. To some degree, PINNs are an example of “grafting together” domain-driven models and data-driven methodologies. However, there are important subtleties with this, and we identify several possible failure modes with a naive approach. We then illustrate possible directions for addressing these failure modes.
Background and problem overview. Many of the problems with a PDE constraint fit the following abstraction: F(u(x, t)) = 0, x ∈ Ω ⊂ Rd, t ∈ [0, T ], (1) where F is a differential operator representing the PDE, u(x, t) is the state variable (i.e., parameter of interest), x/t denote space/time, T is the time horizon, and Ω is the spatial domain. Since F is a differential operator, in general one must specify appropriate boundary and/or initial conditions to ensure the existence/uniqueness of a solution to Eq. 1. In the context of PDEs, F can be taxonomized into a parabolic, hyperbolic, or elliptic differential operator [23]. Quintessential examples of F include: the convection equation (a hyperbolic PDE), where u(x, t) could model fluid movement, e.g., air or some liquid, over space and time; the diffusion equation (a parabolic PDE), where u(x, t) could model the temperature distribution over space and time; and the Laplace equation (an elliptic PDE), where u(x) could model a steady-state diffusion equation, in the limit as t→∞. One possible data-driven approach is to incorporate domain information by applying Eq. 1 as a “hard constraint” when training a NN on the data. This can be formulated as the following constrained optimization problem,
min θ L(u) s.t. F(u) = 0, (2)
where L(u) is the data-fit term (including initial/boundary conditions), and where F is a constraint on the residual of the PDE system under consideration (i.e., the “physics” knowledge in the equation itself). As mentioned before, for many practical use cases, it is not possible to derive closed form solutions for these problems, and it is often quite difficult to solve problems of the form of Eq. 2, with F(u) as a hard constraint. Another (related but different) data-driven approach is to impose the constraint as a “soft constraint” on the outputs of the NN model,
min θ L(u) + λFF(u), (3)
L(u) = Lu0 + Lub . (4) Here, Lu0 and Lub measure the misfit of the NN prediction and the initial/boundary conditions (which are pre-specified/given as input to the problem), and θ denotes the NN parameters (which takes (x, t), and possibly other quantities, as inputs and then outputs u(x, t)). Furthermore, λF is a regularization parameter that controls the emphasis on the PDE based residual (which we ideally want to be zero). The goal is then to use ML methodologies (stochastic optimization, etc.) to train this NN model to minimize the loss in Eq. 3. In particular, the NN is trained to minimize this modified loss function, where the modification is to penalize the violations of F(u) for some λF ≥ 0. However, even with a large training dataset, this approach does not guarantee that the NN will obey the conservation/governing equations in the constraint Eq. 1. In many SciML problems, these sorts of constraints on the system matter, as they correspond to physical mechanisms of the system. For example, if the conservation of energy equation is only approximately satisfied, then the system being simulated may behave qualitatively differently or even result in unrealistic solutions.
We should also note that this approach of incorporating physics-based regularization, where the regularization constraint, LF , corresponds to a differential operator, is very different than incorporating much simpler norm-based regularization (such as L1 or L2 regularization), as is common in ML more generally. Here, the regularization operator, LF is non-trivially structured—it involves a differential operator that could actually be ill-conditioned, and it does not correspond to a nice convex set (as does
a norm ball). Moreover, LF corresponds to actual physical quantities, and there is often an important distinction between satisfying the constraint exactly versus satisfying the constraint approximately (the soft constraint approach doing only the latter).
Main contributions. The contributions of this paper are as follows:
• We analyze PINN models on simple, yet physically relevant, problems of convection, reaction, and reaction-diffusion. We find that the vanilla/regular PINN approach only works for very easy parameter regimes (i.e., small PDE coefficients), but that it fails to learn relevant physics in even moderately more challenging physical regimes, even for problems that have simple closed-form analytical solutions. For many cases, the vanilla PINN approach achieves almost 100% error, as compared to the ground truth solution, even after extensive hyperparameter tuning. (See §3 for details.)
• We analyze the loss landscape of trained PINN models and find that adding/increasing the PDE-based soft constraint regularization (LF in Eq. 3) makes it more complex and harder to optimize, especially for cases with non-trivial coefficients. We also study how the loss landscape changes as the regularization parameter (λF ) is changed. We find that reducing the regularization parameter can help alleviate the complexity of the loss landscape, but this in turn leads to poor solutions with high errors that do not satisfy the PDE/constraint. (See §4 for details.)
• We demonstrate that the NN architecture has the capacity/expressivity to find a good solution, thereby showing that these problems are not due to the limited capacity of the NN architecture. Instead, we argue that the failure is due to optimization difficulties associated with the PINN’s soft PDE constraint. (See §5 for details.)
• We propose two paths forward to address these failure modes through (i) curriculum regularization and (ii) posing the learning problem as a sequence-to-sequence learning task. First, in curriculum regularization, we start by imposing the PDE constraint (LF ) with small coefficients, which are progressively increased to the target problem’s settings as the model gets trained. This gives the NN an opportunity to first train with easier constraints, before it is exposed to the target constraint which could be hard to optimize from the beginning. Second, we show that changing the learning problem to a sequence-to-sequence learning problem can reduce the PINN error, again without any change to the NN architecture. In this setup, the NN is trained on a time segment, instead of the full space-time, which could be more difficult to learn. The task is then to predict the solution and reduce the loss only over smaller time segments. We extensively test both approaches and show that they can reduce the error by up to 1-2 orders of magnitude as compared to regular PINN training, and in many cases can better capture “sharp” features in the solution. (See §5 for details.)
• We have open sourced our framework [26] which is built on top of PyTorch both to help with reproducibility and also to enable other researchers to extend the results.

heading: 2 Related work

text:
There is a large body of related work, and here we briefly discuss the most related lines of work.
Machine learning and PDEs. ML approaches for PDE problems have been increasing rapidly in recent years [13, 19]. A number of tools and methodologies now exist to solve scientific problems by combining ML and domain insights [14, 20, 27, 28, 38]. As mentioned earlier, a popular approach to combine ML and physical knowledge is to include aspects of the PDE term as part of the optimization process via regularization. A notable aspect of such an approach is that the NN can be trained only on data that comes from the governing equation(s) itself (though additional data can be included as well, if available), i.e., with a relatively small amount of data. This has garnered interest and shown successful results in a wide variety of science and engineering problems and applications [3, 11, 16, 29–31, 43].
However, there have also been issues observed with this formulation. For example, it did not work well for stiff ordinary differential equations (ODEs) describing chemical kinetics [15], for certain heterogeneous media problems [7], or for certain fluid flow problems [10]. Furthermore, PINN models have been analyzed in the context of neural tangent kernels (i.e., towards the infinite width
limit) to study their convergence [36, 37]. This work found some cases where the model failed (such as when the target function exhibits “high frequency features”) and showed some preliminary solutions via the lens of the neural tangent kernel. It has been argued that some of these problems may be due an imbalance in back-propagated gradients in the loss function during training, and a learning-rate annealing scheme has been proposed to mitigate this [35].
Physical priors and constraints in NNs. Imposing physical priors and constraints on NN systems is common in SciML problems, as a way to try to enforce a property of interest. This idea has been introduced in different forms in the past (for instance [5, 18, 25, 28, 32]). Some approaches have focused on embedding specialized physical constraints into NNs, such as conservation of energy or momentum [4, 12] or multiscale features [34]. While methods focusing on constraining the output of the NN are more common, it is difficult to enforce such constraints exactly in ML settings. Previous work has tried to impose hard constraints in ML (both within the context of SciML and otherwise) [6, 21, 22, 24, 40], although this can be computationally expensive, and does not guarantee better results or convergence.

heading: 3 Possible failure modes for physics-informed neural networks

text:
In this section, we highlight several examples where the PINN formulation defined in Eq. 3 does not predict the solution well. We first demonstrate this with two different types of simple, canonical PDE/ODE systems which have simple analytical solutions: convection ( §3.1), and reaction ( §A). We then also include a diffusion component by looking at the reaction-diffusion problem ( §3.2). Note that the convection problem has a linear PDE constraint, and reaction/reaction-diffusion problems both have non-linear PDE terms.2 We show that PINNs can only learn simple problems with very small parameter values (e.g., small convection or reaction coefficients). We demonstrate that these models fail to learn the relevant physical phenomena for non-trivial cases (e.g., relatively larger coefficients). As we will see, while adding the physical constraint as a soft regularization may be easier to deploy and optimize with existing unconstrained optimization methods, this approach does come with trade-offs, including that in many cases the optimization problem becomes much more difficult to solve.
Experiment setup. We study both linear and non-linear PDEs/ODEs, and we vary the convection, reaction, and diffusion coefficients for each problem (hereafter, we refer to these as PDE coefficients). For each problem, we aim to minimize the loss function in Eq. 3. We use a 4-layer fully-connected NN with 50 neurons per layer, a hyperbolic tangent activation function, and randomly sample collocation points (x, t) on the domain. Furthermore, all the systems that we consider have periodic boundary conditions. We enforce this through an extra term in the loss function that takes the difference between the predicted NN solution at each boundary. We train this network using the L-BFGS optimizer and sweep over learning rates from 1e−4 to 2.0.3 After training the PINN, we measure the L2 relative and absolute errors between the PINN’s predicted solution and the analytical solution. The L2 relative error is 1
N
∑N i=0
|| û− u ||2 || u ||2 ; and the absolute error is 1 N
∑N i=0 || û− u ||2, where
N is the number of evaluation grid points, û is the predicted solution by the PINN, and u is the true solution. For all cases, we run models at least ten times with different preset random seeds, and we average the relative and absolute errors in u(x, t). For each loss function, û is the output of the NN and shorthand for û = NN(θ, x, t).
2Note that for convection, this does not necessarily mean that the mapping from initial to final solution is also linear. This just means that the terms in the PDE are linear.
3For reasons that are only partially understood, L-BFGS methods tend to perform better for existing PINN problems. While variants of stochastic gradient descent are much more popular in computer vision, natural language processing, and recommendation systems, we found that they underperform in comparison to L-BFGS.

heading: 3.1 Learning convection

text:
Problem formulation. We first consider a one-dimensional convection problem, a hyperbolic PDE which is commonly used to model transport phenomena:
∂u ∂t + β ∂u ∂x = 0, x ∈ Ω, t ∈ [0, T ], (5)
u(x, 0) = h(x), x ∈ Ω. Here, β is the convection coefficient and h(x) is the initial condition. For constant β and periodic boundary conditions, this problem has a simple analytical solution:
uanalytical(x, t) = F −1(F (h(x))e−iβkt), (6)
where F is the Fourier transform, i = √ −1, and k denotes frequency in the Fourier domain. The general loss function for this problem (corresponding to Eq. 3) is
L(θ) = 1 Nu Nu∑ i=1 ( û− ui0 )2 + 1 Nf Nf∑ i=1 λi (∂û ∂t + β ∂û ∂x )2 + LB, (7)
where û = NN(θ, x, t) is the output of the NN, and LB is the boundary loss. For periodic boundary conditions with Ω = [0, 2π), this loss is:
LB = 1
Nb Nb∑ i=1 ( û(θ, 0, t)− û(θ, 2π, t) )2 . (8)
We use the following simple initial and periodic boundary conditions:
u(x, 0) = sin(x),
u(0, t) = u(2π, t). (9)
Observations. We apply the PINN’s soft regularization to this problem, and we optimize the loss function in Eq. 7. After training, we measure the relative and absolute errors between the PINN’s predicted solution and the analytical solution, as reported in Fig. 1(a). As one can see, the PINN is only able to achieve good solutions for small values of convection coefficient, and it fails when β becomes larger, reaching a relative error of almost 100% for β > 10. We also provide visualization of the exact and PINN solution in Fig. 1(b-c). One can clearly see that the PINN is unable to learn the solution. As we will later show, the NN architecture does have enough capacity to find the solution, but the training/optimization problem is very difficult to solve with PINNs (and importantly, it may require extensive hyperparameter tuning which is often not feasible in practice).

heading: 3.2 Learning reaction-diffusion

text:
Problem formulation. We next look at a reaction-diffusion system, where we add a diffusion operator to the reaction equation discussed above. Note that for pure diffusion, the solution dissipates
to a steady-state of uniform/constant distribution, which may be trivial to learn. Therefore, we consider studying the reaction-diffusion system:
∂u ∂t − ν ∂
2u
∂x2 − ρu(1− u) = 0, x ∈ Ω, t ∈ (0, T ], (10)
u(x, 0) = h(x), x ∈ Ω. Here, ν (ν > 0) is the diffusion coefficient. The solution of such a system can be solved for via Strang splitting, i.e., splitting the equation into two separate models (a reaction component and a diffusion component):
du dt = ρu(1− u) du
dt = ν
∂2u ∂x2 .
(11)
For each timestep, we can solve the reaction equation through Eq. 15 (in §A). The diffusion equation has the following analytical solution:
uanalytical(x, t) = F −1(F (u(x, t = tn))e−νk2t), (12)
where u(x, t = tn) is the solution at the nth time step. We solve the reaction equation for each timestep, and then use the reaction solution as the initial condition to solve the diffusion component and get the final solution.
The general loss function for this problem is,
L(θ) = 1 Nu Nu∑ i=1 ( û− ui0 )2 +
1
Nf Nf∑ i=1 λi (∂û ∂t − ν ∂ 2û ∂x2 − ρû(1− û) )2 + LB,
(13)
where LB is the boundary loss. Similar to the previous example, periodic boundary conditions can be enforced by including LB from Eq. 8 as an extra term in the loss.
Observations. Similar to the previous case, we can see that the PINN also fails to learn reactiondiffusion. We illustrate a case in Fig. 2 with ρ = 5, when ν = 5. The PINN achieves a high relative error of 93%. Here, we can clearly see that the PINN is unable to capture either the reaction or diffusion component. Additional figures for different ν values can be seen in Fig. D.1. In particular, for ν = 2 the PINN achieves a relative error of 50%. Here, we see that it is unable to capture the “sharper” transitions, though it can predict the center of the solution a little better.

heading: 4 Diagnosing possible failure modes for physics-informed NNs

text:
Thus far, we have shown that PINNs can result in high errors even for simple physical regimes, in particular for PDEs/ODEs with non-trivial convection/reaction/diffusion coefficients. Here, we
demonstrate that one of the underlying reasons for this arises due to the PDE-based soft constraint of LF , which makes the loss landscape difficult to optimize. We first (in §4.1) analyze the loss landscape to illustrate how increasing this soft regularization can lead to more complex loss landscapes, thus leading to optimization difficulties. We then (in §B) demonstrate how this is related to regularizing with differential operators, which can result in ill-conditioning.

heading: 4.1 Soft PDE regularization and optimization difficulties

text:
Here, we analyze how the loss landscape changes for different regimes for the convection problem in §3.1 with/without the soft regularization in PINNs. We show that adding the soft regularization can actually make the problem harder to optimize, i.e., the regularization leads to less smooth loss landscapes. For all the experiments, we plot the loss landscape by perturbing the (trained) model across the first two dominant Hessian eigenvectors and computing the corresponding loss values. This tends to be more informative than perturbing the model parameters in random directions [41, 42].
Figure 3 shows the loss landscape for the convection problem (discussed in §3.1), for different β values. Interestingly, the loss landscape at a relatively low β = 1 is rather smooth, but increasing β further results in a complex and non-symmetric loss landscape. It is also evident that the optimizer has gotten stuck in a local minima with a very high loss function for large β values.
Finally, we study the impact of changing the weight/multiplier for the soft regularization term (i.e., the λ parameter in Eq. 3), which can be relevant in improving PINN performance [35]. While we find that tuning λ can help change the error, it cannot resolve the problem, as shown in Fig. E.1. Note that as the regularization parameter is increased, the loss landscape becomes increasingly more complex and harder to optimize (additionally, see the z-axis scale).

heading: 5 Expressivity versus optimization difficulty

text: In this section, we first show that the failure modes we observed are not necessarily due to the specific NN architecture that we used in our experiments. In particular, we show that the NN model does have the expressivity/capacity to learn the convection/reaction/diffusion coefficient cases where the vanilla PINN method fails. Additionally, in the process of demonstrating this, we also describe two methods that lead to significantly lower error rates. In particular, we show that changing the learning paradigm to curriculum regularization can make the optimization problem easier to solve (as discussed in §5.1). Second, we show that posing the problem as sequence-to-sequence learning may lead to better results than learning the entire state-space at once (as discussed in §5.2).

heading: 5.1 Curriculum PINN Regularization

text:
One may contend that the failure modes shown in §3 may be because the NN does not have enough capacity. Here, we show that this is not the underlying reason. To do so, we devise a “curriculum
regularization” method to warm start the NN training by finding a good initialization for the weights. Instead of training the PINN to learn the solution right away for cases with higher β/ρ, we start by training the PINN on lower β/ρ (easier for the PINN to learn) and then gradually move to training the PINN on higher β/ρ, respectively. We test these results for the examples in §3.1 and §A. This is somewhat analogous to curriculum learning in ML [1], but applied by progressively making the PDE/ODE harder to solve.
Figure 4 shows the training procedure for an example convection case ( §3.1) with β = 30. As Fig. 4(c) shows, the curriculum regularization approach results in a much more accurate solution than regular PINN training. With curriculum regularization, the relative error is almost two orders of magnitude lower. Additionally, this is true across all the other regimes that we found regular PINNs to fail, as shown in Tab. 1. In Fig. E.2, we also show that curriculum regularization not only decreases error significantly, but also decreases the variance of the error. In Fig. E.3, we see that curriculum regularization results in a much smoother loss landscape as compared to regular PINN training.
Curriculum regularization also works well for the reaction example in §A. In this case, we start by training with a low ρ value (reaction coefficient), and then increase gradually to higher ρ values. The results can be seen in Fig. E.4. We can see that the error is 0.1 - 0.6 orders of magnitude lower for ρ = 2− 4 (when the regular PINN error is not as high), and then greatly decreases error by 1-2 orders of magnitude for ρ = 5− 10. As we discussed before, PINN has difficulty in learning sharp features for high values of ρ. However, the curriculum regularization overcomes this, even for ρ = 10, as seen in Fig. E.4(c).

heading: 5.2 Sequence-to-sequence learning vs learning the entire space-time solution

text:
The original PINN approach of [28] trains the NN model to predict the entire space-time at once (i.e., predict u for all locations and time points). In certain cases, this can be more difficult to learn. Here, we demonstrate that it may be better to pose the problem as a sequence-to-sequence (seq2seq) learning task, where the NN learns to predict the solution at the next time step, instead of all times.
0.0 0.2 0.4 0.6 0.8 1.0 t
0
1
2
3
4
5
6
x
Initial condition points Boundary points Collocation points
(a) Regular PINN training
t 2 t 3 t 4 t 5 t 6 t 7 t 8 t 9 t 10 t t 0
1
2
3
4
5
6
x
(b) Sequence-to-sequence learning (model trained every ∆t)
0.0 0.2 0.4 0.6 0.8 1.0 t
0
1
2
3
4
5
6
x
Initial condition points Boundary points Collocation points
Figure 5: Schematic outlining seq2seq learning. In contrast to regular PINN training, the solution in seq2seq learning is predicted for only one ∆t step at a time. Then, the predicted solution at t = ∆t is used as the initial condition for the next segment. To allow fair comparison, we keep the total number of collocation points to be exactly the same in either approach. That is, we do not increase the number of collocation points for seq2seq learning in the right, and keep it to be the same as in the corresponding segment in the left figure.
Entire state space ∆t = 0.05 ∆t = 0.1 ν = 2, ρ = 5 Relative error 5.07× 10−1 2.04× 10−2 1.18× 10−2
Absolute error 2.70× 10−1 1.06× 10−2 6.41× 10−3
ν = 3, ρ = 5 Relative error 7.98× 10−1 1.92× 10−2 1.56× 10−2 Absolute error 4.79× 10−1 1.01× 10−2 8.17× 10−3
ν = 4, ρ = 5 Relative error 8.84× 10−1 2.37× 10−2 1.59× 10−2 Absolute error 5.74× 10−1 1.15× 10−2 8.01× 10−3
ν = 5, ρ = 5 Relative error 9.35× 10−1 2.36× 10−2 2.39× 10−2 Absolute error 6.46× 10−1 1.09× 10−2 1.15× 10−2
ν = 6, ρ = 5 Relative error 9.60× 10−1 2.81× 10−2 2.69× 10−2 Absolute error 6.84× 10−1 1.17× 10−2 1.28× 10−2
Table 2: Predicting the entire state space versus discretizing the state space (i.e., seq2seq learning) for 1D reaction-diffusion ( §3.2). The seq2seq learning achieves lower error for both ∆t = 0.05 and ∆t = 0.1, in comparison to the PINN’s approach of predicting the entire state space at once.
This way, we can use a marching-in-time scheme to predict different sequences/time points. Note that the only data available here is from the PDE itself, i.e., just the initial condition. We take the prediction at t = ∆t and use this as the initial condition to make a prediction at t = 2∆t, and so on. This is schematically outlined in Fig. 5.
We test this scheme by using the exact same NN architecture as in previous sections, and we report the results in Tab. E.1 for the convection problem of §3.1, Tab. E.2 for the reaction problem of §A, and Tab. 2 for the reaction-diffusion problem of §3.2. We compare the relative/absolute error when the learning is posed as a seq2seq problem (i.e., predicting the state space with a “time marching scheme” of one timestep prediction at a time) to the PINN approach of predicting the whole state space at once.4
We explore the following cases where the PINN does poorly, varying β, ρ, and ν coefficients:
4To have a fair comparison between the two methods (time marching versus predicting entire state space at once), for the time marching method, we use the same number of collocation (interior) points for both. For example, for T = [0, 1], if we use 1000 collocation points to predict the entire state space, then for ∆t = 0.1 we use 100 collocation points per section.
1) For 1D convection ( §3.1), higher β values from 30-40.
2) For 1D reaction ( §A), ρ coefficients from 5-10.
3) For 1D reaction-diffusion ( §3.2), a fixed ρ = 5 and ν coefficients from 2-6.
For these cases, we find that posing the problem as seq2seq learning results in significantly lower error. The difference is particularly striking for the reaction and reaction-diffusion cases, where the seq2seq PINN model decreases error by almost two orders of magnitude. An example case is shown Fig. 6, where the seq2seq approach is able to recover the solution, while regular PINNs does very poorly. Note that this behavior also has analogues with numerical methods used in scientific computing, where space-time problems are typically harder to solve, as compared to time marching methods [8]. Intuitively, since the problem is ill-conditioned, restricting the dimensions is expected to help. Furthermore, the underlying function/mapping of the input to the solution should be much simpler to approximate over a smaller time span, as compared to the full time horizon.
These initial results are promising, and further developments may lead to still better ways of using PINNs and learning PDEs. In particular, using more sophisticated methods to predict timesteps across the state space may provide improved performance, as may including more sophisticated seq2seq approaches and tuning the regularization parameter (i.e., amount of constraint added).

heading: 6 Conclusions

text: PINNs—and SciML more generally—hold great promise for expanding the scope of ML methodology to important problems in science and engineering. For these problems, however, integrating ML methods with PDE-based domain-driven constraints as a soft regularization term can lead to subtle and critical issues. In particular, we show that this approach can have fundamental limitations which results in failure modes for learning relevant physics commonly used in different fields of science. To show this, we picked two fundamental PDE problems of diffusion and convection and showed that the PINN only works for very simple cases, failing to learn the relevant physical phenomena for even moderately more challenging regimes. We then analyzed the problem to characterize the underlying reasons why these failures occur. In particular, we studied the PINN loss landscape behavior and found it becomes it becomes increasingly complex for large values of diffusion or convection coefficients, and with/without non-homogeneous forcing. We also discussed that the problem is not necessarily due to the limited capacity of the NN, but that it is partly an optimization problem resulting in the PDE-based soft constraint used in PINNs. Furthermore, we showed that the PINN approach of solving for the entire space-time at once may not be efficient, and instead posing the problem as a sequence-to-sequence learning task can provide lower error rates. Addressing these and related issues will be critical if we hope to go beyond existing cut-and-paste approaches, toward engineering a more intimate connection between scientific methodologies and ML methodologies. This will be needed to deliver on the promise of PINNs and SciML more generally.

heading: 7 Acknowledgements.

text: We are thankful to Shashank Subramanian for his feedback and contributions. We also acknowledge helpful discussions with Prof. George Biros, Geoffrey Negiar, and Daniel Rothchild. ASK was supported by Laboratory Directed Research and Development (LDRD) funding under Contract Number DE-AC02-05CH11231 at LBNL and the Alvarez Fellowship in the Computational Research Division at LBNL. AG was supported through funding from Samsung SAIT. MWM would also like to acknowledge the UC Berkeley CLTC, ARO, NSF, and ONR. The UC Berkeley team also acknowledges gracious support from Intel corporation, Intel VLAB, Samsung, Amazon AWS, Google Cloud, Google TPU Research Cloud, and Google Brain (in particular Prof. David Patterson, Dr. Ed Chi, and Jing Li). Our conclusions do not necessarily reflect the position or the policy of our sponsors, and no official endorsement should be inferred.

title: Characterizing possible failure modes in physics-informed neural networks


summaries: Proposes cycle self-training (CST) an unsupervised domain adaptation (UDA) algorithm that cycles between standard self-training of a target classifier on (source model generated) target pseudolabels, and a new cycle self-training step that updates backbone representations so as to generalize the target classifier to the source domain. An additional self-training objective based on the Tsallis entropy is introduced. Theoretical and empirical results are presented on several standard UDA benchmarks.

summaries: The paper tackles the problem of domain adaptation with a newly proposed cycle self-training algorithm. Given observed that the pseudo labels are noisy and existing de-noising methods require ad-hoc hyper-parameters for specific tasks, the authors attempt to progressively refine the pseudo labels by the capability of the network itself. With the intuition of transferring the source-domain knowledge to the target domain, the authors propose to regularize the pseudo labels with a cyclic training pipeline. Tsallis Entropy is further introduced to improve the label quality. Experiments on both visual classification and linguistic sentiment classification indicate the effectiveness of the introduced method.

summaries: This paper presents a novel approach for unsupervised domain adaptation (UDA) based on cycle self-training. Instead of conventional self-training, there are a generated and two attached classifiers with different purposes. The source classifier, mainly trained on the source loss, is applied to infer pseudo target labels. The target classifier is trained by the self-training loss based on the pseudo labels provided by the source classifier.  The authors also propose the Tsallis entropy as the regularization term, which can be seen as a side contribution.

summaries: This paper proposes an unsupervised domain adaptation method based on cyclical self-training. There are two main components to the method: 1. a cyclical self-training algorithm that solves a bilevel optimization problem based on an in the inner loop which trains a target classifier with target pseudo-labels  and an outer loop that makes the target classifier perform well on the source domain by updating the shared representations in the outer loop. 2. an uncertainty measure based on the Tsallis entropy, which adaptively minimizes the uncertainty in target pseudo-labels, which replaces the standard Gibbs entropy used by self-training methods. Experiments on computer vision and NLP datasets show that proposed method outperforms baseline methods by a decent margin. Ablation studies show that both components of the algorithm---cyclical self-training and Tsallis entropy---meaningfully contribute to performance improvement. Qualitative and quantitative studies show that target pseudo-label quality is improved by the proposed approach.

abstractText: Mainstream approaches for unsupervised domain adaptation (UDA) learn domaininvariant representations to narrow the domain shift, which are empirically effective but theoretically challenged by the hardness or impossibility theorems. Recently, self-training has been gaining momentum in UDA, which exploits unlabeled target data by training with target pseudo-labels. However, as corroborated in this work, under distributional shift, the pseudo-labels can be unreliable in terms of their large discrepancy from target ground truth. In this paper, we propose Cycle Self-Training (CST), a principled self-training algorithm that explicitly enforces pseudo-labels to generalize across domains. CST cycles between a forward step and a reverse step until convergence. In the forward step, CST generates target pseudo-labels with a source-trained classifier. In the reverse step, CST trains a target classifier using target pseudo-labels, and then updates the shared representations to make the target classifier perform well on the source data. We introduce the Tsallis entropy as a confidence-friendly regularization to improve the quality of target pseudo-labels. We analyze CST theoretically under realistic assumptions, and provide hard cases where CST recovers target ground truth, while both invariant feature learning and vanilla self-training fail. Empirical results indicate that CST significantly improves over the state-of-the-arts on visual recognition and sentiment analysis benchmarks.

authors:
E.A. Albadawy
A. Saha
M.A. Mazurowski

title: Deep learning for segmentation of brain tumors: Impact of cross-institutional training and testing

venue: Medical Physics,

authors:
E. Arazo
D. Ortego
P. Albert
N.E. O’Connor
K. McGuinness

title: Pseudo-labeling and confirmation bias in deep semi-supervised learning

authors:
S. Arora
S.S. Du
W. Hu
Z. Li
R.R. Salakhutdinov
R. Wang

title: On exact computation with an infinitely wide neural net

venue: In NeurIPS,

authors:
P. Bachman
O. Alsharif
D. Precup

title: Learning with pseudo-ensembles

authors:
P.L. Bartlett
S. Mendelson

title: Rademacher and gaussian complexities: Risk bounds and structural results

authors:
S. Ben-David
R. Urner

title: On the hardness of domain adaptation and the utility of unlabeled target samples

venue: In ALT,

authors:
S. Ben-David
J. Blitzer
K. Crammer
A. Kulesza
F. Pereira
J.W. Vaughan

title: A theory of learning from different domains

venue: Machine Learning,

authors:
D. Berthelot
N. Carlini
I. Goodfellow
N. Papernot
A. Oliver
C. Raffel

title: Mixmatch: A holistic approach to semi-supervised learning

authors:
L. Bertinetto
J.F. Henriques
P. Torr
A. Vedaldi

title: Meta-learning with differentiable closed-form solvers

venue: In ICLR,

authors:
J. Blitzer
M. Dredze
F. Pereira

title: Biographies, Bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification

venue: In ACL,

authors:
T. Cai
R. Gao
J.D. Lee
Q. Lei

title: A theory of label propagation for subpopulation

title: Poisoning the unlabeled dataset of semi-supervised learning, 2021

authors:
C. Chen
W. Xie
W. Huang
Y. Rong
X. Ding
Y. Huang
T. Xu
J. Huang

title: Progressive feature alignment for unsupervised domain adaptation

venue: In CVPR,

authors:
Y. Chen
C. Wei
A. Kumar
T. Ma

title: Self-training avoids using spurious features under domain shift

authors:
J. Devlin
Chang
M.-W
K. Lee
K. Toutanova

title: BERT: Pre-training of deep bidirectional transformers for language understanding

venue: In NAACL,

authors:
J. Du
E. Grave
B. Gunel
V. Chaudhary
O. Celebi
M. Auli
V. Stoyanov
A. Conneau

title: Self-training improves pre-training for natural language understanding

authors:
C. Finn
P. Abbeel
S. Levine

title: Model-agnostic meta-learning for fast adaptation of deep networks

venue: In ICML,

authors:
P. Foret
A. Kleiner
H. Mobahi
B. Neyshabur

title: Sharpness-aware minimization for efficiently improving generalization

authors:
S. Frei
D. Zou
Z. Chen
Q. Gu

title: Self-training converts weak learners to strong learners in mixture models

venue: arXiv preprint arXiv:2106.13805,

authors:
G. French
M. Mackiewicz
M. Fisher

title: Self-ensembling for visual domain adaptation

authors:
Y. Ganin
E. Ustinova
H. Ajakan
P. Germain
H. Larochelle
M. Marchand
V. Lempitsky

title: Domain-adversarial training of neural networks

authors:
G. Ghiasi
B. Zoph
E.D. Cubuk
Q.V. Le
Lin
T.-Y

title: Multi-task self-training for learning general representations

venue: In ICCV,

authors:
Y. Grandvalet
Y. Bengio

title: Semi-supervised learning by entropy minimization

venue: In NeurIPS, pp

authors:
C. Guo
G. Pleiss
Y. Sun
K.Q. Weinberger

title: On calibration of modern neural networks

authors:
K. He
X. Zhang
S. Ren
J. Sun

title: Deep residual learning for image recognition

authors:
J. Hoffman
E. Tzeng
T. Park
J. Zhu
P. Isola
K. Saenko
A.A. Efros
T. Darrell

title: Cycada: Cycle-consistent adversarial domain adaptation

authors:
X. Jiang
Q. Lao
S. Matwin
M. Havaei

title: Implicit class-conditioned domain alignment for unsupervised domain adaptation

authors:
F.D. Johansson
D. Sontag
R. Ranganath

title: Support and invertibility in domain-invariant representations

venue: In AISTATS,

authors:
A. Kumar
T. Ma
P. Liang

title: Understanding self-training for gradual domain adaptation

title: Pseudo-label : The simple and efficient semi-supervised learning method for deep neural networks. ICML Workshop: Challenges in Representation Learning (WREPL)

authors:
B. Li
Y. Wang
T. Che
S. Zhang
S. Zhao
P. Xu
W. Zhou
Y. Bengio
K. Keutzer

title: Rethinking distributional matching based domain adaptation

authors:
H. Liu
M. Long
J. Wang
M. Jordan

title: Transferable adversarial training: A general approach to adapting deep classifiers

authors:
M. Long
Y. Cao
J. Wang
M.I. Jordan

title: Learning transferable features with deep adaptation networks

venue: In ICML, pp

authors:
M. Long
H. Zhu
J. Wang
M.I. Jordan

title: Unsupervised domain adaptation with residual transfer networks

title: Deep transfer learning with joint adaptation networks

authors:
M. Long
Z. Cao
J. Wang
M.I. Jordan

title: Conditional adversarial domain adaptation

authors:
Z. Lu
Y. Yang
X. Zhu
C. Liu
Song
Y.-Z
T. Xiang

title: Stochastic classifiers for unsupervised domain adaptation

authors:
A. Mey
M. Loog

title: A soft-labeled self-training approach

venue: In ICPR,

authors:
T. Miyato
S. Maeda
S. Ishii
M. Koyama

title: Virtual adversarial training: A regularization method for supervised and semi-supervised learning

authors:
S. Mukherjee
A. Awadallah

title: Uncertainty-aware self-training for few-shot text classification

authors:
A. Paszke
S. Gross
F. Massa
A. Lerer
J. Bradbury
G. Chanan
T. Killeen
Z. Lin
N. Gimelshein
L. Antiga
A. Desmaison
A. Kopf
E. Yang
Z. DeVito
M. Raison
A. Tejani
S. Chilamkurthy
B. Steiner
L. Fang
J. Bai
S. Chintala

title: Pytorch: An imperative style, high-performance deep learning library

authors:
X. Peng
B. Usman
N. Kaushik
J. Hoffman
D. Wang
K. Saenko

title: Visda: The visual domain adaptation challenge

authors:
X. Peng
Q. Bai
X. Xia
Z. Huang
K. Saenko
B. Wang

title: Moment matching for multi-source domain adaptation

authors:
V. Prabhu
S. Khare
D. Kartik
J. Hoffman

title: Sentry: Selective entropy optimization via committee consistency for unsupervised domain adaptation, 2020

authors:
X. Qu
Z. Zou
Y. Cheng
Y. Yang
P. Zhou

title: Adversarial category alignment network for cross-domain sentiment classification

authors:
J. Quionero-Candela
M. Sugiyama
A. Schwaighofer
N.D. Lawrence

title: Dataset Shift in Machine Learning

authors:
A. Rasmus
M. Berglund
M. Honkala
H. Valpola
T. Raiko

title: Semi-supervised learning with ladder networks

authors:
C. Rosenberg
M. Hebert
H. Schneiderman

title: Semi-supervised self-training of object detection models

venue: In WACV,

authors:
K. Saito
K. Watanabe
Y. Ushiku
T. Harada

title: Maximum classifier discrepancy for unsupervised domain adaptation

authors:
M. Sajjadi
M. Javanmardi
T. Tasdizen

title: Regularization with stochastic transformations and perturbations for deep semi-supervised learning

authors:
R. Shu
H. Bui
H. Narui
S. Ermon

title: A DIRT-t approach to unsupervised domain adaptation

authors:
K. Sohn
D. Berthelot
N. Carlini
Z. Zhang
H. Zhang
C.A. Raffel
E.D. Cubuk
A. Kurakin
Li
C.-L

title: Fixmatch: Simplifying semi-supervised learning with consistency and confidence

venue: NeurIPS,

authors:
C. Szegedy
W. Zaremba
I. Sutskever
J. Bruna
D. Erhan
I. Goodfellow
R. Fergus

title: Intriguing properties of neural networks

title: Upper and lower bounds for stochastic processes: modern methods and classical problems, volume 60

venue: Springer Science & Business Media,

authors:
S. Tan
X. Peng
K. Saenko

title: Class-imbalanced domain adaptation: An empirical odyssey

venue: In ECCV Workshop,

authors:
A. Tarvainen
H. Valpola

title: Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results

title: Possible generalization of boltzmann-gibbs statistics

venue: Journal of Statistical Physics,

authors:
E. Tzeng
J. Hoffman
K. Saenko
T. Darrell

title: Adversarial discriminative domain adaptation

authors:
H. Venkateswara
J. Eusebio
S. Chakraborty
S. Panchanathan

title: Deep hashing network for unsupervised domain adaptation

authors:
T. Vu
Luong
M.-T
Q.V. Le
G. Simon
M. Iyyer

title: Strata: Self-training with task augmentation for better few-shot learning

venue: arXiv preprint arXiv:2109.06270,

authors:
C. Wei
K. Shen
C. Yining
T. Ma

title: Theoretical analysis of self-training with deep networks on unlabeled data

authors:
Q. Xie
M.T. Luong
E. Hovy
Q.V. Le

title: Self-training with noisy student improves imagenet classification

authors:
S.M. Xie
A. Kumar
R. Jones
F. Khani
T. Ma
P. Liang

title: In-n-out: Pre-training and self-training using auxiliary information for out-of-distribution robustness

authors:
R. Xu
G. Li
J. Yang
L. Lin

title: Larger norm more transferable: An adaptive feature norm approach for unsupervised domain adaptation

authors:
J. Yosinski
J. Clune
Y. Bengio
H. Lipson

title: How transferable are features in deep neural networks

authors:
W. Zellinger
T. Grubinger
E. Lughofer
T. Natschläger
S. Saminger-Platz

title: Central moment discrepancy (CMD) for domain-invariant representation learning

authors:
H. Zhang
M. Cisse
Y.N. Dauphin
D. Lopez-Paz

title: mixup: Beyond empirical risk minimization

authors:
Y. Zhang
T. Liu
M. Long
M. Jordan

title: Bridging theory and algorithm for domain adaptation

authors:
H. Zhao
R.T.D. Combes
K. Zhang
G. Gordon

title: On learning invariant representations for domain adaptation

authors:
Y. Ziser
R. Reichart

title: Pivot based language modeling for improved neural domain adaptation

authors:
B. Zoph
G. Ghiasi
Lin
T.-Y
Y. Cui
H. Liu
E.D. Cubuk
Q. Le

title: Rethinking pre-training and self-training

authors:
Y. Zou
Z. Yu
B.V.K. Vijaya Kumar
J. Wang

title: Unsupervised domain adaptation for semantic segmentation via class-balanced self-training

venue: In ECCV,

authors:
Y. Zou
Z. Yu
X. Liu
B.V. Kumar
J. Wang

title: Confidence regularized self-training

venue: In ICCV, October 2019

heading: 1 Introduction

text:
Transferring knowledge from a source domain with rich supervision to an unlabeled target domain is an important yet challenging problem. Since deep neural networks are known to be sensitive to subtle change in underlying distributions [70], models trained on one labeled dataset often fail to generalize to another unlabeled dataset [58, 1]. Unsupervised domain adaptation (UDA) addresses the challenge of distributional shift by adapting the source model to the unlabeled target data [50, 43].
The mainstream paradigm for UDA is feature adaptation, a.k.a. domain alignment. By reducing the distance of the source and target feature distributions, these methods learn invariant representations to facilitate knowledge transfer between domains [34, 22, 36, 54, 37, 73], with successful applications in various areas such as computer vision [63, 27, 77] and natural language processing [75, 49]. Despite their popularity, the impossibility theories [6] uncovered intrinsic limitations of learning invariant representations when it comes to label shift [74, 32] and shift in the support of domains [29].
Recently, self-training (a.k.a. pseudo-labeling) [21, 78, 30, 32, 47, 68] has been gaining momentum as a promising alternative to feature adaptation. Originally tailored to semi-supervised learning, self-training generates pseudo-labels of unlabeled data, and jointly trains the model with source labels and target pseudo-labels [31, 39, 30]. However, the distributional shift in UDA makes pseudo-labeling more difficult. Directly using all pseudo-labels is risky due to accumulated error and even trivial solution [14]. Thus previous works tailor self-training to UDA by selecting trustworthy pseudo-labels. Using confidence threshold or reweighting, recent works try to alleviate the negative effect of domain
⇤Corresponding author: Mingsheng Long (mingsheng@tsinghua.edu.cn)
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
shift in standard self-training [78, 47], but they can be brittle and require expensive tweaking of the threshold or weight for different tasks, and their performance gain is still inconsistent.
In this work, we first analyze the quality of pseudo-labels with or without domain shift to delve deeper into the difficulty of standard self-training in UDA. On popular benchmark datasets, when the source and target are the same, our analysis indicates that the pseudo-label distribution is almost identical to the ground-truth distribution. However, with distributional shift, their discrepancy can be very large with examples of several classes mostly misclassified into other classes. We also study the difficulty of selecting correct pseudo-labels with popular criteria under domain shift. Although entropy and confidence are reasonable selection criteria for correct pseudo-labels without domain shift, the domain shift makes their accuracy decrease sharply.
Our analysis shows that domain shift makes pseudo-labels unreliable and that self-training on selected target instances with accurate pseudo-labels is less successful. Thereby, more principled improvement of standard self-training should be tailored to UDA and address the domain shift explicitly. In this work, we propose Cycle Self-Training (CST), a principled self-training approach to UDA, which overcomes the limitations of standard self-training (see Figure 1). Different from previous works to select target pseudo-labels with hard-to-tweak protocols, CST learns to generalize the pseudo-labels across domains. Specifically, CST cycles between the use of target pseudo-labels to train a target classifier, and the update of shared representations to make the target classifier perform well on the source data. In contrast to the standard Gibbs entropy that makes the target predictions over-confident, we propose a confidence-friendly uncertainty measure based on the Tsallis entropy in information theory, which adaptively minimizes the uncertainty without manually tuning or setting thresholds. Our method is simple and generally applicable to vision and language tasks with various backbones.
We empirically evaluate our method on a series of standard UDA benchmarks. Results indicate that CST outperforms previous state-of-the-art methods in 21 out of 25 tasks for object recognition and sentiment classification. Theoretically, we prove that the minimizer of CST objective is endowed with general guarantees of target performance. We also study hard cases on specific distributions, showing that CST recovers target ground-truths while both feature adaptation and standard self-training fail.

heading: 2 Preliminaries

text:
We study unsupervised domain adaptation (UDA). Consider a source distribution P and a target distribution Q over the input-label space X ⇥ Y . We have access to ns labeled i.i.d. samples bP = {xsi , ysi } ns i=1 from P and nt unlabeled i.i.d. samples bQ = {xti} nt i=1 from Q. The model f comprises a feature extractor h parametrized by and a head (linear classifier) g✓ parametrized by ✓, i.e. f✓, (x) = g✓(h (x)). The loss function is `(·, ·). Denote by LP (✓, ) := E(x,y)⇠P `(f✓, (x), y) the expected error on P . Similarly, we use L bP (✓, ) to denote the empirical error on dataset bP . We discuss two mainstream UDA methods and their formulations: feature adaptation and self-training.
Feature Adaptation trains the model f on the source dataset bP , and simultaneously matches the source and target distributions in the representation space Z = h(X ):
min ✓,
L bP (✓, ) + d(h] bP , h] bQ). (1)
Here, h] bP denotes the pushforward distribution of bP , and d(·, ·) is some distribution distance. For instance, Long et al. [34] used maximum mean discrepancy dMMD, and Ganin et al. [22] approximated the H H-distance dH H [7] with adversarial training. Despite its pervasiveness, recent works have shown the intrinsic limitations of feature adaptation under real-world situations [6, 74, 33, 32, 29].
Self-Training is considered a promising alternative to feature adaptation. In this work we mainly focus on pseudo-labeling [31, 30]. Stemming from semi-supervised learning, standard self-training trains a source model fs on the source dataset bP : min✓s, s L bP (✓s, s). The target pseudo-labels are then generated by fs on the target dataset bQ. To leverage unlabeled target data, self-training trains the model on the source and target datasets together with source ground-truths and target pseudo-labels:
min ✓, L bP (✓, ) + Ex⇠ bQ`(f✓, (x), argmax i {f✓s, s(x)[i]}). (2)
Self-training also uses label-sharpening as a standard protocol [31, 57]. Another popular variant of pseudo-labeling is the teacher-student model [4, 61], which iteratively improves the quality of pseudo-labels via alternatively replacing ✓s and s with ✓ and of the previous iteration.

heading: 2.1 Limitations of Standard Self-Training

text:
Standard self-training with pseudo-labels uses unlabeled data efficiently for semi-supervised learning [31, 39, 57]. Here we carry out exploratory studies on the popular VisDA-2017 [45] dataset using ResNet-50 backbones. We find that domain shift makes the pseudo-labels biased towards several classes and thereby unreliable in UDA. See Appendix C.1 for details and results on more datasets.
Pseudo-label distributions with or without domain shift. We resample the original VisDA-2017 to simulate different relationship between source and target domains: 1) i.i.d., 2) covariate shift, and 3) label shift. We train the model on the three variants of source dataset and use it to generate target pseudo-labels. We show the distributions of target ground-truths and pseudo-labels in Figure 2 (Left). When the source and target distributions are identical, the distribution of pseudo-labels is almost the same as ground-truths, indicating the reliability of pseudo-labels. In contrast, when exposed to label shift or covariate shift, the distribution of pseudo-labels is significantly different from target ground-truths. Note that classes 2, 7, 8 and 12 appear rarely in the target pseudo-labels in the covariate shift setting, indicating that the pseudo-labels are biased towards several classes due to domain shift. Self-training with these pseudo-labels is risky since it may lead to misalignment of distributions and misclassify many examples of classes 2, 7, 8 and 12.
Change of pseudo-label distributions throughout training. To further study the change of pseudolabels in standard self-training, we compute the total variation (TV) distance between target groundtruths and target pseudo-labels: dTV(c, c0) = 12 P i kci c 0
ik, where ci is the ratio of class i. We plot its change during training in Figure 2 (Middle). Although the error rate of pseudo-labels continues to decrease, dTV remains almost unchanged at 0.26 throughout training. Note that dTV is the lower bound of the error rate of the pseudo-labels (shown in Appendix C.1). If dTV converges to 0.26, then the accuracy of pseudo-labels is upper-bounded by 0.74. This indicates that the important denoising ability [66] of pseudo-labels in standard self-training is hindered by domain shift.
Difficulty of selecting reliable pseudo-labels under domain shift. To mitigate the negative effect of false pseudo-labels, recent works proposed to select correct pseudo-labels based on thresholding the entropy or confidence criteria [35, 21, 37, 57]. However, it remains unclear whether these strategies are still effective under domain shift. Here we compare the quality of pseudo-labels selected by
different strategies with or without domain shift. For each strategy, we compute False Positive Rate and True Positive Rate for different thresholds and plot its ROC curve in Figure 2 (Right). When the source and target distributions are identical, both entropy and confidence are reasonable strategies for selecting correct pseudo-labels (AUC=0.89). However, when the target pseudo-labels are generated by the source model, the quality of pseudo-labels decreases sharply under domain shift (AUC=0.78).

heading: 3 Approach

text: We present Cycle Self-Training (CST) to improve pseudo-labels under domain shift. An overview of our method is given in Figure 1. Cycle Self-Training iterates between a forward step and a reverse step to make self-trained classifiers generalize well on both target and source domains.

heading: 3.1 Cycle Self-Training

text:
Forward Step. Similar to standard self-training, we have a source classifier ✓s trained on top of the shared representations on the labeled source domain, and use it to generate target pseudo-labels as
y 0 = argmax i {f✓s, (x)[i]}, (3)
for each x in the target dataset bQ. Traditional self-training methods use confidence thresholding or reweighting to select reliable pseudo-labels. For example, Sohn et al. [57] select pseudo-labels with softmax value and Long et al. [37] add entropy reweighting to rely on examples with more confidence prediction. However, the output of deep networks is usually miscalibrated [25], and is not necessarily related to the ground-truth confidence even on the same distribution. In domain adaptation, as shown in Section 2.1, the discrepancy between the source and target domains makes pseudo-labels even more unreliable, and the performance of commonly used selection strategies is also unsatisfactory. Another drawback is the expensive tweaking in order to find the optimal confidence threshold for new tasks. To better apply self-training to domain adaptation, we expect that the model can gradually refine the pseudo-labels by itself without the cumbersome selection or thresholding.
Reverse Step. We design a complementary step with the following insights to improve self-training. Intuitively, the labels on the source domain contain both useful information that can transfer to the target domain and harmful information that can make pseudo-labels incorrect. Similarly, reliable pseudo-labels on the target domain can transfer to the source domain in turn, while models trained with incorrect pseudo-labels on the target domain cannot transfer to the source domain. In this sense, if we explicitly train the model to make target pseudo-labels informative of the source domain, we can gradually make the pseudo-labels more accurate and learn to generalize to the target domain.
Specifically, with the pseudo-labels y0 generated by the source classifier ✓s at hand as in equation 3, we train a target head ✓̂t( ) on top of the representation with pseudo-labels on the target domain bQ,
✓̂t( ) = argmin ✓ Ex⇠ bQ`(f✓, (x), y 0). (4)
We wish to make the target pseudo-labels informative of the source domain and gradually refine them. To this end, we update the shared feature extractor to predict accurately on the source domain and jointly enforce the target classifier ✓̂t( ) to perform well on the source domain. This naturally leads to the objective of Cycle Self-Training:
minimize ✓s, LCycle(✓s, ) := L bP (✓s, ) + L bP (✓̂t( ), ). (5)
Bi-level Optimization. The objective in equation 5 relies on the solution ✓̂t( ) to the objective in equation 4. Thus, CST formulates a bi-level optimization problem. In the inner loop we generate target pseudo-labels with the source classifier (equation 3), and train a target classifier with target pseudo-labels (equation 4). After each inner loop, we update the feature extractor for one step in the outer loop (equation 5), and start a new inner loop again. However, since the inner loop of the optimization in equation 4 only involves the light-weight linear head ✓t, we propose to calculate the analytical form of ✓̂t( ) and directly back-propagate to the feature extractor instead of calculating the second-order derivatives as in MAML [18]. The resulting framework is as fast as training two heads jointly. Also note that the solution ✓̂t( ) relies on ✓s implicitly through y0. However, both standard self-training and our implementation use label sharpening, making y0 not differentiable. Thus we follow vanilla self-training and do not consider the gradient of ✓̂t( ) w.r.t. y0 in the outer loop optimization. We defer the derivation and implementation of bi-level optimization to Appendix B.2.

heading: 3.2 Tsallis Entropy Minimization

text:
Gibbs entropy is widely used by existing semi-supervised learning methods to regularize the model output and minimize the uncertainty of predictions on unlabeled data [24]. In this work, we generalize Gibbs entropy to Tsallis entropy [62] in information theory. Suppose the softmax output of a model is y 2 RK , then the ↵-Tsallis entropy is defined as
S↵(y) = 1
↵ 1
⇣ 1 X y ↵ [i] ⌘ , (6)
where ↵ > 0 is the entropic-index. Note that lim↵!1 S↵(y) = P
i y[i]log(y[i]) which exactly recovers the Gibbs entropy. When ↵ = 2, S↵(y) becomes the Gini impurity 1 P i y 2 [i].
We propose to control the uncertainty of target pseudo-labels based on Tsallis entropy minimization: L bQ,Tsallis,↵(✓, ) := Ex⇠ bQS↵(f✓, (x)). (9)
Figure 3 shows the change of Tsallis entropy with different entropic-indices ↵ for binary problems. Intuitively, smaller ↵ exerts more penalization on uncertain predictions and larger ↵ allows several scores yi’s to be similar. This is critical in self-training since an overly small ↵ (as in Gibbs entropy) will make the incorrect dimension of pseudo-labels close to 1 and have no chance to be corrected throughout training. In Section 5.4, we further verify this property with experiments.
An important improvement of the Tsallis entropy over Gibbs entropy is that it can choose the suitable measure of uncertainty for different systems to avoid over-confidence caused by overly penalizing the uncertain pseudo-labels. To automatically find the suitable ↵, we adopt a similar strategy as Section 3.1. The intuition is that if we use the suitable entropic-index ↵ to train the source classifier ✓s,↵, the target pseudo-labels generated by ✓s,↵ will contain desirable knowledge of the source dataset, i.e. a target classifier ✓t,↵ trained with these pseudo-labels will perform well on the source domain. Therefore, we semi-supervisedly train a classifier ✓̂s,↵ on the source domain with the ↵-Tsallis entropy regularization L bQ,Tsallis,↵ on the target domain as: ✓̂s,↵ = argmin✓ L bP (✓, )+L bQ,Tsallis,↵(✓, ), from which we obtain the target pseudo-labels. Then we train another head ✓̂t,↵ with target pseudo-labels. We automatically find ↵ by minimizing the loss of ✓̂t,↵ on the source data:
↵̂ = argmin ↵2[1,2] L bP (✓̂t,↵, ) (10)
To solve equation 10, we discretize the feasible region [1, 2] of ↵ and use discrete optimization to lower computational cost. We also update ↵ at the start of each epoch, since we found more frequent
Algorithm 1 Cycle Self-Training (CST) 1: Input: source dataset bP and target dataset bQ. 2: for epoch = 0 to MaxEpoch do 3: Select ↵̂ as equation 10 at the start of each epoch. 4: for t = 0 to MaxIter do 5: Forward Step 6: Generate pseudo-labels on the target domain with and ✓s: y0 = argmaxi{f✓s, (x)[i]}. 7: Reverse Step 8: Train a target head ✓̂t( ) with target pseudo-labels y0 on the feature extractor :
✓̂t( ) = argmin ✓ Ex⇠ bQ`(f✓, (x), y 0).
9: Update the feature extractor and the source head ✓s to make ✓̂t( ) perform well on the source dataset and minimize the ↵̂-Tsallis entropy on the target dataset:
⌘r [L bP (✓s, ) + L bP (✓̂t( ), ) + L bQ,Tsallis,↵̂(✓s, )]. (7)
✓s ✓s ⌘r✓s [L bP (✓s, ) + L bQ,Tsallis,↵̂(✓s, )]. (8)10: end for 11: end for
update leads to no performance gain. Details are deferred to Appendix B.3. Finally, with the optimal ↵̂ found, we add the ↵̂-Tsallis entropy minimization term L bQ,Tsallis,↵̂ to the overall objective:
minimize ✓s, LCycle(✓s, ) + L bQ,Tsallis,↵̂(✓s, ). (11)
In summary, Algorithm 1 depicts the complete training procedure of Cycle Self-Training (CST).

heading: 4 Theoretical Analysis

text: We analyze the properties of CST theoretically. First, we prove that the minimizer of the CST loss LCST(fs, ft) will lead to small target loss ErrQ(fs) under a simple but realistic expansion assumption. Then, we further demonstrate a concrete instantiation where cycle self-training provably recovers the target ground truth, but both feature adaptation and standard self-training fail. Due to space limit, we state the main results here and defer all proof details to Appendix A.

heading: 4.1 CST Provably Works under the Expansion Assumption

text:
We start from a K-way classification model, f : X ! [0, 1]K 2 F and f̃(x) := argmaxi f(x)[i] denotes the prediction. Denote by Pi the conditional distribution of P given y = i. Assume the supports of Pi and Pj are disjoint for i 6= j. The definition is similar for Qi. We further Assume P (y = i) = Q(y = i). For any x 2 X , N (x) is defined as the neighboring set of x with a proper metric d(·, ·), N (x) = {x0 : d(x, x0)  ⇠}. N (A) := [x2AN (x). Denote the expected error on the target domain by ErrQ(f) := E(x,y)⇠QI(f̃(x) 6= y). We study the CST algorithm under the expansion assumption of the mixture distribution [66, 11]. Intuitively, this assumption indicates that the conditional distributions Pi and Qi are closely located and regularly shaped, enabling knowledge transfer from the source domain to the target domain. Definition 1 ((q, ✏)-constant expansion [66]). We say P and Q satisfy (q, ✏)-constant expansion for some constant q, ✏ 2 (0, 1), if for any set A 2 X and any i 2 [K] with 12 > P 12 (Pi+Qi)(A) > q, we have P 1
2 (Pi+Qi) (N (A)\A) > min{✏, P 1 2 (Pi+Qi) (A)}.
Based on this expansion assumption, we consider a robustness-constrained version of CST. Later we will show that the robustness is closely related to the uncertainty. Denote by fs the source model and ft the model trained on the target with pseudo-labels. Let R(ft) := P 1
2 (P+Q) ({x : 9x0 2
N (x), f̃t(x) 6= f̃t(x0)}) represent the robustness [66] of ft on P and Q. Suppose E(x,y)⇠QI(f̃s(x) 6= f̃t(x))  c and R(ft)  ⇢. The following theorem states that when fs and ft behave similarly on the target domain Q and ft is robust to local changes in input, the minimizer of the cycle source error ErrP (ft) will guarantee low error of fs on the target domain Q.
Theorem 1. Suppose Definition 1 holds for P and Q. For any fs, ft satisfying E(x,y)⇠QI(f̃s(x) 6= f̃t(x))  c and R(ft)  ⇢, the expected error of fs on the target domain Q is bounded,
ErrQ(fs)  ErrP (ft) + c+ 2q + ⇢
min{✏, q} . (12)
To further relate the expected error with the CST training objective and obtain finite-sample guarantee, we use the multi-class margin loss: l (f(x), y) := ( M(f(x), y)), where M(v, y) = v[y] maxy0 6=y v[y0] and is the ramp function. We then extend the margin loss: M(v) = maxy(v[y] maxy0 6=y v[y0]) (The difference between the largest and the second largest scores in v), and l (ft(x), fs(x)) := ( M(ft(x), f̃s(x))). Further suppose f[i] is Lf -Lipschitz w.r.t. the metric d(·, ·) and ⌧ := 1 2Lf⇠min{✏, q} > 0. Consider the following training objective for CST, denoted by LCST(fs, ft), where L bP, (ft) := E(x,y)⇠ bP l (ft(x), y) corresponds to the cycle source loss in equation 5, L bQ, (ft, fs) := E(x,y)⇠ bQl (ft(x), fs(x)) is consistent with the target loss in equation 4, and M(ft(x)) is closely related to the uncertainty of predictions in equation 11.
minLCST(fs, ft) := L bP, (ft) + L bQ, (ft, fs) + 1 E(x,y)⇠ 12 ( bP+ bQ)M(ft(x))
⌧ . (13)
The following theorem shows that the minimizer of the training objective LCST(fs, ft) guarantees low population error of fs on the target domain Q.
Theorem 2. bR(F| bP ) denotes the empirical Rademacher complexity of function class F on dataset bP . For any solution of equation 13 and > 0, with probability larger than 1 ,
ErrQ(fs)  LCST(fs, ft) + 2q + 4K h bR(F| bP ) + bR(F̃ ⇥ F| bQ) i + 2
⌧
h bR(F| bP ) + bR(F| bQ) i + ⇣,
where ⇣ = O ⇣p log(1/ )/ns + p log(1/ )/nt ⌘
is a low-order term. F̃ ⇥ F refers to the function class {x! f(x)[f̃ 0(x)] : f, f 0 2 F}.
Main insights. Theorem 2 justifies CST under the expansion assumption. The generalization error of the classifier fs on the target domain is bounded with the CST loss objective LCST(fs, ft), the intrinsic property of the data distribution q, and the complexity of the function classes. In our algorithm, LCST(fs, ft) is minimized by the neural networks and q is a constant. The complexity of the function class can be controlled with proper regularization.

heading: 4.2 Hard Case for Feature Adaptation and Standard Self-Training

text:
To gain more insight, we study UDA in a quadratic neural network f✓, (x) = ✓>( >x) 2, where is element-wise power. In UDA, the source can have multiple solutions but we aim to learn the one working on the target [34]. We design the underlying distributions p and q in Table 6 to reflect this. Consider the following P and Q. x[1] and x[2] are sampled i.i.d. from distribution p on P , and from
Table 1: The design of p and q.
Distribution 1 +1 0 Source p 0.05 0.05 0.90 Target q 0.25 0.25 0.50
q on Q. For i 2 [3, d], x[i] = ix[2] on P and x[i] = ix[1] on Q. i 2 {±1} are i.i.d. and uniform. We also assume realizability: y = x2[1] x 2 [2] for both source and target. Note that y = x 2 [1] x 2 [i]
for all i 2 [2, d] are solutions to P but only y = x2[1] x 2 [2] works on Q. We visualize this specialized setting in Figure 4.
To make the features more tractable, we study the norm-constrained version of the algorithms (details are deferred to Section A.3.2). We compare the features learned by feature adaptation, standard selftraining, and CST. Intuitively, feature adaptation fails because the ideal target solution y = x2[1] x 2 [2] has larger distance in the feature space than other spurious solutions y = x2[1] x 2 [i]. Standard selftraining also fails since it will choose randomly among all solutions. In comparison, CST can recover the ground truth, because it can distinguish the spurious solution resulting in bad pseudo-labels. A classifier trained with those pseudo-labels cannot work on the source domain in turn. This intuition is rigorously justified in the following two theorems. Theorem 3. For ✏ 2 (0, 0.5), the following statements hold for feature adaptation and self-training: • For failure rate ⇠ > 0, and target dataset size nt > ⇥(log 1⇠ ), with probability at least 1 ⇠ over
the sampling of target data, the solution (✓̂FA, ̂FA) found by feature adaptation satisfies
ErrQ(✓̂FA, ̂FA) ✏. (14)
• With probability at least 1 1d 1 , the solution (✓̂ST, ̂ST) of standard self-training satisfies
ErrQ(✓̂ST, ̂ST) ✏. (15)
Theorem 4. For failure rate ⇠ > 0, and target dataset size nt > ⇥(log 1⇠ ), with probability at least

heading: 1 ⇠, the solution of CST (̂CST, ✓̂CST) recovers the ground truth of the target dataset:

text: ErrQ(✓̂CST, ̂CST) = 0. (16)

heading: 5 Experiments

text: We test the performance of the proposed method on both vision and language datasets. Cycle SelfTraining (CST) consistently outperforms state-of-the-art feature adaptation and self-training methods. Code is available at https://github.com/Liuhong99/CST.

heading: 5.1 Setup

text: Datasets. We experiment on visual object recognition and linguistic sentiment classification tasks: Office-Home [64] has 65 classes from four kinds of environment with large domain gap: Artistic (Ar), Clip Art (Cl), Product (Pr), and Real-World (Rw); VisDA-2017 [45] is a large-scale UDA dataset with two domains named Synthetic and Real. The datasets consist of over 200k images from 12 categories of objects; Amazon Review [10] is a linguistic sentiment classification dataset of product reviews in four products: Books (B), DVDs (D), Electronics (E), and Kitchen (K). Implementation. We use ResNet-50 [26] (pretrained on ImageNet [53]) as feature extractors for vision tasks, and BERT [16] for linguistic tasks. On VisDA-2017, we also provide results of ResNet101 to include more baselines. We use cross-entropy loss for classification on the source domain. When training the target head ✓̂t and updating the feature extractor with CST, we use squared loss to get the analytical solution of ✓̂t directly and avoid calculating second order derivatives as metalearning [18]. Details on adapting squared loss to multi-class classification are deferred to Appendix B. We adopt SGD with initial learning rate ⌘0 = 2e 3 for image classification and ⌘0 = 5e 4 for sentiment classification. Following standard protocol in [26], we decay the learning rate by 0.1 each 50 epochs until 150 epochs. We run all the tasks 3 times and report mean and deviation in top-1 accuracy. For VisDA-2017, we report the mean class accuracy. Following Theorem 2, we also enhance CST with sharpness-aware regularization [19] (CST+SAM), which help regularize the Lipschitzness of the function class. Due to space limit, we report mean accuracies in Tables 2 and 3 and defer standard deviation to Appendix C.

heading: 5.2 Baselines

text:
We compare with two lines of works in domain adaptation: feature adaptation and self-training. We also compare with more complex state-of-the-arts and create stronger baselines by combining feature adaptation and self-training.
Feature Adaptation: DANN [22], MCD [54], CDAN [37] (which improves DANN with pseudolabel conditioning), MDD [73] (which improves previous domain adaptation with margin theory), Implicit Alignment (IA) [28] (which improves MDD to deal with label shift).
Self-Training. We include VAT [40], MixMatch [8] and FixMatch [57] in the semi-supervised learning literature as self-training methods. We also compare with self-training methods for UDA: CBST [77], which considers class imbalance in standard self-training, and KLD [78], which improves CBST with label regularization. However, these methods involve tricks specified for convolutional networks. Thus, in sentiment classification tasks where we use BERT backbones, we compare with other consistency regularization baselines: VAT [40], VAT+Entropy Minimization.
Feature Adaptation + Self-Training. DIRT-T [56] combines DANN, VAT, and entropy minimization. We also create more powerful baselines: CDAN+VAT+Entropy and MDD+Fixmatch.
Other SOTA. AFN [69] boosts transferability by large norm. STAR [38] aligns domains with stochastic classifiers. SENTRY [48] selects confident examples with a committee of random augmentations.

heading: 5.3 Results

text:
Results on 12 pairs of Office-Home tasks are shown in Table 2. When domain shift is large, standard self-training methods such as VAT and FixMatch suffer from the decay in pseudo-label quality. CST outperforms feature adaptation and self-training methods significantly in 9 out of 12 tasks. Note that CST does not involve manually setting confidence threshold or reweighting.
Table 4 shows the results on VisDA-2017. CST surpasses state-of-the-arts with ResNet-50 and ResNet101 backbones. We also combine feature adaptation and self-training (DIRT-T, CDAN+VAT+entropy and MDD+FixMatch) to test if feature adaptation alleviates the negative effect of domain shift in standard self-training. Results indicate that CST is a better solution than simple combination.
While most traditional self-training methods include techniques specified for ConvNets such as Mixup [72], CST is a universal method and can directly work on sentiment classification by simply replacing the head and training objective of BERT [16]. In Table 3, most feature adaptation baselines improve over source only marginally, but CST outperforms all baselines on most tasks significantly.
5.4 Analysis
Ablation Study. We study the role of each part of CST in self-training. CST w/o Tsallis removes the Tsallis entropy LTsallis,↵. CST+Entropy replaces the Tsallis entropy with standard entropy. FixMatch+Tsallis adds LTsallis,↵ to standard self-training. Observations are shown in Table 5. CST+Entropy performs 3.7% worse than CST, indicating that Tsallis entropy is a better regularization for pseudolabels than standard entropy. CST performs 5.4% better
than FixMatch, indicating that CST is better adapted to domain shift than standard self-training. While FixMatch+Tsallis outperforms FixMatch, it is still 3.6% behind CST, with much larger total variation distance dTV between pseudo-labels and ground-truths, indicating that CST makes pseudo-labels more reliable than standard self-training under domain shift.
Quality of Pseudo-labels. We visualize the error of pseudo-labels during training on VisDA-2017 in Figure 5 (Left). The error of target classifier ✓t on the source domain decreases quickly in training, when both the error of pseudo-labels (error of ✓s on Q) and the total variation (TV) distance between pseudo-labels and ground-truths continue to decay, indicating that CST gradually refines pseudolabels. This forms a clear contrast to standard self-training as visualized in Figure 2 (Middle), where the distance dTV remains nearly unchanged throughout training.
Comparison of Gibbs entropy and Tsallis entropy. We compare the pseudo-labels learned with standard Gibbs entropy and Tsallis entropy on Ar!Cl with ResNet-50 at epoch 40. We compute the difference between the largest and the second largest softmax scores of each target example and plot the histogram in Figure 5 (Right). Gibbs entropy makes the largest softmax output close to 1, indicating over-confidence. In this case, if the prediction is wrong, it can be hard to correct it using self-training. In contrast, Tsallis entropy allows the largest and the second largest scores to be similar.

heading: 6 Related Work

text:
Self-Training. Self-training is a mainstream technique for semi-supervised learning [13]. In this work, we focus on pseudo-labeling [52, 31, 2], which uses unlabeled data by training on pseudo-labels generated by a source model. Other lines of work study consistency regularization [4, 51, 55, 40]. Recent works demonstrate the power of such methods [67, 57, 23]. Equipped with proper training techniques, these methods can achieve comparable results as standard training that uses much more labeled examples [17]. Zoph et al. [76] compare self-training to pre-training and joint training. Vu et al. [65], Mukherjee & Awadallah [42] show that task-level self-training works well in few-shot learning. These methods are tailored to semi-supervised learning or general representation learning and do not take domain shift into consideration explicitly. Wei et al. [66], Frei et al. [20] provide the first nice theoretical analysis of self-training based on the expansion assumption.
Domain Adaptation. Inspired by the generalization error bound of Ben-David et al. [7], Long et al. [34], Zellinger et al. [71] minimize distance measures between source and target distributions to learn domain-invariant features. Ganin et al. [22] (DANN) proposed to approximate the domain distance by adversarial learning. Follow-up works proposed various improvement upon DANN [63, 54, 37, 73, 28]. Popular as they are, failure cases exist in situation like label shift [74, 32], shift in support of domains [29], and large discrepancy between source and target [33]. Another line of works try to address domain adaptation with self-training. Shu et al. [56] improves DANN with VAT and entropy minimization. French et al. [21], Zou et al. [78], Li et al. [32] incorporated various semi-supervised learning techniques to boost domain adaptation performance. Kumar et al. [30], Chen et al. [15] and Cai et al. [11] showed self-training provably works in domain adaptation under certain assumptions.

heading: 7 Conclusion

text: We propose cycle self-training in place of standard self-training to explicitly address the distribution shift in domain adaptation. We show that our method provably works under the expansion assumption and demonstrate hard cases for feature adaptation and standard self-training. Self-training (or pseudolabeling) is only one line of works in the semi-supervised learning literature. Future work can delve into the behaviors of other semi-supervised learning techniques including consistency regularization and data augmentation under distribution shift, and exploit them extensively for domain adaptation.

heading: Acknowledgements

text: This work was supported by the National Natural Science Foundation of China under Grants 62022050 and 62021002, Beijing Nova Program under Grant Z201100006820041, China’s Ministry of Industry and Information Technology, the MOE Innovation Plan and the BNRist Innovation Fund.

title: Cycle Self-Training for Domain Adaptation