summaries: This paper studies the the statistical error of the Deep Ritz Method and Physics-Informed Neural Networks using neural networks and truncated Fourier basis in solving PDEs. The static Schrodinger equation is used as a prototype PDE. With appropriate assumptions, the authors established upper and lower bounds of the error for both methods. The upper bound derived in this paper improves existing results with a faster rate. The authors also proved that the upper bound of PINN is nearly optimal. Some numerical experiments are conducted to verify the results.

summaries: This paper establishes statistical lower bounds and upper bounds for (a modified) Deep Ritz method and PINNs based learning of solutions of PDEs when the estimators belong either to a class of sparse neural networks or lie in truncated fourier basis. They utilize the fact that the objective in DRM and PINNS is strongly convex, and use it to get a faster generalization bound O(1/n) instead of O(1/\sqrt{n}). Given that the upper bound for the initial non-modified version of DRM does not match the lower bound, they introduced a modified deep ritz method, where the number of samples to estimate the gradient squared is greater than (the ration is provided in the statement) rest of the objective. This enables them to achieve minimax optimality for DRM as well.   Through their experiments they verify that the number of training samples n and the test error follows a power low with \alpha = 1/d as indicated by the derived rates.

summaries: Applying deep learning (DL) to solving PDEs numerically has been a very exciting research directions. The current paper studies certain statistics properties of approximating linear Elliptic PDEs using neural networks (and truncated Fourier series). Under certain (quite strong) assumptions on the function class, the authors proved sharper (in some cases, tight)  bounds for the approximation errors.

summaries: This paper carries out a variety of studies, both theoretical and numerical, on numerical solution of a Schrodinger equation using deep learning inspired methods.  The main results are upper and lower bounds on a power law scaling for sample complexity, function of dimension and regularity, which are tight for one of the methods.  Another method (Deep Ritz) has a proposed improvement.

abstractText: In this paper, we study the statistical limits of deep learning techniques for solving elliptic partial differential equations (PDEs) from random samples using the Deep Ritz Method (DRM) and Physics-Informed Neural Networks (PINNs). To simplify the problem, we focus on a prototype elliptic PDE: the Schrödinger equation on a hypercube with zero Dirichlet boundary condition, which is applied in quantummechanical systems. We establish upper and lower bounds for both methods, which improve upon concurrently developed upper bounds for this problem via a fast rate generalization bound. We discover that the current Deep Ritz Method is sub-optimal and propose a modified version of it. We also prove that PINN and the modified version of DRM can achieve minimax optimal bounds over Sobolev spaces. Empirically, following recent work which has shown that the deep model accuracy will improve with growing training sets according to a power law, we supply computational experiments to show similar-behavior of dimension dependent power law for deep PDE solvers.

authors:
Genming Bai
Ujjwal Koley
Siddhartha Mishra
Roberto Molinaro

title: Physics informed neural networks (pinns) for approximating nonlinear dispersive pdes

venue: arXiv preprint arXiv:2104.05584,

authors:
Peter L Bartlett
Olivier Bousquet
Shahar Mendelson

title: Local rademacher complexities

venue: The Annals of Statistics,

authors:
Kaushik Bhattacharya
Bamdad Hosseini
Nikola B Kovachki
Andrew M Stuart

title: Model reduction and neural networks for parametric pdes

venue: arXiv preprint arXiv:2005.03180,

title: Functional analysis, Sobolev spaces and partial differential equations

venue: Springer Science & Business Media,

authors:
Lawrence D Brown
Mark G Low

title: Asymptotic equivalence of nonparametric regression and white noise

authors:
Jingrun Chen
Rui Du
Keke Wu

title: A comprehensive study of boundary conditions when solving pdes by dnns

venue: arXiv preprint arXiv:2005.04554,

authors:
Minshuo Chen
Haoming Jiang
Wenjing Liao
Tuo Zhao

title: Efficient approximation of deep relu networks for functions on low dimensional manifolds

venue: Advances in Neural Information Processing Systems,

title: Nonparametric regression on lowdimensional manifolds using deep relu networks

venue: arXiv preprint arXiv:1908.01842,

authors:
Yian Chen
Jeremy Hoskins
Yuehaw Khoo
Michael Lindsey

title: Committor functions via tensor networks

venue: arXiv preprint arXiv:2106.12515,

authors:
Yifan Chen
Bamdad Hosseini
Houman Owhadi
Andrew M Stuart

title: Solving and learning nonlinear pdes with gaussian processes

venue: arXiv preprint arXiv:2103.12959,

title: Convergence rates of gradient methods for convex optimization in the space of measures

venue: arXiv preprint arXiv:2105.08368,

title: A practical guide to splines, volume 27

venue: springer-verlag New York,

authors:
Giacomo De Palma
Bobak Toussi Kiani
Seth Lloyd

title: Random deep neural networks are biased towards simple functions

venue: arXiv preprint arXiv:1812.10156,

authors:
Chenguang Duan
Yuling Jiao
Yanming Lai
Xiliang Lu
Zhijian Yang

title: Convergence rate analysis for deep ritz method

venue: arXiv preprint arXiv:2103.13330,

authors:
Weinan E
Bing Yu

title: The deep ritz method: a deep learning-based numerical algorithm for solving variational problems

venue: Communications in Mathematics and Statistics,

title: Partial differential equations

venue: Graduate studies in mathematics,

authors:
Yuwei Fan
Lexing Ying

title: Solving electrical impedance tomography with deep learning

venue: Journal of Computational Physics,

authors:
Max H Farrell
Tengyuan Liang
Sanjog Misra

title: Deep neural networks for estimation and inference

authors:
Jordi Feliu-Faba
Yuwei Fan
Lexing Ying

title: Meta-learning pseudo-differential operators with deep neural networks

authors:
Maximilien Germain
Huyên Pham
Xavier Warin

title: Neural networks-based algorithms for stochastic control and pdes in finance

venue: arXiv preprint arXiv:2101.08068,

authors:
Davis Gilton
Greg Ongie
Rebecca Willett

title: Neumann networks for linear inverse problems in imaging

venue: IEEE Transactions on Computational Imaging,

authors:
Philipp Grohs
Lukas Herrmann

title: Deep neural network approximation for high-dimensional elliptic pdes with boundary conditions

venue: arXiv preprint arXiv:2007.05384,

authors:
Philipp Grohs
Fabian Hornung
Arnulf Jentzen
Philippe Von Wurstemberger

title: A proof that artificial neural networks overcome the curse of dimensionality in the numerical approximation of black-scholes partial differential equations

venue: arXiv preprint arXiv:1809.02362,

authors:
Ingo Gühring
Gitta Kutyniok
Philipp Petersen

title: Error bounds for approximations with deep relu neural networks in w s, p norms

venue: Analysis and Applications,

authors:
Jiequn Han
Arnulf Jentzen
E Weinan

title: Solving high-dimensional partial differential equations using deep learning

venue: Proceedings of the National Academy of Sciences,

authors:
Jiequn Han
Jianfeng Lu
Mo Zhou

title: Solving high-dimensional eigenvalue problems using deep neural networks: A diffusion monte carlo like approach

title: Predicting the impact of dataset composition on model performance, 2021

authors:
Kaiming He
Xiangyu Zhang
Shaoqing Ren
Jian Sun

title: Delving deep into rectifiers: Surpassing human-level performance on imagenet classification

venue: In Proceedings of the IEEE international conference on computer vision,

authors:
James B Heaton
Nick G Polson
Jan Hendrik Witte

title: Deep learning for finance: deep portfolios

venue: Applied Stochastic Models in Business and Industry,

authors:
Joel Hestness
Sharan Narang
Newsha Ardalani
Gregory Diamos
Heewoo Jun
Hassan Kianinejad
Md Patwary
Mostofa Ali
Yang Yang
Yanqi Zhou

title: Deep learning scaling is predictable, empirically

venue: arXiv preprint arXiv:1712.00409,

authors:
Jan-Christian Hütter
Philippe Rigollet

title: Minimax rates of estimation for smooth optimal transport

venue: maps. arXiv preprint arXiv:1905.05828,

authors:
Masaaki Imaizumi
Kenji Fukumizu

title: Advantage of deep neural networks for estimating functions with singularity on curves

venue: arXiv preprint arXiv:2011.02256,

authors:
Yuling Jiao
Yanming Lai
Dingwei Li
Xiliang Lu
Yang Wang
Jerry Zhijian Yang

title: Convergence analysis for the pinns, 2021a

authors:
Yuling Jiao
Yanming Lai
Yisu Luo
Yang Wang
Yunfei Yang

title: Error analysis of deep ritz methods for elliptic equations

venue: arXiv preprint arXiv:2107.14478,

authors:
Yuling Jiao
Guohao Shen
Yuanyuan Lin
Jian Huang

title: Deep nonparametric regression on approximately low-dimensional manifolds

venue: arXiv preprint arXiv:2104.06708,

authors:
Jared Kaplan
Sam McCandlish
Tom Henighan
Tom B Brown
Benjamin Chess
Rewon Child
Scott Gray
Alec Radford
Jeffrey Wu
Dario Amodei

title: Scaling laws for neural language models

authors:
Yuehaw Khoo
Jianfeng Lu
Lexing Ying

title: Solving parametric pde problems with artificial neural networks

venue: arXiv preprint arXiv:1707.03351,

title: Oracle Inequalities in Empirical Risk Minimization and Sparse Recovery Problems: Ecole d’Eté de Probabilités de Saint-Flour XXXVIII-2008, volume 2033

authors:
Samuel Lanthaler
Siddhartha Mishra
George Em Karniadakis

title: Error estimates for deeponets: A deep learning framework in infinite dimensions

venue: arXiv preprint arXiv:2102.09618,

authors:
Michel Ledoux
Michel Talagrand

title: Probability in Banach Spaces: isoperimetry and processes

authors:
Zongyi Li
Nikola Kovachki
Kamyar Azizzadenesheli
Burigede Liu
Kaushik Bhattacharya
Andrew Stuart
Anima Anandkumar

title: Fourier neural operator for parametric partial differential equations

venue: arXiv preprint arXiv:2010.08895,

authors:
Youming Liu
Huiying Wang

title: Convergence order of wavelet thresholding estimator for differential operators on besov spaces

venue: Applied and Computational Harmonic Analysis,

authors:
Zichao Long
Yiping Lu
Xianzhong Ma
Bin Dong

title: Pde-net: Learning pdes from data

venue: In International Conference on Machine Learning,

authors:
Zichao Long
Yiping Lu
Bin Dong

title: Pde-net 2.0: Learning pdes from data with a numericsymbolic hybrid deep network

authors:
Denghui Lu
Han Wang
Mohan Chen
Lin Lin
Roberto Car
E Weinan
Weile Jia
Linfeng Zhang

title: pflops deep potential molecular dynamics simulation of 100 million atoms with ab initio accuracy

venue: Computer Physics Communications,

authors:
Jianfeng Lu
Yulong Lu
Min Wang

title: A priori generalization analysis of the deep ritz method for solving high dimensional elliptic equations

venue: arXiv preprint arXiv:2101.01708,

authors:
Tao Luo
Haizhao Yang

title: Two-layer neural networks for partial differential equations: Optimization and generalization theory

venue: arXiv preprint arXiv:2006.15733,

authors:
Tudor Manole
Sivaraman Balakrishnan
Jonathan Niles-Weed
Larry Wasserman

title: Plugin estimation of smooth optimal transport maps

venue: arXiv preprint arXiv:2107.12364,

authors:
Tanya Marwah
Zachary C Lipton
Andrej Risteski

title: Parametric complexity bounds for approximating pdes with neural networks

venue: arXiv preprint arXiv:2103.02138,

authors:
Hiroaki Mikami
Kenji Fukumizu
Shogo Murai
Shuji Suzuki
Yuta Kikuchi
Taiji Suzuki
Shin-ichi Maeda
Kohei Hayashi

title: A scaling law for synthetic-to-real transfer: A measure of pre-training

venue: arXiv preprint arXiv:2108.11018,

authors:
Hadrien Montanelli
Qiang Du

title: New error bounds for deep relu networks using sparse grids

venue: SIAM Journal on Mathematics of Data Science,

authors:
Hans-Georg Müller
Theo Gasser

title: Optimal convergence properties of kernel estimates of derivatives of a density function

venue: In Smoothing techniques for curve estimation,

authors:
Richard Nickl
Sara van de Geer
Sven Wang

title: Convergence rates for penalized least squares estimators in pde constrained regression problems

venue: SIAM/ASA Journal on Uncertainty Quantification,

authors:
Atsushi Nitanda
Taiji Suzuki

title: Optimal rates for averaged stochastic gradient descent under neural tangent kernel regime

venue: arXiv preprint arXiv:2006.12297,

authors:
Greg Ongie
Rebecca Willett
Daniel Soudry
Nathan Srebro

title: A function space view of bounded norm infinite width relu nets: The multivariate case

title: Nonparametric estimation of the derivatives of a density by the method of wavelets

venue: Bulletin of Informatics and Cybernetics,

authors:
Maziar Raissi
Paris Perdikaris
George E Karniadakis

title: Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations

authors:
Alexander Rakhlin
Karthik Sridharan
Alexandre B Tsybakov

title: Empirical entropy, minimax regret and minimax risk

authors:
Lorenz Richter
Leon Sallandt
Nikolas Nüsken

title: Solving high-dimensional parabolic pdes using the tensor train format

venue: arXiv preprint arXiv:2102.11830,

title: Scaling laws for deep learning

venue: arXiv preprint arXiv:2108.07686,

authors:
Jonathan S Rosenfeld
Amir Rosenfeld
Yonatan Belinkov
Nir Shavit

title: A constructive prediction of the generalization error across scales

title: Nonparametric regression using deep neural networks with relu activation function

venue: Annals of Statistics,

title: Spline functions: basic theory

authors:
Utkarsh Sharma
Jared Kaplan

title: A neural scaling law from the dimension of the data manifold

venue: arXiv preprint arXiv:2004.10802,

authors:
Zuowei Shen
Haizhao Yang
Shijun Zhang

title: Neural network approximation: Three hidden layers are enough

venue: Neural Networks,

authors:
Yeonjong Shin
Zhongqiang Zhang
George Em Karniadakis

title: Error estimates of residual minimization using neural networks for linear pdes

venue: arXiv preprint arXiv:2010.08019,

authors:
Justin Sirignano
Konstantinos Spiliopoulos

title: Dgm: A deep learning algorithm for solving partial differential equations

venue: Journal of computational physics,

authors:
Nathan Srebro
Karthik Sridharan
Ambuj Tewari

title: Smoothness, low noise and fast rates

venue: Advances in neural information processing systems,

title: Adaptivity of deep relu network for learning in besov and mixed smooth besov spaces: optimal rate and curse of dimensionality

venue: arXiv preprint arXiv:1810.08033,

authors:
Taiji Suzuki
Shunta Akiyama

title: Benefit of deep learning with non-convex noisy gradient descent: Provable excess risk bound and superiority to kernel methods

venue: arXiv preprint arXiv:2012.03224,

authors:
Taiji Suzuki
Atsushi Nitanda

title: Deep learning is adaptive to intrinsic dimensionality of model smoothness in anisotropic besov space

title: An introduction to matrix concentration inequalities

venue: arXiv preprint arXiv:1501.01571,

title: Introduction to nonparametric estimation

title: A new approach to least-squares estimation, with applications

title: Exponential convergence of the deep neural network approximation for analytic functions

venue: arXiv preprint arXiv:1807.00297,

title: Some observations on partial differential equations in barron and multi-layer spaces

venue: arXiv preprint arXiv:2012.01484,

title: The finite neuron method and convergence analysis

venue: arXiv preprint arXiv:2010.01458,

authors:
Yunbei Xu
Assaf Zeevi

title: Towards optimal problem dependent generalization error bounds in statistical learning theory

venue: arXiv preprint arXiv:2011.06186,

title: Error bounds for approximations with deep relu networks

authors:
Yaohua Zang
Gang Bao
Xiaojing Ye
Haomin Zhou

title: Weak adversarial networks for highdimensional partial differential equations

authors:
Xiaoshuai Zhang
Yiping Lu
Jiaying Liu
Bin Dong

title: Dynamically unfolding recurrent restorer: A moving endpoint control method for image restoration

venue: arXiv preprint arXiv:1805.07709,

heading: 1 INTRODUCTION

text:
Partial differential equations (PDEs) play a prominent role in many disciplines of science and engineering. The recent deep learning breakthrough and the rapid development of sensors, computational power, and data storage in the past decade has drawn attention to numerically solving PDEs via machine learning methods (Long et al., 2018; 2019; Raissi et al., 2019; Han et al., 2018; Sirignano & Spiliopoulos, 2018; Khoo et al., 2017), especially in high dimensions where conventional methods become impractical. The set of applications that motivate this interest is wide-ranging, including computational physics (Han et al., 2018; Long et al., 2018; Raissi et al., 2019), inverse problem (Zhang et al., 2018; Gilton et al., 2019; Fan & Ying, 2020) and quantitative finance (Heaton et al., 2017; Germain et al., 2021). The numerical methods generated by the use of deep learning techniques are mesh-less methods, see the discussion in (Xu, 2020). A natural deep learning technique in the
problems that are based on a standard feed-forward type of architecture takes advantage (when available) of a variational formulation, whose solution coincides with the solution of the PDE of interest. Despite the success and popularity of adopting neural networks for solving high-dimensional PDEs, the following question still remains poorly answered.
For a given PDE and a data-driven approximation architecture, how large the sample size and how complex the model are needed to reach a prescribed performance level?
In this paper, we aim to establish the numerical analysis of such deep learning based PDE solvers. Inspired by recent works which showed that the empirical performance of a model is remarkably predictable via a power law of the data number, known as the neural scaling law (Kaplan et al., 2020; Hestness et al., 2017; Sharma & Kaplan, 2020), we aim to explore the neural scaling law for deep PDE solvers and compare its performance to Fourier approximation.
Among the various approaches of using deep learning methods for solving PDEs, in this work, we focus on the Deep Ritz method (DRM) (E & Yu, 2018; Khoo et al., 2017) and the Physics-Informed Neural Networks (PINN) approach (Sirignano & Spiliopoulos, 2018; Raissi et al., 2019), both of which are based on minimizing neural network parameters according to some loss functional related to the PDEs. To provide theoretical guarantees for DRM and PINN, following (Lu et al., 2021b; Duan et al., 2021; Bai et al., 2021), we decompose the error into approximation error (Yarotsky, 2017; Suzuki, 2018; Shen et al., 2021) and generalization error (Bartlett et al., 2005; Xu & Zeevi, 2020; Farrell et al., 2021; Schmidt-Hieber et al., 2020; Suzuki, 2018). However, instead of the O(1/ √ n) (n is the number of data sampled) slow rate generalization bounds established in prior work (Lu et al., 2021b; Shen et al., 2021; Xu, 2020; Shin et al., 2020), we utilize the strongly convex structure of the DRM and PINN objectives and provide an O(1/n) fast rate generalization bound (Bartlett et al., 2005; Xu & Zeevi, 2020) that leads us to a non-parametric estimation bound. Our theory also suggests an optimal selection of network size with respect to the number of sampled data. Moreover, to illustrate the optimality of our upper bound, we also establish an information-theoretic lower bound which matches our upper bound for PINN and a modified version of DRM.
We also test our theory by numerical experiments. Recent works (Hestness et al., 2017; Kaplan et al., 2020; Rosenfeld et al., 2019; Mikami et al., 2021) studying a variety of deep learning algorithms all find the same polynomial scaling relation between the testing error and the number of data. As the number of training data n increases, the population loss L of well-trained and well-tuned models scales with n as a power-law L ∝ 1nα for some α. (Sharma & Kaplan, 2020) also scans over a large range of α and problem dimension d and finds an approximately α ∝ 1d scaling law. In Section 4, we conduct numerical experiments to show that this phenomenon still appears for deep PDE solvers and this neural scaling law tests more idiosyncratic features of the theory.

heading: 1.1 RELATED WORKS

text:
Neural Scaling Law The starting point of our work is the recent observation across speech, vision and text (Hestness et al., 2017; Kaplan et al., 2020; Rosenfeld et al., 2019; Rosenfeld, 2021) that the empirical performance of a model satisfies a power law scales as a power-law with model size and dataset size. (Sharma & Kaplan, 2020) further finds out that the power of the scaling law depends on the intrinsic dimension of the dataset. Theoretical works (Schmidt-Hieber et al., 2020; Suzuki, 2018; Suzuki & Nitanda, 2019; Chen et al., 2019b; Imaizumi & Fukumizu, 2020; Farrell et al., 2021; Jiao et al., 2021c) explore the optimal power law under the non-parametric curve estimation setting via a plug-in neural network. Our work extends this line of research to solving PDEs.
Deep Network Based PDE Solver. Solving high dimensional partial differential equations (PDEs) has been a long-standing challenge due to the curse of dimensionality. At the same time, deep learning has shown superior flexibility and adaptivity in approximating high dimensional functions, which leads to state-of-the-art performances in a wide range of tasks ranging from computer vision to natural language processing. Recent years, pioneer works (Han et al., 2018; Raissi et al., 2019; Long et al., 2018; Sirignano & Spiliopoulos, 2018; Khoo et al., 2017) try to utilize the deep neural networks to solve different types of PDEs and achieve impressive results in many tasks (Lu et al., 2021a; Li et al., 2020). Based on the natural idea of representing solutions of PDEs by (deep) neural networks, different loss functions for solving PDEs are proposed. (Han et al., 2018; 2020) utilize the
Feynman-Kac formulation which turns solving PDE to a stochastic control problem and the weak adversarial network (Zang et al., 2020) solves the weak formulations of PDEs via an adversarial network. In this paper, we focus on the convergence rate of the Deep Ritz Method (DRM) (E & Yu, 2018; Khoo et al., 2017) and Physics-Informed neural network (PINN) (Raissi et al., 2019; Sirignano & Spiliopoulos, 2018). DRM (E & Yu, 2018; Khoo et al., 2017) utilizes the variational structure of the PDE, which is similar to the Ritz-Galerkin method in classical numerical analysis of PDEs, and trains a neural network to minimize the variational objective. PINN (Raissi et al., 2019; Sirignano & Spiliopoulos, 2018) trains a neural network directly to minimize the residual of the PDE, i.e., using the strong form of the PDE.
Theoretical Guarantees For Machine Learning Based PDE Solvers. Theoretical convergence results for deep learning based PDE solvers raises wide interest recently. Specifically, (Lu et al., 2021b; Grohs & Herrmann, 2020; Marwah et al., 2021; Wojtowytsch et al., 2020; Xu, 2020; Shin et al., 2020; Bai et al., 2021) investigate the regularity of PDEs approximated by neural network and (Lu et al., 2021b; Luo & Yang, 2020) further provide a generalization analysis. (Nickl et al., 2020) introduces a prior over the solution of the PDE and considers an equivalent white noise model (Brown & Low, 1996). (Nickl et al., 2020) provides the rate of convergence of the posterior. Our paper does not need to introduce the prior on the target function and provides a non-asymptotic guarantee for finite number of data. At the same time, (Nickl et al., 2020) can only be applied to linear PDEs while our proof technique can be extended to nonlinear ones. All these papers also fail to answer the question that how to determine the network size corresponding to the sampled data number to achieve a desired statistical convergence rate. (Hütter & Rigollet, 2019; Manole et al., 2021) consider the similar problem for the optimal transport problem, i.e. Monge-ampere equation. Nevertheless, the variational problem we considered is different from (Hütter & Rigollet, 2019; Manole et al., 2021) and leads to technical difference. The most related works to ours are two concurrent papers (Duan et al., 2021; Jiao et al., 2021a;b). However, our upper bound is faster than (Duan et al., 2021; Jiao et al., 2021a;b). In this paper, we also show that generalization analysis in (Lu et al., 2021b; Duan et al., 2021; Luo & Yang, 2020) are loose due to the lack of a localization technique (De Boor & De Boor, 1978; Bartlett et al., 2005; Koltchinskii, 2011; Xu, 2020). With observation of the strong convexity of the loss function, we follow the fast rate results for ERM (Schmidt-Hieber et al., 2020; Xu & Zeevi, 2020; Farrell et al., 2021) and provide a near optimal bound for both DRM and PINN.

heading: 1.2 CONTRIBUTION

text:
In short, we summarize our contribution as follows
• In this paper, we first considered the statistical limit of learning a PDE solution from sampled observations. The lower bound shows a non-standard exponent different from non-parametric estimation of a function. • Instead of the O(1/ √ n) slow rate generalization bounds in (Lu et al., 2021b; Duan et al.,
2021; Jiao et al., 2021a;c), we utilized the strongly convex nature of the variational form and provided a fast rate generalization bound via the localization methods (Van De Geer, 1987; Bartlett et al., 2005; Koltchinskii, 2011; Srebro et al., 2010; Xu & Zeevi, 2020). We discovered that the current Deep Ritz Methods is sub-optimal and propose a modified version of it. We showed that PINN and the modified version of DRM can achieve nearly min-max optimal convergence rate. Our result is listed in Table 1.
• We tested the recently discovered neural scaling law (Hestness et al., 2017; Kaplan et al., 2020; Rosenfeld et al., 2019; Hashimoto, 2021) for deep PDE solvers numerically. The empirical results verified our theory.

heading: 2 SET-UP

text:
We consider the static Schrödinger equation with zero Dirichlet boundary condition on the domain Ω, which we assume to be the unit hypercube in Rd. In order to precisely introduce the problem, we recall some standard notions. We consider our domain as Ω = [0, 1]d and use L2(Ω) to denote the space of square integrable functions on Ω with respect to the Lebesgue measure. We let L∞(Ω) be
the space of essentially bounded (with respect to the Lebesgue measure) functions on Ω and C(∂Ω) denotes the space of continuous functions on ∂Ω.
Let f ∈ L2(Ω), V ∈ L∞(Ω), and , g ∈ L∞(Ω). Our focus is on the analysis of Deep-Learning-based numerical methods to solve the elliptic PDE:
−∆u+ V u = f in Ω, u = g on ∂Ω.
(2.1)

heading: 2.1 LOSS FUNCTIONS FOR SOLVING PDES AND INDUCED EVALUATION METRIC

text:
In this paper, we mainly focus on analyzing Deep Ritz Methods (DRM) and Physics-Informed Neural Network (PINN). In this subsection, we first introduce the objective function and algorithm of the two methods.
Deep Ritz Methods (E & Yu, 2018; Sirignano & Spiliopoulos, 2018) Recall that the equation 2.1 is equivalent to following variational form
u∗ = arg min H10 (Ω) EDRM(u) := arg min H10 (Ω)
1
2 ∫ Ω ‖∇u‖2 + V |u|2 dx− ∫ Ω fudx, (2.2)
where u is minimized over H10 (Ω) with boundary condition given by g on ∂Ω.
This variational form provides the basis for the DRM type method for solving the static Schrödinger equation based on neural network ansatz. More specifically, the energy functional given in equation 2.2 is viewed as the population risk function to train an optimal estimator approximation of the solution to the PDE within a parameterized hypothesis function class F ⊂ H10 (Ω). In this paper, we also rely on the strong convexity of the DRM objective with respect to the H1 norm.
Proposition 2.1. For DRM, we further assume 0 < Vmin ≤ V (x) ≤ Vmax, then we have
2 max{1, Vmax} ( EDRM(u)−EDRM(u∗) ) ≤ ‖u−u∗‖2H1 ≤
2 max{1, Vmin} ( EDRM(u)−EDRM(u∗) ) holds for all u ∈ H10 (Ω)
Physics-Informed Neural Network (Raissi et al., 2019; Sirignano & Spiliopoulos, 2018). PINN solves 2.1 via minimizing the following objective function
u∗ = arg min H10 (Ω) EPINN(u) := arg min H10 (Ω) ∫ Ω |∆u(x)− V (x)u(x) + f(x)|2dx.
The objective function EPINN can also be viewed as the population risk function and we can train an optimal estimator approximation of the solution to the PDE within a parameterized hypothesis function class F ⊂ H10 (Ω). In this paper, we also rely on the strong convexity of the PINN objective with respect to the H2 norm, for which we need some additional assumptions on the potential.
Proposition 2.2. For PINN, we further assume V ∈ L∞(Ω) with 0 < Cmin < V 2 − ∆V, 0 < Cmin < V (x) ≤ Vmax and −∆V (x) ≤ Vmax, then we have for all u ∈ H10 (Ω)
1
2 (1 + Vmax + V 2max)
( EPINN(u)−EPINN(u∗) ) ≤ ‖u− u∗‖2H2
≤ 2 max{1, Cmin}
( EPINN(u)−EPINN(u∗) ) .

heading: 2.2 ESTIMATOR SETTING

text:
Empirical Loss Minimization In order to access the d-dimensional integrals, DRM (E & Yu, 2018; Khoo et al., 2017) and PINN (Raissi et al., 2019; Sirignano & Spiliopoulos, 2018) employ a Monte-Carlo method for computing the high dimensional integrals, which leads to the so-called empirical risk minimization training for neural networks. To define the empirical loss, let {Xj}nj=1 be an i.i.d. sequence of random variables distributed according to the uniform distribution PΩ in domain Ω. We also have access to noisy observations fj = f(Xj) + ξj (1 ≤ j ≤ n) of the right hand side of the PDE (2.1), where ξj are i.i.d. bounded random variables with zero mean and independent of Xj . Define the empirical losses En by setting
EDRMn (u) = 1
n n∑ j=1 [ |Ω| · (1 2 ‖∇u(Xj)‖2 + 1 2 V (Xj)|u(Xj)|2 − fju(Xj) )] , (2.3)
EPINNn (u) = 1
n n∑ j=1 [ |Ω| · ( ∆u(Xj)− V (Xj)u(Xj) + fj )2] , (2.4)
where |Ω| represents the Lebesgue measure of the domain. Once given an empirical loss En, we apply the empirical loss minimization to seek the estimation un, i.e. un = arg minu∈F En(u), where F is the parametrized hypothesis function space we consider. Some examples can be reproducing kernel Hilbert space (Chen et al., 2021b) and tensor training format (Richter et al., 2021; Chen et al., 2021a). In this paper, we consider sparse neural network and truncated Fourier basis, which can achieve min-max optimal estimation rate for the non-parametric function estimation (Tsybakov, 2008; Schmidt-Hieber et al., 2020; Farrell et al., 2021; Suzuki, 2018; Chen et al., 2019b; Jiao et al., 2021c; Nitanda & Suzuki, 2020).
Sparse Neural Network Function Space In this paper, the hypothesis function space F is expressed by the neural network function space following (Schmidt-Hieber et al., 2020; Suzuki, 2018; Farrell et al., 2021). Let us denote the ReLU3 activation by η3(x) = max{x3, 0} (x ∈ R), which is used in (E & Yu, 2018). For a vector x, η(x) is operated in an element-wise manner. Define the space of all neural networks with height L, width W , sparsity constraint S and norm constraint B as
Φ(L,W,S,B) := {(W(L)η3(·) + b(L)) ◦ · · · (W(2)η3(·) + b(2)) ◦ (W(1)x+ b(1)) | W(L) ∈ R1×W , b(L) ∈ R,W(1) ∈ RW×d, b(1) ∈ RW ,W(l) ∈ RW×W , b(l) ∈ RW (1 < l < L), L∑ l=1 (‖W(l)‖0 + ‖b(l)‖0) ≤ S,max l ‖W(l)‖∞,∞ ∨ ‖b(l)‖∞ ≤ B}, (2.5)
where ◦ denotes the function composition, ‖ · ‖0 is the `0-norm of the matrix (the number of non-zero elements of the matrix) and ‖ · ‖∞,∞ is the `∞-norm of the matrix (maximum of the absolute values of the elements).
Truncated Fourier Basis Estimator We also consider the Truncated Fourier basis as our estimator. Denote the domain we are interested in by Ω ⊆ [0, 1]d. For any z ∈ Nd, we consider the corresponding Fourier basis function φz(x) := e2πi〈z,x〉 (x ∈ Ω). Any function f ∈ L2(Ω) can be represented as a weighted sum of the Fourier basis f(x) := ∑ z∈Nd fzφz(x), where
fz := ∫ Ω f(x)φz(x)dx (∀ z ∈ Nd) is the Fourier coefficient. This inspires us to use the Fourier basis whose index lies in a truncated set Zξ = {z ∈ Z|‖z‖∞ ≤ ξ} to represent the function class F as Fξ = { ∑ ‖z‖∞≤ξ azφz|az ∈ R, ‖z‖∞ ≤ ξ}.

heading: 3 LOWER BOUND

text:
In this section, we aim to consider the statistical limit of learning the solution of a PDE. As discussed in Propositions 2.1 and 2.2, we directly consider the H1 norm for DRM and H2 norm for PINN as the evaluation metric. The lower bounds are shown as follows. Theorem 3.1 (Lower bound). We denote u∗(f) to be the solution of the PDE 2.1 and we can access randomly sampled data {Xi, fi}i=1,··· ,n as described in Section 2.2. We further assume u∗(f) ∈ Hs for a given s ∈ Z+. Then we have the following lower bounds. DRM Lower Bound. For all estimators ψ : ( Rd )⊗n × R⊗n → Hs(Ω), we have
inf ψ sup u∗∈Hs(Ω)
E‖ψ({Xi, fi}i=1,··· ,n)− u∗(f)‖2H1 & n − 2s−2d+2s−4 . (3.1)
PINN Lower Bound. For all estimators ψ : ( Rd )⊗n × R⊗n → Hs(Ω), we have
inf ψ sup u∗∈Hs(Ω)
E‖ψ({Xi, fi}i=1,··· ,n)− u∗(f)‖2H2 & n − 2s−4d+2s−4 . (3.2)
Given that n− 2(β−k) d+2β is the minimax rate of estimating the k-th derivative of a β-smooth density in L2 (Liu & Wang, 2012; Prakasa Rao, 1996; Müller & Gasser, 1979), the lower bound obtained here is the rate of estimating the right hand side function f in terms of the H−1 norm. Given the H−1 norm error estimate on f , we can achieve an estimate of u, which provides an alternative way to understand our upper bound. (See discussion in Appendix E.) The lower bound is non-standard, for the 2s− 2 in the numerator is different from the 2s− 4 in the denominator.

heading: 4 UPPER BOUND

text:
To theoretically understand the empirical success of Physics-Informed Neural Networks and the Deep Ritz solver, in this section, we aim to prove that the excess risk ∆En := E(un)−E(u∗) of a well-trained neural networks on the PINN/DRM loss function will follow precise power-law scaling relations with the size of the training dataset. Similar to (Xu, 2020; Lu et al., 2021b; Duan et al., 2021; Jiao et al., 2021a;b), we decompose the excess risk into approximation error and generalization error. Different from the concurrent bound (Duan et al., 2021; Jiao et al., 2021a), we provide a fast rate O(1/n) by utilizing the strong convexity of the objective function established in Section 2.1 and achieve a faster and near optimal upper bound. We show that the generalization error can be bounded by the fixed point (i.e. the solution of φ(r) = r) of the local Rademacher complexity
ψ(r) = Rn({I(u) | ‖u− u∗‖2A ≤ r}), where Rn is the Rademacher complexity, I(u) = ∆u + V u, ‖ · ‖A = ‖ · ‖H2 for PINN and I(u) = ‖∇u‖2 +V u, ‖ · ‖A = ‖ · ‖H1 for DRM. We put detailed definition and analysis in Appendix B.4. We first provide a meta theorem to decompose the error into approximation and a fast rate generalization error. Then we plug in the approximation and generalization error calculated in Appendix B.3 and Appendix B.2 and finally achieve the following upper bounds. We also put a more detailed proof sketch in Appendix A.2.
Physics Informed Neural Network. Theorem 4.1. (Informal Upper Bound of PINN with Deep Neural Network Estimator) With proper assumptions, consider the sparse Deep Neural Network function space Φ(L,W,S,B) with parameters L = O(1), W = O(n d d+2s−4 ), S = O(n d d+2s−4 ), B = O(n d d+2s−4 ), then the Physics Informed estimator ûDNNPINN = minu∈Φ(L,W,S,B) E PINN n (u) satisfies the following upper bound with high probability
‖ûDNNPINN − u∗‖2H2 . n − 2s−4d+2s−4 log n.
Theorem 4.2. (Informal Upper Bound of PINN with Truncated Fourier Series Estimator) With proper assumptions, consider the Physics Informed Neural Network objective with a plug-in Fourier Series estimator ûFourierPINN = minu∈Fξ(Ω) E PINN n (u) with ξ = Θ(n 1 d+2s−4 ), then with high probability we have
‖ûFourierPINN − u∗‖2H2 . n − 2s−4d+2s−4 .

heading: Deep Ritz Methods.

text:
Theorem 4.3. (Informal Upper Bound of DRM with Deep Neural Network Estimator) With proper assumptions, consider the sparse Deep Neural Network function space Φ(L,W, S,B) with parameters L = O(1), W = O(n d d+2s−2 ), S = O(n d d+2s−2 ), B = O(n d d+2s−2 ), then the Deep ritz estimator ûDNNDRM = minu∈Φ(L,W,S,B) E DRM n (u) satisfies the following upper bound with high probability
‖ûDNNDRM − u∗‖2H1 . n − 2s−2d+2s−2 log n.
Theorem 4.4. (Informal Upper Bound of DRM with Truncated Fourier Series Estimator) With proper assumptions, consider the Deep Ritz objective with a plug in Fourier Series estimator ûFourierDRM = minu∈Fξ(Ω) E DRM n (u) with ξ = Θ(n 1 d+2s−2 ), then with high probability we have
‖ûFourierDRM − u∗‖2H1 . n − 2s−2d+2s−2 .

heading: Remark.

text:
• There is a common belief that Machine learning based PDE solvers can break the curse of dimensionality (E & Yu, 2018; Grohs et al., 2018; Lanthaler et al., 2021). However, we obtained an n− 2s−2 2s−4+d convergence rate, which can become super slow in high dimension.
Our analysis showed that it is essential to constrain the function space to break the curse of dimensionality. (Lu et al., 2021b) considered the DRM in Barron spaces. (Ongie et al., 2019) showed that functions in the Barron space enjoy a smoothness s at the same magnitude as d , which will also lead to convergence rate independent of the dimension using our upper bound. Neural network can also approximate mixed sparse grid spaces (Montanelli & Du, 2019; Suzuki, 2018) and functions on manifold (Nitanda & Suzuki, 2020; Chen et al., 2019b) without curse of dimensionality. Combined with these approximation bounds, we can also achieve a bound that breaks the curse of dimensionality using Theorem B.12 and B.9. In this paper, we aim to consider the statistical power of the loss function in common function spaces and leave the curse of dimensionality as a separate topic.
• Our bound is faster than the concurrent bound (Duan et al., 2021; Jiao et al., 2021a) for we provided a fast rate O(1/n) by utilizing the strong convexity of the objective function and improved the convergence rate from n− 2s−2 d+4s−4 to n− 2s−2 d+2s−2 for Deep Ritz and from
n− 2s−4 d+4s−8 to n− 2s−4 d+2s−4 for PINN. Compared to the lower bound provided in Section 3, our bounds for PINN is near-optimal while the upper bound for DRM is sub-optimal. We believe our bound is tight and put the discussion in Appendix E. We’ll propose a modified version of DRM to match the upper and lower bound in the next section.
• For upper bound of DRM, due to a technical issue, we assumed that the observation we access is clean, i.e fi = f(Xi). We conjecture that add noising on observation will not effect the rate and leave this to future work.

heading: 5 MODIFIED DEEP RITZ METHODS

text:
Comparing the lower bound in Section 3 and the upper bound in Section 4, we find out that the Physics-Informed Neural Network achieves min-max optimality while the Deep Ritz Method does not. In this section, we propose a modified version of Deep Ritz which can be statistically optimal.
As discussed in Appendix E, the reason behind the suboptimality of DRM comes from the high complexity introduced via the uniform concentration bound of the gradient term in the variational form. At the same time, we further observed that the ∫ ‖∇u‖2dx does not require any query from the right hand side function f , which means that we can easily make another splitted sample to approximate the ∫ ‖∇u‖2dx term more precisely.
EMDRMN,n (u) = 1
N N∑ j=1 [ |Ω| · 1 2 ‖∇u(X ′j)‖2 ] + 1 n n∑ j=1 [ |Ω| · (1 2 V (Xj)|u(Xj)|2 − fju(Xj) )] (5.1)
Once we sampled more data for approximating ∫ |∇u|2dx, we can achieve an near optimal bound for the Truncated Fourier Estimator when Nn & n 2
d+2s−4 . Theorem 5.1. (Informal Upper Bound of DRM with Truncated Fourier Series Estimator)With proper assumptions, consider the Deep Ritz objective with a plug in Fourier Series estimator ûFourierMDRM = minu∈Fξ(Ω) E MDRM N,n (u) with ξ = Θ(n 1 d+2s−4 ) and Nn & n 2 d+2s−4 , then we have
‖ûFourierMDRM − u∗‖2H1 . n − 2s−2d+2s−4 .
Remark. We still cannot achieve optimal rate for neural network even with modified DRM methods. The reason is because the number of neurons is not a good complexity measure for the gradient of the function. Thus, the bound for ψ(r) = Rn({I(u) | ‖u− u∗‖2H1 ≤ r}) is not enough for achieving optimal convergence rate. However, following (Schmidt-Hieber et al., 2020; Suzuki, 2018; Imaizumi & Fukumizu, 2020; Chen et al., 2019b; Farrell et al., 2021) to use deep networks for estimating functions, we optimize the best neural network with constrained sparsity in our paper. Here we conjecture that there exists a computable complexity measure that can make DRM statistically optimal and leave finding the right complexity of the neural network’s gradient to be future work.

heading: 6 EXPERIMENTS

text: In this section, we conduct several numerical experiments to verify our theory. We follow the neural network and hyper-parameter setting in (Chen et al., 2020). Due to the page limit, we only put the experiments for Deep Ritz Methods here.

heading: 6.1 THE MODIFIED DEEP RITZ METHODS

text: In this section, we conduct experiments which substantiate our theoretical results for modified Deep Ritz methods. For simplicity, we take V (x) = 1 in our experiment. We conduct experiment in 2-dimension and select the solution of the PDE as u∗ = ∑ z ‖z‖−sφz(x) ∈ Hs. We show the log-log plot of the H1 loss against the number of sampled data for s = 4 in Figure 1. We use an OLS estimator to fit the log-log plot and put the estimated slope and corresponding R2 score in Figure 1. As our theory predicts, the modified Deep Ritz Method converges faster than the original one. All the derivation of the two estimators is listed in Appendix E.

heading: 6.2 DIMENSION DEPENDENT SCALING LAW.

text:
We conduct experiments to illustrate that the population loss of well-trained and well-tuned Deep Ritz method will scale with the ddimensional training data number N as a powerlaw L ∝ 1Nα . We also scan over a range of d and α and verify an approximately α ∝ 1d scaling law as our theory suggests. We use the same test function in Section 6.1 as the solution of our PDE. For simplicity, we take V (x) = 1 in our experiment. We train the deep Ritz method on 20, 80, 320, 1280, 10240 sampled data points for 5,6,7,8,9,10 dimensional problems and we plot our results on the log-log scale. Results are shown in Figure 2. We discover the L ∝ n 1
d+2 scaling law in practical situations.

heading: 6.3 ADAPTATION TO THE SIMPLER FUNCTIONS.

text:
(Sharma & Kaplan, 2020) showed that the neural scaling law will adapt to the structure that the target function enjoys. This adaptivity enables the neural network to break the cure of the dimensionality for simple functions in high dimension. (Suzuki & Nitanda, 2019; Chen et al., 2019a) also observed this theoretically. For solving PDEs, we also observed this adaptivity in practice. Here we tested the following two hypothesis
• Random Neural Network Teacher. Following (Sharma & Kaplan, 2020), we also tested random neural network using He initialization (He et al., 2015) as the ground turth solution u∗. (De Palma et al., 2018) showed that random deep neural networks are biased towards simple functions and in practice we observed a scaling law at the parametric rate. Specifically, we obtained a linear estimate with slope α = −0.50679429 and a R2 score = 0.96 in the log-log plot. See Figure 3(a).
• Simple Polynomials. Neural network can approximate simple polynomials exponentially fast (Wang et al., 2018). Thus, we select the ground truth solution to be the following simple polynomial in 10 dimensional spaces u∗(x) = x1x2 + · · ·+ x9x10. In this example, we obtained a linear estimate with slope α = −0.49755418 and a R2 score = 0.99 in the log-log plot. See Figure 3(b).
7 CONCLUSION AND DISCUSSION
L og of R elative Testing L oss
Log of Training Data Number.
(a) Random Neural Network Teacher (b) y = x1x2 + x3x4 + ⋯ + x9x10
Log of Training Data Number.
L og of R elative Testing L oss
Figure 3: Neural networks have the ability to adapt to simple functions and achieve convergence without curse of dimensionality, showing the median error over 5 replicates.
Conclusion In this paper, We considered the statistical min-max optimality of solving a PDE from random samples. We improved the previous bounds (Xu, 2020; Lu et al., 2021b; Duan et al., 2021; Jiao et al., 2021a) by providing the first fast rate generalization bound for learning PDE solutions via the strongly convex nature of the two objective functions. We achieved the optimal rate via the PINN and a modified Deep Ritz method. We verified our theory via numerical experiments and explored the dimension dependent scaling laws of Deep PDE solvers.
Discussion and Future Work Here we discuss several drawbacks of our theory
• We restricted our target function and estimators in W 1,∞ instead of H1 due to boundedness assumption made in the local Rademacher complexity arguments. However, typical functional used in physics is always unbounded, such as the Newtonian potential 1‖x−y‖d−2 , which limits the application of our theory.
• This paper did not discuss any optimization aspect of the deep PDE solvers and always assumed that global optimum can be achieved. However, it is important to investigate whether the optimization error (Suzuki & Akiyama, 2020; Chizat, 2021) will finally dominate.
• Instead of solving a single PDE, recent works (Long et al., 2018; 2019; Li et al., 2020; Lanthaler et al., 2021; Bhattacharya et al., 2020; Fan & Ying, 2020; Feliu-Faba et al., 2020) considered the so-called "operator learning", which aims to learn a family of PDE/inverse
problems using a single network. It is interesting to investigate the generalization bound and neural scaling law there.
• We find out that the sparsity of the neural network is not a good complexity measure of neural network’s gradient. We conjecture that there exists an oracle complexity measure, whose approximation and generalization bounds can lead Modified DRM to achieve the optimal convergence rate.

heading: ACKNOWLEDGMENTS

text: Yiping Lu is supported by the Stanford Interdisciplinary Graduate Fellowship (SIGF). Jianfeng Lu is supported in part by National Science Foundation via grants DMS-2012286 and CCF1934964. Lexing Ying is supported by National Science Foundation under award DMS-2011699. Jose Blanchet is supported in part by the Air Force Office of Scientific Research under award number FA9550-201-0397 and NSF grants 1915967, 1820942, 1838576. Yiping Lu also thanks Taiji Suzuki, Atsushi Nitanda, Yifan Chen, Junbin Huang, Wenlong Ji, Greg Yang, Yufan Chen, Zong Shang, Denny Wu, Jikai Hou, Jun Hu, Fang Yao, Bin Dong and George Em Karniadakis for helpful comments and feedback.

heading: A APPENDIX ORGANIZATION AND PROOF SKETCH

heading: A.1 NOTATIONS

text:
In this section, we provide all the notations we need in the proof. Let Ω ⊂ Rd be some open set. We denote C(Ω) the space of continuous functions on Ω and Ck(Ω) the space of all functions that are k times continuously differentiable on Ω (∀ k ∈ Z+). For any n ∈ N0 (N0 := Z+ ∪ {0} is the set of all non-negative integers) and 1 ≤ p ≤ ∞, we define the Sobolev space Wn,p(Ω) by
Wn,p(Ω) := {f ∈ Lp(Ω) : Dαf ∈ Lp(Ω), ∀α ∈ Nd0 with |α| ≤ n}. In particular, when p = 2, we define Hn(Ω) := Wn,2(Ω) for any n ∈ N0. Moreover, for any f ∈Wn,p(Ω) with 1 ≤ p <∞, we define the Sobolev norm by:
‖f‖Wn,p(Ω) := ( ∑
0≤|α|≤n
‖Dαf‖pLp(Ω) ) 1 p .
In particular, when p =∞, we have: ‖f‖Wn,∞(Ω) := max
0≤|α|≤n ‖Dαf‖L∞(Ω). Consider the Fourier expansion f := ∑ z∈Nd fzφz(x) of the function f ∈ Wn,p(Ω). We can equivalently express the Sobolev norm as:
‖f‖Wn,p(Ω) = (∑
z
‖z‖np|fz|p )1/p ,
where fz = ∫
Ω f(x)φz(x)dx = ∫ Ω f(x)e−2πi〈z,x〉dx (x ∈ Ω) is the z−th Fourier coefficient of f .
Moreover, we use W 1,p0 (Ω) to denote the closure of C 1 c (Ω) in W 1,p(Ω). In particular, when p = 2, we define H10 (Ω) := W 1,2 0 (Ω). Furthermore, we use ‖ · ‖ to present the vector 2 norm and, given a data sample {Xi}ni=1 ⊂ Ω, ‖ · ‖n,p = (En·p)1/p denote the empirical p norm, where En : L2(Ω) → R is the corresponding empirical average operator defined as Enf := 1n ∑n i=1 f(Xi), ∀ f ∈ L2(Ω). Given two quantities X and Y , we write X . Y when the inequality X ≤ CY holds, where C is some constant. For two functions f and g mapping from R+ to R, we write f = O(g) when there exist two constants C ′ and x0 independent of f and g, such that the inequality f(x) ≤ C ′g(x) holds for any x ≥ x0. We use X ' Y to denote X . Y and Y . X .

heading: A.2 APPENDIX ORGANIZATION AND PROOF SKETCH

text:
In this section, we list the organization of the appendix and put a more detailed proof sketch of our main results. We put all the proof of upper bounds in Appendix B and all the proof of lower bounds in Appendix D. All the proof of the results about modified Deep Ritz method is given in Appendix C. The proof of lower bounds is based on standard Fano method. In this section, we focus on the proof sketch of the fast rate upper bound.
Error Decomposition. We first decompose the excess risk ∆En := E(un) − E(u∗) of a welltrained neural network on the PINN/DRM loss function into approximation error and generalization error, similar to (Xu, 2020; Lu et al., 2021b; Duan et al., 2021; Jiao et al., 2021a;b). The regularity results used in the decomposition are proved in Appendix B.1. Explicitly, for any uF ∈ F (Ω), we can decompose the excess risk as
∆E(n)(û) = E(û)−E(u?) = [ E(û)−En(û) ] + [ En(û)−En(uF ) ] + [ En(uF )−E(uF ) ] + [ E(uF )−E(u?) ] ≤ [ E(û)−En(û) ] + [ En(uF )−E(uF )
]︸ ︷︷ ︸ Generalization Error + [ E(uF )−E(u?) ]︸ ︷︷ ︸ Approximation Error ,
(A.1)
where the expectation is on uniformly sampled data, F (Ω) is the space of parametrized estimators we used like truncated Fourier series or sparse neural networks, û is the minimizer of the empirical loss En in F (Ω) and u∗ is the minimizer of the population loss E (i.e, ground truth solution). The inequality in the third line follows from the fact that û is the minimizer of the empirical loss En in the space F (Ω), which implies En(û) ≤ En(uF ). We call the first term generalization error as it’s measuring the difference between En and E. We call the second term approximation error as it seeks for a parametrized estimator uF that approximates the ground truth solution u∗ well in F (Ω). The upper bounds on generalization and approximation error that we achieved in this paper are listed in Table 2.
Let n denote the number of sampled datapoints. For the generalization error, different from the concurrent upper bound O( 1√
n ) (Duan et al., 2021; Jiao et al., 2021a), we provide a faster and near
optimal upper bound O( 1n ) by utilizing the strong convexity of the objective function established in Appendix B.1. Via using the Peeling Lemma (Lemma B.4, for completeness, we also provide a proof), we show that the generalization error can be bounded by the fixed point of the local Rademacher complexity φ(r) = Rn({I(u) | ‖u− u∗‖2A ≤ r}), where Rn is the Rademacher complexity, I(u) = ∆u + V u, ‖ · ‖A = ‖ · ‖H2 for PINN and I(u) = ‖∇u‖2 + V u, ‖ · ‖A = ‖ · ‖H1 for DRM. Once we show that φ(r) is of magnitude O( √ r n ),
we can achieve the O( 1n ) convergence rate via solving the fix point equation φ(r) = O( √ r n ) = r ⇒ r = O( 1n ). Using the solution of the fixed point equation of the local Rademacher complexity to bound the generalization error is a standard result in empirical process Bartlett et al. (2005); Srebro et al. (2010); Koltchinskii (2011); Xu & Zeevi (2020); Farrell et al. (2021). The difference is that we used the H1/H2 norm to define the localized set, while the previous papers used the `2 distance. The way to obtain the fast rate generalization bound is using the Peeling Lemma.
We present the error decomposition results as a meta theorem, which is shown in Theorem B.12 for PINN, Theorem B.9 for DRM and Theorem C.1 for MDRM, respectively. To make the final rate depend on the data number only, we need bounds of the approximation error in Appendix B.3 and bounds of the local Rademacher complexity in Appendix B.2.
Approximation Error. The proof of the approximation results of truncated Fourier series is easy and intuitive. For completeness, we provide it in Appendix B.3.1. The proof of the approximation results of neural networks follows from the fact that a B-spline approximation can be formulated as a ReLU3 neural network efficiently. Our proof basically follows Duan et al. (2021); Jiao et al. (2021a), while the only difference is the activation function. Our proof is also very similar to (Yarotsky, 2017; Suzuki, 2018), but the depth of our network is of constant magnitude instead of O( 1log ) magnitude, where denotes the desired approximation error. Such improvement of depth results from the fact that ReLU3 activations can approximate B-splines more easily than the ReLU activations, which is useful in our generalization analysis. Although the proof of the approximation results of neural networks in the Sobolev space is standard, we still list it in Appendix B.3.2.
Generalization Error. As we discussed above, the generalization error can be bounded by the fix point of the local Rademacher complexity, i.e. the solution of φ(r) = r. Once we have a O( √ r n ) bound of φ(r), we can achieve the O( 1n ) fast rate generalization bound we want. It remains to upper bound the the local Rademacher complexity φ(r).
For the upper bound on the local Rademacher complexity of truncated Fourier series estimators, our proof technique is similar to that of the kernel estimators, whose Rademacher complexity can be bounded by the trace of the Gram matrix (i.e. the effective number of basis). One interesting thing we
showed is that the final upper bound of the Rademacher complexity localized by H1 norm is √
ξd−2r n .
The term ξd−2 in the numerator is smaller than ξd, which is the exact number of Fourier basis. This improvement results from the H1 norm localization. The detailed proof is given in Lemma B.6, Lemma B.7 and Lemma B.8.
For the upper bound on the local Rademacher complexity bound for neural network, we follow Schmidt-Hieber et al. (2020); Suzuki (2018); Farrell et al. (2021) to use a Dudley integral theorem and a covering number argument. The covering number arguments are shown in Theorem B.4, Theorem
B.5 and Theorem B.6. The final local Rademacher complexity bounds are given in Lemma B.17 and Lemma B.18. The difference is that the complexity of gradient of ReLU3 activation function makes the covering number depend exponentially on the neural network’s depth. However, the improvement of neural network’s depth to constant magnitude mentioned above in the approximation results saves this problem. One drawback of our proof is that the H1 norm localization wouldn’t improve the bound for Rademacher complexity and leads to sub-optimal upper bounds. We hypothesize that our bound is tight for sparse neural network and put seeking a right complexity measure of neural network for solving PDEs as a future work.

heading: B PROOF OF THE UPPER BOUNDS

heading: B.1 REGULARITY RESULT FOR THE PDE MODEL.

heading: Regularity Results of the DRM Objective Function

text:
Theorem B.1. We consider the static Schrödinger equation on the unit hypercube on Rd with the zero Direchlet boundary condition:
−∆u+ V u = f on Ω, u = 0 on ∂Ω.
(B.1)
where f ∈ L2(Ω) and V ∈ L∞(Ω) with 0 < Vmin ≤ V (x) ≤ Vmax > 0. There exists a unique weak solution u∗S to the equivalent variational problem (Evans, 1998):
u∗S = arg min u∈H10 (Ω) EDRM(u) := arg min u∈H10 (Ω)
{1 2 ∫ Ω [ ‖∇u‖2 + V |u|2 ] dx− ∫ Ω fudx } . (B.2)
Then for any u ∈ H1(Ω), we have:
min(1, Vmin)
2 ‖u− u∗S‖2H1(Ω) ≤ E
DRM(u)−EDRM(u∗S) ≤ max(1, Vmax)
2 ‖u− u∗S‖2H1(Ω). (B.3)
Proof. To show that u∗S satisfies estimate B.3, we first claim that for any u ∈ H1(Ω),
EDRM(u)−EDRM(u∗S) = 1
2 ∫ Ω ‖∇u−∇u∗S‖2dx+ 1 2 ∫ Ω V (u∗S − u)2 dx. (B.4)
In fact, by plugging in the first equation of B.1, one has that
EDRM(u∗S) = 1
2 ∫ Ω ‖∇u∗S‖2dx+ 1 2 ∫ Ω V |u∗S |2dx− ∫ Ω fu∗Sdx
= 1
2 ∫ Ω ‖∇u∗S‖2dx+ 1 2 ∫ Ω V |u∗S |2dx+ ∫ Ω (∆u∗S − V u∗S)u∗Sdx
= 1
2 ∫ Ω ‖∇u∗S‖2dx+ ∫ Ω (∆u∗S)u ∗ Sdx− 1 2 ∫ Ω V |u∗S |2dx.
Furthermore, applying Green’s formula to the true solution u∗S yields:
EDRM(u∗S) = 1
2 ∫ Ω ‖∇u∗S‖2dx+ ∫ Ω (∆u∗S)u ∗ Sdx− 1 2 ∫ Ω V |u∗S |2dx
= ∫ ∂Ω ∂u∗S ∂n u∗Sdx− 1 2 ∫ Ω ‖∇u∗S‖2dx− 1 2 ∫ Ω V |u∗S |2dx
= −1 2 ∫ Ω ‖∇u∗S‖2dx− 1 2 ∫ Ω V |u∗S |2dx,
where the last identity above follows from the second equality in B.1. Now for any u ∈ H1(Ω), applying Green’s formula to u and the true solution u∗S implies:
EDRM(u)−EDRM(u∗S) = 1
2 ∫ Ω ‖∇u‖2dx+ 1 2 ∫ Ω V |u|2dx− ∫ Ω fudx+ 1 2 ∫ Ω ‖∇u∗S‖2dx+ 1 2 ∫ Ω V |u∗S |2dx
= 1
2 ∫ Ω ‖∇u‖2dx+ 1 2 ∫ Ω V |u|2dx+ ∫ Ω (∆u∗S − V u∗S)udx+ 1 2 ∫ Ω ‖∇u∗S‖2dx+ 1 2 ∫ Ω V |u∗S |2dx
= 1
2 ∫ Ω ‖∇u‖2dx+ ∫ Ω (∆u∗S)udx+ 1 2 ∫ Ω ‖∇u∗S‖2dx+ 1 2 ∫ Ω V ( u∗S − u )2 dx
= 1
2 ∫ Ω ‖∇u‖2dx+ ∫ ∂Ω ∂u∗S ∂n udx− ∫ Ω ∇u∗S · ∇udx+ 1 2 ∫ Ω ‖∇u∗S‖2dx+ 1 2 ∫ Ω V ( u∗S − u )2 dx
= 1
2 ∫ Ω ‖∇u−∇u∗S‖2dx+ 1 2 ∫ Ω V (u∗S − u)2 dx,
where the last identity above again follows from the second equality in B.1. This completes our proof of identity B.4. Using the assumptions on the potential function V then implies:
EDRM(u)−EDRM(u∗S) ≤ max(1, Vmax)
2
[ ∫ Ω ‖∇u−∇u∗S‖2dx+ ∫ Ω (u∗S − u)2 dx ]
= max(1, Vmax)
2 ‖u− u∗S‖2H1(Ω),
EDRM(u)−EDRM(u∗S) ≥ min(1, Vmin)
2
[ ∫ Ω ‖∇u−∇u∗S‖2dx+ ∫ Ω (u∗S − u)2 dx ]
= min(1, Vmin)
2 ‖u− u∗S‖2H1(Ω).
This completes our proof of B.1.

heading: Regularity Results of the PINN Objective Function

text:
Theorem B.2. We consider the static Schrödinger equation on the unit hypercube on Rd with the Neumann boundary condition:
−∆u+ V u = f on Ω, u = 0 on ∂Ω.
(B.5)
where f ∈ L2(Ω) and V ∈ L∞(Ω) with V 2 − ∆V > Cmin, 0 < Cmin < V (x) ≤ Vmax and −∆V (x) ≤ Vmax. Then there exists a unique solution u∗S ∈ H10 (Ω) to the following minimization problem (Brezis, 2010):
u∗S = arg min u∈H10 (Ω) EPINN(u) := arg min u∈H10 (Ω) {∫ Ω |∆u− V u+ f |2dx } . (B.6)
Then for any u ∈ H10 (Ω), we have:
min{1, Cmin}‖u− u∗S‖2H2(Ω) ≤ E PINN(u)−EPINN(u∗S) ≤ 2(1 + Vmax + V 2max)‖u− u∗S‖2H2(Ω).
(B.7)
Proof. For any u ∈ H10 (Ω), we let ũ = u− u∗, then we have ũ ∈ H10 (Ω). EPINN(u)−EPINN(u∗S) = ∫
Ω
|∆u− V u−∆u∗ + V u∗|2dx = ∫
Ω
|∆ũ− V ũ|2dx
= ∫ Ω (∆ũ)2dx+ ∫ Ω V 2ũ2dx− 2 ∫ Ω V ũ∆ũdx.
(B.8)
Using Green’s formula, we have:∫ Ω V ũ∆ũdx+ ∫ Ω ∇(V ũ) · ∇ũdx = ∫ ∂Ω ∂ũ ∂n V ũds = 0,
where the last equality above follows from the fact that ũ ∈ H10 (Ω). This further implies: EPINN(u)−EPINN(u∗S) = ∫
Ω
(∆ũ)2dx+ ∫ Ω V 2ũ2dx+ 2 ∫ Ω ∇(V ũ) · ∇ũdx
= ∫ Ω (∆ũ)2dx+ ∫ Ω V 2ũ2dx+ 2 ∫ Ω V ‖∇ũ‖2dx+ 2 ∫ Ω ũ∇V · ∇ũdx.
Using Green’s formula again, we have:
2 ∫ Ω ũ∇V · ∇ũdx = ∫ Ω ∇(u2) · ∇V dx = ∫ ∂Ω ∂V ∂n ũ2ds− ∫ Ω ũ2∆V dx = − ∫ Ω ũ2∆V dx.
Then we can further deduce that: EPINN(u)−EPINN(u∗S) = ∫
Ω
(∆ũ)2dx+ ∫ Ω (V 2 −∆V )ũ2dx+ 2 ∫ Ω V ‖∇ũ‖2dx.
For we have assumed V ∈ L∞(Ω) with 0 < Cmin < V 2 − ∆V, 0 < Cmin < V (x) ≤ Vmax and −∆V (x) ≤ Vmax, thus we have
min{1, Cmin}‖u− u∗S‖2H2(Ω) ≤ E PINN(u)−EPINN(u∗S) ≤ 2(1 + Vmax + V 2max)‖u− u∗S‖2H2(Ω).
(B.9)

heading: B.2 AUXILIARY DEFINITIONS AND LEMMATA ON GENERALIZATION ERROR

text:
To bound the generalization error, we use the localized Rademacher complexity (Bartlett et al., 2005). Recall that the Rademacher complexity of a function class G is defined by
Rn(G) = EZEσ [
sup g∈G ∣∣∣ 1 n n∑ j=1 σjg(Zj) ∣∣∣ ∣∣∣ Z1, · · · , Zn],
where Zi are i.i.d samples according to the data distributions and σj are i.i.d Rademacher random variables which take the value 1 with probability 12 and value −1 with probability 1 2 .
The following important symmetrization lemma makes the connection between the uniform law of large numbers and the Rademacher complexity.
Lemma B.1 (Symmetrization Lemma). Let F be a set of functions. Then
E sup u∈F ∣∣∣ 1 n n∑ j=1 u(Xj)− EX∼PΩu(X) ∣∣∣ ≤ 2Rn(F ).
Lemma B.2 (Ledoux-Talagrand contraction (Ledoux & Talagrand, 2013, Theorem 4.12)). Assume that φ : R→R is L-Lipschitz with φ(0) = 0. Let {σi}ni=1 be independent Rademacher random variables. Then for any T ⊂ Rn
Eσ [
sup (t1,··· ,tn)∈T n∑ i=1 σiφ(ti) ] ≤ 2L · Eσ [ sup (t1,··· ,tn)∈T n∑ i=1 σiti ] .
Let (E, ρ) be a metric space with metric ρ. A δ-cover of a set A ⊂ E with respect to ρ is a collection of points {x1, · · · , xn} ⊂ A such that for every x ∈ A, there exists i ∈ {1, · · · , n} such that ρ(x, xi) ≤ δ. The δ-covering number N(δ, A, ρ) is the cardinality of the smallest δ-cover of the set A with respect to the metric ρ. Equivalently, the δ-covering number N(δ, A, ρ) is the minimal number of balls Bρ(x, δ) of radius δ needed to cover the set A. Theorem B.3 (Dudley’s Integral theorem). Let F be a function class such that supf∈F ‖f‖n,2 ≤M . Then the Rademacher complexity Rn(F ) satisfies that
Rn(F ) ≤ inf 0≤δ≤M
{ 4δ +
12√ n ∫ M δ √ logN( ,F , ‖ · ‖n,2) d } .
Lemma B.3 (Talagrand Concentration Inequality). Consider a function class F defined on a probability measure µ such that for all f ∈ F , we have ‖f‖∞ ≤ β,Eµ[f ] = 0,Eµ[f2] ≤ σ2. Then for any t > 0, we can have the following concentration results.
Pz1,··· ,zn∼µ [ sup f∈F 1 n n∑ i=1 f(zi) ≥ 2 sup f∈F Ez′1,··· ,z′n∼µ 1 n n∑ i=1 f(z′i) + √ 2tσ2 n + 2tβ n ] ≤ e−t.
Lemma B.4 (Peeling lemma (Bartlett et al., 2005)). Consider some measurable function class F . Assume that there exists a sub-root function φ(r) satisfying
Rn({f ∈ F | E[f ] ≤ r}) ≤ φ(r) (∀ r > 0). (B.10) Then we have
Eσi,zn [ sup f∈F 1 n ∑n i=1 σif(zi) E[f ] + r ] ≤ 4φ(r) r .
Proof. Denote F(r) = {f ∈ F | E[f ] ≤ r} to be the localized set with radius r. Then for a fixed set of datapoints {zi}ni=1 and a fixed set of Rademacher random variables {σi}ni=1, we have:
Eσi,zn [ sup f∈F 1 n ∑n i=1 σif(zi) E[f ] + r ] ≤ Eσi,zn [ sup f∈F(r) 1 n ∑n i=1 σif(zi) r ] + ∞∑ j=0 Eσi,zn [ sup f∈F(r4j+1)\F(r4j) 1 n ∑n i=1 σif(zi) r4j + r ]
≤ Rn(F(r)) r + ∞∑ j=0 Rn(F(r4j+1)) r4j + r ≤ φ(r) r + ∞∑ j=0 φ(r4j+1) r4j + r
≤ φ(r) r + ∞∑ j=0 2j+1φ(r) r4j + r ≤ 4φ(r) r .
We also modify the peeling lemma above, as we aim to apply it to derive the upper bound for the Modified Deep Ritz Method (MDRM). Lemma B.5 (Peeling Lemma For MDRM). Given some measurable function class F and two continuous mappings g, h : F → R, we define a class F of vector-valued functions by:
F := {(g ◦ f, h ◦ f) | f ∈ F }. For any f ∈ F , we use gf and hf to denote the two compositions g ◦ f and h ◦ f , respectively. For any r > 0, the localized set Fr is defined by: Fr = {(gf , hf ) ∈ F | Ex[gf (x)] + Ey[hf (y)] ≤ r}. Moreover, the modified Rademacher Complexity of Fr is defined by:
Rn,m(Fr) := Rn ( {gf |(gf , hf ) ∈ Fr} ) +Rm({hf |(gf , hf ) ∈ Fr} ) .
Assume that there exists some function φ : [0,∞) → [0,∞) and some r? > 0, such that for any r > r?, we have: φ(4r) ≤ 2φ(r) and Rn,m(Fr) ≤ φ(r). Then for any r > r?, we have:
Eσ,τ [ Ex,y[sup
f∈F
1 n ∑n i=1 σigf (xi) + 1 m ∑m j=1 τjhf (yj) Ex[gf (x)] + Ey[hf (y)] + r ] ] ≤ 4φ(r) r .
Proof. The proof is the same as the original peeling lemma, thus we omit the detailed proof here.

heading: B.2.1 LOCAL RADEMACHER COMPLEXITY OF TRUNCATED FOURIER BASIS

text:
Definition B.1. (Fourier Series) Given a domain Ω ⊆ [0, 1]d. For any z ∈ Nd, we consider the corresponding Fourier basis function φz(x) := e2πi〈z,x〉 (x ∈ Ω). With respect to the Fourier basis, any function f ∈ L2(Ω) can be decomposed as the following sum:
f(x) := ∑ z∈Nd fzφz(x). (B.11)
where for any z ∈ Nd, the Fourier coefficient fz = ∫
Ω f(x)φz(x)dx.
Definition B.2. (Truncated Fourier Series) For a fixed positive integer ξ ∈ Z+, we define the space Fξ(Ω) of truncated Fourier series as follows:
Fξ(Ω) := { f = ∑ z∈Nd fzφz ∣∣∣ fz = 0, ∀ ‖z‖∞ > ξ}. (B.12) Equivalently, we can decompose any f ∈ Fξ(Ω) as f := ∑ ‖z‖∞≤ξ fzφz .
Lemma B.6. (Local Rademacher Complexity of Localized Truncated Fourier Series) For a fixed ξ ∈ Z+, we consider a localized class of functions Fρ,ξ(Ω) = { f ∈ Fξ(Ω) ∣∣∣ ‖f‖2H1(Ω) ≤ ρ}, where ρ > 0 is fixed. Then we have the following upper bound on the local Rademacher complexity:
Rn(Fρ,ξ(Ω)) = EX [ Eσ [
sup f∈Fρ,ξ(Ω)
1
n n∑ i=1
σif(Xi) ∣∣∣ X1, · · · , Xn]] .√ ρ
n ξ d−2 2 . (B.13)
Proof. Take an arbitrary function f ∈ Fρ,ξ(Ω). Let f = ∑ ‖z‖∞≤ξ fzφz be the Fourier basis
expansion of f . ρ ≥ ‖f‖2H1(Ω) implies constraint ∑ ‖z‖∞≤ξ |fz|
2‖z‖2 . ρ on the Fourier coefficients(Adams & Fournier, 2003).
On the other hand, substituting the Fourier expansion into the average sum 1n ∑n i=1 σif(Xi) and using Cauchy-Schwarz inequality imply:
1
n n∑ i=1 σif(Xi) = 1 n n∑ i=1 σi ∑ ‖z‖∞≤ξ fzφz(Xi) = 1 n ∑ ‖z‖∞≤ξ n∑ i=1 σifzφz(Xi)
≤ 1 n ( ∑ ‖z‖∞≤ξ |fz|2‖z‖2 ) 1 2 ( ∑ ‖z‖∞≤ξ ∣∣∣ n∑ i=1 σi ‖z‖ φz(Xi) ∣∣∣2) 12 . √ ρ
n ( ∑ ‖z‖∞≤ξ ∣∣∣ n∑ i=1 σi ‖z‖ φz(Xi) ∣∣∣2) 12 .
where we have used the constraint ∑ ‖z‖∞≤ξ |fz|
2‖z‖2 . ρ in the last step above. Moreover, by taking expectation with respect to the i.i.d Rademacher random variables σi (1 ≤ i ≤ n) and the uniformly sampled data points {Xi}ni=1 on both sides and applying Jensen’s inequality, we can deduce that:
EXEσ [ 1 n n∑ i=1 σif(Xi) ] . √ ρ n EX,σ ( ∑ ‖z‖∞≤ξ ∣∣∣ n∑ i=1 σi ‖z‖ φz(Xi) ∣∣∣2) 12  ≤ √ ρ
n EX,σ[ ∑ ‖z‖∞≤ξ ∣∣∣ n∑ i=1 σi ‖z‖ φz(Xi) ∣∣∣2]  12 .
Using independence between the random variables σi (1 ≤ i ≤ n), we can further simplify the expectation inside the square root above as below:
EX,σ [ ∑ ‖z‖∞≤ξ ∣∣∣ n∑ i=1 σi ‖z‖ φz(Xi) ∣∣∣2] = ∑ ‖z‖∞≤ξ EX,σ [∣∣∣ n∑ i=1 σi ‖z‖ φz(Xi) ∣∣∣2]
= ∑ ‖z‖∞≤ξ n∑ i=1 EX,σ [ σ2i ‖z‖2 ∣∣∣φz(Xi)∣∣∣2]
= ∑ ‖z‖∞≤ξ n∑ i=1 |Ω| ‖z‖2 . n ∑ ‖z‖∞≤ξ 1 ‖z‖2 . n ξd ξ2 = nξd−2.
Combining the two bounds above yields the desired upper bound:
EX [ Eσ [
sup f∈Fρ,ξ(Ω)
1
n n∑ i=1
σif(Xi) ∣∣∣ X1, · · · , Xn]] . √ρ
n
√ nξd−2 = √ ρ
n ξ d−2 2 .
Lemma B.7. (Local Rademacher Complexity of Localized Truncated Fourier Series’ Gradient) For a fixed ξ ∈ Z+, we consider a localized class of functions Gρ,ξ(Ω) = {‖∇f‖ | f ∈ Fρ,ξ(Ω)}, where ρ > 0 is fixed. Then for any sample {Xi}ni=1 ⊂ Ω, we have the following upper bound on the local Rademacher complexity:
Rn(Gρ,ξ(Ω)) = EX [ Eσ [
sup f∈Fρ,ξ(Ω)
1
n n∑ i=1
σi‖∇f(Xi)‖ ∣∣∣ X1, · · · , Xn]] .√ ρ
n ξ d 2 . (B.14)
Proof. Take an arbitrary function f ∈ Fρ,ξ(Ω). Let f = ∑ ‖z‖∞≤ξ fzφz be the Fourier basis expansion of f . Similarly, the norm restriction condition ‖f‖2H1(Ω) ≤ ρ can be reduced to the following condition about Fourier coefficients:∑
‖z‖∞≤ξ
|fz|2‖z‖2 . ρ.
Moreover, substituting the Fourier expansion into the average sum 1n ∑n i=1 σi‖∇f(Xi)‖ and using Cauchy-Schwarz inequality imply:
1
n n∑ i=1 σi‖∇f(Xi)‖ = 1 n n∑ i=1 σi‖ ∑ ‖z‖∞≤ξ fz∇φz(Xi)‖ ≤ 1 n ∑ ‖z‖∞≤ξ n∑ i=1 σi‖fz∇φz(Xi)‖
≤ 1 n ( ∑ ‖z‖∞≤ξ |fz|2‖z‖2 ) 1 2 ( ∑ ‖z‖∞≤ξ ∣∣∣ n∑ i=1 σi ‖z‖ ‖∇φz(Xi)‖ ∣∣∣2) 12 . √ ρ
n ( ∑ ‖z‖∞≤ξ ∣∣∣ n∑ i=1 σi ‖z‖ ‖∇φz(Xi)‖ ∣∣∣2) 12 . where we have used the constraint ∑ ‖z‖∞≤ξ |fz|
2‖z‖2 . ρ in the last step above. Moreover, by taking expectation with respect to the i.i.d Rademacher random variables σi (1 ≤ i ≤ n) and the uniformly sampled data points {Xi}ni=1 on both sides and applying Jensen’s inequality, we can deduce that:
EXEσ [ 1 n n∑ i=1 σi‖∇f(Xi)‖ ] . √ ρ n EX,σ ( ∑ ‖z‖∞≤ξ ∣∣∣ n∑ i=1 σi ‖z‖ ‖∇φz(Xi)‖ ∣∣∣2) 12
≤ √ ρ
n EX,σ[ ∑ ‖z‖∞≤ξ ∣∣∣ n∑ i=1 σi ‖z‖ ‖∇φz(Xi)‖ ∣∣∣2]  12 .
Using independence between the random variables σi (1 ≤ i ≤ n), we can further simplify the expectation inside the square root above as below:
EX,σ [ ∑ ‖z‖∞≤ξ ∣∣∣ n∑ i=1 σi ‖z‖ ‖∇φz(Xi)‖ ∣∣∣2] = ∑ ‖z‖∞≤ξ EX,σ [∣∣∣ n∑ i=1 σi ‖z‖ ‖∇φz(Xi)‖ ∣∣∣2]
= ∑ ‖z‖∞≤ξ n∑ i=1 EX,σ [ σ2i ‖z‖2 ‖∇φz(Xi)‖2 ]
= ∑ ‖z‖∞≤ξ n∑ i=1 |Ω|4π 2‖z‖2 ‖z‖2 . n ∑ ‖z‖∞≤ξ 1 . nξd.
Combining the two bounds above yields the desired upper bound:
EX [ Eσ [
sup f∈Fρ,ξ(Ω)
1
n n∑ i=1
σi‖∇f(Xi)‖ ∣∣∣ X1, · · · , Xn]] . √ρ
n
√ nξd = √ ρ
n ξ d 2 .
Lemma B.8. (Local Rademacher Complexity of Localized Truncated Fourier Series’ Laplacian) For a fixed ξ ∈ Z+, we consider a localized class of functions Jρ,ξ(Ω) := { f ∈ Fξ(Ω) ∣∣∣ ‖f‖2H2(Ω) ≤ ρ}, where ρ > 0 is fixed. Correspondingly, we define a localized class of Laplacians Kρ,ξ(Ω) := {∆f | f ∈ Jρ,ξ(Ω)}. Then for any sample {Xi}ni=1 ⊂ Ω, we have the following upper bound on the local Rademacher complexity:
Rn(Kρ,ξ(Ω)) = EX [ Eσ [
sup f∈Fρ,ξ(Ω)
1
n n∑ i=1
σi∆f(Xi) ∣∣∣ X1, · · · , Xn]] .√ ρ
n ξ d 2 . (B.15)
Proof. Take an arbitrary function f ∈ Jρ,ξ(Ω). Let f = ∑ ‖z‖∞≤ξ fzφz be the Fourier basis expansion of f . Similarly, the norm restriction condition ‖f‖2H2(Ω) ≤ ρ can be reduced to the following condition about Fourier coefficients:∑
‖z‖∞≤ξ
|fz|2‖z‖4 . ρ.
Moreover, substituting the Fourier expansion into the average sum 1n ∑n i=1 σi∆f(Xi) and using Cauchy-Schwarz inequality imply:
1
n n∑ i=1 σi∆f(Xi) = 1 n n∑ i=1 σi ∑ ‖z‖∞≤ξ fz∆φz(Xi) = 1 n ∑ ‖z‖∞≤ξ n∑ i=1 σifz∆φz(Xi)
≤ 1 n ( ∑ ‖z‖∞≤ξ |fz|2‖z‖4 ) 1 2 ( ∑ ‖z‖∞≤ξ ∣∣∣ n∑ i=1 σi ‖z‖2 ∆φz(Xi) ∣∣∣2) 12 . √ ρ
n ( ∑ ‖z‖∞≤ξ ∣∣∣ n∑ i=1 σi ‖z‖2 ∆φz(Xi) ∣∣∣2) 12 .
where we have used the constraint ∑ ‖z‖∞≤ξ |fz|
2‖z‖4 . ρ in the last step above. Moreover, by taking expectation with respect to the i.i.d Rademacher random variables σi (1 ≤ i ≤ n) and the uniformly sampled data points {Xi}ni=1 on both sides and applying Jensen’s inequality, we can deduce that:
EXEσ [ 1 n n∑ i=1 σi∆f(Xi) ] . √ ρ n EX,σ ( ∑ ‖z‖∞≤ξ ∣∣∣ n∑ i=1 σi ‖z‖2 ∆φz(Xi) ∣∣∣2) 12  ≤ √ ρ
n EX,σ[ ∑ ‖z‖∞≤ξ ∣∣∣ n∑ i=1 σi ‖z‖2 ∆φz(Xi) ∣∣∣2]  12 .
Using independence between the random variables σi (1 ≤ i ≤ n), we can further simplify the expectation inside the square root above as below:
EX,σ [ ∑ ‖z‖∞≤ξ ∣∣∣ n∑ i=1 σi ‖z‖2 ∆φz(Xi) ∣∣∣2] = ∑ ‖z‖∞≤ξ EX,σ [∣∣∣ n∑ i=1 σi ‖z‖2 ∆φz(Xi) ∣∣∣2]
= ∑ ‖z‖∞≤ξ n∑ i=1 EX,σ [ σ2i ‖z‖4 |∆φz(Xi)|2 ]
= ∑ ‖z‖∞≤ξ n∑ i=1 |Ω|16π 4‖z‖4 ‖z‖4 . n ∑ ‖z‖∞≤ξ 1 . nξd.
Combining the two bounds above yields the desired upper bound:
EX [ Eσ [
sup f∈Fρ,ξ(Ω)
1
n n∑ i=1
σi∆f(Xi) ∣∣∣ X1, · · · , Xn]] . √ρ
n
√ nξd = √ ρ
n ξ d 2 .

heading: B.2.2 LOCAL RADEMACHER COMPLEXITY OF THE DEEP NEURAL NETWORK MODEL

text:
In this section we aim to bound the local Rademacher Complexity of a Deep Neural Network. We first bound the covering number of the function space composed by the gradient of all possible neural networks and then apply a Duley Integral to achieve the final bound. Definition B.3. Let ηl denote the l-ReLU activiation function. Here we use η3 := max{0, x}3(E & Yu, 2018) as the activation function to ensure smoothness. We can define the space consisting of all neural network models with depth L, width W , sparsity constraint S and norm constraint B as follows:
Φ(L,W, S,B) := { (W(L)η3(·) + b(L)) · · · (W(1)x+ b(1)) | W(L) ∈ R1×W , b(L) ∈ R, (B.16)
W(1) ∈ RW×d, b(1) ∈ RW ,W(l) ∈ RW×W , b(l) ∈ RW (1 < l < L), (B.17) L∑ l=1 (‖W(l)‖0 + ‖b(l)‖0) ≤ S,max l ‖W(l)‖∞,∞ ∨ ‖b(l)‖∞ ≤ B } . (B.18)
where ‖ · ‖0 measures the number of nonzero entries in a matrix and ‖ · ‖∞,∞ measures the maximum of the absolute values of the entries in a matrix. For any d ∈ Z+, we refer to an arbitrary element in Φ(L,W, S,B) as a ReLU3 Deep Neural Network. Then for any index 1 ≤ k ≤ L, we use Fk to denote the k−ReLU3 Deep Neural Network composed by the first k layers, i.e:
Fk(x) := (W(k)F η3(·) + b (k) F ) · · · (W (1) F x+ b (1) F ).
Also, we use Φk(L,W, S,B) to denote the space consisting of all Fk. In particular, when k = L, we have:
F (x) := FL(x) = (W(L)F η3(·) + b (L) F ) · · · (W (1) F x+ b (1) F ), and ΦL(L,W,S,B) = Φ(L,W,S,B).
Furthermore, given that the domain Ω ⊂ [0, 1]d is bounded, we have supx∈Ω ‖x‖∞ = 1. Lemma B.9. (Upper bound on ∞-norm of functions in DNN space) For any 1 ≤ k ≤ L, the following inequality holds:
sup x∈Ω, Fk∈Φk(L,W,S,B)
‖Fk(x)‖∞ ≤W 3k−1−1 2 (B ∨ d) 5·3k−1−1 2 2 3k−1 2 −k+1.
Proof. We use induction to prove this claim. Base cases: When k = 1, we have that for any x ∈ Ω and any F1 ∈ Φ1(L,W, S,B), the following holds:
‖F1(x)‖∞ = ‖W(1)F x+ b (1) F ‖∞ ≤ ‖W (1) F ‖∞‖x‖∞ + ‖b (1) F ‖∞
≤ d‖W(1)F ‖∞,∞ +B ≤ dB +B ≤ 2(B ∨ d) 2.
(B.19)
When k = 2, we have that for any x ∈ Ω and any F2 ∈ Φ2(L,W, S,B), the following holds:
‖F2(x)‖∞ = ‖W(2)F η3(F1(x)) + b (2) F ‖∞ ≤ ‖W (2) F ‖∞‖η3(F1(x))‖∞ + ‖b (2) F ‖∞ ≤W‖W (2) F ‖∞,∞‖F1(x)‖ 3 ∞ +B.
By applying the bound proved in the case when k = 1, we have:
‖F2(x)‖∞ ≤WB(dB +B)3 +B = WB4(d+ 1)3 +B = WB4(d3 + 3d2 + 3d+ 1) +B ≤ 8W (B ∨ d)7.
where the last inequality follows from the assumption that W ≥ 2. Inductive Step: Now we assume that the claim has been proved for k−1, where 3 ≤ k ≤ L. Similarly, for any x ∈ Ω and any Fk ∈ Φk(L,W,S,B), we have:
‖Fk(x)‖∞ = ‖W(k)F η3(Fk−1(x)) + b (k) F ‖∞ ≤ ‖W (k) F ‖∞‖η3(Fk−1(x))‖∞ + ‖b (k) F ‖∞
≤W‖W(k)F ‖∞,∞‖Fk−1(x)‖ 3 ∞ +B ≤WB‖Fk−1(x)‖3∞ +B.
Using inductive hypothesis, we can further deduce that:
‖Fk(x)‖∞ ≤WB ×W 3k−1−3 2 (B ∨ d) 5·3k−1−3 2 2 3k−3 2 −3k+6 +B
≤W 3k−1−1 2 (B ∨ d) 5·3k−1−1 2 2 3k−3 2 −3k+6 +B ∨ d
≤W 3k−1−1 2 (B ∨ d) 5·3k−1−1 2 [2 3k−3 2 −3k+6 + 1]
≤W 3k−1−1 2 (B ∨ d) 5·3k−1−1 2 2 3k−3 2 −k+2 (k ≥ 3)
= W 3k−1−1 2 (B ∨ d) 5·3k−1−1 2 2 3k−1 2 −k+1.
Taking supremum with respect to x ∈ Ω and Fk ∈ Φk(L,W,S,B) on the LHS implies that the given upper bound also holds for k. By induction, the claim is proved.
We also need to show that the ReLU3 activation function is a Lipschitzness functions over a bounded domain.
Lemma B.10. For any k ∈ Z+, consider the k−ReLU activation function ηk defined on some bounded domain D ⊂ Rd (i.e, supx∈D ‖x‖∞ ≤ C for some C > 0). Then we have that for any x, y ∈ D, the following inequalities hold:
‖η1(x)− η1(y)‖∞ ≤ ‖x− y‖∞, ‖η2(x)− η2(y)‖∞ ≤ 2C‖x− y‖∞, ‖η3(x)− η3(y)‖∞ ≤ 3C2‖x− y‖∞.
Proof. This is because |∇η1(x)| = |max{1, 0}| = 1, |∇η2(x)| = |2 max{x, 0}| ≤ 2C and |∇η3(x)| = |3 max{x, 0}2| ≤ 3C2.
Lemma B.11. (Relation between the covering number of DNN space and parameter space) For any 1 ≤ k ≤ L, suppose that a pair of different two networks Fk, Gk ∈ Φk(L,W,S,B) are given by:
Fk(x) := (W(k)F η3(·) + b (k) F ) · · · (W (1) F x+ b (1) F ),
Gk(x) := (W(k)G η3(·) + b (k) G ) · · · (W (1) G x+ b (1) G ).
Furthermore, assume that the ‖ ‖∞ norm of the distance between the parameter spaces of Fk and Gk is uniformly upper bounded by δ, i.e
‖W (l)F −W (l) G ‖∞,∞ ≤ δ, ‖b (l) F − b (l) G ‖∞ ≤ δ, (∀ 1 ≤ l ≤ k). (B.20)
Then we have:
sup x∈Ω ‖Fk(x)−Gk(x)‖∞ ≤ δW
3k−1−1 2 (B ∨ d) 5·3k−1−1 2 2 3k−1 2 −k+13k−1. (B.21)
Proof. Let’s prove the claim by using induction on k. Base Case: When k = 1, we have that for any x ∈ Ω and any F1, G1 ∈ Φ1(L,W, S,B) satisfying constraint B.20, the following holds:
‖F1(x)−G1(x)‖∞ = ‖W(1)F x+ b (1) F −W (1) G x− b (1) G ‖∞
≤ ‖W(1)F −W (1) G ‖∞‖x‖∞ + ‖b (1) F − b (1) G ‖∞ ≤ δd+ δ = δ(d+ 1) ≤ 2δ(B ∨ d) ≤ 2δ(B ∨ d)2.
(B.22)
When k = 2, we have that for any x ∈ Ω and any F2, G2 ∈ Φ2(L,W,S,B) satisfying constraint B.20, the following inequality holds:
‖F2(x)−G2(x)‖∞ = ‖W(2)F η3(F1(x)) + b (2) F −W (2) G η3(G1(x))− b (2) G ‖∞
≤ ‖W(2)F η3(F1(x))−W (2) G η3(G1(x))‖∞ + ‖b (2) F − b (2) G ‖∞
≤ ‖W(2)F η3(F1(x))−W (2) G η3(F1(x))‖∞ + ‖W (2) G η3(F1(x))−W (2) G η3(G1(x))‖∞ + δ.
By applying the upper bound proved in equation B.19, we can upper bound the first part ‖W(2)F η3(F1(x))−W (2) G η3(F1(x))‖∞ by:
‖W(2)F η3(F1(x))−W (2) G η3(F1(x))‖∞ ≤ ‖W (2) F −W (2) G ‖∞‖η3(F1(x))‖∞
≤Wδ‖F1(x)‖3∞ ≤ δW [2(B ∨ d)2]3.
By applying the Lipschitz condition proved in Lemma B.10 and the bound proved in equation B.22, we can further upper bound the second part ‖W(2)G η3(F1(x))−W (2) G η3(G1(x))‖∞ by:
‖W(2)G η3(F1(x))−W (2) G η3(G1(x))‖∞ ≤ ‖W (2) G ‖∞‖η3(F1(x))− η3(G1(x))‖∞
≤WB × 3 sup F1∈Φ1(L,W,S,B) ‖F1(x)‖2∞ × ‖F1(x)−G1(x)‖∞
≤WB × 3[2(B ∨ d)2]2 × 2δ(B ∨ d) ≤ 24δW (B ∨ d)6.
Summing the two upper bounds above yields:
‖F2(x)−G2(x)‖∞ ≤ 8δW (B ∨ d)6 + 24δW (B ∨ d)6 + δ ≤ 24δW (B ∨ d)7.
where we again use the assumption d ≥ 2 in the last step. Inductive Step: Now we assume that the claim has been proved for k − 1, where k ≥ 3. For any x ∈ Ω and Fk ∈ Φk(L,W, S,B), we have that:
‖Fk(x)−Gk(x)‖∞ = ‖W(k)F η3(Fk−1(x)) + b (k) F −W (k) G η3(Gk−1(x))− b (k) G ‖∞
≤ ‖W(k)F η3(Fk−1(x))−W (k) G η3(Gk−1(x))‖∞ + ‖b (k) F − b (k) G ‖∞
≤ ‖W(k)F η3(Fk−1(x))−W (k) G η3(Gk−1(x))‖∞ + δ.
Applying triangle inequality helps us upper bound the first term above as follows:
‖W(k)F η3(Fk−1(x))−W (k) G η3(Gk−1(x))‖∞
≤ ‖W(k)F η3(Fk−1(x))−W (k) G η3(Fk−1(x))‖∞ + ‖W (k) G η3(Fk−1(x))−W (k) G η3(Gk−1(x))‖∞
≤ ‖W(k)F −W (k) G ‖∞‖η3(Fk−1(x))‖∞ + ‖W (k) G ‖∞‖η3(Fk−1(x))− η3(Gk−1(x))‖∞ ≤ δW‖Fk−1(x)‖3∞ +BW‖η3(Fk−1(x))− η3(Gk−1(x))‖∞.
From Lemma B.9, we can upper bound the first term δW‖Fk−1(x)‖3∞ by:
δW‖Fk−1(x)‖3∞ ≤ δW 3k−1−1 2 (B ∨ d) 5·3k−1−3 2 2 3k−3 2 −3k+6.
Moreover, applying Lemma B.10 and the inductive hypothesis let us upper bound the second term BW‖η3(Fk−1(x))− η3(Gk−1(x))‖∞ as follows:
BW‖η3(Fk−1(x))− η3(Gk−1(x))‖∞ ≤ BW × 3 sup
x∈Ω, Fk−1∈Φk−1(L,W,S,B) ‖Fk−1(x)‖2∞ × ‖Fk−1(x)−Gk−1(x)‖∞
≤ 3BW ×W 3 k−2−1(B ∨ d)5×3 k−2−123 k−1−1−2k+4‖Fk−1(x)−Gk−1(x)‖∞
≤ 3BW ×W 3 k−2−1(B ∨ d)5×3 k−2−123 k−1−1−2k+4 × δW 3k−2−1 2 (B ∨ d) 5·3k−2−1 2 2 3k−1−1 2 −k+23k−2
≤ 3k−1δW 3k−1−1 2 (B ∨ d) 5×3k−1−1 2 2 3k−1 2 −3k+5.
Combining the two upper bounds derived above yields:
‖Fk(x)−Gk(x)‖∞ ≤ δW 3k−1−1 2 (B ∨ d) 5·3k−1−3 2 2 3k−3 2 −3k+6
+ 3k−1δW 3k−1−1 2 (B ∨ d) 5×3k−1−1 2 2 3k−1 2 −3k+5 + δ
≤ δ3k−1W 3k−1−1 2 (B ∨ d) 5×3k−1−1 2 2 3k−1 2 −k+1,
where the last inequality above follows from k ≥ 3. Taking supremum with respect to x ∈ Ω on the LHS implies the given upper bound also holds for k. By induction, the claim is proved.
Theorem B.4. (Bounding the DNN space covering number) Fix some sufficiently large N ∈ Z+. Consider a Deep Neural Network space Φ(L,W, S,B) with L = O(1),W = O(N), S = O(N) and B = O(N). Then the log value of the covering number of this DNN space with respect to the inf-norm ‖F (x)‖∞ := supx∈Ω |F (x)|, which is denoted by N (δ,Φ(L,W,S,B), ‖ · ‖∞), can be upper bounded by:
logN (δ,Φ(L,W, S,B), ‖ · ‖∞) = O ( S [ log(δ−1) + 3L log(WB) ]) . (B.23)
Proof. We firstly fix a sparsity pattern (i.e, the locations of the non-zero entries are fixed). By picking k = L in Lemma B.11, we get the following upper bound on the covering number with respect to ‖ · ‖∞: ( δ
3L−1W 3L−1−1 2 (B ∨ d) 5×3 L−1−1 2 2 3L−1 2 −L+2
)−S .
Furthermore, note that the number of feasible configurations is upper bounded by ( (W+1)L
S
) ≤
(W + 1)LS .(Schmidt-Hieber et al., 2020; Farrell et al., 2021) Plug in the previous inequality and yields:
logN (δ,Φ(L,W, S,B), ‖ · ‖∞) ≤ log [ (W + 1)LS ( δ 3L−1W 3L−1−1 2 (B ∨ d) 5×3 L−1−1 2 2 3L−1 2 −L+1 )−S] ≤ S log [ δ−1(W + 1)L3L−1W 3L−1−1 2 (B ∨ d) 5×3L−1−1 2 2 3L−1 2 −L+1
] . S [ log(δ−1) + L log(3W ) + 3L log(W (B ∨ d)) + 3L log 2 ] .
Note that here the dimension d is some constant. Thus, by plugging in thee given magnitudes L = O(1),W = O(N), S = O(N) and B = O(N), we can further deduce that:
logN (δ,Φ(L,W, S,B), ‖ · ‖∞) . S [ log(δ−1) + 3L log(WB) ] .
This finishes our proof.
Now let’s consider upper bounding the covering number of the l2 norm of the sparse Deep Neural Networks’ gradients. Note that for any 1 ≤ k ≤ L − 1, any k−ReLU3 Deep Neural Network Fk ∈ Φk(L,W,S,B) is a map from Rd to RW . For any 1 ≤ l ≤ W , we use Fk,l(x) to denote the
l-th component of the map Fk. This helps us write the map Fk(x) and its Jacobian matrix J [Fk](x) explicitly as:
Fk(x) = [Fk,1(x), Fk,2(x), · · · , Fk,W (x)]T ∈ RW .
J [Fk](x) =  ∂ ∂x1 Fk,1(x) ∂ ∂x2 Fk,1(x) · · · ∂∂xdFk,1(x) ∂ ∂x1 Fk,2(x) ∂ ∂x2 Fk,2(x) · · · ∂∂xdFk,2(x)
· · · · · · . . .
∂ ∂x1 Fk,W (x) ∂ ∂x2 Fk,W (x) · · · ∂∂xdFk,W (x)  ∈ RW×d. In particular, when k = L, we have that any FL ∈ ΦL(L,W,S,B) = Φ(L,W, S,B) is a map from Rd to R. Thus, its Jacobian can be explicitly written as the following row vector:
J [FL](x) = [ ∂
∂x1 FL(x),
∂
∂x2 FL(x), · · ·
∂
∂xd FL(x)] ∈ R1×d.
Lemma B.12. (Upper bound on∞-norm of Jacobian/Gradient of elements in the DNN space) For any 1 ≤ k ≤ L, the following inequality holds:
sup x∈Ω,Fk∈Φk(L,W,S,B)
‖J [Fk](x)‖∞ ≤W 3k−1−1 2 (B ∨ d) 5·3k−1−1 2 2 3k−1 2 −k+13k−1.
Proof. We use induction on k to prove the claim. Base case: k = 1. By the definition of Jacobian matrix, we have that for any x ∈ Ω and any F1 ∈ Φ1(L,W, S,B), the following holds:
‖J [F1](x)‖∞ = ‖W(1)F ‖∞ ≤ dB ≤ 2(B ∨ d) 2.
Inductive Step: Assume that the claim has been proved for k − 1, where 2 ≤ k ≤ L. For any x ∈ Ω and any Fk ∈ Φk(L,W, S,B), by applying the Chain Rule, we can write the Jacobian matrix J [Fk](x) as J [Fk](x) =W(k)F J [η3 ◦ Fk−1](x), where the ReLU3 activation function η3 is applied to each component Fk−1,l (1 ≤ l ≤W ) of the map Fk. Then we have the following upper bound:
‖J [Fk](x)‖∞ ≤ ‖W(k)F ‖∞‖J [η3 ◦ Fk−1](x)‖∞ ≤WB‖J [η3 ◦ Fk−1](x)‖∞. (B.24)
Note that the composition η3 ◦ Fk−1 is a map from Rd to RW . Hence, the Jacobian matrix J [η3 ◦ Fk−1](x) is of shape RW×d. Applying the Chain Rule again implies:
‖J [η3 ◦ Fk−1](x)‖∞ = sup 1≤l≤W ( d∑ j=1 |3η2(Fk−1,l(x)) ∂Fk−1,l(x) ∂xj |).
Furthermore, for any 1 ≤ l ≤W , the summation on the RHS above can be upper bounded by: d∑ j=1 |3η2(Fk−1,l(x)) ∂Fk−1,l(x) ∂xj | ≤ 3‖Fk−1(x)‖2∞( d∑ j=1 | ∂ ∂xj Fk−1,l(x)|) ≤ 3‖Fk−1(x)‖2∞‖J [Fk−1](x)‖∞.
Now let’s take supremum with respect to l and apply the inductive hypothesis and Lemma B.9. This yields:
‖J [η3 ◦ Fk−1](x)‖∞ ≤ 3W 3 k−2−1(B ∨ d)5·3 k−2−123 k−1−1−2k+4 ×W 3k−2−1 2 (B ∨ d) 5·3k−2−1 2 2 3k−1−1 2 −k+23k−2
= W 3k−1−1 2 −1(B ∨ d) 5·3k−1−1 2 −12 3k−1
2 −3k+53k−1. (B.25)
By substituting equation B.25 into equation B.24, we can derive the final bound:
‖J [Fk](x)‖∞ ≤WB‖J [η3 ◦ Fk−1](x)‖∞ ≤W 3k−1−1 2 (B ∨ d) 5·3k−1−1 2 2 3k−1 2 −3k+53k−1
≤W 3k−1−1 2 (B ∨ d) 5·3k−1−1 2 2 3k−1 2 −k+13k−1.
where the last inequality above follows from k ≥ 2. Taking supremum with respect to x ∈ Ω and Fk ∈ Φk(L,W, S,B) on the LHS implies that the given upper bound also holds for k. By induction, the claim is proved.
For the convenience of the following proof, we first prove this lemma for vector 2 norm and∞ norm. Lemma B.13. Given any two row vectors u,v ∈ R1×d, we have:∣∣∣‖u‖ − ‖v‖∣∣∣ ≤ ‖u− v‖∞. Proof. Assume that the two vectors u,v ∈ Rd can be explicitly written as u = [u1, u2, · · · , ud] and v = [v1, v2, · · · , vd], respectively. By applying Cauchy-Schwarz inequality, we have:
∣∣∣‖u‖ − ‖v‖∣∣∣2 = ∣∣∣∣∣∣ √√√√ d∑
i=1
u2i − √√√√ d∑ i=1 v2i ∣∣∣∣∣∣ 2
= d∑ i=1 u2i + d∑ i=1 v2i − 2 √√√√ d∑ i=1 u2i √√√√ d∑ i=1 v2i ≤ d∑ i=1 u2i + d∑ i=1 v2i − 2 d∑ i=1 uivi = d∑ i=1 |ui − vi|2
≤ ( d∑ i=1 |ui − vi| )2 = ‖u− v‖2∞.
Taking the square root on both sides yields the desired inequality.
Then we upper bound the Lipschitz constant of the gradient of the neural network. Given a DNN space Φ(L,W, S,B), we define a corresponding DNN Gradient space∇Φ(L,W, S,B) as:
∇Φ(L,W,S,B) := {‖∇F‖ | F ∈ Φ(L,W,S,B)}. (B.26)
Lemma B.14. (Relation between the covering number of the DNN Gradient space and parameter space) For any 1 ≤ k ≤ L, suppose that a pair of different two networks Fk, Gk ∈ Φk(L,W, S,B) are given by:
Fk(x) := (W(k)F η3(·) + b (k) F ) · · · (W (1) F x+ b (1) F ),
Gk(x) := (W(k)G η3(·) + b (k) G ) · · · (W (1) G x+ b (1) G ).
Furthermore, assume that the ‖ ‖∞ norm of the distance between the parameter spaces is uniformly upper bounded by δ, i.e
‖W (l)F −W (l) G ‖∞,∞ ≤ δ, ‖b (l) F − b (l) G ‖∞ ≤ δ, (∀ 1 ≤ l ≤ k). (B.27)
Then we have:
sup x∈Ω ‖J [Fk](x)− J [Gk](x)‖∞ ≤ δW
3k−1−1 2 (B ∨ d) 5·3k−1−1 2 2 3k−1 2 −k+132k−2. (B.28)
In particular, when k = L, we have:
sup x∈Ω ∣∣∣‖∇FL(x)‖ − ‖∇GL(x)‖∣∣∣ ≤ δW 3L−1−12 (B ∨ d) 5·3L−1−12 2 3L−12 −L+132L−2. (B.29) Proof. We use induction on k to prove the claim. Base case: When k = 1, we have that for any x ∈ Ω and any F1, G1 ∈ Φ1(L,W,S,B), the following holds:
‖J [F1](x)− J [G1](x)‖∞ = ‖W(1)F −W (1) G ‖∞ ≤ δd ≤ 2δ(B ∨ d) 2.
Inductive Step: assume that the claim has been proved for k − 1, where 2 ≤ k ≤ L. Then for any x ∈ Ω and Fk, Gk ∈ Φk(L,W, S,B) satisfying constraint B.27, applying the Chain Rule and
triangle inequality help us upper bound the inf-norm ‖J [Fk](x)− J [Gk](x)‖∞ by:
‖J [Fk](x)− J [Gk](x)‖∞ = ‖W(k)F J [η3 ◦ Fk−1](x)−W (k) G J [η3 ◦Gk−1](x)‖∞
≤ ‖W(k)F J [η3 ◦ Fk−1](x)−W (k) G J [η3 ◦ Fk−1](x)‖∞ + ‖W (k) G J [η3 ◦ Fk−1](x)−W (k) G J [η3 ◦Gk−1](x)‖∞
≤ ‖W(k)F −W (k) G ‖∞‖J [η3 ◦ Fk−1](x)‖∞ + ‖W (k) G ‖∞‖J [η3 ◦ Fk−1](x)− J [η3 ◦Gk−1](x)‖∞ ≤ δW‖J [η3 ◦ Fk−1](x)‖∞ +BW‖J [η3 ◦ Fk−1](x)− J [η3 ◦Gk−1](x)‖∞. (B.30)
Using equation B.25 helps us upper bound the first term by:
δW‖J [η3 ◦ Fk−1](x)‖∞ ≤ δW 3k−1−1 2 (B ∨ d) 5·3k−1−1 2 −12 3k−1
2 −3k+53k−1. (B.31) Note that the two compositions η3 ◦ Fk−1 and η3 ◦Gk−1 both map from Rd to RW . Hence, the two Jacobian matrices J [η3 ◦ Fk−1](x) and J [η3 ◦Gk−1](x) are of shape RW×d. Applying the Chain Rule again implies:
‖J [η3 ◦ Fk−1](x)− J [η3 ◦Gk−1](x)‖∞ = sup 1≤l≤W ( d∑ j=1 |3η2(Fk−1,l(x)) ∂Fk−1,l(x) ∂xj − 3η2(Gk−1,l(x)) ∂Gk−1,l(x) ∂xj |).
For any 1 ≤ l ≤W , the summation on the RHS above can be upper bounded by: d∑ j=1 |3η2(Fk−1,l(x)) ∂Fk−1,l(x) ∂xj − 3η2(Gk−1,l(x)) ∂Gk−1,l(x) ∂xj |
≤ d∑ j=1 |3η2(Fk−1,l(x)) ∂Fk−1,l(x) ∂xj − 3η2(Gk−1,l(x)) ∂Fk−1,l(x) ∂xj |
+ d∑ j=1 |3η2(Gk−1,l(x)) ∂Fk−1,l(x) ∂xj − 3η2(Gk−1,l(x)) ∂Gk−1,l(x) ∂xj | ≤ d∑ j=1 |3η2(Fk−1,l(x))− 3η2(Gk−1,l(x))‖ ∂Fk−1,l(x) ∂xj |+ d∑ j=1 |3η2(Gk−1,l(x))‖ ∂Fk−1,l(x) ∂xj − ∂Gk−1,l(x) ∂xj |.
We denote the two summations above by T1 and T2, respectively:
T1 := d∑ j=1 |3η2(Fk−1,l(x))− 3η2(Gk−1,l(x))‖ ∂Fk−1,l(x) ∂xj |,
T2 := d∑ j=1 |3η2(Gk−1,l(x))‖ ∂Fk−1,l(x) ∂xj − ∂Gk−1,l(x) ∂xj |.
For the first sum T1, applying Lemma B.9, Lemma B.10, Lemma B.11 and Lemma B.12 yields the following upper bound:
T1 ≤ 6 (
sup x∈Ω, Fk−1∈Φk−1(L,W,S,B)
‖Fk−1(x)‖∞ ) ‖Fk−1(x)−Gk−1(x)‖∞ d∑ j=1 |∂Fk−1,l(x) ∂xj |
≤ 6 (
sup x∈Ω, Fk−1∈Φk−1(L,W,S,B)
‖Fk−1(x)‖∞ ) ‖Fk−1(x)−Gk−1(x)‖∞‖J [Fk−1](x)‖∞
≤ 3×W 3k−2−1 2 (B ∨ d) 5·3k−2−1 2 2 3k−1−1 2 −k+2 × δW 3k−2−1 2 (B ∨ d) 5·3k−2−1 2 2 3k−1−1 2 −k+23k−2
×W 3k−2−1 2 (B ∨ d) 5·3k−2−1 2 2 3k−1−1 2 −k+23k−2 = δW 3k−1−3 2 (B ∨ d) 5·3k−1−3 2 2 3k−3 2 −3k+632k−3.
For the second sum T2, applying Lemma B.9 and inductive hypothesis yields: T2 ≤ 3 (
sup x∈Ω, Gk−1∈Φk−1(L,W,S,B)
‖Gk−1(x)‖∞ )2 ‖J [Fk−1](x)− J [Gk−1](x)‖∞
≤ 3×W 3 k−2−1(B ∨ d)5·3 k−2−123 k−1−1−2k+4 × δW 3k−2−1 2 (B ∨ d) 5·3k−2−1 2 2 3k−1−1 2 −k+232k−4
= δW 3k−1−3 2 (B ∨ d) 5·3k−1−3 2 2 3k−3 2 −3k+632k−3.
Combining the two upper bounds on T1 and T2 yields: d∑ j=1 |3η2(Fk−1,l(x)) ∂Fk−1,l(x) ∂xj − 3η2(Gk−1,l(x)) ∂Gk−1,l(x) ∂xj |
≤ T1 + T2 ≤ 2× δW 3k−1−3 2 (B ∨ d) 5·3k−1−3 2 2 3k−3 2 −3k+632k−3.
By taking supremum with respect to 1 ≤ l ≤W on the LHS yields:
BW‖J [η3 ◦ Fk−1](x)− J [η3 ◦Gk−1](x)‖∞ ≤ δW 3k−1−1 2 (B ∨ d) 5·3k−1−1 2 2 3k−1
2 −3k+632k−3. (B.32)
By adding the two upper bounds in B.31 and B.32, we can deduce that:
‖J [Fk](x)− J [Gk](x)‖∞ ≤ δW 3k−1−1 2 (B ∨ d) 5·3k−1−1 2 −12 3k−1 2 −3k+53k−1
+ δW 3k−1−1 2 (B ∨ d) 5·3k−1−1 2 2 3k−1 2 −3k+632k−3
≤ δW 3k−1−1 2 (B ∨ d) 5·3k−1−1 2 2 3k−1 2 −k+132k−2.
where the last inequality above follows from k ≥ 2. Taking supremum with respect to x ∈ Ω on the LHS implies the given upper bound also holds for k. By induction, the claim is proved. In particular, when k = L, we have ∇FL(x) = J [FL](x)T for any x ∈ Ω. Applying Lemma B.13 then yields:
sup x∈Ω ∣∣∣‖∇FL(x)‖ − ‖∇GL(x)‖∣∣∣ = sup x∈Ω ∣∣∣‖∇J [FL](x)T ‖ − ‖∇J [GL](x)T ‖∣∣∣ ≤ sup x∈Ω ‖J [FL](x)− J [GL](x)‖∞
≤ δW 3L−1−1 2 (B ∨ d) 5·3L−1−1 2 2 3L−1 2 −k+132L−2.
This finishes our proof of the Lemma.
Theorem B.5. (Bounding the DNN Gradient space covering number) Fix some sufficiently largeN ∈ Z+. Consider a Deep Neural Network space Φ(L,W, S,B) with L = O(1),W = O(N), S = O(N) and B = O(N). Then the log value of the covering number of the DNN Gradient space with respect to the ‖ · ‖∞ norm ‖F (x)‖∞ := supx∈Ω |F (x)|, which is denoted byN (δ,∇Φ(L,W, S,B), ‖ · ‖∞), can be upper bounded by:
logN (δ,∇Φ(L,W, S,B), ‖ · ‖∞) = O ( S [ log(δ−1) + 3L log(WB) ]) . (B.33)
Proof. We firstly fix a sparsity pattern (i.e, the locations of the non-zero entries are fixed). Using equation B.29 in Lemma B.14, yields the following upper bound on the covering number with respect to ‖ · ‖∞: ( δ
W 3L−1−1 2 (B ∨ d) 5·3 L−1−1 2 2 3L−1 2 −L+132L−2
)−S .
Furthermore, note that the number of feasible configurations is upper bounded by: ( (W+1)L
S
) ≤
(W + 1)LS .(Schmidt-Hieber et al., 2020; Farrell et al., 2021). Plug this inequality into the previous estimation then yields:
logN (δ,Φ(L,W, S,B), ‖ · ‖∞) ≤ log [ (W + 1)LS ( δ W 3L−1−1 2 (B ∨ d) 5·3 L−1−1 2 2 3L−1 2 −L+132L−2 )−S] ≤ S log [ δ−1(W + 1)L32L−2W 3L−1−1 2 (B ∨ d) 5·3L−1−1 2 2 3L−1 2 −L+1
] . S [ log(δ−1) + 2L log(3W ) + 3L log(W (B ∨ d)) + 3L log 2 ] .
Note that here the dimension d is some constant. Thus, by plugging in thee given magnitudes L = O(1),W = O(N), S = O(N) and B = O(N), we can further deduce that:
logN (δ,Φ(L,W, S,B), ‖ · ‖∞) . S [ log(δ−1) + 3L log(WB) ] .
This finishes our proof.
Now let’s consider upper bounding the covering number of the Laplacian of the sparse Deep Neural Networks. Note that for any 1 ≤ k ≤ L − 1, any k−ReLU3 Deep Neural Network Fk ∈ Φk(L,W, S,B) is a vector-valued function mapping from Rd to RW . Moreover, we define the Laplacian of Fk(x), which is denoted by ∆[Fk](x), as follows:
∆[Fk](x) = [∆Fk,1(x),∆Fk,2(x), · · · ,∆Fk,W (x)]T ∈ RW , where for any 1 ≤ l ≤W , we have:
∆Fk,l(x) = d∑ j=1 ∂2 ∂x2j Fk,l(x).
In particular, when k = L, we have that any FL ∈ ΦL(L,W, S,B) = Φ(L,W,S,B) is a scalarvalued function mapping from Rd to R. Thus, its Laplacian can be explicitly written as:
∆[FL](x) = ∆FL(x) = d∑ j=1 ∂2 ∂x2j FL(x).
For both Lemma B.15 and Lemma B.16 below, we consider a fixed Deep Neural Network space Φ(L,W, S,B) with L = O(1),W = O(N), S = O(N) and B = O(N), where N ∈ Z+ is fixed and sufficiently large. Lemma B.15. (Upper bound on ∞-norm of Laplacian of elements in the DNN space) For any 1 ≤ k ≤ L, we have the following upper bound:
sup x∈Ω,Fk∈Φk(L,W,S,B)
‖∆[Fk](x)‖∞ = O ( W 3k−1−1 2 (B ∨ d) 5·3k−1−1 2 ) .
Proof. We use induction on k to prove the claim. Base case: k = 1. Note that any F1 ∈ Φ1(L,W, S,B) is a linear transform, so the Laplacian ∆[F1](x) must be the zero vector for any x ∈ Ω. This implies:
‖∆[F1](x)‖∞ = 0 . (B ∨ d)2. Inductive Step: Assume that the claim has been proved for k − 1, where 2 ≤ k ≤ L. For any x ∈ Ω and any Fk ∈ Φk(L,W, S,B), using linearity of the Laplacian operator implies:
∆[Fk](x) =W(k)F ∆[η3 ◦ Fk−1](x). Taking the inf-norm on both sides of the identity above implies:
‖∆[Fk](x)‖∞ ≤ ‖W(k)F ‖∞‖∆[η3 ◦ Fk−1](x)‖∞ ≤WB‖∆[η3 ◦ Fk−1](x)‖∞. It now remains to upper bound the term ‖∆[η3 ◦ Fk−1](x)‖∞. For any 1 ≤ l ≤W , we will use the Chain Rule to write the l-th component ( ∆[η3 ◦ Fk−1](x) ) l
in an explicit form. For any 1 ≤ j ≤ d, we have:
∂
∂xj η3[Fk−1,l(x)] = 3η2[Fk−1,l(x)]
∂
∂xj Fk−1,l(x).
Differentiating with respect to xj on both sides above yields:
∂2
∂x2j η3[Fk−1,l(x)] = 6η1[Fk−1,l(x)] ( ∂ ∂xj Fk−1,l(x) )2 + 3η2[Fk−1,l(x)] ∂2 ∂x2j Fk−1,l(x). (B.34)
Summing the expression above from j = 1 to j = d implies:∣∣∣(∆[η3 ◦ Fk−1](x)) l ∣∣∣ = ∣∣∣∣∣∣ d∑ j=1 ∂2 ∂x2j η3[Fk−1,l(x)] ∣∣∣∣∣∣ =
∣∣∣∣∣∣6η1[Fk−1,l(x)] d∑ j=1 ( ∂ ∂xj Fk−1,l(x) )2 + 3η2[Fk−1,l(x)] d∑ j=1 ∂2 ∂x2j Fk−1,l(x) ∣∣∣∣∣∣ ≤ 6 ∣∣∣η1[Fk−1,l(x)]∣∣∣
 d∑ j=1 ∣∣∣ ∂ ∂xj Fk−1,l(x) ∣∣∣ 2 + 3∣∣∣η2[Fk−1,l(x)]∣∣∣ ∣∣∣∣∣∣ d∑ j=1 ∂2 ∂x2j Fk−1,l(x) ∣∣∣∣∣∣ .
We denote the two summations above by U1 and U2, respectively:
U1 := 6 ∣∣∣η1[Fk−1,l(x)]∣∣∣  d∑ j=1 ∣∣∣ ∂ ∂xj Fk−1,l(x) ∣∣∣ 2 . U2 := 3 ∣∣∣η2[Fk−1,l(x)]∣∣∣
∣∣∣∣∣∣ d∑ j=1 ∂2 ∂x2j Fk−1,l(x) ∣∣∣∣∣∣ . On the one hand, by applying Lemma B.9 and Lemma B.12, we can upper bound U1 by:
U1 ≤ 6 (
sup x∈Ω, Fk−1∈Φk−1(L,W,S,B)
‖Fk−1(x)‖∞ )(
sup x∈Ω,Fk−1∈Φk−1(L,W,S,B)
‖J [Fk−1](x)‖∞ )2
≤ 6×W 3k−2−1 2 (B ∨ d) 5·3k−2−1 2 2 3k−1−1 2 −k+2 ×W 3 k−2−1(B ∨ d)5·3 k−2−123 k−1−1−2k+432k−4
.W 3k−1−3 2 (B ∨ d) 5·3k−1−3 2 ,
where the last step above follows from k ≤ L and L = O(1). On the other hand, by applying Lemma B.9 and the inductive hypothesis, we have:
U2 ≤ 3 (
sup x∈Ω, Fk−1∈Φk−1(L,W,S,B)
‖Fk−1(x)‖∞ )2 ‖∆[Fk−1](x)‖∞
. 3×W 3 k−2−1(B ∨ d)5·3 k−2−123 k−1−1−2k+4 ×W 3k−2−1 2 (B ∨ d) 5·3k−2−1 2
.W 3k−1−3 2 (B ∨ d) 5·3k−1−3 2 ,
where the last step above follows from k ≤ L and L = O(1). Summing the two bounds on U1 and U2 implies that for any 1 ≤ l ≤W , we have:∣∣∣(∆[η3 ◦ Fk−1](x))
l ∣∣∣ ≤ U1 + U2 .W 3k−1−32 (B ∨ d) 5·3k−1−32 . (B.35) Taking supremum with respect to 1 ≤ l ≤W then yields:
‖∆[Fk](x)‖∞ ≤WB‖∆[η3 ◦ Fk−1](x)‖∞ .W 3k−1−1 2 (B ∨ d) 5·3k−1−1 2 .
Taking supremum with respect to x ∈ Ω and Fk ∈ Φk(L,W,S,B) on the LHS implies that the given upper bound also holds for k. By induction, the claim is proved.
Lemma B.16. (Relation between the covering number of the DNN Laplacian space and parameter space) For any 1 ≤ k ≤ L, suppose that a pair of different two networks Fk, Gk ∈ Φk(L,W, S,B) are given by:
Fk(x) := (W(k)F η3(·) + b (k) F ) · · · (W (1) F x+ b (1) F ),
Gk(x) := (W(k)G η3(·) + b (k) G ) · · · (W (1) G x+ b (1) G ).
Furthermore, assume that the ‖ ‖∞ norm of the distance between the parameter spaces is uniformly upper bounded by δ, i.e
‖W (l)F −W (l) G ‖∞,∞ ≤ δ, ‖b (l) F − b (l) G ‖∞ ≤ δ, (∀ 1 ≤ l ≤ k). (B.36)
Then we have:
sup x∈Ω ‖∆[Fk](x)−∆[Gk](x)‖∞ = O
( δW 3k−1−1 2 (B ∨ d) 5·3k−1−1 2 ) . (B.37)
Proof. We use induction on k to prove the claim. Base case: k = 1. Note that any F1 ∈ Φ1(L,W, S,B) is a linear transform, so the Laplacian ∆[F1](x) must be the zero vector for any x ∈ Ω. Hence, for any x ∈ Ω and any F1, G1 ∈ Φ1(L,W, S,B), we have:
‖∆[F1](x)−∆[G1](x)‖∞ = 0 . δ(B ∨ d)2.
Inductive Step: assume that the claim has been proved for k − 1, where 2 ≤ k ≤ L. Then for any x ∈ Ω and Fk, Gk ∈ Φk(L,W, S,B) satisfying constraint B.36, applying linearity of the Laplacian operator indicates:
‖∆[Fk](x)−∆[Gk](x)‖∞ = ‖W(k)F ∆[η3 ◦ Fk−1](x)−W (k) G ∆[η3 ◦Gk−1](x)‖∞ = ∥∥∥(W(k)F −W(k)G )∆[η3 ◦ Fk−1](x)∥∥∥∞
+ ∥∥∥W(k)G (∆[η3 ◦ Fk−1](x)−∆[η3 ◦Gk−1](x))∥∥∥∞ ≤ ‖W(k)F −W (k) G ‖∞‖∆[η3 ◦ Fk−1](x)‖∞
+ ‖W(k)G ‖∞‖∆[η3 ◦ Fk−1](x)−∆[η3 ◦Gk−1](x)‖∞.
For the first term ‖W(k)F −W (k) G ‖∞‖∆[η3 ◦ Fk−1](x)‖∞, applying the bound in equation B.35 and equation B.36 yields:
‖W(k)F −W (k) G ‖∞‖∆[η3 ◦ Fk−1](x)‖∞ . δW ×W
3k−1−3 2 (B ∨ d) 5·3k−1−3 2
= δW 3k−1−1 2 (B ∨ d) 5·3k−1−3 2 . (B.38)
For the second term ‖W(k)G ‖∞‖∆[η3 ◦ Fk−1](x) −∆[η3 ◦ Gk−1](x)‖∞, we need to upper bound the norm ‖∆[η3 ◦ Fk−1](x) −∆[η3 ◦ Gk−1](x)‖∞ at first. Note that for any 1 ≤ l ≤ W , we can use equation B.34 to write the l-th component of ∆[η3 ◦ Fk−1](x)−∆[η3 ◦Gk−1](x) as:(
∆[η3 ◦ Fk−1](x)−∆[η3 ◦Gk−1](x) ) l = d∑ j=1 ∂2 ∂x2j η3[Fk−1,l(x)]− d∑ j=1 ∂2 ∂x2j η3[Gk−1,l(x)]
= 6η1[Fk−1,l(x)] d∑ j=1 ( ∂ ∂xj Fk−1,l(x) )2 − 6η1[Gk−1,l(x)] d∑ j=1 ( ∂ ∂xj Gk−1,l(x) )2
+ 3η2[Fk−1,l(x)] d∑ j=1 ∂2 ∂x2j Fk−1,l(x)− 3η2[Gk−1,l(x)] d∑ j=1 ∂2 ∂x2j Gk−1,l(x)
= 6η1[Fk−1,l(x)] d∑ j=1 ( ∂ ∂xj Fk−1,l(x) )2 − 6η1[Gk−1,l(x)] d∑ j=1 ( ∂ ∂xj Fk−1,l(x) )2
+ 6η1[Gk−1,l(x)] d∑ j=1 ( ∂ ∂xj Fk−1,l(x) )2 − 6η1[Gk−1,l(x)] d∑ j=1 ( ∂ ∂xj Gk−1,l(x) )2
+ 3η2[Fk−1,l(x)] d∑ j=1 ∂2 ∂x2j Fk−1,l(x)− 3η2[Gk−1,l(x)] d∑ j=1 ∂2 ∂x2j Fk−1,l(x)
+ 3η2[Gk−1,l(x)] d∑ j=1 ∂2 ∂x2j Fk−1,l(x)− 3η2[Gk−1,l(x)] d∑ j=1 ∂2 ∂x2j Gk−1,l(x).
We denote the four summations above by V1, V2, V3 and V4, respectively:
V1 := 6η1[Fk−1,l(x)] d∑ j=1 ( ∂ ∂xj Fk−1,l(x) )2 − 6η1[Gk−1,l(x)] d∑ j=1 ( ∂ ∂xj Fk−1,l(x) )2 ,
V2 := 6η1[Gk−1,l(x)] d∑ j=1 ( ∂ ∂xj Fk−1,l(x) )2 − 6η1[Gk−1,l(x)] d∑ j=1 ( ∂ ∂xj Gk−1,l(x) )2 ,
V3 := 3η2[Fk−1,l(x)] d∑ j=1 ∂2 ∂x2j Fk−1,l(x)− 3η2[Gk−1,l(x)] d∑ j=1 ∂2 ∂x2j Fk−1,l(x),
V4 := 3η2[Gk−1,l(x)] d∑ j=1 ∂2 ∂x2j Fk−1,l(x)− 3η2[Gk−1,l(x)] d∑ j=1 ∂2 ∂x2j Gk−1,l(x).
By applying Lemma B.9, Lemma B.10, Lemma B.11 and Lemma B.12, we can upper bound V1 by:
V1 = 6 ( η1[Fk−1,l(x)]− η1[Gk−1,l(x)] ) d∑ j=1 ( ∂ ∂xj Fk−1,l(x) )2
≤ 6|Fk−1,l(x)−Gk−1,l(x)|  d∑ j=1 ∣∣∣ ∂ ∂xj Fk−1,l(x) ∣∣∣ 2 ≤ 6‖Fk−1(x)−Gk−1(x)‖∞‖J [Fk−1](x)‖2∞
. δW 3k−2−1 2 (B ∨ d) 5·3k−2−1 2 2 3k−1−1 2 −k+23k−2 ×W 3 k−2−1(B ∨ d)5·3 k−2−123 k−1−1−2k+432k−4
. δW 3k−1−3 2 (B ∨ d) 5·3k−1−3 2 .
where the last step above follows from k ≤ L and L = O(1). Furthermore, note that for any 1 ≤ j ≤ d, we can upper bound the difference ( ∂ ∂xj Fk−1,l(x) )2 −(
∂ ∂xj
Gk−1,l(x) )2
as follows:( ∂ ∂xj Fk−1,l(x) )2 − ( ∂ ∂xj Gk−1,l(x) )2 ≤ ∣∣∣∣( ∂∂xj Fk−1,l(x) )2 − ( ∂ ∂xj Gk−1,l(x) )2∣∣∣∣
= ∣∣∣∣ ∂∂xj Fk−1,l(x) + ∂∂xjGk−1,l(x) ∣∣∣∣ ∣∣∣∣ ∂∂xj Fk−1,l(x)− ∂∂xjGk−1,l(x) ∣∣∣∣ ≤ (∣∣∣ ∂ ∂xj Fk−1,l(x) ∣∣∣+ ∣∣∣ ∂ ∂xj Gk−1,l(x) ∣∣∣) ∣∣∣∣ ∂∂xj Fk−1,l(x)− ∂∂xjGk−1,l(x) ∣∣∣∣ .
(B.39)
Note that η1(Gk−1,l(x)) ≥ 0. Combining the non-negativity with equation B.39, Lemma B.9, Lemma B.12 and Lemma B.14 helps us upper bound V2 by:
V2 = 6η1[Gk−1,l(x)] d∑ j=1 [( ∂ ∂xj Fk−1,l(x) )2 − ( ∂ ∂xj Gk−1,l(x) )2]
≤ 6‖Gk−1(x)‖∞ d∑ j=1 (∣∣∣ ∂ ∂xj Fk−1,l(x) ∣∣∣+ ∣∣∣ ∂ ∂xj Gk−1,l(x) ∣∣∣) ∣∣∣∣ ∂∂xj Fk−1,l(x)− ∂∂xjGk−1,l(x) ∣∣∣∣
≤ 6‖Gk−1(x)‖∞  d∑ j=1 ∣∣∣ ∂ ∂xj Fk−1,l(x) ∣∣∣+ d∑ j=1 ∣∣∣ ∂ ∂xj Gk−1,l(x) ∣∣∣  d∑ j=1 ∣∣∣ ∂ ∂xj Fk−1,l(x)− ∂ ∂xj Gk−1,l(x) ∣∣∣
≤ 6‖Gk−1(x)‖∞ ( ‖J [Fk−1](x)‖∞ + ‖J [Gk−1](x)‖∞ )∥∥∥J [Fk−1](x)− J [Gk−1](x)∥∥∥ ∞ ≤ 6W 3k−2−1 2 (B ∨ d) 5·3k−2−1 2 2 3k−1−1 2 −k+2 × 2W 3k−2−1 2 (B ∨ d) 5·3k−2−1 2 2 3k−1−1 2 −k+23k−2
× δW 3k−2−1 2 (B ∨ d) 5·3k−2−1 2 2 3k−1−1 2 −k+132k−4 . δW 3k−1−3 2 (B ∨ d) 5·3k−1−3 2 .
where the last step above follows from k ≤ L and L = O(1). Moreover, using Lemma B.9, Lemma B.10 and Lemma B.15 helps us upper bound V3 by:
V3 = ( 3η2[Fk−1,l(x)]− 3η2[Gk−1,l(x)] ) d∑ j=1 ∂2 ∂x2j Fk−1,l(x)
≤ ∣∣∣3η2[Fk−1,l(x)]− 3η2[Gk−1,l(x)]∣∣∣∣∣∣ d∑
j=1
∂2
∂x2j Fk−1,l(x) ∣∣∣ ≤ 6 (
sup x∈Ω, Fk−1∈Φk−1(L,W,S,B)
‖Fk−1(x)‖∞ ) ‖Fk−1(x)−Gk−1(x)‖∞‖∆[Fk−1](x)‖∞
. 6W 3k−2−1 2 (B ∨ d) 5·3k−2−1 2 2 3k−1−1 2 −k+2 × δW 3k−2−1 2 (B ∨ d) 5·3k−2−1 2 2 3k−1−1 2 −k+23k−2
×W 3k−2−1 2 (B ∨ d) 5·3k−2−1 2 . δW 3k−1−3 2 (B ∨ d) 5·3k−1−3 2 .
where the last step above follows from k ≤ L and L = O(1). Finally, applying Lemma B.9 and inductive hypothesis helps us upper bound V4 by:
V4 = 3η2[Gk−1,l(x)] ( d∑ j=1 ∂2 ∂x2j Fk−1,l(x)− d∑ j=1 ∂2 ∂x2j Gk−1,l(x) ) ≤ 3‖Gk−1(x)‖2∞‖∆[Fk−1](x)−∆[Gk−1](x)‖∞
. 3W 3 k−2−1(B ∨ d)5·3 k−2−123 k−1−1−2k+4 × δW 3k−2−1 2 (B ∨ d) 5·3k−2−1 2
. δW 3k−1−3 2 (B ∨ d) 5·3k−1−3 2 .
where the last step above follows from k ≤ L and L = O(1). Combining the four bounds on V1, V2, V3 and V4 implies:(
∆[η3 ◦ Fk−1](x)−∆[η3 ◦Gk−1](x) ) l = 4∑ i=1 Vi . δW 3k−1−3 2 (B ∨ d) 5·3k−1−3 2 .
Taking supremum with respect to 1 ≤ l ≤ W gives us an upper bound on the second term ‖W(k)G ‖∞‖∆[η3 ◦ Fk−1](x)−∆[η3 ◦Gk−1](x)‖∞:
‖W(k)G ‖∞‖∆[η3 ◦ Fk−1](x)−∆[η3 ◦Gk−1](x)‖∞ .WB × δW 3k−1−3 2 (B ∨ d) 5·3k−1−3 2
= δW 3k−1−1 2 (B ∨ d) 5·3k−1−1
2 . (B.40)
Combining the two bounds derived in equation B.38 and equation B.40 then implies:
‖∆[Fk](x)−∆[Gk](x)‖∞ . δW 3k−1−1 2 (B ∨ d) 5·3k−1−3 2 + δW 3k−1−1 2 (B ∨ d) 5·3k−1−1 2
. δW 3k−1−1 2 (B ∨ d) 5·3k−1−1 2 .
Taking supremum with respect to x ∈ Ω on the LHS implies that the given upper bound also holds for k. By induction, the claim is proved.
Given a Neural Network function space Φ(L,W, S,B), we define a corresponding Neural Network Laplacian space ∆Φ(L,W, S,B) as:
∆Φ(L,W, S,B) := {∆F | F ∈ Φ(L,W, S,B)}. (B.41) Theorem B.6. (Bounding the Neural Network Laplacian space covering number) Fix some sufficiently large N ∈ Z+. Consider a Deep Neural Network space Φ(L,W, S,B) with L = O(1),W = O(N), S = O(N) and B = O(N). Then the log value of the covering number of the DNN Laplacian space with respect to the ‖ · ‖∞ norm ‖F (x)‖∞ := supx∈Ω |F (x)|, which is denoted by N (δ,∆Φ(L,W,S,B), ‖ · ‖∞), can be upper bounded by:
logN (δ,∆Φ(L,W, S,B), ‖ · ‖∞) = O ( S [ log(δ−1) + 3L log(WB) ]) . (B.42)
Proof. We firstly fix a sparsity pattern (i.e, the locations of the non-zero entries are fixed). Applying Lemma B.16 yields that there exists some constant C = O(1), such that the covering number with respect to ‖ · ‖∞ can be upper bounded by:( δ
CW 3L−1−1 2 (B ∨ d) 5·3 L−1−1 2
)−S .
Furthermore, note that the number of feasible configurations is upper bounded by ( (W+1)L
S
) ≤
(W + 1)LS(Schmidt-Hieber et al., 2020; Farrell et al., 2021). Then we plug this into the pervious estimation and yields:
logN (δ,Φ(L,W,S,B), ‖ · ‖∞) ≤ log [ (W + 1)LS ( δ CW 3L−1−1 2 (B ∨ d) 5·3 L−1−1 2 )−S] ≤ S log [ δ−1(W + 1)LW 3L−1−1 2 (B ∨ d) 5·3L−1−1 2
] . S [ log(δ−1) + L log(W ) + 3L log(W (B ∨ d)) ] .
Note that here the dimension d is some constant. Thus, by plugging in thee given magnitudes L = O(1),W = O(N), S = O(N) and B = O(N), we can further deduce that:
logN (δ,∆Φ(L,W, S,B), ‖ · ‖∞) . S [ log(δ−1) + 3L log(WB) ] .
This finishes our proof.
Lemma B.17 (Local Rademacher Complexity Bound of DNN Estimator for Deep Ritz Method). Consider a Deep Neural Network space F (Ω) = Φ(L,W,S,B) with L = O(1),W = O(N), S = O(N) and B = O(N), where N ∈ Z+ is fixed to be sufficiently large. Moreover, assume that the gradients and function value of F (Ω), u∗, V and f are uniformly bounded
max {
sup u∈F (Ω) ‖u‖L∞(Ω), sup u∈F (Ω)
‖∇u‖L∞(Ω), ‖u∗‖L∞(Ω), ‖∇u∗‖L∞(Ω), Vmax, ‖f‖L∞(Ω) } ≤ C.
(B.43) For any ρ > 0, we consider a localized set Lρ defined by:
Lρ(Ω) := {u : u ∈ F (Ω), ‖u− u∗‖2H1 ≤ ρ}.
Then for any ρ & n−2, the Rademacher complexity of a localized function space Sρ(Ω) :={ h := |Ω| · [ 1 2 ( ‖∇u‖2 − ‖∇u∗‖2 ) + 12V (|u| 2 − |u∗|2)− f(u− u∗) ] ∣∣∣ u ∈ Lρ(Ω)} can be upper bounded by a sub-root function
φ(ρ) := O
(√ S3Lρ
n log (BWn)
) .
i.e. we have φ(4ρ) ≤ 2φ(ρ) and Rn(Sρ(Ω)) ≤ φ(ρ). (B.44)
holds for all ρ & n−2.
Proof. Firstly, we will check that for any u ∈ Lρ(Ω), the corresponding function h in Sρ(Ω) is Lipschitz with respect to u − u∗ and ‖∇u‖ − ‖∇u∗‖. Note that for any u1, u2 ∈ Lρ(Ω) with corresponding functions h1, h2 ∈ Sρ(Ω), applying boundedness condition B.43 yields:
|h1(x)− h2(x)| ≤ 1
2 ∣∣∣‖∇u1(x)‖2 − ‖∇u2(x)‖2∣∣∣+ 1 2 |V (x)||u1(x)2 − u2(x)2|+ |f(x)||u1(x)− u2(x)|
≤ C ∣∣∣‖∇u1(x)‖ − ‖∇u2(x)‖∣∣∣+ (C2 + C)|u1(x)− u2(x)|
= C ∣∣∣(‖∇u1(x)‖ − ‖∇u∗(x)‖)− (‖∇u2(x)‖ − ‖∇u∗(x)‖)∣∣∣
+ (C2 + C) ∣∣∣(u1(x)− u∗(x))− (u2(x)− u∗(x))∣∣∣.
Let’s pick L = C2 + C > C. Applying the Talagrand Contraction Lemma B.2 helps us upper bound the local Rademacher complexity Rn(Sρ(Ω)) by
Rn(Sρ(Ω)) = ExEσ [ sup
u∈Lρ(Ω)
1
n n∑ i=1 σi [1 2 ( ‖∇u‖2 − ‖∇u∗‖2 ) + 1 2 V (|u|2 − |u∗|2)− f(u− u∗)
]]
≤ 2LExEσ [ sup
u∈Lρ(Ω)
1
n n∑ i=1 σi ( u(xi)− u∗(xi)
)]
+ 2LEx′Eσ′ [
sup u∈Lρ(Ω)
1
n n∑ i=1 σ′i ( ‖∇u(x′i)‖ − ‖∇u∗(x′i)‖ )] . Rn ({ u− u∗ : u ∈ Lρ }) +Rn ({ ‖∇u‖ − ‖∇u∗‖ : u ∈ Lρ }) .
From the localization constraint ρ ≥ ‖u−u∗‖2H1(Ω) = ‖u−u ∗‖2L2(Ω) + ‖∇u−∇u ∗‖2L2(Ω), we can deduce that
‖u− u∗‖L2(Ω) ≤ √ ρ and ‖∇u−∇u∗‖L2(Ω) ≤ √ ρ. (B.45)
Moreover, note that Ω ⊂ [0, 1]d. Applying triangle inequality yields:∥∥∥‖∇u‖ − ‖∇u∗‖∥∥∥2 L2(Ω) = ∫ Ω ∣∣∣‖∇u(x)‖ − ‖∇u∗(x)‖∣∣∣2dx ≤ ∫ Ω ‖∇u(x)−∇u∗(x)‖2dx
= ‖∇u−∇u∗‖2L2(Ω) ≤ ρ⇒ ∥∥∥‖∇u‖ − ‖∇u∗‖∥∥∥
L2(Ω) ≤ √ρ.
(B.46) Using inequality B.45 and inequality B.46, we have:
Rn(Sρ(Ω)) . Rn ({ u− u∗ : u ∈ Lρ }) +Rn ({ ‖∇u‖ − ‖∇u∗‖ : u ∈ Lρ }) ≤ Rn ({ u− u∗ : u ∈ Φ(L,W,S,B), ‖u− u∗‖L2(Ω) ≤ √ ρ })
+Rn
({ ‖∇u‖ − ‖∇u∗‖ : u ∈ Φ(L,W, S,B), ∥∥∥‖∇u‖ − ‖∇u∗‖∥∥∥ L2(Ω) ≤ √ρ }) .
(Geer & van de Geer, 2000; Rakhlin et al., 2017) showed a “upper isometry” property, where the metric ‖ · ‖L2 is equivalent to ‖ · ‖n,2 with high probability. Combining this fact with Theorem B.3, we can bound the local Rademacher complexities using Dudley integral:
Rn(Sρ(Ω)) . Rn ({ u− u∗ : u ∈ Φ(L,W,S,B), ‖u− u∗‖L2(Ω) ≤ √ ρ })
+Rn
({ ‖∇u‖ − ‖∇u∗‖ : u ∈ Φ(L,W, S,B), ∥∥∥‖∇u‖ − ‖∇u∗‖∥∥∥ L2(Ω) ≤ √ρ })
≤ Rn ({ u− u∗ : u ∈ Φ(L,W,S,B), ‖u− u∗‖n,2 ≤ 2 √ ρ })
+Rn
({ ‖∇u‖ − ‖∇u∗‖ : u ∈ Φ(L,W, S,B), ∥∥∥‖∇u‖ − ‖∇u∗‖∥∥∥ n,2 ≤ 2√ρ }) . inf
0<α<2 √ ρ
{ 4α+
12√ n ∫ 2√ρ α √ logN (δ,Φ(L,W, S,B), ‖ · ‖n,2)dδ } + inf
0<α<2 √ ρ
{ 4α+
12√ n ∫ 2√ρ α √ logN (δ,∇Φ(L,W,S,B), ‖ · ‖n,2)dδ } . inf
0<α<2 √ ρ
{ 4α+
12√ n ∫ 2√ρ α √ logN (δ,Φ(L,W, S,B), ‖ · ‖∞)dδ } + inf
0<α<2 √ ρ
{ 4α+
12√ n ∫ 2√ρ α √ logN (δ,∇Φ(L,W,S,B), ‖ · ‖∞)dδ } .
For any ρ & 1n2 , we pick α = 1 n .
√ ρ and plug in the upper bounds proved in Theorem B.4 and
Theorem B.5, which implies:
Rn(Sρ(Ω)) . 1
n + 1√ n ∫ 2√ρ 1 n √ S [ log(δ−1) + 3L log(WB) ] dδ + 1√ n ∫ 2√ρ 1 n √ S [ log(δ−1) + 3L log(WB) ] dδ
.
√ S3Lρ
n log (BWn).
Lemma B.18 (Local Rademacher Complexity Bound of DNN Estimator for Physics Informed Neural Network). Consider a Deep Neural Network space F (Ω) = Φ(L,W, S,B) with L = O(1),W = O(N), S = O(N) and B = O(N), where N ∈ Z+ is fixed to be sufficiently large. Moreover, assume that the gradients and function value of F (Ω), u∗, V and f are uniformly bounded
max {
sup u∈F (Ω) ‖u‖L∞(Ω), sup u∈F (Ω)
‖∆u‖L∞(Ω), ‖u∗‖L∞(Ω), ‖∆u∗‖L∞(Ω), Vmax, ‖f‖L∞(Ω) } ≤ C.
(B.47) For any ρ > 0, we consider a localized set Mρ defined by:
Mρ(Ω) := {u : u ∈ F (Ω), ‖u− u∗‖2H2 ≤ ρ}.
Then for any ρ & n−2, the Rademacher complexity of a localized function space Tρ(Ω) := { h :=
|Ω| · [ (∆u− V u+ f)2 − (∆u∗ − V u∗ + f)2 ] ∣∣∣ u ∈Mρ(Ω)} can be upper bounded by a sub-root function
φ(ρ) := O
(√ S3Lρ
n log (BWn)
) .
i.e. we have
φ(4ρ) ≤ 2φ(ρ) and Rn(Tρ(Ω)) ≤ φ(ρ). (B.48)
holds for all ρ & n−2.
Proof. Firstly, we will check that for any u ∈ Lρ(Ω), the corresponding function h in Sρ(Ω) is Lipschitz with respect to u−u∗ and ∆u−∆u∗. Note that for any u1, u2 ∈ Lρ(Ω) with corresponding functions h1, h2 ∈ Sρ(Ω), applying boundedness condition B.47 yields:
|h1(x)− h2(x)| ≤ |∆u1 −∆u2 − V (u1 − u2)||∆u1 − V u1 + ∆u2 − V u2 + 2f | ≤ (2C2 + 4C) (|∆u1 −∆u2|+ C|u1 − u2|)
= (2C2 + 4C) ∣∣∣(∆u1(x)−∆u∗(x))− (∆u2(x)−∆u∗(x))∣∣∣
+ (2C3 + 4C2) ∣∣∣(u1(x)− u∗(x))− (u2(x)− u∗(x))∣∣∣.
Let’s pick L = max{2C2 +4C, 2C3 +4C2}. Applying the Talagrand Contraction Lemma B.2 helps us upper bound the local Rademacher complexity Rn(Tρ(Ω)) by
Rn(Tρ(Ω)) = ExEσ [
sup u∈Mρ(Ω)
1
n n∑ i=1 σi [ (∆u− V u+ f)2 − (∆u∗ − V u∗ + f)2 ] ] ≤ 2LExEσ [ sup
u∈Mρ(Ω)
1
n n∑ i=1 σi ( u(xi)− u∗(xi)
)]
+ 2LEx′Eσ′ [
sup u∈Mρ(Ω)
1
n n∑ i=1 σ′i ( ∆u(x′i)−∆u∗(x′i) )] . Rn ({ u− u∗ : u ∈Mρ }) +Rn ({ ∆u−∆u∗ : u ∈Mρ
}) . Rn ({ u− u∗ : u ∈ Φ(L,W, S,B), ‖u− u∗‖L2(Ω) ≤ √ ρ })
+Rn ({ ∆u−∆u∗ : u ∈ Φ(L,W, S,B), ‖∆u−∆u∗‖L2(Ω) ≤ √ ρ })
≤ Rn ({ u− u∗ : u ∈ Φ(L,W, S,B), ‖u− u∗‖n,2 ≤ 2 √ ρ })
+Rn ({ ∆u−∆u∗ : u ∈ Φ(L,W, S,B), ‖∆u−∆u∗‖n,2 ≤ 2 √ ρ })
. inf 0<α<2 √ ρ
{ 4α+
12√ n ∫ 2√ρ α √ logN (δ,Φ(L,W,S,B), ‖ · ‖n,2)dδ } + inf
0<α<2 √ ρ
{ 4α+
12√ n ∫ 2√ρ α √ logN (δ,∆Φ(L,W, S,B), ‖ · ‖n,2)dδ } . inf
0<α<2 √ ρ
{ 4α+
12√ n ∫ 2√ρ α √ logN (δ,Φ(L,W,S,B), ‖ · ‖∞)dδ } + inf
0<α<2 √ ρ
{ 4α+
12√ n ∫ 2√ρ α √ logN (δ,∆Φ(L,W, S,B), ‖ · ‖∞)dδ } .
For any ρ & 1n2 , we pick α = 1 n .
√ ρ and plug in the upper bounds proved in Theorem B.4 and
Theorem B.5, which implies:
Rn(Tρ(Ω)) . 1
n + 1√ n ∫ 2√ρ 1 n √ S [ log(δ−1) + 3L log(WB) ] dδ + 1√ n ∫ 2√ρ 1 n √ S [ log(δ−1) + 3L log(WB) ] dδ
.
√ S3Lρ
n log (BWn).

heading: B.3 AUXILIARY DEFINITIONS AND LEMMATA ON APPROXIMATION ERROR

heading: B.3.1 APPROXIMATION USING TRUNCATED FOURIER BASIS

text:
Lemma B.19. Given α > 0 and a fixed integer ξ ∈ Z+. For any function f ∈ Hα(Ω) , we let fξ = ∑ ‖z‖∞≤ξ fzφz be the best approximation of f in the space Fξ(Ω). Then for any 0 < β ≤ α, we have the following inequality:
‖f − fξ‖2Hβ(Ω) ≤ ξ −2(α−β)‖f‖2Hα .
Proof. For f ∈ Hα(Ω), we know the Fourier coefficient satisfies∑ ‖z‖∞≥ξ |fz|2‖z‖2α . ‖f‖2Hα .
We directly construct fξ = ∑ ‖z‖∞≤ξ fzφz to be the truncated Fourier series of the function f , then we have ‖f − fξ‖2Hβ(Ω) .
∑ ‖z‖∞≥ξ |fz|2‖z‖2β ≤ ξ−2(α−β) ∑ ‖z‖∞≥ξ |fz|2‖z‖2α ≤ ξ−2(α−β)‖f‖2Hα .

heading: B.3.2 APPROXIMATION USING NEURAL NETWORK

text:
In this section, we aim to provide approximation bound for deep neural network. Our proof of the approximation upper bound is based on the observation that the B-spline approximation(De Boor & De Boor, 1978; Schumaker, 2007) can be formulated as a ReLU3 neural network efficiently(Suzuki, 2018; Gühring et al., 2020; Duan et al., 2021; Jiao et al., 2021a). Although the proof of the approximation of the neural network to the Sobolev spaces is a standard approach, we still demonstrate the proof sketch here. Definition B.4. (Univariate and Multivariate B-splines) Fix an arbitrary integer l ∈ Z+. Consider a corresponding uniform partition πl of [0, 1]:
πl : 0 = t (l) 0 < t (l) 1 < · · · < t (l) l−1 < t (l) l = 1,
where t(l)i = i l (∀ 0 ≤ i ≤ l). Now for any k ∈ Z +, we can define an extended partition πl,k as:
πl,k : t (l) −k+1 = · · · t (l) −1 = 0 = t (l) 0 < t (l) 1 < · · · < t (l) l−1 < t (l) l = 1 = t (l) l+1 = · · · = t (l) l+k−1.
Based on the extended partition πl,k, the univariate B-splines of order k with respect to partition πl are defined by:
N (k) l,i (x) := (−1) k(t (l) i+k− t (l) i ) · [ t (l) i , · · · , t (l) i+k ] max{(x− t), 0}k−1, x ∈ [0, 1], i ∈ Il,k, (B.49)
where Il,k = {−k + 1,−k + 2, · · · , l − 1} and [ t (l) i , · · · , t (l) i+k] denotes the divided difference operator. Equivalently, for any x ∈ [0, 1], we can rewrite the univariate B-splines N (k)l,i (x) in an explicit form:
N (k) l,i (x) =  lk−1 (k−1)! ∑k j=0(−1)j ( k j ) max { x− i+jl , 0 }k−1 , (0 ≤ i ≤ l − k + 1)∑k−1 j=0 aij max { x− jl , 0 }k−1 + ∑k−2 n=1 binx n + bi0, (−k + 1 ≤ i ≤ 0)∑l j=l−k+1 cij max { x− jl , 0 }k−1 , (l − k + 1 ≤ i ≤ l − 1)
(B.50)
where {aij | − k + 1 ≤ i ≤ 0, 0 ≤ j ≤ k − 1}, {bin | − k + 1 ≤ i ≤ 0, 1 ≤ n ≤ k − 2} and {cij | l − k + 1 ≤ i ≤ l − 1, l − k + 1 ≤ j ≤ l − 1} are some fixed constants. For any index vector i = (i1, i2, · · · , id) ∈ Idl,k, we can define a corresponding multivariate B-spline as a product of univariate B-splines:
N (k) l,i (x) := Π d j=1N (k) l,ij (xj). (B.51)
Definition B.5. (Interpolation Operator(Schumaker, 2007)) Take some domain Ω ⊂ [0, 1]d and two arbitrary integers k, l ∈ Z+. Consider the extended partition πl,k and the corresponding set of multivariate B-splines {N (k)l,i (x)}i∈Idl,k defined in Definition B.4. For any i ∈ I d l,k, we define the domain Ωi := {x ∈ Ω : xj ∈ [tij , tij+k], 1 ≤ j ≤ d}. There exists a set of linear functionals {λi}i∈Idk,l , where λi : L 1(Ω)→ R (∀ i ∈ Idk,l), such that for any i ∈ Idk,l and p ∈ [1,∞], we have:
λi(N (k) l,j ) = δi,j and |λi(f)| ≤ 9 d(k−1)(2k + 1)d (k l )− dp ‖f‖Lp(Ωi), ∀ f ∈ Lp(Ω). (B.52) The corresponding interpolation operator Qk,l is defined as:
Qk,lf := ∑ i∈Idk,l λi(f)N (k) l,i , ∀ f ∈ L 1(Ω).
Theorem B.7. [(Schumaker, 2007)] Fix f ∈ W s(Ω) with Ω ⊆ [0, 1]d, s ∈ Z+ and p ∈ [1,∞). Then for any k, l, r ∈ Z+ with k ≥ s and 0 ≤ r ≤ s, we have that there exists some constant C = C(k, s, r, p, d), such that:
‖f −Qk,lf‖Hr(Ω) ≤ C (1 l )s−r ‖f‖Hs(Ω).
Theorem B.8. (Approximation result of Deep Neural Network) Fix some dimension d ∈ Z+, some domain Ω ⊆ [0, 1]d. We pick some l = N 1d ≥ 2, for any s, r ∈ Z+ with 0 ≤ r ≤ s and any function u∗ ∈ Hs(Ω), there exists some sparse Deep Neural Network uDNN ∈ Φ(L,W,S,B) with L = O(1),W = O(N), S = O(N), B = O(N), such that:
‖uDNN − u∗‖Hr(Ω) . N− s−r d ‖u∗‖Hs(Ω). (B.53)
Proof. We firstly show that the given function u∗ can be approximated well by some linear combination of multivariate splines, which is denoted by usp. Note that N is assumed to be sufficiently large. Hence, we may pick l = dN 1d e = Θ(N 1d ) ∈ Z+ to be the partition size of the B-splines. Moreover, by picking k = 4 and p = 2 in Theorem B.7, we have that the linear combination usp := Q4,lu ∗ = ∑
i∈Id4,l λi(u
∗)N (4) l,i satisfies:
‖u∗ − usp‖Hr(Ω) = ‖u∗ −Q4,lu∗‖Hr(Ω) ≤ C (1 l )s−r ‖u∗‖Hs(Ω) = CN− s−r d ‖u∗‖Hs(Ω).
We will then show that the linear combination usp = ∑
i∈Id4,l λi(f)N
(4) l,i can be implemented by
some Deep Neural Network uDNN ∈ Φ(L,W, S,B) with L = O(1),W = O(N), S = O(N) and B = O(logN). Firstly, note that for x ≥ 0, both x and x2 can be expressed in terms of the ReLU3 activation function η3 with no error:
x = − 1 12 [η3(x+ 3)− 5η3(x+ 2) + 7η3(x+ 1)− 3η3(x) + 6],
x2 = −1 6 [η3(x+ 2)− 4η3(x+ 1) + 3η3(x)− 4].
Applying the explicit formula listed in equation B.50 implies that for any −3 ≤ i ≤ l − 1, the univariate B-spline function N (4)l,i (x) (x ∈ [0, 1]) can be implemented by some ReLU3 Deep Neural Network vDNN with both scalar input and scalar output. We have that for vDNN, the depth Lv is 2 and the maximum width Wv is upper bounded by 11.
Secondly, for any x, y ≥ 0, we have that the product operation x · y can be expressed in terms of the ReLU3 activation function η3 with no error:
x · y = 1 2 [(x+ y)2 − x2 − y2]
= − 1 12
[ η3(x+ y + 2)− 4η3(x+ y + 1) + 3η3(x+ y)
− η3(x+ 2) + 4η3(x+ 1)− 3η3(x)− η3(y + 2) + 4η3(y + 1)− 3η3(y) + 4 ] .
In (Schumaker, 2007), it has been proved that the B-splines are always non-negative, i.e N (4)l,i (x) ≥ 0, ∀ x ∈ [0, 1]. Therefore, by multiplying the non-negative univariate B-splines, we can implement any multivariate B-spline N (4)l,i = Π d j=1N (4) l,ij
(xj) with some ReLU3 Deep Neural Network pDNN. We have that for pDNN, the depth Lp = dlog2 de+ 2 and the maximum width Wp = max{11d, 92d}. Hence, we can further claim that u∗ = ∑ i∈Id4,l λi(u ∗)N (4) l,i , which is a linear combination of the multivariate B-splines N (4)l,i , can be implemented by some ReLU3 Deep Neural Network uDNN. It remains to check that uDNN ∈ Φ(L,W,S,B) with L = O(1),W = O(N), S = O(N) and B = O(N). Note that we can ensure that the hidden layers of uDNN are of the same dimension W by adding inactive neurons. For the depth L of uDNN, we have that L is equal to Lp + 1, where Lp denotes the depth of the ReLU3 Deep Neural Network pDNN. Thus, we have L = Lp + 1 = dlog2 de+ 3, which implies that L = O(1). For the width W of uDNN, we have that W ≤ |Idk,l|Wp, where Wp denotes the width of the ReLU3 Deep Neural Network pDNN. This implies:
W ≤ |Idk,l| × 11d = 11d(l + k)d = 11d(l + 4)d = O(ld)⇒W = O(N). For the sparsity constraint S of uDNN, starting from the third layer, the number of activated neurons is half of the number of activated neurons at previous layer. This yields the following upper bound on S:
S ≤ 2(W +W + L−2∑ j=0 W 2j ) ≤ 8W ⇒ S = O(W ) = O(N).
For the norm constraint B of uDNN, we have the following upper bound on B from equation B.50 and equation B.52:
B = O(max{lk−1, sup i∈Idk,l λi(u ∗)}) = O(max{l3, ld}) = O(N).
Now we have shown that parameters L,W, S,B of the Deep Neural Network uDNN are of the desired magnitude, which completes our proof.

heading: B.4 FINAL UPPER BOUND

text: In this subsection, we provide the proof of upper bounds for PINN and DRM. For both estimator, we first provide a meta-theorem to illustrate the approximation and generalization decomposition with a O(1/n) fast rate generalization bound(Bartlett et al., 2005; Xu, 2020). Then we use truncated Fourier basis estimator and neural network estimator as example to obtain the final rate.

heading: B.4.1 DEEP RITZ METHODS

text:
Theorem B.9 (Meta-theorem for Upper Bounds of Deep Ritz Methods). Let u∗ ∈ Hs(Ω) denote the true solution to the PDE model with Dirichlet boundary condition:
−∆u+ V u = f on Ω, u = 0 on ∂Ω,
(B.54)
where f ∈ L2(Ω) and V ∈ L∞(Ω) with 0 < Vmin ≤ V (x) ≤ Vmax > 0. In Theorem B.1, it has been proved that u∗ can be obtained by minimizing the loss E(u):
u∗ = arg min u∈H10 (Ω) E(u) := arg min u∈H10 (Ω)
{1 2 ∫ Ω [ ‖∇u‖2 + V |u|2 ] dx− ∫ Ω fudx } .
For a fixed function space F (Ω), consider the empirical loss induced by the Deep Ritz Method:
En(u) = 1
n n∑ j=1 [ |Ω| · (1 2 ‖∇u(Xj)‖2 + 1 2 V (Xj)|u(Xj)|2 − f(Xj)u(Xj) )] , (B.55)
where {Xj}nj=1 are datapoints uniformly sampled from the domain Ω. Then the Deep Ritz estimator associated with function space F (Ω) is defined as the minimizer of En(u) over the function space F (Ω):
ûDRM = min u∈F (Ω) En(u).
Moreover, we assume that there exists some constant C > 0 such that all function u in the function space F (Ω), the real solution u∗ and f, V satisfy the following two conditions.
• The gradients and function value are uniformly bounded max {
sup u∈F (Ω) ‖u‖L∞(Ω), sup u∈F (Ω)
‖∇u‖L∞(Ω), ‖u∗‖L∞(Ω), ‖∇u∗‖L∞(Ω), Vmax, ‖f‖L∞(Ω) } ≤ C.
(B.56)
• All the functions in the function space F (Ω) satisfies the boundary condition
u = 0 on ∂Ω.
At the the same time, for any ρ > 0, we assume the Rademacher complexity of a localized function space Sρ(Ω) := { h := |Ω| · [ 1 2 ( ‖∇u‖2 − ‖∇u∗‖2 ) + 12V (|u| 2 − |u∗|2)− f(u− u∗) ] ∣∣∣ ‖u −
u∗‖2H1 ≤ ρ } can be upper bounded by a sub-root function φ = φ(ρ) : [0,∞)→ [0,∞), i.e.
φ(4ρ) ≤ 2φ(ρ) and Rn(Sρ(Ω)) ≤ φ(ρ) (∀ ρ > 0). (B.57)
For all constant t > 0. We denote r∗ to be the solution of the fix point equation of local Rademacher complexity r = φ(r). There exist two constants Cp, Cq such that with probability 1−Cp exp(−Cqt), we have the following upper bound for the Deep Ritz Estimator
‖ûDRM − u∗‖2H1 . inf uF∈F (Ω)
( E(uF )−E(u?) ) + max { r∗, t
n
} .
Proof. To upper bound the excess risk ∆E(n) := E(ûDRM)−E(u∗), following(Xu, 2020; Lu et al., 2021b; Duan et al., 2021), we decompose the excess risk into approximation error and generalization error with probability 1− e−Cqt, where Cq > 0 is some constant:
∆E(n)(ûDRM) = E(ûDRM)−E(u?) = [ E(ûDRM)−En(ûDRM) ] + [ En(ûDRM)−En(uF ) ] + [ En(uF )−E(uF ) ] + [ E(uF )−E(u?)
] ≤ [ E(ûDRM)−En(ûDRM) ] + [ En(uF )−E(uF ) ] + [ E(uF )−E(u?)
] ≤ [ E(ûDRM)−E(u∗) + En(u∗)−En(ûDRM)
] + 3
2
[ E(uF )−E(u?) ] + t
2n ,
(B.58) where the expectation is on all sampled data. The inequality of the third line is because ûDRM is the minimizer of the empirical loss En in the solution set F (Ω), so we have En(ûDRM) ≤ En(uF ). The last inequality is based on the Bernstein inequality. The variance of h = |Ω| ·[
1 2
( ‖∇u‖2 − ‖∇u∗‖2 ) + 12V (|u| 2 − |u∗|2)− f(u− u∗) ] can be upper bounded by α ′ α [ E(uF )−
E(u?) ]
due to the strong convexity of the variation objective (B.60). According to the Bernstein inequality, there exists some constant Cq > 0, such that with probability 1− e−Cqt we have:
En(uF )−En(u∗)−E(uF ) + E(u∗) ≤
√ t [ E(uF )−E(u?) ] n ≤ 1 2 [ E(uF )−E(u?) ] + t 2n .
Note that B.58 holds for all function lies in the function space F . Thus, we can take uF := arg minu0∈F (Ω) ( E(u0)−E(u?) ) and finally get
∆E(n) ≤ E(ûDRM)−E(u∗) + En(u∗)−En(u)︸ ︷︷ ︸ ∆Egen + 3 2 inf uF∈F (Ω)
( E(uF )−E(u?) ) ︸ ︷︷ ︸
∆Eapp
+ t
2n .
This inequality decompose the excess risk to the generalization error ∆Egen := E(ûDRM)−E(u∗) + En(u ∗)−En(ûDRM) and the approximation error ∆Eapp = infuF∈F (Ω) ( E(uF )−E(u?) ) . From the lemmata proved in Section B.3, we already have an estimation of the approximation error’s convergence rate. So now we’ll focus on providing fast rate upper bounds of the generalization error for the two estimators using the localization technique(Bartlett et al., 2005; Xu, 2020). To achieve the fast generalization bound, we focus on the following normalized empirical process
S̃r(Ω) := { h̃(x) :=
E[h]− h(x) E[h] + r
| h ∈ S(Ω) } (r > 0).
First, we try to bound the expectation of the normalized empirical process. Applying the Symmetrization Lemma B.1, we can first bound the expectation as
sup h̃∈S̃r(Ω)
Ex′ [ 1
n n∑ i=1 h̃(x′i)
] ≤ Ex′ [ sup
h∈S(Ω)
∣∣∣ 1 n n∑ i=1 h(x′i)− E[h] E[h] + r ∣∣∣] ≤ 2Rn(Ŝr(Ω)). where the function class Ŝr(Ω) is defined as:
Ŝr(Ω) := { ĥ(x) := h(x)
E[h] + r | h ∈ S(Ω)
} ,
where S(Ω) = { h := |Ω| · [ 1 2 ( ‖∇u‖2 − ‖∇u∗‖2 ) + 12V (|u| 2 − |u∗|2)− f(u− u∗) ]}
. Then applying the Peeling Lemma B.4 to any function h ∈ S(Ω) helps us upper bound the local Rademacher complexity Rn(Ŝr(Ω)) with the function φ defined in equation B.57:
Rn(Ŝr(Ω)) = Eσ [ Ex [
sup h∈S(Ω)
1 n ∑n i=1 σih(xi)
E[h] + r
]] ≤ 4φ(r)
r .
Combining all inequalities derived above yields:
sup h̃∈S̃r(Ω)
Ex′ [ 1
n n∑ i=1 h̃(x′i)
] ≤ 2Rn(Ŝr(Ω)) ≤ 8φ(r)
r (r > 0). (B.59)
Secondly we’ll apply the Talagrand concentration inequality, which requires us to verify the condition needed. We will first check that the expectation value E[h] is always non-negative for any h ∈ S(Ω):
E[h] = 1
|Ω| ∫ Ω |Ω| · (1 2 ‖∇u(x)‖2 + 1 2 V (x)|u(x)|2 − f(x)u(x))dx
− 1 |Ω| ∫ Ω |Ω| · (1 2 ‖∇u?(x)‖2 + 1 2 V (x)|u?(x)|2 − f(x)u?(x))dx = E(u)−E(u?) ≥ 0⇒ E[h] ≥ 0.
We will proceed to verify that any h̃ = E[h]−hE[h]+r ∈ S̃r(Ω) is of bounded inf-norm. We need to prove that any h ∈ S(Ω) is of bounded inf-norm beforehand. Using boundedness condition listed in
equation B.56 implies:
‖h‖∞ = |Ω|‖ 1
2
( ‖∇u‖2 − ‖∇u∗‖2 ) + 1
2 V (|u|2 − |u∗|2)− f(u− u∗)‖∞
≤ |Ω| 2
( ‖∇u‖2∞ + ‖∇u∗‖2∞ ) + |Ω| 2 Vmax ( ‖u‖2∞ + ‖u∗‖2∞ ) + |Ω|‖f‖∞ ( ‖u‖∞ + ‖u∗‖∞ ) ≤ |Ω|
2 × 2C2 + |Ω| 2 Vmax × 2C2 + 2|Ω|C2 = |Ω|(Vmax + 3)C2.
By taking M := |Ω|(Vmax + 3)C2, we then have ‖h‖∞ ≤ M for all h ∈ S(Ω). Note that the denominator can be lower bounded by |E[h] + r| ≥ r > 0. Combining these two inequalities help us upper bound the inf-norm ‖h̃‖∞ = supx∈Ω |h̃(x)| as follows:
‖h̃‖∞ = ‖E[h]− h‖∞ |E[h] + r| ≤ 2‖h‖∞ r ≤ 2M r =: β.
We will then check the normalized functions E[h]−h(x)E[h]+r in S̃r(Ω) have bounded second moment, which is satisfied because of the regularity results of the PDE. We aim to show that there exist some constants α, α′ > 0, such that for any h ∈ S(Ω), the following inequality holds:
αE[h2] ≤ ‖u− u∗‖2H1(Ω) ≤ α ′E[h]. (B.60)
The RHS of the inequality follows from strong convexity of the DRM objective function proved in Theorem B.1:
E[h] = E(u)−E(u∗) ≥ min{1, Vmin} 4 ‖u− u∗‖2H1(Ω).
The LHS of the inequality follows from boundedness condition listed in equation B.56 and the QM-AM inequality:
E[h2] = ∫
Ω
[ 1
2
( ‖∇u‖2 − ‖∇u∗‖2 ) + 1
2 V (|u|2 − |u∗|2)− f(u− u∗)
]2 dx
≤ 3 4 ∫ Ω ( ‖∇u‖2 − ‖∇u∗‖2 )2 dx+ 3 4 ∫ Ω V 2(|u|2 − |u∗|2)2dx+ 3 ∫ Ω f2(u− u∗)2dx
≤ 3 4 ∫ Ω ∣∣∣‖∇u‖ − ‖∇u∗‖∣∣∣2(‖∇u‖+ ‖∇u∗‖)2dx+ 3 4 V 2max ∫ Ω ∣∣∣|u| − |u∗|∣∣∣2(|u|+ |u∗|)2dx + 3C2
∫ Ω (u− u∗)2dx ≤ 3C2 ∫ Ω ‖∇u−∇u∗‖2dx+ 3C2(1 + V 2max) ∫ Ω |u− u∗|2dx
≤ 3C2(1 + V 2max)‖u− u∗‖2H1(Ω).
By picking α′ = 4min{1,Vmin} and α = 1 3C2(1+V 2max) , we have finished proving inequality B.60. Then we can can upper bound the expectation E[h̃2] as:
E[h̃2] = E[(h− E[h])2] |E[h] + r|2 = E[h2]− E[h]2 |E[h] + r|2 ≤ E[h 2] |E[h] + r|2 .
Using the fact that E[h] ≥ 0 and inequality B.60, we can lower bound the denominator |E[h] + r|2 as follows:
|E[h] + r|2 ≥ 2E[h]r ≥ 2rα α′ E[h2].
Therefore, we can deduce that:
E[h̃2] ≤ E[h 2] |E[h] + r|2 ≤ E[h 2] 2rα α′ E[h2] = α′ 2rα =: σ2.
Hence, any function in the localized class S̃r(Ω) is of bounded second moment.
It is easy to check that for any h̃ ∈ S̃r(Ω), we have
E[h̃] = E[h]− E[h] E[h] + r = 0,
i.e. any function in the localized class S̃r(Ω) is of zero mean.
Now we have verified that any function h̃ ∈ S̃r(Ω) satisfies all the required conditions. By taking µ to be the uniform distribution on the domain Ω and applying Talagrand’s Concentration inequality given in Lemma B.3, we have:
Px [ sup
h̃∈S̃r(Ω)
1
n n∑ i=1 h̃(xi) ≥ 2 sup h̃∈S̃r(Ω) Ex′ [ 1 n n∑ i=1 h̃(x′i) ] + √ 2tσ2 n + 2tβ n ] ≤ e−t.
By using the upper bound deduced above and plugging in the expressions of β and σ, we can rewrite Talagrand’s Concentration Inequality in the following way. With probability at least 1 − e−t, the inequality below holds:
1
n n∑ i=1 h̃(xi) ≤ sup h̃∈S̃r(Ω) 1 n n∑ i=1 h̃(xi) ≤ 2 sup h̃∈S̃r(Ω) Ex′ [ 1 n n∑ i=1 h̃(x′i) ] + √ 2tσ2 n + 2tβ n
≤ 16φ(r) r +
√ tα′
nαr +
4Mt
nr =: ψ(r).
Let’s pick the critical radius r0 to be:
r0 = max{214r∗, 24Mt
n ,
36α′t
αn }. (B.61)
Note that concavity of the function φ implies that φ(r) ≤ r for any r ≥ r∗. Combining this with the first inequality listed in B.57 yields:
16φ(r)
r ≤
211φ( r0214 )
214 r0214 =
1 8 × φ( r0214 ) r) 214 ≤ 1 8 .
On the other hand, applying equation B.61 yields:√ α′t nαr0 ≤ √ α′t nα αn 36α′t = 1 6 ,
4Mt nr0 ≤ 4Mt n × n 24Mt = 1 6 .
Summing the three inequalities above implies:
ψ(r0) = 16φ(r0)
r0 +
√ tα′
nαr0 +
4Mt nr0 ≤ 1 8 + 1 6 + 1 6 < 1 2 .
By picking r = r0, we can further deduce that for any function u ∈ F (Ω), the following inequality holds with probability 1− e−t:
E(u)−E(u∗)−En(u) + En(u∗) E(u)−E(u∗) + r0 = 1 n n∑ i=1 h̃(xi) ≤ ψ(r0) < 1 2 .
Multiplying the denominator on both sides indicates:
∆Egen = E(u)−E(u∗)−En(u) + En(u∗) ≤ 1
2
[ E(u)−E(u∗) ] + 1
2 r0 =
1 2 ∆E(n) + 1 2 r0.
Substituting the upper bound above into the decomposition ∆E(n) ≤ ∆Egen + 32∆Eapp + t 2n yields that with probability 1− 2e−min{Cq,1}t, we have:
∆E(n) ≤ ∆Egen + 3
2 ∆Eapp +
t 2n ≤ 1 2 ∆E(n) + 1 2 r0 + 3 2 ∆Eapp + t 2n .
Simplifying the inequality above yields that with probability 1− 2e−min{Cq,1}t, we have:
∆E(n) ≤ r0 + 3∆Eapp + t
n = 3 inf uF∈F (Ω)
( E(uF )−E(u?) ) + max{214r∗, 24M t
n ,
36α′
α
t n }+ t n
. inf uF∈F (Ω)
( E(uF )−E(u?) ) + max { r∗, t
n
} .
Moreover, using strong convexity of the DRM objective function proved in Theorem B.1 implies:
∆E(n) = E(ûDRM)−E(u∗) ≥ min{1, Vmin}
4 ‖ûDRM − u∗‖2H1(Ω).
Combining the two bounds above yields that with probability 1− 2e−min{Cq,1}t, we have:
‖ûDRM − u∗‖2H1(Ω) . inf uF∈F (Ω)
( E(uF )−E(u?) ) + max { r∗, t
n
} .
Deep Neural Network Estimator. For any N ∈ Z+, there exists some Deep Neural Network in Φ(L,W, S,B) with L = O(1), W = O(N), S = O(N), B = O(N), such that the approximation error ∆Eapp = O(N− 2(s−1) d ) and generalization error ∆Egen = O(N logNn ). With optimal selection N = n d d+2s−2 to balance the bias and variance, we can achieve n− 2s−2 d+2s−2 log n convergence rate for the DNN estimator. Theorem B.10. (Final Upper Bound of DRM with Deep Neural Network Estimator) Under the assumptions in Theorem B.9, we consider the Deep Ritz objective with the sparse Deep Neural Network function space Φ(L,W,S,B), where the parameters L = O(1), W = O(n d d+2s−2 ), S = O(n d d+2s−2 ), B = O(n d d+2s−2 ). Then we have that the DNN estimator ûDNNDRM = minu∈Φ(L,W,S,B) E DRM n (u) satisfies the following upper bound with high probability:
‖ûDNNDRM − u∗‖2H1 . n − 2s−2d+2s−2 log n.
Proof. On the one hand, by taking s = 1 and p = 2 in Theorem B.8 proved above, we have that there exists some Deep Neural Network uDNN ∈ Φ(L,W, S,B) with L = O(1),W = O(N), S = O(N), B = O(N), such that.
‖uDNN − u∗‖2H1(Ω) ≤ N − 2s−2d ‖u∗‖Hs(Ω).
Applying strong convexity of the DRM objective function proved in Theorem B.1 yields the following upper bound on the approximation error of DRM:
∆Eapp . ‖uDNN − u∗‖2H1(Ω) ≤ N − 2s−2d .
On the other hand, from Lemma B.17 proved above, we know that the function φ(ρ) that upper bounds the local Rademacher complexity of the Deep Neural Network space is of the same magnitude as √
S3Lρ n log (BWn). By plugging in the magnitudes of L,W, S,B, we can determine the critical
radius r∗:√ r∗3LS
n log(BWn) '
√ r∗N
n (2 logN + log n) ' r∗ ⇒ r∗ ' N(logN + log n) n .
Combining the two bounds above with Theorem B.9 yields that with high probability, we have:
‖ûDNNDRM − u∗‖2H1 . ∆Eapp + r∗ . N− 2(s−1) d + N(logN + log n)
n .
By equating the two terms above, we can solve for the optimal N that yields the desired bound:
N− 2(s−1) d ' N
n ⇒ N ' n
d d+2s−2 .
Plugging in the optimal N gives us the magnitudes of the four parameters L = O(1), W = O(n d d+2s−2 ), S = O(n d d+2s−2 ), B = O(n d d+2s−2 ), as well as the final rate:
‖ûDNNDRM − u∗‖2H1 . N− 2(s−1) d + N logN n . n− 2(s−1) d+2(s−1) log n.
Truncated Fourier Series Estimator. For any ξ ∈ Z+, there exists some Truncated Fourier Series in Fξ(Ω) with approximation error ∆Eapp = O(ξ−2(s−1)) and generalization error ∆Egen = O( ξ d
n ).
With optimal selection ξ = n 1 d+2s−2 to balance the bias and variance, we can achieve n− 2s−2 d+2s−2 convergence rate for the Fourier estimator. Theorem B.11. (Final Upper Bound of DRM with Truncated Fourier Series Estimator) Under the assumptions in Theorem B.9, we consider the Deep Ritz objective with the Truncated Fourier Series function space Fξ(Ω), where the parameter ξ = Θ(n 1 d+2s−2 ). Then we have that the Fourier estimator ûFourierDRM = minu∈Fξ(Ω) E DRM n (u) satisfies the following upper bound with high probability:
‖ûFourierDRM − u∗‖2H1 . n − 2s−2d+2s−2 .
Proof. Let’s firstly derive the function φ(ρ) that upper bounds the local Rademacher complexity of Sρ(Ω) := { h := |Ω| · [ 1 2 ( ‖∇u‖2 − ‖∇u∗‖2 ) + 12V (|u| 2 − |u∗|2)− f(u− u∗) ] ∣∣∣ u ∈ Fρ,ξ(Ω)},
where Fρ,ξ(Ω) = { v ∈ Fξ(Ω) ∣∣∣ ‖v‖2H1(Ω) ≤ ρ} denotes the localized Truncated Fourier Series space. From Talagrand Contraction Lemma B.2, Lemma B.6 and Lemma B.7 proved above, we have
Rn(Sρ(Ω)) . Rn ({ u− u∗ : u ∈ Fρ,ξ(Ω), ‖u− u∗‖H1(Ω) ≤ √ ρ })
+Rn ({ ‖∇u−∇u∗‖ : u ∈ Fρ,ξ(Ω), ‖u− u∗‖H1(Ω) ≤ √ ρ })
. EX [ Eσ [
sup u∈Fρ,ξ(Ω)
1
n n∑ i=1 σi(u(Xi)−Πξu∗(Xi)) ∣∣∣‖u−Πξu∗‖2H1(Ω) ≤ ρ]
]
+ EX [ Eσ [
sup u∈Fρ,ξ(Ω)
1
n n∑ i=1 σi‖∇u(Xi)−∇Πξu∗(Xi)‖ ∣∣∣‖u−Πξu∗‖2H1(Ω) ≤ ρ]
]
+ EX [ Eσ [ 1 n n∑ i=1 σi‖∇Π>ξu∗(Xi)‖ ]] + EX [ Eσ [ 1 n n∑ i=1 σiΠ>ξu ∗(Xi) ]]
. EX [ Eσ [
sup v∈Fρ,ξ(Ω)
1
n n∑ i=1 σiv(Xi) ∣∣∣‖v‖2H1(Ω) ≤ ρ]
]
+ EX [ Eσ [
sup v∈Fρ,ξ(Ω)
1
n n∑ i=1 σi‖∇v(Xi)‖ ∣∣∣‖v‖2H1(Ω) ≤ ρ]
]
+ EX [ Eσ [ 1 n n∑ i=1 σi‖∇Π>ξu∗(Xi)‖ ]] + EX [ Eσ [ 1 n n∑ i=1 σiΠ>ξu ∗(Xi) ]]
.
√ ρ
n ξ d 2 +
√ ‖Π>ξu∗‖2H1
n .
√ ρ
n ξ d 2 +
√ ξ−2(s−1)
n .
√ ρ
n ξ d 2 +
1 n + ξ−2(s−1),
(B.62) where Πξu := ∑ ‖z‖∞≤ξ uzφz(x) is the projection to the Fourier basis whose frequency is smaller
than ξ and Π>ξu := ∑ ‖z‖∞≤ξ uzφz(x) is the projection to the Fourier basis whose frequency is larger than ξ. Then, the critical radius r∗ can be determined as follows:√ r∗
n ξ d 2 +
1 n + ξ−2(s−1) ' r∗ ⇒ r∗ ' ξ
d
n +
1 n + ξ−2(s−1).
Moreover, by taking α = s and β = 1 in Lemma B.19 and applying strong convexity of the DRM objective function proved in Theorem B.1, we can upper bound the approximation error ∆Eapp as below:
∆Eapp . ξ −2(s−1).
Combining the two bounds above with Theorem B.9 yields that with high probability, we have:
‖ûFourierDRM − u∗‖2H1 . ∆Eapp + r∗ . ξd
n +
1 n + ξ−2(s−1).
By equating two of the three terms above, we can solve for ξ that yields the desired bound:
ξd n ' ξ−2(s−1) ⇒ ξ ' n 1 d+2s−2 .
Plugging in the optimal ξ gives us the final rate:
‖ûFourierDRM − u∗‖2H1 . ξd
n + ξ−2(s−1) +
1 n . n− 2s−2 d+2s−2 .

heading: B.4.2 PHYSICS INFORMED NEURAL NETWORK

text:
Theorem B.12 (Meta-theorem for Upper Bounds of Physics Informed Neural Network). Let u∗ ∈ Hs(Ω) denote the true solution to the PDE model with Dirichlet boundary condition:
−∆u+ V u = f on Ω, u = 0 on ∂Ω,
(B.63)
where f ∈ L2(Ω) and V ∈ L∞(Ω) with V − 12∆V > Cmin, 0 < Cmin < V (x) ≤ Vmax and −∆V (x) ≤ Vmax. In Theorem B.2, it has been proved that u∗ can be obtained by minimizing the loss E(u):
u∗ = arg min u∈H10 (Ω) E(u) := arg min u∈H10 (Ω) {∫ Ω |∆u− V u+ f |2dx } .
For a fixed function space F (Ω), consider the empirical loss induced by the Physics Informed Neural Network:
En(u) = 1
n n∑ j=1 [ |Ω| · ( ∆u(Xj)− V (Xj)u(Xj) + f(Xj) )2] , (B.64)
where {Xj}nj=1 are datapoints uniformly sampled from the domain Ω. Then the Physics Informed Neural Network estimator associated with function space F (Ω) is defined as the minimizer of En(u) over the function space F (Ω):
ûPINN = min u∈F (Ω) En(u).
Moreover, we assume that there exists some constant C > 0 such that all function u in the function space F (Ω), the real solution u∗ and f, V satisfy the following two conditions.
• The gradients and function value are uniformly bounded max {
sup u∈F (Ω) ‖u‖L∞(Ω), sup u∈F (Ω) ‖∇u‖L∞(Ω), sup u∈F (Ω) ‖∆u‖L∞(Ω),
‖u∗‖L∞(Ω), ‖∇u∗‖L∞(Ω), ‖∆u∗‖L∞(Ω), Vmax, ‖f‖L∞(Ω) } ≤ C.
(B.65)
• All the functions in the function space F (Ω) satisfies the boundary condition
u = 0 on ∂Ω.
At the the same time, for any ρ > 0, we assume the Rademacher complexity of a localized function space Tρ(Ω) := { h := |Ω| · [ (∆u− V u+ f)2 − (∆u∗ − V u∗ + f)2 ] ∣∣∣ ‖u − u∗‖2H2 ≤ ρ} can be upper bounded by a sub-root function φ = φ(ρ) : [0,∞)→ [0,∞), i.e.
φ(4ρ) ≤ 2φ(ρ) and Rn(Tρ(Ω)) ≤ φ(ρ) (∀ ρ > 0). (B.66) For all constant t > 0. We denote r∗ to be the solution of the fix point equation of local Rademacher complexity r = φ(r). There exists two constants Cp, Cq such that with probability 1−Cp exp(−Cqt), we have the following upper bound for the Physics Informed Neural Network Estimator
‖ûPINN − u∗‖2H2 . inf uF∈F (Ω)
( E(uF )−E(u?) ) + max { r∗, t
n
} .
Proof. To upper bound the excess risk ∆E(n), following(Xu, 2020; Lu et al., 2021b; Duan et al., 2021), we decompose the excess risk into approximation error and generalization error with probability 1− e−Cqt, where Cq > 0 is some constant:
∆E(n)(ûPINN) = E(ûPINN)−E(u?) = [ E(ûPINN)−En(ûPINN) ] + [ En(ûPINN)−En(uF ) ] + [ En(uF )−E(uF ) ] + [ E(uF )−E(u?)
] ≤ [ E(ûPINN)−En(ûPINN) ] + [ En(uF )−E(uF ) ] + [ E(uF )−E(u?)
] ≤ [ E(ûPINN)−E(u∗) + En(u∗)−En(ûPINN)
] + 3
2
[ E(uF )−E(u?) ] + t
2n ,
(B.67) where the expectation is on all sampled data. The inequality of the third line is because ûPINN is the minimizer of the empirical loss En in the solution set F (Ω), so we have En(ûPINN) ≤ En(uF ). The last inequality is based on the Bernstein inequality. The variance of h = |Ω| ·[ (∆u− V u+ f)2 − (∆u∗ − V u∗ + f)2 ] can be bounded by α ′
α
[ E(uF )−E(u?) ] due to the strong
convexity of the variation objective (B.69). According to the Bernstein inequality, there exists some constant Cq > 0, such that with probability 1− e−Cqt we have:
En(uF )−En(u∗)−E(uF ) + E(u∗) ≤
√ t [ E(uF )−E(u?) ] n ≤ 1 2 [ E(uF )−E(u?) ] + t 2n .
Note that C.5 holds for all function lies in the function space F . Thus, we can take uF := arg minu0∈F (Ω) ( E(u0)−E(u?) ) and finally get
∆E(n) ≤ E(ûPINN)−E(u∗) + En(u∗)−En(ûPINN)︸ ︷︷ ︸ ∆Egen + 3 2 inf uF∈F (Ω)
( E(uF )−E(u?) ) ︸ ︷︷ ︸
∆Eapp
+ t
2n .
This inequality decompose the excess risk to the generalization error ∆Egen := E(ûPINN)−E(u∗) + En(u ∗)−En(ûPINN) and the approximation error ∆Eapp = infuF∈F (Ω) ( E(uF )−E(u?) ) . From the lemmata proved in Section B.3, we already have an estimation of the approximation error’s convergence rate. So now we’ll focus on providing fast rate upper bounds of the generalization error for the two estimators using the localization techinque(Bartlett et al., 2005; Xu, 2020). To achieve the fast generalization bound, we focus on the following normalized empirical process
T̃r(Ω) := { h̃(x) :=
E[h]− h(x) E[h] + r
| h ∈ T (Ω) } (r > 0).
First, we try to bound the expectation of the normalized empirical process. Applying the Symmetrization Lemma B.1, we can first bound the expectation as
sup h̃∈T̃r(Ω)
Ex′ [ 1
n n∑ i=1 h̃(x′i)
] ≤ Ex′ [ sup
h∈T (Ω)
∣∣∣ 1 n n∑ i=1 h(x′i)− E[h] E[h] + r ∣∣∣] ≤ 2Rn(T̂r(Ω)). where the function class Ŝr(Ω) is defined as:
T̂r(Ω) := { ĥ(x) := h(x)
E[h] + r | h ∈ T (Ω)
} ,
where T (Ω) = { h := |Ω| · [ (∆u− V u+ f)2 − (∆u∗ − V u∗ + f)2 ] } . Then applying the Peeling
Lemma B.4 to any function h ∈ T (Ω) helps us upper bound the local Rademacher complexity Rn(T̂r(Ω)) with the function φ defined in equation B.66:
Rn(T̂r(Ω)) = Eσ [ Ex [
sup h∈T (Ω)
1 n ∑n i=1 σih(xi)
E[h] + r
]] ≤ 4φ(r)
r .
Combining all inequalities derived above yields:
sup h̃∈T̃r(Ω)
Ex′ [ 1
n n∑ i=1 h̃(x′i)
] ≤ 2Rn(T̂r(Ω)) ≤ 8φ(r)
r (r > 0). (B.68)
Secondly we’ll apply the Talagrand concentration inequality, which requires us to verify the condition needed. We will first check that the expectation value E[h] is always non-negative for any h ∈ S(Ω):
E[h] = 1
|Ω| ∫ Ω |Ω| · (∆u− V u+ f)2dx− 1 |Ω| ∫ Ω |Ω| · (∆u∗ − V u∗ + f)2dx
= E(u)−E(u?) ≥ 0⇒ E[h] ≥ 0.
We will proceed to verify that any h̃ = E[h]−hE[h]+r ∈ T̃r(Ω) is of bounded inf-norm. We need to prove that any h ∈ T (Ω) is of bounded inf-norm beforehand. Using boundedness condition listed in equation B.65 implies:
‖h‖∞ = |Ω| · ‖(∆u− V u+ f)2 − (∆u∗ − V u∗ + f)2‖∞ = |Ω| · ‖(∆u− V u+ f)2‖∞ ≤ |Ω| · (‖∆u‖∞ + Vmax‖u‖∞ + ‖f‖∞)2 ≤ |Ω|(Vmax + 2)2C2.
By taking M := |Ω|(Vmax + 2)2C2, we then have ‖h‖∞ ≤ M for all h ∈ T (Ω). Note that the denominator can be lower bounded by |E[h] + r| ≥ r > 0. Combining these two inequalities help us upper bound the inf-norm ‖h̃‖∞ = supx∈Ω |h̃(x)| as follows:
‖h̃‖∞ = ‖E[h]− h‖∞ |E[h] + r| ≤ 2‖h‖∞ r ≤ 2M r =: β.
We will then check the normalized functions E[h]−h(x)E[h]+r in T̃r(Ω) have bounded second moment, which is satisfied because of the regularity results of the PDE. We aim to show that there exist some constants α, α′ > 0, such that for any h ∈ T (Ω), the following inequality holds:
αE[h2] ≤ ‖u− u∗‖2H2(Ω) ≤ α ′E[h]. (B.69)
The RHS of the inequality follows from strong convexity of the PINN objective function proved in Theorem B.2:
E[h] = E(u)−E(u∗) ≥ min{1, Cmin}‖u− u∗‖2H2(Ω).
The LHS of the inequality follows from boundedness condition listed in equation B.65 and the QM-AM inequality:
E[h2] = ∫
Ω
[ (∆u− V u+ f)2 − (∆u∗ − V u∗ + f)2 ]2 dx = ∫ Ω (∆u− V u+ f)4dx
≤M2 ∫
Ω
(∆u− V u−∆u∗ + V u∗)2dx ≤ 2M2 ∫
Ω
[(∆u−∆u∗)2 + V 2(u− u∗)2]dx
≤ 2M2 max{1, V 2max}‖u− u∗‖2H2(Ω)
By picking α′ = 1min{1,Cmin} and α = 1 2M2 max{1,V 2max} , we have finished proving inequality B.69. Then we can can upper bound the expectation E[h̃2] as:
E[h̃2] = E[(h− E[h])2] |E[h] + r|2 = E[h2]− E[h]2 |E[h] + r|2 ≤ E[h 2] |E[h] + r|2 .
Using the fact that E[h] ≥ 0 and inequality B.69, we can lower bound the denominator |E[h] + r|2 as follows:
|E[h] + r|2 ≥ 2E[h]r ≥ 2rα α′ E[h2].
Therefore, we can deduce that:
E[h̃2] ≤ E[h 2] |E[h] + r|2 ≤ E[h 2] 2rα α′ E[h2] = α′ 2rα =: σ2.
Hence, any function in the localized class T̃r(Ω) is of bounded second moment.
It is easy to check that for any h̃ ∈ T̃r(Ω), we have
E[h̃] = E[h]− E[h] E[h] + r = 0,
i.e. any function in the localized class S̃r(Ω) is of zero mean.
Now we have verified that any function h̃ ∈ S̃r(Ω) satisfies all the required conditions. By taking µ to be the uniform distribution on the domain Ω and applying Talagrand’s Concentration inequality given in Lemma B.3, we have:
Px [ sup
h̃∈T̃r(Ω)
1
n n∑ i=1 h̃(xi) ≥ 2 sup h̃∈T̃r(Ω) Ex′ [ 1 n n∑ i=1 h̃(x′i) ] + √ 2tσ2 n + 2tβ n ] ≤ e−t.
By using the upper bound deduced above and plugging in the expressions of β and σ, we can rewrite Talagrand’s Concentration Inequality in the following way. With probability at least 1 − e−t, the inequality below holds:
1
n n∑ i=1 h̃(xi) ≤ sup h̃∈S̃r(Ω) 1 n n∑ i=1 h̃(xi) ≤ 2 sup h̃∈S̃r(Ω) Ex′ [ 1 n n∑ i=1 h̃(x′i) ] + √ 2tσ2 n + 2tβ n
≤ 16φ(r) r +
√ tα′
nαr +
4Mt
nr =: ψ(r).
Let’s pick the critical radius r0 to be:
r0 = max{214r∗, 24Mt
n ,
36α′t
αn }. (B.70)
Note that concavity of the function φ implies that φ(r) ≤ r for any r ≥ r∗. Combining this with the first inequality listed in B.66 yields:
16φ(r)
r ≤
211φ( r0214 )
214 r0214 =
1 8 × φ( r0214 ) r) 214 ≤ 1 8 .
On the other hand, applying equation B.70 yields:√ α′t nαr0 ≤ √ α′t nα αn 36α′t = 1 6 ,
4Mt nr0 ≤ 4Mt n × n 24Mt = 1 6 .
Summing the three inequalities above implies:
ψ(r0) = 16φ(r0)
r0 +
√ tα′
nαr0 +
4Mt nr0 ≤ 1 8 + 1 6 + 1 6 < 1 2 .
By picking r = r0, we can further deduce that for any function u ∈ F (Ω), the following inequality holds with probability 1− e−t:
E(u)−E(u∗)−En(u) + En(u∗) E(u)−E(u∗) + r0 = 1 n n∑ i=1 h̃(xi) ≤ ψ(r0) < 1 2 .
Multiplying the denominator on both sides indicates:
∆Egen = E(u)−E(u∗)−En(u) + En(u∗) ≤ 1
2
[ E(u)−E(u∗) ] + 1
2 r0 =
1 2 ∆E(n) + 1 2 r0.
Substituting the upper bound above into the decomposition ∆E(n) ≤ ∆Egen + 32∆Eapp + t 2n yields that with probability 1− 2e−min{1,Cq}t, we have:
∆E(n) ≤ ∆Egen + 3
2 ∆Eapp +
t 2n ≤ 1 2 ∆E(n) + 1 2 r0 + 3 2 ∆Eapp + t 2n .
Simplifying the inequality above yields that with probability 1− 2e−min{1,Cq}t, we have:
∆E(n) ≤ r0 + 3∆Eapp + t
n = 3 inf uF∈F (Ω)
( E(uF )−E(u?) ) + max{214r∗, 24M t
n ,
36α′
α
t n }+ t n
. inf uF∈F (Ω)
( E(uF )−E(u?) ) + max { r∗, t
n
} .
Moreover, using strong convexity of the PINN objective function proved in Theorem B.2 implies:
∆E(n) = E(ûPINN)−E(u∗) ≥ min{1, Cmin}‖ûPINN − u∗‖2H2(Ω).
Combining the two bounds above yields that with probability 1− 2e−min{1,Cq}t, we have:
‖ûPINN − u∗‖2H2(Ω) . inf uF∈F (Ω)
( E(uF )−E(u?) ) + max { r∗, t
n
} .
Deep Neural Network Estimator. For any N ∈ Z+, there exists some Deep Neural Network in Φ(L,W, S,B) with L = O(1), W = O(N), S = O(N), B = O(N), such that the approximation error ∆Eapp = O(N− 2(s−2) d ) and generalization error ∆Egen = O(N logNn ). With optimal selection N = n d d+2s−4 to balance the bias and variance, we can achieve n− 2s−4 d+2s−4 log n convergence rate for PINN estimator.
Theorem B.13. (Final Upper Bound of PINN with Deep Neural Network Estimator) Under the assumptions in Theorem B.12, we consider the PINN objective with the sparse Deep Neural Network function space Φ(L,W,S,B), where the parameters L = O(1), W = O(n d d+2s−4 ), S = O(n d d+2s−4 ) and B = O(n d d+2s−4 ). Then we have that the DNN estimator ûDNNPINN = minu∈Φ(L,W,S,B) E PINN n (u) satisfies the following upper bound with high probability:
‖ûDNNPINN − u∗‖2H2 . n − 2s−4d+2s−4 log n.
Proof. On the one hand, by taking s = 2 and p = 2 in Theorem B.8 proved above, we have that there exists some Deep Neural Network uDNN ∈ Φ(L,W, S,B) with L = O(1),W = O(N), S = O(N), B = O(N), such that.
‖uDNN − u∗‖2H2(Ω) ≤ N − 2s−4d ‖u‖Hs(Ω).
Applying strong convexity of the PINN objective function proved in Theorem B.2 yields the following upper bound on the approximation error of PINN:
∆Eapp . ‖uDNN − u∗‖2H2(Ω) ≤ N − 2s−4d .
On the other hand, from lemma B.18 proved above, we know that the function φ(ρ) that upper bounds the local Rademacher complexity of the Deep Neural Networks uDNN is of the same magnitude as√
S3Lρ n log (BWn). By plugging in the magnitudes of L,W,S,B, we can determine the critical
radius r∗:√ r∗3LS
n log(BWn) '
√ r∗N
n (2 logN + log n) ' r∗ ⇒ r∗ ' N(logN + log n) n .
Combining the two bounds above with Theorem B.12 yields that with high probability, we have:
‖ûDNNPINN − u∗‖2H2 . ∆Eapp + r∗ . N− 2(s−2) d + N(logN + log n)
n .
By equating the two terms above, we can solve for the optimal N that yields the desired bound:
N− 2(s−2) d ' N
n ⇒ N ' n
d d+2s−4 .
Plugging in the optimal N gives us the magnitudes of the four parameters L = O(1), W = O(n d d+2s−4 ), S = O(n d d+2s−4 ), B = O(n d d+2s−4 ), as well as the final rate:
‖ûDNNPINN − u∗‖2H2 . N− 2(s−2) d + N logN n . n− 2(s−2) d+2(s−2) log n.
Truncated Fourier Series Estimator. For any ξ ∈ Z+, there exists some Truncated Fourier Series in Fξ(Ω) with approximation error ∆Eapp = O(ξ−2(s−2)) and generalization error ∆Egen = O( ξ d
n ).
With optimal selection ξ = n 1 d+2s−4 to balance the bias and variance, we can achieve n− 2s−4 d+2s−4 convergence rate for the Fourier estimator.
Theorem B.14. (Final Upper Bound of PINN with Truncated Fourier Series Estimator) Under the assumptions in Theorem B.12, we consider the PINN objective with the Truncated Fourier Series function space Fξ(Ω), where the parameter ξ = Θ(n 1 d+2s−4 ). Then we have that the Fourier estimator ûFourierPINN = minu∈Fξ(Ω) E PINN n (u) satisfies the following upper bound with high probability:
‖ûFourierPINN − u∗‖2H2 . n − 2s−4d+2s−4
Proof. Let’s firstly derive the function φ(ρ) that upper bounds the local Rademacher complexity of Tρ(Ω) := { h := |Ω| · [ (∆u− V u+ f)2 − (∆u∗ − V u∗ + f)2 ] ∣∣∣ u ∈ Jρ,ξ(Ω)}, where Jρ,ξ(Ω) = { v ∈ Fξ(Ω)
∣∣∣ ‖v‖2H2(Ω) ≤ ρ} denotes the localized Truncated Fourier Series space. From Talagrand Contraction Lemma B.2, Lemma B.6 and Lemma B.8 proved above, we have
Rn(Tρ(Ω)) . Rn ({ u− u∗ : u ∈ Jρ,ξ(Ω), ‖u− u∗‖H2(Ω) ≤ √ ρ })
+Rn ({ ∆u−∆u∗ : u ∈ Jρ,ξ(Ω), ‖u− u∗‖H2(Ω) ≤ √ ρ })
. EX [ Eσ [
sup u∈Fρ,ξ(Ω)
1
n n∑ i=1 σi(u(Xi)−Πξu∗(Xi)) ∣∣∣‖u−Πξu∗‖2H1(Ω) ≤ ρ]
]
+ EX [ Eσ [
sup u∈Fρ,ξ(Ω)
1
n n∑ i=1 σi(∆u(Xi)−∆Πξu∗(Xi)) ∣∣∣‖u−Πξu∗‖2H1(Ω) ≤ ρ]
]
+ EX [ Eσ [ 1 n n∑ i=1 σi∆Π>ξu ∗(Xi) ]] + EX [ Eσ [ 1 n n∑ i=1 σiΠ>ξu ∗(Xi) ]]
. EX [ Eσ [
sup v∈Fρ,ξ(Ω)
1
n n∑ i=1 σiv(Xi) ∣∣∣‖v‖2H2(Ω) ≤ ρ]
]
+ EX [ Eσ [
sup v∈Fρ,ξ(Ω)
1
n n∑ i=1 σi∆v(Xi) ∣∣∣‖v‖2H2(Ω) ≤ ρ]
]
+ EX [ Eσ [ 1 n n∑ i=1 σi∆Π>ξu ∗(Xi) ]] + EX [ Eσ [ 1 n n∑ i=1 σiΠ>ξu ∗(Xi) ]]
.
√ ρ
n ξ d 2 +
√ ‖Π>ξu∗‖2H2
n .
√ ρ
n ξ d 2 +
√ ξ−2(s−2)
n .
√ ρ
n ξ d 2 +
1 n + ξ−2(s−2),
(B.71) where Πξu := ∑ ‖z‖∞≤ξ uzφz(x) is the projection to the Fourier basis whose frequency is smaller
than ξ and Π>ξu := ∑ ‖z‖∞≤ξ uzφz(x) is the projection to the Fourier basis whose frequency is
larger than ξ. Then, the critical radius r∗ can be determined as follows:√ r∗
n ξ d 2 +
1 n + ξ−2(s−2) ' r∗ ⇒ r∗ ' ξ
d
n +
1 n + ξ−2(s−2),
Moreover, by taking α = s and β = 1 in Lemma B.19 and applying strong convexity of the DRM objective function proved in Theorem B.2, we can upper bound the approximation error ∆Eapp as below:
∆Eapp . ξ −2(s−2).
Combining the two bounds above with Theorem B.12 yields that with high probability, we have:
‖ûFourierPINN − u∗‖2H2 . ∆Eapp + r∗ . ξd
n +
1 n + ξ−2(s−2).
By equating two of the three terms above, we can solve for ξ that yields the desired bound:
ξd n ' ξ−2(s−2) ⇒ ξ ' n 1 d+2s−4 .
Plugging in the optimal ξ gives us the final rate:
‖ûFourierPINN − u∗‖2H2 . ξd
n + ξ−2(s−2) +
1 n . n− 2s−4 d+2s−4 .

heading: C PROOF OF MODIFIED DRM

text:
In this section, we provide the proof of the modified deep Ritz method here. We first provide a similar meta-theorem as we did for DRM. Theorem C.1 (Meta-theorem for Upper Bounds of Modified Deep Ritz Method). Let u∗ ∈ Hs(Ω) denote the true solution to the PDE model with Dirichlet boundary condition:
−∆u+ V u = f on Ω, u = 0 on ∂Ω,
(C.1)
where f ∈ L2(Ω) and V ∈ L∞(Ω) with 0 < Vmin ≤ V (x) ≤ Vmax > 0. In Theorem B.1, it has been proved that u∗ can be obtained by minimizing the loss E(u):
u∗ = arg min u∈H10 (Ω) E(u) := arg min u∈H10 (Ω)
{1 2 ∫ Ω [ ‖∇u‖2 + V |u|2 ] dx− ∫ Ω fudx } .
For a fixed function space F (Ω), consider the empirical loss induced by the Modified Deep Ritz Method (N ≥ n):
EN,n(u) = 1
N N∑ i=1 [ |Ω| · 1 2 ‖∇u(X ′i)‖2 ] + 1 n n∑ j=1 [ |Ω| · (1 2 V (Xj)|u(Xj)|2 − f(Xj)u(Xj) )] ,
(C.2) where {X ′i}Ni=1 and {Xj}nj=1 are datapoints uniformly and independently sampled from the domain Ω. Then the Modified Deep Ritz estimator associated with function space F (Ω) is defined as the minimizer of EN,n(u) over the function space F (Ω):
ûMDRM = min u∈F (Ω) EN,n(u).
Moreover, we assume that there exists some constant C > 0 such that all function u in the function space F (Ω), the real solution u∗ and f, V satisfy the following two conditions.
• The gradients and function value are uniformly bounded max {
sup u∈F (Ω) ‖u‖L∞(Ω), sup u∈F (Ω)
‖∇u‖L∞(Ω), ‖u∗‖L∞(Ω), ‖∇u∗‖L∞(Ω), Vmax, ‖f‖L∞(Ω) } ≤ C.
(C.3)
• All the functions in the function space F (Ω) satisfy the boundary condition
u = 0 on ∂Ω.
At the the same time, for any ρ > 0, we assume the Rademacher complexity of the following vector-valued function space
Sρ(Ω) := { (h1, h2) ∣∣h1 := |Ω| · [1
2
( ‖∇u‖2 − ‖∇u∗‖2 )] ,
h2 := |Ω| · [ 1
2 V (|u|2 − |u∗|2)− f(u− u∗)
] , ‖u− u∗‖2H1 ≤ ρ } .
can be upper bounded by a sub-root function φ = φ(ρ) : [0,∞)→ [0,∞), i.e.
φ(4ρ) ≤ 2φ(ρ) and RN,n(Sρ(Ω)) ≤ φ(ρ) (∀ ρ > 0), (C.4)
where RN,n(S) := RN ({h1|(h1, h2) ∈ S}) +Rn({h2|(h1, h2) ∈ S}). For all constant t > 0. We denote r∗ to be the solution of the fix point equation of local Rademacher complexity r = φ(r). There exists two constants Cp, Cq such that with probability 1 − Cp exp(−Cqt), we have the following upper bound for the Modified Deep Ritz Estimator
‖ûMDRM − u∗‖2H1 . inf uF∈F (Ω)
( E(uF )−E(u?) ) + max { r∗, t
n
} .
Proof. To upper bound the excess risk ∆E(N,n) := E(ûMDRM) − E(u∗), following(Xu, 2020; Lu et al., 2021b; Duan et al., 2021), we decompose the excess risk into approximation error and generalization error with probability 1− e−t:
∆E(N,n) = [ E(ûMDRM)−E(u?) ] = [ E(ûMDRM)−EN,n(ûMDRM) ] + [ EN,n(ûMDRM)−EN,n(uF ) ] + [ EN,n(uF )−E(uF ) ] + [ E(uF )−E(u?)
] ≤ [ E(ûMDRM)−EN,n(ûMDRM) ] + [ EN,n(uF )−E(uF ) ] + [ E(uF )−E(u?)
] ≤ [ E(ûMDRM)−E(u∗) + EN,n(u∗)−EN,n(ûMDRM)]
] + 2 [ E(uF )−E(u?) ] + 4t
min{N,n} ,
(C.5) where the expectation is on all sampled data. The inequality of the third line is because ûMDRM is the minimizer of the empirical loss En in the solution set F (Ω), so we have EN,n(ûMDRM) ≤ EN,n(uF ). The last inequality is based on the Bernstein inequality. For any uF ∈ F (Ω), we use hF ,1, hF ,2 to denote the following two functions:
hF ,1 : = 1
2
( ‖∇uF ‖2 − ‖∇u∗‖2 ) ,
hF ,2 : = 1
2 V (|uF |2 − |u∗|2)− f(uF − u∗).
Applying Bernstein’s inequality twice to hF ,1 and hF ,2 implies that there exists some constant Cq, such that with probability 1− 2e−Cqt, the following two inequalities hold simultaneously:
EN (hF ,1)−E(hF ,1) ≤
√ t αα′E[h 2 F ,1]
N ,
En(hF ,2)−E(hF ,2) ≤
√ t αα′E[h 2 F ,2]
n .
Note that the variance sum E[h2F ,1] + E[h 2 F ,2] can be upper bounded by
α′ α [ E(uF )−E(u?) ] due
to the strong convexity of the variation objective (C.9). Adding the two inequalities above implies
with probability 1− 2e−Cqt we have: EN,n(uF )−EN,n(u∗)−E(uF ) + E(u∗) = EN (hF ,1)−E(hF ,1) + En(hF ,2)−E(hF ,2)
≤
√ t αα′E[h 2 F ,1]
N +
√ t αα′E[h 2 F ,2]
n
≤ √√√√2t αα′(E[h2F ,1] + E[h2F ,1]) min{N,n}
≤
√ 2t [ E(uF )−E(u?) ] min{N,n} ≤ [ E(uF )−E(u?) ] +
4t
min{N,n} .
Note that C.5 holds for all function lies in the function space F . Thus, we can take uF := arg minuF∈F (Ω) ( E(uF )−E(u?) ) and finally get:
∆E(N,n) ≤ E(ûMDRM)−E(u∗) + EN,n(u∗)−EN,n(ûMDRM)︸ ︷︷ ︸ ∆Egen +2 inf uF∈F (Ω)
( E(uF )−E(u?) ) ︸ ︷︷ ︸
∆Eapp
+ 4t
n .
This inequality decomposes the excess risk to the generalization error ∆Egen := E(ûMDRM) − E(u∗) +EN,n(u ∗)−EN,n(ûMDRM) and the approximation error ∆Eapp = infuF∈F (Ω) ( E(uF )−
E(u?) )
. From the lemmata proved in Section B.3, we already have an estimation of the approximation error’s convergence rate. So now we’ll focus on providing fast rate upper bounds of the generalization error for the two estimators using the localization techinque(Bartlett et al., 2005; Xu, 2020). To achieve the fast generalization bound, we focus on the following two normalized empirical processes:
S̃r,1(Ω) := { h̃1(x) :=
E[h1]− h1(x) E[h1] + E[h2] + r
| (h1, h2) ∈ S(Ω) } (r > 0),
S̃r,2(Ω) := { h̃2(x) :=
E[h2]− h2(x) E[h1] + E[h2] + r
| (h1, h2) ∈ S(Ω) } (r > 0).
where the space S(Ω) is defined as: S(Ω) := { (h1, h2) ∣∣h1 := |Ω| · [1
2
( ‖∇u‖2 − ‖∇u∗‖2 )] ,
h2 := |Ω| · [ 1
2 V (|u|2 − |u∗|2)− f(u− u∗)
] , u ∈ F (Ω) } .
First, we try to bound the expectation of the two normalized empirical processes. Applying the Symmetrization Lemma B.1, we can first bound the two expectations as:
sup h̃1∈S̃r,1(Ω)
Ey′ [ 1
N N∑ i=1 h̃1(y ′ i)
] ≤ Ey′ [ sup
h1∈S1(Ω)
∣∣∣ 1 N N∑ i=1 h1(y ′ i)− E[h1] E[h1] + E[h2] + r ∣∣∣] ≤ 2RN (Ŝr,1(Ω)), sup
h̃2∈S̃r,2(Ω) Ey
 1 n n∑ j=1 h̃2(yj)  ≤ Ey [ sup h2∈S2(Ω) ∣∣∣ 1 n n∑ i=1 h2(yj)− E[h2] E[h1] + E[h2] + r ∣∣∣] ≤ 2Rn(Ŝr,2(Ω)). where the function classes Ŝr,k(Ω) (1 ≤ k ≤ 2) are defined as:
Ŝr,1(Ω) := { ĥ1(x) :=
h1(x)
E[h1] + E[h2] + r | (h1, h2) ∈ S(Ω)
} ,
Ŝr,2(Ω) := { ĥ2(x) :=
h2(x)
E[h1] + E[h2] + r | (h1, h2) ∈ S(Ω)
} .
Applying the modified Peeling Lemma B.5 to any function h = (h1, h2) ∈ S(Ω) helps us upper bound the sum of the two local Rademacher complexities RN (Ŝr,1(Ω)) + Rn(Ŝr,2(Ω)) with the function φ defined in equation C.4:
RN (Ŝr,1(Ω)) +Rn(Ŝr,2(Ω)) = Eσ [ Ey [
sup h∈S(Ω)
1 N ∑N i=1 σih1(yi)
E[h1] + E[h2] + r
]] + Eτ [ Ey′ [
sup h∈S(Ω)
1 n ∑n j=1 τjh2(y ′ j)
E[h1] + E[h2] + r
]]
= Eσ [ Ey,y′ [ sup
h∈S(Ω)
1 N ∑N i=1 σih1(yi)
E[h1] + E[h2] + r + sup h∈S(Ω)
1 n ∑n j=1 τjh2(y ′ j)
E[h1] + E[h2] + r
]]
= RN,n(Ŝr(Ω)) ≤ 4φ(r)
r .
Combining all inequalities derived above yields:
sup h̃1∈S̃r,1(Ω)
Ey′ [ 1
N N∑ i=1 h̃1(y ′ i) ] + sup h̃2∈S̃r,2(Ω) Ey  1 n n∑ j=1 h̃2(yj)  ≤ 2RN (Ŝr,1(Ω)) + 2Rn(Ŝr,2(Ω)) = 2RN,n(Ŝr(Ω)) ≤ 8φ(r)
r (r > 0).
(C.6)
Secondly we’ll apply the Talagrand concentration inequality to the two function classes S̃r,1(Ω) and S̃r,2(Ω), which requires us to verify the conditions needed. We will first check that the expectation sum E[h1] + E[h2] is always non-negative for any (h1, h2) ∈ S(Ω):
E[h1] + E[h2] = 1
|Ω| ∫ Ω |Ω| · (1 2 ‖∇u(x)‖2 + 1 2 V (x)|u(x)|2 − f(x)u(x))dx
− 1 |Ω| ∫ Ω |Ω| · (1 2 ‖∇u?(x)‖2 + 1 2 V (x)|u?(x)|2 − f(x)u?(x))dx = E(u)−E(u?) ≥ 0⇒ E[h1] + E[h2] ≥ 0.
Next, We will verify that S̃r,1(Ω) satisfies all three requirements. At first, we will show that any h̃1 =
E[h1]−h1 E[h1]+E[h2]+r ∈ S̃r,1(Ω) is of bounded inf-norm. We need to prove that any h1 ∈ S1(Ω) is of
bounded inf-norm beforehand. Using boundedness condition listed in equation C.3 implies:
‖h1‖∞ = ‖ 1
2
( ‖∇u‖2 − ‖∇u∗‖2 ) ‖∞ ≤ 1
2
( ‖∇u‖2∞ + ‖∇u∗‖2∞ ) ≤ C2.
By taking M1 := C2, we then have ‖h1‖∞ ≤M1 for all h1 ∈ S1(Ω). Note that the denominator of h̃1 can be lower bounded by |E[h1] + E[h2] + r| ≥ r > 0. Combining these two inequalities help us upper bound the inf-norm ‖h̃1‖∞ = supx∈Ω |h̃1(x)| as follows:
‖h̃1‖∞ = ‖E[h1]− h1‖∞ |E[h1] + E[h2] + r| ≤ 2‖h1‖∞ r ≤ 2M1 r =: β1.
Also, it is easy to check that for any h̃1 ∈ S̃r,1(Ω), we have
E[h̃1] = E[h1]− E[h1]
E[h1] + E[h2] + r = 0,
i.e. any function in the localized class S̃r,1(Ω) is of zero mean. Moreover, we take σ21 = suph̃1∈S̃r,1(Ω) E[h̃ 2 1] to be the upper bound on the second moment of functions in S̃r,1(Ω). Now we have verified that any function h̃1 ∈ S̃r,1(Ω) satisfies all the required conditions. By taking µ to be the uniform distribution on the domain Ω and applying Talagrand’s Concentration inequality given in Lemma B.3, we have:
Px [ sup
h̃1∈S̃r,1(Ω)
1
N N∑ i=1 h̃1(xi) ≥ 2 sup h̃1∈S̃r,1(Ω) Ey [ 1 N N∑ i=1 h̃1(yi) ] + √ 2tσ21 N + 2tβ1 N ] ≤ e−t.
(C.7)
Moreover, We will verify that S̃r,2(Ω) also satisfies all three requirements. At first, we will show that any h̃2 =
E[h2]−h2 E[h1]+E[h2]+r ∈ S̃r,2(Ω) is of bounded inf-norm. We need to prove that any h2 ∈ S2(Ω)
is of bounded inf-norm beforehand. Using boundedness condition listed in equation C.3 implies:
‖h2‖∞ = ‖ 1
2 V (|u|2 − |u∗|2)− f(u− u∗)‖∞
≤ 1 2 Vmax
( ‖u‖2∞ + ‖u∗‖2∞ ) + ‖f‖∞ ( ‖u‖∞ + ‖u∗‖∞ ) ≤ 1
2 Vmax × 2C2 + 2C2 = (Vmax + 2)C2.
By taking M2 := (Vmax + 2)C2, we then have ‖h2‖∞ ≤ M2 for all h2 ∈ S2(Ω). Note that the denominator of h̃2 can be lower bounded by |E[h1] + E[h2] + r| ≥ r > 0. Combining these two inequalities help us upper bound the inf-norm ‖h̃2‖∞ = supx∈Ω |h̃2(x)| as follows:
‖h̃2‖∞ = ‖E[h2]− h2‖∞ |E[h1] + E[h2] + r| ≤ 2‖h2‖∞ r ≤ 2M2 r =: β2.
Also, it is easy to check that for any h̃2 ∈ S̃r,2(Ω), we have
E[h̃2] = E[h2]− E[h2]
E[h1] + E[h2] + r = 0,
i.e. any function in the localized class S̃r,2(Ω) is of zero mean. Moreover, we take σ22 = suph̃2∈S̃r,2(Ω) E[h̃ 2 2] to be the upper bound on the second moment of functions in S̃r,2(Ω). Now we have verified that any function h̃2 ∈ S̃r,2(Ω) satisfies all the required conditions. By taking µ to be the uniform distribution on the domain Ω and applying Talagrand’s Concentration inequality given in Lemma B.3, we have:
Px′  sup h̃2∈S̃r,2(Ω) 1 n n∑ j=1 h̃2(x ′ j) ≥ 2 sup h̃2∈S̃r,2(Ω) Ey′ [ 1 n n∑ j=1 h̃2(y ′ j) ] + √ 2tσ22 n + 2tβ2 n  ≤ e−t. (C.8) By applying a union bound to the two inequalities derived in C.7 and C.8, we can derive that with probability at least 1− 2e−t, the inequality below holds:
1
N N∑ i=1 h̃1(x ′ i) + 1 n n∑ j=1 h̃(xj) ≤ sup h̃1∈S̃r,1(Ω) 1 N N∑ i=1 h̃1(xi) + sup h̃2∈S̃r,2(Ω) 1 n n∑ j=1 h̃2(x ′ j)
≤ 2 sup h̃1∈S̃r,1(Ω)
Ey [ 1 N N∑ i=1 h̃1(yi) ] + √ 2tσ21 N + 2tβ1 N
+ 2 sup h̃2∈S̃r,2(Ω)
Ey′ [ 1 n n∑ j=1 h̃2(y ′ j) ] + √ 2tσ22 n + 2tβ2 n
≤ 16φ(r) r +
√ 2t
n (σ1 + σ2) +
2t(β1 + β2)
n .
By the definition of β1 and β2, we have that the term 2t(β1+β2)
n can be upper bounded by:
2t(β1 + β2)
n =
4t(M1 +M2) nr ≤ 4(Vmax + 3)C
2t
nr .
Now we will derive some upper bound on the sum σ1 + σ2. By definition we have that:
(σ1 + σ2) 2 ≤ 2(σ21 + σ22) = 2
[ sup
h̃1∈S̃r,1(Ω) E[h̃21] + sup h̃2∈S̃r,2(Ω) E[h̃22] ] = 2 [
sup h∈S(Ω)
E[h21]− E[h1]2
|E[h1] + E[h2] + r|2 + sup h∈S(Ω)
E[h22]− E[h2]2 |E[h1] + E[h2] + r|2 ]
≤ 4 sup h∈S(Ω) E[h21] + E[h22] |E[h1] + E[h2] + r|2 .
Now it suffices to derive an upper bound of E[h 2 1]+E[h 2 2]
|E[h1]+E[h2]+r|2 for any h ∈ S(Ω). The existence of such an upper bound is guaranteed because of the regularity results of the PDE. We aim to show that there exist some constants α, α′ > 0, such that for any h ∈ S(Ω), the following inequality holds:
α(E[h21] + E[h22]) ≤ ‖u− u∗‖2H1(Ω) ≤ α ′(E[h1] + E[h2]). (C.9)
The RHS of the inequality follows from strong convexity of the DRM objective function proved in Theorem B.1:
E[h1] + E[h2] = E(u)−E(u∗) ≥ min{1, Vmin}
4 ‖u− u∗‖2H1(Ω).
The LHS of the inequality follows from boundedness condition listed in equation C.3 and the QM-AM inequality:
E[h21] + E[h22] = ∫
Ω
1
4
( ‖∇u‖2 − ‖∇u∗‖2 )2 dx+ ∫ Ω [ 1 2 V (|u|2 − |u∗|2)− f(u− u∗) ]2 dx
≤ 1 4 ∫ Ω ( ‖∇u‖2 − ‖∇u∗‖2 )2 dx+ 1 2 ∫ Ω V 2(|u|2 − |u∗|2)2dx+ 2 ∫ Ω f2(u− u∗)2dx
≤ 1 4 ∫ Ω ∣∣∣‖∇u‖ − ‖∇u∗‖∣∣∣2(‖∇u‖+ ‖∇u∗‖)2dx+ 1 2 V 2max ∫ Ω ∣∣∣|u| − |u∗|∣∣∣2(|u|+ |u∗|)2dx + 2C2
∫ Ω (u− u∗)2dx ≤ C2 ∫ Ω ‖∇u−∇u∗‖2dx+ 2C2(1 + V 2max) ∫ Ω |u− u∗|2dx
≤ 2C2(1 + V 2max)‖u− u∗‖2H1(Ω).
By picking α′ = 4min{1,Vmin} and α = 1 2C2(1+V 2max) , we have finished proving inequality C.9. Then we can can upper bound the term E[h 2 1]+E[h 2 2]
|E[h1]+E[h2]+r|2 as:
E[h21] + E[h22] |E[h1] + E[h2] + r|2
≤ α′ α
( E[h1] + E[h2] ) 2r ( E[h1] + E[h2] ) ≤ α′ 2αr .
Combining the bounds derived above helps us upper bound the term √
2t n (σ1 + σ2) as below:√
2t n (σ1 + σ2) ≤
√ 8t
n
√ sup
h∈S(Ω) E[h21] + E[h22] |E[h1] + E[h2] + r|2
≤ √ 4α′t
nαr .
Thus, using the two upper bounds on √
2t n (σ1 + σ2) and 2t(β1+β2) n , we have
1
N N∑ i=1 h̃1(x ′ i) + 1 n n∑ j=1 h̃(xj) ≤ 16φ(r) r +
√ 2t
n (σ1 + σ2) +
2t(β1 + β2)
n
≤ 16φ(r) r +
√ 4α′t
nαr +
4(Vmax + 3)C 2t
nr = ψ(r).
Let’s pick the critical radius r0 to be:
r0 = max{214r∗, 24Mt
n ,
144α′t
αn }. (C.10)
Note that concavity of the function φ implies that φ(r) ≤ r for any r ≥ r∗. Combining this with the first inequality listed in C.4 yields:
16φ(r0)
r0 ≤
211φ( r0214 )
214 r0214 =
1 8 × φ( r0214 ) r) 214 ≤ 1 8 .
On the other hand, applying equation C.10 yields:√ 4α′t nαr0 ≤ √ 4α′t nα αn 144α′t = 1 6 ,
4(Vmax + 3)C 2t
nr0 ≤ 4(Vmax + 3)C
2t
n × n 24(Vmax + 3)C2t = 1 6 .
Summing the three inequalities above implies:
ψ(r0) = 16φ(r0)
r0 +
√ 4α′t
nαr0 +
4(Vmax + 3)C 2t
nr0 ≤ 1 8 + 1 6 + 1 6 < 1 2 .
By picking r = r0, we can further deduce that for any function u ∈ F (Ω), the following inequality holds with probability 1− 2e−t:
E(u)−E(u∗) + EN,n(u∗)−EN,n(u) E(u)−E(u∗) + r0 = 1 n n∑ i=1 h̃(xi) ≤ ψ(r0) < 1 2 .
Multiplying the denominator on both sides indicates:
∆Egen = E(u)−E(u∗) + EN,n(u∗)−EN,n(u) ≤ 1
2
[ E(u)−E(u∗) ] + 1
2 r0 =
1 2 ∆E(n) + 1 2 r0.
Substituting the upper bound above into the decomposition ∆E(n) ≤ ∆Egen + 2∆Eapp + 4tn yields that with probability 1− 4e−min{1,Cq}t, we have:
∆E(n) ≤ ∆Egen + 3
2 ∆Eapp +
t 2n ≤ 1 2 ∆E(n) + 1 2 r0 + 2∆Eapp + 4t n .
Simplifying the inequality above yields that with probability 1− 4e−min{1,Cq}t, we have:
∆E(n) ≤ r0 + 4∆Eapp + 8t
n = 3 inf uF∈F (Ω)
( E(uF )−E(u?) ) + max{214r∗, 24M t
n ,
36α′
α
t n }+ 8t n
. inf uF∈F (Ω)
( E(uF )−E(u?) ) + max { r∗, t
n
} .
Moreover, using strong convexity of the DRM objective function proved in Theorem B.1 implies:
∆E(n) = E(ûMDRM)−E(u∗) & ‖ûMDRM − u∗‖2H1(Ω).
Combining the two bounds above yields that with probability 1− 4e−min{1,Cq}t, we have:
‖ûMDRM − u∗‖2H1(Ω) . inf uF∈F (Ω)
( E(uF )−E(u?) ) + max { r∗, t
n
} .
Truncated Fourier Series Estimator. Next we aim to show that the truncated Fourier series estimator can achieve the min-max optimal rate using the MDRM objective function. For any ξ ∈ Z+ satisfying ξ2 < Nn , there exists some Truncated Fourier Series in Fξ(Ω) with approximation error ∆Eapp = O(ξ−2(s−1)) and generalization error ∆Egen = O( ξ d−2
n ). With optimal selection
ξ = n 1 d+2s−4 to balance the bias and variance, we can achieve n− 2s−2 d+2s−4 convergence rate for the Fourier estimator.
Theorem C.2. (Final Upper Bound of MDRM with Truncated Fourier Series Estimator) Under the assumptions in Theorem C.1, we consider the Modified Deep Ritz objective with the Truncated Fourier Series function space Fξ(Ω), where the parameter ξ = Θ(n 1 d+2s−2 ). By assuming ξ d N < ξd−2
n , we have that the Fourier estimator ûFourierMDRM = minu∈Fξ(Ω) E MDRM N,n (u) satisfies the following upper bound with high probability:
‖ûFourierMDRM − u∗‖2H1 . n − 2s−2d+2s−4
Proof. Following the same proof as shown for the DRM upper bound in Theorem B.11, we firstly need to determine the critical radius r∗:√
r∗ n ξ d−2 2 +
√ r∗
N ξ d 2 +
1 n + 1 N + ξ−2(s−1) ' r∗.
For we have assumed ξ d N < ξd−2 n , the solution of the fixed point equation is r ∗ ' ξ d−2 n + 1 n+ξ −2(s−1). On the other hand, by taking α = s and β = 1 in Lemma B.19 and applying strong convexity of the DRM objective function proved in Theorem B.1, we can upper bound the approximation error ∆Eapp as below:
∆Eapp . ξ −2(s−1).
Combining the two bounds above with Theorem C.1 yields that with high probability, we have:
‖ûFourierMDRM − u∗‖2H1 . ∆Eapp + r∗ . ξd−2
n +
1 n + ξ−2(s−1).
By equating the two of the three terms above, we can solve for ξ that yields the desired bound:
ξd−2
n ' ξ−2(s−1) ⇒ ξ ' n 1 d+2s−4 .
Plugging in the expression of ξ gives the final upper bound:
‖ûFourierMDRM − u∗‖2H1 . ξd−2
n + ξ−2(s−2) +
1 n . n− 2s−2 d+2s−4 .

heading: D PROOF OF THE LOWER BOUNDS

heading: D.1 PRELIMINARIES ON TOOLS FOR LOWER BOUNDS

text:
In this section, we list the standard tools we use to establish the lower bound. The main tool we use is the Fano’s inequailty and the Varshamov-Gilber Lemma. Lemma D.1 (Fano’s methods). Assume that V is a unifrom random variable over set V , then for any markov chain V → X → V̂ , we always have:
P(V̂ 6= V ) ≥ 1− I(V ;X) + log 2 log(|V|) .
Lemma D.2 (Varshamov-Gillbert Lemma,(Tsybakov, 2008) Theorem 2.9). Let D ≥ 8. There exists a subset V = {τ (0), · · · , τ (2D/8)} of D−dimensional hypercube HD = {0, 1}D such that τ (0) = (0, 0, · · · , 0) and the `1 distance between every two elements is larger than D8 :
D∑ l=1 ‖τ (j) − τ (k)‖`1 ≥ D 8 , for all 0 ≤ j, k ≤ 2D/8.

heading: D.2 PROOF OF LOWER BOUND

text:
In this section, we provide the proof of the lower bound for learning a PDE. Our proof uses standard Fano method to establish minimax lower bound but finally leads to a non-standard convergence rate. We state standard results for Fano methods in Appendix D.1. Following is the proof our main lower bound. Theorem D.1 (Lower bound). We denote u∗(f) to be the solution of the PDE 2.1 and we can access randomly sampled data {Xi, Yi}i=1,··· ,n as described in Section 2.2. DRM Lower Bound. For all estimators ψ : ( Rd )⊗n × R⊗n → Hs(Ω), we have
inf ψ sup u∗∈Hs(Ω)
E‖ψ({Xi, fi}i=1,··· ,n)− u∗(f)‖2H1 & n − 2s−2d+2s−4 . (D.1)
PINN Lower Bound. For all estimators ψ : ( Rd )⊗n × R⊗n → Hs(Ω), we have
inf ψ sup u∗∈Hs(Ω)
E‖ψ({Xi, fi}i=1,··· ,n)− u∗(f)‖2H2 & n − 2s−4d+2s−4 . (D.2)
Proof. We construct the following bump function to construct the multiple hypothesis test used for proving the lower bound. Consider a simple C∞ bump function supported on [0, 1]d
g(x) = d∏ i=1 ξ(xi), x = (x1, · · · , xd),
where ξ : R → R is a non-zero funtion in C∞(R) with support contained in [0, 1] and satisfies ξ(x) 6= 0, ddxξ(x) 6= 0. Then ∇g(x) 6= 0 and the support of function g is [0, 1] d.
Next, we takem = [n 1
2s−4+d ] and consider a regular gird x(j), j ∈ [m]d. According to the VarshamovGilbert lemma, there exist 2m d/8 (0, 1)-sequences τ (1), · · · , τ (2m d/8) ∈ {0, 1}md such that ‖τ (k) − τ (k ′)‖2 ≥ m d
8 for all 0 < k 6= k ′ ≤ 2md/8. Then we construct the multiple hypothesis as
uk(x) = ∑ j∈[m]d τ (k) j ω ms+ d 2 g(m(x− x(j))), k = 1, 2, · · · , 2m d/8,
where ω is a constant to be determined later. It is easy to find out that uk ∈ Cs. Then we reduce solving the PDE to a multiple hypothesis testing problem, which considers all mappings from n sampled data to the constructed hypothesis Ψ : ( Rd )⊗n × R⊗n → V := {ui|i = 1, 2, · · · , 2md/8}. Then we apply the local Fano method and check that we can obtain a constant lower bound of P(V̂ 6= V ) for any estimator V̂ . From the local Fano method, we know that
I(V ;X) ≤ 1 |V|2 ∑ z ∑ v 6=v′ DKL(Pv||P ′v),
where Pk denotes the joint distribution of the sampled data (x, y). In specific, x follows a uniform distribution on [0, 1]d and y = f(X) + , where is independently sampled from a standard Gaussian distribution N(0, 1). Then we have
KL(Pk||Pk′) = E log( dPk dPk′ ) = ‖∆uk + V uk‖2L2 ≤ Cω m2s−4 .
Using Fano inequality, if we select m ∝ [n 1
2s−4+d ] then we have the following lower bound when ω is taken to be sufficiently large:
P(V̂ 6= V ) ≥ 1− I(V ;X) + log 2 log(|V|)
≥ 1− 8Cω m2s−4
md log 2 ≥ 1/2.
At the same time, we can estimate the separation of the hypotheses in two different norms:
• Deep Ritz Method:∫ [0,1]d ‖∇uk−∇uk′‖2dx = κ2 m2s−2+d ∑ j∈[m]d ‖τ (k)j −τ (k′) j ‖1 ∫ Rd ‖∇g(x)‖2dx & 1 m2s−2 .
• Physic Informed Neural Network:∫ [0,1]d ‖∆uk −∆uk′‖2dx = κ2 m2s−4+d ∑ j∈[m]d ‖τ (k)j − τ (k′) j ‖1 ∫ Rd ∆g(x)2dx & 1 m2s−4 .
Plugging in m ∝ [n 1 2s−4+d ], we know that with constant probability we have
inf ψ sup u∗∈Hs(Ω)
E‖ψ({Xi, Yi}i=1,··· ,n)− u∗(f)‖2H1 & n − 2s−2d+2s−4 , (D.3)
inf ψ sup u∗∈Hs(Ω)
E‖ψ({Xi, Yi}i=1,··· ,n)− u∗(f)‖2H2 & n − 2s−4d+2s−4 . (D.4)
E INTUITION BEHIND THE SUB-OPTIMALITY OF THE UNMODIFIED DEEP RITZ METHODS
In this section, we aim to discuss the intuition behind the sub-optimality of the unmodified DRM via using the truncation Fourier basis. To simplify the notation, in this section we consider the following simplest Poisson equation ∆u = f on the hypercube with zero Dirichlet boundary condition. To illustrate the necessity of the modification we made, we consider the difference between the following two estimators
• Estimator 1. We use the truncated Fourier basis estimator to learn the right hand side function f and then we invert the PDE exactly to get the estimated u.
• Estimator 2. We plug in a parametrization of the truncated fourier basis into the empirical DRM objective
We would like to point out that estimator 1 isn’t build for computational consideration. Instead, we use it to consider the statistical limit of our sampled data. We first show that the estimator 1 can achieve the minimax optimal estimation error.
Error Of Estimator 1 Firstly, we show that if one wants to learn the function u in H10 norm, one need to learn the right hand side function f in H−10 norm. The H −1 0 norm is defined as the dual norm of the H1 norm, i.e. ‖u‖H−10 = max‖v‖H10≤1 〈u, v〉. Once we assume we have an estimate f̂ of f in H−10 , we can have an estimate of u via û := (∆) −1 f̂ , whose distance to u in the H1 norm satisfies:
‖∇u−∇û‖H10 = max‖v‖ H10 ≤1 〈∇u−∇û,∇v〉
= max ‖v‖ H10 ≤1 〈∆u−∆û, v〉
= max ‖v‖
H10 ≤1
〈 f − f̂ , v 〉 = ‖f − f̂‖H−1 .
Estimator 1 using the truncated Fourier estimator to estimate the right hand side function f . Suppose we can access a random sample of observed data as {xi, f(xi)}ni=1, then the Fourier coefficient fz := 〈u, φz〉 can be estimated as f̂z := 1n ∑n i=1 f(xi)φz(xi). To bound the estimation error of
f̂ := ∑ ‖z‖∞≤Z f̂zφz in H −1 0 , we first apply the bias-variance decomposition:
E‖f̂ − f‖2 H−10 ≤ ‖Ef̂ − f̂‖2 H−10 + E‖f − Ef̂‖2 H−10
We first bound the bias term ‖Ef̂ − f‖2H−1 . Given Ef̂ = ∑ ‖z‖∞≤Z fzφz , we have that for a truncation set Z of the from Z := {z ∈ Nd|‖z‖∞ ≤ Z}, the bias term can be controlled by:
‖ ∑
‖z‖∞>Z
fzφz‖2H−1 ≤ C ∑
‖z‖∞>Z
f2z z −2 ≤ ‖z‖−2(s−1)‖f‖2Hα−2
Next we estimate the variance of the estimator by decomposing the variance into the following sum: E‖f − f̂‖2H−1 ≤ E ∑
‖z‖∞≤Z
(f̂z − fz)2‖φz‖2H−1 ≤ ∑
‖z‖∞≤Z
|z|−1Var(f̂z).
Finally we achieve a Z−2(s−1) + Z d−2
n upper bound for estimator 1. With optimal selection of Z, we
can achieve the min-max optimal convergence rate n− 2s−2 d+2s−4 .
Difference Between Estimator 1 and Estimator 2 Next we aim to understand the Deep Ritz Method objective function via plugging in a truncated Fourier series estimator. We consider an estimator of the form u = ∑ ûzφz(x), which lies in the space of truncated Fourier series. Then the empirical DRM objective function can be expressed as
1
2n n∑ i=1 (∑ z ûz∇φz(xi) )2 + ∑ z ûzφz(xi)f(xi). (E.1)
We observe that (E.1) is a quadratic formula with respect to the Fourier coefficients u := (uz)‖z‖∞≤Z . Thus, we can rewrite it as the following matrix form
min 1
2 u>Âu + u>f̂ , where Â =
( 1
n n∑ i=1 ∇φi(xi)∇φj(xi) ) ‖i‖∞≤Z,‖j‖∞≤Z . (E.2)
Based on the matrix formulation E.2, we can compare the solution given by the two estimators
• Estimator 1: The Fourier coefficients of the solution of Estimator 1 are û1 = diag ( ‖z‖2 )−1 ‖z‖∞≤Z f̂ . (E.3)
• Estimator 2: The Fourier coefficients of the solution of Estimator 2 are
û2 = Â −1f̂ . (E.4)
Note that EÂ = ( ‖z‖2 ) ‖z‖∞≤Z
. Thus, we can further introduce another variance from the sampling of A. By directly estimating û1 − û2, we will show that this term will be larger than the final convergence rate. Notice that
‖û1 − û2‖2H1 = f> ( (EÂ)−1 − Â−1 )> diag ( ‖z‖2 ) ‖z‖∞≤Z ( (EÂ)−1 − Â−1 ) f (E.5)
Next we aim to bound ( (EÂ)−1 − Â−1 ) . We first use the Matrix Bernstein Inequality(Tropp, 2015)
to bound the H1 distance between û1 and û2. According to the Matrix Bernstein Inequality, we have that with probability 1− e−t, the following inequality holds∥∥∥((EÂ)− Â)∥∥∥ H ≤ √ Zd n + t n , (E.6)
where ‖ · ‖H is the matrix operator norm respect to the vector ‖ · ‖H defined as ‖z‖2H = z>diag ( ‖z‖2 )−1 ‖z‖∞≤Z
z. Note that( I + (EÂ)−1 ( Â− (EÂ) ))( (EÂ)−1 − Â−1 ) = (EÂ)−1 ( Â− (EÂ) ) (EÂ)−1 (E.7)
When n is large enough, we know that 12I 6 I + (EÂ) −1 ( Â− (EÂ) ) 6 I with high probability.
Thus the term ‖û1−û2‖2H1 is at the scale of ∥∥∥((EÂ)− Â)∥∥∥2
H ≈ Z
d
n , which is of the same magnitude
as what we get from the empirical process approach in our main proof. It is also larger than Z d−2
n , which is the magnitude of the variance term for û1. Therefore, here we conjecture that the our bound for DRM itself is tight and leads to the sub-optimal convergence rate.

title: MACHINE LEARNING FOR ELLIPTIC PDES: FAST RATE GENERALIZATION BOUND, NEURAL SCAL-