summaries: The authors consider a binary classification task. As a model the authors use a deep fully-connected neural network and train it to separate the submanifolds, representing different classes. They assume that sub-manifolds belong to the unit sphere. Also, the authors restrict their analysis to a one-dimensional case. The main claim is that by increasing depth we can improve model generalization of a network, trained by SGD.

summaries: The paper studies the conditions for a deep fully-connected network to separate low-dimensional data classes. A binary classification setting is considered, where the two classes are modelled as two different manifolds. The manifolds are assumed to be one-dimensional for the ease of analysis. It is shown that the network depth should be sufficiently large so as to adapt to the geometrical properties of data (e.g. the manifold curvature); the network width should increase polynomially with the network depth; and the number of data samples should also scale at a polynomial rate with the network depth. The authors show that if these conditions are met, with high probability a randomly initialized network converges to a classifier that separates the two class manifolds. The proof technique relies on conditioning the network parameters of the l-th layer on the parameters of the previous layers using a Martingale model, which gives sharp concentration guarantees.

summaries: The paper under review studies the question of whether gradient descent can solve the problem of calibrating a deep neural network for separating two submanifolds of the sphere. The problem studied in the paper is very interesting and as been the subject of recent increasing interest in the machine learning community. The contribution is restricted to a simple set up and addresses the question in the finite sample regime. The framework of the analysis hinges on the Neural Tangent Kernel approximation of Jacot et al.

abstractText: We study the multiple manifold problem, a binary classification task modeled on applications in machine vision, in which a deep fully-connected neural network is trained to separate two low-dimensional submanifolds of the unit sphere. We provide an analysis of the one-dimensional case, proving for a simple manifold configuration that when the network depth L is large relative to certain geometric and statistical properties of the data, the network width n grows as a sufficiently large polynomial in L, and the number of i.i.d. samples from the manifolds is polynomial in L, randomly-initialized gradient descent rapidly learns to classify the two manifolds perfectly with high probability. Our analysis demonstrates concrete benefits of depth and width in the context of a practically-motivated model problem: the depth acts as a fitting resource, with larger depths corresponding to smoother networks that can more readily separate the class manifolds, and the width acts as a statistical resource, enabling concentration of the randomlyinitialized network and its gradients. The argument centers around the “neural tangent kernel” of Jacot et al. and its role in the nonasymptotic analysis of training overparameterized neural networks; to this literature, we contribute essentially optimal rates of concentration for the neural tangent kernel of deep fullyconnected ReLU networks, requiring width n ≥ Lpoly(d0) to achieve uniform concentration of the initial kernel over a d0-dimensional submanifold of the unit sphere Sn0−1, and a nonasymptotic framework for establishing generalization of networks trained in the “NTK regime” with structured data. The proof makes heavy use of martingale concentration to optimally treat statistical dependencies across layers of the initial random network. This approach should be of use in establishing similar results for other network architectures.

authors:
P-A Absil
R Mahony
R Sepulchre

title: Optimization Algorithms on Matrix Manifolds

authors:
Zeyuan Allen-Zhu
Yuanzhi Li

title: What can resnet learn efficiently, going beyond kernels

venue: In Advances in Neural Information Processing Systems,

title: Backward feature correction: How deep learning performs deep learning

venue: arXiv preprint arXiv:2001.04413,

authors:
Zeyuan Allen-Zhu
Yuanzhi Li
Yingyu Liang

title: Learning and generalization in overparameterized neural networks, going beyond two layers

authors:
Zeyuan Allen-Zhu
Yuanzhi Li
Zhao Song

title: A convergence theory for deep learning via OverParameterization

venue: In Proceedings of the 36th International Conference on Machine Learning,

authors:
Sanjeev Arora
Simon Du
Wei Hu
Zhiyuan Li
Ruosong Wang

title: Fine-Grained analysis of optimization and generalization for overparameterized Two-Layer neural networks

authors:
Sanjeev Arora
Simon S Du
Wei Hu
Zhiyuan Li
Russ R Salakhutdinov
Ruosong Wang

title: On exact computation with an infinitely wide neural net

title: Weighted sums of certain dependent random variables

venue: Tohoku Mathematical Journal, Second Series,

authors:
Stéphane Boucheron
Maud Thomas

title: Concentration inequalities for order statistics

venue: Electronic Communications in Probability,

authors:
Stéphane Boucheron
Gábor Lugosi
Pascal Massart

title: Concentration Inequalities: A Nonasymptotic Theory of Independence

venue: OUP Oxford,

title: Functional Analysis, Sobolev Spaces and Partial Differential Equations

authors:
Joan Bruna
Stéphane Mallat

title: Invariant scattering convolution networks

venue: IEEE Trans. Pattern Anal. Mach. Intell.,

authors:
Yuan Cao
Quanquan Gu

title: Generalization bounds of stochastic gradient descent for wide and deep neural networks

authors:
Zixiang Chen
Yuan Cao
Difan Zou
Quanquan Gu

title: How much over-parameterization is sufficient to learn deep ReLU networks

venue: In International Conference on Learning Representations,

authors:
Lénaïc Chizat
Francis Bach

title: A note on lazy training in supervised differentiable programming

title: Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss

venue: In Proceedings of Thirty Third Conference on Learning Theory, volume 125 of Proceedings of Machine Learning Research,

authors:
Youngmin Cho
Lawrence K Saul

title: Kernel methods for deep learning

venue: In Advances in neural information processing systems,

authors:
Taco Cohen
Max Welling

title: Group equivariant convolutional networks

venue: In Proceedings of The 33rd International Conference on Machine Learning,

authors:
R M Corless
G H Gonnet
D E G Hare
D J Jeffrey
D E Knuth

title: On the LambertW function

venue: Adv. Comput. Math.,

title: Order Statistics, pp. 1039–1040

title: A general class of exponential inequalities for martingales and ratios

venue: Ann. Probab.,

authors:
David L Donoho
Carrie Grimes

title: Image manifolds which are isometric to euclidean space

venue: J. Math. Imaging Vis.,

authors:
Simon S Du
Xiyu Zhai
Barnabas Poczos
Aarti Singh

title: Gradient descent provably optimizes over-parameterized neural networks

authors:
Lawrence Craig Evans
Ronald F Gariepy

title: Measure Theory and Fine Properties of Functions

authors:
Cong Fang
Jason D Lee
Pengkun Yang
Tong Zhang

title: Modeling from features: a mean-field framework for over-parameterized deep neural networks

venue: arXiv preprint arXiv:2007.01452,

authors:
Charles Fefferman
Sanjoy Mitter
Hariharan Narayanan

title: Testing the manifold hypothesis

venue: J. Amer. Math. Soc.,

title: On tail probabilities for martingales

authors:
Federica Gerace
Bruno Loureiro
Florent Krzakala
Marc Mezard
Lenka Zdeborova

title: Generalisation error in learning with random features and the hidden manifold model

venue: In Proceedings of the 37th International Conference on Machine Learning,

authors:
Behrooz Ghorbani
Song Mei
Theodor Misiakiewicz
Andrea Montanari

title: Linearized two-layers neural networks in high dimension

title: When do neural networks outperform kernel methods

authors:
Sebastian Goldt
Marc Mézard
Florent Krzakala
Lenka Zdeborová

title: Modeling the influence of data structure on learning in neural networks: The hidden manifold model

venue: Phys. Rev. X,

authors:
Boris Hanin
Mihai Nica

title: Finite depth and width corrections to the neural tangent kernel

title: A Basis Theory Primer: Expanded Edition

venue: Birkhäuser Boston,

authors:
Roger A Horn
Charles R Johnson

title: Topics in matrix analysis

venue: Cambridge university press,

authors:
Arthur Jacot
Franck Gabriel
Clement Hongler

title: Neural tangent kernel: Convergence and generalization in neural networks

authors:
Ziwei Ji
Matus Telgarsky

title: Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networks

authors:
Michel Ledoux
Michel Talagrand

title: Probability in Banach Spaces: Isoperimetry and Processes

authors:
Jaehoon Lee
Lechao Xiao
Samuel Schoenholz
Yasaman Bahri
Roman Novak
Jascha SohlDickstein
Jeffrey Pennington

title: Wide neural networks of any depth evolve as linear models under gradient descent

title: Introduction to Riemannian Manifolds. Springer, Cham, 2 edition, 2018

authors:
Yuanzhi Li
Tengyu Ma
Hongyang R Zhang

title: Learning Over-Parametrized Two-Layer neural networks beyond NTK

venue: In Proceedings of Thirty Third Conference on Learning Theory,

authors:
Tengyuan Liang
Alexander Rakhlin
Xiyu Zhai

title: On the multiple descent of Minimum-Norm interpolants and restricted lower isometry of kernels

title: Group invariant scattering

venue: Commun. Pure Appl. Math.,

authors:
Song Mei
Andrea Montanari
Phan-Minh Nguyen

title: A mean field view of the landscape of two-layer neural networks

venue: Proc. Natl. Acad. Sci. U. S. A.,

authors:
Song Mei
Theodor Misiakiewicz
Andrea Montanari

title: Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit

venue: In Proceedings of the Thirty-Second Conference on Learning Theory,

authors:
Takeru Miyato
Toshiki Kataoka
Masanori Koyama
Yuichi Yoshida

title: Spectral normalization for generative adversarial networks

authors:
Andrea Montanari
Yiqiao Zhong

title: The interpolation phase transition in neural networks: Memorization and generalization under lazy training

venue: arXiv preprint arXiv:2007.12826,

title: Aspects of Multivariate Statistical Theory. Wiley Series in Probability and Statistics

authors:
Quynh Nguyen
Marco Mondelli
Guido Montufar

title: Tight bounds on the smallest eigenvalue of the neural tangent kernel for deep ReLU networks

venue: arXiv preprint arXiv:2012.11654,

authors:
Atsushi Nitanda
Taiji Suzuki

title: Optimal rates for averaged stochastic gradient descent under neural tangent kernel regime

authors:
Samet Oymak
Zalan Fabian
Mingchen Li
Mahdi Soltanolkotabi

title: Generalization guarantees for neural networks via harnessing the low-rank structure of the jacobian

venue: arXiv preprint arXiv:1906.05392,

authors:
Mark Rudelson
Roman Vershynin

title: Non-asymptotic theory of random matrices: Extreme singular values

venue: In Proceedings of the International Congress of Mathematicians 2010 (ICM

authors:
Ernest Ryu
Jialin Liu
Sicheng Wang
Xiaohan Chen
Zhangyang Wang
Wotao Yin

title: Plugand-Play methods provably converge with properly trained denoisers

title: Curves of finite total curvature

venue: In Discrete differential geometry,

authors:
Y Sun
J Liu
U S Kamilov

title: Block coordinate regularization by denoising

venue: IEEE Transactions on Computational Imaging,

title: Generalization bound of globally optimal non-convex neural network training: Transportation map estimation by infinite dimensional langevin dynamics

title: Concentration of measure and isoperimetric inequalities in product spaces

venue: Publications Mathématiques de l’Institut des Hautes Etudes Scientifiques,

authors:
Belinda Tzen
Maxim Raginsky

title: A mean-field theory of lazy training in two-layer neural nets: entropic regularization and controlled McKean-Vlasov dynamics

venue: arXiv preprint arXiv:2002.01987,

title: High-dimensional probability: An introduction with applications in data science, volume 47

authors:
Michael B Wakin
David L Donoho
Hyeokho Choi
Richard G Baraniuk

title: The multiscale structure of non-differentiable image manifolds

venue: In Wavelets XI,

authors:
Xu Wang
Konstantinos Slavakis
Gilad Lerman

title: Multi-Manifold Modeling in Non-Euclidean spaces

venue: In Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics,

title: Sharp asymptotic and finite-sample rates of convergence of empirical measures in wasserstein distance

title: The comparison geometry of ricci curvature

title: d, and δ in the theorem statement suffice to satisfy the hypotheses of Lemma B.7. We have already satisfied the conditions on ε and q. We notice that (B.2) implies that it suffices to enforce simply N ≥ L10, and following Lemma C.4, we can bound CM

title: The constant CM in Lemma C.5 can be sharpened if more is known about the curvature ofM: if Ric ≥ 0, the exponential dependence on curvature and diameter can be removed (intuitively, taking k ↘ 0 “recovers” this from the proved result), and if Ric > 0, the dependence on diameter can be completely removed using Myers’ theorem (Zhu

title: S̆ Proof. This result is a generalization of a well known comparison theorem of Schur’s to higher dimensions following the proof in (Sullivan, 2008), where we additionally specialize to the case where one of the curves is an arc. Given a curve γ satisfying the conditions of the lemma, we first consider an arc γ̃ of a circle

heading: 1 INTRODUCTION

text:
Data in many applications in machine learning and computer vision exhibit low-dimensional structure (Fig. 1a). Although deep neural networks achieve state-of-the-art performance on tasks in these areas, rigorous explanations for their performance remain elusive, in part due to the complex interaction between models, architectures, data, and algorithms in neural network training. There is a need for model problems that capture essential features of applications (such as low dimensionality), but are simple enough to admit rigorous end-to-end performance guarantees. In addition to helping to elucidate the mechanisms by which deep networks succeed, this approach has the potential to clarify the roles of various network properties and how these should reflect the properties of the data.
These considerations lead us to formulate the multiple manifold problem (Fig. 1b), a binary classification problem in which the classes are two disjoint submanifolds of the unit sphere Sn0−1, and the classifier is a deep fully-connected ReLU network of depth L and width n trained on N i.i.d. samples from a distribution supported on the manifolds. The goal is to articulate conditions on the network architecture and number of samples under which the learned classifier provably separates the two manifolds, guaranteeing perfect generalization to unseen data. The difficulty of an instance of the multiple manifold problem is controlled by the dimension of the manifolds d0, their separation ∆, and their curvature κ, allowing us to study the constraints imposed by these intrinsic properties of the data on the settings of the neural network’s architectural hyperparameters such that the two manifolds can be separated by training with a gradient-based method.
Our main result is an analysis of the one-dimensional case of the multiple manifold problem, which reduces the analysis of the gradient descent dynamics to the construction of a certificate—showing that a certain deterministic integral equation involving the network architecture and the structure of the data admits a solution of small norm. We construct such a certificate for the simple geometry in Fig. 3, guaranteeing generalization in this setting.
Theorem 1 (informal). Let d0 = 1. Suppose a certificate forM exists. Then if the network depth satisfies L ≥ poly(κ,Cρ, log(n0)), the width satisfies n ≥ poly(L, log(Ln0)), and the number of training samples satisfies N ≥ poly(L), randomly-initialized gradient descent on N i.i.d. samples rapidly learns a network that separates the two manifolds with overwhelming probability. The constants Cρ, κ depend only on the data density and the regularity of the manifolds. In addition, if L & ∆−1, then a certificate exists for the configuration ofM shown in Fig. 3.
Theorem 1 gives a provable generalization guarantee for a model classification problem with deep networks on structured data that depends only on the architectural hyperparameters and properties of the data. In addition, it provides an interpretable tradeoff between the architectural settings necessary to separate the two manifolds: the network depth needs to be set according to the intrinsic difficulty of the problem, and the network width needs to grow with the depth. Our analysis gives further insight into the independent roles played by each of these parameters in solving the problem, with the depth acting as a ‘fitting resource’, making the network’s output more regular and easier to change, and the width acting as a ‘statistical resource’, granting concentration of the network over the random initialization around a well-behaved object that we can analyze. Moreover, the sample complexity of Theorem 1 is dictated by the intrinsic difficulty of the problem instance which is set by the geometry of the data. As a consequence, we avoid any dependence of the width of the network on the number of samples, which is common in deep network convergence results in the literature (e.g. (Allen-Zhu et al., 2019b; Du et al., 2019), (Chen et al., 2021, Theorem 3.4)). As is the case in practice, given a fixed architecture, more data doesn’t have a detrimental effect on fitting 1.
Theorem 1 is modular, in the sense that a generalization guarantee is ensured for any geometry for which one can construct a certificate. The key to our approach will be to approximate the gradient
1When using data augmentation, for example, the number of samples is effectively infinite yet highly structured, enabling convergence and generalization.
descent dynamics with a linear discrete dynamical system defined in terms of the so-called neural tangent kernel Θ(x,x′) defined on the manifolds. Due to the structure in the data, diagonalizing the operator corresponding to this kernel is intractable in general, but we show that constructing a certificate—arguably an easier task, because it requires producing a bound on the norm of a solution to an equation rather than producing the solution itself—suffices to guarantee that the error decreases rapidly during training given a suitably structured network.
We summarize the primary contributions of this work below.
• Generalization in deep networks: There are few generalization results for deep networks trained efficiently with gradient descent available in the literature.2 Theorem 1 provides such a guarantee that does not depend on any property of the trained network (e.g., norms of final weights) that is not readily available before training. In this context, the certificate condition is equivalent to the initial network function having a controlled norm in a certain RKHS; this condition is natural in the training regime we consider, and appears ubiquitously in works on generalization in shallower networks (Ghorbani et al., 2020; Ji & Telgarsky, 2020; Nitanda & Suzuki, 2021).
• Uniform concentration of the neural tangent kernel for deep ReLU networks: As an intermediate step in the proof of Theorem 1, we establish essentially optimal rates of uniform concentration for the neural tangent kernel of an arbitrarily deep network (Theorem 2) using martingale concentration, where we require the width to grow only linearly with the depth. We expect this martingale approach to be applicable to essentially any other compositionally-structured network architecture. Our uniform result generalizes prior results on pointwise concentration (Arora et al., 2019b; Allen-Zhu et al., 2019b), analogous to our Theorem B.3, and proves useful in establishing generalization.
• Strong regularity estimates for random ReLU networks: As a further consequence of the uniform concentration framework we have developed, we obtain depth-logarithmic Lipschitz estimates for random ReLU networks of arbitrary depth and linear width, as well as (for still wider networks) a uniform approximation for the network output by a constant which improves with depth, both with overwhelming probability (Section 3.3). We also control the evolution of the Lipschitz constant during NTK regime training (Lemma B.7), showing that it scales polynomially in the depth. These results may be of interest in applications where guaranteeing a Lipschitz property for networks is important, such as GAN training (Miyato et al., 2018) or denoising (Ryu et al., 2019; Sun et al., 2020).

heading: 1.1 RELATED WORK

text:
Deep networks and low-dimensional structure. The notion of modeling data as low-dimensional submanifolds has been widely considered in the context of clustering (Wang et al., 2015) and manifold learning (Donoho & Grimes, 2005; Fefferman et al., 2016). Goldt et al. (2020) independently proposed the “hidden manifold model”, a model problem for learning shallow neural networks for binary classification of structured data with motivations very similar to ours and which admits a mean-field analysis (Gerace et al., 2020). The data model consists of gaussian samples from a low-dimensional subspace passed through a nonlinear function acting coordinatewise in the standard basis; although this models statistical variations around a base domain, a feature of real data that the model we study here lacks, we believe that the study of an arbitrary density supported on two Riemannian manifolds lends our data model increased structural generality. In the context of kernel regression with the kernel given by the NTK of a two-layer neural network, Ghorbani et al. (2020) study a data generating model that consists of uniform samples from a low-dimensional subsphere corrupted additively by independent uniform samples from a subsphere in the orthogonal complement, and a target mapping that depends only on the low-dimensional part. The authors obtain asymptotic generalization guarantees for this data model that reveal conditions under which the corruption degrades the performance of neural tangent methods.
2The closest result we are aware of is (Chen et al., 2021, Theorem 3.4); this result involves a-priori assumptions on the trained network weights, which are only resolved for two-layer networks, and entails an unnatural relationship between n and N and a possible exponential dependence of N on L, which Theorem 1 avoids.
Analyses of neural network training. To reason analytically about the complicated training process, we adopt the neural tangent kernel approach (Jacot et al., 2018). The first works to instantiate these ideas in a nonasymptotic setting obtained convergence guarantees for training deep neural networks on finite datasets (Allen-Zhu et al., 2019b; Du et al., 2019). By exploiting more structure in the data, generalization results have been obtained (Allen-Zhu et al., 2019a; Arora et al., 2019a; Ji & Telgarsky, 2020; Oymak et al., 2019; Cao & Gu, 2019; Suzuki, 2020; Li et al., 2020; AllenZhu & Li, 2020) that apply to shallow networks, teacher-student learning scenarios, and/or hold conditional on the existence of certain small-norm interpolators. Other works have obtained generalization guarantees using generalization bounds for kernel methods (Ghorbani et al., 2019; Liang et al., 2020; Ghorbani et al., 2020; Montanari & Zhong, 2020) using the fact that the linearized predictor in the NTK regime can be linked to a kernel method (Arora et al., 2019b). A parallel line of works (Mei et al., 2018; Tzen & Raginsky, 2020; Mei et al., 2019; Chizat & Bach, 2020; Fang et al., 2020) approach the problem by studying an infinite-width limit of neural network training that yields a different training dynamics. Approaches of this type are of interest because there is no restriction to short-time dynamics, and the limit of the dynamics can often be characterized in terms of a well-structured object, such as a max-margin classifier (Chizat & Bach, 2020). On the other hand, it is often difficult to prove finite-time convergence to the limit.

heading: 2 PROBLEM FORMULATION AND MAIN RESULTS

heading: 2.1 DATA MODEL AND NETWORK DEFINITIONS

text:
We consider data supported on the union of two class manifoldsM =M+ ∪M−, whereM+ and M− are two disjoint smooth regular simple curves taking values in Sn0−1, with n0 ≥ 3. We denote the data measure supported onM that generates our samples as µ∞, with corresponding density ρ, and write ρmin = infx∈M ρ(x) and ρmax = supx∈M ρ(x). We denote by κ a uniform bound on the (extrinsic) curvature of the two curves, we write ∆ = minx∈M+,x′∈M− ∠(x,x
′) for the separation between class manifolds, where ∠(x,x′) = cos-1〈x,x′〉 for unit vectors, and to have a quantitative characterization of ‘how simple’ the curves are, we assume there exist constants 0 < cλ ≤ 1, Kλ ≥ 1 such that for every 0 < s ≤ cλ/κ and every x,x′ in a common connected component of M, one has distM(x,x′) ≤ Kλs if ∠(x,x′) ≤ s, where distM denotes the Riemannian distance. Our target function is the signed indicator for each class manifold f?(x) = 1M+(x)− 1M−(x). The model we consider is a fully-connected neural network with ReLU activations and access to i.i.d. samples from µ∞ and their corresponding labels. We parameterize our neural network with weights W 1 ∈ Rn×n0 , W ` ∈ Rn×n if ` ∈ {2, . . . , L}, and WL+1 ∈ R1×n, which we collect as θ = (W 1, . . . ,WL+1), and write the iterates of the forward pass as α`θ(x) = [ W `α`−1θ (x) ] +
for ` = 1, . . . , L with α0θ(x) = x, which we also refer to as features or activations, with the network output written as fθ(x) = WL+1αLθ (x), and the prediction error as ζθ(x) = fθ(x)− f?(x).
For an i.i.d. sample (x1, . . . ,xN ) from µ∞, we write µN = 1N ∑N i=1 δxi for the empirical measure
associated to the sample, and we consider the training objective LµN (θ) = 12 ∫ M (ζθ(x)) 2 dµN (x), i.e. the empirical risk evaluated with the square loss. We train with vanilla gradient descent with constant step size τ > 0: after randomly initializing the parameters θN0 as W
` ∼i.i.d. N (0, 2/n) if ` ∈ [L] andWL+1 ∼i.i.d. N (0, 1) we consider the sequence of iterates θNk+1 = θNk −τ∇̃LµN (θNk ), where ∇̃LµN represents a ‘formal gradient’ of the empirical loss, which we define in detail in Appendix A.1. We say the parameters obtained at iteration k of gradient descent separate the manifolds M if the classifier implemented by the neural network with the parameters θNk labels the two manifolds correctly, i.e. if f?(x) sign(fθNk (x)) = 1 for every x ∈ M. As a shorthand, we will denote quantities evaluated at θNk with a subscript k; an omitted subscript will denote k = 0, and we will write explicitly θ0 = θN0 . Additional notation is provided in Appendix A.5.1.

heading: 2.2 ERROR DYNAMICS AND CERTIFICATES

text:
Because it is difficult to endow the network parameters generated by the gradient iteration with a particular interpretation, we prefer to reason about how the network error ζNk evolves under gradient
descent. We calculate (in Lemma B.8)
ζNk+1(x) = ζ N k (x)− τ ∫ M ΘNk (x,x ′)ζNk (x ′) dµN (x′), (2.1)
where we have defined the integral kernel ΘNk (x,x ′) = ∫ 1 0 〈∇̃fθNk (x ′), ∇̃fθNk −tτ∇̃LµN (θNk )(x)〉dt, where ∇̃fθ0 denotes a formal gradient of the initial network function with respect to the parameters, which is defined in detail in Appendix A.1. We then define a nominal error evolution by
ζ∞k+1(x) = ζ ∞ k (x)− τ ∫ M Θ(x,x′)ζ∞k (x ′) dµ∞(x′) (2.2)
with identical initial conditions ζ∞0 = ζ and where Θ(x,x ′) = 〈∇̃fθ0(x), ∇̃fθ0(x′)〉 is the socalled neural tangent kernel with associated integral operator Θ. We prove that the error evolution (2.1) is well-approximated by the nominal error evolution under suitable conditions on the network width, step size, and number of samples, which together ensure that training proceeds in the “NTK regime” where ΘNk stays close to Θ. As for the nominal error evolution (2.2), we note that this system is linear, time-invariant, and stable when τ is set appropriately small, so the norm of the nominal error is guaranteed to decrease rapidly if the initial error ζ aligns well with eigenfunctions of Θ corresponding to large eigenvalues. However, computation of these eigenfunctions is intractable for general data geometries and distributions because the operator Θ is not generally translationally invariant on M. To overcome this issue, we prove this alignment implicitly by constructing an approximate solution to the linear integral equation Θ[g] = ζ such that ‖g‖L2
µ∞ is sufficiently
small. To be precise, g ∈ L2µ∞ will be called a δ1, δ2-certificate for the dynamics (2.2) if
‖Θ[g]− ζ‖L2 µ∞ ≤ δ1; ‖g‖L2 µ∞ ≤ δ2. (2.3)

heading: 2.3 MAIN RESULTS AND PROOF OUTLINE

text:
Our main result is that conditional on the existence of a certificate of suitably small norm forM, gradient descent provably separates the two manifolds in time polynomial in the network depth.
Theorem 1. LetM be a one-dimensional Riemannian manifold satisfying our regularity assumptions. For any 0 < δ ≤ 1/e, choose
L ≥ C1 max { Cµ∞ log 9(1/δ) log24 (Cµ∞n0 log(1/δ)) , κ 2K2λ/c 2 λ } ,
n = C2L 99 log9(1/δ) log18(Ln0),
N ≥ L10,
and fix τ such that C3nL2 ≤ τ ≤ C4 nL .
Then if there exists a certificate in the sense of (2.3) with δ1 = C5C 1/2 ρ √ log(1/δ) log(nn0)/L and
δ2 = C6 √ log(1/δ) log(nn0)/(nρ 1/2 min), with probability at least 1−δ over the random initialization of the network and the i.i.d. sample from µ∞, the parameters obtained at iteration bL39/44/(nτ)c of gradient descent on the finite sample loss LµN yield a classifier that separates the two manifolds. The constants C1, . . . , C6 are suitably chosen absolute constants, the constants κ, Kλ, cλ are respectively the extrinsic curvature constant and the global regularity constant defined in Section 2.1, the constant Cρ is defined as max{ρmin, ρ−1min}, and the constant Cµ∞ is defined as C15ρ (1 + ρmax) 6 (min {µ∞(M+), µ∞(M−)})−11/2.
For one-dimensional instances of the two manifold problem with sufficiently deep and overparameterized networks trained in the small-step-size regime, Theorem 1 completely reduces the analysis of the gradient iteration to the certificate problem. From a qualitative perspective, the network resource constraints imposed by Theorem 1 are natural:
(i) The network depth L is set by geometric and statistical properties of the data with only a mild polylogarithmic dependence on the ambient dimension n0, which reflects the role of depth in controlling the capability of the network to fit functions.
(ii) The network width n is set by the depth L: the inductive structure of the network causes quantities that depend on the initial random weights θ0 to concentrate worse as the depth is increased, which can be counteracted by setting the width appropriately large.
(iii) The sample complexity of N ≥ L10 reflects the capacity of the network via the depth, and is in particular independent of the width n, which can thus be interpreted as purely a statistical resource.
In addition, the conclusion of Theorem 1 implies not just that the expected generalization error with respect to µ∞ of a binary classifier is zero, but the stronger separation property, i.e. that the generalization error will be zero for any choice of test distribution supported onM simultaneously. We give a brief sketch of the proof of Theorem 1 in Appendix A.4. To obtain a generalization guarantee from Theorem 1, it only remains to construct a certificate forM. We demonstrate this for the family of simple, highly-symmetric geometries shown in Figure 3, and leave the case of general one-dimensional manifolds for future work.
Proposition 1. Let M be an r-instance of the two circles geometry studied in Appendix C.1.1 and shown in Figure 3, with r ≥ 1/2. For any 0 < δ ≤ 1/e, if L ≥ C1∆−1 and n ≥ C2L
5 log4(1/δ) log4(Ln0 log(1/δ)), then there exists a certificate in the sense of (2.3) satisfying the requirements of Theorem 1 with probability at least 1− 3δ for some absolute constants C1, C2 > 0.
Taking a union bound, Proposition 1 shows that under the hypotheses of Theorem 1, with probability at least 1 − 4δ a certificate exists for the geometry shown in Figure 3 as soon as L is larger than a constant multiple of the inverse separation ∆−1, even as the separation approaches zero. We conjecture that a similar phenomenon holds for more general geometries, possibly with additional dependencies on the curvature and global regularity parameters ofM. The dependence of L on the geometry is due to the “sharpening” effect the depth has on the kernel Θ governing the dynamics and thus on the fitting capabilities of the network, as illustrated in Figure 2a.
To prove that the nominal error evolution (2.2) decreases rapidly and approximates the actual error evolution (2.1) throughout training, it is essential to have a precise characterization of the ‘initial’ neural tangent kernel Θ. One of our main technical contributions is to show concentration of Θ in the regime where the width n scales linearly with the depth L.
Theorem 2. For any d0 ∈ N, let M be a d0-dimensional complete Riemannian submanifold of Sn0−1. Then if n ≥ C1Ld40 log
4(CMn0L), one has with probability at least 1 − n−10 that for every (x,x′) ∈ M×M∣∣∣∣∣Θ(x,x′)− n2 L−1∑ `=0 cos ( ϕ(`)(ν) ) L−1∏ `′=` ( 1− ϕ (`′)(ν) π )∣∣∣∣∣ ≤ C2 √ nL3d40 log 4(CMnn0),
where we write ν = ∠(x,x′) with an abuse of notation, ϕ(`) denotes the `-fold composition of ϕ(ν) = cos−1 ( (1− νπ ) cos ν + sin ν π ) , the constants C1, C2 > 0 are absolute, and the constant CM > 0 depends only on the diameters and curvatures of the class manifolds (Lemma C.4). 3
For networks of uniform width that are wider than they are deep by a certain constant factor, we believe that the scalings in Theorem 2 are essentially optimal: the variance calculations of Hanin & Nica (2020) give some heuristic evidence here, and we believe the idea of using diagonal concentration to prove deviation lower bounds could be generalized to rigorously establish optimality. Figure 2b illustrates the phenomenon underlying Theorem 2. We discuss the proof of Theorem 2 in more detail in Sections 3.1 and 3.3.

heading: 3 KEY PROOF ELEMENTS

heading: 3.1 CONCENTRATION AT INITIALIZATION: MARTINGALES AND ANGLE CONTRACTION

text:
The initial kernel Θ is a complicated random process defined over the weights (W 1, . . . ,WL+1). To control it, we first show for fixed (x,x′) that Θ(x,x′) concentrates with high probability, and then leverage approximate continuity properties to pass to uniform control of Θ. Here we describe our approach to pointwise control; uniformization is discussed in Section 3.3. The kernel can be written in the form
Θ(x,x′) = 〈αL(x),αL(x′)〉+ L−1∑ `=0 〈α`(x),α`(x′)〉〈β`(x),β`(x′)〉,
where β`(x) = (WL+1PIL(x) · · ·W `+2PI`+1(x))∗ will be referred to as backward features, and PI`(x) is a projection onto {α`(x) > 0}. We consider 〈β0(x),β0(x′)〉 as a representative example: up to a small residual term, this random variable can be expressed as a sum of martingale differences. Formally, for ` ∈ [L], let F` denote the σ-algebra generated by all weight matrices up to layer `, with F0 denoting the trivial σ-algebra. We can then write∣∣〈β0(x),β0(x′)〉 − g0(ν0)∣∣ ≤
∣∣∣∣∣ L+1∑ `=1 g`(W `, . . . ,W 1, ν0)− E [ g`(W `, . . . ,W 1, ν0) ∣∣ F`−1]∣∣∣∣∣+R
(3.1) for some functions g` and controllable residualR, where ν0 = ∠(x,x′). If we fix all the variables in F`−1, the fluctuations in the `-th summand will be due to W ` alone. Intuitively, since each weight matrix appears at most once in β0(x),4 it will appear at most twice in g`, and therefore g` will have a subexponential distribution conditioned on F`−1 and concentrate well around its conditional expectation. This property stems from the compositional structure of the network, with independent sources of randomness introduced at every layer, and is essentially agnostic to other details of the architecture. The concentration of the summands in (3.1) implies concentration of the sum: even though the summands are not independent, they can be controlled using concentration inequalities analogous to those for sums of independent variables (Azuma, 1967; Freedman, 1975).
Showing that terms of the form 〈α`(x),α`(x′)〉 concentrate in the linear regime gives rise to additional challenges. Here we exploit an essential difference between the concentration properties
3Since we do not use the “NTK parameterization”, the norm of our NTK scales like nL rather than L. Due to our scaling of the weights (Section 2.1) the contribution of the final layer to the NTK is negligible and can be dropped. This leads to discrepancies between the expression above and similar expressions found in the literature—we show essential equivalence between our NTK and others in Appendix A.3.
4Technically, the features α`(x) depend on all the weights up to layer ` and hence so does the projection matrixPI`(x), but our analysis shows that this dependence has only a minor effect on the statistical fluctuations.
of the angles between features ν` = ∠(α`(x),α`(x′)) relative to those of the correlation process 〈α`(x),α`(x′)〉 studied in prior works on concentration of Θ: when ν`−1 = 0, we have that ν` = 0 deterministically, whereas the correlation process behaves like a subexponential random variable with small but nonzero deviations. Together with smoothness, this clamping phenomenon allows us to show concentration of the angle at layer ` around the function ϕ(`)(ν0), which is no larger than a constant multiple of `−1. This contraction of the angles with depth is the key to establishing Theorem 2; in addition, it gives the invariant kernel Θ̂ (see Figure 2b) its sharpness at zero and localization properties, both of which increase as the depth is increased and which we exploit in the proof of Proposition 1. We provide full details of our approach in Appendices D and E.
3.2 CERTIFICATE CONSTRUCTION: GENERAL FORMULATION AND A SIMPLE EXAMPLE
If the network is sufficiently deep, the norm of the nominal error can thus be made arbitrarily small in a number of iterations that scales only polynomially with the problem parameters.
Because our formulation of the certificate problem (2.3) accommodates approximate solutions, under a minor condition on the network width n (see Proposition 1) it suffices to solve an auxiliary system Θ̂[g] = ζ̂, where Θ̂ and ζ̂ are analytically-convenient approximations to Θ and ζ produced by our concentration analysis, including Theorem 2. For the simple geometry in Fig. 3, we show in Appendix C.1.1 how to solve this auxiliary system using Fourier analysis, where we require L & ∆−1. The depth of the network is thus determined by the geometry of the data, and specifically by the inter-manifold distance which intuitively sets the “difficulty” of the fitting problem. In Section 4 we discuss approaches to constructing certificates for general smooth curves.

heading: 3.3 UNIFORM CONCENTRATION AND ITS CONSEQUENCES

text:
To uniformize the pointwise estimates of Section 3.1, we must overcome the issue that the backward features β`(x) are not continuous functions of the input, due to the matrices PI`(x). Our approach is to discretize the input space, control the number of features that can change sign near each point in the discretization, then extend the pointwise estimates of Section 3.1 to the setting where a small number of features have changed sign—again, we find martingale concentration a necessity to achieve linear width-depth scaling. We give full details in Appendix D.3.
Although Theorem 2 is the main application of these estimates—with uniform control of Θ, we can prove operator norm bounds on its corresponding integral operator Θ, which is of great help in proving generalization results—they also imply useful regularity estimates for the initial random network fθ0 . For example, we prove that networks of uniform width n n40L are with high probability √ n0(log n0)(logL)-Lipschitz as functions on Rn0 (Theorem B.5)—in particular, the Lipschitz constant depends only logarithmically on depth, in contrast to existing results in the literature (Nguyen et al., 2020). For networks of larger width n & d30L
5, we prove that with high probability the network fθ0 is approximately constant on the domainM⊂ Sn0−1 (Lemma D.11):
sup x∈M ∣∣∣∣fθ0(x)− ∫ M fθ0(x ′) dµ∞(x′) ∣∣∣∣ . 1L. We find this result to be useful in simplifying the certificate construction problem of Section 3.2.

heading: 4 DISCUSSION

text:
Certificates for curves. The most urgent task toward expanding the scope of Theorem 1 is the construction of certificates for geometries beyond the coaxial circles of Proposition 1. The proof
of Proposition 1 relies heavily on translation invariance of the intra- and inter-manifold distances in the coaxial circles geometry in order to avoid the need for certain sharp technical estimates for the decay of the kernel Θ̂. With sharper control of the decay of the kernel Θ̂, it is possible to select the network depth in a way that grants appropriate worst-case control of the magnitude of the crossmanifold integrals in the action of Θ̂ (as in Figure 2a), allowing us to reduce to what is essentially a one-manifold certificate construction problem that can be solved with harmonic analysis. Beyond these considerations, it is important to extend Theorem 1 to manifolds of dimension d0 > 1, which should be relatively straightforward. Our concentration results, notably including Theorem 2, are already applicable to manifolds of arbitrary dimension.
Convolutional networks and non-differentiable manifolds. Although we have motivated our data model in the multiple manifolds problem using applications in computer vision, it is important to note that the spatially-structured image articulation manifolds that arise as data in these contexts do not carry a differentiable structure (Wakin et al., 2005), so the assumption of bounded curvature may not be realistic here. On the other hand, in these applications it is standard to employ a convolutional network architecture. We anticipate that our martingale concentration framework can be extended to these architectures, and beyond establishing analogues of Theorem 1 in this setting, we believe it should be possible to obtain similar guarantees for models of image articulation manifolds. In particular, one might expect randomly-initialized convolutional networks to enjoy local invariance properties, like the scattering networks of Mallat (Mallat, 2012; Bruna & Mallat, 2013), which could achieve a degree of invariant classification without expending additional network resources computing convolutions over general LCA groups (Cohen & Welling, 2016).
The importance of being low-dimensional. Ghorbani et al. (2019) show that kernel ridge regression with any rotationally invariant kernel on Sd (including that of a deep network) is equivalent to polynomial regression with a degree p polynomial if the number of samples is bounded by dp+1 and d→∞. For data lying on a low-dimensional manifold, as we consider here, one would expect less pessimistic rates; indeed, in a subsequent work (Ghorbani et al., 2020) the authors establish similar guarantees to Ghorbani et al. (2019) for a linear data model with low-dimensional structure in terms of a smaller “effective dimension”. In comparison, although our present certificate construction argument only implies dynamics for the restrictive coaxial circles geometry of Figure 3, for which one can obtain guarantees for kernel regression with a shallow NTK by the results of Ghorbani et al. (2020), the general multiple manifold problem formulation allows one to model nonlinear structure in the data, and measures fitting difficulty through intrinsic parameters like the curvature and separation. The guarantees in Ghorbani et al. (2019; 2020) depend on the degree of approximability of the target function by low-degree polynomials, and although this achieves additional generality over our model, it seems more challenging to relate this to geometric or other types of nonlinear low-dimensional structure.
The NTK regime and beyond. In recent years there has been much work devoted to the analysis of networks trained in the regime where the changes in ΘNk remain small and the dynamics in (2.1) are close to linear (Jacot et al., 2018; Lee et al., 2019; Arora et al., 2019b; Allen-Zhu & Li, 2019) (referred to as the NTK/“overparametrized”/kernel regime). Concurrently, there have also been results highlighting the limitations of this regime. In (Chizat & Bach, 2018) the authors coin the term “lazy training” in referring to dynamics where the relative change in the differential of the network function is small compared to the change in the objective during gradient descent. While the dynamics we study indeed fall into this category, the analysis makes it evident that not all lazy training regimes are created equal. Our performance guarantees depend on the structure of the kernel Θ̂, and on controlling the fluctuations of ΘNk around it. We are able to control these only if the width of the network is sufficiently large compared to the depth. In contrast, lazy training can also be achieved in homogeneous models by simply scaling the output of the model (Chizat & Bach, 2018), in which case one cannot argue that the kernel has the decay properties that enable it to fit data.
Our analysis hinges on staying in the NTK regime during training. We obtain suboptimal scaling of n with L in Theorem 1 because we treat all changes that occur in ΘNk during training as being adversarial to the algorithm’s ability to generalize. It is likely that if an improved understanding of feature learning can be incorporated into an analysis of the dynamics, the resulting scaling requirements would be more realistic.

heading: ACKNOWLEDGMENTS

text: This work was supported by the grants NSF 1733857, NSF 1838061, NSF 1740833, NSF 1740391, NSF NeuroNex Award DBI-1707398 (DG), the Gatsby Charitable Foundation (DG) and a Swartz fellowship (DG), and by a fellowship award (SB) through the National Defense Science and Engineering Graduate (NDSEG) Fellowship Program, sponsored by the Air Force Research Laboratory (AFRL), the Office of Naval Research (ONR) and the Army Research Office (ARO). The authors would like to thank Ethan Dyer, Guy Gur-Ari, Quynh Nguyen, Jeffrey Pennington, Sam Schoenholz, Daniel Soudry, and Tingran Wang for helpful discussions/feedback.

heading: CONTENTS

heading: A Extended Problem Formulation 16

text:
A.1 Regarding the Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
A.2 Regarding the Data Manifolds . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
A.3 Regarding the Initialization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
A.4 Proof Outline for Theorem 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
A.5 Notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
A.5.1 General Notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
A.5.2 Summary of Operator and Error Definitions . . . . . . . . . . . . . . . . . 21

heading: B Proofs of the Main Results 23

text:
B.1 Main Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
B.2 Supporting Results on Dynamics . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
B.3 Auxiliary Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

heading: C Skeleton Analysis and Certificate Construction 61

text:
C.1 Certificate Construction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
C.1.1 Two Circles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
C.2 Auxiliary Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
C.2.1 Geometric Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
C.2.2 Analysis of the Skeleton . . . . . . . . . . . . . . . . . . . . . . . . . . . 71

heading: D Concentration at Initialization 83

text:
D.1 Notation and Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
D.2 Pointwise Concentration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
D.2.1 Forward Concentration . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
D.2.2 Backward Feature Control . . . . . . . . . . . . . . . . . . . . . . . . . . 94
D.3 Uniformization Estimates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
D.3.1 Nets and Covering Numbers . . . . . . . . . . . . . . . . . . . . . . . . . 101
D.3.2 Controlling Support Changes Uniformly . . . . . . . . . . . . . . . . . . . 101
D.3.3 Uniformizing Forward Features Under SSC . . . . . . . . . . . . . . . . . 108
D.3.4 Small Support Change Residuals . . . . . . . . . . . . . . . . . . . . . . . 120
D.4 Auxiliary Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148

heading: E Sharp Bounds on the One-Step Angle Process 159

text:
E.1 Definitions and Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
E.2 Main Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
E.3 Supporting Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
E.3.1 Core Supporting Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
E.3.2 Proving Lemma E.6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
E.3.3 Proving Lemma E.7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179
E.3.4 General Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186
E.3.5 Differentiation Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190
E.3.6 Miscellaneous Analytical Results . . . . . . . . . . . . . . . . . . . . . . 202
E.4 Deferred Proofs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232

heading: F Controlling Changes During Training 235

text:
F.1 Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235
F.2 Changes in Feature Supports During Training . . . . . . . . . . . . . . . . . . . . 235
F.3 Changes in Features During Training . . . . . . . . . . . . . . . . . . . . . . . . . 237
F.4 Changes in ΘNk During Training . . . . . . . . . . . . . . . . . . . . . . . . . . . 241
F.5 Auxiliary Lemmas and Proofs . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242

heading: G Auxiliary Results 249

heading: APPENDICES: SUMMARY OF CONTENTS

text:
We briefly summarize the contents of each of the subsequent appendices.
A. We discuss the contents of the problem formulation section from the main body, Section 2.1, in more technical detail, in particular giving technical definitions for formal gradients, regularity conditions, and so on. We provide a proof sketch to offer some intuitions about the proof of the main result. We also summarize notation and the key operator definitions that appear throughout the paper.
B. We give proofs for our main results. We provide supporting results on the NTK regime dynamics of gradient descent and other relevant technical lemmas, as discussed in the proof sketch of Section 2.3.
C. We give technical definitions relevant to the cross-manifold perspective on certificate construction, construct a certificate for the two circles geometry of Figure 3, and provide technical estimates on the kernels ψ1 and ψ that remain after applying our measure concentration arguments to the NTK Θ.
D. We collect results on measure concentration relevant to proving our main uniform concentration result for the NTK, Theorem 2. Some of these results are also relevant for controlling changes during training.
E. We collect results relevant to proving a certain concentration result for the angles between features as they propagate across one layer of the initial neural network. The main results of this section are fundamental to the study of the concentration of angles in Appendix D, and we provide them in a separate appendix due to their length.
F. We establish results on uniform control of the changes during training of the NTK ΘNk from its “initial value” of Θ. These are a key ingredient in our dynamics arguments in Appendix B.
G. We provide statements of general technical lemmas that are of a classical nature, which we rely on throughout the other appendices.

heading: A EXTENDED PROBLEM FORMULATION

heading: A.1 REGARDING THE ALGORITHM

text:
We analyze a gradient-like method for the minimization of the empirical loss LµN . After randomly initializing the parameters θN0 as W
` ∼i.i.d. N (0, 2/n) if ` ∈ [L] and WL+1 ∼i.i.d. N (0, 1), independently of the samples x1, . . . ,xN , we consider the sequence of iterates
θNk+1 = θ N k − τ∇̃LµN (θNk ), (A.1)
where τ > 0 is a step size, and ∇̃LµN represents a ‘formal gradient’ of the loss LµN , which we define as follows: first, we define formal gradients of the network output by
∇̃W `fθ(x) = β`−1θ (x)α `−1 θ (x) ∗
for ` ∈ [L] and x ∈M, where we have introduced the definitions β`θ(x) = ( WL+1PIL(x)W LPIL−1(x) . . .W `+2PI`+1(x) )∗ for ` = 0, 1, . . . , L− 1, and where we additionally define
I`(x) = supp ( 1α`θ(x)>0 ) , PI`(x) = ∑ i∈I`(x) eie ∗ i
for the orthogonal projection onto the set of coordinates where the `-th activation at input x is positive. We call the vectors β`θ(x) the backward features or backward activations—they correspond to the backward pass of our neural network. We also define
∇̃WL+1fθ(x) = αLθ (x)∗.
We then define the formal gradient of the loss LµN by ∇̃LµN (θ) = ∫ M ∇̃fθ(x)ζθ(x) dµN (x).
Let us emphasize again that the expressions above are definitions, not gradients in the analytical sense: we introduce these definitions to cope with nonsmoothness of the ReLU [ · ]+. On the other hand, our formal gradient definitions coincide with the expressions one obtains by applying the chain rule to differentiate LµN at points where the ReLU is differentiable, and we will make use of this fact to proceed with these formal gradients in a manner almost identical to the differentiable setting.
We reiterate here our notational conventions for quantities evaluated at these iterates: we denote evaluation of quantities such as the features and prediction error at parameters along the gradient descent trajectory using a subscript k, with an omitted subscript denoting evaluation at the initial k = 0 parameters, and we add a superscript N to parameters such as the prediction error to emphasize that they are evaluated at the parameters generated by (A.1). For example, in this notation we express ζθNk as ζ N k . In addition, we use θ0 to denote the initial parameters θ N 0 . We emphasize the dependence of certain quantities on these random initial parameters notationally, including the initial network function fθ0 .

heading: A.2 REGARDING THE DATA MANIFOLDS

text:
We now provide additional details regarding our assumptions on the data manifolds. For background on curves and more broadly Riemannian manifolds, we refer the reader to (Lee, 2018; Absil et al., 2009). We assume thatM =M+∪M−, whereM+ andM− are two disjoint complete connected5 Riemannian submanifolds of the unit sphere Sn0−1, with n0 ≥ 3. In particular,M± are compact. We take as metric on these manifolds the metric induced by that of the sphere, which we take in turn as that induced by the euclidean metric on Rn0 . We write µ∞+ and µ∞− for the measures on M+
5Certain parts of our argument, such as the concentration result Theorem B.2, are naturally applicable to cases whereM± themselves have a finite number of connected components with a mild dependence on this number, and we state them as such. We skip this extra generality in our dynamics arguments to avoid an additional ‘juggling act’ that would obscure the main ideas.
andM− (respectively) induced by the data measure µ∞, and we assume that µ∞ admits a density ρ with respect to the Riemannian measure onM, writing ρ+ and ρ− for the densities onM± induced by the density ρ. When d0 = 1, we add additional structural assumptions to the above: we assume thatM± are smooth, simple, regular curves. Concretely, thatM admits a density ρ with respect to the Riemannian measure means that
1 = ∫ M dµ∞(x) = ∫ M+ ρ+(x) dV+(x) + ∫ M− ρ−(x) dV−(x).
When d0 = 1, becauseM± are smooth regular curves, they admit global unit-speed parameterizations with respect to arc length γ± : I± → Sn0−1, where I± are intervals of the form [0, len(M±)]. In this setting, the curvature constraint is expressed as
max { sup s∈I+ ∥∥γ′′+(s)∥∥2 , sup s∈I− ∥∥γ′′−(s)∥∥2 } ≤ κ,
and we observe that the fact thatM± are sphere curves implies κ ≥ 1.6 Exploiting the coordinate representation of the Riemannian measure and the fixed inherited metric from Rn0 , we thus have∫
M± ρ±(x) dV±(x) = ∫ I± ρ± ◦ γ±(t)‖γ′±(t)‖2 dt = ∫ I± ρ± ◦ γ±(t) dt.
We will exploit this formula in the sequel to compare between Lpµ(M) and Lp(M) norms of functions defined on the manifold. More generally, we will frequently make use of similar reasoning that leverages the existence of unit-speed parameterizations for the curves.
For clarity we rewrite the global regularity condition: we assume there exist constants 0 < cλ ≤ 1, Kλ ≥ 1 such that
∀s ∈ (0, cλ/κ] , (x,x′) ∈M? ×M?, ? ∈ {+,−} : ∠(x,x′) ≤ s⇒ distM(x,x′) ≤ Kλs, (A.2)
where distM denotes the Riemannian distance between points in a common connected component, and we define Cλ = K2λ/c 2 λ. Because M± are simple curves, they do not self-intersect; the assumption (A.2) gives a quantitative characterization of how far the curves are from self-intersecting. We illustrate how the associated constants can be obtained from the assumption that the manifolds are simple curves: for either ? ∈ {+,−}, consider a connected componentM? ⊂ M, and for any 0 < s ≤ len(M?), define
r?(s) = inf x,x′∈M?×M?, distM(x,x ′)>s
∠(x,x′).
If r?(s) = 0, by compactness we can construct a sequence of pairs of points that converges to r?(s), but this would imply thatM? is self-intersecting, contradicting our assumption that it is simple. It follows that r?(s) > 0 for any value of s. If we now define K̃s = r?(s)/s, it follows that for any (x,x′) ∈M? ×M?, ∠(x,x′) ≤ s⇒ distM(x,x′) ≤ K̃ss. Our regularity assumption implies that a single such constant holds for a range of scales below the curvature scale, which is a mild assumption since K̃s approaches 1 as s approaches 0.

heading: A.3 REGARDING THE INITIALIZATION

text:
The manner in which we have defined our initial random neural network fθ0 is sometimes referred to as “fan-out initialization” in the literature—it guarantees that feature norms are preserved from layer to layer in the network, and thereby avoids the vanishing and exploding gradient problems. The difference between this initialization and the so called “standard" or “fan-in” initialization is only in the first and last layer weights, yet in a sufficiently deep network trained in the NTK regime the effect of any single layer is negligible and the dynamics of our network will be essentially identical to one with standard initialization. On the other hand, following the work of Jacot et al.
6We point out that the curvature of the manifolds does not enter into the proof of the concentration result Theorem B.2, so there is no ambiguity in discussing curvature only in the context of curves.
(2018), it has become common in the theoretical literature to consider a different construction of the neural network called “NTK parameterization”, which is in some ways more convenient for theoretical analysis. In particular, Arora et al. (2019b) prove their results on NTK concentration using this parameterization; to facilitate a comparison between our concentration result (Theorem 2) and theirs, we discuss the connection between fan-out and NTK parameterization in this section. This material is well-known and no doubt can be found already in the literature, but we believe it may be helpful to translate it into our notation.
Recall our definitions for the weights and features in our neural network: we have W ` ∼i.i.d. N (0, 2/n) if ` ∈ {0, 1 . . . , L} andWL+1 ∼i.i.d. N (0, 1), with features defined for ` = 0, 1, . . . , L by
α`(x) = { x ` = 0[ W `α`−1(x) ] + otherwise,
and output fθ0(x) = W L+1αL(x). Within this section—and only within this section—we shall define auxiliary weights byG(1) ∈ Rn×n0 ,G(`) ∈ Rn×n for integer 1 < ` < L+ 1, andG(L+1) ∈ R1×n, with distributions G(`) ∼i.i.d. N (0, 1) for ` ∈ {0, 1 . . . , L + 1}, independent of everything else in the problem. As before, for ` ∈ {0, 1, . . . , L}we useα(`)NTK(x) to denote the layer-` features:
α (`) NTK(x) = { x ` = 0[ G(`)α
(`−1) NTK (x) ] + otherwise.
This network’s output will be written
fNTK(x) = ( L∏ `=1 √ 2 n ) G(L+1)α (L) NTK(x).
By 1-homogeneity (absolute) of σ, it follows that fθ0 d = fNTK. As the notation suggests, the network fNTK corresponds to a “NTK parameterization” network—although this network and fθ0 are equivalent in terms of predictions, their “gradients” are not equivalent. The NTK for the NTK parameterization network is obtained by differentiating (at points of differentiability): after calculating (as in Lemma B.8), we introduce notation as we did for the fan-out parameterization network in Appendix A.1, so that
ΘNTK(x,x ′) = 〈 ∇̃fNTK(x), ∇̃fNTK(x′) 〉 ,
with (for ` = 1, . . . , L+ 1)
∇̃G(`)fNTK(x) = ( L∏ `=1 √ 2 n ) β (`−1) NTK (x)α (`−1) NTK (x) ∗
where
β (`) NTK(x) =
{( G(L+1)PINTKL (x)G (L)PINTKL−1 (x) · · ·G (`+2)PINTK`+1 (x) )∗ ` = 0, 1, . . . , L− 1
1 ` = L,
and INTK` (x) = supp ( 1 α
(`) NTK(x)>0
) .
We shall relate the NTK parameterization NTK ΘNTK to our fan-out parameterization NTK Θ using homogeneity of the ReLU. First, let us observe that{
i ∈ [n] ∣∣ (α`(x)) i > 0 } d = { i ∈ [n] ∣∣∣ (α(`)NTK(x)) i > 0 } .
because [ · ]+ is 1-homogeneous and we haveG(`) d = √ n/2W ` when ` ≤ L. For ` ∈ {0, 1, . . . , L}, we note that both α`(x) and α(`)NTK(x) depend only on the parameters (W 1, . . . ,W `) and (G(1), . . . ,G(`)), respectively. If we write θ = (W 1, . . . ,WL) and θNTK = (G(1), . . . ,G(L)), then it follows that α`(x) is a `-homogeneous function of θ (and likewise for α(`)NTK(x)). In
addition, the projection matrices PI`(x) are 0-homogeneous functions of θ, and so taking ` ∈ {0, 1, . . . , L − 1} and counting parameters in the definitions of β`(x) and β(`)NTK(x) implies that these two functions are (L− `−1)-homogeneous functions of θ and θNTK, respectively. Of course, for ` = L, they are 0-homogeneous. Thus, using thatG(`) d= √ n/2W ` for ` ≤ L again, we obtain
∇̃G(`)fNTK(x) d = {√ 2/n∇̃W `fθ0(x) ` = 0, 1, . . . , L ∇̃W `fθ0(x) ` = L+ 1.
Although we have argued equidistributionality above for each index ` separately for simplicity, the elementary nature of our arguments (we are just moving scalars around) and the statistical dependencies across gradients allows us to apply the same argument ‘in parallel’ to the sum of inner products between gradients, yielding
ΘNTK(x,x ′) d = 〈 αL(x),αL(x′) 〉 + 2
n L∑ `=1 〈 α`−1(x),α`−1(x′) 〉〈 β`−1(x),β`−1(x′) 〉 .
This expression makes it immediately clear that our concentration framework proves sharp concentration of the NTK of a uniform-width NTK parameterization feedforward ReLU network that improves over the results of Arora et al. (2019b) when the data are on the sphere7 —a simple adaptation of the proof of Theorem B.3 will suffice.

heading: A.4 PROOF OUTLINE FOR THEOREM 1

text:
In Appendix B, we prove a slightly more general version of Theorem 1 in Theorem B.1. Here, we give a brief outline of the proof of this result.
Proving the separation property essentially requires us to obtain control of ‖ζNk ‖L∞(M), and by an interpolation inequality (Lemma B.14) it suffices to control the generalization error ‖ζNk ‖L2µ∞ and the smoothness (measured through the Lipschitz constant) of ζNk . We start with the generalization error, picking up from where we left off at the end of Section 2.2: the triangle inequality gives∥∥ζNk ∥∥L2 µ∞ ≤ ‖ζ∞k ‖L2 µ∞ + ∥∥ζ∞k − ζNk ∥∥L2 µ∞ , (A.3)
which allows us to divide the analysis into two subproblems: characterizing the nominal dynamics (Lemmas B.6 and B.12), and the nominal-to-finite transition (Lemma B.7). Beginning with the nominal dynamics, we use (2.2) to write
ζ∞k = (Id−τΘ) k [ζ] ,
where Θ denotes the operator on L2µ∞ corresponding to integration against the kernel Θ and Id denotes the identity operator. The definition of Θ and compactness ofM imply that Θ is a positive, compact operator (Lemma B.9), so these dynamics are stable when τ is chosen larger than the operator norm of Θ. However, the rate of decrease of ‖ζ∞k ‖L2µ∞ with k could still be extremely slow if the initial error ζ has significant components in the direction of eigenfunctions of Θ corresponding to small eigenvalues, and because Θ acts roughly like a convolution operator, we expect there to exist eigenvalues arbitrarily close to zero. By solving the certificate problem (2.3), we can assert that misalignment does not occur. To solve the certificate problem, as we describe in Section 3.2, we work with analytically-convenient approximations for Θ and ζ: the exact definitions of these approximations Θ̂ and ζ̂ are given in Appendix A.5.2, and we prove their suitability as approximations in Theorem B.2 (a slightly more general version of Theorem 2) and Lemma D.11, respectively. As we have discussed in Section 3.1, our rates of concentration for Θ about Θ̂ are essentially optimal—the poor rates that end up appearing in Theorem B.1 are set by later parts of the argument.
With our approximation to Θ justified, we show that for any sufficiently small step size τ and number of iterations k, solving the certificate problem (2.3) guarantees appropriate decrease of the
7The results of Arora et al. (2019b) apply to data of norm no larger than 1, but it is straightforward to extend our results for spherical data to this setting, using the 1-homogeneity of Θ in each argument (as a kernel on the entire ambient space Rn0 × Rn0 ) to write Θ(x,x′) = ‖x‖2‖x′‖2Θ(x/‖x‖2,x′/‖x′‖2).
nominal generalization error; additional details are discussed in Section 3.2. The key property that we use in constructing certificates in Proposition B.4 (the ‘appendix version’ of Proposition 1) is that as the depth L increases, the kernel Θ̂ sharpens and localizes (Fig. 2a): the conditions on L in Theorem B.1 guarantee that the sharpness is sufficient to ensure that the cross-manifold integrals in the certificate problem are small in magnitude, which leads to rapid decrease of the nominal error. Our precise characterization of this phenomenon is presented in Appendix C.
To complete the proof, we will justify the nominal-to-finite transition in (A.3). Starting from the update equations (2.1) and (2.2), subtracting and rearranging gives an update equation for the difference:
ζNk − ζ∞k = (Id−τΘ) [ ζNk−1 − ζ∞k−1 ] − τΘNk−1 [ ζNk−1 ] + τΘ [ ζNk−1 ] .
In particular, if τ is chosen less than the operator norm of Θ, we can take norms on both sides of the previous equation, apply the triangle inequality, then exploit a telescoping series cancellation to obtain the difference bound∥∥ζ∞k − ζNk ∥∥L2 µ∞ ≤ τ k−1∑ s=0 ∥∥∥∥∫ M ΘNs ( · ,x′)ζNs (x′) dµN (x′)− ∫ M Θ( · ,x′)ζNs (x′) dµ∞(x′) ∥∥∥∥ L2 µ∞ . (A.4) There are two obstacles to controlling the norm terms on the RHS of (A.4): the kernels ΘNs are distinct from the kernel Θ due to changes in the weights that occur during training, and the empirical measure µN incurs a sampling error relative to the population measure µ∞. To address the first challenge, we measure the changes to the NTK during training in a worst-case fashion as
∆Nk = max i∈{0,1,...,k} ∥∥ΘNi −Θ∥∥L∞(M×M), and train in the NTK regime, where the network width n is larger than a large polynomial in the depth L and the total training time kτ is no larger than L/n. These conditions imply that with high probability ∆Nk is no larger than a constant multiple of n
1−δ poly(L, d0) for a small constant δ > 0, so that the amortized changes during training kτ∆Nk can be made small by sufficient overparameterization. We provide full details of this argument in Appendix F. By the preceding argument, we can use the triangle inequality and Jensen’s inequality to pass from the norm term in (A.4) to a difference-of-measures term which integrates against Θ, and by Theorem B.2, we can replace the integration against Θ by an integration against a smooth, deterministic kernel, which leads to a bound∥∥ζ∞k − ζNk ∥∥L2 µ∞ ≤ Rk(n,L, d0) + τ k−1∑ s=0 ∥∥∥∥∫ M ψ1(∠( · ,x′))ζNs (x′) ( dµN (x′)− dµ∞(x′) )∥∥∥∥ L2 µ∞ ,
where Rk is a residual term that we argue is small in the NTK regime with high probability, and for concision we write ψ1 to denote the function of ∠(x,x′) that appears in Theorem B.2. To control the remaining term, we make use of a basic result from optimal transport theory, which states that for any probability measure µ on the Borel sets of a metric space X and corresponding empirical measure µN , one has for every Lipschitz function f∫
X
f(x) ( dµ(x)− dµN (x) ) ≤ ‖f‖LipW ( µ, µN ) ,
whereW( · , · ) denotes the 1-Wasserstein metric, and concentration inequalities for empirical measures in the 1-Wasserstein metric (Weed & Bach, 2019). To apply this result to our setting, it is necessary to control the change throughout training of the Lipschitz constant of ζNk , and one must also account for the fact that the metric space in our setting isM, which has two distinct connected components. We treat the first issue using an inductive argument, and our treatment of the second issue (Lemma B.13) leads to the dependence on the degree of class imbalance demonstrated in the constant Cµ∞ in Theorem B.1.

heading: A.5 NOTATION

heading: A.5.1 GENERAL NOTATION

text:
If n ∈ N, we write [n] = {1, . . . , n}. We generally use bold notation x,A for vectors, matrices, and operators and non-bold notation for scalars and scalar-valued functions. For a vector x or a matrix
A, we will write entries as either xj or Aij , or (x)j or (A)ij ; we will occasionally index the rows or columns of A similarly as (A)i or (A)j , with the particular meaning made clear from context. We write [x]+ = max{x, 0} for the ReLU activation function; if x is a vector, we write [x]+ to denote the vector given by the application of [ · ]+ to each coordinate of x, and we will generally adopt this convention for applying scalar functions to vectors. If x,x′ ∈ Rn are nonzero, we write ∠(x,x′) = cos-1(〈x,x′〉/‖x‖2‖x′‖2) for the angle between x and x′. The vectors (ei) denote the canonical basis for Rn. We write 〈x,y〉 = ∑ i xiyi for the euclidean
inner product on Rn, and if 0 < p < +∞ we write ‖x‖p = ( ∑ i|xi|p)1/p for the `p norms (when p ≥ 1) on Rn. We also write ‖x‖0 = |{i ∈ [n] | xi 6= 0}| and ‖x‖∞ = maxi∈[n] |xi|. The unit ball in Rn is written Bn = {x ∈ Rn | ‖x‖2 ≤ 1}, and we denote its (topological) boundary, the unit sphere, as Sn−1. We reserve the notation ‖ · ‖ for the operator norm of a m × n matrix A, defined as ‖A‖ = sup‖x‖2≤1 ‖Ax‖2; more generally, we write ‖A‖`p→`q = sup‖x‖p≤1 ‖Ax‖q for the corresponding induced matrix norm. For m × n matrices A and B, we write 〈A,B〉 = tr(A∗B) for the standard inner product, where A∗ denotes the transpose of A, and ‖A‖F = √ 〈A,A〉 for the Frobenius norm ofA.
The Banach space of (equivalence classes of) real-valued measurable functions on a measure space (X,µ) satisfying ( ∫ X |f |p dµ)1/p < +∞ is written Lpµ(X) or simply Lp if the space and/or measure is clear from context; we write ‖ · ‖Lp for the associated norm, and 〈 · , · 〉L2 for the associated inner product when p = 2, with the adjoint operation denoted by ∗. For an operator T : Lpµ → Lqν , we write T [f ] to denote the image of f under T , T i to denote the operator that applies T i times, and ‖T ‖Lpµ→Lqν = sup‖f‖Lpµ≤1 ‖T [f ]‖L q ν . We use Id to denote the identity operator, i.e. Id[g] = g for every g ∈ Lpµ. We say that T is positive if 〈f, T [f ]〉L2 ≥ 0 for all f ∈ L2; for example, the identity operator is positive.
For an event E in a probability space, we write 1E to denote the indicator random variable that takes the value 1 if ω ∈ E and 0 otherwise. If σ > 0, by g ∼ N (0, σ2I) we mean that g ∈ Rn is distributed according to the standard i.i.d. gaussian law with variance σ2, i.e., it admits the density (2πσ2)−n/2 exp(−‖x‖22/(2σ2)) with respect to Lebesgue measure on Rn; we will occasionally write this equivalently as g ∼i.i.d. N (0, σ2). We use d = to denote the “identically-distributed” equivalence relation.
We use “numerical constant” and “absolute constant” interchangeably for numbers that are independent of all problem parameters. Throughout the text, unless specified otherwise we use c, c′, c′′, C,C ′, C ′′, K,K ′,K ′′, and so on to refer to numerical constants whose value may change from line to line within a proof. Numerical constants with numbered subscripts C1, C2, . . . and so on will have values fixed at the scope of the proof of a single result, unless otherwise specified. We generally use lower-case letters to refer to numerical constants whose value should be small, and upper case for those that should be large; we will generally use K, K ′ and so on to denote numerical constants involved in lower bounds on the size of parameters required for results to be valid. If f and g are two functions, the notation f . g means that there exists a numerical constant C > 0 such that f ≤ Cg; the notation f & g means that there exists a numerical constant C > 0 such that f ≥ Cg; and when both are true simultaneously we write f g. If f is a real-valued function with sufficient differentiability properties, we will write both f ′ and ḟ for the derivative of f , and when higher derivatives are available we will occasionally denote them by f (n), with this usage specifically made clear in context. For a metric space X and a Lipschitz function f : X → R, we write ‖f‖Lip to denote the minimal Lipschitz constant of f .

heading: A.5.2 SUMMARY OF OPERATOR AND ERROR DEFINITIONS

text:
We collect some of the important definitions that appear throughout the main text and the appendices in this section. We begin with the NTK-type operators that appear in our analysis. Recall from Appendix A.1 our definition for the backward features: we have
β`θ(x) = ( WL+1PIL(x)W LPIL−1(x) . . .W `+2PI`+1(x) )∗ for ` = 0, 1, . . . , L− 1, and where we additionally define
I`(x) = supp ( 1α`θ(x)>0 ) , PI`(x) = ∑ i∈I`(x) eie ∗ i
for the orthogonal projection onto the set of coordinates where the `-th activation at input x is positive. “The” neural tangent kernel is defined as
Θ(x,x′) = 〈 ∇̃fθ0(x), ∇̃fθ0(x′) 〉 = 〈 αL(x),αL(x′) 〉 +
L−1∑ `=0 〈 α`(x),α`(x′) 〉〈 β`(x),β`(x′) 〉 ,
with corresponding operator on L2µ∞(M)
Θ[g](x) = ∫ M Θ(x,x′)g(x′) dµ∞(x′).
As shown in Lemma B.8, this is not exactly the kernel that governs the dynamics of gradient descent: the relevant kernels in this context are defined as
ΘNk (x,x ′) = ∫ 1 0 〈 ∇̃fθNk (x ′), ∇̃fθNk −tτ∇̃LµN (θNk )(x) 〉 dt.
We define operators ΘNk on L 2 µN (M) corresponding to integration against these kernel in a manner analogous to the definition of Θ:
ΘNk [g](x) = ∫ M ΘNk (x,x ′)g(x′) dµN (x′).
We then move to the deterministic approximations for Θ that we develop: we define
ϕ(ν) = cos-1((1− ν/π) cos ν + (1/π) sin ν),
which governs the angle evolution process in the initial random network, as studied in Appendix E, and write ϕ(`) to denote `-fold composition of ϕ with itself. We define
ψ1(ν) = n
2 L−1∑ `=0 cos ( ϕ(`)(ν) ) L−1∏ `′=` ( 1− ϕ (`′)(ν) π ) ,
which is the “output” of our main result on concentration, Theorem B.2, and
ψ(ν) = n
2 L−1∑ `=0 L−1∏ `′=`
( 1− ϕ (`′)(ν)
π
) ,
which is at the core of the certificate construction problem. We think of ψ as an analytically-simpler version of ψ1, with an approximation guarantee given in Lemma C.11. Throughout these appendices, we will make use of basic properties of ψ1 and ψ that follow from properties of ϕ without explicit reference; the source material for these types of claims is Lemma E.5, which gives elementary properties of ϕ (for example, that it takes values in [0, π/2], which implies that ψ and ψ1 are no larger than nL/2). For derived estimates, we call the reader’s attention to the contents of Appendix C.2.2; we will make explicit reference to these results when we need them, however. Although we have mentioned approximations Θ̂ and Θ̂ in the main text, we will prefer in these appendices to explicitly reference ψ and ψ1 to avoid confusion; as an exception, we will use the Θ̂ notation in Appendix C as discussed there. Our approximation for the initial prediction error is
ζ̂(x) = −f?(x) + ∫ M fθ0(x ′) dµ∞(x′), (A.5)
where we recall fθ0 denotes the network function with the initial (random) weights. In particular, this approximates the network function with a constant, and the error as a piecewise constant function onM±. This approximation is justified in Lemma D.11.

heading: B PROOFS OF THE MAIN RESULTS

heading: B.1 MAIN RESULTS

text:
Theorem B.1. LetM be a one-dimensional Riemannian manifold satisfying our regularity assumptions. For any 0 < δ ≤ 1/e, choose L so that
L ≥ C1 max { Cµ∞ log 9(1/δ) log24 (Cµ∞n0 log(1/δ)) , κ 2Cλ } ,
let N ≥ L10, set n = C2L99 log9(1/δ) log18(Ln0), and fix τ > 0 such that C3 nL2 ≤ τ ≤ C4 nL .
Then if there exists a function g ∈ L2µ∞(M) such that
‖Θ[g]− ζ‖L2 µ∞ (M)
≤ C5 √ log(1/δ) log(nn0)
Lmin { ρqcertmin , ρ −qcert min } ; ‖g‖L2 µ∞ (M)
≤ C6 √ log(1/δ) log(nn0)
nρqcertmin ,
(B.1) with probability at least 1−δ over the random initialization of the network and the i.i.d. sample from µ∞, the parameters obtained at iteration bL39/44/(nτ)c of gradient descent on the finite sample loss LµN yield a classifier that separates the two manifolds. The constants C1, . . . , C4 > 0 depend only on the constants qcert, C5, C6 > 0, the constants κ, Cλ are respectively the extrinsic curvature constant and the global regularity constant defined in Section 2.1, and the constant Cµ∞ is defined as max{ρqmin, ρ −q min}(1 + ρmax) 6 (min {µ∞(M+), µ∞(M−)})−11/2, where q = 11 + 8qcert.
Proof. The proof is an application of Lemma B.7, with suitable instantiations of the parameters of that result; to avoid clashing with the probability parameter δ in this theorem, we use ε for the parameter δ appearing in Lemma B.7. Define Cρ = max{ρmin, ρ−1min}. We will pick q = 39/44 and ε = 5/47, so that the relevant hypotheses of Lemma B.7 become (after worst-casing in the bound on N somewhat for readability)
d ≥ K log(nn0CM)
n ≥ K ′max { L99d9 log9 L, κ2/5, ( κ
cλ )1/3} L ≥ K ′′max{C2qcertρ d, κ2Cλ}
N ≥ K ′′′C 133/18+(152/27)qcert ρ (1 + ρmax) 133/54
min {µ∞(M+), µ∞(M−)}19/18 d8/3L9 log3 L,
and the conclusion we will appeal to becomes
P [∥∥∥ζNbL39/44/(nτ)c∥∥∥ L∞(M) ≤ CC 1+2qcert/3 ρ (1 + ρmax) 1/2 min {µ∞(M+), µ∞(M−)}1/2 d3/4 log4/3 L L1/11 ] ≥ 1− C ′Le−cd nτ .
Under our choice of τ and enforcing
L ≥ (2C) 11C 11+22qcert/3 ρ (1 + ρmax) 11/2d33/4 log44/3 L
(min {µ∞(M+), µ∞(M−)})11/2 , (B.2)
we have the equivalent result P [∥∥∥ζNbL39/44/(nτ)c∥∥∥
L∞(M) ≤ 1 2
] ≥ 1− L3e−cd
≥ 1− e−c ′d,
where the last bound holds when d ≥ K logL, which is redundant with the hypotheses on n and d required to use Lemma B.7. Thus, when in addition d ≥ (1/c′) log(1/δ), we obtain
P [∥∥∥ζNbL39/44/(nτ)c∥∥∥
L∞(M) ≤ 1 2
] ≥ 1− δ. (B.3)
Therefore to conclude, we need only argue that our choices of n, N , L, d, and δ in the theorem statement suffice to satisfy the hypotheses of Lemma B.7. We have already satisfied the conditions on ε and q. We notice that (B.2) implies that it suffices to enforce simply N ≥ L10, and following Lemma C.4, we can bound CM as in (B.62) in the proof of Lemma B.7 by
CM ≤ 1 + len(M+) µ∞(M+) + len(M−) µ∞(M−) ≤ 21 + ρmax ρmin .
Because n ≥ L99 and L ≥ Cρ(1 + ρmax), we can eliminate CM from the lower bound on d while paying only an extra factor of 2 in the constant. In addition, because κ ≥ 1 andCλ ≥ max{1, 1/cλ},
we can remove the κ2/5 and ( κ cλ )1/3 lower bounds on n, since they are enforced through L already via the bound L ≥ K ′′κ2Cλ, worsening the absolute constant if needed. These simplifications lead us to the sufficient conditions (plus the certificate existence hypotheses)
d ≥ K max{log(1/δ), log(nn0)} n ≥ K ′L99d9 log9 L
L ≥ K ′′max
{ C 11+22qcert/3 ρ (1 + ρmax) 11/2d33/4 log44/3 L
(min {µ∞(M+), µ∞(M−)})11/2 , κ2Cλ } N ≥ L10.
We ignore the condition on N below, since it matches with the theorem statement. When δ ≤ 1/e, given that n0 ≥ 3 we have nn0 ≥ e and max{log(1/δ), log(nn0)} ≤ log(1/δ) log(nn0). For the sake of simplicity, we can also round up the fractional constants in the lower bound on L. We can eliminate d from these sufficient conditions by substituting the lower bound into the conditions on n and L, and this also implies that our conditions on certificate existence in the theorem statement suffice for the certificate existence hypothesis for Lemma B.7. Thus, we have the remaining sufficient conditions
n ≥ KL99 log9(1/δ) log9(nn0) log9 L
L ≥ K ′max
{ C11+8qcertρ (1 + ρmax) 6 log9(1/δ) log9(nn0) log 15 L
(min {µ∞(M+), µ∞(M−)})11/2 , κ2Cλ
} .
Using Lemma B.15 and choosing L larger than a sufficiently large absolute constant and larger than log(1/δ), we obtain that it suffices to enforce for n
n ≥ KL99 log9(1/δ) log18(Ln0). In the hypotheses of the theorem, we have chosen the equality n = KL99 log9(1/δ) log18(Ln0) in the last bound. This implies log(nn0) ≤ C log(Ln0), so it suffices to enforce the L lower bound
L ≥ K ′max
{ C11+8qcertρ (1 + ρmax) 6 log9(1/δ) log24(Ln0)
(min {µ∞(M+), µ∞(M−)})11/2 , κ2Cλ
} .
Defining, as in the theorem
Cµ∞ = C11+8qcertρ (1 + ρmax) 6
(min {µ∞(M+), µ∞(M−)})11/2 ,
and using Cµ∞ ≥ 1, we can worsen the absolute constant K ′ in order to apply Lemma B.15 once again, obtaining the simplified condition
L ≥ CK ′max { Cµ∞ log 9(1/δ) log24 (Cµ∞n0 log(1/δ)) , κ 2Cλ } .
These conditions reflect what is stated in the lemma.
Theorem B.2. Let M be a d0-dimensional Riemannian submanifold of Sn0−1. For any d ≥ Kd0 log(nn0CM), if n ≥ K ′d4L then one has on an event of probability at least 1− e−cd
sup (x,x′)∈M×M ∣∣∣∣∣Θ(x,x′)− n2 L−1∑ `=0 cos ( ϕ(`)(ν) ) L−1∏ `′=` ( 1− ϕ (`′)(ν) π )∣∣∣∣∣ ≤ √d4nL3, where we write ν = ∠(x,x′) in context with an abuse of notation, c,K,K ′ > 0 are absolute constants, and CM > 0 depends only on the number of connected components of M and their diameters and curvatures (Lemma C.4).
Proof. We have by the definition of Θ
Θ(x,x′) = 〈αL(x),αL(x′)〉+ L−1∑ `=0 〈α`(x),α`(x′)〉〈β`(x),β`(x′)〉. (B.4)
Under the stated hypotheses, Lemmas D.10 and D.13 give uniform control of each of the terms appearing in this expression with suitable probability to tolerate 2L+ 1 union bounds, which gives simultaneous uniform control of the factors on an event E with probability at least 1−e−cd. Starting from (B.4), we can write with the triangle inequality∣∣∣∣∣Θ(x,x′)− n2 L−1∑ `=0 cos ( ϕ(`)(ν) ) L−1∏ `′=` ( 1− ϕ (`′)(ν) π
)∣∣∣∣∣ ≤ ∣∣〈αL(x),αL(x′)〉∣∣ +
L−1∑ `=0 ∣∣∣∣∣〈α`(x),α`(x′)〉〈β`(x),β`(x′)〉 − n2 L−1∑ `=0 cos ( ϕ(`)(ν) ) L−1∏ `′=` ( 1− ϕ (`′)(ν) π )∣∣∣∣∣. (B.5)
By the triangle inequality, we have∣∣∣∣∣〈α`(x),α`(x′)〉〈β`(x),β`(x′)〉 − n2 cos(ϕ(`)(ν)) L−1∏ `′=` ( 1− ϕ (`′)(ν) π )∣∣∣∣∣ ≤ ∣∣〈α`(x),α`(x′)〉∣∣∣∣∣∣∣〈β`(x),β`(x′)〉 − n2 L−1∏ `′=` ( 1− ϕ (`′)(ν) π
)∣∣∣∣∣ +
∣∣∣∣∣n2 L−1∏ `′=` ( 1− ϕ (`′)(ν) π )∣∣∣∣∣∣∣∣〈α`(x),α`(x′)〉 − cos(ϕ(`)(ν))∣∣∣. Under the conditions on n, L, and d, we have on the event E that for each `
sup (x,x′)∈M ∣∣〈α`(x),α`(x′)〉∣∣ ≤ 2, so we can conclude that on E∣∣∣∣∣〈α`(x),α`(x′)〉〈β`(x),β`(x′)〉 − n2 cos(ϕ(`)(ν)) L−1∏ `′=` ( 1− ϕ (`′)(ν) π
)∣∣∣∣∣ ≤ 3√d4nL. The conditions on n, d, and L imply that this residual is larger than that incurred by the level-L features, which is no larger than 2. Returning to (B.5), we have shown that on E∣∣∣∣∣Θ(x,x′)− n2 L−1∑ `=0 cos ( ϕ(`)(ν) ) L−1∏ `′=` ( 1− ϕ (`′)(ν) π
)∣∣∣∣∣ ≤ C√d4nL3. After adjusting the other absolute constants to absorb C into d, this gives the claim.
Theorem B.3 (Pointwise Version of Theorem B.2). Let M be a d0-dimensional Riemannian submanifold of Sn0−1. For any d ≥ K log n, if n ≥ K ′max{1, d4L} then one has for any (x,x′) ∈M×M
P [∣∣∣∣∣Θ(x,x′)− n2 L−1∑ `=0 cos ( ϕ(`)(ν) ) L−1∏ `′=` ( 1− ϕ (`′)(ν) π )∣∣∣∣∣ ≤ √d4nL3 ] ≥ 1− e−cd,
where we write ν = ∠(x,x′) in context with an abuse of notation, and c,K,K ′ > 0 are absolute constants.
Proof. Follow the proof of Theorem B.2, but invoke the pointwise versions of the uniform concentration results used there (i.e., Lemmas D.1 and D.4) after rescaling d to relocate the log n terms.
Proposition B.4. Let M be an r-instance of the two circles geometry studied in Appendix C.1.1, with r ≥ 1/2. For any 0 < δ ≤ 1/e, if n ≥ KL5 log4(1/δ) log4(Ln0 log(1/δ)) and L ≥ K ′(1 − r2)−1/2, then there exist absolute constants C5, C6 > 0 and a function g such that (B.1) is satisfied with the choice qcert = 1/2 with probability at least 1− 3δ. The constants K,K ′ > 0 are absolute.
Proof. Given r ≥ 12 and L ≥ max{K, (π/2)(1−r 2)−1/2}, we have by Lemma C.1 that there exists g such that ∫ M ψ ◦ ∠( · ,x ′)g(x′) dµ∞(x′) = ζ̂, with
‖g‖L2 µ∞ ≤ (64/ √ π) ‖ζ̂‖L∞(M) nρ
1/2 min
. (B.6)
By this bound, the triangle inequality, the Minkowski inequality, and the fact that µ∞ is a probability measure, we have
‖Θ[g]− ζ‖L2 µ∞ ≤ ‖Θ− ψ ◦ ∠‖L∞(M×M)‖g‖L2 µ∞ + ‖ζ − ζ̂‖L2 µ∞
≤ C‖Θ− ψ ◦ ∠‖L∞(M×M) ‖ζ̂‖L∞(M) nρ
1/2 min
+ ‖ζ − ζ̂‖L∞(M). (B.7)
An application of Theorem B.2 and Lemma C.11 gives that on an event of probability at least 1− e−cd
‖Θ− ψ ◦ ∠‖L∞(M×M) ≤ Cn/L
if d ≥ Kd0 log(nn0CM) and n ≥ K ′d4L5. An application of Lemma D.11 gives
P [∥∥∥ζ̂ − ζ∥∥∥ L∞(M) ≤ √ 2d L ] ≥ 1− e−cd
and P [ ‖ζ‖L∞(M) ≤ √ d ] ≥ 1− e−cd
as long as n ≥ Kd4L5 and d ≥ K ′d0 log(nn0CM), where we use these conditions to simplify the residual that appears in Lemma D.11. In particular, combining the previous two bounds with the triangle inequality and a union bound and then rescaling d, which worsens the constant c and the absolute constants in the preceding conditions, gives
P [∥∥∥ζ̂∥∥∥
L∞(M) ≤ √ d
] ≥ 1− 2e−cd.
Combining these bounds using a union bound and substituting into (B.7), we get that under the preceding conditions, on an event of probability at least 1− 3e−cd we have
‖Θ[g]− ζ‖L2 µ∞ ≤ C √ d
L
( 1 + 1
ρ 1/2 min
)
≤ C √ d
L max{ρ1/2min, ρ −1/2 min },
where we worst-case the density constant in the second line, and in addition, on the same event, we have by (B.6)
‖g‖L2 µ∞ ≤ (64/
√ π)
√ d
nρ 1/2 min
.
To conclude, we simplify the preceding conditions on n and turn the parameter d into a parameter δ > 0 in order to obtain the claimed form of the result. We have in this setting d0 = 1, and also that CM is bounded by an absolute constant; since n0 ≥ 3, we can thus eliminate the parameter CM from our hypotheses by adding an extra absolute constant factor. Choosing d ≥ (1/c) log(1/δ), we obtain that the previous two bounds hold on an event of probability at least 1− 3δ. When δ ≤ 1/e, given that n0 ≥ 3 we have nn0 ≥ e and max{log(1/δ), log(nn0)} ≤ log(1/δ) log(nn0), so that it suffices to enforce the requirement d ≥ K log(1/δ) log(nn0) for a certain absolute constant K > 0. We can then substitute this lower bound on d into the two certificate bounds above to obtain the form claimed in (B.1) with qcert = 1/2. For the hypothesis on n, we substitute this lower bound on d into the condition on n to obtain the sufficient condition n ≥ K ′L5 log4(1/δ) log4(nn0). Using Lemma B.15 and possibly worsening absolute constants, we then get that it suffices to enforce n ≥ K ′L5 log4(1/δ) log4(Ln0 log(1/δ)), which is the hypothesis in the result.
Theorem B.5. There exist absolute constants c, C,K,K ′ > 0 such that for any d ≥ Kn0 log n, if n ≥ K ′d4L, then on an event of probability at least 1− e−cd the natural extension of fθ0 to Rn0 is 3 √ d-Lipschitz.
Proof. The proof is a simple application of Lemma B.17, which (because fθ0 is 1-nonnegatively homogeneous and so are all its intermediate feature maps α`θ0(x)) implies that it suffices to control the Lipschitz constants of the maps and bound them on the unit sphere, together with Lemmas D.11 and D.12. In particular, for any d ≥ Kn0 log(n) and any n ≥ K ′d4L, we have that there exists an event of probability at least 1− e−cd on which
‖fθ‖L∞(Sn0−1) ≤ √ d,
and ∥∥fθ∣∣Sn0−1∥∥Lip ≤ √d. Applying Lemma B.17, it follows that fθ0 : Rn0 → R is 3 √ d-Lipschitz on an event of probability at least 1− e−cd.

heading: B.2 SUPPORTING RESULTS ON DYNAMICS

text:
Lemma B.6 (Nominal). Suppose Cerr, Ccert, qcert > 0 are absolute constants. Then there exist absolute constants c, c′, C ′, C ′′, C ′′′ > 0 and absolute constants K,K ′,K ′′ > 0 such that for any d ≥ Kd0 log(nn0CM) and any 1/2 ≤ q ≤ 1, if n ≥ K ′d4L5, if L ≥ K ′′dC2qcertρ , and if additionally there exists g ∈ L2µ∞(M) satisfying
‖Θ[g]− ζ‖L2 µ∞ (M) ≤ CerrCqcertρ
√ d
L ; ‖g‖L2 µ∞ (M) ≤ Ccertρ−qcertmin
√ d
n
and τ > 0 is chosen such that
τ ≤ c ′
nL ,
then one has
P  ⋂ 0≤k≤Lq/(nτ) { ‖ζ∞k ‖L2 µ∞ (M) ≤ √ d } ≥ 1− e−cd,
and in addition
P  ⋂ C′ √ d/(nτρ
qcert min )≤k≤Lq/(nτ)
{ ‖ζ∞k ‖L2 µ∞ (M) ≤ C ′′Cqcertρ √ d logL
nkτ } ≥ 1− e−cd.

heading: Moreover, one has

text: P  ⋂ 0≤k≤Lq/(nτ) { k∑ s=0 ‖ζ∞s ‖L2 µ∞ (M) ≤ C2qcertρ C ′′′d log2 L nτ } ≥ 1− e−cd.

heading: The constant Cρ = max{ρmin, ρ−1min}.

text:
Proof. We will combine Lemma B.12 with various probabilistic results to obtain a simple final form for the bound from this result.
Invoking Lemma B.12, we can assert that for any step size τ > 0 satisfying
τ < 1
‖Θ‖L2 µ∞ (M)→L 2 µ∞ (M)
, (B.8)
and for any k satisfying
kτ ≥ √ 3e
2
‖g‖L2 µ∞ (M)
‖ζ‖L∞(M) , (B.9)
the population dynamics satisfy
‖ζ∞k ‖L2 µ∞ (M)
≤ √
3‖Θ[g]− ζ‖L2 µ∞ (M)
− 3‖g‖L2 µ∞ (M)
kτ log
(√ 3
2
‖g‖L2 µ∞ (M)
‖ζ‖L∞(M)kτ
) . (B.10)
We state the bounds we will apply to simplify this expression. An application of Lemma D.11 gives
P [∥∥∥ζ̂ − ζ∥∥∥ L∞(M) ≤ √ 2d L ] ≥ 1− e−cd (B.11)
and P [ ‖ζ‖L∞(M) ≤ √ d ] ≥ 1− e−cd (B.12)
as long as n ≥ Kd4L5 and d ≥ K ′d0 log(nn0CM), where we use these conditions to simplify the residual that appears in the version of (B.11) quoted in Lemma D.11. In particular, combining (B.11) and (B.12) with the triangle inequality and a union bound and then rescaling d, which worsens the constant c and the absolute constants in the preceding conditions, gives
P [∥∥∥ζ̂∥∥∥
L∞(M) ≤ √ d
] ≥ 1− 2e−cd. (B.13)
In addition, we can write using the triangle inequality ‖ζ‖L∞(M) ≥ ∥∥∥ζ̂∥∥∥ L∞(M) − ∥∥∥ζ − ζ̂∥∥∥ L∞(M) , and ∥∥∥ζ̂∥∥∥ L∞(M) = sup x∈M ∣∣∣∣f?(x)− ∫ M fθ0(x ′) dµ∞(x′)
∣∣∣∣ = max
{∣∣∣∣∫ M fθ0(x ′) dµ∞(x′)− 1 ∣∣∣∣, ∣∣∣∣∫ M fθ0(x ′) dµ∞(x′) + 1 ∣∣∣∣} ≥ 1,
so that, by (B.11), we have if L ≥ 2 √ d
P [ ‖ζ‖L∞(M) ≥ 1
2
] ≥ 1− e−cd. (B.14)
Because µ∞ is a probability measure, Jensen’s inequality, the Schwarz inequality, and the triangle inequality give
‖Θ‖L2 µ∞ (M)→L 2 µ∞ (M) ≤ sup (x,x′)∈M×M |Θ(x,x′)|
≤ sup(x,x′)∈M×M |Θ(x,x ′)− ψ1 ◦ ∠(x,x′)|
+ sup(x,x′)∈M×M |ψ1 ◦ ∠(x,x′)|, and an application of Theorem B.2 and Lemma E.5 then gives that on an event of probability at least 1− e−cd
‖Θ‖L2 µ∞ (M)→L 2 µ∞ (M) ≤ CnL (B.15)
provided d ≥ Kd0 log(nn0CM) and n ≥ K ′d4L. We will write E for the event consisting of the union of the events invoked for the bounds (B.11) to (B.15), which has probability at least 1− e−cd by a union bound and a choice of d ≥ K. We will conclude by simplifying (B.10) on E . First, we note that by (B.15), the step size condition (B.8) is satisfied on E provided
τ ≤ c nL , (B.16)
which holds under our hypotheses. Next, on E , we write using decreasingness of x 7→ − log x and (B.12)
− 3‖g‖L2 µ∞ (M)
kτ log
(√ 3
2
‖g‖L2 µ∞ (M)
‖ζ‖L∞(M)kτ
) ≤ − 3‖g‖L2 µ∞ (M)
kτ log
(√ 3
2
‖g‖L2 µ∞ (M)
kτ √ d
)
= − √ 6d
√ 3‖g‖L2
µ∞ (M)√ 2kτ √ d log
(√ 3
2
‖g‖L2 µ∞ (M)
kτ √ d
) .
(B.17)
By the hypothesis on g, we have on E
‖g‖L2 µ∞ (M)
≤ Cρ−qcertmin
√ d
n , (B.18)
and so it follows that on E √ 3
2
‖g‖L2 µ∞ (M)
kτ √ d
≤ C nkτρqcertmin .
The function x 7→ −x log x is a strictly increasing function on [0, e−1], so when k is chosen such that
Ce
nτρqcertmin ≤ k, (B.19)
we have on E by (B.17)
− 3‖g‖L2 µ∞ (M)
kτ log
(√ 3
2
‖g‖L2 µ∞ (M)
‖ζ‖L∞(M)kτ
) ≤ C √ 6d
nkτρqcertmin log ( C−1nkτρqcertmin ) . (B.20)
Additionally, in the context of the condition (B.9), notice that by (B.14) and (B.18), on E we have√ 3e
2
‖g‖L2 µ∞ (M)
τ‖ζ‖L∞(M) ≤ Ce
√ d
nτρqcertmin ,
so that given d ≥ 1, we have that the choice
k ≥ Ce √ d
nτρqcertmin (B.21)
implies both conditions (B.9) and (B.19). We can simplify (B.20) using the hypothesis kτ ≤ Lq/n with 1/2 ≤ q ≤ 1: we get
nkτρqcertmin C ≤ L qρqcertmin C ≤ L1+q,
where the last inequality requires L ≥ ρqcertmin /C, which implies
− 3‖g‖L2 µ∞ (M)
kτ log
(√ 3
2
‖g‖L2 µ∞ (M)
‖ζ‖L∞(M)kτ
) ≤ C ′ √ d logL
nkτρqcertmin . (B.22)
The conditions we need to satisfy on kτ can be stated together as
Ce √ d nρqcertmin ≤ kτ ≤ Lq/n,
and it is possible to satisfy these conditions simultaneously as long as
L ≥
( Ce √ d
ρqcertmin
)1/q .
We obtain an upper bound C 2e2d
ρ 2qcert min for the quantity on the RHS of this inequality from q ≥ 1/2; it suffices to choose L larger than this upper bound instead. The other simplifications are easier: using the assumption on the norm of Θ[g]− ζ, we have
‖Θ[g]− ζ‖L2 µ∞ (M)
≤ Cqcertρ C √ d
Lρ 1/2 min
.
Worst-casing terms using our hypotheses on d and L to obtain a simplified bound, on E , we have thus shown that when (B.21) is satisfied, we have
‖ζ∞k ‖L2 µ∞ (M)
≤ CCqcertρ √ d
( 1
L +
logL
nkτ
) .
We have 1
L ≤ logL nkτ ⇐⇒ L logL n ≥ kτ,
which is implied by the hypothesis kτ ≤ Lq/n as long as L ≥ e. So we can simplify to
‖ζ∞k ‖L2 µ∞ (M)
≤ CCqcertρ
√ d logL
nkτ .
We also need a bound that works for k that do not satisfy (B.21). From the update equation for the dynamics in the proof of Lemma B.12 and the choice of τ , we also have
‖ζ∞k ‖L2 µ∞ (M) ≤ ‖ζ‖L2 µ∞ (M) ≤
√ d,
where the last bound is valid on E . Finally, we can obtain the claimed sum bound by calculating using our ‘small-k’ and ‘large-k’ bounds:
k∑ s=0 ‖ζ∞s ‖L2 µ∞ (M) = bC √ d/(nτρ qcert min )c∑ s=0 ‖ζ∞s ‖L2 µ∞ (M) + k∑ s=dC √ d/(nτρ
qcert min )e
‖ζ∞s ‖L2 µ∞ (M)
≤ √ d ( 1 + C ′ √ d
nτρqcertmin
) + C ′′Cqcertρ √ d logL
nτ
k∑ s=dC √ d/(nτρ
qcert min )e
1
s
≤ C ′d nτρqcertmin + C ′′Cqcertρ
√ d logL
nτ
( nτρqcertmin C √ d + ∫ k C √ d/(nτρ
qcert min )
ds
s
)
≤ Cd nτρqcertmin
+ C ′max { ρ2qcertmin , 1 } logL+ C ′′Cqcertρ √ d log2 L
nτ ,
where the second inequality uses standard estimates for the harmonic numbers and C ′ √ d/(nτρqcertmin ) ≥ 1, which follows from τ ≤ c′/(nL), d ≥ 1 and L ≥ Kρ qcert min for a suitable absolute constant K; and the third inequality integrates and simplifies, using kτ ≤ L/n and again d ≥ 1 and L ≥ Cρqcertmin . Worst-casing constants and using nτ ≤ 1, we simplify this last bound to
k∑ s=0 ‖ζ∞s ‖L2 µ∞ (M) ≤ max { ρ2qcertmin , 1 ρ2qcertmin } Cd log2 L nτ .
To see that the conditions on L in the statement of the result suffice, note that we have to satisfy (say) L ≥ Kρqcertmin and L ≥ K ′ρ −qcert min ; the first of these lower bounds is tighter when ρmin ≥ 1, and the second when ρmin < 1, and so it suffices to require L ≥ Kρ2qcertmin and L ≥ K ′ρ −2qcert min instead.
Lemma B.7 (Nominal to Finite). Let d0 = 1, and suppose Cerr, Ccert, qcert > 0 are absolute constants. Then there exist absolute constants c, c′, C ′, C ′′, C ′′′ > 0 and absolute constants K,K ′,K ′′,K ′′′ > 0 such that for any d ≥ Kd0 log(nn0CM), any 1/2 ≤ q < 1 and any 0 < δ ≤ 1, if L ≥ K ′max{C2qcertρ d, κ2Cλ}, if
n ≥ K ′′max { e252/δL60+44qd9 log9 L, κ2/5, ( κ
cλ
)1/3} ,
and if
N1/(2+δ) ≥ K ′′′C 7/2+8qcert/3 ρ (1 + ρmax) 7/6e119/(3δ) min { µ∞(M+)1/2, µ∞(M−)1/2 } d5/4L5/2+2q logL, and if additionally there exists g ∈ L2µ∞(M) satisfying
‖Θ[g]− ζ‖L2 µ∞ (M) ≤ CerrCqcertρ
√ d
L ; ‖g‖L2 µ∞ (M) ≤ Ccertρ−qcertmin
√ d
n
and τ > 0 is chosen such that
τ ≤ c ′
nL ,
then one has generalization in L2µ∞(M):
P [∥∥∥ζNbLq/(nτ)c∥∥∥ L2 µ∞ (M) ≤ C ′Cqcertρ √ d logL Lq ] ≥ 1− C ′′′Le−cd nτ ,
and in addition, one has generalization in L∞(M):
P [∥∥∥ζNbLq/(nτ)c∥∥∥ L∞(M) ≤ C ′′C 1+2qcert/3 ρ (1 + ρmax) 1/2e14/(3δ) min {µ∞(M+), µ∞(M−)}1/2 d3/4 log4/3 L L(4q−3)/6 ] ≥ 1− C ′′′Le−cd nτ .

heading: The constant Cρ = max{ρmin, ρ−1min}.

text:
Proof. The proof controls the L∞ norm of the error evaluated along the finite sample dynamics using an interpolation inequality for Lipschitz functions on an interval (Lemma B.14), which relates the L∞ norm to a certain combination of the predictor’s Lipschitz constant and its L2µ∞ norm. We can control these two quantities at time zero using our measure concentration results; to control them for larger times 0 < k ≤ Lq/(nτ), we set up a system of coupled ‘discrete integral equations’ for the generalization error of the finite sample predictor and the Lipschitz constant of the finite sample predictor, and use the fact that kτ is not large to argue by induction that not much blow-up can occur. Along the way, we control the generalization error of the finite sample predictor by linking it to the generalization error of the nominal predictor as controlled in Lemma B.6; the residual that arises is shown to be small by applying Corollary B.11 and applying basic results from optimal transport theory adapted to our setting, encapsulated in Lemmas B.13 and B.16.
To begin, we will lay out the probabilistic bounds we will rely on for simplifications, so that the rest of the proof can proceed without interruption. We will want to satisfy
τ < 1
max {∥∥ΘµN∥∥L2 µN (M)→L2 µN (M), ‖Θµ∞‖L2µ∞ (M)→L2µ∞ (M) } , (B.23) following the notation of Lemma B.10. Using Jensen’s inequality, the Schwarz inequality, and the triangle inequality, we have for ? ∈ {N,∞}
‖Θµ?‖L2 µ? (M)→L2 µ? (M) = sup ‖g‖
L2 µ?
(M)≤1
∥∥∥∥∫ M Θ(x,x′)g(x′) dµ?(x′) ∥∥∥∥ L2 µ? (M)
≤ ‖g‖L1 µ? (M) sup (x,x′)∈M×M
|Θ(x,x′)|
≤ sup(x,x′)∈M×M |Θ(x,x ′)− ψ1 ◦ ∠(x,x′)|
+ sup(x,x′)∈M×M |ψ1 ◦ ∠(x,x′)|, (B.24)
where the notation ψ1 follows the definition in Appendix C.2.2. The first term in (B.24) can be controlled using Theorem B.2: we obtain that on an event of probability at least 1− e−cd
‖Θ− ψ1 ◦ ∠‖L∞(M×M) ≤ √ d4nL3 (B.25)
if d ≥ Kd0 log(nn0CM) and n ≥ K ′d4L. The second term in (B.24) can be controlled using the triangle inequality, Lemma E.5, and the definition of ψ1: we obtain that it is no larger than nL/2. Combining these two bounds, we have on an event of probability at least 1− e−cd
max {∥∥ΘµN∥∥L2 µN (M)→L2 µN (M), ‖Θµ∞‖L2µ∞ (M)→L2µ∞ (M) } ≤ CnL (B.26)
provided d ≥ Kd0 log(nn0CM) and n ≥ K ′d4L. Thus, with probability at least 1 − e−cd, our choice of step size τ ≤ c/(nL) satisfies (B.23). Under our hypotheses on the function g in the statement of the result and taking a union bound with the event in (B.26), we can invoke Lemma B.6 to obtain
P  ⋂ C √ d/(nτρ
qcert min )≤k≤Lq/(nτ)
{ ‖ζ∞k ‖L2 µ∞ (M) ≤ C ′Cqcertρ √ d logL
nkτ
} ≥ 1− C ′′′Le−cd nτ (B.27)
and
P  ⋂ 0≤k≤Lq/(nτ) { k∑ s=0 ‖ζ∞s ‖L2 µ∞ (M) ≤ C2qcertρ C ′′d log2 L nτ } ≥ 1− C ′′′Le−cd nτ
(B.28)
provided d ≥ Kd0 log(nn0CM), 1/2 ≤ q < 1, n ≥ K ′d4L5, and L ≥ K ′′C2qcertρ d. We have by Lemmas B.6 and B.10, a union bound with (B.26), and our condition on τ that
P  ⋂ 0≤k≤Lq/(nτ) { ‖ζ∞k ‖L2 µ∞ (M) ≤ √ d } ∩ ⋂ 0≤k≤Lq/(nτ) {∥∥ζNk ∥∥L2 µN (M) ≤ √ d } ≥ 1− CLe−cd nτ (B.29) as long as d ≥ Kd0 log(nn0CM) and n ≥ K ′L48+20qd9 log9 L, and where we used our conditions on τ and q to obtain that Lq/nτ ≥ 1 and simplify the probability bound; and, following the notation of Corollary B.11, we have by this result (again under our condition on τ and a union bound) that there is an event of probability at least 1− CLe−cd/(nτ) on which
∆NbLq/(nτ)c−1 ≤ ( n11L48+8qd9 log9 L )1/12 (B.30)
under the previous conditions on n and d. In addition, applying Lemma D.12 and a union bound gives that on an event of probability at least 1− Ce−cd
max {∥∥∥ζ∣∣M+∥∥∥Lip,∥∥∥ζ∣∣M−∥∥∥Lip } ≤ √ d (B.31)
provided d ≥ Kd0 log(nn0CM) and n ≥ K ′max{d4L, (κ/cλ)1/3, κ2/5}. Finally, we have by Lemma B.13 that for any 0 < δ ≤ 1
P  ⋂ f∈Lip(M)  ∣∣∫ M f(x) dµ ∞(x)− ∫ M f(x) dµ N (x) ∣∣ ≤ 2‖f‖L∞(M) √ d N + e14/δCµ∞,M √ dmax?∈{+,−} ∥∥∥∥f∣∣∣M? ∥∥∥∥ Lip N1/(2+δ)   ≥ 1− 8e−d, (B.32) as long as d ≥ 1 and N ≥ 2 √ d/min{µ∞(M+), µ∞(M−)}. We let E(q, δ) denote the event consisting of the union of the events appearing in the bounds (B.25) to (B.32) hold; by a union bound and the previous observation that Lq/nτ ≥ 1, we have
P[E ] ≥ 1− C ′Le−cd
nτ .
In the sequel, we will use the events defining E to simplify our residuals without explicitly referencing that our bounds hold only on E to save time. We start from the dynamics update equations given by Lemma B.8, which we use to write
ζ∞k − ζNk = (Id−τΘ) [ ζ∞k−1 − ζNk−1 ] + τΘNk−1 [ ζNk−1 ] − τΘ [ ζNk−1 ] ,
where Θ is defined as in Lemma B.12. Under the choice of τ and positivity of Θ (Lemma B.9), we apply the triangle inequality and a telescoping series with the common initial conditions to obtain∥∥ζ∞k − ζNk ∥∥L2 µ∞ (M) ≤ τ k−1∑ s=0 ∥∥ΘNs [ζNs ]−Θ [ζNs ]∥∥L2 µ∞ (M) . (B.33)
We can write ΘNs [ ζNs ] (x) = ∫ M ΘNs (x,x ′)ζNs (x ′) dµN (x′)
=
∫ M ( ΘNs (x,x ′)− ψ1 ◦ ∠(x,x′) ) ζNs (x
′) dµN (x′) + ∫ M ψ1 ◦ ∠(x,x ′)ζNs (x ′) dµN (x′),
and analogously Θ [ ζNs ] (x) = ∫ M (Θ(x,x′)− ψ1 ◦ ∠(x,x′)) ζNs (x′) dµ∞(x′)
+ ∫ M ψ1 ◦ ∠(x,x′)ζNs (x′) dµ∞(x′).
Using Jensen’s inequality and the Schwarz inequality, we have∥∥∥∥∫ M ( ΘNs (x,x ′)− ψ1 ◦ ∠(x,x′) ) ζNs (x ′) dµN (x′) ∥∥∥∥ L2 µ∞ (M)
≤ ∫ M ∥∥ΘNs ( · ,x′)− ψ1 ◦ ∠( · ,x′)∥∥L2 µ∞ (M) |ζNs (x′)|dµN (x′)
≤ ∥∥ΘNs − ψ1 ◦ ∠∥∥L∞(M×M)‖ζNs ‖L1µN (M)
≤ ∥∥ΘNs − ψ1 ◦ ∠∥∥L∞(M×M)‖ζNs ‖L2µN (M),
since µN is a probability measure. Repeating an analogous calculation with µ∞ for the other term and applying the triangle inequality, we have
∥∥ΘNs [ζNs ]−Θ [ζNs ]∥∥L2 µ∞ (M) ≤ ‖Θ− ψ1 ◦ ∠‖L∞(M×M)  ∥∥ζ∞s − ζNs ∥∥L2 µ∞ (M) +‖ζ∞s ‖L2 µ∞ (M)
+ ∥∥ζNs ∥∥L2
µN (M)  + ∥∥ΘNs −Θ∥∥L∞(M×M)∥∥ζNs ∥∥L2
µN (M)
+ ∥∥∥∥∫ M ψ1 ◦ ∠( · ,x′)ζNs (x′) ( dµ∞(x′)− dµN (x′) )∥∥∥∥ L2 µ∞ (M) .
(B.34)
We detour briefly to simplify residuals appearing in (B.34) before using the result to update (B.33). Using (B.25) and (B.30), we get
‖Θ− ψ1 ◦ ∠‖L∞(M×M) (∥∥ζ∞s − ζNs ∥∥L2 µ∞ (M) + ‖ζ∞s ‖L2 µ∞ (M) + ∥∥ζNs ∥∥L2 µN (M) ) + ∥∥ΘNs −Θ∥∥L∞(M×M)∥∥ζNs ∥∥L2
µN (M)
≤
√ d4nL3 (∥∥ζ∞s − ζNs ∥∥L2 µ∞ (M) + ‖ζ∞s ‖L2 µ∞ (M) + ∥∥ζNs ∥∥L2 µN (M) ) + ( n11L48+8qd9 log9 L
)1/12 ∥∥ζNs ∥∥L2 µN (M)
≤ ( n11L48+8qd9 log9 L )1/12(∥∥ζ∞s − ζNs ∥∥L2 µ∞ (M) + ‖ζ∞s ‖L2 µ∞ (M) + 2 ∥∥ζNs ∥∥L2 µN (M) ) .
(B.35)
where the final bound holds when n ≥ d3. Using (B.29), we can further simplify the RHS of the last bound above to(
n11L48+8qd9 log9 L )1/12(∥∥ζ∞s − ζNs ∥∥L2 µ∞ (M) + ‖ζ∞s ‖L2 µ∞ (M) + 2 ∥∥ζNs ∥∥L2 µN (M) ) ≤ 2 ( n11L48+8qd15 log9 L )1/12 + ( n11L48+8qd9 log9 L )1/12 ∥∥ζ∞s − ζNs ∥∥L2 µ∞ (M) .
With this last bound and (B.34), we can use kτ ≤ Lq/n to simplify (B.33) to
∥∥ζ∞k − ζNk ∥∥L2 µ∞ (M) ≤ C ( L48+20d15 log9 L n )1/12 +τ ( n11L48+8qd9 log9 L )1/12∑k−1 s=0 ∥∥ζ∞s − ζNs ∥∥L2 µ∞ (M)
+ τ k−1∑ s=0 ∥∥∥∥∫ M ψ1 ◦ ∠( · ,x′)ζNs (x′) ( dµ∞(x′)− dµN (x′) )∥∥∥∥ L2 µ∞ (M) .
(B.36)
To control the remaining term in (B.36), we split the error ζNs into a Lipschitz component whose evolution is governed by the nominal kernel ψ1 ◦ ∠ and a nonsmooth component which is small in
L∞. Formally, we define Θnom : L2µN (M)→ L 2 µN (M) by
Θnom [g] (x) = ∫ M ψ1 ◦ ∠(x,x′)g(x′) dµN (x′),
and use the update equation from Lemma B.8 to write
ζNs = ζ − τ s−1∑ i=0 ΘNi [ ζNi ]
= ζ − τ s−1∑ i=0 Θnom [ ζNi ]
︸ ︷︷ ︸ ζN,Lips
+ τ s−1∑ i=0 ( Θnom −ΘNi ) [ ζNi ]
︸ ︷︷ ︸ δNs
,
so that ζNs = ζ N,Lip s + δ N s , and ζ N,Lip 0 = ζ, δ N 0 = 0. It is straightforward to control δ N s in L ∞: we have (as usual) by the triangle inequality, Jensen’s inequality, and the Schwarz inequality
∥∥δNs ∥∥L∞(M) ≤ τ s−1∑ i=0 ∫ M ∥∥ψ1 ◦ ∠( · ,x′)−ΘNi ( · ,x′)∥∥L∞(M)∣∣ζNi (x′)∣∣dµN (x′) ≤ τ
s−1∑ i=0 ∥∥ψ1 ◦ ∠−ΘNi ∥∥L∞(M×M)∥∥ζNi ∥∥L2 µN (M),
and then the triangle inequality together with (B.25), (B.29) and (B.30) yield∥∥δNs ∥∥L∞(M) ≤ sτ√d(√d4nL3 + (n11L48+8qd9 log9 L)1/12) ≤ sτ √ d ( n11L48+8qd9 log9 L )1/12 , (B.37)
where the second line applies the same simplifications that led us to (B.35). The triangle inequality gives ∥∥∥∥∫
M ψ1 ◦ ∠( · ,x′)δNs (x′)
( dµ∞(x′)− dµN (x′) )∥∥∥∥ L2 µ∞ (M)
≤ ∑
?∈{N,∞}
∥∥∥∥∫ M ψ1 ◦ ∠( · ,x′)δNs (x′) dµ?(x′) ∥∥∥∥ L2 µ∞ (M) ,
and simplifying as usual using Jensen’s inequality and the Hölder inequality, we obtain∥∥∥∥∫ M ψ1 ◦ ∠( · ,x′)δNs (x′) ( dµ∞(x′)− dµN (x′) )∥∥∥∥ L2 µ∞ (M) ≤ nL‖δNs ‖L∞(M)
≤ sτ ( n23L60+8qd15 log9 L )1/12 ,
where the last bound uses (B.37). Then using the triangle inequality and kτ ≤ Lq/n to simplify in (B.36), we obtain ∥∥ζ∞k − ζNk ∥∥L2 µ∞ (M) ≤ C ( L60+32qd15 log9 L n
)1/12 + τ ( n11L48+8qd9 log9 L
)1/12 k−1∑ s=0 ∥∥ζ∞s − ζNs ∥∥L2 µ∞ (M)
+ τ k−1∑ s=0 ∥∥∥∥∫ M ψ1 ◦ ∠( · ,x′)ζN,Lips (x′) ( dµ∞(x′)− dµN (x′) )∥∥∥∥ L2 µ∞ (M) .
(B.38)
To simplify the remaining term in (B.38), we aim to apply (B.32); to do this we will need to justify the notation and establish that ζN,Lips ∈ Lip(M) regardless of the random sample from µ∞ and the
random instance of the weights. Because ζN,Lips is a sum of functions, we can bound its minimal Lipschitz constant by the sum of bounds on the Lipschitz constants of each summand. We always have for either ? ∈ {+,−}∥∥∥ζN,Lips ∣∣M?∥∥∥Lip ≤ ∥∥∥ζ∣∣M?∥∥∥Lip + τ s−1∑ i=0 ∥∥∥∥∫ M ψ1 ◦ ∠( · ,x′)ζNi (x′) dµN (x′) ∥∥∥∥ Lip . (B.39)
We note that because the ReLU [ · ]+ is 1-Lipschitz as a map on Rn, we have∥∥∥ζ∣∣M?∥∥∥Lip ≤ ∥∥WL+1∥∥2 L∏ `=1
∥∥W `∥∥ < +∞, so we need only develop a Lipschitz property for the summands in the second term of (B.39). To do this, we will start by showing that t 7→ ψ1 ◦ cos-1〈γ?(t),x′〉 is absolutely continuous for each x′. Continuity is immediate. The only obstruction to differentiability comes from the inverse cosine, which fails to be differentiable at ±1, and because M ⊂ Sn0−1 we have 〈γ?(t),x′〉 = ±1 only if γ?(t) = ±x′; because γ? are simple curves, this shows that there are at most two points of nondifferentiability in [0, len(M?)]. At points of differentiability, we calculate using the chain rule the derivative
t 7→ − ( ψ′1 ◦ cos-1〈γ?(t),x′〉 )〈 γ′?(t)√ 1− 〈γ?(t),x′〉2 ,x′ 〉 ,
and because γ? is a sphere curve, it holds (I − γ?(t)γ∗?(t))γ′?(t) = γ′?(t) for all t, whence by Cauchy-Schwarz∣∣∣∣∣ 〈 γ′?(t)√ 1− 〈γ?(t),x′〉2 ,x′ 〉∣∣∣∣∣ = ∣∣∣∣∣ 〈 (I − γ?(t)γ?(t)∗)x′√ 1− 〈γ?(t),x′〉2 ,γ′?(t)
〉∣∣∣∣∣ ≤ ‖(I − γ?(t)γ?(t)∗)x′‖2√
1− 〈γ?(t),x′〉2 ≤ 1, (B.40)
where we also used that γ? are unit-speed curves. In particular, the derivative is bounded, hence integrable on [0, len(M?)], and so an application of (Cohn, 2013, Theorem 6.3.11) establishes that t 7→ ψ1 ◦ cos-1〈γ?(t),x′〉 is absolutely continuous, with the expansion∣∣ψ1 ◦ cos-1〈γ?(t),x′〉 − ψ1 ◦ cos-1〈γ?(t′),x′〉∣∣
= ∣∣∣∣∣ ∫ t′ t ( ψ′1 ◦ cos-1〈γ?(t′′),x′〉 )〈 γ′?(t′′)√ 1− 〈γ?(t′′),x′〉2 ,x′ 〉 dt′′ ∣∣∣∣∣, which gives an avenue to establish Lipschitz estimates for t 7→ ψ1 ◦ cos-1〈γ?(t),x′〉. Because x′ 7→ ζNi (x′) is continuous and i ≤ s ≤ k ≤ Lq/(nτ) < +∞, an application of Fubini’s theorem enables us to also use this result to obtain Lipschitz estimates for the summands examined in (B.39), to wit ∥∥∥∥∫
M ψ1 ◦ ∠( · ,x′)ζNi (x′) dµN (x′) ∥∥∥∥ Lip
≤ sup x∈M? ∫ M |ψ′1 ◦ ∠(x,x′)| ∣∣ζNi (x′)∣∣ dµN (x′) ≤ ‖ζNi ‖L2
µN (M) sup
x∈M? (∫ M (ψ′1 ◦ ∠(x,x′)) 2 dµN (x′) )1/2 (B.41)
after using the bound (B.40) in the first inequality and the Schwarz inequality for the second. Before proceeding with further simplifications, we note that theC2 property of ψ1, continuity of ζNi , boundedness of i, and compactness of M let us assert using (B.41) and (B.39) that ζN,Lips ∈ Lip(M) whether or not we are working on the event E . Continuing, we develop a bound for the RHS of (B.41) that is valid on E . Using the triangle inequality and the Minkowski inequality, we have for
the second term on the RHS of the last bound in (B.41)
sup x∈M? (∫ M (ψ′1 ◦ ∠(x,x′)) 2 dµN (x′) )1/2 ≤ sup x∈M? (∣∣∣∣∫ M (ψ′1 ◦ ∠(x,x′)) 2 ( dµN (x′)− dµ∞(x′) )∣∣∣∣)1/2
+ sup x∈M? (∫ M (ψ′1 ◦ ∠(x,x′)) 2 dµ∞(x′) )1/2 .
(B.42)
For the first term in (B.42), we use Lemmas C.7, C.22 and C.24 to obtain that x′ 7→ (ψ′1◦∠(x,x′))2 is bounded by Cn2L4 and C ′n2L5-Lipschitz for every x, and then applying (B.32) gives
sup x∈M? (∣∣∣∣∫ M (ψ′1 ◦ ∠(x,x′)) 2 ( dµN (x′)− dµ∞(x′) )∣∣∣∣)1/2
≤
( Cn2L4 √ d
N + e14/δCµ∞,MC
′n2L5 √ d
N1/(2+δ)
)1/2
≤ C (1 + Cµ ∞,M)
1/2e7/δ
N1/(4+2δ) nL5/2d1/4. (B.43)
For the second term in (B.42), we apply Lemmas C.8 and C.22 together with the choiceL ≥ Kκ2Cλ to get
sup x∈M? (∫ M (ψ′1 ◦ ∠(x,x′)) 2 dµ∞(x′) )1/2 ≤ CnL2 sup x∈M± (∫ M
dµ∞(x′)
(1 + (L/π)∠(x,x′))2 )1/2 ≤ CnL3/2ρ1/2max(len(M+) + len(M−))1/2
≤ Cρ1/2maxC 1/2 µ∞,MnL 3/2. (B.44)
Combining (B.43) and (B.44) to control the RHS of (B.42), we obtain from (B.41)∥∥∥∥∫ M ψ1 ◦ ∠( · ,x′)ζNi (x′) dµN (x′) ∥∥∥∥ Lip
≤ C ∥∥ζNi ∥∥L2
µN (M)
( (1 + Cµ∞,M) 1/2e7/δ
N1/(4+2δ) nL5/2d1/4 + ρmaxCµ∞,MnL 3/2 ) ≤ C
∥∥ζNi ∥∥L2 µN (M) (1 + Cµ∞,M) 1/2 e7/δ(1 + ρmax) 1/2d1/4nL3/2, (B.45)
where in the second line we used N ≥ L4+2δ . Plugging (B.45) into (B.39) and applying in addition (B.31), we get∥∥∥ζN,Lips ∣∣M?∥∥∥Lip ≤ √d+ Cτe7/δ (1 + Cµ∞,M)1/2 (1 + ρmax)1/2d1/4nL3/2 s−1∑ i=0 ∥∥ζNi ∥∥L2 µN (M).
(B.46)
Let us briefly pause to reorient ourselves. We do not have control of the empirical losses appearing in (B.46) by an outside result, so we need to make some further simplifications to this bound. We will control the sum of empirical losses term in (B.46) by linking it to the difference population error, which we last saw in (B.38), and the population error using the triangle inequality and a change of measure inequality. Meanwhile, with the Lipschitz property of ζN,Lips we have shown, we will be able to obtain a bound in terms of simpler quantities for the last term on the RHS of (B.38) using (B.32). The two resulting bounds will give us a system of two coupled ‘discrete integral equations’ for the difference population error and the Lipschitz constants of ζN,Lips , which we will solve inductively.
First, we continue simplifying (B.46). The triangle inequality and the fact that µN is a probability measure give ∥∥ζNi ∥∥L2
µN (M) ≤ ∥∥∥ζN,Lipi ∥∥∥ L2 µN (M) + ∥∥δNi ∥∥L∞(M), (B.47)
and we have by the triangle inequality and Hölder- 12 continuity of x 7→ √ x∥∥∥ζN,Lipi ∥∥∥
L2 µN
(M) ≤ ∥∥∥ζN,Lipi ∥∥∥
L2 µ∞ (M)
+ ∣∣∣∣∣∥∥∥ζN,Lipi ∥∥∥L2 µN (M) − ∥∥∥ζN,Lipi ∥∥∥ L2 µ∞ (M) ∣∣∣∣∣ ≤ ∥∥∥ζN,Lipi ∥∥∥
L2 µ∞ (M)
+ √∣∣∣∣∫ M ( ζN,Lipi (x) )2 (dµ∞(x)− dµN (x)) ∣∣∣∣. (B.48) We have shown that ζN,Lipi ∈ Lip(M) and ζ N,Lip i ∈ L∞(M) above, and so ( ζN,Lipi )2 ∈ Lip(M)
as well, with ∥∥∥∥∥(ζN,Lipi )2 ∣∣∣∣ M? ∥∥∥∥∥ Lip ≤ 2 ∥∥∥ζN,Lipi ∥∥∥ L∞(M) ∥∥∥∥ζN,Lipi ∣∣∣M? ∥∥∥∥ Lip . Applying the previous equation with (B.32) to control (B.48), we get∥∥∥ζN,Lipi ∥∥∥ L2 µN (M)
≤
∥∥∥ζN,Lipi ∥∥∥ L2 µ∞ (M)
+Cd1/4
√ ‖ζN,Lipi ‖2L∞(M)
N + e14/δCµ∞,M‖ζN,Lipi ‖L∞(M) max?∈{+,−}
∥∥∥∥ζN,Lipi ∣∣∣M? ∥∥∥∥
Lip
N1/(2+δ)
≤
∥∥∥ζN,Lipi ∥∥∥ L2 µ∞ (M)
+Cd1/4 ‖ζN,Lipi ‖L∞(M)√ N + e7/δC 1/2 µ∞,M‖ζN,Lipi ‖ 1/2 L∞(M) max?∈{+,−} ∥∥∥∥ζN,Lipi ∣∣∣M? ∥∥∥∥1/2 Lip N1/(4+2δ)  , where the second line applies the Minkowski inequality. Using the triangle inequality and that µ∞ is a probability measure, we have∥∥∥ζN,Lipi ∥∥∥
L2 µ∞ (M)
≤ ∥∥ζNi ∥∥L2 µ∞ (M) + ∥∥δNi ∥∥L2 µ∞ (M)
≤ ‖ζ∞i ‖L2 µ∞ (M) + ∥∥ζNi − ζ∞i ∥∥L2 µ∞ (M) + ∥∥δNi ∥∥L∞(M). (B.49)
Substituting (B.49) into (B.47) and using (B.37) to simplify gives∥∥ζNi ∥∥L2 µN (M)
≤ ∥∥ζNi − ζ∞i ∥∥L2 µ∞ (M) + ‖ζ∞i ‖L2 µ∞ (M) + 2iτ √ d ( n11L48+8qd9 log9 L )1/12
+ Cd1/4  ∥∥∥ζN,Lipi ∥∥∥ L∞(M)√ N + e7/δC 1/2 µ∞,M ∥∥∥ζN,Lipi ∥∥∥1/2 L∞(M) max?∈{+,−} ∥∥∥∥ζN,Lipi ∣∣∣M? ∥∥∥∥1/2 Lip N1/(4+2δ)  . (B.50)
Following (B.46), we need to sum the previous bound over i. To simplify residuals, we use (B.28) to get
Cs2τ √ d ( n11L48+8qd9 log9 L )1/12 + s−1∑ i=0 ‖ζ∞i ‖L2 µ∞ (M)
≤ Cs2τ √ d ( n11L48+8qd9 log9 L )1/12 + C2qcertρ C ′d log2 L
nτ
≤ 2C2qcertρ C ′d log2 L
nτ ,
where the second bound uses the control sτ ≤ kτ ≤ Lq/n and holds under the condition n ≥ (C/C ′)12L48+32qd3. Summing in (B.50) and using the previous bound, it follows
s−1∑ i=0 ∥∥ζNi ∥∥L2 µN (M)
≤ CC2qcertρ d log 2 L
nτ + s−1∑ i=0 ∥∥ζNi − ζ∞i ∥∥L2 µ∞ (M)
+ Cd1/4  ∥∥∥ζN,Lipi ∥∥∥ L∞(M)√ N + e7/δC 1/2 µ∞,M ∥∥∥ζN,Lipi ∥∥∥1/2 L∞(M) max?∈{+,−} ∥∥∥∥ζN,Lipi ∣∣∣M? ∥∥∥∥1/2 Lip N1/(4+2δ)  . (B.51)
Plugging (B.51) into (B.46), we obtain∥∥∥ζN,Lips ∣∣M?∥∥∥Lip ≤ C1d1/4L3/2  d log 2 L+ nτ ∑s−1 i=0 ∥∥ζNi − ζ∞i ∥∥L2 µ∞ (M)
+nτd1/4 ∑s−1 i=0 ‖ζN,Lipi ‖L∞(M)√ N + ‖ζN,Lipi ‖1/2L∞(M) max?∈{+,−}
∥∥∥∥ζN,Lipi ∣∣∣M? ∥∥∥∥1/2
Lip
N1/(4+2δ)
 , (B.52)
where for concision we have defined
C1(δ, µ ∞) = CC2qcertρ e 14/δ (1 + Cµ∞,M) (1 + ρmax) 1/2 (B.53)
and simplified the √ d residual in (B.46) by worst-casing with the larger residual from the population error term in (B.51), and made other simplifications by worst-casing some constants. We simplify (B.38) next: we have shown that ζN,Lips ∈ Lip(M) and ζN,Lips ∈ L∞(M) above, and so for every x ∈M, we have
ψ1 ◦ ∠(x, · )ζN,Lips ∈ Lip(M)
as well, with∥∥∥ψ1 ◦ ∠(x, · )ζN,Lips ∣∣M?∥∥∥Lip ≤ CnL max?′∈{+,−}∥∥∥ζN,Lips ∣∣M?′∥∥∥Lip + C ′nL2∥∥ζN,Lips ∥∥L∞(M) (B.54) using the definition of ψ1, Lemmas E.5, C.7 and C.22, and∥∥ψ1 ◦ ∠(x, · )ζN,Lips ∥∥L∞(M) ≤ CnL∥∥ζN,Lips ∥∥L∞(M). (B.55) The bounds (B.54) and (B.55) retain no x dependence. Applying (B.32) and integrating over x, we obtain from (B.54) and (B.55)∥∥∥∥∫
M ψ1 ◦ ∠( · ,x′)ζN,Lips (x′)
( dµ∞(x′)− dµN (x′) )∥∥∥∥ L2 µ∞ (M)
≤ CnL √ d ∥∥ζN,Lips ∥∥L∞(M) N
+ CnLe14/δCµ∞,M
√ dmax?∈{+,−} ∥∥∥ζN,Lips ∣∣M?∥∥∥Lip N1/(2+δ)
+ CnL2e14/δCµ∞,M
√ d ∥∥ζN,Lips ∥∥L∞(M)
N1/(2+δ) ,
and we can combine the first and third terms on the RHS of the previous bound by worst-casing, giving∥∥∥∥∫
M ψ1 ◦ ∠( · ,x′)ζN,Lips (x′)
( dµ∞(x′)− dµN (x′) )∥∥∥∥ L2 µ∞ (M)
≤ C √ dnLe14/δ (1 + Cµ∞,M)
N1/(2+δ)
( max
?∈{+,−} ∥∥∥ζN,Lips ∣∣M?∥∥∥Lip + L∥∥ζN,Lips ∥∥L∞(M) ) .
Plugging the previous bound into (B.38), we obtain
∥∥ζ∞k − ζNk ∥∥L2 µ∞ (M)
≤ C ( L60+32qd15 log9 L
n
)1/12 + τ ( n11L48+8qd9 log9 L )1/12 k−1∑ s=0 ∥∥ζ∞s − ζNs ∥∥L2 µ∞ (M)
+ Cτ √ dnLe14/δ (1 + Cµ∞,M)
N1/(2+δ)
k−1∑ s=0 ( max ?∈{+,−} ∥∥∥ζN,Lips ∣∣M?∥∥∥Lip + L∥∥ζN,Lips ∥∥L∞(M) ) .
(B.56)
To finish coupling (B.52) and (B.56), we need to remove the L∞(M) terms. We accomplish this using Lemma B.14, which gives
∥∥ζN,Lips ∥∥L∞(M) ≤ CC1/22 ∥∥ζN,Lips ∥∥L2 µ∞ (M) + C
ρ 1/3 min
∥∥ζN,Lips ∥∥2/3L2 µ∞ (M) max ?∈{+,−} ∥∥∥ζN,Lips ∣∣M?∥∥∥1/3Lip , (B.57)
where we have defined
C2(µ ∞) = ρmax ρmin min {µ∞(M+), µ∞(M−)} . (B.58)
For coupling purposes, it will suffice to use a version of (B.57) obtained by simplifying with some coarse estimates. Using (B.49), (B.37) and (B.29), we have∥∥∥ζN,Lipi ∥∥∥
L2 µ∞ (M)
≤ √ d+ iτ √ d ( n11L48+8qd9 log9 L )1/12 + ∥∥ζNi − ζ∞i ∥∥L2
µ∞ (M)
≤ 2 √ d+ ∥∥ζNi − ζ∞i ∥∥L2 µ∞ (M) ,
using iτ ≤ Lq/n and n ≥ L48+20qd9 log9 L in the second line, and plugging this into (B.57) and using the Minkowski inequality gives
∥∥ζN,Lips ∥∥L∞(M) ≤ CC1/22 √d+ CC1/22 ∥∥ζNi − ζ∞i ∥∥L2 µ∞ (M) + Cd1/3
ρ 1/3 min
max ?∈{+,−} ∥∥∥ζN,Lips ∣∣M?∥∥∥1/3Lip + C
ρ 1/3 min
∥∥ζNi − ζ∞i ∥∥2/3L2 µ∞ (M) max ?∈{+,−} ∥∥∥ζN,Lips ∣∣M?∥∥∥1/3Lip . (B.59)
To make some of the subsequent bounds more concise, we introduce additional notation
Λs = max ?∈{+,−} ∥∥∥ζN,Lips ∣∣M?∥∥∥Lip.
Plugging (B.59) into (B.52) and using the Minkowski inequality, we obtain
Λs ≤ CC1d1/4L3/2 ( d log2 L+ C 1/2 2 d
3/4nsτ√ N + nτ
( 1 + C 1/2 2 d 1/4
√ N ) s−1∑ i=0 ∥∥ζNi − ζ∞i ∥∥L2 µ∞ (M)
+ nτd7/12
ρ 1/3 min
√ N s−1∑ i=0 Λ 1/3 i + C 1/4 2 nτd 1/2 N1/(4+2δ) s−1∑ i=0 Λ 1/2 i
+ nτd5/12
ρ 1/6 minN 1/(4+2δ) s−1∑ i=0 Λ 2/3 i + nτd1/4 ρ 1/3 min √ N s−1∑ i=0 ∥∥ζNi − ζ∞i ∥∥2/3L2 µ∞ (M) Λ 1/3 i
+ C
1/4 2 nτd 1/4
N1/(4+2δ) s−1∑ i=0 ∥∥ζNi − ζ∞i ∥∥1/2L2 µ∞ (M) Λ 1/2 i
+ nτd1/4
ρ 1/6 minN 1/(4+2δ) s−1∑ i=0 ∥∥ζNi − ζ∞i ∥∥1/3L2 µ∞ (M) Λ 2/3 i
) .
(B.60)
To simplify (B.60), we use sτ ≤ Lq/n, C2 ≥ 1, and q ≤ 1, and so if additionally we choose N ≥ C2 max{ √ d, L2} we obtain
Λs ≤ CC1d1/4L3/2 ( d log2 L+ nτ
s−1∑ i=0 ∥∥ζNi − ζ∞i ∥∥L2 µ∞ (M)
+ nτd1/4
ρ 1/6 minN 1/(4+2δ) s−1∑ i=0 ∥∥ζNi − ζ∞i ∥∥1/3L2 µ∞ (M) Λ 2/3 i + nτd7/12 ρ 1/3 min √ N s−1∑ i=0 Λ 1/3 i
+ C
1/4 2 nτd 1/2
N1/(4+2δ) s−1∑ i=0 Λ 1/2 i +
nτd5/12
ρ 1/6 minN 1/(4+2δ) s−1∑ i=0 Λ 2/3 i
+ nτd1/4
ρ 1/3 min
√ N s−1∑ i=0 ∥∥ζNi − ζ∞i ∥∥2/3L2 µ∞ (M) Λ 1/3 i
+ C
1/4 2 nτd 1/4
N1/(4+2δ) s−1∑ i=0 ∥∥ζNi − ζ∞i ∥∥1/2L2 µ∞ (M) Λ 1/2 i
) .
(B.61)
Meanwhile, we recall
Cµ∞,M = len(M+) µ∞(M+) + len(M−) µ∞(M−) ,
and an integration in coordinates gives
µ∞(M±) = ∫ len(M±)
0
ρ± ◦ γ±(t) dt ≥ ρmin len(M±),
so that
Cµ∞,M ≤ 2
ρmin . (B.62)
Using (B.62) and plugging (B.59) into (B.56), we obtain∥∥ζ∞k − ζNk ∥∥L2 µ∞ (M)
≤ C ( L60+32qd15 log9 L
n )1/12 + τ (( n11L48+8qd9 log9 L )1/12 + C ′C 1/2 2 √ dnL2e14/δ
ρminN1/(2+δ) ) k−1∑ s=0 ∥∥ζ∞s − ζNs ∥∥L2 µ∞ (M)
+ C ′τ √ dnLe14/δ
ρminN1/(2+δ) k−1∑ s=0
( Λs + LC 1/2 2 √ d
+Ld1/3ρ −1/3 min Λ 1/3 s + Lρ −1/3 min ∥∥ζ∞s − ζNs ∥∥2/3L2 µ∞ (M) Λ 1/3 s
) .
(B.63)
In (B.61) and (B.63), we now have a suitable system of coupled discrete integral equations for ‖ζ∞k − ζNk ‖L2µ∞ (M) and Λk. We will solve these equations by positing bounds for each parameter that are valid for all indices 0 ≤ k ≤ bLq/(nτ)c based on inspection of (B.61) and (B.63), then proving the bounds hold by induction on k. Positing the bounds is not too hard, because each term in (B.61) and (B.63) with a factor of N in its denominator can be forced to be small by requiring N to be large enough. For all 0 ≤ k ≤ bLq/(nτ)c, we claim
∥∥ζ∞k − ζNk ∥∥L2 µ∞ (M) ≤ Cdiff max  ClipC1C 1/2 2 C 4/3 ρ d7/4L5/2+q log2 L N1/(2+δ) ,( L60+32qd15 log9 L
n
)1/12  (B.64)
Λk ≤ ClipC1d5/4L3/2 log2 L, (B.65) where Cdiff and Clip are two absolute constants that we will specify in our arguments below. We prove (B.64) and (B.65) by induction on k. The case of k = 0 is immediate, since ζ∞0 = ζ N 0 for (B.64); and by construction ζN,Lip0 = ζ, and (B.31) and d ≥ 1 then gives (B.65) if L ≥ e. We therefore move to the induction step, assuming that (B.64) and (B.65) hold for k − 1 and showing that this implies the bounds for k. We begin by verifying (B.64). Applying the induction hypothesis for k − 1 via (B.65), we can write
Λs + LC 1/2 2
√ d+ L ( dΛs ρmin )1/3 ≤ ClipC1d5/4L3/2 log2 L+ LC1/22 √ d+ ( ClipC1 ρmin )1/3 d3/4L3/2 log2/3 L
≤ ClipC1C1/22 C1/3ρ d5/4L3/2 log 2 L,
where we worst-cased in the second line using Clip ≥ 1 and C1 ≥ 1, C2 ≥ 1, which follow from (B.53) and (B.58). We use kτ ≤ Lq/n with the last bound to note that
C ′τ √ dnLe14/δ
ρminN1/(2+δ) k−1∑ s=0 ClipC1C 1/2 2 C 1/3 ρ d 5/4L3/2 log2 L ≤ C ′′ClipC1C1/22 C4/3ρ d7/4L5/2+q log2 L N1/(2+δ) ,
where C ′′ ≥ 1. Using this bound and (B.65) once more, we can simplify (B.63) to∥∥ζ∞k − ζNk ∥∥L2 µ∞ (M)
≤ C ′′ClipC1C1/22 C4/3ρ d7/4L5/2+q log2 L
N1/(2+δ) + C
( L60+32qd15 log9 L
n )1/12 + τ (( n11L48+8qd9 log9 L )1/12 + C ′C 1/2 2 √ dnL2e14/δ
ρminN1/(2+δ) ) k−1∑ s=0 ∥∥ζ∞s − ζNs ∥∥L2 µ∞ (M)
+ C ′C 1/3 lip C 1/3 1 e 14/δ
ρ 4/3 min
τd11/12nL5/2 log2/3 L
N1/(2+δ)
k−1∑ s=0 ∥∥ζ∞s − ζNs ∥∥2/3L2 µ∞ (M) .
(B.66)
Noticing that the RHS of the bound (B.64) does not depend on k, let us momentarily denote it by CdiffM (i.e., the part of the RHS of this bound that does not involve Cdiff is denoted asM ). Plugging into (B.66) and using kτ ≤ Lq/n, we obtain
∥∥ζ∞k − ζNk ∥∥L2 µ∞ (M) ≤ C ′′ClipC1C1/22 C4/3ρ d7/4L5/2+q log2 L N1/(2+δ) + C
( L60+32qd15 log9 L
n )1/12 + Cdiff (( L48+20qd9 log9 L
n
)1/12 + C ′C 1/2 2 √ dL2+qe14/δ
ρminN1/(2+δ)
) M
+ C ′C 2/3 diff C 1/3 lip C 1/3 1 e 14/δ
ρ 4/3 min
d11/12L5/2+q log2/3 L
N1/(2+δ) M2/3.
In particular, if Cdiff = 6 max{C,C ′′} (for the constants in the first line of the previous bound), we can bound the RHS of the previous bound and obtain∥∥ζ∞k − ζNk ∥∥L2 µ∞ (M) ≤ CdiffM 3 + Cdiff (( L48+20qd9 log9 L n )1/12 + C ′C 1/2 2 √ dL2+qe14/δ
ρminN1/(2+δ)
) M
+ C ′C 2/3 diff C 1/3 lip C 1/3 1 e 14/δ
ρ 4/3 min
d11/12L5/2+q log2/3 L
N1/(2+δ) M2/3.
(B.67)
We can conclude (B.64) from (B.67) provided we can show the second and third terms are no larger than CdiffM/3. For the second term in (B.67), if we choose N such that
N1/(2+δ) ≥ 6C ′C1/22 ρ −1 mine 14/δd1/2L2+q
and n such that n ≥ 612L48+20qd9 log9 L
then we have
Cdiff
(( L48+20qd9 log9 L
n
)1/12 + C ′C 1/2 2 √ dL2+qe14/δ
ρminN1/(2+δ)
) M ≤ CdiffM
3 .
For the third term in (B.67), we proceed in cases: first, when
ClipC1C 1/2 2 C 4/3 ρ
d7/4L5/2+q log2 L N1/(2+δ) ≤ ( L60+32qd15 log9 L n )1/12 , (B.68)
we have by (B.64)
M =
( L60+32qd15 log9 L
n
)1/12 ,
and if we require additionally Cdiff ≥ 1, it follows that
C ′C 2/3 diff C 1/3 lip C 1/3 1 e 14/δ
ρ 4/3 min
d11/12L5/2+q log2/3 L
N1/(2+δ) M2/3
≤ C ′CdiffClipC1C1/22 C4/3ρ e14/δ d7/4L5/2+q log2 L
N1/(2+δ) M2/3
≤ C ′Cdiffe14/δM1+2/3, using C1 ≥ 1, C2 ≥ 1, and Clip ≥ 1 and worst-casing exponents on d and logL in the first line, and (B.68) in the second line. In particular, by the value of M in this regime, if
n ≥ (3C ′e14/δ)18L60+32qd15 log9 L then we obtain for the third term in (B.67)
C ′C 2/3 diff C 1/3 lip C 1/3 1 e 14/δ
ρ 4/3 min
d11/12L5/2+q log2/3 L
N1/(2+δ) M2/3 ≤ CdiffM 3 ,
as desired. Next, we consider the remaining case
ClipC1C 1/2 2 C 4/3 ρ
d7/4L5/2+q log2 L N1/(2+δ) ≥ ( L60+32qd15 log9 L n )1/12 , (B.69)
which by (B.64) implies
M = ClipC1C 1/2 2 C 4/3 ρ
d7/4L5/2+q log2 L
N1/(2+δ) .
With this setting of M , the third term in (B.67) can be bounded as
C ′C 2/3 diff C 1/3 lip C 1/3 1 e 14/δ
ρ 4/3 min
d11/12L5/2+q log2/3 L
N1/(2+δ) M2/3
= C ′C 2/3 diff ClipC1C 1/3 2 C 4/3+8/9 ρ e
14/δ d 7/4+1/3L5/2+qL5/3+2q/3 log2 L
N1/(2+δ)+2/(6+3δ)
≤ C ′CdiffC8/9ρ e14/δ d1/3L5/3+2q/3
N2/(6+3δ) M,
and using the RHS of the final bound in the previous expression, we see that if we choose
N1/(2+δ) ≥ (3C ′)3/2C4/3ρ e21/δd1/2L5/2+q,
then we have for the case (B.69)
C ′C 2/3 diff C 1/3 lip C 1/3 1 e 14/δ
ρ 4/3 min
d11/12L5/2+q log2/3 L
N1/(2+δ) M2/3 ≤ CdiffM 3 .
Combining the bounds on the third term in (B.67) over both cases (B.68) and (B.69), we have shown∥∥ζ∞k − ζNk ∥∥L2 µ∞ (M) ≤ CdiffM,
which proves (B.64). Next, to verify (B.65), we proceed with a similar idea: the bound claimed in (B.65) corresponds to a constant multiple of the first term in parentheses in (B.61), so to establish (B.65) it suffices to show that each of the other terms in (B.61) is no larger than a certain constant. To work with the maximum operation in (B.64), we will again split the analysis into two cases. First, we consider the case where (B.69) holds, so that the maximum in (B.64) is achieved by the second argument. Plugging (B.64) and (B.65) into (B.61) and using kτ ≤ Lq/n, we get
Λk ≤ CC1d1/4L3/2 ( d log2 L+ CdiffClipC1C 1/2 2 C 4/3 ρ d7/4L5/2+2q log2 L
N1/(2+δ)
+ C 1/3 diff ClipC 11/18 ρ C1C 1/6 2
d5/3L11/6+4q/3 log2 L
N6/(10+5δ)
+ C 1/3 lip C 1/3 1 C 1/3 ρ dL1/2+q log2/3 L√ N + C 1/2 lip C 1/2 1 C 1/4 2 d9/8L3/4+q logL N1/(4+2δ)
+ C 2/3 lip C 2/3 1 C 1/6 ρ
d5/4L1+q log4/3 L
N1/(4+2δ)
+ C 2/3 diff ClipC1C 1/3 2 C 5/3 ρ
d11/6L13/6+5q/3 log2 L
N (5+δ)/(4+2δ)
+ C 1/2 diff ClipC1C 1/2 2 C 2/3 ρ
d7/4L2+3q/2 log2 L
N1/(2+δ)
) .
Using Clip ≥ 1, C1 ≥ 1, Cρ ≥ 1, and C2 ≥ 1, we can worst-case constants in the previous expression to simplify. We can then do some selective worst-casing of the exponents on d, N , and L in all except the first term: we have evidently (to combine the first and last terms)
d7/4L2+3q/2 log2 L
N1/(2+δ) ≤ d
7/4L5/2+2q log2 L
N1/(2+δ)
and (to combine the first and second terms)
d5/3L11/6+4q/3 log2 L
N6/(10+5δ) ≤ d
7/4L5/2+2q log2 L
N1/(2+δ) ,
and because 0 < δ ≤ 1, we have 1/(2 + δ) ≤ 1/2 and (5 + δ)/(4 + 2δ) ≥ 1, and if N ≥ d1/12 this implies (to combine the first and second-to-last terms)
d11/6L13/6+5q/3 log2 L
N (5+δ)/(4+2δ) ≤ d
7/4L5/2+2q log2 L
N1/(2+δ) .
We can worst-case the remaining three terms, and we thus obtain Λk ≤ CC1d1/4L3/2 ( d log2 L+ 4CdiffClipC1C 1/2 2 C 5/3 ρ d1+3/4L5/2+2q log2 L N1/(2+δ)
+3C 2/3 lip C 2/3 1 C 1/4 2 C 1/3 ρ d1+1/4L1+q log4/3 L N1/(4+2δ)
) .
We can then pick Clip = 3C, and if
N1/(4+2δ) ≥ 3(3C)2/3C2/31 C 1/4 2 C 1/3 ρ d 1/4L1+q,
and N1/(2+δ) ≥ 12CCdiffC1C1/22 C5/3ρ d3/4L5/2+2q,
then it follows from the previous bound
Λk ≤ 3CC1d5/4L3/2 log2 L,
which establishes (B.65) in the first case, where (B.69) holds. Next, we consider the remaining case where (B.68) holds, so that the maximum in (B.64) is saturated by the first argument. We start by grouping some terms in (B.61) so that it will be slightly easier to simplify later: we can write
Λk ≤ CC1d1/4L3/2 ( d log2 L+ nτ
k−1∑ s=0 ∥∥ζNs − ζ∞s ∥∥L2 µ∞ (M) +
nτd1/4
ρ 1/6 minN 1/(4+2δ) k−1∑ s=0 (∥∥ζNs − ζ∞s ∥∥1/3L2 µ∞ (M) + d1/6 ) Λ2/3s
+ nτd1/4
ρ 1/3 min
√ N k−1∑ s=0 (∥∥ζNs − ζ∞s ∥∥2/3L2 µ∞ (M) + d1/3 ) Λ1/3s
+ C
1/4 2 nτd 1/4
N1/(4+2δ) k−1∑ s=0 (∥∥ζNs − ζ∞s ∥∥1/2L2 µ∞ (M) + d1/4 ) Λ1/2s ) .
(B.70)
By the case-defining condition (B.68) and (B.64), enforcing
n ≥ C12diffL60+32qd9 log 9 L
implies ∥∥ζNs − ζ∞s ∥∥L2 µ∞ (M) + d1/2 ≤ 2d1/2,
and we can use this to simplify (B.70), obtaining Λk ≤ CC1d1/4L3/2 ( d log2 L+ nτ
k−1∑ s=0 ∥∥ζNs − ζ∞s ∥∥L2 µ∞ (M) + 2nτd5/12 ρ 1/6 minN 1/(4+2δ) k−1∑ s=0 Λ2/3s
+ 2nτd7/12
ρ 1/3 min
√ N k−1∑ s=0 Λ1/3s + 2C 1/4 2 nτd 1/2 N1/(4+2δ) k−1∑ s=0 Λ1/2s
) .
(B.71)
Plugging (B.64) and (B.65) into (B.71) and using kτ ≤ Lq/n and (B.68), we get the bound Λk ≤ CC1d1/4L3/2 ( d log2 L+ Cdiff ( L60+44qd15 log9 L
n
)1/12 +
2C 2/3 lip C 2/3 1 C 1/6 ρ
d1+1/4L1+q log2/3 L
N1/(4+2δ)
+ 2C 1/3 lip C 1/3 1 C 1/3 ρ dL1/2+q log2/3 L√ N
+ 2C 1/2 lip C 1/2 1 C 1/4 2
d1+1/8L3/4+q logL
N1/(4+2δ)
) .
(B.72)
From (B.72), we see that if we choose n such that n ≥ (2Cdiff)12L60+44qd3
and we choose N such that N1/(2+δ) ≥ 16C4/3LipC 4/3 1 C 1/2 2 C 1/3 ρ d 1/2L2+2q
then (B.72) implies the bound Λk ≤ 3CC1d5/4L3/2 log2 L, which agrees with the previous choice Clip = 3C and thus proves (B.65) in the remaining case of (B.71). By induction, then, we have proved that (B.64) and (B.65) hold for each index 0 ≤ k ≤ bLq/(nτ)c. We can now wrap up the proof: we will obtain the desired conclusion by plugging the results we have developed into (B.57) and simplifying. Plugging (B.37), (B.27) and (B.64) into (B.49) and bounding the maximum by the sum, we get∥∥∥ζN,LipbLq/(nτ)c∥∥∥
L2 µ∞ (M) ≤ ∥∥∥ζ∞bLq/(nτ)c∥∥∥
L2 µ∞ (M)
+ ∥∥∥ζNbLq/(nτ)c − ζ∞bLq/(nτ)c∥∥∥
L2 µ∞ (M)
+ ∥∥∥δNbLq/(nτ)c∥∥∥
L∞(M)
≤ CCqcertρ
√ d logL nτbLq/(nτ)c + C ′ ( L60+32qd15 log9 L n )1/12 +C ′′C1C 1/2 2 C 4/3 ρ d7/4L5/2+q log2 L N1/(2+δ)
≤ CCqcertρ
√ d logL
Lq + C ′
( L60+32qd15 log9 L
n
)1/12 + C ′′C1C 1/2 2 C 4/3 ρ d7/4L5/2+q log2 L
N1/(2+δ)
≤ CCqcertρ
√ d logL
Lq , (B.73)
where in the third inequality we apply bLq/(nτ)c ≥ Lq/(2nτ), which follows from our choice of step size, and in the fourth inequality we simplify residuals using n ≥ (C ′/C)12d9L60+44q and N1/(2+δ) ≥ C ′′C1C1/22 C 4/3 ρ d5/4L5/2+2q logL. Applying (B.73), the triangle inequality (with (B.37) and the fact that µ∞ is a probability measure) and our previous choice of large n, we get∥∥∥ζNbLq/(nτ)c∥∥∥ L2 µ∞ (M) ≤ CCqcertρ √ d logL Lq , (B.74) i.e. generalization in L2µ∞(M). We can bootstrap generalization in L∞(M) from (B.73) using the triangle inequality and (B.57): we get∥∥∥ζNbLq/(nτ)c∥∥∥
L∞(M) ≤ CC1/22 ∥∥∥ζN,LipbLq/(nτ)c∥∥∥
L2 µ∞ (M)
+ C
ρ 1/3 min
∥∥∥ζN,LipbLq/(nτ)c∥∥∥2/3 L2 µ∞ (M) Λ 1/3 bLq/(nτ)c + ∥∥∥δNbLq/(nτ)c∥∥∥ L∞(M)
≤ CC
1/2 2 C qcert ρ
√ d logL
Lq + C ′C 1/3 1
ρ 1/3 min
d3/4L(3−4q)/6 log4/3 L min { ρ
1/3 min, 1
} ,
where in the second line we apply (B.37) and our previous choice of large n to absorb the residual from δNbLq/(nτ)c, and apply (B.65) to bound the Λ 1/3 bLq/(nτ)c term. Worst-casing the errors in the previous bound, we obtain∥∥∥ζNbLq/(nτ)c∥∥∥ L∞(M) ≤ C ( Cqcertρ C 1/2 2 + C 1/3 1 C 2/3 ρ ) d3/4 log4/3 L L(4q−3)/6 .
To conclude, we will tally dependencies and make some simplifications to show the conditions stated in the result suffice. Recalling (B.53) and (B.58) and using (B.62), we have
C1 ≤ CC2qcert+1ρ (1 + ρmax) 1/2 e14/δ,
so we can simplify to
C1/2ρ C 1/2 2 + C 1/3 1 C 2/3 ρ ≤ Cρ
( ρmax
min {µ∞(M+), µ∞(M−)} )1/2 + CC1+2qcert/3ρ (1 + ρmax) 1/6e14/(3δ)
≤ CC 1+2qcert/3 ρ (1 + ρmax) 1/2e14/(3δ)
min {µ∞(M+), µ∞(M−)}1/2 .
We can use this to obtain a simplified generalization inL∞(M) bound from our previous expression: it becomes∥∥∥ζNbLq/(nτ)c∥∥∥
L∞(M) ≤ CC
1+2qcert/3 ρ (1 + ρmax) 1/2e14/(3δ) min {µ∞(M+), µ∞(M−)}1/2 d3/4 log4/3 L L(4q−3)/6 , (B.75)
which can be made nonvacuous when q > 3/4. Tallying dependencies, we find after worst-casing (and using q ≥ 1/2 and some interdependences between parameters to simplify) that it suffices to choose N such that
N1/(2+δ) ≥ CC4/31 C 1/2 2 C 5/3 ρ e 21/δd5/4L5/2+2q logL,
the depth L such that L ≥ C max{C2qcertρ d, κ2Cλ},
the width n such that
n ≥ C max { e252/δL60+44qd9 log9 L, κ2/5, ( κ
cλ
)1/3} ,
and d such that d ≥ Cd0 log(nn0CM). Unpacking the constants in the condition on N , we see that it suffices to choose N such that
N1/(2+δ) ≥ CC 7/2+8qcert/3 ρ (1 + ρmax) 7/6e119/(3δ)
min {µ∞(M+), µ∞(M−)}1/2 d5/4L5/2+2q logL.

heading: B.3 AUXILIARY RESULTS

text:
Lemma B.8. Defining a kernel
ΘNk (x,x ′) = ∫ 1 0 〈 ∇̃fθNk (x ′), ∇̃fθNk −tτ∇̃LµN (θNk )(x) 〉 dt
and corresponding operator on L2µN (M)
ΘNk [g](x) = ∫ M ΘNk (x,x ′)g(x′) dµN (x′),
we have that ΘNk is bounded, and ζNk+1 = ( Id−τΘNk ) [ ζNk ] .
Proof. By the definition of the gradient iteration, we have that
ζNk+1 − ζNk = fθNk −τ∇̃LµN (θNk ) − fθNk .
The total number of trainable parameters in the network is M = n(n(L − 1) + n0 + 1), and the euclidean space in which θ lies is isomorphic to RM . For k ∈ N0, define paths γNk : [0, 1] → RM by
γNk (t) = θ N k − tτ∇̃LµN (θNk ),
so that ζNk+1 − ζNk = fγNk (1) − fγNk (0). We will justify a first-order Taylor representation in integral form based on the previous expression by arguing that for every x ∈ M, t 7→ fγNk (t)(x) is absolutely continuous on [0, 1], by checking the hypotheses of (Cohn, 2013, Theorem 6.3.11). Because γNk is smooth and f( · )(x) is continuous, fγNk (t) is also continuous. Continuity of the features as a function of the parameters and of γ N k implies that for every ` ≥ 0, the image of [0, 1] under the map
t 7→ α`γNk (t)(x)
is compact. By repeated application of Lemma E.21, we conclude that t 7→ fγNk (t)(x) is differentiable at all but countably many points of [0, 1]. Following the proof of Lemma E.21, we see that the points of nondifferentiability of t 7→ fγNk (t)(x) are contained in the set of points of [0, 1] where there exists a layer ` at which at least one of the coordinates of α`
γNk ( · ) (x) vanishes. Applying the
chain rule at points of differentiability of the ReLU [ · ]+ and assigning 0 otherwise, it follows that the derivative of t 7→ fγNk (t)(x) at t ∈ [0, 1] is equal to
−τ 〈 ∇̃LµN (θNk ), ∇̃fγNk (t)(x) 〉 at all but countably many points t ∈ [0, 1]. We finally need to check integrability of this derivative on [0, 1]. We have by linearity
− τ 〈 ∇̃LµN (θNk ), ∇̃fγNk (t)(x) 〉 = −τ ∫ M ζθNk (x ′) 〈 ∇̃fθNk (x ′), ∇̃fγNk (t)(x) 〉 dµN (x′), (B.76)
and by definition〈 ∇̃fγNk (t)(x), ∇̃fθNk (x ′) 〉
= 〈 αLγNk (t) (x),αLθNk (x′) 〉 + L−1∑ `=0 〈 α`γNk (t) (x),α`θNk (x′) 〉〈 β`γNk (t) (x),β`θNk (x′) 〉 .
By construction of the network, the feature maps (t,x) 7→ α` γNk (t) (x) are continuous. For the backward feature maps, we can write for any θ1 = (W 11 , . . . ,W L+1 1 ) and any θ2 = (W 1 2 , . . . ,W L+1 2 ) using Cauchy-Schwarz
∣∣〈β`θ1(x),β`θ2(x′)〉∣∣ ≤ L∏ `′=`+1 ∥∥∥W `′+11 ∥∥∥∥∥∥W `′+12 ∥∥∥, and the RHS of this bound is a continuous function of (θ,x). Because our domain of interest [0, 1] ×M is compact, we have from the triangle inequality, the previous bound on the backward feature inner products and the Weierstrass theorem
sup t∈[0,1],x∈M ∣∣∣〈∇̃fγNk (t)(x), ∇̃fθNk (x′)〉∣∣∣ < +∞, (B.77) so that in particular, we can bound our expression for the derivative of t 7→ fγNk (t)(x) using the triangle inequality as∣∣∣−τ〈∇̃LµN (θNk ), ∇̃fγNk (t)(x)〉∣∣∣ ≤ Cτ ∫M ∣∣∣ζθNk (x′)∣∣∣dµN (x′)
for some constantC > 0. The RHS of the previous bound does not depend on t, so by an application of (Cohn, 2013, Theorem 6.3.11), it follows that t 7→ fγNk (t)(x) is absolutely continuous, and we have the representation
ζNk+1(x)− ζNk (x) = −τ ∫ 1
0
〈 ∇̃LµN (θNk ), ∇̃fγNk (t)(x) 〉 dt.
Using (B.76), we can express this as ζNk+1(x)− ζNk (x) = −τ ∫ 1
0 (∫ M ζθNk (x ′) 〈 ∇̃fθNk (x ′), ∇̃fγNk (t)(x) 〉 dµN (x′) ) dt.
To conclude, it will be convenient to switch the order of integration appearing in the previous expression. Applying (B.77), we have∣∣∣ζθNk (x′)〈∇̃fθNk (x′), ∇̃fγNk (t)(x)〉∣∣∣ ≤ C∣∣∣ζθNk (x′)∣∣∣, and the RHS of this bound is integrable over [0, 1]×M because the network is a continuous function of the input. By Fubini’s theorem, it follows
ζNk+1(x)− ζNk (x) = −τ ∫ M (∫ 1 0 〈 ∇̃fθNk (x ′), ∇̃fγNk (t)(x) 〉 dt ) ζθNk (x ′) dµN (x′) (B.78)
Defining
ΘNk (x,x ′) = ∫ 1 0 〈 ∇̃fθNk (x ′), ∇̃fγNk (t)(x) 〉 dt
and using (B.77), we can define bounded operators ΘNk : L 2 µN (M)→ L 2 µN (M) by
ΘNk [g](x) = ∫ M ΘNk (x,x ′)g(x′) dµN (x′),
and with this definition, (B.78) becomes ζNk+1 − ζNk = −τΘNk [ ζNk ] ,
as claimed.
Lemma B.9. For any network parameters θ, define kernels
Θθ(x,x ′) = 〈 ∇̃fθ(x′), ∇̃fθ(x) 〉 ,
and for ? ∈ {N,∞}, define corresponding bounded operators on L2µ?(M) by
Θθ,µ? [g](x) = ∫ M Θθ(x,x ′)g(x′) dµ?(x′).
For any settings of the parameters θ, the operators Θθ,µ? are self-adjoint, positive, and compact.

heading: In particular, they diagonalize in a countable orthonormal basis of L2µ?(M) functions with corresponding nonnegative eigenvalues.

text:
Proof. When ? = N , an identification reduces the operators Θθ,µ? to operators on finitedimensional vector spaces, and the claims follow immediately from general principles and the finite-dimensional spectral theorem. We therefore only work out the details for the case ? = ∞. Boundedness follows from an argument identical to the one developed in the proof of Lemma B.8, in particular to develop an estimate analogous to (B.77). This estimate, together with separability and compactness of M, also establishes that Θθ,∞ is compact, by standard results for HilbertSchmidt operators (Heil, 2011, §B). In addition, this estimate allows us to apply Fubini’s theorem to write for any g1, g2 ∈ L2µ∞(M)
〈g1,Θθ,∞[g2]〉L2 µ∞ (M) = ∫∫ M×M Θθ(x,x ′)g1(x)g2(x ′) dµ∞(x) dµ∞(x′) = 〈g2,Θθ,∞[g1]〉
since Θθ(x,x′) = Θθ(x′,x). A similar calculation establishes positivity: we have for any g ∈ L2µ∞(M)
〈g,Θθ,∞[g]〉L2 µ∞ (M) = ∫∫ M×M 〈 ∇̃fθ(x′), ∇̃fθ(x) 〉 g(x)g(x′) dµ∞(x) dµ∞(x′)
= 〈∫ M ∇̃fθ(x)g(x) dµ∞(x), ∫ M ∇̃fθ(x)g(x) dµ∞(x) 〉 ≥ 0,
where we applied Fubini’s theorem and linearity of the integral. These facts and the spectral theorem for self-adjoint compact operators on a Hilbert space imply in particular that the operator Θθ,∞ can be diagonalized in a countable orthonormal basis of eigenfunctions (vi)i∈N ⊂ L2µ∞(M) with corresponding nonnegative eigenvalues (λi)i∈N ⊂ [0,+∞).
Lemma B.10. Write ΘµN for the operator defined in Lemma B.9, with the parameters θ set to the initial random network weights and the measure set to µN . There exist absolute constants c,K,K ′ > 0 such that for any q ≥ 0 and any d ≥ Kd0 log(nn0CM), if
τ < 1∥∥ΘµN∥∥L2
µN (M)→L2 µN (M)
and if in addition n ≥ K ′L48+20qd9 log9 L, then one has
P  ⋂ 0≤k≤Lq/(nτ) {∥∥ζNi ∥∥L2 µN (M) ≤ √ d } ≥ 1− (1 + 2Lq nτ ) e−cd.
Proof. Consider the nominal error evolution ζN,nomk , defined iteratively as
ζN,nomk+1 = ζ N,nom k − τΘµN [ ζN,nomk ] ;
ζN,nom0 = ζ
for a step size τ > 0, which satisfies
τ < 1∥∥ΘµN∥∥L2
µN (M)→L2 µN (M)
.
We will prove the claim by showing that this auxiliary iteration is monotone decreasing in the loss, and close enough to the gradient-like iteration of interest that we can prove that the gradient-like iteration also retains a controlled loss. These dynamics satisfy the ‘update equation’
ζN,nomk = ( Id−τΘµN )k [ζ] .
Because M is compact and ζ is a continuous function of the input, we have ζ ∈ L∞(M) for all values of the random weights. Because µN is a probability measure, this means ζ has finiteLp
µN (M)
norm for every p > 0. Meanwhile, the choice of τ and positivity of the operator (by Lemma B.9) guarantees ∥∥Id−τΘµN∥∥L2
µN (M)→L2 µN (M) ≤ 1,
from which it follows from the update equation∥∥∥ζN,nomk ∥∥∥ L2 µN (M) ≤ ‖ζ‖L2 µN (M) ≤ ‖ζ‖L∞(M), (B.79)
where the last inequality uses that µN is a probability measure. In particular, this nominal error evolution is nonincreasing in the relevant loss. Now, we recall the update equation for the finitesample dynamics
ζNk+1 = ( Id−τΘN ) [ ζNk ] ,
which follows from Lemma B.8. Subtracting and rearranging, this gives an update equation for the difference:
ζNk+1 − ζ N,nom k+1 = ( Id−τΘµN ) [ ζNk − ζ N,nom k ] − τ ( ΘNk −ΘµN ) [ ζNk ] . (B.80)
Under our hypothesis on τ , (B.80) and the triangle inequality imply the bound∥∥∥ζNk+1 − ζN,nomk+1 ∥∥∥ L2 µN (M)
≤ ∥∥∥ζNk − ζN,nomk ∥∥∥
L2 µN
(M) + τ ∥∥ζNk ∥∥L2 µN (M) ∥∥ΘNk −ΘµN∥∥L2 µN (M)→L2 µN (M).
Using Jensen’s inequality and the Schwarz inequality, we have∥∥ΘNk −ΘµN∥∥L2 µN (M)→L2 µN (M)
≤ sup ‖g‖
L2 µN
(M)≤1
∫ M ∥∥ΘNk ( · ,x′)−Θ( · ,x′)∥∥L2 µN (M)|g(x ′)|dµN (x′)
≤ sup ‖g‖
L2 µN
(M)≤1 ∥∥ΘNk −Θ∥∥L∞(M×M)‖g‖L1µN (M) ≤ ∥∥ΘNk −Θ∥∥L∞(M×M),
since µN is a probability measure. Defining
∆Nk = max i∈{0,1,...,k} ∥∥ΘNi −Θ∥∥L∞(M×M), by a telescoping series and the identical initial conditions, we thus obtain∥∥∥ζNk+1 − ζN,nomk+1 ∥∥∥
L2 µN
(M) ≤ τ∆Nk k∑ i=0 ∥∥ζNi ∥∥L2 µN (M),
and the triangle inequality and (B.79) then yield∥∥ζNk+1∥∥L2 µN (M) ≤ ‖ζ‖L∞(M) + τ∆ N k k∑ i=0 ∥∥ζNi ∥∥L2 µN (M).
Using a discrete version of (the standard) Gronwall’s inequality, the previous bound implies
∥∥ζNk ∥∥L2 µN (M) ≤ ‖ζ‖L∞(M) + ‖ζ‖L∞(M) k−1∑ i=0 τ∆Nk−1 exp  k−1∑ j=i+1 τ∆Nk−1  ≤ ‖ζ‖L∞(M) ( 1 + kτ∆Nk−1 exp ( kτ∆Nk−1 )) . (B.81)
To conclude, we will use Lemma F.5 and an inductive argument based on (B.81). Let us first observe that by Lemma D.11 (with a rescaling of d, which worsens the absolute constants), we have
P [ ‖ζ‖L∞(M) ≤ √ d
2
] ≥ 1− e−cd (B.82)
as long as n ≥ Kd4L and d ≥ K ′d0 log(nn0CM). Define events ENk by ENk = {∥∥ζNk ∥∥L2
µN (M) >
√ d } ,
where d > 0 is sufficiently large to satisfy the conditions on d given above. We are interested in controlling the probability of ⋃k i=0 ENk for k ∈ N0. We can write
P [ k⋃ i=0 ENi ] = P [ k−1⋃ i=0 ENi ] + P [ ENk ∩ k−1⋂ i=0 ( ENi )c] ,
and unraveling, we obtain
P [ k⋃ i=0 ENk ] = k∑ i=0 P ENi ∩ i−1⋂ j=0 ( ENj )c.
In words, it is enough to control the sum of the measures of the parts of ENk that are common with the part of the space where none of the past events occurs. First, note that (B.82) implies
P [ EN0 ] ≤ e−cd,
and so assume i > 0 below. For any q > 0, if kτ ≤ Lq/n, n ≥ KL36+8qd9 and d ≥ K ′d0 log(nn0CM), Lemma F.5 gives that there are events BNi that respectively contain the sets {∆Ni−1 > CL4+2q/3d3/4n11/12 log 3/4 L}, and which satisfy in addition
P BNi ∩ i−1⋂ j=0 ( ENj )c ≤ e−cd.
We thus have by this last bound, a partition, and intersection monotonicity
P ENi ∩ i−1⋂ j=0 ( ENj )c ≤ e−cd + P[ENi ∩ (BNi )c],
and by construction, one has ∆Ni−1 ≤ CL4+2q/3d3/4n11/12 log 3/4 L on ( BNi )c
. Another partition and (B.82) give
P [ ENi ∩ ( BNi )c] ≤ e−cd + P[ENi ∩ (BNi )c ∩ { ‖ζ‖L∞(M) ≤ √ d
2
}] .
When the two events on the RHS of the last bound are active, we can obtain using (B.81)∥∥ζNk ∥∥L2 µN (M)
≤ √ d
2
( 1 + kτCL4+2q/3d3/4n11/12 log3/4 L exp ( kτCL4+2q/3d3/4n11/12 log3/4 L )) .
Given that kτ ≤ Lq/n, we have
kτCL4+2q/3d3/4n11/12 log3/4 L ≤ ( C12L48+20qd9 log9 L
n
)1/12 ≤ 1/e,
where the last bound holds provided n ≥ KL48+20qd9 log9 L . Thus, on the event ( BNi )c ∩{‖ζ‖L∞(M) ≤ √d2 } , we have ∥∥ζNk ∥∥L2 µN (M) ≤ √ d,
and thus
P [ ENi ∩ ( BNi )c ∩{‖ζ‖L∞(M) ≤ √d2 }] = 0.
By our previous reductions, we conclude
P ENi ∩ i−1⋂ j=0 ( ENj )c ≤ 2e−cd,
and in particular
P [ k⋃ i=0 ENk ] ≤ (2k + 1)e−cd.
The claim is then established by taking k as large as Lq/(nτ).
Corollary B.11. Write ΘµN for the operator defined in Lemma B.9, with the parameters θ set to the initial random network weights θ0 and the measure set to µN , and define for k ∈ N0
∆Nk = max i∈{0,1,...,k} ∥∥ΘNi −Θ∥∥L∞(M×M). There exist absolute constants c, C,C ′,K,K ′ > 0 such that for any q ≥ 0 and any d ≥ Kd0 log(nn0CM), if
τ < 1∥∥ΘµN∥∥L2
µN (M)→L2 µN (M)
.
and if in addition n ≥ K ′L48+20qd9 log9 L, then one has on an event of probability at least 1 − C ′(1 + Lq/(nτ))e−cd
∆NbLq/(nτ)c−1 ≤ C log 3/4(L)d3/4L4+2q/3n11/12.
Proof. Use Lemma B.10 to remove the hypothesis about boundedness of the errors from Lemma F.5, then apply this result together with a union bound.
Lemma B.12. Write Θ for the operator defined in Lemma B.9, with the parameters θ set to the initial random network weights and the measure set to µ∞. Consider the (population) nominal error evolution ζ∞k , defined iteratively as
ζ∞k+1 = ζ ∞ k − τΘ [ζ∞k ] ;
ζ∞0 = ζ
for a step size τ > 0, which satisfies
τ < 1
‖Θ‖L2 µ∞ (M)→L 2 µ∞ (M)
.
Then for any g ∈ L2µ∞(M) and any k satisfying kτ ≥ √ 3e
2
‖g‖L2 µ∞ (M)
‖ζ‖L∞(M) ,
we have
‖ζ∞k ‖L2 µ∞ (M)
≤ √
3‖Θ[g]− ζ‖L2 µ∞ (M)
− 3‖g‖L2 µ∞ (M)
kτ log
(√ 3
2
‖g‖L2 µ∞ (M)
‖ζ‖L∞(M)kτ
) .
Proof. The dynamics satisfy the ‘update equation’
ζ∞k = (Id−τΘ) k [ζ] .
Because M is compact and ζ is a continuous function of the input, we have ζ ∈ L∞(M) for all values of the random weights. Because µ∞ is a probability measure, this means ζ has finite Lpµ∞(M) norm for every p > 0. Using the eigendecomposition of Θ as developed in Lemma B.9, we can therefore write
ζ = ∞∑ i=0 〈vi, ζ〉L2 µ∞ (M)vi
in the sense of L2µ∞(M). Because Θ and Id−τΘ diagonalize simultaneously, we obtain
‖ζ∞k ‖ 2 L2 µ∞ (M) = ∞∑ i=1 (1− τλi)2k〈vi, ζ〉2L2 µ∞ (M) ≤ ∞∑ i=1 e−2kτλi〈vi, ζ〉2L2 µ∞ (M) ,
where the inequality follows from the elementary estimate 1 − x ≤ e−x for x ≥ 0 and our choice of τ , which guarantees that 1 − τλi > 0 for all i ∈ N so that the elementary estimate is valid after squaring. We can split this last sum into two parts: for any λ ∈ R, we have
‖ζ∞k ‖ 2 L2 µ∞ (M)
= ∑
i :λi≥λ
e−2kτλi〈vi, ζ〉2L2 µ∞ (M)
+ ∑
i :λi<λ
e−2kτλi〈vi, ζ〉2L2 µ∞ (M) .
Because Θ is positive, we have further that λi ≥ 0 for all i, so we can take λ ≥ 0. The first sum consists of large eigenvalues: we use exp(−2kτλi) ≤ exp(−2kτλ) to preserve their effect, and then upper bound the remainder of the sum by the squared L2µ∞ norm of ζ. The second sum consists of small eigenvalues: we replace exp(−2kτλi) ≤ 1, and then plug in ζ = Θ[g] − (Θ[g] − ζ) and use bilinearity, self-adjointness of Θ, and the triangle inequality to get∣∣∣〈vi, ζ〉L2
µ∞ (M) ∣∣∣ ≤ ∣∣∣λ〈vi, g〉L2 µ∞ (M) ∣∣∣+ ∣∣∣〈vi,Θ[g]− ζ〉L2 µ∞ (M) ∣∣∣. We then square both (nonnegative) sides of the inequality and use Cauchy-Schwarz to replace the squared sum with the sum of squares times a constant, obtaining
‖ζ∞k ‖ 2 L2 µ∞ (M) ≤ e−2kτλ‖ζ‖2L2 µ∞ (M) + 3λ2‖g‖2L2 µ∞ (M) + 3‖Θ[g]− ζ‖2L2 µ∞ (M)
after re-adding indices i to the sum to obtain the third residual. We will choose λ ≥ 0 to minimize the sum of the first and second terms. Differentiating and setting to zero gives the critical point equation
2
3
‖ζ‖2 L2 µ∞ (M) (kτ)2
‖g‖2 L2 µ∞ (M)
= (2tλ)e2kτλ,
which can be inverted to give the unique critical point
λ = 1
2kτ W 2 3 ‖ζ‖2 L2 µ∞ (M) (kτ)2
‖g‖2 L2 µ∞ (M)  , where W is the Lambert W function, defined as the principal branch of the inverse of z 7→ zez; we know that this critical point is a minimizer because the function of λ we differentiated diverges as λ→∞. Plugging this point into the sum of the first two terms gives
‖ζ∞k ‖ 2 L2 µ∞ (M) ≤ 3‖Θ[g]− ζ‖2L2 µ∞ (M)
+ ‖g‖2 L2 µ∞ (M)
(2/3)(kτ)2 1 + 1 2 W ‖ζ‖2L2µ∞ (M)(kτ)2 (3/2)‖g‖2
L2 µ∞ (M)
W ‖ζ‖2L2µ∞ (M)(kτ)2
(3/2)‖g‖2 L2 µ∞ (M)  . For x ≥ 0, the function x 7→ W (x) is strictly increasing, as the inverse of y 7→ yey; by definition W (e) = 1; and we have the representation W (z)+ logW (z) = log z (Corless et al., 1996), whence W (x) ≤ log x if x ≥ e. Because µ∞ is a probability measure, we have
‖ζ‖2L2 µ∞ (M) ≤ ‖ζ‖2L∞ ,
and therefore if
kτ ≥ √ 3e
2
‖g‖L2 µ∞ (M)
‖ζ‖L∞(M) ,
we can simplify the previous bound to
‖ζ∞k ‖ 2 L2 µ∞ (M) ≤ 3‖Θ[g]− ζ‖2L2 µ∞ (M)
+ 9‖g‖2 L2 µ∞ (M)
4(kτ)2 log2 3 2 ‖g‖2 L2 µ∞ (M)
‖ζ‖2L∞(M)(kτ)2  , using also properties of the logarithm. Taking square roots and using the Minkowski inequality then yields
‖ζ∞k ‖L2 µ∞ (M)
≤ √
3‖Θ[g]− ζ‖L2 µ∞ (M)
− 3‖g‖L2 µ∞ (M)
kτ log
(√ 3
2
‖g‖L2 µ∞ (M)
‖ζ‖L∞(M)kτ
) ,
where we used the previous lower bound on kτ to determine the sign that the absolute value of the logarithm takes. This gives the claim.
Lemma B.13 (Kantorovich-Rubinstein Duality). Let Lip(M) denote the class of functions f : M→ R such that both f ∣∣ M±
are Lipschitz with respect to the Riemannian distances onM±. For any d ≥ 1, any 0 < δ ≤ 1 and any N ≥ 2 √ d/min{µ∞(M+), µ∞(M−)}, one has that on an event of probability at least 1− 8e−d, simultaneously for all f ∈ Lip(M)∣∣∣∣∫ M f(x) dµ∞(x)− ∫ M f(x) dµN (x)
∣∣∣∣ ≤ 2‖f‖L∞(M) √ d
N + e14/δCµ∞,M
√ dmax?∈{+,−} ∥∥∥f ∣∣M?∥∥∥Lip N1/(2+δ) ,
where
Cµ∞,M = len (M+) µ∞(M+) + len (M−) µ∞(M−) .
Proof. The proof is an application of the Kantorovich-Rubinstein duality theorem for the 1- Wasserstein distance (Weed & Bach, 2019, eq. (1)), which states that for any two Borel probability measures µ, ν onM±, one has
W (µ, ν) = sup ‖f‖Lip≤1 ∣∣∣∣∣ ∫ M± f(x) dµ(x)− ∫ M± f(x) dν(x) ∣∣∣∣∣, whereM± denotes either ofM+ orM−, and ‖·‖Lip is the minimal Lipschitz constant with respect to the Riemannian distance onM±. Therefore for any f :M± → R Lipschitz, we have∣∣∣∣∣ ∫ M± f(x) dµ(x)− ∫ M± f(x) dν(x)
∣∣∣∣∣ ≤ ‖f‖LipW (µ, ν) , (B.83) where one checks separately the case where ‖f‖Lip = 0 to see that this bound holds there as well. To go from (B.83) to the desired conclusion, we need to pass from the measures µ∞ and µN , both supported onM, to measures µ?± (with ? ∈ {N,∞}), supported on the manifoldsM± (which we will define in detail below); the challenge here is that the number of ‘hits’ of each manifold M± that show up in the finite sample measure µN is a random variable, which requires a small detour to control. Let us define random variables N+, N− by
N+ = Nµ N (M+) ; N− = NµN (M−) ,
so that N± have support in {0, 1, . . . , N}, and N+ +N− = N . Define in addition p+ = µ
∞ (M+) ; p− = µ∞ (M−) , which represent the degree of imbalance between the positive and negative classes in the data. By definition of the i.i.d. sample, we have thatN+ ∼ Binom(N, p+). UsingN+ andN−, we can define ‘conditional’ finite sample measures µN+ and µ N − by
µN+ = 1 max{1, N+} ∑
i∈[N ] :xi∈M+
δ{xi}; µ N − =
1 max{1, N−} ∑
i∈[N ] :xi∈M−
δ{xi},
so that (N+/N)µN+ + (N−/N)µ N − = µ N ,8 and µN+ and µ N − are both probability measures except when N+ ∈ {0, N}, in which case exactly one is a probability measure. By the triangle inequality, we have for any continuous f :M→ R∣∣∣∣∫
M f(x) dµ∞(x)− ∫ M f(x) dµN (x) ∣∣∣∣ ≤
∑ ?∈{+,−} ∣∣∣∣p? ∫ M? f(x) dµ∞? (x) p? − N? N ∫ M? f(x) dµN? (x) ∣∣∣∣ ≤
∑ ?∈{+,−} ‖f‖L∞(M) ∣∣∣∣N?N − p? ∣∣∣∣+ ∣∣∣∣∫ M? f(x) dµ∞? (x) p? − ∫ M? f(x) dµN? (x) ∣∣∣∣. (B.84) 8Here we treat the empty sum as the appropriate ‘zero element’ of the space of finite signed Borel measures
onM±, namely the trivial measure that assigns zero to every Borel subset ofM±.
By Lemma G.1, we have
P [∣∣∣∣N?N − p? ∣∣∣∣ ≤ √ d N ] ≥ 1− 2e−2d. (B.85)
Using that N −N+ = N− and 1− p+ = p−, the bound (B.85) implies if N ≥ 2 √ d/min{p+, p−}
P [ p? 2 ≤ N? N ≤ 1− p? 2 ] ≥ 1− 2e−2d. (B.86)
Now fix an arbitrary f ∈ Lip(M). For either ? ∈ {+,−}, we can write∫ M? f(x) dµN? (x) = 1 max{1, N?} ∑
i :xi∈M?
f(xi) = 1 max{1, ∑N i=1 1xi∈M?} N∑ i=1 1xi∈M?f(xi),
and sinceM+ andM− are separated by a positive distance ∆ > 0, we have that xi 7→ 1xi∈M? are continuous functions onM. Since f is continuous onM by the same reasoning and the fact that M is compact, it follows that the functions (x1, . . . ,xN ) 7→ ∫ M? f(x) dµ N ? (x) are continuous on
M× · · · ×M as well, and in particular for any t > 0 the sets{∣∣∣∣∫ M? f(x) dµ∞? (x) p? − ∫ M? f(x) dµN? (x) ∣∣∣∣ > t} are open in M, and so is their union over all f ∈ Lip(M). By conditioning, we can then apply (B.86) to write
P  ⋃ f∈Lip(M) {∣∣∣∣∫ M? f(x) dµ∞? (x) p? − ∫ M? f(x) dµN? (x) ∣∣∣∣ > t}
≤ 2e−2d + dN(1−p?)/2e∑ k=bNp?/2c P  ⋃ f∈Lip(M) {∣∣∣∣∣ ∫ M? f(x) dµ∞? (x) p? − ∫ M? f(x) dµ N ? (x) ∣∣∣∣∣ > t } ∣∣∣∣∣∣N? = k P[N? = k]. (B.87)
Conditioned on {N? = k} with 0 < k < N , the measure µN? is distributed as an empirical measure of sample size k from the probability measure µ∞? /p? supported on M?. For bNp?/2c ≤ k ≤ dN(1− p?)/2e, any δ > 0 and any d ≥ 1 we have for both possible values of ?√
de14/δ len(M?) k1/(2+δ) ≤ √ de14/δ len(M?)(⌊ Np?
2 ⌋)1/(2+δ) ≤ √
2de14/δ len(M?) (Np?) 1/(2+δ) ,
and so an application of Lemma B.16 thus gives for any 0 < δ ≤ 1 and any d ≥ 2
P [ W ( dµ∞? (x)
p? ,dµN?
) >
√ de14/δ len(M?) (Np?) 1/(2+δ) ∣∣∣∣∣N? = k ] ≤ e−d.
Combining this last bound with (B.83) and (B.87) gives
P  ⋃ f∈Lip(M)  ∣∣∣∫M? f(x)dµ∞? (x)p? − ∫M? f(x) dµN? (x)∣∣∣
>
√ de14/δ ∥∥∥∥f∣∣∣M? ∥∥∥∥
Lip N1/(2+δ) len(M?) p?
  ≤ 3e−d.
where we used max{p+, p−} ≤ 1 to remove the exponent of 1/(2 + δ) on these terms. Taking a max over the Lipschitz constants and combining this bound with (B.84) and (B.85) and a union bound, we obtain
P  ⋃ f∈Lip(M)  ∣∣∫ M f(x) dµ ∞(x)− ∫ M f(x) dµ N (x) ∣∣ > 2‖f‖L∞(M) √ d N
+ e14/δCµ∞,M
√ dmax?∈{+,−} ∥∥∥∥f∣∣∣M? ∥∥∥∥
Lip
N1/(2+δ)
  ≤ 8e−d,
where the constant is defined as in the statement of the lemma.
Lemma B.14. Let d0 = 1. There is an absolute constant C > 0 such that for any function f : M→ R with f ∣∣ M± Lipschitz with respect to the Riemannian distances onM±, one has
‖f‖L∞ ≤ C max  ρ1/2max‖f‖L2 µ∞ (M) ρ 1/2 min(min {µ∞(M+),µ∞(M−)}) 1/2 , ‖f‖2/3 L2 µ∞ (M) max?∈{+,−} ∥∥∥∥f∣∣∣M? ∥∥∥∥1/3 Lip
ρ 1/3 min
.
Proof. For any T > 0 and a nonconstant Lipschitz function f : [0, T ] → R, we will establish the inequality
‖f‖L∞ ≤ C max { ‖f‖L2√
T , ‖f‖2/3L2 ‖f‖ 1/3 Lip
} , (B.88)
where the constant C > 0 is absolute. We can use this result to establish the claim. We start by writing
‖f‖L∞ = max ?∈{+,−} ∥∥∥f ∣∣M?∥∥∥L∞ , and for ? ∈ {+,−}, we have ∥∥∥f ∣∣M?∥∥∥L∞ = ‖f ◦ γ?‖L∞ , (B.89) where γ? : [0, len(M?)] → M? are the smooth unit-speed curves parameterized with respect to arc length parameterizing the manifolds. Similarly, the curves’ parameterization with respect to arc length implies
‖f ◦ γ?‖Lip ≤ ∥∥∥f ∣∣M?∥∥∥Lip. (B.90)
Applying (B.88) with (B.89) and (B.90), we obtain∥∥∥f ∣∣M?∥∥∥L∞ ≤ C max { ‖f ◦ γ?‖L2√ len(M?) , ‖f ◦ γ?‖2/3L2 ∥∥∥f ∣∣M?∥∥∥1/3Lip } .
For the first term in the max, we have
‖f ◦ γ?‖2L2 len(M?)
= ∣∣∣∣∣ 1len(M?) ∫ len(M?)
0
f ◦ γ?(t)2 dt ∣∣∣∣∣ ≤ ∣∣∣∣∣ ∫ len(M?)
0
f ◦ γ?(t)2ρ? ◦ γ?(t) ρ? ◦ γ?(t)− 1len(M?)
ρ? ◦ γ?(t) dt ∣∣∣∣∣ + ∣∣∣∣∣ ∫ len(M?)
0
f ◦ γ?(t)2ρ? ◦ γ?(t) dt ∣∣∣∣∣ using the triangle inequality. For the second term in the last bound, we note that∣∣∣∣∣ ∫ len(M?) 0 f ◦ γ?(t)2ρ? ◦ γ?(t) dt
∣∣∣∣∣ ≤ ∣∣∣∣∣ ∫ len(M+)
0
f ◦ γ+(t)2ρ+ ◦ γ+(t) dt ∣∣∣∣∣+ ∣∣∣∣∣ ∫ len(M−)
0
f ◦ γ−(t)2ρ− ◦ γ−(t) dt ∣∣∣∣∣ ≤ ‖f‖2L2
µ∞ (M) , (B.91)
and for the first term, we have
max t∈[0,len(M?)] ∣∣∣∣∣ρ? ◦ γ?(t)− 1 len(M?) ρ? ◦ γ?(t) ∣∣∣∣∣ ≤ max t∈[0,len(M?)] ∣∣∣ρ? ◦ γ?(t)− ρ?◦γ?(t)µ∞(M?) ∣∣∣+ ∣∣∣ ρ?◦γ?(t)µ∞(M?) − 1len(M?) ∣∣∣ ρ? ◦ γ?(t) ≤ 1− µ ∞(M?)
µ∞(M?) + ρmax µ∞(M?)ρmin
≤ 2ρmax µ∞(M?)ρmin , (B.92)
where in the first inequality we used the triangle inequality, and for the second we used that ρ?◦γ? integrates to µ∞(M?) over [0, len(M?)], which implies that there exists at least one t ∈ [0, len(M?)] at which ρ? ◦ γ?(t) ≥ µ∞(M?)/ len(M?), so that the maximum of the difference in the second term on the RHS of the first inequality is bounded by the maximum of the density term. Thus, by Hölder’s inequality and (B.91) and (B.92), we have∣∣∣∣∣ ∫ len(M?) 0 f ◦ γ?(t)2ρ? ◦ γ?(t) ρ? ◦ γ?(t)− 1len(M?) ρ? ◦ γ?(t) dt
∣∣∣∣∣ ≤ 3ρmax ρmin min {µ∞(M+), µ∞(M−)} ‖f‖2L2 µ∞ (M)
Similarly, for the second term in the max, we have
‖f ◦ γ?‖2/3L2
= (∫ len(M?) 0 f ◦ γ?(t)2 dt )1/3
≤ (∫ len(M+) 0 f ◦ γ+(t)2 dt+ ∫ len(M−) 0 f ◦ γ−(t)2 dt )1/3
≤ 1 ρ
1/3 min
(∫ len(M+) 0 f ◦ γ+(t)2ρ+ ◦ γ+(t) dt+ ∫ len(M−) 0 f ◦ γ−(t)2ρ− ◦ γ−(t) dt )1/3
≤ ‖f‖2/3 L2 µ∞ (M)
ρ 1/3 min
.
Thus, we have ∥∥∥f ∣∣M?∥∥∥L∞ ≤ C max
ρ 1/2 max‖f‖L2
µ∞ (M)
ρ 1/2 min (min {µ∞(M+), µ∞(M−)})
1/2 , ‖f‖2/3 L2 µ∞ (M) ∥∥∥f ∣∣M?∥∥∥1/3Lip ρ
1/3 min , and taking a maximum over ? ∈ {+,−} establishes the claim. To prove (B.88), consider first the trivial case where ‖f‖L∞ = 0: here the LHS and RHS of (B.88) are identical, and the proof is immediate. When ‖f‖L∞ > 0, the Weierstrass theorem implies that there exists t ∈ [0, T ] such that |f(t)| = ‖f‖L∞ ; we consider the case sign(f(t)) > 0. For any t′ ∈ [0, T ], we can write by the Lipschitz property
f(t′) ≥ ‖f‖L∞ − ‖f‖Lip|t− t′|,
and the RHS of the previous bound is nonnegative on the intersection of the interval [t − ‖f‖L∞‖f‖−1Lip, t+‖f‖L∞‖f‖ −1 Lip] with the domain [0, T ] (with standard extended-valued arithmetic
conventions when ‖f‖Lip = 0). This gives the bound
‖f‖2L2 ≥ ∫ min{t+ ‖f‖L∞‖f‖Lip ,T}
max { t− ‖f‖L∞‖f‖Lip ,0 } (‖f‖L∞ − ‖f‖Lip|t− t′|)2 dt′
= ∫ min{ ‖f‖L∞‖f‖Lip ,T−t} max { − ‖f‖L∞‖f‖Lip ,−t
} (‖f‖L∞ − ‖f‖Lip|t′|)2 dt′, where the second line follows from the changes of variables t′ 7→ t′ + t. The integrand on the RHS of the second line in the previous bound is even-symmetric, and max { −‖f‖L∞‖f‖Lip ,−t } =
−min { ‖f‖L∞ ‖f‖Lip , t } , so we can discard one side of the interval of integration to get
∫ min{ ‖f‖L∞‖f‖Lip ,T−t} max { − ‖f‖L∞‖f‖Lip ,−t } (‖f‖L∞ − ‖f‖Lip|t′|)2 dt′ (B.93)
≥ ∫ min{ ‖f‖L∞‖f‖Lip ,max{t,T−t}}
0
(‖f‖L∞ − ‖f‖Lipt′) 2 dt′. (B.94)
We proceed analyzing two distinct cases. First, if ‖f‖L∞ ≤ max{t, T − t}‖f‖Lip, then we must have ‖f‖Lip > 0; integrating the RHS of (B.94), we obtain
‖f‖2L2 ≥ ‖f‖3L∞ 3‖f‖Lip ,
or equivalently ‖f‖L∞ ≤ 31/3‖f‖2/3L2 ‖f‖ 1/3 Lip . (B.95)
Next, we consider the case ‖f‖L∞ > max{t, T − t}‖f‖Lip. We split on two sub-cases: when ‖f‖Lip = 0, integrating (B.94) gives
‖f‖2L2 ≥ ‖f‖2L∞ max{t, T − t} ≥ T‖f‖2L∞
2 , (B.96)
where we used max{t, T − t} ≥ T/2. When ‖f‖Lip > 0, integrating (B.94) gives
‖f‖2L2 ≥ 1
3‖f‖Lip
( ‖f‖3L∞ − (‖f‖L∞ − ‖f‖Lip max{t, T − t}) 3 )
= max{t, T − t}
3
2∑ k=0 ‖f‖2−kL∞ (‖f‖L∞ − ‖f‖Lip max{t, T − t}) k
≥ T‖f‖ 2 L∞
6 , (B.97)
where the second line uses a standard algebraic identity, and the third line uses max{t, T−t} ≥ T/2 together with the definition of the case to get that ‖f‖L∞ − ‖f‖Lip max{t, T − t} > 0 in order to discard all but the k = 0 summand. Combining (B.97) and (B.96), we obtain for this case
‖f‖L∞ ≤ √
6‖f‖L2√ T , (B.98)
and combining (B.95) and (B.98) gives unconditionally
‖f‖L∞ ≤ max {√ 6‖f‖L2√ T , 31/3‖f‖2/3L2 ‖f‖ 1/3 Lip } ,
which establishes (B.88). For the case sign(f(t)) < 0, apply the preceding argument to −f to conclude. See (Brezis, 2011, Exercise 8.15) for a sketch of a proof that leads to more general versions of (B.88).
Lemma B.15. For any p ∈ N, if C ≥ (4p)4p, then one has
n ≥ C logp n if n ≥ 2pC logp(2pC).
Proof. We first give a proof for p = 1, then build off this proof for the general case. Consider the function f(x) = cx − log x. We have f ′(x) = c − 1/x, which is nonnegative for every x ≥ 1/c, so in particular f is increasing under this condition. By concavity of the logarithm, we have log x ≤ log(2/c) + (c/2)(x− 2/c), whence
f(x) ≥ 1 + cx/2− log(2/c).
The RHS of this bound is equal to zero at x = (2/c)(log(2/c)− 1), and
2
c
( log ( 2
c ) − 1 ) ≥ 1 c
⇐⇒ c ≤ 2e−3/2.
In particular, we have f(x) ≥ 0 for every x ≥ (2/c) log(2/c). Rearranging this bound, we can assert the desired conclusion that if C ≥ 3, then n ≥ C log n for every n ≥ 2C log 2C. Equivalently, we have for all such n that Cn−1 log n ≤ 1. Next, we consider the case of p > 1. We will show
C logp n
n ≤ 1
under suitable conditions. Let us consider the choice n = KC logpKC, where K > 0 is a constant we will specify below. Consider the function f(x) = Cx−1 logp x, which satisfies
f ′(x) = C logp−1(x)(p− logp−1(x))
x2 .
In particular, f is decreasing as soon as p ≤ logp−1(x). Now, we can calculate
f (KC logpKC) = 1
K
( 1 + p log logKC
logKC
)p ,
and by our result for the case p = 1, we have for all p ≥ 2
p log logKC
logKC ≤ 1 if logKC ≥ 4p log 4p.
This condition is satisfied for KC ≥ (4p)4p, so if we set K = 2p, we obtain the above conclusion when C ≥ (4p)4p. Under these conditions, we then get
f(KC logpKC) ≤ 1.
Similarly, we have logp−1(KC logKC) ≥ logp−1((4p)4p) = (4p)p−1 logp−1(4p), which is larger than p because 4p ≥ e. It follows that f(x) ≤ 1 for every x ≥ KC logKC, which completes the proof.
Lemma B.16 (Concentration of Empirical Measure in Wasserstein Distance (Weed & Bach, 2019)). Let d0 = 1. For either ? ∈ {+,−}, let µ be a Borel probability measure on M?, and write µN for the empirical measure corresponding to N i.i.d. samples from µ. Then for any d ≥ 1 and any 0 < δ ≤ 1, one has
P [ W(µ, µN ) ≤
√ de14/δ len(M?) N1/(2+δ)
] ≥ 1− e−2d,
where the 1-Wasserstein distance is taken with respect to the Riemannian distance.
Proof. The proof is a direct application of the results of (Weed & Bach, 2019) on concentration of empirical measures in Wasserstein distance. For the duration of the proof, we will work on the metric space (M?, len(M?)−1 distM?( · )), i.e., the same metric space scaled to have unit diameter; we will then obtain the result in terms of the unscaled metric by the definition of the 1-Wasserstein distance.
Because d0 = 1 andM? can be given as a unit-speed curve parameterized with respect to arc length, we have for any Borel S ⊂ [0, 1] and any ε > 0
Nε(S) ≤ 1
ε ,
where Nε(S) denotes the ε-covering number of S by closed balls in the rescaled metric. Following the notation of (Weed & Bach, 2019, §4.1), we then obtain for any s > 2
dε(µ, ε s/(s−2)) =
log inf { Nε(S) ∣∣ µ(S) ≥ 1− εs/(s−2)} − log ε ≤ 1.
Invoking (Weed & Bach, 2019, Proposition 5), we obtain after some simplifications of the constants that for any 0 < δ ≤ 1 (putting s = δ + 2 in the previous estimates)
E [ W ( µ, µN )] ≤ 311/δN−1/(2+δ) + 36N−1/2 ≤ e14/δN−1/(2+δ),
where the final inequality worst-cases constants for convenience. Using (Weed & Bach, 2019, Proposition 20), we have
P [ W(µ, µN ) + E [ W(µ, µN ) ] ≥ √ d
N
] ≤ e−2d,
and hence
P [ W(µ, µN ) ≥ √ de14/δ
N1/(2+δ)
] ≤ P [ W(µ, µN ) ≥ e 14/δ
N1/(2+δ) +
√ d
N
]
≤P [ W(µ, µN ) + E [ W(µ, µN ) ] ≥ √ d
N
] ≤ e−2d
if d ≥ 1.
Lemma B.17. Let n,m ∈ N. Let F : Rn → Rm be 1-nonnegatively homogeneous, and suppose there exist M,L ≥ 0 such that
1. ∥∥‖F ∣∣Sn−1‖2∥∥L∞ ≤M ;
2. F ∣∣ Sn−1 is L-Lipschitz.
Then for any x,x′ ∈ Rn, one has ‖F (x)− F (x′)‖2 ≤ (2L+M)‖x− x
′‖2, so that F is (2L+M)-Lipschitz.
Proof. For any numbers a, b ≥ 0 and any u,v ∈ Rm, one has by the triangle inequality ‖au− bv‖2 ≤ min{a‖u− v‖2 + |a− b|‖v‖2, b‖u− v‖2 + |a− b|‖u‖2}. Using an elementary property of the min and the max, we thus have ‖au− bv‖2 ≤ min{a, b}‖u− v‖2 + max{‖u‖2, ‖v‖2}|a− b|. (B.99) Now we proceed to show the claim. Noting that the case where both x,x′ are zero is trivial, first consider the case where x is nonzero and x′ is zero. By nonnegative homogeneity, it suffices to proceed as
‖F (x)− F (x′)‖2 = ‖F (x)‖2 = ‖x‖2 ∥∥∥∥F ( x‖x‖2 )∥∥∥∥ 2 ≤M‖x‖2 = M‖x− x′‖2
to conclude; for the inequality we used the boundedness assumption on F . Now fix x,x′ ∈ Rn nonzero. The inequality (B.99) can be applied to get
‖F (x)− F (x′)‖2 = ∥∥∥∥‖x‖2F ( x‖x‖2 ) − ‖x′‖2F ( x′ ‖x′‖2 )∥∥∥∥ 2
≤ min{‖x‖2, ‖x′‖2} ∥∥∥∥F ( x‖x‖2 ) − F ( x′ ‖x′‖2 )∥∥∥∥ 2
+ max {∥∥∥∥F ( x‖x‖2 )∥∥∥∥ 2 , ∥∥∥∥F ( x′‖x′‖2 )∥∥∥∥ 2 } ‖x− x′‖2,
where in the inequality we also applied the `2 triangle inequality. Using the assumed properties of F , we thus have
‖F (x)− F (x′)‖2 ≤ Lmin{‖x‖2, ‖x ′‖2} ∥∥∥∥ x‖x‖2 − x ′ ‖x′‖2 ∥∥∥∥ 2 +M‖x− x′‖2.
By a classical inequality (e.g. proved in (E.15)), one has∥∥∥∥ x‖x‖2 − x ′ ‖x′‖2 ∥∥∥∥ 2 ≤ 2 ‖x− x ′‖2 max{‖x‖2, ‖x′‖2} , whence ‖F (x)− F (x′)‖2 ≤ (2L+M)‖x− x
′‖2, as was to be shown.

heading: C SKELETON ANALYSIS AND CERTIFICATE CONSTRUCTION

text:
In this section, we construct a certificate g for the certificate problem (B.1) in the context of a simple model geometry. We also collect technical estimates relevant to the analysis of the skeleton ψ. We point to Appendix A.5.2 for a summary of the operator and function definitions relevant to the certificate problem that we will use below. We will use the notation
Θ̂[g](x) = ∫ M ψ ◦ ∠(x,x′)g(x′) dµ∞(x′)
in this section; we call explicit attention to this notation to avoid confusion with the kernel Θ̂ = ψ1 ◦ ∠ that we have defined in the main text for convenience of exposition.

heading: C.1 CERTIFICATE CONSTRUCTION

text:
To construct a certificate, it suffices to solve the integral equation
ζ̂ = Θ̂[g] (C.1) for a function g ∈ L2µ∞(M) and obtain estimates on the norm of g. It is useful to consider separately the contributions of integration over the class manifolds M± in the action of the operator Θ̂: we can write for any g
Θ̂[g](x) = ∫ M+ ψ ◦ ∠(x,x′)g(x′) dµ∞+ (x′) + ∫ M− ψ ◦ ∠(x,x′)g(x′) dµ∞− (x′),
and it then makes sense to further subdivide based on whether the evaluation point x lies in M+ orM−, and to introduce the density ρ explicitly by a change of variables. With a slight abuse of notation, we will write dx′ to denote the Riemannian measure on M+ and M−, for concision. Because the kernel ψ ◦ ∠ is symmetric, if we define an operator Θ̂+ : L2(M+)→ L2(M+) by
Θ̂+[g+](x) = ∫ M+ ψ ◦ ∠(x,x′)g+(x′) dx′,
an operator Θ̂− : L2(M−)→ L2(M−) by
Θ̂−[g−](x) = ∫ M− ψ ◦ ∠(x,x′)g−(x′) dx′,
and an operator Θ̂± : L2(M+)→ L2(M−) by
Θ̂±[g+](x) = ∫ M+ ψ ◦ ∠(x,x′)g+(x′) dx′,
then we can write the certificate system (C.1) equivalently as the 2× 2 block operator equation[ ζ̂+ ζ̂− ] = [ Θ̂+ Θ̂ ∗ ± Θ̂± Θ̂− ] [ ρ+g+ ρ−g− ] ,
where we write ρ+ and ρ− for the restriction of the density ρ to M+ and M−, respectively, and where the adjoint operation is viewed as occurring with operators between L2(M+) and L2(M−) (both Hilbert spaces). We will make use of this notation in the sequel.

heading: C.1.1 TWO CIRCLES

text:
The two circles geometry is a highly-symmetric geometry whereM+ andM− are coaxial circles in the upper and lower hemispheres of S2, each of radius 0 < r < 1. Here we note that since the skeleton ψ depends only on the angle between points of Sn0−1, the particular embedding of this geometry into Sn0−1 is irrelevant, and it is without loss of generality to consider the geometry in S2 once we have restricted ourselves to this configuration. We have unit-speed charts, for t ∈ [0, 2πr]
γ+(t) = r cos t/rr sin t/r√ 1− r2  , γ−(t) =  r cos t/rr sin t/r − √ 1− r2  , which implies specific forms of the spherical distances
∠ (γ+(t),γ+(t ′)) = cos-1 ( r2 cos ∣∣∣∣ t− t′r ∣∣∣∣+ (1− r2)) (C.2)
and
∠ (γ+(t),γ−(t ′)) = cos-1 ( r2 cos ∣∣∣∣ t− t′r ∣∣∣∣− (1− r2)) , (C.3)
with the analogous results for the remaining possible combinations of domains, by symmetry. Because ζ̂ is piecewise constant on each connected component ofM, there are constants C+, C− such that C+ = ζ̂ onM+ and C− = ζ̂ onM−. The block-structured system we are interested in solving is then [
C+ C−
] = [ Θ̂+ Θ̂ ∗ ±
Θ̂± Θ̂−
] [ ρ+g+ ρ−g− ] , (C.4)
where subscripts are used to denote the domain of each component of the certificate. The coordinate representations (C.2) and (C.3) show that each of the operators appearing in the 2×2 matrix in (C.4) is invariant on the circle; we can obtain some useful simplifications by identifying these operators with their coordinate representations. Defining
fr(t) = cos -1 ( r2 cos t+ (1− r2) ) ,
gr(t) = cos -1 ( r2 cos t− (1− r2) ) ,
and (self-adjoint) operators on 2π-periodic functions g by A[g](t) = ∫ 2π
0
ψ ◦ fr(t− t′)g(t′) dt′,
X [g](t) = ∫ 2π
0
ψ ◦ gr(t− t′)g(t′) dt′,
by a change of coordinates, it is equivalent to solve the system[ r−1C+ r−1C− ] = [ A X X A ] [ ρ+g+ ρ−g− ] , (C.5)
where we have identified ρ+ and ρ− with their coordinate representations, and with an abuse of notation used the same notation for the certificate as in (C.4). We can use symmetry properties to determine gr(t) = π − fr(t− π), so for purposes of analysis we need only consider fr. Each of the invariant operators in (C.5) diagonalizes in the Fourier basis, and because the target ζ̂ is a piecewise constant function, we only need to use the first Fourier coefficient. In other words, we can solve this system by first inverting the invariant operator, which responds to only the constant component of the target, and then inverting the density multiplication operators. This approach is made precise in the following lemma.
Lemma C.1. There is an absolute constant K > 0 such that if L ≥ max{K, (π/2)(1 − r2)−1/2} and r ≥ 12 , then the system (C.4) has a solution that satisfies∥∥∥∥[g+g− ]∥∥∥∥ L2 µ∞ ≤ 64 ∥∥∥ζ̂∥∥∥ L∞(M) nπ1/2ρ 1/2 min .
Proof. Following the discussion by (C.5), it is equivalent to solve the system in the Fourier basis, with only the DC component. We thus start by solving the system[
r−1C+ r−1C−
] = [ 2 ∫ π 0 ψ ◦ fr(t) dt 2 ∫ π 0 ψ ◦ gr(t) dt
2 ∫ π
0 ψ ◦ gr(t) dt 2 ∫ π 0 ψ ◦ fr(t) dt ] [ G+ G− ] ,
where G+ and G− are constants that we will show exist. This is a 2 × 2 system, and the matrix is symmetric, with minimum eigenvalue 2 ∫ π 0
(ψ ◦ fr − ψ ◦ gr)(t) dt. Using Lemma C.2, we have if L ≥ max{K, (π/2)(1− r2)−1/2} and r ≥ 12
2 ∫ π 0 (ψ ◦ fr − ψ ◦ gr)(t) dt ≥ πn 32r ,
so the 2×2 matrix is invertible, and by an operator norm bound on its inverse we have the regularity estimate (
G2+ +G 2 − )1/2 ≤ 32
πn
( C2+ + C 2 − )1/2 .
It follows that the function [ g+ g− ] = [ G+ ρ+ G− ρ− ] solves the system (C.4). We conclude∥∥∥∥[g+g− ]∥∥∥∥2 L2 µ∞ = ∫ 2π 0 ( G+ ρ+ ◦ γ+(t) )2 ρ+ ◦ γ+(t) dt+ ∫ 2π 0 ( G− ρ− ◦ γ−(t) )2 ρ− ◦ γ−(t) dt
≤ 2 11
πn2ρmin
( C2+ + C 2 − ) .
Taking square roots on both sides of the expression resulting from the last inequality will give the claim, after we simplify the expression √ C2+ + C
2 −. Since√
C2+ + C 2 − ≤ √ 2 max{C+, C−} = √ 2 ∥∥∥ζ̂∥∥∥
L∞(M) ,
we can conclude after adjusting constants.
Lemma C.2. There exists an absolute constantK > 0 such that ifL ≥ max{K, (π/2)(1−r2)−1/2} and if r ≥ 12 , one has
2 ∫ [0,π] (ψ ◦ fr − ψ ◦ gr)(t) dt ≥ πn 32r .
Proof. Write σr = ψ ◦ fr − ψ ◦ gr for brevity, which is nonnegative, by Lemma C.3. We consider the tangent line to the graph of σr at 0; by Lemma C.3, this line has the form t 7→ σr(0)− tnrL(L+ 1)/4π, and its graph hits the horizontal axis at t = 4πσr(0)/nrL(L + 1). Using that σr(0) ≤ ψ(0) = nL/2, we see that this point of intersection is no larger than 2π/r(L + 1), which can be made less than K by choosing L ≥ K ′, where K > 0 is the absolute constant appearing in the convexity bound of Lemma C.3, and K ′ > 0 is an absolute constant. Under this condition, we obtain using Lemma C.3 σr(t) ≥ σr(0)− tnrL(L+ 1)/4π, and so ∫
[0,π]
σr(t) dt ≥ ∫
[0,4πσr(0)/nrL(L+1)]
(σr(0)− ntrL(L+ 1)/4π) dt
= 2πσr(0)
2
nrL(L+ 1) .
We have σr(0) = nL/2− ψ(cos-1(2r2 − 1)), and using the estimate of Lemma C.20, we get
ψ(ν) ≤ nL 2 1 + Lν/2π 1 + Lν/π .
Together with the estimate cos-1(2r2 − 1) ≥ 2 √ 1− r2, we obtain
σr(0) = nL 2 − ψ(cos-1(2r2 − 1)) ≥ nL 2
( L √ 1− r2
π + 2L √ 1− r2
) ≥ nL
8 ,
where the final inequality requires the choice L ≥ π/2 √
1− r2. Thus, we have shown∫ [0,π] σr(t) dt ≥ πn 64r ,
as claimed.
Lemma C.3. There is an absolute constant 0 < K ≤ π/2 such that if L ≥ 3, one has for all r ∈ (0, 1):
(i) ψ ◦ fr − ψ ◦ gr ≥ 0 on [0, π];
(ii) (ψ ◦ fr − ψ ◦ gr)′(0) = −nrL(L+ 1)/4π;
(iii) ψ ◦ fr − ψ ◦ gr is convex on [0,K].
Proof. In this proof, we will make use of basic results on the skeleton ψ, namely Lemmas E.5, C.17 and C.18 without making explicit reference to them. Property (i) follows from the fact that ψ is decreasing, cos-1 is decreasing, and the definitions of fr and gr. We note that fr is smooth on (0, π); to prove property (ii), it will suffice to show that fr admits a right derivative at 0 and π and apply the chain rule. We have if t ∈ (0, π)
f ′r(t) = r2 sin t√
1− (r2 cos t+ (1− r2))2 =
1√ 2 + r2(cos t− 1) r sin t√ 1− cos t
after some rearranging, and by periodicity and symmetry properties of fr, we have
lim t↘0 g′r(t) = lim t↗π f ′r(t) = 0.
We Taylor expand sin t(1−cos t)−1/2 in a neighborhood of zero to evaluate the derivatives there. We have sin t = t−t3/6+O(t5) and 1−cos t = t2/2(1−t2/2+O(t4)); by the binomial series, we have (1−cos t)−1/2 = √ 2/t(1+ t2/4+O(t4)), whence sin t(1−cos t)−1/2 = √ 2+ √
2t2/12+O(t4), and
lim t↘0
f ′r(t) = r.
Thus
(ψ ◦ fr − ψ ◦ gr)′(0) = ψ′(0)f ′r(0) = − nrL(L+ 1)
4π .
For property (iii), now consider t ∈ [0, π/2] when necessary. The chain rule gives
(ψ ◦ fr − ψ ◦ gr)′′ = [(ψ′ ◦ fr)f ′′r − (ψ′ ◦ gr)g′′r ] + [(ψ′′ ◦ fr)(f ′r)2 − (ψ′′ ◦ gr)(g′r)2], and we have if t ∈ (0, π)
f ′′r (t) = r4(1− cos t)
[ cos t− (r2 cos t+ (1− r2)) ]( 1− (r2 cos t+ (1− r2))2
)3/2 after some rearranging of the numerator. We have 1− cos t ≥ 0, and so the estimate r2 cos t+ (1− r2) ≥ cos t (with equality only at t = 0) yields f ′′r (t) ≤ 0 (with a strict inequality if 0 < t < π). By symmetry, this implies that g′′r ≥ 0, and using that ψ′ ≤ 0, we obtain
(ψ ◦ fr − ψ ◦ gr)′′ ≥ (ψ′′ ◦ fr)(f ′r)2 − (ψ′′ ◦ gr)(g′r)2. By symmetry, we have gr(t) = fr(π − t) on [0, π], and because fr is strictly concave we know as well that f ′r is strictly decreasing; it follows that f ′ r − g′r is also strictly decreasing, and its unique zero satisfies the equation 1− cos t 1 + cos t = 2− r2(1 + cos t) 2− r2(1− cos t) .
Noting that t = π/2 satisfies this equation, we conclude that f ′r ≥ g′r on [0, π/2], so that on this interval we have (ψ ◦ fr − ψ ◦ gr)′′ ≥ (g′r)2 ((ψ′′ ◦ fr)− (ψ′′ ◦ gr)) . By Lemma C.19, if L ≥ 3 there is an absolute constant K > 0 such that ... ψ ≤ 0 on [0,K]. The previous bound then yields (ψ ◦ fr − ψ ◦ gr)′′ ≥ 0,
as claimed.

heading: C.2 AUXILIARY RESULTS

heading: C.2.1 GEOMETRIC RESULTS

text:
Lemma C.4. LetM be a complete Riemannian submanifold of the unit sphere Sn0−1 (with respect to the spherical metric induced by the euclidean metric on Rn0 ) with finitely many connected components K. If d0 = 1, assume moreover that each connected component ofM is a smooth regular curve. Then for every 0 < ε ≤ 1, there is a ε-net forM in the euclidean metric ‖ · ‖2 having cardinality no larger than (CM/ε)d0 , where CM ≥ 1 is a constant depending only on the diameters supx,x′∈Mi distMi(x,x
′) and, when d0 ≥ 2, additionally on the extremal Ricci curvatures ofMi. Moreover, these nets have the property that if x ∈ M is given, there is a point in the net x̄ within euclidean distance ε of x such that x̄ lies in the same connected component ofM as x.
Proof. Consider a fixed connected componentMi with i ∈ [K]. We write the Riemannian distance of Mi as distMi ; because Mi is a Riemannian submanifold of Rn0 , we have distMi(x,y) ≥ ‖x − y‖2 for every x,y in Mi. Because distMi(x,y) ≥ ‖x − y‖2, it suffices to estimate the covering number in terms of the Riemannian distance. We will consider distinctly the cases d0 = 1 and d0 ≥ 2, starting with d0 = 1. When d0 = 1, we have assumed that Mi are regular curves, so it is without loss of generality to assume they are moreover unit-speed curves parameterized by arc length, with lengths len(Mi). It follows that we can obtain an ε-net forMi in terms of distMi having cardinality at most len(Mi)/ε when 0 < ε ≤ 1, and by the submanifold property these sets also constitute ε-nets forMi in terms of the `2 distance. Covering each connected component Mi in this way gives a ε-net for M by taking the union of each connected component’s net.
When d0 ≥ 2, we make use of standard results relating the covering number to the curvature and diameter ofM. Let diam(Mi) = supx,x′∈Mi distMi(x,x
′), and let Rici denote the Ricci curvature tensor ofMi (recall that we assume the metric onM is the one induced by the euclidean metric). Then becauseM is compact, (1) maxi∈[K] diam(Mi) < +∞; and (2) because Rici is moreover continuous, there are constants ki > 0 such that Rici ≥ −(d0 − 1)ki for each i ∈ [K]. Applying Lemma C.5, it follows that for any ε > 0, there is a ε-net forMi in terms of distMi with cardinality no larger than (CMi/ε) d0 , where CMi . diam(Mi)e2 diam(Mi) √ ki .
Thus, for any i ∈ [K], any d0 ≥ 1 and any 0 < ε ≤ 1, we can conclude that there is a ε-net forMi in the euclidean metric having cardinality no larger than (CMi/ε) d0 , where
CMi = { len(Mi) d0 = 1 16 diam(Mi)e2 diam(Mi) √ ki d0 ≥ 2.
Taking the union of these nets and applying Lemma G.10 for simplicity, we conclude that for any d0 ≥ 1 and any 0 < ε ≤ 1, there is a ε-net for M in the euclidean metric having cardinality no larger than (CM/ε)d0 , where
CM =
{ 1 + ∑K i=1 len(Mi) d0 = 1
1 + 16 ∑K i=1 diam(Mi)e2 diam(Mi) √ ki d0 ≥ 2.
The additional property claimed is satisfied by our construction of the nets.
Lemma C.5. Given k > 0 and integer d ≥ 2, suppose thatM is a d-dimensional complete Riemannian manifold with Ricci curvature tensor satisfying Ric ≥ −(d − 1)k. Then for any r, ε > 0
and any p ∈ M, there exists an ε-net (measured in the Riemannian distance distM) of the metric ball {x ∈ M | distM(p,x) ≤ r} with cardinality at most (CM/ε)d, where CM > 0 is a constant depending only on k and r.
Proof. The proof is essentially an application of (Zhu, 1997, Lemma 3.6) together with some calculations on volumes of geodesic balls in hyperbolic space that we record here for completeness, although they are classical. For any r > 0 and any p ∈M, write
Br(p) = {x ∈M | distM(p,x) ≤ r}. Fix p ∈ M and r, ε > 0. The hypotheses of the lemma make (Zhu, 1997, Lemma 3.6) applicable, whence
inf card(S) ∣∣∣∣∣∣ S ⊂ Br(p), Br(p) ⊂ ⋃ p′∈S Bε(p ′)  ≤ vol(Bk(2r))vol(Bk(ε/4)) , where card(S) denotes the cardinality of a set S, and for all ε > 0, vol(Bk(ε)) denotes the volume of a geodesic ball of radius r in the d-dimensional simply-connected hyperbolic space of constant sectional curvature −k; these spaces are homogeneous and isotropic so the base point does not matter (c.f. (Lee, 2018, Proposition 3.9)). In particular, we can calculate these volumes in any model of hyperbolic space and anchored at any base point; we choose the Poincaré ball model and the base point 0, where the maximal unit-speed geodesics take the simple form
γ(t) = k−1/2v tanh
√ kt
2 for v ∈ Sd and t ∈ R (Lee, 2018, Theorem 3.7, Proposition 5.28). Integrating the volume form in coordinates, we then get for any ε > 0
vol(Bk(ε)) = ∫ (k−1/2 tanh √ kε/2))Bd ( 2/k 1/k − ‖x‖22 )d dx
= k−d/2 ∫
(tanh √ kε/2)Bd
( 2
1− ‖x‖22
)d dx
where the second line changes coordinates x 7→ k−1/2x. Changing to polar coordinates in the last expression, we get
vol(Bk(ε)) = k−d/2 vol(Sd−1) ∫
[0,tanh √ kε/2]
xd−1 ( 2
1− x2
)d dx,
and then changing coordinates x 7→ tanhx, we obtain after applying several trigonometric identities vol(Bk(ε)) = k−d/2 vol(Sd−1) ∫
[0, √ kε/2]
2 sinhd−1(2x) dx
= k−d/2 vol(Sd−1) ∫
[0, √ kε]
sinhd−1(x) dx,
whence vol(Bk(2r))
vol(Bk(ε/4)) =
∫ [0,2r √ k]
sinhd−1(x) dx∫ [0,ε √ k/4] sinhd−1(x) dx .
We have bounds x ≤ sinhx ≤ xex for nonnegative x,9 which gives after integration
vol(Bk(2r))
vol(Bk(ε/4)) ≤
∫ [0,2r
√ k] xd−1e(d−1)x dx∫
[0,ε √ k/4] xd−1 dx ≤ d (2r √ k)d ∫ [0,1] xd−1e2r √ k(d−1)x dx
(ε √ k/4)d
≤
( 16re2r √ k
ε
)d ,
9The lower bound is implied by coshx ≥ 1; the upper bound follows from writing sinhx = 0.5ex(1 − e−2x) and using e−x ≥ 1− x.
where in the second line we change coordinates x 7→ (2r √ k)x, and then use L∞ control of the (monotone increasing) integrand in the second line to move to the expression in the third line.
Remark C.6. The constant CM in Lemma C.5 can be sharpened if more is known about the curvature ofM: if Ric ≥ 0, the exponential dependence on curvature and diameter can be removed (intuitively, taking k ↘ 0 “recovers” this from the proved result), and if Ric > 0, the dependence on diameter can be completely removed using Myers’ theorem (Zhu, 1997, Theorem 3.4(1)).
Lemma C.7. For any x,x′, x̄, x̄′ in Sn0−1, one has
|∠(x,x′)− ∠(x̄, x̄′)| ≤ √ 2|‖x− x′‖2 − ‖x̄− x̄′‖2|.
Proof. Writing∠(x,x′) = cos-1〈x,x′〉 = cos-1(1−(1/2)‖x−x′‖22), consider the function f(x) = cos-1(1− (1/2)x2) for x ∈ [− √ 2, √ 2], which is differentiable except possibly at 0. We calculate
f ′(x) = x√
1− (1− 12x2)2 = signx√ 1− 14x2 ,
and taking limits at 0 shows that f admits left and right derivatives on all of [− √ 2, √
2]. f ′ is evensymmetric, so by checking values at 0 and √ 2 we conclude that |f ′| ≤
√ 2, which shows that f is√
2-Lipschitz. The claim follows.
Lemma C.8. Let d0 = 1. Choose L so that L ≥ Kκ2Cλ, where κ and Cλ are respectively the curvature and global regularity constants defined in Section 2.1, and K,K ′ > 0 are absolute constants. Then
sup x∈M± ∫ M
dµ∞(x′) (1 + (L/π)∠(x,x′))2 ≤ Cρmax(len(M+) + len(M−)) L ,
where C is an absolute constant andM± denotes eitherM+ orM−.
Proof. Recall that γ+,γ− denote unit-speed curves parameterized with respect to arc length whose images areM+,M−. For convenience, define g(ν) = 1/(1 + Lν/π). We have
sup x∈M± ∫ M (g(∠(x,x′))) 2 dµ∞(x′) ≤ sup x∈M±
∫ M+ (g(∠(x,x′)))2 dµ∞+ (x ′)
+ sup x∈M± ∫ M− (g(∠(x,x′)))2 dµ∞− (x ′).
(C.6)
First, we note that |g| is strictly decreasing. We claim that for any x ∈ M−, there is a x? ∈ M+ such that ∠(x?,x′) ≤ ∠(x,x′) for all x′ ∈M+; it is easy to see this is the case by choosing x? to achieve the minimum in minx′∈M+ ∠(x,x
′) and arguing by contradiction. By monotonicity of the integral, this implies
sup x∈M± ∫ M+ (g(∠(x,x′))) 2 dµ∞+ (x ′) ≤ sup x∈M+ ∫ M+ (g(∠(x,x′))) 2 dµ∞+ (x ′), (C.7)
and similarly for the term involving integration overM−. Therefore
sup x∈M± ∫ M (g(∠(x,x′))) 2 dµ∞(x′) ≤ sup x∈M+
∫ M+ (g(∠(x,x′)))2 dµ∞+ (x ′)
+ sup x∈M− ∫ M− (g(∠(x,x′)))2 dµ∞− (x ′),
(C.8)
and it suffices to analyze these two terms. We bound the first term, since the second can be bounded by an identical argument. By compactness, the supremum in this term is attained at some x ∈M+. Taking t such that γ+(t) = x, we can write
sup x∈M+ ∫ M+ g(∠(x,x′))2 dµ∞+ (x ′) ≤ ρmax ∫ S+ 0 g(∠(γ(t), γ(s)))2 ds. (C.9)
We split the interval [0, S+] into two disjoint sub-intervals [0, S+]∩ [t−Kτ/ √ L, t+Kτ/ √ L] and
[0, S+] \ [t − Kτ/ √ L, t + Kτ/ √ L], corresponding to “large scale” and “small scale” behavior, where Kλ is the global regularity constant defined in (A.2). If we now assume 1√L ≤ cλ κ , then from (A.2) we obtain
∠(x,x′) ≤ 1√ L ⇒ distM(x,x′) ≤ Kλ√ L
and hence distM(x,x
′) > Kλ√ L ⇒ ∠(x,x′) > 1√ L .
From the definition of g it follows that
g( 1√ L ) = 1 1 + √ L/π ≤ π√ L .
Since |g| is a monotonically decreasing function we can bound the second integral in (C.9), obtaining∫ s∈[0,S+]\[t∗−
Kλ√ L ,t∗+ Kλ√ L ]
(g(∠(γ(s),γ(t∗)))2 ds ≤ len(M+)C ′/L. (C.10)
We next consider the remaining interval of integration in (C.9). Defining
S++ = min { Kλ√ L , S+ − t∗ } , S−+ = min { Kλ√ L , t∗ } ,
and ν±(s) = ∠(γ+(t∗ ± s),γ+(t∗)), the integral of interest can be written as
ρmax ∫ s∈[0,S+]∩[t∗−Kτ√
L ,t∗+Kτ√ L ]
(g(∠(γ(s),γ(t∗))))2 ds = ρmax S++∫ s=0 (g(ν+(s))) 2 ds
+ρmax S−+∫ s=0 (g(ν−(s))) 2 ds.
(C.11)
It will be sufficient to consider the first integral here since the second one can be bounded in an identical fashion. We aim to show that the integral above is not too large. This will be the case if ν+(s) stays very small for a large range of values of s. To show that this is does not occur, we will use our bounds on the curvature of M to bound ν+(s) uniformly from below, which will in turn provide an upper bound on the integral. We will require an application of Lemma C.9, which will be applicable if S++ ≤ πκ . If L ≥ κ2K2τ π2 we have
S++ ≤ Kλ√ L ≤ π κ .
It follows immediately that Lemma C.9 applies to any restriction of γ+ of length no larger than πκ . Next define by γ̃ : [0, S++ ]→ Sn0−1 an unit-speed arc of curvature κ, and ν̃(s) = ∠(γ̃(0), γ̃(s)). We claim that ∀s ∈ [0, S++ ] : ν+(s) ≥ ν̃(s). (C.12) The proof is by contradiction. Assume there is some r such that
ν+(r) < ν̃(r). (C.13)
Now define by γr : [0, r]→ Sn0−1 a restriction of γ+ such that γr(0) = γ+(t∗),γr(s) = γ+(t∗ + s), by γ̆r an arc with curvature κ and the same endpoints as γr, and by γ̃r a restriction of γ̃ with
len(γ̃r) = len(γr) = r.
Note that ∠(γ̃r(0), γ̃r(s)) = ν̃(r). However, an application of Lemma C.9 gives
len(γr) ≤ len(γ̆r) < len(γ̃r)
where the second inequality is because γ̆r and γ̃r have identical curvature at every point, and by assumption (C.13) the endpoints of γ̃r are a greater geodesic (and hence euclidean) distance from each other than the endpoints of γ̆r (which are a distance ν+(r) apart). This inequality contradicts the equality above it, and we conclude that no such r exists, and (C.12) holds.
We have that |g| is a monotonically decreasing function, hence we can write for the first integral in (C.11)
S++∫ s=0 (g(ν+(s))) 2 ds ≤ S++∫ s=0 (g(ν̃(s))) 2 ds.
We now bound this integral. Since γ̃ is an arc with curvature κ, from the proof of Lemma C.3 we have that ν̃ is concave, and since ν̃(0) = 0 we can write
ν̃(s) ≥ ν̃(S++)
S++ s,
and since |g| is monotonically decreasing S++∫ s=0 (g(ν̃(s))) 2 ds ≤ S++∫ s=0 ( g( ν̃(S++) S++ s) )2 ds = S++ ν̃(S++) ν̃(S++)∫ s=0 (g(s)) 2 ds
= S++
ν̃(S++)
ν̃(S++)
1 + Lν̃(S++)/π ≤π S++ Lν̃(S++)
where we used the definition of g. It remains to show that S++ and ν̃(S + +) are close. Since γ̃ is an arc with curvature κ and length S++ , if we additionally assume L ≥ Kκ2Cλ for some K chosen so that κ‖γ̃(0)−γ̃(S++)‖2
2 ≤ κS++
2 ≤ κCλ 2 √ L ≤ 12 , we obtain∥∥γ̃(0)− γ̃(S++)∥∥2 ≤S++ = 2
κ sin−1
( κ ∥∥γ̃(0)− γ̃(S++)∥∥2
2
)
≤ ∥∥γ̃(0)− γ̃(S++)∥∥2 + κ24 ∥∥γ̃(0)− γ̃(S++)∥∥32∣∣∥∥γ̃(0)− γ̃(S++)∥∥2 − S++ ∣∣ ≤ κ24 ∥∥γ̃(0)− γ̃(S++)∥∥32 ≤ κ24 (S++)3 ≤ κ24 K2L S++
where in the first line we used sin−1(x) ≤ x+ x3 for x. Since∥∥γ̃(0)− γ̃(S++)∥∥2 ≤ ∠(γ̃(0), γ̃(S++)) = ν̃(S++) ≤ S++ we obtain ∣∣ν̃(S++)− S++ ∣∣ ≤ κ24 K2λL S++ and hence
S++ ν̃(S++) ≤ S++
S++ − κ 2
4 K2λ L S + +
= 1
1− κ24 K2λ L
.
We now choose L ≥ Kκ2K2τ for some K, so that the above term is smaller than 2. We therefore have
Si∫ s=0 (g(ν̃(s))) 2 ds ≤ C/L
for some C. We can bound the second integral in (C.11) in an identical fashion. Combining this result with (C.10) and recalling (C.9), we obtain
sup x∈M+ ∫ M+ (g(∠(x,x′))) 2 dµ∞(x′) ≤ C ′ρmax(len(M+) + len(M−))/L
for some constant, which completes the proof.
Lemma C.9. Given a smooth, simple open curve in Rn of length S with unit-speed parametrization γ : [0, S]→ Rn such that for some κ > 0
1. ‖γ̈‖2 ≤ κ
2. S ≤ πκ
define by γ̆ an arc of any circle of radius 1κ such that γ̆(0) = γ(0), γ̆(S̆) = γ(S), S̆ ≤ π κ 10 . We then have
S ≤ S̆
Proof. This result is a generalization of a well known comparison theorem of Schur’s to higher dimensions following the proof in (Sullivan, 2008), where we additionally specialize to the case where one of the curves is an arc.
Given a curve γ satisfying the conditions of the lemma, we first consider an arc γ̃ of a circle of radius 1κ and length S, with a unit-speed parametrization. At the midpoint of this arc, the tangent vector γ̃′(S2 ) is parallel to γ̃(S)− γ̃(0), hence
‖γ̃(S)− γ̃(0)‖2 = 〈 γ̃′( S
2 ), γ̃(S)− γ̃(0)
〉 = 〈 γ̃′( S
2 ), S∫ 0 γ̃′(t)dt 〉 .
Similarly, for the curve γ we have ‖γ(S)− γ(0)‖2 ≥ 〈 γ′( S
2 ),γ(S)− γ(0)
〉 = 〈 γ′( S
2 ), S∫ 0 γ′(t)dt 〉 .
Denoting the angle between tangent vectors 〈γ′(a),γ′(b)〉 = cos θ(a, b), we use the fact that for any smooth curve with unit-speed parametrization ‖γ′′(t)‖2 = ∣∣dθ ds (t, s)s=t ∣∣ .= |θ′(t)|. This gives for any t ∈ [0, S/2]〈 γ′( S
2 ),γ′(
S 2 + t)
〉 = cos  S 2 +t∫ S 2 θ′(t′)dt′
≥ cos  S 2 +t∫ S 2 |θ′(t′)| dt′  = cos  S 2 +t∫ S 2 ‖γ′′(t)‖2 dt ′
≥ cos κ S 2 +t∫ S 2 dt′  = cos  S 2 +t∫ S 2 ‖γ̃′′(t)‖2 dt ′  = 〈γ̃′(S 2 ), γ̃′( S 2 + t) 〉 where we have used monotonicity of cos over the relevant range which is ensured by assumption 2, and a similar argument follows for t ∈ (0,−S/2]. Combining these inequalities gives
‖γ(S)− γ(0)‖2 ≥ ‖γ̃(S)− γ̃(0)‖2 . We have shown that, unsurprisingly, if the curvature of γ is bounded and it is not too long, then the distance between its endpoints is greater than that of a curve of equal length but larger curvature - namely the arc γ̃. We now consider the arc γ̆ defined in the lemma statement. If S > S̆, due to assumption 2 this would imply
‖γ̃(S)− γ̃(0)‖2 > ∥∥∥γ̆(S̆)− γ̆(0)∥∥∥
2 = ‖γ(S)− γ(0)‖2
contradicting the inequality proved above. It follows that S ≤ S̆. 10For any circle and choice of endpoints there will be two such arcs, and the last condition implies that we choose the shorter of the two.

heading: C.2.2 ANALYSIS OF THE SKELETON

text:
Notation. Define ϕ(0) = Id, and for ` ∈ N define ϕ(`) as the `-fold composition of ϕ with itself, where ϕ(ν) = cos-1 (( 1− π−1ν ) cos ν + π−1 sin ν )
is the heuristic angle evolution function. We will make use of basic properties of this function such as smoothness (established in Lemma E.5) below. In this section, we will study the skeleton
ψ1(ν) = n
2 L−1∑ `=0 cosϕ(`)(ν) L−1∏ `′=` ( 1− π−1ϕ(` ′)(ν) ) , ν ∈ [0, π],
where we have not included the additive factor cosϕ(L)(ν), as it is easily removed along the lines of Theorem B.2. We define
ξ(`)(ν) = L−1∏ `′=` ( 1− π−1ϕ(` ′)(ν) ) , ` = 0, . . . , L− 1,
so that
ψ1(ν) = n
2 L−1∑ `=0 cosϕ(`)(ν)ξ(`)(ν). (C.14)
We will also establish a convenient approximation to the skeleton. Define
ψ(ν) = n
2 L−1∑ `=0 ξ(`)(ν).
Lemma C.10 implies that ψ is convex; it is less trivial to obtain the same for ψ1. We will prove several estimates below for the terms ξ(`) and their derivatives that can be used to immediately obtain useful estimates for ψ and its derivatives.
Lemma C.10. For each ` = 0, 1, . . . , L, the functions ϕ(`) are nonnegative, strictly increasing, and concave (positive and strictly concave on (0, π)); if 0 ≤ ` < L, the functions ξ(`), are nonnegative, strictly decreasing, and convex (positive and strictly convex on (0, π)).
Proof. These claims are a consequence of some general facts for smooth functions that we articulate here so that we can rely on them often in the sequel. First, we have for any smooth function f : (0, π)→ R
(f ◦ f)′ = (f ′ ◦ f)f ′, (f ◦ f)′′ = (f ′ ◦ f)f ′′ + (f ′)2(f ′′ ◦ f).
These equations show that if f > 0, f ′ > 0, and f ′′ < 0, then f ◦ f also satisfies these three properties. Lemma E.5 shows that ϕ satisfies these three properties on (0, π); we conclude from the mean value theorem and a simple induction the same for ϕ(`), as claimed. Meanwhile, if f, g are smooth real-valued functions on (0, π), we have
(fg)′ = f ′g + g′f,
(fg)′′ = f ′′g + g′′f + 2f ′g′.
Thus, if f and g are both positive, strictly decreasing, strictly convex functions on (0, π), then fg also satisfies these three properties. Lemma E.5 implies that 0 < 1−π−1ϕ(`) < 1 on (0, π), and the first and second derivatives are scaled and negated versions of those of ϕ(`); we conclude by another induction that the same three properties apply to the functions ξ(`).
Lemma C.11. There is an absolute constant C > 0 such that if L ≥ 12 and n ≥ L, then one has
‖ψ1 − ψ‖L∞ ≤ Cn
L .
Proof. We have from the triangle inequality
‖ψ1 − ψ‖L∞ ≤ sup ν∈[0,π]
( n
2 L−1∑ `=0
∣∣∣cosϕ(`)(ν)− 1∣∣∣ξ(`)(ν))
≤ n 2 L−1∑ `=0 sup ν∈[0,π] (∣∣∣cosϕ(`)(ν)− 1∣∣∣ξ(`)(ν)) , where we use Lemma C.10 to take ξ(`) outside the absolute value. Notice that (cosϕ(`) − 1)ξ(`) ≤ 0, so to control the L∞ norm of this term it suffices to bound it from below. We will show the monotonicity property
(cosϕ(`) − 1)ξ(`) − (cosϕ(`+1) − 1)ξ(`+1) ≥ 0, (C.15) from which it follows
‖ψ1 − ψ‖L∞ ≤ nL
2 sup
ν∈[0,π] ∣∣∣cosϕ(L−1)(ν)− 1∣∣∣, using also ξ(L−1)(ν) ≤ 1. Since cosx ≥ 1− (1/2)x2, and since Lemma C.12 gives that ϕ(L−1) ≤ C/(L− 1) (and also estimates the constant), we have as soon as L ≥ 1 + C/ √ 2
‖ψ1 − ψ‖L∞ ≤ C2nL
4(L− 1)2
which gives the claim provided L ≥ 2 and n ≥ L. So to conclude, we need only establish (C.15). To this end, write the LHS of (C.15) as
(cosϕ(`)−1)ξ(`)−(cosϕ(`+1)−1)ξ(`+1) = [ (cosϕ(`) − cosϕ(`+1))− ϕ (`)
π (cosϕ(`) − 1)
] ξ(`+1)
to notice that it suffices to prove nonnegativity of the bracketed quantity. In addition, since ` ≥ 0 and ϕ(ν) ≤ ν by Lemma E.5, we can instead prove the inequality
(cosx− cosϕ(x))− x π (cosx− 1) ≥ 0
for all x ∈ [0, π]. Using the closed-form expression for cosϕ(x) in Lemma E.2, we can plug into the previous inequality and cancel to get the equivalent inequality
x− sinx ≥ 0. But this is immediate from the concavity estimate sinx ≤ x, and (C.15) is proved.
Lemma C.12. If ` ∈ N0, one has the “fluid” estimate for the angle evolution function
ϕ(`)(ν) ≤ ν 1 + c`ν ,
where c > 0 is an absolute constant. In particular, if ` ∈ N one has ϕ(`) ≤ 1/c`.
Proof. The second claim follows from the first claim and 1 + c`ν ≥ c`ν, so we will focus on establishing the first estimate. The proof is by induction on ` ∈ N, since the case of ` = 0 is immediate. By Lemma E.5, there is a constant c1 > 0 such that ϕ(ν) ≤ ν(1 − c1ν), and using the numerical inequality x(1− x) ≤ x(1 + x)−1, valid for x ≥ 0, we get
ϕ(ν) ≤ ν 1 + c1ν , (C.16)
which establishes the claim in the case ` = 1. Assuming the claim holds for `− 1, we calculate
ϕ(`)(ν) ≤ ϕ (`−1)(ν)
1 + c1ϕ(`−1)(ν) ≤
ν 1+c1(`−1)ν
1 + c1 ν 1+c1(`−1)ν ,
where the first inequality uses (C.16), and the second inequality uses the induction hypothesis and the relation x(1 + x)−1 = 1 − (1 + x)−1 to see that x 7→ x(1 + c1x)−1 is increasing. Clearing denominators in the numerator and denominator of the RHS of this last bound, we see that it is equal to ν/(1 + `ν/π), and the claim follows by induction.
Lemma C.13. If ` ∈ N0, the iterated angle evolution function satisfies the estimate
ϕ(`)(ν) ≥ ν 1 + `ν/π .
Proof. The proof is by induction on ` ∈ N, since the case ` = 0 is immediate. The case ` = 1 follows from Lemma C.14. Assuming the claim holds for `− 1, we calculate
ϕ(`)(ν) ≥ ϕ (`−1)(ν)
1 + ϕ(`−1)(ν)/π ≥
ν 1+(`−1)ν/π
1 + 1π ν 1+(`−1)ν/π ,
where the first inequality applies Lemma C.14, and the second uses the fact that the RHS of the bound in Lemma C.14 is strictly increasing and the induction hypothesis. Clearing denominators in the numerator and denominator of the RHS of this last bound, we see that it is equal to ν/(1+`ν/π), and the claim follows by induction.
Lemma C.14. It holds ϕ(ν) ≥ ν
1 + ν/π .
Proof. After some rearranging using Lemma E.2, it suffices to prove( 1− ν
π
) cos ν + sin ν
π ≤ cos
( πν
π + ν
) . (C.17)
Using Lemma E.5, we see that both the LHS and RHS of this bound are nonincreasing. We will prove the estimate in three stages, using “small angle”, “large angle”, and “intermediate angle” estimates of the quantities on both sides of (C.17). Since πν/(π + ν) ∈ [0, π/2], we can use standard estimates for cos to get RHS estimates
cos
( πν
π + ν
) ≥ 1− 1
2
( πν
π + ν
)2 (C.18)
and
cos
( πν
π + ν ) ≥ π − ν π + ν . (C.19)
As for the LHS, we can obtain an estimate near ν = π in a straightforward way. Transforming the domain by ν 7→ π − ν, it suffices to get estimates on sin ν − ν cos ν near ν = 0, then divide by π. Using cos ν ≥ 1− (1/2)ν2 and sin ν ≤ ν, it follows that sin ν − ν cos ν ≤ (1/2)ν3. We conclude(
1− ν π
) cos ν + sin ν
π ≤ 1 2π (π − ν)3. (C.20)
We will develop a second-order approximation to the LHS near 0 for the small-angle estimates. The first, second, and third derivatives of the LHS are (1 − ν/π) sin ν, (1/π) sin ν − (1 − ν/π) cos ν, and (2/π) cos ν + (1 − ν/π) sin ν, respectively. To bound the third derivative, we will use the estimate cos ν ≤ 1 − ν2/3 on [0, π/2]. To prove this, note that Taylor’s formula implies the bound cos ν ≤ 1 − ν2/3 on [0, cos-1(2/3)]; because cos is concave on [0, π/2], we also have the tangent line bound cos(ν) ≤ −ν √ 5/3 + (2/3 + √ 5 cos-1(2/3)/3) on [0, π/2]. We can then solve for the zeros of the quadratic polynomial 1 − ν2/3 + ( √ 5/3)ν − (2/3 + √
5 cos-1(2/3)/3); a numerical evaluation shows that both roots are real and outside the interval [cos-1(2/3), π/2]. Since the tangent line touches the graph of cos at ν = cos-1(2/3), this proves that cos ν ≤ 1 − ν2/3 on [0, π/2]. We can therefore write
2 cos ν + (π − ν) sin ν ≤ 2(1− ν2/3) + ν(π − ν), ν ∈ [0, π/2].
The RHS of this inequality is a concave quadratic; we calculate its maximum analytically as 2 + 3π2/20. Meanwhile, if ν ∈ [π/2, π], we have 2 cos ν ≤ 0, and (π − ν) sin ν ≤ π/2. We conclude that (2/π) cos ν+(1−ν/π) sin ν ≤ 2+3π2/20 on [0, π]. Writing c = 1/(3π)+π/40, this implies an estimate (
1− ν π
) cos ν + sin ν
π ≤ 1− ν
2
2 + cν3. (C.21)
Finally, we will need some estimates for interpolating the small and large angle regimes. We note that the second derivative (1/π) sin ν − (1 − ν/π) cos ν of the LHS of (C.17) is nonnegative if ν ≥ π/2, because cos ≥ 0 here; meanwhile, the third derivative (2/π) cos ν + (1 − ν/π) sin ν of the LHS of (C.17) is nonnegative if 0 ≤ ν ≤ π/2, since cos ≥ 0 here, and it follows that the second derivative is increasing on [0, π/2]. Checking numerically that the value of the second derivative at 1.42 is positive, we conclude that the LHS of (C.17) is convex on [1.42, π]. In addition, we use calculus to evaluate the first and second derivative of the RHS of (C.18) as −νπ3/(π + ν)3 and −π3(π− 2ν)/(π+ ν)4, respectively; this shows that the RHS of (C.18) is convex for ν ≥ π/2, and concave for ν ≤ π/2. Taking a tangent line to the graph of the RHS of (C.18) at π/2, it follows that the function
g(x) = { 1− (π2/2)ν2/(π + ν)2 x ≤ π/2 −(4π/27)ν + (1 + π2/54) x ≥ π/2 (C.22)
is a concave lower bound for the RHS of (C.18) on [0, π].
We proceed to using the estimates developed in the previous paragraph to prove (C.17). We first argue that for ν in a neighborhood of 0, we have
1− ν2/2 + cν3 ≤ 1− (π2/2)ν2/(π + ν)2, which will in turn prove (C.17) in the same neighborhood. Cancelling and rearranging, it is equivalent to show (2/π − 2c)− (4c/π − 1/π2)ν − (2c/π2)ν2 ≥ 0. The LHS is a concave quadratic, with value 2/π − 2c > 0 at 0; we calculate its two distinct roots numerically as lying in the intervals [−5.1,−5] and [1.42, 1.43], respectively. It follows that (C.17) holds for ν ∈ [0, 1.42]. Next, we argue that for ν in a neighborhood of π, we have
1 2π (π − ν)3 ≤ π − ν π + ν ,
which will in turn prove (C.17) in the same neighborhood. Transforming with ν 7→ π − ν and rearranging, it is equivalent to show ν2(2π− ν) ≤ 4π2 in a neighborhood of 0. The LHS of this last inequality is 0 at 0, and nonnegative on [0, π]; its first and second derivatives are ν(4π − 3ν) and 4π − 6ν, respectively, which shows that it is a strictly increasing function of ν on [0, π]. Verifying numerically the three distinct real roots of ν3 − 2πν2 + 1 = 0 and transferring the result back via another transformation ν 7→ π − ν, we conclude that (C.17) holds on [π − 1.1, π]. To obtain that (C.17) holds on [1.42, π−1.1], we use that the function g defined in (C.22) is a concave lower bound for the RHS of (C.18), so that it suffices to show that the LHS of (C.17) is upper bounded by g on [1.42, π − 1.1]. The LHS of (C.17) is convex on [1.42, π], so it follows that it is sufficient to show that the values of the LHS of (C.17) at 1.42 and at π − 1.1 are upper bounded by those of g at the same points. Confirming this numerically, we can conclude the proof.
Lemma C.15. If ` ∈ N0, one has ∣∣∣ϕ̇(`)(ν)∣∣∣ ≤ 1 1 + (c/2)`ν ,
where c > 0 is the absolute constant also appearing in Lemma E.5 (property 4), and in particular c/2 is equal to the absolute constant appearing in Lemma C.12. In particular, if ` ∈ N and ν ∈ [0, π] we have the estimate ∣∣∣νϕ̇(`)(ν)∣∣∣ ≤ 2
c` .
Proof. The case of ` = 0 follows directly (as an equality) from ϕ(0)(ν) = ν. Now we assume ` ∈ N. Smoothness of ϕ(`) follows from Lemma E.5. Applying the chain rule and an induction, we have
ϕ̇(`) = ( ϕ̇ ◦ ϕ(`−1) ) ϕ̇(`−1) = `−1∏ `′=0 ϕ̇ ◦ ϕ(` ′), (C.23)
and applying the chain rule also gives ϕ̈(`) = ( ϕ̇(`−1) )2 ( ϕ̈ ◦ ϕ(`−1) ) + ( ϕ̈(`−1) )( ϕ̇ ◦ ϕ(`−1) ) . (C.24)
By Lemma E.5, we have ϕ̇ > 0 on [0, π]), and the formula (C.23) then implies that ϕ̇(`) > 0 on [0, π]) as well. Considering only angles in this half-open interval and distributing, it follows
ϕ̈(`)( ϕ̇(`) )2 = ϕ̈ ◦ ϕ(`−1)( ϕ̇ ◦ ϕ(`−1) )2 + 1ϕ̇ ◦ ϕ(`−1) ϕ̈(`−1)(ϕ̇(`−1))2 = ϕ̈
ϕ̇2 ◦ ϕ(`−1) + 1 ϕ̇ ◦ ϕ(`−1) ϕ̈(`−1)( ϕ̇(`−1) )2 . Applying an induction using the previous formula and distributing in the result, we obtain
ϕ̈(`)( ϕ̇(`) )2 = `−1∑ `′=0 ( 1∏`−1 `′′=`′+1 ϕ̇ ◦ ϕ(` ′′) ) ϕ̈ ϕ̇2 ◦ ϕ(` ′). (C.25)
By Lemma E.5, we have 0 < ϕ̇ ≤ 1 on [0, π) and ϕ̈ ≤ 0. Thus
− ϕ̈ (`)( ϕ̇(`) )2 ≥ − `−1∑
`′=0
ϕ̈ ◦ ϕ(` ′).
When `′ > 0, we have ϕ(` ′) ≤ π/2, and by Lemma E.5, we have ϕ̈ ≤ −c < 0 on [0, π/2]; thus, −ϕ̈ ◦ ϕ(`′) ≥ c if `′ > 0. When `′ = 0, we can use the fact that ϕ̈ ≤ 0 on [0, π] to get a bound ϕ̈ ≤ −c1[0,π/2]. We conclude
− ϕ̈ (`)( ϕ̇(`) )2 ≥ c(`− 1) + c1[0,π/2]. (C.26)
Next, we notice using the chain rule that( 1
ϕ̇(`)
)′ = − ϕ̈
(`)( ϕ̇(`) )2 , and using (C.23) and Lemma E.5, we have that ϕ̇(`)(0) = 1. For any ν ∈ [0, π), we integrate both sides of (C.26) from 0 to ν to obtain using the fundamental theorem of calculus
1
ϕ̇(`)(ν) − 1 ≥ c(`− 1)ν + c ∫ ν 0 1[0,π/2](t) dt
= c(`− 1)ν + cmin{ν, π/2}
≥ c`ν 2 ,
where in the final inequality we use the inequality min{ν, π/2} ≥ ν/2, valid for ν ∈ [0, π]. Rearranging, we conclude for any 0 ≤ ν < π
ϕ̇(`)(ν) ≤ 1 1 + (c/2)`ν ,
and noting that the LHS of this bound is equal to 0 at ν = π and the RHS is positive, we conclude the claimed bound for every ν ∈ [0, π]. The second estimate claimed follows by multiplying this bound by ν on both sides, and using 1 + (c/2)`ν ≥ (c/2)`ν.
Lemma C.16. If ` ∈ N, one has∣∣∣ϕ̈(`)(ν)∣∣∣ ≤ C 1 + (c/8)`ν ( 1 + 1 (c/8)ν log (1 + (c/8)(`− 1)ν) ) ,
where C > 0 is an absolute constant, and c > 0 is the absolute constant also appearing in Lemma E.5 (property 4), and in particular c/2 is equal to the absolute constant appearing in Lemma C.12. If ν ∈ [0, π], the RHS of this upper bound is a decreasing function of ν, and moreover we have the estimates
|ϕ̈(`)| ≤ C`, ∣∣∣ν2ϕ̈(`)(ν)∣∣∣ ≤ Cπν
1 + (c/8)`ν
( 1 + 8 log `
cπ
) ≤ 8πC
c` +
64C log `
c2` .
Proof. Smoothness follows from Lemma E.5; we make use of some results from the proof of Lemma C.15, in particular (C.23) and (C.25). We treat the case of ` = 1 first. By Lemma E.5, we have |ϕ̈| ≤ C for an absolute constant C > 0, and since 1/(1 + (c/2)ν) ≥ 1/(3/2) = 2/3 by the numerical estimate of the absolute constant c > 0 in Lemma E.5, it follows
|ϕ̈(ν)| ≤ 3C/2 1 + (c/2)ν ,
which establishes the claim when ` = 1 (after worst-casing constants if necessary). Next, we assume ` > 1. Multiplying both sides of (C.25) by (ϕ̇(`))2 and cancelling using (C.23), we obtain
ϕ̈(`) = `−1∑ `′=0
∏`−1 `′′=0 ( ϕ̇ ◦ ϕ(`′′) )2 ∏`−1 `′′=`′+1 ϕ̇ ◦ ϕ(` ′′) ϕ̈ ◦ ϕ(`′)( ϕ̇ ◦ ϕ(`′)
)2 =
`−1∑ `′=0  `′−1∏ `′′=0 ( ϕ̇ ◦ ϕ(` ′′) )2( `−1∏ `′′=`′+1 ϕ̇ ◦ ϕ(` ′′) ) ϕ̈ ◦ ϕ(` ′)
= ϕ̇(`) `−1∑ `′=0 ϕ̇(` ′) ϕ̈ ◦ ϕ(` ′) ϕ̇ ◦ ϕ(`′) ,
(C.27)
where the last equality holds at least on [0, π), by Lemmas E.5 and C.15, and where empty products are defined to be 1. If `′ > 0, we have ϕ(`
′) ≤ π/2, and by Lemma E.5 we have that |ϕ̈| ≤ C and ϕ̇ ≥ c′ > 0 on [0, π/2] for absolute constants C,C ′ > 0. Separating the `′ = 0 summand, this gives a bound ∣∣∣ϕ̈(`)∣∣∣ ≤ C ( `−1∏
`′=1
ϕ̇ ◦ ϕ(` ′) ) + C
c′ ϕ̇(`) `−1∑ `′=1 ϕ̇(` ′). (C.28)
By Lemma C.15, we have ϕ̇(ν) ≤ 1/(1 + (c/2)ν), and by Lemma E.5, we have ϕ(ν) ≤ ν, hence ϕ(` ′)(ν) ≤ ν. Using concavity of ϕ, nonincreasingness of ϕ̇ and nondecreasingness of ϕ(`′) (which follow from Lemma E.5) and a simple re-indexing, we can write
`−1∏ `′=1 ϕ̇ ◦ ϕ(` ′)(ν) = `−2∏ `′=0 ϕ̇ ◦ ϕ(` ′+1)(ν) = `−2∏ `′=0 ϕ̇ ◦ ϕ(` ′) ◦ ϕ(ν)
≤ `−2∏ `′=0 ϕ̇(ϕ(` ′)(ν/2)) = ϕ̇(`−1)(ν/2)
≤ 1 1 + (c/4)(`− 1)ν
≤ 1 1 + (c/8)`ν
where the third-to-last line follows from (C.23), the second-to-last line follows from Lemma C.15, and the last line follows from the inequality ` − 1 ≥ `/2 if ` ≥ 2. Following on from (C.28), we conclude by an application of Lemma C.15
∣∣∣ϕ̈(`)(ν)∣∣∣ ≤ C 1 + (c/8)`ν + C/c′ 1 + (c/2)`ν `−1∑ `′=1
1
1 + (c/2)`′ν
≤ C c′
( 1
1 + (c/8)`ν `−1∑ `′=0
1
1 + (c/8)`′ν
) ,
where the last line simply worst-cases the constants. For any `′ ∈ N0, the function x 7→ 1/(1 + (c/8)`′x) is nonincreasing, so we can estimate the sum in the previous statement using an integral,
obtaining ∣∣∣ϕ̈(`)(ν)∣∣∣ ≤ C/c′ 1 + (c/8)`ν ( 1 + ∫ `−1 0
1
1 + cνx dx
)
≤ C/c ′
1 + (c/8)`ν
( 1 + 1
(c/8)ν log (1 + (c/8)(`− 1)ν) ) after evaluating the integral—we define the quantity inside the parentheses on the RHS of the final inequality to be `− 1 when ν = 0, which agrees with the integral representation in the previous line and with the unique continuous extension of the function on (0, π] to [0, π]—which establishes the first claim.
We now move on to the study of the bound we have derived. For decreasingness, we note that the functions
ν 7→ C/c ′
1 + (c/8)`ν , ν 7→ 1 + 1 (c/8)ν log (1 + (c/8)(`− 1)ν) , (C.29)
whose product is equal to our upper bound, are evidently both smooth nonnegative functions of ν at least on (0, π], so that by the product rule for differentiable functions it suffices to prove that these two functions are themselves decreasing functions of ν. The first function is evidently decreasing as an increasing affine reparameterization of ν 7→ 1/ν; for the second function, after multiplying by the constant ` − 1 and rescaling by a positive number (when ` = 1, the function is identically zero on (0, π], and the function’s continuous extension as defined above equals 0 at 0 as well), we observe that it suffices to prove that x 7→ x−1 log(1+x) is a decreasing function of ν on (0,∞). The derivative of this function is x 7→ (x− (1 + x) log(1 + x))/(x2(1 + x)), so it suffices to show that x− (1 + x) log(1 + x) ≤ 0. Noting that the function x 7→ x log x is convex (its second derivative is 1/x), it follows that x−(1+x) log(1+x) is concave as a sum of concave functions, and is therefore has its graph majorized by its supporting hyperplanes; its derivative is equal to − log(1 + x), which equals 0 at 0, and we therefore conclude from our previous reduction that the second function in (C.29) is decreasing, and that our composite upper bound is as well. For the remaining estimates, we use the concavity estimate log(1 + x) ≤ x to obtain from our previous result∣∣∣ϕ̈(`)(ν)∣∣∣ ≤ C`
1 + (c/8)`ν ≤ C`,
since the function x 7→ C/(1 + cx) is nonincreasing for any choice of the constants. Next, we use the expression we have derived in the first claim to obtain∣∣∣ν2ϕ̈(`)(ν)∣∣∣ ≤ Cν
1 + (c/8)`ν
( ν + 1
(c/8) log (1 + (c/8)(`− 1)ν)
) .
For any K > 0, the function x 7→ x/(1 + Kx) is nondecreasing, and using the numerical estimate π(c/8) < 1 that follows from Lemma E.5, we obtain in addition 1 + π(c/8)(`− 1) ≤ ` for ` ∈ N. Thus ∣∣∣ν2ϕ̈(`)(ν)∣∣∣ ≤ Cπ2
1 + (c/8)`π
( 1 + log `
cπ/8 ) ≤ 8πC
c` +
64C log `
c2` ,
as claimed.
Lemma C.17. One has for every ` ∈ {0, 1, . . . , L}
ϕ(`)(0) = 0; ϕ̇(`)(0) = 1; ϕ̈(`)(0) = − 2` 3π ,
and for every ` ∈ [L] ϕ̇(`)(π) = ϕ̈(`)(π) = 0.
Finally, we have ϕ̇(0)(π) = 1 and ϕ̈(0)(π) = 0.
Proof. The claims are consequences of Lemma E.5 when ` = 1, and of ϕ(0) = Id for smaller `; assume ` > 1 below. The claim for ϕ(`)(0) follows from the fact that ϕ(0) = 0 and induction. For the claim about ϕ̇(`)(0), we calculate using the chain rule
ϕ̇(`)(0) = ϕ̇(ϕ(`−1)(0))ϕ̇(`−1)(0)
= ϕ̇(0)ϕ̇(`−1)(0)
= ϕ̇(`−1)(0).
By induction and Lemma E.5, we obtain ϕ̇(`)(0) = 1. The claim about ϕ̇(`)(π) follows from the same argument. For the remaining claims about ϕ̈(`), we calculate using the chain rule
ϕ̈(`) = ( ϕ̇(`−1) )2 ϕ̈ ◦ ϕ(`−1) + ( ϕ̈(`−1) ) ϕ̇ ◦ ϕ(`−1),
whence ϕ̈(`)(0) = ϕ̈(0) + ϕ̈(`−1)(0).
Using Lemma E.5 to get ϕ̈(0) = −2/(3π), this yields
ϕ̈(`)(0) = − 2` 3π .
Similarly, since we have shown ϕ̇(`−1)(π) = 0, we obtain ϕ̈(`)(π) = 0.
Lemma C.18. For first and second derivatives of ξ(`), one has
ξ̇(`) = −π−1 L−1∑ `′=` ϕ̇(` ′) L−1∏ `′′=` `′′ 6=`′ ( 1− π−1ϕ(` ′′) ) ,
and
ξ̈(`) (C.30)
= −π−1 L−1∑ `′=` ϕ̈(`′) L−1∏ `′′=` `′′ 6=`′ ( 1− π−1ϕ(` ′′) ) − π−1ϕ̇(` ′) L−1∑ `′′=` `′′ 6=`′ ϕ̇(` ′′)
L−1∏ `′′′=`
`′′′ 6=`′,`′′′ 6=`′′
( 1− π−1ϕ(` ′′′) ) ,
(C.31)
where empty sums are interpreted as zero, and empty products as 1. In particular, one calculates
ξ(`)(0) = 1; ξ̇(`)(0) = −L− ` π ; ξ̈(`)(0) = (L− `)(L− `− 1) π2 + L(L− 1)− `(`− 1) 3π2 ,
and ξ(0)(π) = 0; ξ̇(`)(π) = − 1
π ξ(1)(π)1`=0; ξ̈ (`)(π) = 0.
Proof. The two derivative formulas are direct applications of the Leibniz rule to ξ(`). The claims about values at 0 follow from plugging the results of Lemma C.17 into our derivative formulas and the definition of ξ(`). For values at π, we first note that ϕ(0)(π) = π, from which it follows ξ(0)(π) = 0. Next, we use Lemma C.17 to get that ϕ̈(`)(π) = 0 for all ` ∈ {0, 1, . . . , L} and ϕ̇(`)(π) = 1`=0 to get ξ̇(`)(π) = −π−1ξ(1)(π)1`=0. For ξ̈(`)(π), we have
ξ̈(`) = π−2 L−1∑ `′=` ϕ̇(` ′)(π) ∑ `′′ 6=`′ ϕ̇(` ′′)(π) ∏ `′′′ 6=`′′,`′′′ 6=`′ ( 1− π−1ϕ(` ′′′)(π) )
= π−21`=0 L−1∑ `′=1 ϕ̇(` ′)(π) ∏ `′′ 6=`′,`′′ 6=0 ( 1− π−1ϕ(` ′′′)(π) ) .
If L = 1, the sum in the last expression is empty, and this quantity is 0. If L > 1, the sum is nonempty, and every summand is equal to zero by Lemma C.17. We conclude ξ̈(`)(π) = 0.
Lemma C.19. If L ≥ 3, there exists an absolute constant 0 < C ≤ π/2 such that on the interval [0, C], one has for every ` = 0, 1, . . . , L− 1
... ξ (`) ≤ 0.
Proof. We consider functions only on [0, π/2] in this proof. Following the calculations in the proof of Lemma C.15, we have the expression(
ϕ ◦ ϕ(`−1) )′′′
(C.32) = ( ϕ̇ ◦ ϕ(`−1) ) ... ϕ(`−1) + 3 ( ϕ̈ ◦ ϕ(`−1) ) ϕ̇(`−1)ϕ̈(`−1) + (... ϕ ◦ ϕ(`−1) )( ϕ̇(`−1) )3 . (C.33)
Using as well Lemma E.5, we have first and second derivative estimates
0 ≤ ϕ̇(`) ≤ 1
and −C2` ≤ ϕ̈(`) ≤ −c2`. By Lemma E.5, ... ϕ extends to a continuous function on [0, π/2], so in addition there exists a δ > 0 such that on [0, δ] we have ... ϕ ≥ − 1
2π2 (C.34)
We lower bound (C.33) on [0, δ] using these estimates. For ` = 1, we can do no better than (C.34). For ` > 1, we can write(
ϕ ◦ ϕ(`−1) )′′′ ≥ ( ϕ̇ ◦ ϕ(`−1) ) ... ϕ(`−1) + 3ϕ̇(`−1) (( ϕ̈ ◦ ϕ(`−1) ) ϕ̈(`−1) − 1
6π2
( ϕ̇(`−1) )2) ≥ ( ϕ̇ ◦ ϕ(`−1) ) ... ϕ(`−1) + 3ϕ̇(`−1) ( c22(`− 1)− 1
6π2
) .
We have the numerical estimate c2 = 0.14 from Lemma E.5, and we check numerically that (0.14)2 > 1/6π2. This implies that on [0, δ] and for every ` ≥ 2, ... ϕ(`) is lower bounded by a positive number plus a scaled version of ... ϕ(`−1). We check precisely using the original formula (C.33) and Lemma E.5 for ` = 2
... ϕ(2)(0) = 2 ... ϕ(0) + 3ϕ̈(0)2 = 2
3π2 > 0,
so that in particular ... ϕ(2)(0) + ... ϕ(1)(0) = 1
3π2 > 0.
By continuity, it follows that there is a neighborhood [0, δ′] on which we have ... ϕ(2) + ... ϕ(1) > 0. Thus, on [0,min{δ, δ′}], we guarantee that simultaneously ... ϕ(`) > 0 if ` ≥ 2; ... ϕ(2) + ... ϕ(1) > 0.
Now we consider the third derivative of the skeleton summands ξ(`). Following the calculations of Lemmas C.10 and C.18, in particular applying the Leibniz rule, we observe that every term in the sum defining ... ξ (`) that does not involve a third derivative of one of the factors (1 − (1/π)ϕ(`′)) will be nonpositive, because (1 − (1/π)ϕ(`′)) ≥ 0, ϕ̇(`′) ≥ 0, and ϕ̈(`′) ≤ 0. Meanwhile, by our calculations above, on the interval [0,min{δ, δ′}], the only terms that can be positive are those with ` = 0 or ` = 1 where we differentiate the `′ = 1 factor three times, i.e., the `′ = 1 term in the sum
− 1 π L−1∑ `′=` ... ϕ(` ′) L−1∏ `′′=` `′′ 6=`′
( 1− ϕ (`′′)
π
)
with ` = 0 or ` = 1. We will compare the `′ = 1 summand with the `′ = 2 summand: we have that the sum of these two terms equals
− 1 π  L−1∏ `′′=` `′′ 6=1,2 ( 1− ϕ (`′′) π )(...ϕ(1− ϕ(2)π ) + ... ϕ(2) ( 1− ϕ π )) . (C.35)
At 0, the quantity inside the right parentheses is equal to ... ϕ + ... ϕ(2) > 0, by our calculations above. Thus, by continuity, there is a possibly smaller δ′′ > 0 such that on [0, δ′′], the sum of terms (C.35) is negative. We conclude that on [0,min{δ, δ′, δ′′}], we have for every ` ≥ 0
... ξ (`) ≤ 0,
and since we have chosen the neighborhood sizes δ, δ′, δ′′ independently of the depth L, we can conclude.
Lemma C.20. For all ` ∈ {0, . . . , L− 1}, one has
ξ(`)(ν) ≤ 1 + `ν/π 1 + Lν/π .
Proof. We have
ξ(`)(ν) = L−1∏ `′=`
( 1− ϕ (`′)(ν)
π
) ≤ ( 1− 1
π(L− `) L−1∑ `′=` ϕ(` ′)(ν)
)L−`
≤ exp ( − 1 π L−1∑ `′=` ϕ(` ′)(ν) ) ,
(C.36)
where the first inequality applies the AM-GM inequality, and the second uses the standard exponential convexity estimate. Using Lemma C.13, we have
− L−1∑ `′=` ϕ(` ′)(ν) ≤ − L−1∑ `′=`
ν
1 + `′ν/π ≤ − ∫ L `
ν
1 + `′ν/π d`′,
where the last inequality uses the fact that `′ 7→ ν/(1 + `′ν/π) is nonincreasing for every ν ∈ [0, π] together with a standard estimate from the integral test. We calculate∫ L
`
ν
1 + `′ν/π d`′ = π log
( 1 + Lν/π
1 + `ν/π
) ,
which gives the claim after substituting into (C.36).
Lemma C.21. For all ` ∈ {0, . . . , L− 1}, one has
|ξ̇(`)(ν)| ≤ 3 L− ` 1 + Lν/π .
Proof. Using Lemma C.18, we have
ξ̇(`) = −ξ (1)
π 1`=0 −
ξ(`)
π L−1∑ `′=max{`,1}
ϕ̇(` ′)
1− ϕ(`′)/π
where we directly treat the case ` = 0 to avoid dividing by zero at ν = π. The triangle inequality and Lemmas E.5 and C.20 then give
|ξ̇(`)(ν)| ≤ 2 π  1 1 + Lν/π 1`=0 + ξ (`)(ν) L−1∑ `′=max{`,1} ϕ̇(` ′)(ν)  .
Using Lemma C.15, we have L−1∑ `′=` ϕ̇(` ′)(ν) ≤ L−1∑ `′=`
1 1 + c`′ν ≤ 1 1 + c`ν + ∫ L−1 `
1
1 + c`′ν d`′,
where the last inequality uses the fact that `′ 7→ 1/(1 + c`′ν) is nonincreasing for every ν ∈ [0, π] together with a standard estimate from the integral test. Evaluating the integral, we obtain
L−1∑ `′=` ϕ̇(` ′)(ν) ≤ 1 1 + c`ν + 1 cν log ( 1 + c(L− 1)ν 1 + c`ν ) ,
where the second term on the RHS is defined at ν = 0 by continuity. Using the standard concavity estimate log(1 + x) ≤ x, we have
1
cν log
( 1 + c(L− 1)ν
1 + c`ν
) = 1
cν log
( 1 +
(L− `− 1)cν 1 + c`ν
) ≤ L− `− 1
1 + c`ν ,
whence L−1∑ `′=` ϕ̇(` ′)(ν) ≤ L− ` 1 + c`ν . (C.37) Combined with the result of Lemma C.20, we conclude
ξ(`)(ν) L−1∑ `′=` ϕ̇(` ′)(ν) ≤ 1 cπ L− ` 1 + Lν/π .
The numerical estimate c = 0.07 in Lemma E.5 then allows us to conclude
|ξ̇(`)(ν)| ≤ 3 L− ` 1 + Lν/π ,
as claimed.
Lemma C.22. One has |ψ′1(ν)| ≤ 5nL2
1 + Lν/π ,
and
|ψ′(ν)| ≤ (3/2)nL 2
1 + Lν/π .
Proof. We calculate using the chain rule
ψ′1 = n
2 L−1∑ `=0 ξ̇(`) cosϕ(`) − ξ(`)ϕ̇(`) sinϕ(`),
and the triangle inequality gives
|ψ′1| ≤ n
2 L−1∑ `=0 ∣∣∣ξ̇(`)∣∣∣+ ξ(`)ϕ̇(`). Applying Lemmas C.15, C.20 and C.21 and Lemma E.5 to estimate the constant c in Lemma C.15, we then obtain
|ψ′1(ν)| ≤ n
2(1 + Lν/π) L−1∑ `=0 3(L− `) + 1 + `ν/π 1 + `ν/(5π)
≤ n 2(1 + Lν/π) L−1∑ `=0 3(L− `) + 1 + 4 `ν/(5π) 1 + `ν/(5π)
≤ n 2(1 + Lν/π)
( 3L2
2 + 5L ) ≤ 5nL 2
1 + Lν/π .
The proof of the second claim is nearly identical, since in this case we need only use the bounds on |ξ̇(`)|.
Lemma C.23. There are absolute constants c, C > 0 such that for all ` ∈ {0, . . . , L− 1}, one has∣∣∣ξ̈(`)∣∣∣ ≤ CL(L− `)(1 + `ν/π) (1 + cLν)2 + C (L− `)2 (1 + cLν)(1 + c`ν) .
Proof. By Lemmas E.5 and C.18, we can write
ξ̈(`) =− ξ (`)
π L−1∑ `′=max{1,`} ϕ̈(` ′) 1− ϕ(` ′) π
+ 1
π2 2ξ(1)1`=0 L−1∑ `′=1 ϕ̇(` ′) 1− ϕ (`′)
π
+ ξ(`) L−1∑
`′=max{1,`} L−1∑ `′′=max{1,`}
`′′ 6=`′
ϕ̇(` ′)ϕ̇(` ′′)( 1− ϕ(` ′)
π
)( 1− ϕ(` ′′)
π
)
Focusing first on the second term, we have using Lemma E.5, (C.37) and Lemma C.20
2ξ(1)1`=0 L−1∑ `′=1 ϕ̇(` ′) 1− ϕ (`′)
π
+ ξ(`) L−1∑
`′=max{1,`} L−1∑ `′′=max{1,`}
`′′ 6=`′
ϕ̇(` ′)ϕ̇(` ′′)( 1− ϕ(` ′)
π
)( 1− ϕ(` ′′)
π
)
≤4ξ(1)1`=0 L−1∑ `′=1 ϕ̇(` ′) + 4ξ(`) L−1∑ `′=max{1,`} L−1∑ `′′=max{1,`}
`′′ 6=`′
ϕ̇(` ′)ϕ̇(` ′′).
We can then write using nonnegativity
L−1∑ `′=` L−1∑ `′′=` `′′ 6=`′ ϕ̇(` ′)ϕ̇(` ′′) ≤ L−1∑ `′=` L−1∑ `′′=` ϕ̇(` ′)ϕ̇(` ′′) = ( L−1∑ `′=` ϕ̇(` ′) )2 ,
and using (C.37) and Lemma C.20, we obtain thus
ξ(1)1`=0 L−1∑ `′=1 ϕ̇(` ′) + ξ(`) L−1∑ `′=max{1,`} L−1∑ `′′=max{1,`}
`′′ 6=`′
ϕ̇(` ′)ϕ̇(` ′′) ≤ 3 cπ
(L− `)2
(1 + Lν/π)(1 + c`ν) .
Regarding the first term, we have using Lemma C.16
L−1∑ `′=` |ϕ̈(` ′)| ≤ C L−1∑ `′=`
`′ 1 + (c/4)`′ν ≤ C L(L− `) 1 + (c/4)Lν ,
because the function `′ 7→ `′/(1 + c`′ν) is nondecreasing. Applying also Lemma C.20, we obtain using the triangle inequality and worst-casing constants∣∣∣ξ̈(`)∣∣∣ ≤ C1L(L− `)(1 + `ν/π)
(1 + cLν)2 + C2
(L− `)2
(1 + cLν)(1 + c`ν) .
Lemma C.24. One has
|ψ′′1 (ν)| ≤ CnL3
1 + cLν ,
and
|ψ′′(ν)| ≤ CnL 3
1 + cLν ,
where c, C > 0 are absolute constants.
Proof. We calculate using the chain rule
ψ′′1 = n
2 L−1∑ `=0 ξ̈(`) cosϕ(`) − 2ξ̇(`)ϕ̇(`) sinϕ(`) − ξ(`)ϕ̈(`) sinϕ(`) − ξ(`) ( ϕ̇(`) )2 cosϕ(`),
and the triangle inequality gives
|ψ′′1 | ≤ n
2 L−1∑ `=0 ξ̈(`) + 2 ∣∣∣ξ̇(`)∣∣∣ϕ̇(`) + ξ(`)∣∣∣ϕ̈(`)∣∣∣+ ξ(`) (ϕ̇(`))2 .
Using Lemmas C.15, C.16, C.20, C.21 and C.23 and worst-casing constants for convenience, we obtain from the last estimate
|ψ′′1 (ν)| ≤ Cn L−1∑ `=0  L(L−`)(1+`ν/π)(1+cLν)2 + (L−`)2(1+cLν)(1+c`ν) + L−`(1+Lν/π)(1+c`ν) + 1+`ν/π 1+Lν/π ( 1 (1+c`ν)2 + ` )  ≤ CnL3 1 + cLν ,
where in the second line we made some estimates along the lines of the proof of Lemma C.22 and worsened the constant C. The proof for ψ follows from the same argument, since in this case we have the same sum of ξ̈(`) terms but none of the extra residuals.

heading: D CONCENTRATION AT INITIALIZATION

heading: D.1 NOTATION AND FRAMEWORK

text:
We recall the expression for the neural tangent kernel, as summarized in Appendix A.5.2: Θ(x,x′) = 〈 ∇̃fθ0(x), ∇̃fθ0(x′) 〉 = 〈 αL(x),αL(x′) 〉 +
L−1∑ `=0 〈 α`(x),α`(x′) 〉〈 β`(x),β`(x′) 〉 ,
The objective of this section is to establish supporting results for the proof of Theorem B.2, which gives uniform concentration of Θ(x,x′) overM×M around the deterministic skeleton kernel. We take a pointwise-uniformize approach to proving this result: Appendix D.2 establishes concentration results for the constituents of Θ(x,x′) when x,x′ are fixed, and Appendix D.3 develops results that control the number of local support changes near points in a discretization of M ×M in order to provide a suitable stand-in for the continuity properties necessary to uniformize these pointwise results. We collect relevant technical results and their proofs in Appendix D.4.

heading: D.2 POINTWISE CONCENTRATION

text: We fix (x,x′) in this section, and generally suppress notation involving the specific points for concision. We separate our analysis into two distinct sub-problems: “forward concentration”, which consists of the study of the correlations 〈α`(x),α`(x′)〉, and “backward concentration”, which consists of the study of the backward feature correlations 〈β`(x),β`(x′)〉. Forward concentration is a prerequisite of our approach to backward concentration, so we begin there.

heading: D.2.1 FORWARD CONCENTRATION

text:
Notation. For ` = 0, 1, . . . , L, define random variables z`1 = ‖α`(x)‖2 and z`2 = ‖α`(x′)‖2. With the convention 0 ·+∞ = 0, we define for ` = 0, . . . , L, random variables ν` by
ν` = cos-1 ( 1z`1>0 1z`2>0 〈 α`(x)
‖α`(x)‖2 , α`(x′) ‖α`(x′)‖2
〉 − 1{z`1=0}∪{z`2=0} ) .
These definitions guarantee that ν` = π whenever either feature norm z`i vanishes. These random variables are significant toward controlling Θ(x,x′) because, for each `
〈α`(x),α`(x′)〉 = z`1z`2 cos ν`.
Let us define pairs of gaussian vectors g`1, g ` 2 ∼i.i.d. N (0, (2/n)I) that are independent of everything else in the problem. For ` ≥ 1, we have by rotational invariance of the Gaussian distribution and the probability chain rule
z`1 = ∥∥∥[W `α`−1(x)]
+ ∥∥∥ 2 d = ∥∥∥[g`1]+∥∥∥2z`−11 .
Since α0(x) = x and ‖x‖2 = 1, we have by an induction with analogous definitions
z`1 d = ∏̀ `′=1 ∥∥∥∥[g`′1 ] + ∥∥∥∥ 2 .
Similarly, we have
z`2 d = ∏̀ `′=1 ∥∥∥∥[g`′1 ] + ∥∥∥∥ 2 .
As for the angles, we have by rotational invariance
z`1z ` 2 = ∥∥∥[W `α`−1(x)] + ∥∥∥ 2 ∥∥∥[W `α`−1(x′)] + ∥∥∥ 2
d = ∥∥∥[g`1]+∥∥∥2∥∥∥[g`1 cos ν`−1 + g`2 sin ν`−1]+∥∥∥2z`−11 z`−12 ,
so that an inductive argument gives
z`1z ` 2 d = (∏̀ `′=1 ∥∥∥∥[g`′1 ] + ∥∥∥∥ 2 )(∏̀ `′=1 ∥∥∥∥[g`′1 cos ν`′−1 + g`′2 sin ν`′−1]+ ∥∥∥∥ 2 ) .
We will write
z̄`1 = ∏̀ `′=1 ∥∥∥∥[g`′1 ] + ∥∥∥∥ 2 , z̄`2 = ∏̀ `′=1 ∥∥∥∥[g`′1 cos ν`′−1 + g`′2 sin ν`′−1]+ ∥∥∥∥ 2 ,
and similarly
ν̄` = cos-1 1z̄`1z̄`2>0 〈 [ g`1 ] +∥∥∥[g`1]+∥∥∥2 , [ g`1 cos ν `−1 + g`2 sin ν `−1] +∥∥∥[g`1 cos ν`−1 + g`2 sin ν`−1]+∥∥∥2 〉 − 1{z̄`1=0}∪{z̄`2=0}  , so that we obtain for the angles by a similar inductive argument
ν` d = ν̄`. (D.1)
For technical reasons, it will be convenient to consider an auxiliary angle process, defined for ` ≥ 1 as
ν̂` = cos-1 1Ē(g`1, g`2) 〈 [ g`1 ] +∥∥∥[g`1]+∥∥∥2 , [ g`1 cos ν̂ `−1 + g`2 sin ν̂ `−1] +∥∥∥[g`1 cos ν̂`−1 + g`2 sin ν̂`−1]+∥∥∥2 〉 , (D.2)
where we define with notation from Appendix E.1 Ē = ⋂ i∈[n] { (g1, g2) ∣∣∣ ∀ν ∈ [0, 2π], 12 ≤ ∥∥I[n]\{i}[g1 cos ν + g2 sin ν]+∥∥2 ≤ 2}, and ν̂0 = ν0 = 〈x,x′〉. We then observe
∏̀ `′=1 1Ē ( g` ′ 1 , g `′ 2 ) ≤ ∏̀ `′=1 1z̄`1z̄ ` 2>0 ,
since the inductive structure of z̄`i implies that all feature norms are nonvanishing if and only if the top-level feature norms z̄Li are nonvanishing, and since the statement ∏` `′=1 1Ē ( g` ′ 1 , g `′ 2 ) = 1
implies by definition that z̄L1 ≥ 2−L and z̄L2 ≥ 2−L. By Lemma E.16, as long as n ≥ 21 the event Ē has overwhelming probability, and in particular a union bound implies
P [∏̀ `′=1 1z̄`1z̄ ` 2>0 = 1 ] ≥ P [∏̀ `′=1 1Ē ( g` ′ 1 , g `′ 2 ) = 1 ] ≥ 1− CLe−cn, (D.3)
so that P [ ∀` = 1, 2, . . . , L, ν̂` = ν̄` ] ≥ 1− CLe−cn. (D.4)
We can therefore pass from ν̄` to ν̂` with negligible error.
From the expression for ν̂`, we see that the angles ν̂0 → ν̂1 → · · · → ν̂L form a Markov chain, and we will control them using martingale techniques. For ` = 0, 1, . . . , L, we write F` to denote the σ-algebra generated by the gaussian vectors (g11 , g 1 2 , g 2 1 , g 2 2 , . . . , g ` 1, g ` 2), so that (F0, . . . ,FL) is a filtration, and the sequences of random variables (ν̂1, . . . , ν̂L) and (ν̄1, . . . , ν̄L) are adapted to (F1, . . . ,FL). Moreover, with these definitions we have
E [ ν̂` ∣∣ F`−1] = ϕ̄(ν̂`−1),
where ϕ̄ is the angle evolution function defined in Appendix E.1, which is well-approximated by the function ϕ(ν) = cos-1 ((
1− ν π
) + sin ν
π ) (see Lemmas E.1 and E.2). In the sequel, we will employ the notation ϕ(`) to denote the `-fold composition of ϕ with itself. By Lemma E.5, the function ϕ is smooth, and the chain rule implies the same for ϕ(`); we will employ the notation ϕ̇(`) and ϕ̈(`) for the first and second derivatives of ϕ(`), respectively.

heading: Main results.

text:
Lemma D.1. There are absolute constants c, C,C ′ > 0 and absolute constants K,K ′ > 0 such that for any d ≥ K, if n ≥ K ′max{1, d4 log4 n, d3L log3 n} then one has for any ` = 1, . . . , L
P ∣∣∣〈α`(x),α`(x′)〉− cosϕ(`)(∠(x,x′))∣∣∣ > C √ d3` log3 n
n  ≤ C ′n−cd. Proof. We have 〈
α`(x),α`(x′) 〉 d
= z`1z ` 2 cos ν `,
and the triangle inequality (applied twice) then yields∣∣∣〈α`(x),α`(x′)〉− cosϕ(`)(ν0)∣∣∣ ≤ ∣∣cos ν`∣∣∣∣z`1z`2 − 1∣∣+ ∣∣∣cos ν` − cosϕ(`)(ν0)∣∣∣ ≤ ∣∣z`2∣∣∣∣z`1 − 1∣∣+ ∣∣z`2 − 1∣∣+ ∣∣∣ν` − ϕ(`)(ν0)∣∣∣,
where we also use |cos| ≤ 1 and that cos is 1-Lipschitz. Since z`i d = z̄`i for i = 1, 2, we obtain using Lemma D.2 and the choice n ≥ KdL
P [∣∣z`i − 1∣∣ > C√d`n ] ≤ C ′`e−d,
and as long as n ≥ C2dL, we obtain on one of the same events P [ z`2 ≤ 2 ] ≥ 1− C ′`e−d.
By a union bound, we obtain
P [∣∣z`2∣∣∣∣z`1 − 1∣∣+ ∣∣z`2 − 1∣∣ ≤ 3C√d`n ] ≥ 1− 2C ′`e−d,
so that if we put d′ = d log n and therefore choose n ≥ C2dL log n, we have
P [∣∣z`2∣∣∣∣z`1 − 1∣∣+ ∣∣z`2 − 1∣∣ ≤ 3C√d` log nn ] ≥ 1− 2C ′`n−d ≥ 1− 2C ′n−d,
with the second bound holding if d ≥ 1 and n ≥ L. For the remaining term, we have by the triangle inequality ∣∣∣ν` − ϕ(`)(ν0)∣∣∣ ≤ ∣∣ν` − ν̂`∣∣+ ∣∣∣ν̂` − ϕ(`)(ν0)∣∣∣. By (D.4), the first term on the RHS of the previous expression is equal to zero with probability at least 1−CLe−cn as long as n ≥ 21. The second term can be controlled with Lemma D.3 provided we select n,L, d to satisfy the hypotheses of that lemma. We thus obtain via an additional union bound
P ∣∣∣〈α`(x),α`(x′)〉− cosϕ(`)(ν0)∣∣∣ > 3C√d` log n n + C ′ √ d3 log3 n n`  ≤ C ′′n−cd+C ′′′`e−c′n. If n ≥ (2/c′) logL and n ≥ (2c/c′)d log n, we have C ′′n−cd + C ′′′`e−c′n ≤ (C ′′ + C ′′′)n−cd. The previous bound then becomes
P ∣∣∣〈α`(x),α`(x′)〉− cosϕ(`)(ν0)∣∣∣ > 3C√d` log n n + C ′ √ d3 log3 n n`  ≤ (C ′′ + C ′′′)n−cd, and if we worst-case the dependence on ` and d in the residual in the previous bound, we obtain
P ∣∣∣〈α`(x),α`(x′)〉− cosϕ(`)(ν0)∣∣∣ > (3C + C ′) √ d3` log3 n
n  ≤ (C ′′ + C ′′′)n−cd, as claimed.
Lemma D.2. There are absolute constants c, C,C ′ > 0 and an absolute constant K > 0 such that for i = 1, 2, every ` = 1, . . . , L, and any d > 0, if n ≥ max{Kd`, 4}, then one has
P [∣∣z`i − 1∣∣ > C√d`n ] ≤ C ′`e−cd.
Proof. Because z`i d = z̄`1, it suffices to show
P [∣∣∣∣∣−1 + ∏̀ `′=1 ∥∥∥∥[g`′1 ] + ∥∥∥∥ 2 ∣∣∣∣∣ > C √ d` n ] ≤ C ′`e−cd. (D.5)
The proof will proceed by showing concentration of the squared quantity ∏` `′=1 ∥∥∥∥[g`′1 ] + ∥∥∥∥2 2 around 1, so that we can appeal to results like Lemma D.26, and then conclude by applying an inequality for the square root to pass to the actual quantity of interest. To enter the setting of Lemma D.26, it makes sense to normalize the factors in the product by their degree, but we must avoid dividing by zero. We
have ∏` `′=1 ∥∥∥∥[g`′1 ] + ∥∥∥∥ 0 = 0 if and only if ∏` `′=1 ∥∥∥∥[g`′1 ] + ∥∥∥∥ 2 = 0, and whenever ∏` `′=1 ∥∥∥∥[g`′1 ] + ∥∥∥∥ 2 6= 0, we can write
∏̀ `′=1 ∥∥∥∥[g`′1 ] + ∥∥∥∥ 2 = (∏̀ `′=1 ∥∥∥∥[g`′1 ] + ∥∥∥∥2 2 )1/2 (D.6)
= (∏̀ `′=1 2 n ∥∥∥∥[g`′1 ] + ∥∥∥∥ 0 )1/2∏̀ `′=1 1∥∥∥√n2 [g`′1 ]+∥∥∥0 ∥∥∥∥√n2 [g`′1 ]+ ∥∥∥∥2 2 1/2 , (D.7)
using 0-homogeneity of the `0 “norm”. This leads to an extra product-of-degrees term; we will make use of Lemma D.27 to show that the product of degrees itself concentrates. We will also show that the event where a degree is zero is extremely unlikely and proceed with the degree-normalized main term by conditioning. By symmetry, the random variables ‖ [ g`1 ] + ‖0 are i.i.d. sums of n Bernoulli random variables with rate 12 . By Lemma G.1, we then have
P [∥∥∥[g`1]+∥∥∥0 < n/2− t] ≤ e−2t2/n,
and so P [
min `′=1,...,` ∥∥∥∥[g`′1 ] + ∥∥∥∥ 0 < n/2− t ] = P [ ∃`′ ∈ {1, . . . , `} : ∥∥∥∥[g`′1 ] + ∥∥∥∥ 0 < n/2− t ]
≤ `P [∥∥∥[g`1]+∥∥∥0 < n/2− t] ≤ `e−2t2/n,
where the first inequality applies a union bound. Putting t = n/4, we conclude P [
min `′=1,...,` ∥∥∥∥[g`′1 ] + ∥∥∥∥ 0 < n/4 ] ≤ `e−n/8,
so that whenever n ≥ 16 log `, we have ‖ [ g` ′
1 ] + ‖0 ≥ n/4 for every `′ ≤ ` with probability at least
1−e−n/16. This gives us enough to begin working on showing concentration of the squared version of (D.5): partitioning, we can use the previous simplified bound to write
P [∣∣∣∣∣−1 + ∏̀ `′=1 ∥∥∥∥[g`′1 ] + ∥∥∥∥2 2 ∣∣∣∣∣ > C √ d` n ] (D.8)
≤ e−n/16 + P [ min
`′=1,...,` ∥∥∥∥[g`′1 ] + ∥∥∥∥ 0 ≥ n/4, ∣∣∣∣∣−1 + ∏̀ `′=1 ∥∥∥∥[g`′1 ] + ∥∥∥∥2 2 ∣∣∣∣∣ > C √ d` n ] . (D.9)
Using (D.7) and the triangle inequality, we can write whenever no terms in the product vanish∣∣∣∣∣−1 + ∏̀ `′=1 ∥∥∥∥[g`′1 ] + ∥∥∥∥2 2 ∣∣∣∣∣ = ∣∣∣∣∣∣ (∏̀ `′=1 2 n ∥∥∥∥[g`′1 ] + ∥∥∥∥ 0 )∏̀ `′=1 1∥∥∥√n2 [g`′1 ]+∥∥∥0 ∥∥∥∥√n2 [g`′1 ]+ ∥∥∥∥2 2 − 1 ∣∣∣∣∣∣
≤ ∣∣∣∣∣ (∏̀ `′=1 2 n ∥∥∥∥[g`′1 ]+ ∥∥∥∥ 0 )∣∣∣∣∣ ∣∣∣∣∣∣ ∏̀ `′=1 1∥∥∥√n2 [g`′1 ]+∥∥∥0 ∥∥∥∥√n2 [g`′1 ]+ ∥∥∥∥2 2 − 1 ∣∣∣∣∣∣
+ ∣∣∣∣∣ (∏̀ `′=1 2 n ∥∥∥∥[g`′1 ] + ∥∥∥∥ 0 ) − 1 ∣∣∣∣∣. (D.10) Moreover, we have by Lemma D.27
P [∣∣∣∣∣−1 + ∏̀ `′=1 2 n ∥∥∥∥[g`′1 ] + ∥∥∥∥ 0 ∣∣∣∣∣ > 4 √ d` n ] ≤ 4`e−cd
as long as n ≥ 128d`. Choosing in addition n ≥ 4d` and using nonnegativity, this implies
P [∣∣∣∣∣ ∏̀ `′=1 2 n ∥∥∥∥[g`′1 ]+ ∥∥∥∥ 0 ∣∣∣∣∣ > 2 ] ≤ 4`e−cd,
occurring on the same event. Combining the previous two bounds with (D.10) and (D.9) via another partition, we get
P [∣∣∣∣∣−1 + ∏̀ `′=1 ∥∥∥∥[g`′1 ] + ∥∥∥∥2 2 ∣∣∣∣∣ > C √ d` n ] ≤ e−n/16 + 4`e−cd
+ P  min`′=1,...,` ∥∥∥∥[g`′1 ] + ∥∥∥∥ 0
≥ n/4,∣∣∣∣∣−1 +∏``′=1 1∥∥∥√n2 [g`′1 ]+∥∥∥0 ∥∥∥∥√n2 [g`′1 ] + ∥∥∥∥2 2 ∣∣∣∣∣ > (C/2 + 2)√d`n , (D.11)
where we use here that on the event {min`′=1,...,` ‖ [ g` ′
1 ] + ‖0 ≥ n/4}, the quantity ∏` `′=1 ∥∥∥∥[g`′1 ] + ∥∥∥∥ 2
is nonzero almost surely, which allowed us to invoke the identities (D.7). For (k1, . . . , k`) ∈ [n]`, we define events Ek11 , . . . , E k` ` by
Ek`′`′ = {∥∥∥∥√n2 [g`′1 ]+ ∥∥∥∥ 0 = k`′ } .
Conditioning and then relaxing the bounds, we can write
P  min `′=1,...,` ∥∥∥∥[g`′1 ] + ∥∥∥∥ 0 ≥ n/4, ∣∣∣∣∣∣−1 + ∏̀ `′=1 1∥∥∥√n2 [g`′1 ]+∥∥∥0 ∥∥∥∥√n2 [g`′1 ]+ ∥∥∥∥2 2 ∣∣∣∣∣∣ > C ′ √ d` n
≤ ∑
(k1,...,k`)∈[n]` k`′≥dn/4e
P [∣∣∣∣∣−1 +∏``′=1 1∥∥∥√n2 [g`′1 ]+∥∥∥0 ∥∥∥∥√n2 [g`′1 ]+ ∥∥∥∥2 2 ∣∣∣∣∣ > C ′√d`n ∣∣∣∣∣ Ek11 , . . . , Ek`` ] ∗P [ Ek11 , . . . , E k` `
] . Conditioned on Ek11 , . . . , E
k` ` with k`′ > 0, the random variable∏` `′=1‖ √ n 2 [ g` ′ 1 ] + ‖22/‖ √ n 2 [ g` ′ 1 ] + ‖0 is distributed as a product of independent degree-normalized standard χ2 random variables with minimum degree min{k1, . . . , k`}. An application of Lemma D.26 then yields immediately
P ∣∣∣∣∣∣−1 + ∏̀ `′=1 1∥∥∥√n2 [g`′1 ]+∥∥∥0 ∥∥∥∥√n2 [g`′1 ]+ ∥∥∥∥2 2 ∣∣∣∣∣∣ > C ′ √ d` n ∣∣∣∣∣∣ Ek11 , . . . , Ek``  ≤ C ′′`e−cd
as long as n ≥ K ′′d`, whence
P  min `′=1,...,` ∥∥∥∥[g`′1 ] + ∥∥∥∥ 0 ≥ n/4, ∣∣∣∣∣∣−1 + ∏̀ `′=1 1∥∥∥√n2 [g`′1 ]+∥∥∥0 ∥∥∥∥√n2 [g`′1 ]+ ∥∥∥∥2 2 ∣∣∣∣∣∣ > C ′ √ d` n  ≤ C ′′`e−cd.
Combining this previous bound with (D.11) yields
P [∣∣∣∣∣−1 + ∏̀ `′=1 ∥∥∥∥[g`′1 ] + ∥∥∥∥2 2 ∣∣∣∣∣ > C √ d` n ] ≤ e−n/16 + C ′`e−cd,
where we worst-cased constants in the probability bound. If we choose n ≥ 4C2d`, we have C √ d`/n ≤ 1/2, and we obtain on the event in the previous bound
P [∣∣∣∣∣−1 + ∏̀ `′=1 ∥∥∥∥[g`′1 ] + ∥∥∥∥2 2 ∣∣∣∣∣ > 12 ] ≤ e−n/16 + C ′`e−cd.
In particular, on the complement of the event in the previous bound, the product lies in [1/2, 3/2]. To conclude, we can linearize the square root near 1 to obtain an analogous bound for the product of the norms. Taylor expansion of the smooth function x 7→ x1/2 about the point 1 gives
√ x− 1 = 1 2 (x− 1)− 1 8 k−3/2 (x− 1)2 ,
where k lies between x and 1. In particular, if x ≥ 12 , we have 1
2 (x− 1)− 1√ 2 (x− 1)2 ≤ √ x− 1 ≤ 1 2 (x− 1) ,
so that ∣∣∣∣(√x− 1)− 12 (x− 1) ∣∣∣∣ ≤ 1√2 (x− 1)2 .
Thus, when x ≥ 12 we have by the triangle inequality∣∣√x− 1∣∣ ≤ 1√ 2 (x− 1)2 + 1 2 |x− 1|.
from which we conclude based on a partition and our previous choices of large n
P [∣∣∣∣∣−1 + ∏̀ `′=1 ∥∥∥∥[g`′1 ] + ∥∥∥∥ 2 ∣∣∣∣∣ > 2C √ d` n ] ≤ 2e−n/16 + 2C ′`e−cd,
which yields the claimed probability bound when n ≥ 16d.
Lemma D.3. There are absolute constants c, C,C0 > 0 and absolute constants K,K ′ > 0 such that for any Lmax ∈ N and any d ≥ K, if n ≥ K ′max{1, d4 log4 n, d3Lmax log3 n}, then one has
P ∃L ∈ [Lmax] : ∣∣∣ν̂L − ϕ(L)(ν0)∣∣∣ > C0 √ d3 log3 n
nL  ≤ Cn−cd. (D.12) Proof. The proof uses a recursive construction involving L ∈ [Lmax]. Before beginning the main argument, we will define the key quantities that appear and enforce bounds on the parameters to obtain certain estimates. For each L ∈ [Lmax], we define the event
EL = ∣∣∣ν̂L − ϕ(L)(ν0)∣∣∣ > C0 √ d3 log3 n nL , where C0 > 0 is an absolute constant whose value we will specify below, so that EL ∈ FL, and our task is to produce an appropriate measure bound on ⋃ L∈[Lmax] EL. For notational convenience, we also define E0 = ∅. For each L ∈ [Lmax] and each ` ∈ [L], we define
∆`L = ϕ (L−`)(ν̂`)− ϕ(L−`+1)(ν̂`−1),
so that for every L, ∆1L, . . . ,∆ L L is adapted to the sequence F1, . . . ,FL, and we have the decomposition
ν̂L − ϕ(L)(ν0) = L∑ `=1 ∆`L.
In particular, we have
EL =  ∣∣∣∣∣ L∑ `=1 ∆`L ∣∣∣∣∣ > C0 √ d3 log3 n nL . The sequences (∆`L)`∈L are not quite martingale difference sequences, but we will show they are very nearly so: writing
∆`L = ( ∆`L − E [ ∆`L ∣∣ F`−1])︸ ︷︷ ︸ ∆̄`L +E [ ∆`L ∣∣ F`−1], we have that (∆̄`L)`∈L is a martingale difference sequence, which can be controlled using truncation and martingale techniques, and the extra conditional expectation term can be controlled analytically. In particular, we have the following estimates: by Lemma D.24, we have if n ≥ max{K1 log4 n,K2Lmax} that for every L ∈ [Lmax] and every ` ∈ [L]∣∣E[∆`L ∣∣ F`−1]∣∣ ≤ C1 log nn ν̂`−11 + (c0/64)(L− `)ν̂`−1 (1 + logL) + C2 1n2 ; (D.13) by the first result in Lemma D.25 we have for every d ≥ max{K3, 6/c1} that if n ≥ K4d4 log4 n, then for every L ∈ [Lmax] and every ` ∈ [L] (and after worsening constants)
P [∣∣∆`L∣∣ > 2C3√d log nn ν̂`−11 + (c0/64)(L− `)ν̂`−1 + 2C2n2 ∣∣∣∣∣ F`−1 ] ≤ C5n−c1d; (D.14)
and by the second result in Lemma D.25, we have by our previous choices of n, d, and Lmax that for every L ∈ [Lmax] and every ` ∈ [L] (after worsening constants)
E [( ∆`L )2 ∣∣∣ F`−1] ≤ 4C23 d log nn ( ν̂`−1 1 + (c0/64)(L− `)ν̂`−1 )2 + C4 n4 . (D.15)
The main line of the argument will consist of showing that a measure bound of the form (D.12) on⋃ `∈[L−1] E` implies one of the same form on ⋃ `∈[L] E`. For any L ∈ [Lmax], on the event EcL we have
ν̂L ≤ ϕ(L)(ν̂0) + C0
√ d3 log3 n
nL
≤ 2 c0L + C0
√ d3 log3 n
nL
≤ 3 c0L ,
(D.16)
where the second inequality follows from Lemma C.12, and the third follows from the choice n ≥ (C0c0)
2d3L log3 n. In particular, if we make the choice n ≥ (C0c0)2d3Lmax log3 n, we have (D.16) on EcL for every L ∈ [Lmax]. Accordingly, for every L ∈ [Lmax] and every ` ∈ [L] we define truncation events G`L by
G`L = {∣∣∆`L∣∣ ≤ 2C3√d log nn ν̂`−11 + (c0/64)(L− `)ν̂`−1 + 2C2n2 } ∩ Ec`−1. (D.17)
We have G`L ∈ F`, and a union bound and (D.14) imply
P  ⋂ `∈[L] G`L c ∣∣∣∣∣∣ FL−1  ≤ C5Ln−c1d + P  ⋃ `′∈[L−1] E`′ ∣∣∣∣∣∣ FL−1
= C5Ln −c1d + 1⋃
`′∈[L−1] E`′ ,
where the second line uses the fact that E`′ ∈ F` ′ . In particular, taking expectations recovers
P  ⋂ `∈[L] G`L c ≤ C5Ln−c1d + P  ⋃ `′∈[L−1] E`′ . (D.18) In addition, by (D.16) we have on Ec`−1
ν̂`−1
1 + (c0/64)(L− `)ν̂`−1 ≤ 3 c0
1
(`− 1) + (3/64)(L− `)
= 3
c0
1
(3/64)L+ (61/64)`− 1
≤ 64 c0(L− 1)
≤ 128 c0L ,
where the final inequality requires L ≥ 2. Thus, when L ≥ 2, we have on G`L that∣∣∆`L∣∣ ≤ 256C3c0L √ d log n n + 2C2 n2
≤ 512C3 c0︸ ︷︷ ︸
2K0
√ d log n
nL2 ,
(D.19)
where the final inequality holds when d ≥ 1 and n ≥ (C2c0/128C3)2/3L2/3. Similarly, when L ≥ 2, on Ec`−1 we have by (D.15)
E [( ∆`L )2 ∣∣∣ F`−1] ≤ 216C23
c20
d log n nL2 + C4 n4
≤ 2 17C23 c20 d log n nL2 = 2K20 d log n nL2 ,
(D.20)
where the second inequality holds when d ≥ 1 and n ≥ (C4c20/217C23 )1/3L2/3; and in the same setting we have by (D.13)∣∣E[∆`L ∣∣ F`−1]∣∣ ≤ 128C1c0 (1 + logL) log nnL + C2n2
≤ 256C1 c0 (1 + logL) log n nL ,
(D.21)
where the second inequality holds when n ≥ (C2c0/128C1)L. In particular, if we enforce these conditions with Lmax in place of L, we have that (D.19) to (D.21) hold for all 2 ≤ L ≤ Lmax (with (D.20) and (D.21) holding on Ec`−1).
We begin the recursive construction. We will enforce C0 = max{4πC3, 6K0} for the absolute constant in the definition of E`. The main tool is the elementary identity
P  ⋃ `∈[L] E`  = P  ⋃ `∈[L−1] E` + P EL ∩ ⋂ `∈[L−1] Ec` , (D.22) which allows us to leverage an inductive argument provided we can control P[EL ∩ ⋂ `∈[L−1] Ec` ], the probability that the L-th angle deviates above its nominal value subject to all prior angles being controlled. The case L = 1 can be addressed directly: (D.14) gives
P [∣∣∆11∣∣ > 2πC3√d log nn + 2C2n2 ] ≤ C5n−c1d,
and as long as d ≥ 1 and n ≥ (C2/πC3)2/3, this implies
P [∣∣∆11∣∣ > 4πC3√d log nn ] ≤ C5n−c1d. (D.23)
This gives a suitable measure bound on E1, after choosing d ≥ 1 and n ≥ e so that d3 log3 n ≥ d log n. We now assume L ≥ 2. By the triangle inequality, we have∣∣∣∣∣ L∑ `=1 ∆`L ∣∣∣∣∣ ≤ ∣∣∣∣∣ L∑ `=1 ∆̄`L ∣∣∣∣∣+ L∑ `=1
∣∣E[∆`L ∣∣ F`−1]∣∣, (D.24) and we therefore have for any t > 0
P {∣∣∣∣∣ L∑ `=1 ∆`L ∣∣∣∣∣ > t } ∩ ⋂ `∈[L−1] Ec`  ≤ P
{∣∣∣∣∣ L∑ `=1 ∆̄`L ∣∣∣∣∣+ L∑ `=1 ∣∣E[∆`L ∣∣ F`−1]∣∣ > t } ∩ ⋂ `∈[L−1] Ec`  = P [ 1⋂
`∈[L−1] Ec` ∣∣∣∣∣ L∑ `=1 ∆̄`L ∣∣∣∣∣+ 1⋂`∈[L−1] Ec` L∑ `=1 ∣∣E[∆`L ∣∣ F`−1]∣∣ > t ] .
(D.25)
By (D.21), we have
1⋂ `∈[L−1] Ec` L∑ `=1 ∣∣E[∆`L ∣∣ F`−1]∣∣ ≤ 256C1c0 (1 + logL) log nn . (D.26)
For the remaining term, we have by the triangle inequality∣∣∣∣∣ L∑ `=1 ∆̄`L ∣∣∣∣∣ ≤ ∣∣∣∑L`=1 ∆`L −∆`L1G`L∣∣∣+ ∣∣∣∑L`=1 ∆`L1G`L − E[∆`L1G`L ∣∣∣ F`−1]∣∣∣ + ∣∣∣∑L`=1 E[∆`L1G`L ∣∣∣ F`−1]− E[∆`L ∣∣ F`−1]∣∣∣. (D.27)
By (D.17), an integration of (D.14), and a union bound, we have
P [ 1⋂
`∈[L−1] Ec` ∣∣∣∣∣ L∑ `=1 ∆`L −∆`L1G`L ∣∣∣∣∣ > 0 ]
≤ P  ⋃ `∈[L] {∣∣∆`L∣∣ > 2C3√d log nn ν̂`−11 + (c0/64)(L− `)ν̂`−1 + 2C2n2 }
≤ C5Ln−c1d,
(D.28)
and we have∣∣∣∣∣ L∑ `=1 E [ ∆`L1G`L ∣∣∣ F`−1]− E[∆`L ∣∣ F`−1] ∣∣∣∣∣
≤ L∑ `=1 E [∣∣∆`L∣∣1(G`L)c ∣∣∣ F`−1]
≤ π L∑ `=1 P [( G`L )c ∣∣∣ F`−1]
≤ π L∑ `=1 P [{∣∣∆`L∣∣ > 2C3√d log nn ν̂`−11 + (c0/64)(L− `)ν̂`−1 + 2C2n2 } ∪ E`−1 ∣∣∣∣∣ F`−1 ]
≤ πC5Ln−c1d + π L−1∑ `=1 1E` ,
where the first line uses linearity of the conditional expectation and the triangle inequality for sums and for the integral; the second line uses the worst-case bound of π on the magnitude of the increments ∆`L; the third line uses (D.17); and the fourth line uses a union bound, E`−1 ∈ F`−1, and (D.14). Multiplying both sides of the final bound by 1⋂
`∈[L−1] Ec` , we conclude
1⋂ `∈[L−1] Ec` ∣∣∣∣∣ L∑ `=1 E [ ∆`L1G`L ∣∣∣ F`−1]− E[∆`L ∣∣ F`−1] ∣∣∣∣∣ ≤ 1⋂`∈[L−1] Ec`πC5Ln−c1d ≤ πC5Ln−c1d. (D.29) For the remaining term in (D.27), we first observe
E [( ∆`L1G`L − E [ ∆`L1G`L ∣∣∣ F`−1])2 ∣∣∣∣ F`−1] ≤ E[(∆`L)2 1G`L ∣∣∣ F`−1] ≤ E [( ∆`L )2 ∣∣∣ F`−1],
where the first line uses the centering property of the L2 norm, and the second line uses ( ∆`L )2 ≥ 0 to drop the indicator for G`L. For notational simplicity, we define
V L = L∑ `=1 E [( ∆`L1G`L − E [ ∆`L1G`L ∣∣∣ F`−1])2 ∣∣∣∣ F`−1], so that our previous bound and (D.20) imply⋂
`∈[L−1]
Ec` ⊂ { V L ≤ 2K20 d log n
nL
} .
This implies that for any t > 0
P [ 1⋂
`∈[L−1] Ec` ∣∣∣∣∣ L∑ `=1 ∆`L1G`L − E [ ∆`L1G`L ∣∣∣ F`−1]∣∣∣∣∣ > t ]
= P  ⋂ `∈[L−1] Ec`  ∩ {∣∣∣∣∣ L∑ `=1 ∆`L1G`L − E [ ∆`L1G`L ∣∣∣ F`−1]∣∣∣∣∣ > t }
≤ P [{ V L ≤ 2K20 d log n
nL
} ∩ {∣∣∣∣∣ L∑ `=1 ∆`L1G`L − E [ ∆`L1G`L ∣∣∣ F`−1]∣∣∣∣∣ > t }] .
The previous term can be controlled using Lemma G.5 and (D.19):
P [{ V L ≤ 2K20 d log n
nL
} ∩ {∣∣∣∣∣ L∑ `=1 ∆`L1G`L − E [ ∆`L1G`L ∣∣∣ F`−1]∣∣∣∣∣ > t }]
≤ 2 exp − t2/2 2K20 d logn nL + (2K0/3)t √ d logn nL2  . Setting t = 3K0 √ d3 log3 n/nL, we obtain
P 1⋂ `∈[L−1] Ec` ∣∣∣∣∣ L∑ `=1 ∆`L1G`L − E [ ∆`L1G`L ∣∣∣ F`−1]∣∣∣∣∣ > 3K0 √ d3 log3 n nL  ≤ 2 exp ( −9
4
d2 log2 n
1 + d logn√ L ) ≤ 2n−(9/8)d,
(D.30)
where the last line uses the bounds L ≥ 1 and d log n/(1 + d log n) ≥ 12 if d ≥ 1 and n ≥ e. Combining (D.28) to (D.30) in (D.27) via a union bound, we obtain
P 1⋂ `∈[L−1] Ec` ∣∣∣∣∣ L∑ `=1 ∆̄`L ∣∣∣∣∣ > 3K0 √ d3 log3 n nL + πC5Ln −c1d  ≤ C5Ln−c1d + 2n−(9/8)d. Applying this result and (D.26) to (D.25) via a union bound, we obtain
P  ∣∣∣∣∣ L∑ `=1 ∆`L ∣∣∣∣∣ > 3K0 √ d3 log3 n nL + πC5Ln −c1d + 256C1 c0 (1 + logL) log n n  ∩ ⋂ `∈[L−1] E`  ≤ C5Ln−c1d + 2n−(9/8)d.
If d ≥ 2/c1 and n ≥ Lmax, we have C5Ln−c1d ≤ C5n−c1d/2; under these condition on d and n, we have πC5Ln−c1d ≤ πc5n−1, and so πC5n−c1d/2 + (256C1/c0)(1 + logL)(log n)/n ≤ C(1 + logL)(log n)/n; and if d ≥ 1 and n ≥ Lmax, we have
3K0
√ d3 log3 n
nL ≥ C (1 + logL) log n n
provided n ≥ C ′(C/3K0)2Lmax logLmax. Under these conditions, our previous bound simplifies to
P  ∣∣∣∣∣ L∑ `=1 ∆`L ∣∣∣∣∣ > 6K0 √ d3 log3 n nL  ∩ ⋂ `∈[L−1] E`  ≤ (2 + C5)n−min{c1/2,9/8}d.
In particular, applying this bound to (D.22), we have shown that for any L ≥ 2
P  ⋃ `∈[L] E`  = P  ⋃ `∈[L−1] E` + (2 + C5)n−min{c1/2,9/8}d. Unraveling the recursion with (D.23) (and worst-casing the constants there), we conclude
P  ⋃ `∈[L] E`  ≤ (2 + C5)Ln−min{c1/2,9/8}d, which proves the claim, after possibly choosing n to be larger than another absolute constant multiple of Lmax to remove the leading L factor.

heading: D.2.2 BACKWARD FEATURE CONTROL

text:
Having established concentration of the feature norms and the angles between them, it remains to control the inner products of backward features that appear in Θ. The core of the technical approach will once again be martingale concentration. We establish the following control on the backward feature inner products:
Lemma D.4. Fix x,x′ ∈ Sn0−1 and denote ν = ∠(x,x′). If n ≥ max {KL log n,K ′Ldb,K ′′}, db ≥ K ′′′ logL for suitably chosen K,K ′,K ′′,K ′′′ then
P [ L−1⋂ `=0 {∥∥β`(x)∥∥2 2 ≤ Cn }] ≥ 1− e−c nL .
If additionally n,L, d satisfy the requirements of lemmas D.3 and E.16, we have
P [ L−1⋂ `=0 {∣∣∣∣∣〈β`(x),β`(x′)〉− n2 L−1∏ i=` ( 1− ϕ (i)(ν) π )∣∣∣∣∣ ≤ log2(n)√d4Ln }] ≥ 1− e−cd
where ϕ(i) denotes i applications of the angle evolution function defined in lemma E.2, and c > 0, C are absolute constants.
Proof. For ` ∈ [L], write F` for the σ-algebra generated by all the weights up to layer ` in the network, i.e., W 1, . . . ,W `, with F0 given by the trivial σ-algebra. Consider some 〈 β` ′ (x), β` ′ (x′) 〉 for 0 ≤ `′ ≤ L− 1. Defining
Γ`:` ′ (x) = PI`(x)W `PI`−1(x) . . .PI`′ (x)W `′ ,
B`:` ′ xx′ = Γ `:`′+2(x)PI`′+1(x)PI`′+1(x′)Γ `:`′+2∗(x′),
for ` ∈ {`′ + 1, . . . , L}, and setting Γ`′+1:`′+2(x) = I ,B`′:`′xx′ = 12I , we define the event
ẼL+1:` ′
B = {∥∥∥BL:`′xx′ ∥∥∥2 F ≤ C2nL } ∩ {∥∥∥BL:`′xx′ ∥∥∥ ≤ CL} ∩ {tr [BL:`′xx′ ] ≤ Cn} . Since 〈 β` ′ (x),β` ′ (x′) 〉 = WL+1BL:` ′
xx′W L+1∗ is a Gaussian chaos in terms of the WL+1 vari-
ables (and recalling WL+1 ∼ iid N (0, I)) and ẼLB is FL-measurable, the Hanson-Wright inequality (lemma G.4) gives
P [ 1ẼL+1:`′B ∣∣∣〈β`′(x),β`′(x′)〉− tr [BL:`′xx′ ]∣∣∣ ≥ C√tnL] ≤ 2 exp ( −cmin { t, √ tn
L
}) ≤ 2e−ct.
Using lemma D.28 to bound P [( ẼL+1:` ′
B
)c] from above gives
P [∣∣∣〈β`′(x),β`′(x′)〉− tr [BL:`′xx′ ]∣∣∣ > C√tnL]
≤P [ 1ẼL+1:`′B ∣∣∣〈β`′(x),β`′(x′)〉− tr [BL:`′xx′ ]∣∣∣ ≥ C√tnL] + P [ 1(ẼL+1:`′B )c
∣∣∣〈β`′(x),β`′(x′)〉− tr [BL:`′xx′ ]∣∣∣ ≥ C√tnL] ≤P [ 1ẼL+1:`′B
∣∣∣〈β`′(x),β`′(x′)〉− tr [BL:`′xx′ ]∣∣∣ ≥ C√tnL]+ P [(ẼL+1:`′B )c] ≤2e−ct + Cn−c ′ n L ≤ C ′e−c ′′t
(D.31)
for appropriately chosen constant, if t . n log n/L. Choosing t = n/L in the bound above and using the bound on tr [ BL:` ′
xx
] from lemma D.28 we obtain
P [∥∥∥β`′(x)∥∥∥2
2 ≥ 2Cn
] ≤ P [∥∥∥β`′(x)∥∥∥2 2 − tr [ BLxx ] > Cn ] + P [ tr [ BLxx ] > Cn ] ≤ Ce−c ′ n L + C ′nL2e−c ′′ n L ≤ C ′′nL2e−c ′′ n L
for appropriate constants. Taking a union bound over the possible values of `′ proves the first part of the lemma.
We next control ∣∣∣∣tr [BL:`′xx′ ]− n2 L−1∏
`=`′
( 1− ϕ
(`)(ν) π )∣∣∣∣ using martingale concentration (in a similar manner to the control of the angles established in previous sections). We write
tr [ BL:` ′
xx′
] − n
2 L−1∏ i=`′ ( 1− ϕ (i)(ν) π ) (D.32)
= L−1∑ `=`′ L−1∏ i=`+1 ( 1− ϕ (i)(ν) π )( tr [ B`+1:` ′ xx′ ] − ( 1− ϕ (`)(ν) π ) tr [ B`:` ′ xx′ ]) ≡ L∑ `=`′+1 ∆` (D.33)
(note the change in the indexing). Consider the filtration F0 ⊂ · · · ⊂ FL and adapted sequence ∆` = ∆` − E [ ∆`|F`−1 ] , (D.34)
so that L∑
`=`′+1
∆` = L∑ `=`′+1 ∆` + L∑ `=`′+1 E [ ∆`|F`−1 ] . (D.35)
We begin by considering the first term in the sum since it takes a distinct form. Denoting byW ` ′+1
(:,i)
the i-th column ofW ` ′+1, rotational invariance of the Gaussian distribution gives tr [ B` ′+1:`′
xx′ ] =tr [ PI`′+1(x)PI`′+1(x′) ] =tr [ PW `′+1α`′ (x)>0PW `′+1α`′ (x′)>0
] d =tr [ P W ` ′+1
(:,1) >0 P W ` ′+1 (:,1) cos ν`′+W ` ′+1 (:,2) sin ν`′>0 ] and hence
E [ tr [ B` ′+1:`′
xx′ ]∣∣∣F`′] = E W `′+1 tr [ B` ′+1:`′ xx′ ] = n E g1,g2 1g1>01g1 cos ν`′+g1 sin ν`′>0
where (g1, g2) ∼ N (0, I). Moving to spherical coordinates, we obtain
E W `′+1
tr [ B` ′+1:`′
xx′
] = n
2π ∞∫ 0 π/2∫ −π2 +ν` ′ e−r 2/2rdrdθ = n 2 ( 1− ν `′ π ) .
We now note that conditioned on F`′ , tr [ PW 0
(:,1) >0PW 0 (:,1) cos ν+W 0 (:,2) sin ν>0
] is a sum of n in-
dependent variables taking values in {0, 1}. An application of Bernstein’s inequality for bounded random variables (lemma G.3) then gives
P [∣∣∆`′+1∣∣ > √nd] =P[ L−1∏
i=`′+1
( 1− ϕ (i)(ν)
π ) ∣∣∣∣∣tr [B`′+1:`′xx′ ]− n2 ( 1− ν `′ π )∣∣∣∣∣ > √nd ]
≤2 exp ( −c nd
n+ √ nd
) ≤ 2e−c ′d
(D.36) for some c′, where we used the fact that the angle evolution function ϕ is bounded by π/2. Note also from Lemma D.3 that
P ∣∣∣∣∣E [∆`′+1|F`′]− n2 L−1∏ i=`′ ( 1− ϕ (i)(ν) π )∣∣∣∣∣ > √ d3 log3(n)n L  =P n 2 L−1∏ i=`′+1 ( 1− ϕ (i)(ν) π ) ∣∣∣∣∣ν` ′ π − ϕ (`′)(ν) π ∣∣∣∣∣ > √ d3 log3(n)n L
 ≤e−cd
(D.37)
for some constant c, where we assumed d > K log n for some K.
Having controlled the first term in (D.35), we now proceed to bound the remaining terms. We define events
E`:` ′
B =
{∥∥α`−1(x)∥∥ 2 ∥∥α`−1(x′)∥∥ 2 > 0 } ∩ {∣∣ϕ(`−1)(ν)− ν`−1∣∣ ≤ C√d3 log3 nn` ∣∣∣∣} ∩ { tr [ B`−1:` ′
xx′
] ≤ Cn } ∩ {∥∥∥B`−1:`′xx′ ∥∥∥2 F ≤ C2n` } ∩ {∥∥∥B`−1:`′xx′ ∥∥∥ ≤ C`} ,
(which from lemma D.28 hold with high probability). Note that as a consequence of the first event in E`:`′B the angle ν` is well-defined. Note that E`:` ′ B is F`−1-measurable.
We will first control (D.35) by considering each summand truncated on the respective event E`:`′B . Our task is therefore to control
L∑ `=`′+2 1E`:`′B ∆` + L∑ `=`′+2 1E`:`′B E∆`|F`−1.
Since E [ 1E`:`′B ∆` ] = E [ E [ 1E`:`′B ∆` ∣∣∣F`−1]] = E [1E`:`′B E [∆`∣∣F`−1]] = 0, the first sum is over a zero-mean adapted sequence and hence a martingale, and can thus be controlled using the Azuma-Hoeffding inequality. We will first show that the remaining term is small. We begin by computing
1E`:`′B Etr [ B`:` ′ xx′ ] |F`−1 = E W ` tr [ 1E`:`′B B`−1:` ′ xx′ W `∗PW `α`−1(x′)>0PW `α`−1(x)>0W ` ]
where we used the fact that E`:`′B ∈ F`−1 and is thus independent ofW `. There exists a matrixR such that
Rα`−1(x) = ∥∥α`−1(x)∥∥
2 ê1,Rα
`−1(x′) = ∥∥α`−1(x′)∥∥
2
( ê1 cos ν `−1 + ê2 sin ν `−1) .
Rotational invariance of the Gaussian distribution gives
W `α`−1(x) d = W `(:,1) ∥∥α`(x)∥∥ 2 ,
W `α`−1(x′) d = ∥∥α`−1(x′)∥∥
2
( W `(:,1) cos ν `−1 +W `(:,2) sin ν `−1 ) ,
where we denote byW `(:,i) the i-th column ofW `. Defining B̃`−1:`
′ xx′ = RB `−1:`′ xx′ R ∗ we have
Etr [ B`:` ′
xx′
] |F`−1 = E W ` tr [ 1E`:`′B B̃`−1:` ′ xx′ W `∗PW ` (:,1) >0PW ` (:,1) cos ν`−1+W ` (:,2) sin ν`−1>0W ` ]
=1E`:`′B B̃`−1:`
′:`′
xx′ji E W ` ∑ ijk W `ki1W `k1>01W `k1 cos ν`−1+W `k2 sin ν`−1>0W ` kj
=1E`:`′B n∑ i,j=1 B̃`−1:` ′:`′ xx′ji E W ` nW `1i1W `11>01W `11 cos ν`−1+W `12 sin ν`−1>0W ` 1j
. = n∑ i,j=1 1E`:`′B B̃`−1:` ′:`′ xx′ji Q `−1 ij .
If i /∈ {1, 2} we get (with the square brackets denoting indicators)
Q`−1ij = E W `
2δij [ W `11 > 0 ] [ W `11 cos ν `−1 +W `12 sin ν `−1 > 0 ] = δij ( 1− ν `−1
π
) .
If i ∈ {1, 2} then the Q`−1ij 6= 0 only if j ∈ {1, 2}. In these cases we have
Q`−111 = E W `
n ( W `11 )2 [ W `11 > 0 ] [ W `11 cos ν `−1 +W `12 sin ν `−1 > 0 ] =2 E
W ` g21 [g1 > 0]
[ g1 cos ν `−1 + g2 sin ν `−1 > 0 ] where (g1, g2) ∼ N (0, I). Moving to spherical coordinates, we obtain
Q`−111 = 1
π ∞∫ 0 π/2∫ −π2 +ν`−1 e−r 2/2r3 cos2 θdrdθ = π − ν`−1 + sin ν`−1 cos ν`−1 π ,
and similarly
Q`−122 = 1
π ∞∫ 0 π/2∫ −π2 +ν e−r 2/2r3 sin2 θdrdθ = π − ν`−1 − sin ν`−1 cos ν`−1 π
Q`−112 = Q `−1 21 = E W ` nW `11
[ W `11 > 0 ] [ W `11 cos ν `−1 +W `12 sin ν `−1 > 0 ] W `12
= 1
π ∞∫ 0 π/2∫ −π2 +ν`−1 e−r 2/2r3 sin θ cos θdrdθ = 1 2π π/2∫ −π2 +ν`−1 sin θ cos θdθ = sin2 ν`−1 2π .
Combining terms and using tr [ B`−1:` ′
xx′
] = tr [ B̃`−1:` ′
xx′
] we obtain
1E`:`′B E [ tr [ B`:` ′ xx′ ] |F`−1 ] = 1E`:`′B  π − ν`−1 π tr [ B`−1:` ′ xx′ ] + sin ν`−1 cos ν`−1 π ( B̃`−1:` ′ xx′11 − B̃ `−1:`′ xx′22
) + sin2 ν`−1
2π
( B̃`−1:` ′
xx′12 + B̃ `−1:`′ xx′21
)
 ,
hence
1E`:`′B E W `
[ tr [ B`:` ′
xx′
] − ( 1− ϕ `−1(ν)
π
) tr [ B`−1:` ′
xx′
]]
=1E`:`′B  ϕ`−1(ν)−ν`−1 π tr [ B`−1:` ′ xx′ ] + sin ν `−1 cos ν`−1 π ( B̃`−1:` ′ xx′11 − B̃ `−1:`′ xx′22
) + sin 2 ν`−1
2π
( B̃`−1:` ′
xx′12 + B̃ `−1:`′ xx′21
)
On E`:`′B , the bound on ∣∣ϕ`−1(ν)− ν`−1∣∣ and lemma C.12 give ν`−1 ≤ C` a.s.. Additionally, on this
event max i,j∈[n] ∣∣∣B̃`−1:`′xx′ij ∣∣∣ ≤ ∥∥∥B`−1:`′xx′ ∥∥∥ ≤ C` a.s.. It follows that∣∣∣∣1E`:`′B EW ` [ tr [ B`:` ′ xx′ ] − ( 1− ϕ `−1(ν) π ) tr [ B`−1:` ′ xx′
]]∣∣∣∣ (D.38) ≤ C2 √ d3n log3 n
` +
2C2 π + C3 π` ≤ C ′
√ d3n log3 n
` (D.39)
almost surely, and hence restoring a constant factor with magnitude bounded by 1, we have
1E`:`′B ∣∣E∆`|F`−1∣∣ (D.40) =
L−1∏ i=` ( 1− ϕ (i)(ν) π ) ∣∣∣∣1E`:`′B EW ` [ tr [ B`:` ′ xx′ ] − ( 1− ϕ `−1(ν) π ) tr [ B`−1:` ′ xx′ ]]∣∣∣∣ (D.41) ≤ a.s. C ′ √ d3n log3 n ` . (D.42)
Using lemma D.28 to bound P [( E`:`′B )c] from above then gives
P ∣∣E∆`|F`−1∣∣ > C ′ √ d3n log3 n
`  < P [(E`:`′B )c] ≤ C ′n−cd. (D.43) An application of the triangle inequality and union bound then give
P [∣∣∣∣∣ L∑
`=`′+2
E∆`|F`−1 ∣∣∣∣∣ > C ′ √ d3Ln log3 n ] ≤P [ L∑
`=`′+2
∣∣E∆`|F`−1∣∣ > C ′√d3Ln log3 n]
≤ L∑
`=`′+2
P ∣∣E∆`|F`−1∣∣ > C ′ √ d3n log3 n
L  ≤
L∑ `=`′+2 P [( E`:` ′ B )c] ≤CLn−cd
(D.44) for some constants c, C. We proceed to control the remaining terms in (D.35), namely L∑
`=`′+2
∆`. Aiming to apply martingale
concentration, we require an almost sure bound on the summands, which we achieve by truncation. Towards this end, we define an event
G` = |∆`| ≤ C√d`+ C ′ √ d3n log3 n `  . Combining (D.43) and the result of lemma D.29 (after taking an expectation) we have
P [G`] ≥ 1− P ∣∣E∆`|F`−1∣∣ > C ′ √ d3n log3 n
`
− P [∣∣∆`∣∣ > C√d`]
≥ 1− C ′′n−cd − C ′′′e−c ′d ≥ 1− C ′′′′e−c ′d (D.45)
for appropriate constants. We now decompose the sum that we would like to bound:
∣∣∣∣∣ L∑
`=`′+2
∆` ∣∣∣∣∣ ≤
∣∣∣∣∣ L∑
`=`′+2
∆` −∆`1G` ∣∣∣∣∣︸ ︷︷ ︸ Σ1
+ ∣∣∣∣∣ L∑
`=`′+2
∆`1G` − E [ ∆`1G` |F`−1 ]∣∣∣∣∣︸ ︷︷ ︸ Σ2
+ ∣∣∣∣∣ L∑
`=`′+2
E [ ∆`1G` |F`−1 ] − E [ ∆`|F`−1 ]∣∣∣∣∣︸ ︷︷ ︸ Σ3
. (D.46)
Since each summand in Σ1 are equal to zero on the respective truncation event, a union bound and (D.45) give
P [∣∣∣∣∣ L∑
`=`′+2
∆` −∆`1G` ∣∣∣∣∣ > 0 ] ≤ P [ L⋃
`=`′+2
Gc`
] ≤
L∑ `=`′+2 P [Gc` ] ≤ LCe−cd (D.47)
for some constants. The term Σ2 is a sum of almost surely bounded martingale differences. We can apply the Azuma-Hoeffding inequality (lemma G.8) directly to conclude
P [∣∣∣∣∣ L∑
`=`′+2
∆`1G` − E∆`1G` |F`−1 ∣∣∣∣∣ > d2 √ nL log3 n logL ] (D.48)
≤ exp − d4nL log3 n logL 2
L∑ `=`′+2
( C √ d`+ C ′ √ d3n log3 n
`
)2  ≤ e−cd. (D.49)
Considering a single summand in Σ3, Jensen’s inequality and the Cauchy-Schwarz inequality give∣∣E [∆`1G` −∆`| F`−1]∣∣ = ∣∣E [∆`1Gc` ∣∣F`−1]∣∣ ≤ a.s. E [ |∆`|1Gc` ∣∣F`−1] ≤ a.s. ( E [ 1Gc`
∣∣F`−1])1/2 (E [∆2` ∣∣F`−1])1/2 . (D.50) This is an F`−1-measurable function, and we can show that it is small on the event E`:`′B ∈ F`−1. To control the first factor, we note that
1E`:`′B E [ 1Gc` ∣∣F`−1] =E [1Gc`∩E`:`′B ∣∣∣F`−1] =P |∆`| > C√d`+ C ′ √ d3n log3 n `  ∩ E`:`′B ∣∣∣∣∣∣F`−1
 ≤ P [{∣∣∣1E`:`′B ∆`∣∣∣ > C√d`+ C ′√d3n log3 n` } ∩ E`:`′B
∣∣∣∣F`−1] +P [{∣∣∣1(E`:`′B )c∆`∣∣∣ > 0} ∩ E`:`′B ∣∣∣F`−1]
≤P ∣∣∣1E`:`′B ∆`∣∣∣ > C√d`+ C ′ √ d3n log3 n
`
∣∣∣∣∣∣F`−1
≤ P [∣∣∣1E`:`′B ∆`∣∣∣ > C√d`∣∣∣F`−1]
+P [ 1E`:`′B ∣∣∣E∆`|F`−1∣∣∣ > C ′√d3n log3 n` ∣∣∣∣F`−1] ≤ a.s. P [∣∣∣1E`:`′B ∆`∣∣∣ > C√d`] ≤a.s. C ′e−cd
(D.51)
where to obtain the second to last line we used the definition of ∆`, then used Lemma D.29 and (D.40) to bound the first and second term almost surely.
We proceed to control the second factor in (D.50), by bounding 1E`:`′B E [ ∆2` ∣∣F`−1]
=1E`:`′B L−1∏ i=` ( 1− ϕ (i)(ν) π )2 E W ` [( tr [ B`:` ′ xx′ ] − ( 1− ϕ (`−1)(ν) π ) tr [ B`−1:` ′ xx′ ])2]
≤ a.s. 1E`:`′B E W `
[( tr [ B`:` ′
xx′
] − ( 1− ϕ (`−1)(ν)
π
) tr [ B`−1:` ′
xx′
])2]
≤ a.s. 4 E W `
 ( 1E`:`′B [ tr [ B`:` ′ xx′ ] − E W ` [ tr [ B`:` ′ xx′ ]]])2 + ( 1E`:`′B [ E W ` [ tr [ B`:` ′ xx′ ]] − ( 1− ϕ (`−1)(ν) π ) tr [ B`−1:` ′ xx′ ]])2  .
Using (D.38) and (D.148) to bound the integrand above, we have P [ 1E`:`′B E [ ∆2` ∣∣F`−1] > C (d`+ d3n log3 n
`
)] ≤ C ′e−cd
for appropriate constants. Combining (D.51) and the above bound gives
P 1E`:`′B ∣∣E [1G`∆` −∆`| F`−1]∣∣ > C √ d`+ d3n log3 n ` e−cd  ≤ C ′e−c′d for some c, c′, C, C ′, and using lemma D.28
P ∣∣E [1G`∆` −∆`| F`−1]∣∣ > C √ d`+ d3n log3 n
` e−cd  ≤ C ′e−c′d + P [(E`:`′B )c] ≤ C ′e−c ′d + C ′′n−c ′′d ≤ C ′′′e−c ′′′d
for appropriate constants. An application of the triangle inequality and a union bound (and introducing some slack to simplify the expression) then gives
P [∣∣∣∣∣ L∑
`=`′+2
E [ 1G`∆` −∆`| F`−1 ]∣∣∣∣∣ > CL √ d3n log3 ne−cd ] ≤ C ′Le−cd
for some constants. Combining this bound with (D.47) and (D.48) gives
P [∣∣∣∣∣ L∑
`=`′+2
∆` ∣∣∣∣∣ > d2 √ nL log3 n logL+ CL √ d3n log3 ne−cd ] ≤ C ′Le−c ′d + e−cd + C ′′Le−c ′′d ≤ C ′′′Le−c ′′′d
P [∣∣∣∣∣ L∑
`=`′+2
∆` ∣∣∣∣∣ > Cd2 √ Ln log3 n logL ] ≤ C ′Le−cd.
where in the last inequality we assumed K logL ≤ d. Combining this with (D.44), we obtain
P [∣∣∣∣∣ L∑
`=`′+2
∆` ∣∣∣∣∣ > Cd2 √ Ln log3 n logL ] ≤ C ′′Ln−c ′d + C ′′′Le−c ′′d ≤ C ′Le−cd
for appropriate constants. This bound all the terms in the sum (D.32) aside from the first one. The first term is bounded in (D.36), (D.37), and the fluctuations due to the last layer are bounded in (D.31). Combining all of these gives
P [∣∣∣∣∣〈β`′(x),β`′(x′)〉− n2 L−1∏ i=`′ ( 1− ϕ (i)(ν) π )∣∣∣∣∣ > Cd2 √ Ln log3 n logL ]
≤P [∣∣∣〈β`′(x),β`′(x′)〉− tr [BL−1:`′xx′ ]∣∣∣ > C4 d2 √ Ln log3 n logL ] + P [ |∆`′+1| ≤ C 3 d2 √ Ln log3 n logL ] + P [∣∣∣∣∣ L∑
`=`′+2
∆` ∣∣∣∣∣ ≤ C4 d2 √ Ln log3 n logL ]
+ P [∣∣∣∣∣ L∑
`=`′+2
E∆`|F`−1 ∣∣∣∣∣ ≤ C4 d2 √ Ln log3 n logL ] ≤ C ′Le−cd
after worsening constants. A final union bound over `′ and assuming d ≥ K logL gives
P [ L−1⋂ `′=0 {∣∣∣∣∣〈β`′(x),β`′(x′)〉− n2 L−1∏ i=`′ ( 1− ϕ (i)(ν) π )∣∣∣∣∣ ≥ Cd2 √ Ln log3 n logL }] ≥ 1−C ′e−cd
for appropriately chosen c, C ′,K. If we additionally assume n > Lwe obtain the desired result.

heading: D.3 UNIFORMIZATION ESTIMATES

heading: D.3.1 NETS AND COVERING NUMBERS

text: We appeal to Lemma C.4 to obtain estimates for the covering number of M, which we will use throughout this section. In the remainder of this section, we will use the notation Nε to denote the ε-nets for M constructed in Lemma C.4, and for any x̄ ∈ Nε, we will also use the notation Nε(x̄) = B(x̄, ε)∩M , where ∈ {+,−} is the component of x̄, to denote the relevant connected neighborhood of the specific point in the net we are considering. Here we are implicitly assuming thatM± are themselves connected, but this construction evidently generalizes to cases whereM± themselves have a positive number of connected components, as treated in Lemma C.4. Focusing on this simpler case in the sequel will allow us to keep our notation concise.

heading: D.3.2 CONTROLLING SUPPORT CHANGES UNIFORMLY

text:
The quantities we have studied in Appendix D.2 are challenging to uniformize due to discontinuities in the support projections PI`( · ). We will get around this difficulty by carefully tracking (with high probability) how much the supports can change by when we move away from the points in our net Nε. It seems intuitively obvious that when ε is exponentially small in all problem parameters, there should be almost no support changes when moving away from our net; the challenge is to show that this property also holds when ε is not so relatively small.
Introduce the following notation for the network preactivations at level `, where ` ∈ [L]:
ρ`(x) = W `α`−1(x),
so that α`(x) = [ρ`(x)]+. We also let F` denote the σ-algebra generated by all weight matrices up to level ` in the network, and let F0 denote the trivial σ-algebra.
Definition D.1. Let ε,∆ > 0, and let x̄ ∈ Nε. For ` ∈ [L], a feature (α`(x̄))i is called ∆-risky if |(ρ`(x̄))i| ≤ ∆; otherwise, it is called ∆-stable. If for all x ∈ Nε(x̄) we have
∀`′ ∈ [`], ∥∥ρ`(x)− ρ`(x̄)∥∥∞ ≤ ∆,
we say that stable sign consistency holds up to layer `. We abbreviate this condition as SSC(`, ε,∆) at x̄, with the dependence on x̄, ε, and ∆ suppressed when it is clear from context.
If SSC(`) holds at x̄ and if (α` ′ (x̄))i is stable, we can write for any x ∈ Nε(x̄) sign ( (ρ` ′ (x))i ) = sign ( (ρ` ′ (x̄))i + ( (ρ` ′ (x))i − (ρ` ′ (x̄))i )) = sign ( (ρ` ′ (x̄))i ) ,
so that no stable feature supports change on Nε(x̄), and we only need to consider changes due to the risky features. Moreover, observe that
P [ (ρ`(x̄))i ∈ {±∆} ] = E [ P [ ‖α`−1(x)‖2〈ei, g〉 ∈ {±∆} ∣∣ F`−1]] = 0, (D.52)
where g ∼ N (0, (2/n)I) is independent of everything else in the problem, since ∆ > 0. It follows that when considering the network features over any countable collection of points x̄ ∈M, we have almost surely that the risky features are witnessed in the interior of [−∆,+∆]. Below, we will show that with appropriate choices of ε and ∆, with very high probability: (i) each point in the net x̄ has very few risky features; and (ii) SSC(L) holds uniformly over the net under reasonable conditions involving n,L, d. We write R`(x̄,∆) ⊂ [n] for the random variable consisting of the set of indices of ∆-risky features at level ` with input x̄ ∈ Nε.
Lemma D.5. There is an absolute constant K > 0 such that for any x̄ ∈ M and any d > 0, if n ≥ max{KdL, 4} and ∆ ≤ d log n/(6n3/2L), then one has
P [ L∑ `=1 |R`(x̄,∆)| > d log n ] ≤ 2n−d + L2e−cn/L.
Proof. For any x̄ ∈ Nε, Lemma D.2 (with a suitable choice of d in that context) gives P [∣∣∥∥α`(x̄)∥∥ 2 − 1 ∣∣ > 1
2
] ≤ C`e−cn` ,
so that if additionally n ≥ (2/c)` log(C), one has P [∣∣∥∥α`(x̄)∥∥ 2 − 1 ∣∣ > 1
2
] ≤ `e−cn` . (D.53)
Let G` = {1/2 ≤ ‖α`(x̄)‖2 ≤ 2}, so that G` is F`-measurable, and G = ∩`∈[L−1]G`; then by (D.53) and a union bound, we have P[G] ≥ 1 − L2e−cn/L. We also let G0 = ∅c. For i ∈ [n] and ` ∈ [L], consider the random variables Xi` = |(ρ`(x̄))i|, and moreover define
X̃i` = Xi`
‖α`−1(x̄)‖2 1G`−1 . We have ∑ i,` 1Xi`≤∆ = ∑ `|R`(x̄)|, which is the total number of ∆-risky features at x̄, and the
corresponding sum with the random variables X̃i` is thus an upper bound on the number of risky features at x̄. Notice that Xi` and X̃i` are F`-measurable, and additionally notice that on G, we have Xi`/2 ≤ X̃i` ≤ 2Xi`. For any K ∈ {0, 1, . . . , nL − 1, nL}, we have by disjointness of the events in the union and a partition
P ∑ i,` 1Xi`≤∆ > K  ≤ L2e−cn/L + nL∑ k=K+1 P G ∩ ∑ i,` 1Xi`≤∆ = k
≤ L2e−cn/L + nL∑
k=K+1
P G ∩ ∑
i,`
1X̃i`≤2∆ = k  ,
so it is essentially equivalent to consider the X̃i`. By another partitioning we can write
P G ∩ ∑
i,`
1X̃i`≤2∆ = k   = ∑
S∈{0,1}n×L : ‖S‖2F=k
E [ L∏ `=1 ( 1G`−1 n∏ i=1 11X̃i`≤2∆=Si` )]
where {0, 1}n×L is the set of n × L matrices with entries in {0, 1}. Using the tower rule and FL−1-measurability of all factors with ` < L, we can then write
E [ L∏ `=1 1G`−1 n∏ i=1 11X̃i`≤2∆=Si` ]
= E [ E [ L∏ `=1 ( 1G`−1 n∏ i=1 11X̃i`≤2∆=Si` ) ∣∣∣∣∣ FL−1 ]]
= E [( L−1∏ `=1 ( 1G`−1 n∏ i=1 11X̃i`≤2∆=Si` )) 1GL−1E [ n∏ i=1 11X̃iL≤2∆=SiL ∣∣∣∣∣ FL−1 ]] .
We study the inner conditional expectation as follows: because ρL(x̄) = WLαL−1(x̄), we can apply rotational invariance in the conditional expectation to obtain
1GL−1E [ n∏ i=1 11X̃iL≤2∆=SiL ∣∣∣∣∣ FL−1 ] = 1GL−1E [ n∏ i=1 11|(wL)i|1G`−1≤2∆ =SiL ∣∣∣∣∣ FL−1 ]
= 1GL−1E [ n∏ i=1 11|(wL)i|≤2∆=SiL ∣∣∣∣∣ FL−1 ] ,
where wL ∼ N (0, (2/n)I) is the first column of WL, and the last equality takes advantage of the presence of the indicator for 1G`−1 multiplying the conditional expectation. We then write using independence
E [ n∏ i=1 11|(wL)i|≤2∆=SiL ∣∣∣∣∣ FL−1 ] = P [ n⋂ i=1 { 1|(wL)i|≤2∆ = SiL } ∣∣∣∣∣ FL−1 ]
= n∏ i=1 P [ 1|(wL)i|≤2∆ = SiL ∣∣ FL−1], and putting pL = P[|(wL)1| ≤ 2∆], we have by identically-distributedness
n∏ i=1 P [ 1|(wL)i|≤2∆ = SiL ∣∣ FL−1] = n∏ i=1 pSiLL (1− pL) 1−SiL .
After removing the indicator for GL−1 by nonnegativity of all factors in the expectation, this leaves us with
E [ L∏ `=1 1G`−1 n∏ i=1 11X̃i`≤2∆=Si` ] ≤ ( n∏ i=1 pSiLL (1− pL) 1−SiL ) E [( L−1∏ `=1 1G`−1 n∏ i=1 11X̃i`≤2∆=Si` )] .
This process can evidently be iteratedL−1 additional times with analogous definitions—we observe that the fact that all weight matricesW ` have the same column distribution implies that p1 = · · · = pL, so we write p = p1 henceforth—and by this we obtain
E [ L∏ `=1 1G`−1 n∏ i=1 11X̃i`≤2∆=Si` ] ≤ L∏ `=1 n∏ i=1 pSiL(1− p)1−SiL ,
and in particular
P G ∩ ∑
i,`
1X̃i`≤2∆ = k   ≤ ∑
S∈{0,1}n×L : ‖S‖F=k
L∏ `=1 n∏ i=1 pSiL(1− p)1−SiL .
For i ∈ [n] and ` ∈ [L], let Yi` denote nL i.i.d. Bern(p) random variables; we recognize this last sum as the probability that ∑ i,` Yi` = k. In particular, using our previous work we can assert for any t > 0
P ∑ i,` 1Xi`≤∆ > t  ≤ P ∑ i,` Yi` > t + L2e−cn/L, so to conclude it suffices to articulate some binomial tail probabilities and estimate p. We have
p = P[|(wL)i| ≤ 2∆] = √ n
2π ∫ 2∆ 0 exp ( −nt 2 4 ) dt
≤ ∆ √ 2n
π ,
(D.54)
and we can write with the triangle inequality and a union bound
P ∑ i,` Yi` > t  ≤ P ∣∣∣∣∣∣ ∑ i,` Yi` − E[Yi`] ∣∣∣∣∣∣ > t + P ∑ i,` E[Yi`] > t .
By (D.54), we have ∑ i,` E[Yi`] ≤ n3/2L∆. We calculate using independence
E  ∑
i,`
Yi` − E[Yi`] 2  ≤∑
i,`
E[Yi`] ≤ n3/2L∆,
so an application of Lemma G.3 yields
P ∣∣∣∣∣∣ ∑ i,` Yi` − E[Yi`] ∣∣∣∣∣∣ > t  ≤ 2 exp(− t2/2 n3/2L∆ + t/3 ) .
For any d > 0, if we choose t = d log n and enforce ∆ ≤ d log n/(6n3/2L), we obtain
P ∑ i,` Yi` > d log n  ≤ 2n−d, from which we conclude as sought
P ∑ i,` 1Xi`≤∆ > d log n  ≤ 2n−d + L2e−cn/L.
The next task is to study the stable sign condition at a point x̄ as a function of ε and ∆, assuming ∆ at least satisfies the hypotheses of Lemma D.5. In particular, we will be interested in conditions under which we can guarantee that SSC(`− 1) holding implies that SSC(`) holds. Let S`(x̄,∆) = [n] \R`(x̄,∆) denote the ∆-stable features at level ` with input x̄, and define for 0 ≤ `′ ≤ ` ≤ L
T`,` ′
x = PS`(x̄)PI`(x)W `PS`−1(x̄)PI`−1(x)W `−1 . . .PS`′+1(x̄)PI`′+1(x)W `′+1;
Φ`,` ′ x = W `T`−1,` ′ x , (D.55)
so that Φ`,` ′ x x carries an input x ∈ Nε(x̄) applied at the features at level `′ (in particular, `′ = 0 corresponds to α0(x) = x, the network input) to the preactivations at level ` in a network restricted to only the stable features at x̄. We can write
ρ`(x) = W `PI`−1(x)W `−1 . . .PI1(x)W 1x;
α`(x) = PI`(x)W `PI`−1(x)W `−1 . . .PI1(x)W 1x,
which gives us a useful representation if we disregard all levels with no risky features: let r =∑L `=1 1|R`(x̄,∆)|>0 be the number of levels in the network with risky features, and let `1 < `2 < · · · < `r denote the levels at which risky features occur. If no risky features occur at a level `, we of course have PS`(x̄) = I . Assume to begin that ` > `r, and start by writing
ρ`(x) = Φ`,`rx ( PS`r (x̄) + PR`r (x̄) ) PI`r (x)Φ `r,`r−1 x ( PS`r−1(x̄) + PR`r−1(x̄) ) PI`r−1(x) . . .
. . .Φ`2,`1x ( PS`1 (x̄) + PR`1 (x̄) ) PI`1 (x)Φ `1,0 x .
Now we distribute from left to right, and recombine everything to right on the term corresponding to the projection onto the risky features at `r; this gives
ρ`(x) = Φ`,`rx PR`r (x̄)α `r (x) + Φ`,`r−1x ( PS`r−1(x̄) + PR`r−1(x̄) ) PI`r−1(x) . . .
. . .Φ`2,`1x ( PS`1 (x̄) + PR`1 (x̄) ) PI`1 (x)Φ `1,0 x .
We can write Φ`,`rx PR`r (x̄)α `r (x) = Φ`,`rx ∣∣ R`r (x̄) α`r (x) ∣∣ R`r (x̄) ,
where the restriction notation emphasizes that we are considering a column submatrix of the transfer operator induced by the risky features. Iterating the previous argument, we obtain
ρ`(x) = Φ`,0x x+ r∑ i=1 Φ`,`ix ∣∣ R`i (x̄) α`i(x) ∣∣ R`i (x̄) .
It is clear that an analogous argument can be used in the case where ` ≤ `r by adapting which risky features can be visited: we can thus assert
ρ`(x) = Φ`,0x x+ ∑
i∈[r] : `i<`
Φ`,`ix ∣∣ R`i (x̄) α`i(x) ∣∣ R`i (x̄) . (D.56)
Furthermore, we note that under SSC(` − 1), no stable feature supports change on Nε(x̄), and so one has for every x ∈ Nε(x̄)
Φ`,` ′ x̄ = Φ `,`′ x ,
so under SSC(`− 1) we have by (D.56) ρ`(x)− ρ`(x̄) = Φ`,0x̄ (x− x̄) + ∑
i∈[r] : `i<`
Φ`,`ix̄ ∣∣∣ R`i (x̄) ( α`i(x) ∣∣ R`i (x̄) −α`i(x̄) ∣∣ R`i (x̄) ) .
The ReLU [·]+ is 1-Lipschitz with respect to ‖·‖∞, and by monotonicity of the max under restriction and SSC(`− 1) we have∥∥∥α`i(x)∣∣
R`i (x̄) −α`i(x̄) ∣∣ R`i (x̄) ∥∥∥ ∞ ≤ ∥∥∥ρ`i(x)∣∣ R`i (x̄) − ρ`i(x̄) ∣∣ R`i (x̄) ∥∥∥ ∞
≤ ∥∥ρ`i(x)− ρ`i(x̄)∥∥∞ ≤ ∆.
Thus, by the triangle inequality, we have under SSC(`− 1) a bound∥∥ρ`(x)− ρ`(x̄)∥∥∞ ≤ ε∥∥∥Φ`,0x̄ ∥∥∥`2→`∞ + ∆ ∑ i∈[r] : `i<` ∥∥∥∥Φ`,`ix̄ ∣∣∣ R`i (x̄) ∥∥∥∥ `∞→`∞ . (D.57)
This suggests an inductive approach to establishing SSC(`) provided we have established it at previous layers—we just need to control the transfer coefficients in (D.57).
Lemma D.6. There are absolute constants c, c′, C, C ′, C ′′, C ′′′ > 0 and absolute constants K,K ′ > 0 such that for any 1 ≤ `′ < ` ≤ L, any d ≥ K log n and any x̄ ∈ Sn0−1, if ∆ ≤ cn−5/2 and n ≥ K ′max{d4L, 1}, then one has
P [∥∥∥Φ`,0x̄ ∥∥∥
`2→`∞ ≤ C(1 + √ n0 n ) ] ≥ 1− C ′′e−cd,
and for any fixed S ⊂ [n], one has
P [∥∥∥Φ`,`′x̄ ∣∣∣ S ∥∥∥ `∞→`∞ ≤ C ′|S| √ d n ] ≥ 1− C ′′′e−c ′d.
Proof. We will use Lemmas D.14 and D.23 to bound the transfer coefficients, so let us first verify the hypotheses of these lemmas. In our setting, the transfer matrices differ only from the ‘nominal’ transfer matrices by restriction to the stable features at x̄; we have S`(x̄) ∩ I`(x̄) = [n] \ R`(x̄), which is an admissible support random variable for Lemmas D.14 and D.23, and in particular∥∥(PS`(x̄)PI`(x̄) − PI`(x̄))ρ`(x̄)∥∥2 = ∥∥PR`(x̄)α`(x̄)∥∥2 ≤ √n∆ by Lemma G.10 and the definition ofR`(x̄). Additionally, using Lemma D.5, we have if d ≥ 1, n ≥ KdL, and ∆ ≤ cd/n5/2, there is an event E with measure at least 1− 2e−d − L2e−cn/L on which there are no more than d risky features at x̄. Worsening constants in the scalings of n if necessary and requiring moreover d ≥ K ′ log n and n ≥ K ′′d4, it follows that we can invoke Lemmas D.14 and D.23 to bound the probability of events involving transfer coefficients multiplied by 1E . Let us also check the residuals we will obtain when applying Lemma D.23: in the notation there, the vector d has as its `-th entryR`(x̄), and so we have bounds ‖d‖1/2 ≤ ‖d‖21 and ‖d‖1 = ∑ `R`(x̄),
which means on the event E , the residual is dominated by the C √ d4nL term in the scalings we have assumed.
The `2 → `∞ operator norm of a matrix is the maximum `2 norm of a row of the matrix, and the `∞ → `∞ operator norm is the maximum `1 norm of a row. Thus∥∥∥Φ`,0x̄ ∥∥∥
`2→`∞ = max i=1,...,n ∥∥∥e∗iΦ`,0x̄ ∥∥∥ 2
= max i=1,...,n ∥∥∥(W `)∗iT`−1,0x̄ ∥∥∥ 2
where (W `)∗i is the i-th row of W `, which is n0-dimensional when ` = 1 and n-dimensional otherwise. In particular, we have∥∥∥Φ1,0x̄ ∥∥∥ `2→`∞ = max i=1,...,n
∥∥(W 1)i∥∥2, and so taking a square root and applying Lemma G.2 and independence of the rows ofW 1, we have
P [∥∥∥Φ1,0x̄ ∥∥∥
`2→`∞ ≤ 2(1 + √ n0 n ) ] ≥ 1− 2ne−cn,
for c > 0 an absolute constant. When ` > 1, we can write
max i=1,...,n ∥∥∥(W `)∗iT`−1,0x̄ ∥∥∥ 2 = max i=1,...,n ∥∥∥(W `)∗iT`−1,1x̄ PI1(x̄)W 1∥∥∥ 2
≤ ∥∥W 1∥∥ max
i=1,...,n ∥∥∥(W `)∗iT`−1,1x̄ PI1(x̄)∥∥∥ 2 ,
where the second line applies Cauchy-Schwarz. Using, say, rotational invariance, Gauss-Lipschitz concentration, and Lemma E.48 (or (Vershynin, 2018, Theorem 4.4.5)), we have
P [∥∥W 1∥∥ > C(1 +√n0
n )
] ≤ 2e−cn
for absolute constants c, C > 0. On the other hand, note that ‖(W `)∗iT `−1,1 x̄ PI1(x̄)‖2 has the same distribution as the square root of one of the index-0 diagonal terms studied in Lemma D.23 in a network truncated at level `− 1 instead of L and scaled by 2/n; and so applying this result together with a union bound and the choice n ≥ max{K,K ′d4L} for absolute constants K,K ′ > 0 gives
P [ 1E max
i=1,...,n ∥∥∥(W `)∗iT`−1,1x̄ PI1(x̄)∥∥∥ 2 > C ′ ] ≤ C ′′ne−c ′d
where C ′, c′, C ′′ > 0 are absolute constants. We conclude by another union bound P [∥∥∥Φ`,0x̄ ∥∥∥
`2→`∞ ≤ C(1 + √ n0 n ) ] ≥ 1− 2e−cn − C ′ne−c ′d − C ′′L2e−c ′′n/L.
We can reduce the study of the partial risky propagation coefficients to a similar calculation. We have ∥∥∥Φ`,`′x̄ ∣∣∣
S ∥∥∥ `∞→`∞ = max j=1,...,n ∥∥∥(W `)∗j (T`−1,`′x̄ ∣∣∣ S )∥∥∥ 1 ,
where by construction we have that ` > `′. In the case ` = `′ + 1, the form is slightly different; we have ∥∥∥Φ`′+1,`′x̄ ∣∣∣
S ∥∥∥ `∞→`∞ = max j=1,...,n ∥∥∥∥(W `′+1∣∣∣ S ) j ∥∥∥∥ 1 ≤ |S| max j=1,...,n ∥∥∥∥(W `′+1∣∣∣ S ) j ∥∥∥∥ ∞ ,
where the inequality uses Lemma G.10. The classical estimate for the gaussian tail gives
P [∣∣∣(W `′+1)jk∣∣∣ >√2d n ] ≤ 2e−d, (D.58)
for each k ∈ [n], so a union bound gives
P [ max
j=1,...,n ∥∥∥∥(W `′+1∣∣∣ S ) j ∥∥∥∥ ∞ > √ 2d n ] ≤ 2ne−d,
and we conclude
P [∥∥∥Φ`′+1,`′x̄ ∣∣∣ S ∥∥∥ `∞→`∞ ≤ |S| √ 2d n ] ≥ 1− 2e−d/2,
where the final bound holds if d ≥ 2 log n. Next, we assume ` > `′ + 1. In this case, Lemma G.10 gives ∥∥∥Φ`,`′x̄ ∣∣∣
S ∥∥∥ `∞→`∞ = max j=1,...,n ∥∥∥(W `)∗j (T`−1,`′x̄ ∣∣∣ S )∥∥∥ 1
≤ |S| max j=1,...,n ∥∥∥(W `)∗j ((T`−1,`′x̄ ∣∣∣ S ))∥∥∥ ∞ .
For the second term on the RHS of the inequality, we write
max j=1,...,n ∥∥∥(W `)∗jT`−1,`′x̄ ∣∣∣ S ∥∥∥ ∞ = max j=1,...,n max k∈S ∣∣∣(W `)∗jT`−1,`′x̄ ek∣∣∣ then apply rotational invariance of the distribution of (W `)j and F`−1-measurability of T`−1,` ′
x̄
∣∣∣ S
to obtain
max j=1,...,n max k∈S ∣∣∣(W `)∗jT`−1,`′x̄ ek∣∣∣ = max j=1,...,n
max k∈S : ∥∥∥T`−1,`′x̄ ek∥∥∥ 2 >0 ∣∣∣(W `)∗jT`−1,`′x̄ ek∣∣∣ d = max j=1,...,n max k∈S |gj | ∥∥∥T`−1,`′x̄ ek∥∥∥ 2 ,
where g ∼ N (0, (2/n)I) is independent of everything else in the problem. We have by Lemma D.14 based on our previous choices of n and d
P [ 1E ∥∥∥T`−1,`′x̄ ek∥∥∥ 2 ≤ C ] ≥ 1− C ′e−cn/L,
and (D.58) applied to g controls the remaining term. Taking a union bound over [n] in these two estimates and partitioning with E , we conclude
P [ max
j=1,...,n ∥∥∥(W `)∗j (T`−1,`′x̄ ∣∣∣ S )∥∥∥ ∞ > C √ d n ] ≤ 2ne−d + C ′ne−cn/L + 2e−d + L2e−c ′n/L
and thus
P [∥∥∥Φ`,`′x̄ ∣∣∣ S ∥∥∥ `∞→`∞ > C|S| √ d n ] ≤ C ′e−d/2 + C ′′n2e−cn/L,
where the last bound holds if d ≥ 2 log n. Choosing n ≥ KL log n for a suitable absolute constant K > 0 allows us to simplify the residual terms in the probability bounds to the forms we have claimed.
Lemma D.7. There is an absolute constant c > 0 and absolute constants k, k′,K,K ′ > 0 such that for any d ≥ K log n, if n ≥ K ′max{d4L, 1}, ∆ ≤ kn−5/2, and ε ≤ k′∆ ( 1 + √ n0 n )−1 , then one has for any x̄ ∈ Nε
P [ {SSC(L) holds at x̄} ∩ { L∑ `=1 |R`(x̄,∆)| ≤ d }] ≥ 1− e−cd.
Proof. We start by constructing a high-probability event on which we have control of every possible propagation coefficient. For any d ≥ K log n, choosing ∆ ≤ cn−5/2 and n ≥ K ′d4L and applying the first conclusion in Lemma D.6 and a union bound, we have
P [ ∃` ∈ [L], ∥∥∥Φ`,0x̄ ∥∥∥ `2→`∞ > C1(1 + √ n0 n ) ] ≤ Ce−cd (D.59)
and under the same hypotheses, for any 1 ≤ `′ < ` ≤ L and any S ⊂ [n], the second conclusion in Lemma D.6 gives
P [∥∥∥Φ`,`′x̄ ∣∣∣ S ∥∥∥ `∞→`∞ > C2|S| √ d n ] ≤ C ′e−16d.
Using Lemma D.5, we have if n ≥ max{KdL, 4} and ∆ ≤ K ′/n5/2
P [ L∑ `=1 |R`(x̄,∆)| > d ] ≤ 2e−d + L2e−cn/L. (D.60)
Denote the complement of the event in the previous bound as E . On E , there are no more than d levels in the network with risky features. There are at most ∑dde k=0 ( n k ) ways to choose a subset S ⊂ [n] with cardinality at most dde; using n ≥ e and d ≥ 1, we have dde∑ k=0 ( n k ) ≤ 1 + ddendde ≤ 4dn2d.
In addition, there are at most L2 ways to pick two indices 1 ≤ `′ < ` ≤ L. Using n ≥ L, this yields at most 4dn2+2d ≤ n8d items to union bound over, i.e.,
P  ⋃ S⊂[n] |S|≤dde ⋃ 1≤`<`′≤L {∥∥∥Φ`,`′x̄ ∣∣∣ S ∥∥∥ `∞→`∞ > C2|S| √ d n } ≤ Ce−8d (D.61) if d ≥ K log n and n ≥ max{K ′d4L, n0}. Denote the complement of the union of the events in the bounds (D.59) to (D.61) and E as G; taking additional union bounds and worst-casing absolute constants, we have shown P[G] ≥ 1− Ce−cd. If we enumerate the levels of the network that have risky features as 1 ≤ `1 < · · · < `r ≤ L, it follows from our previous counting argument that on G, we have transfer coefficient bounds (for any ` ∈ [L] and any `i < `)∥∥∥Φ`,0x̄ ∥∥∥
`2→`∞ ≤ C1(1 + √ n0 n ),
∥∥∥∥Φ`,`ix̄ ∣∣∣ R`i (x̄) ∥∥∥∥ `∞→`∞ ≤ C2|R`i(x̄)| √ d n .
Now we begin the induction. Let x ∈ Nε(x̄). For SSC(1), we have from the definitions∥∥ρ1(x)− ρ1(x̄)∥∥∞ ≤ ε∥∥∥Φ1,0x̄ ∥∥∥`2→`∞ ≤ C1(1 + √ n0 n )ε,
where the last inequality holds on G. So we have SSC(1) on G if ε ≤ ∆(C1(1 + √ n0 n )) −1. Continuing, we suppose that we have established SSC(` − 1) on G. We can therefore apply (D.57) together with our transfer coefficient bounds to get∥∥ρ`(x)− ρ`(x̄)∥∥∞ ≤ C1(1 +√n0n )ε+ C2∆ √ d n ∑ i∈[r] : `i<` |R`i(x̄)|
≤ C1(1 + √ n0 n )ε+ C2∆ √ d3 n .
Notice that the last bound does not depend on `. Thus, if we choose ε ≤ ∆(2C1(1 + √ n0 n )) −1 and
n ≥ 4C22d3, we obtain ∥∥ρ`(x)− ρ`(x̄)∥∥∞ ≤ ∆; by induction, we can conclude that SSC(L) holds on G, which implies the claim; we obtain the final simplified probability bound by worsening the constant in the exponent.

heading: D.3.3 UNIFORMIZING FORWARD FEATURES UNDER SSC

text:
Under the SSC(L) condition, we can uniformize forward and backward features. A prerequisite of our approach, which we also used to establish SSC(L) in the previous section, is control of certain residuals that appear when a small number of supports can change off the nominal forward and backward correlations. These estimates are studied in the next section, Appendix D.3.4.
In previous sections, most of our results (e.g. Lemma D.1) feature a lower bound of the type n ≥ K in their hypotheses. After uniformizing, we will discard this hypothesis using our extra assumption
that n0 ≥ 3, which gives us a lower bound on the logarithmic terms of the form log(Cnn0) that appear as lower bounds on d after uniformizing, and the fact that our lower bounds on n always involve a polynomial in d. Thus, by adjusting absolute constants, we can achieve the same effect as previously.
Lemma D.8. There are absolute constants c, C > 0 and an absolute constant K,K ′ > 0 such that for any d ≥ Kd0 log(nn0CM), if n ≥ K ′d4L then one has
P  ⋂ x̄∈N
n−3n−1/20
{ SSC(L, n−3n
−1/2 0 , Cn
−3) holds at x̄ } ∩ { L∑ `=1 |R`(x̄, Cn−3)| ≤ d } ≥ 1− e−cd.
Proof. Following the discussion in Appendix D.3.1, if 0 < ε ≤ 1 then |Nε| ≤ ed0 log(CM/ε); to apply Lemma D.7 we at least need ∆ ≤ kn−5/2 and ε ≤ k′∆ ( 1 + √ n0 n )−1 , so it suffices to put ∆ = Cn−3 when n is chosen larger than an absolute constant, and require ε ≤ min { 1, k′Cn−5/2 ( 1 + √ n0 n )−1} . Fixing ε = n−3n−1/20 , which again is admissible when n is sufficiently large compared to an absolute constant, for any d ≥ Kd0 log(nn0CM), we choose n ≥ K max{1, d4L} and take a union bound to obtain the claim (using here that CM ≥ 1).
Lemma D.9. There is an absolute constant c > 0 and absolute constants K,K ′,K ′′ > 0 such that for any d ≥ Kd0 log(nn0CM), if n ≥ Kd4L, then one has
P [ ⋂ x∈M { ∀` ∈ [L], ∣∣‖α`(x)‖2 − 1∣∣ ≤ 1 2 }] ≥ 1− e−cd.
Proof. Let x̄ ∈ N n−3n
−1/2 0
. Lemma D.2 and a union bound give
P [ ∃` ∈ [L] : ∣∣∥∥α`(x̄)∥∥ 2 − 1 ∣∣ > 1
4
] ≤ C ′L2e−cn/L ≤ e−c ′d
if n ≥ KdL and d ≥ K ′ log n. If additionally d ≥ K ′d0 log(nn0CM), we obtain by the discussion in Appendix D.3.1 and another union bound
P  ⋃ x̄∈N
n−3n−1/20
{ ∃` ∈ [L] : ∣∣∥∥α`(x̄)∥∥ 2 − 1 ∣∣ > 1
4 } ≤ e−cd. Let E denote the event studied in Lemma D.8; choose d ≥ Kd0 log(nn0CM) and n sufficiently large to make the measure bound applicable. A union bound gives
P Ec ∪ ⋃ x̄∈N
n−3n−1/20
{ ∃` ∈ [L] : ∣∣∥∥α`(x̄)∥∥ 2 − 1 ∣∣ > 1
4 } ≤ e−cd. Let G denote the complement of the event in the previous bound. For any x ∈ M, we can find a point x̄ ∈ N
n−3n −1/2 0 ∩N n−3n −1/2 0 (x). On G, SSC(L, n−3n−1/20 , Cn−3) holds at every point in the net N
n−3n −1/2 0
, which implies that on G
∀` ∈ [L], ∥∥ρ`(x)− ρ`(x̄)∥∥∞ ≤ Cn3 , (D.62)
and by the 1-Lipschitz property of [ · ]+ and Lemma G.10, this also implies that on G
∀` ∈ [L], ∥∥α`(x)−α`(x̄)∥∥
2 ≤ C n5/2 .
Choosing n ≥ (4C)2/5, the RHS of this bound is no larger than 1/4. We write using the triangle inequality ∣∣‖α`(x)‖2 − 1∣∣ ≤ ∣∣‖α`(x)‖2 − ‖α`(x̄)‖2∣∣+ ∣∣‖α`(x̄)‖2 − 1∣∣. Using the triangle inequality again, the first term on the RHS is no larger than 1/4 for any ` on G. The second term is also no larger than 1/4 on G by control over the net. We conclude that on G
∀` ∈ [L], ∣∣∥∥α`(x)∥∥ 2 − 1 ∣∣ ≤ 1
2 .
This implies that the event G is contained in the set⋂ x∈M { ∀` ∈ [L], ∣∣‖α`(x)‖2 − 1∣∣ ≤ 1 2 } ,
which is closed, by continuity of ‖ · ‖2 and of the features as a function of the parameters, and is therefore also an event. The claim follows.
Lemma D.10. There are absolute constants c, C > 0 and absolute constants K,K ′ > 0 such that for any d ≥ Kd0 log(nn0CM), if n ≥ K ′d4L, then one has
P  ⋂ (x,x′)∈M×M { ∀` ∈ [L], ∣∣∣〈α`(x),α`(x′)〉− cosϕ(`)(∠(x,x′))∣∣∣ ≤ C√d3` n } ≥ 1− e−cd. Proof. Let x̄, x̄′ ∈ N
n−3n −1/2 0
. Lemma D.1 and a union bound give
P [ ∃` ∈ [L] : ∣∣∣〈α`(x̄),α`(x̄′)〉− cosϕ(`)(∠(x̄, x̄′))∣∣∣ > C√d3` n ] ≤ C ′Le−cd ≤ e−c ′d
if d ≥ K log n and n ≥ max{K ′d3L,K ′′d4,K ′′′}. If additionally d ≥ Kd0 log(nn0CM), we obtain by the discussion in Appendix D.3.1 and another union bound
P  ⋃ (x̄,x̄′)∈N×21
n3 √ n0
{ ∃` ∈ [L] : ∣∣∣〈α`(x̄),α`(x̄′)〉− cosϕ(`)(∠(x̄, x̄′))∣∣∣ > C√d3` n } ≤ e−cd, where with an abuse of notation we write S×2 to denote S × S for a set S. Let E1 denote the event studied in Lemma D.8, and let E2 denote the event studied in Lemma D.9; choose n sufficiently large to make the measure bounds applicable. A union bound gives
P Ec1 ∪ Ec2 ∪ ⋃ (x̄,x̄′)∈N×21
n3 √ n0
{ ∃` ∈ [L] : ∣∣∣〈α`(x̄),α`(x̄′)〉− cosϕ(`)(∠(x̄, x̄′))∣∣∣ > C√d3` n } ≤ e−cd
after adjusting constants. Let G denote the complement of the event in the previous bound. For any (x,x′) ∈ M × M, we can find a point x̄ ∈ N
n−3n −1/2 0
∩ N n−3n
−1/2 0
(x) and a point
x̄′ ∈ N n−3n −1/2 0 ∩ N n−3n −1/2 0 (x′). On G, SSC(L, n−3n−1/20 , Cn−3) holds at every point in the net N
n−3n −1/2 0
, which implies that on G
∀` ∈ [L], ∥∥ρ`(x)− ρ`(x̄)∥∥∞ ≤ Cn3 , and ∀` ∈ [L], ∥∥ρ`(x′)− ρ`(x̄′)∥∥∞ ≤ Cn3 ,
and by the 1-Lipschitz property of [ · ]+ and Lemma G.10, this also implies that on G
∀` ∈ [L], ∥∥α`(x)−α`(x̄)∥∥
2 ≤ C n5/2
, and ∀` ∈ [L], ∥∥α`(x′)−α`(x̄′)∥∥
2 ≤ C n5/2 . (D.63)
For any ` ∈ [L], we write using the triangle inequality∣∣∣〈α`(x),α`(x′)〉− cosϕ(`)(∠(x,x′))∣∣∣ ≤ ∣∣〈α`(x),α`(x′)〉− 〈α`(x̄),α`(x′)〉∣∣ + ∣∣〈α`(x̄),α`(x′)〉− 〈α`(x̄),α`(x̄′)〉∣∣
+ ∣∣∣〈α`(x̄),α`(x̄′)〉− cosϕ(`)(∠(x̄, x̄′))∣∣∣
+ ∣∣∣cosϕ(`)(∠(x̄, x̄′))− cosϕ(`)(∠(x,x′))∣∣∣.
(D.64)
Using Cauchy-Schwarz, we have on G∣∣〈α`(x),α`(x′)〉− 〈α`(x̄),α`(x′)〉∣∣ ≤ ∥∥α`(x)−α`(x̄)∥∥ 2 ∥∥α`(x′)∥∥ 2 ≤ 2C n5/2 ,
with the same bound holding for the second term in (D.64) by an analogous argument. For the third term, we have on G ∣∣∣〈α`(x̄),α`(x̄′)〉− cosϕ(`)(∠(x̄, x̄′))∣∣∣ ≤ C√d3`
n .
For the last term, we use 1-Lipschitzness of cos and 1-Lipschitzness of the ϕ(`), which follows from Lemma E.5 and the chain rule, to obtain∣∣∣cosϕ(`)(∠(x̄, x̄′))− cosϕ(`)(∠(x,x′))∣∣∣ ≤ |∠(x̄, x̄′)− ∠(x,x′)|. Using Lemma C.7 and several applications of the triangle inequality, we get
|∠(x̄, x̄′)− ∠(x,x′)| ≤ √
2|‖x− x′‖2 − ‖x̄− x̄′‖2| ≤ √
2‖(x− x′)− (x̄− x̄′)‖2
≤ √ 2‖x− x̄‖2 + √ 2‖x′ − x̄′‖2 ≤ 2 √ 2
n3 ,
and so returning to (D.64), we have shown∣∣∣〈α`(x),α`(x′)〉− cosϕ(`)(∠(x,x′))∣∣∣ ≤ C√d3` n + C ′ n5/2 ≤ (C + C ′) √ d3` n
for every ` ∈ [L]. This implies that the event G is contained in the set⋂ (x,x′)∈M×M { ∀` ∈ [L], ∣∣∣〈α`(x),α`(x′)〉− cosϕ(`)(∠(x,x′))∣∣∣ ≤ C√d3` n } ,
which is closed, by continuity of the inner product and of the features as a function of the parameters, and is therefore also an event. The claim follows.
Lemma D.11. Assume n,L, d satisfy the requirements of lemma D.10 and additionally d ≥ 1, n ≥ K √ L for some K. Then
P [ ‖fθ0‖L∞ ≤ √ d ] ≥ 1− e−cd,
P [ ‖ζ‖L∞ ≤ √ d ] ≥ 1− e−cd.
Define
ζ̂(x) = −f?(x) + ∫ M fθ0(x ′)dµ∞(x′).
Then under the same assumptions
P ∥∥∥ζ̂ − ζ∥∥∥ L∞ ≤ √ d L2 + d5/2 √ L n  ≥ 1− e−cd for some numerical constant c.
Proof. At some x ∈M, we note that
fθ0(x) = W L+1αL(x) d = g ∥∥αL(x)∥∥ 2
(D.65)
where g is a standard normal independent of the other variables in the problem. Similarly
fθ0(x)− ∫ M fθ0(x ′)dµ∞(x′) = WL+1 αL(x)− ∫ M αL(x′)dµ∞(x′)  d = g′
∥∥∥∥∥∥αL(x)− ∫ M αL(x′)dµ∞(x′) ∥∥∥∥∥∥ 2 ,
(D.66)
where g′ is also standard normal. With respect to the randomness of WL+1 , these two objects are Gaussian processes with variances ∥∥αL(x)∥∥2
2 and ∥∥∥∥αL(x)− ∫ M αL(x′)dµ∞(x′) ∥∥∥∥2 2 respectively.
We next note that∥∥∥∥∥∥αL(x)− ∫ M αL(x′)dµ∞(x′) ∥∥∥∥∥∥ 2
2
= ∥∥∥∥∥∥ ∫ M αL(x)−αL(x′)dµ∞(x′) ∥∥∥∥∥∥ 2
2 ≤ ∫ M ∥∥αL(x)−αL(x′)∥∥2 2 dµ∞(x′)
= ∫ M ∥∥αL(x)∥∥2 2 + ∥∥αL(x′)∥∥2 2 − 2 〈 αL(x),αL(x′) 〉 dµ∞(x′)
≤ sup x∈M
∣∣∣∥∥αL(x)∥∥2 2 + ∥∥αL(x′)∥∥2 2 − 2 〈 αL(x),αL(x′) 〉∣∣∣ ≤ sup x∈M  ∣∣∣∣∣∣∣ ∥∥αL(x)∥∥2 2 + ∥∥αL(x′)∥∥2 2 − 2 〈 αL(x),αL(x′) 〉 −(2− 2 cosϕ(L)(ν(x,x′))) ∣∣∣∣∣∣∣ + ∣∣2− 2 cosϕ(L)(ν(x,x′))∣∣
 where the first inequality comes from an application of Jensen’s inequality. Assuming n, d satisfy the requirements of lemma D.10 and denote the event defined in it by G. On G, angles between features concentrate uniformly around a simple function of the angle evolution function ϕ, in the sense that, for all x ∈M simultaneously,∣∣∣∥∥αL(x)∥∥2 2 + ∥∥αL(x′)∥∥2 2 − 2 〈 αL(x),αL(x′) 〉 − (2− 2 cosϕ(L)(ν(x,x′)))
∣∣∣ ≤ ∣∣∣∥∥αL(x)∥∥2 2 − 1 ∣∣∣+ ∣∣∣∥∥αL(x′)∥∥2 2 − 1 ∣∣∣+ 2 ∣∣∣〈αL(x),αL(x′)〉− cosϕ(L)(ν(x,x′))∣∣∣
≤C ′ √ d3L
n
(D.67)
for some constant C ′. From lemma C.12, there exists a constant c0 > 0 such that for all ν ∈ [0, π],
0 ≤ ϕ(L)(ν) ≤ 1 c0L .
Using cosx ≥ 1− x2/2 and the above bound gives
1 ≥ cosϕ(L)(ν) ≥ 1− (ϕ (L)(ν))2 2 ≥ 1− 1
2c20L 2
and thus ∣∣∣2− 2 cosϕ(L)(ν(x,x′))∣∣∣ ≤ 1 c20L 2 .
Combining the above bound with D.67 and recalling the probability of G holding, we have
P [ sup x∈M ∣∣∣∥∥αL(x)∥∥2 2 + ∥∥αL(x′)∥∥2 2 − 2 〈 αL(x),αL(x′) 〉∣∣∣ > 1 c20L 2 + C ′ √ d3L n ] ≤ e−cd.
(D.68) On the same event we have P [
sup x∈M
∥∥αL(x)∥∥2 2 > 2 ] ≤ e−cd.
Thus on G the variances of the Gaussian processes (D.65), (D.66) are uniformily bounded by 2 and 1 c20L 2 + C
′ √
d3L n respectively. Writing for concision in the subsequent expression
E∗  ∣∣∣∣∣∣fθ0(x)− ∫ M fθ0(x ′)dµ∞(x′) ∣∣∣∣∣∣ ≤ √ d 2 √ 1 L2 + √ d3L n  taking a union bound over all points on the net N
n−3n −1/2 0
gives
P  ⋂ x∈N
n−3n−1/20
{ |fθ0(x)| ≤ √ d
2
} ∩ E∗ ∣∣∣∣∣∣∣G
≥1− ∣∣∣Nn−3n−1/20 ∣∣∣ e−cd ≥ 1− e−c′d,
(D.69)
for some constants, since d was chosen to satisfy the conditions of lemma D.10.
In addition, we see from (D.63) that on the same event, for every x ∈M there exists x ∈ N n−3n
−1/2 0
such that
|fθ0(x)− fθ0(x)| = ∣∣∣∣∣∣fθ0(x)− ∫ M fθ0(x ′)dµ∞(x′)− fθ0(x)− ∫ M fθ0(x ′)dµ∞(x′) ∣∣∣∣∣∣ = ∣∣WL+1 (αL(x)−αL(x))∣∣ ≤ ∥∥WL+1∥∥
2 ∥∥αL(x)−αL(x)∥∥ 2 ≤ C ∥∥WL+1∥∥ 2
n5/2 . Bernstein’s inequality also gives P [∥∥WL+1∥∥ 2 > C √ n ] ≤ e−cn for some constants. Denoting the complement of this event by E we have that on E ∩G, for every x ∈M there exists x ∈ N n−3n
−1/2 0
such that
‖fθ0(x)− fθ0(x)‖2 ≤ C ′ n2 ≤ √ d
2∥∥∥∥∥∥fθ0(x)− ∫ M fθ0(x ′)dµ∞(x′)− fθ0(x)− ∫ M fθ0(x ′)dµ∞(x′) ∥∥∥∥∥∥ 2 ≤ C ′ n2 ≤ √ d 2 √ 1 L2 + √ d3L n
where we assumed d ≥ 1, n ≥ K √ L for some K. Combining the above bound with (D.69) and taking a union bound over the complements of E ,G gives
P  ⋂ x∈M { |fθ0(x)| ≤ √ d } ∩  ∣∣∣∣∣∣fθ0(x)− ∫ M fθ0(x ′)dµ∞(x′) ∣∣∣∣∣∣ ≤ √ d L2 + d5/2 √ L n
≥1− e−cd − P [Gc]− P [Ec]
≥1− e−c ′d − e−c ′′n ≥ 1− e−c ′′′d.
From the first result, we also obtain that with the the same probability ‖ζ‖L∞ ≤ 1 + √ d. By
worsening the constant in the tail we can simplify this to ‖ζ‖L∞ ≤ √ d.
Defining
ζ̂(x) = −f?(x) + ∫ M fθ0(x ′)dµ∞(x′),
since ζ̂(x)− ζ(x) = fθ0(x)− ∫ M fθ0(x ′)dµ∞(x′), it follows that
P ∥∥∥ζ̂ − ζ∥∥∥ L∞ ≤ √ d L2 + d5/2 √ L n  ≥ 1− e−c′′′d.
Lemma D.12. For some integer d0, assume n,L, d satisfy the requirements of lemmas D.8 and D.14, meaning that there exist absolute constants K,K ′,K ′′,K ′′′,K ′′′′ > 0 such that for any d ≥ max{Kd0 log(nn0CM),K ′ logL}, if n ≥ max{K ′′d4L,K ′′′L log n,K ′′′′} then
1. If d0 = 1 and n ≥ K ′′′′′max { d2L, ( κ cτ )1/3 , κ2/5 } where K ′′′′′ is some absolute con-
stant. κ and cλ are the extrinsic curvature and injectivity coefficient defined in Section 2.1, then on an event of probability at least 1− e−cd, one has∥∥∥fθ0 ∣∣M±∥∥∥Lip ≤ √d for a numerical constant c, and where the Lipschitz seminorm is taken with respect to the Riemannian distance onM±.
2. IfM = Sn0−1 so that d0 = n0 − 1, then on an event of probability at least 1− e−cd, one has ∥∥fθ0∣∣Sn0−1∥∥Lip ≤ √d for a numerical constant c.
Proof. We recall fθ0(x) = W L+1αL(x). Let M? denote a connected component of M. Let x1,x2 ∈ M?, and fix a smooth unit-speed geodesic γ : [0,distM?(x1,x2)] → M? such that γ(0) = x1 and γ(distM?(x1,x2)) = x2. The absolute continuity of fθ0 ∣∣ M± ◦ γ follows from an argument almost identical to the one employed in the proof of Lemma B.8, and we obtain in particular the bound
|fθ0(x1)− fθ0(x2)| = ∣∣∣∣∣ ∫ distM? (x1,x2)
0
〈 γ′(t), ( W 1 )∗ β0(γ(t)) 〉 dt ∣∣∣∣∣. Because γ is a unit-speed geodesic, we have for all t
PTγ(t)M?γ ′(t) = (γ′(t)γ′(t)∗) γ′(t) = γ′(t),
and so in particular, by the triangle inequality and Cauchy-Schwarz
|fθ0(x1)− fθ0(x2)| ≤ distM?(x1,x2) sup x∈M? ∥∥∥PTxM? (W 1)∗ β0(x)∥∥∥ 2 . (D.70)
This implies ∥∥∥fθ0∣∣M?∥∥∥Lip ≤ supx∈M? ∥∥PTxM?W 1∗β0(x)∥∥2 . We next write W 1 = W 1xx∗ +W 1 (I − xx∗) = G1 +H1. TxM? can be identified with a linear subspace of Rn0 of dimension d0. Since it is also a subspace of TxSn0−1, PTxM?x = 0 and hence
PTxM?W 1∗ = PTxM?H 1∗ d= PTxM? (I − xx∗) W̃ 1∗ = PTxM?W̃ 1∗
where W̃ 1∗ is a copy of W 1∗ that is independent of all the other variables in the problem (since β0(x) depends only onG1).
We first consider the case d0 = n0 − 1, and subsequently the case d0 = 1. We note that∥∥∥PTxM?W̃ 1∗β0(x)∥∥∥2 2 ≤ ∥∥∥W̃ 1∗β0(x)∥∥∥2 2 = n0∑ i=1 ( W̃ 1∗i β 0(x) )2 , (D.71)
Considering a single summand in the above expression, repeated application of the rotational invariance of the gaussian distribution gives(
W̃ 1∗i β 0(x)
)2 d = 2 ∥∥β0(x)∥∥2 2
n g2i,I(x)
= 2
n ∥∥WL+1ΓL:2(x)PI1(x)∥∥22 g2i,I(x) ≤ 2
n
∥∥WL+1ΓL:2(x)∥∥2 2 g2i,I(x)
d =
2
n
∥∥ΓL:2(x)WL+1∥∥2 2 g2i,I(x)
d =
2
n ∥∥ΓL:2(x)e1∥∥22 ∥∥WL+1∥∥22 g2i,I(x) (D.72) where gi,I(x) is a standard normal variable that depends on i and the support patterns I(x) = {I1(x), . . . , IL(x)}, since β0(x) depends on x only through I(x) Similarly, the dependence on x in the first factor in (D.72) is only through I(x). We now show how to control such terms uniformly onM?.
Define a net N n−3n
−1/2 0 as in Appendix D.3.1. According to Lemma C.4, ∣∣∣Nn−3n−1/20 ∣∣∣ ≤
e3 log(CM?nn0)d0 . Assume n,L, d satisfy the requirements of Lemma D.8 and denote this event defined in that lemma by E . We also define sets of support patterns
J (x) = { J = {J1, . . . , JL} ∣∣∣∣ L∑ `=1 |J` I`(x)| ≤ d } , (D.73)
J (N n−3n
−1/2 0
) = ⋃
x∈N n−3n−1/20
J (x).
On E , ⋃
x∈M? {I(x)} ⊆ J (N n−3n −1/2 0 ), and additionally for any x ∈ M?, then there exists some
x ∈ N n−3n
−1/2 0
such that x ∈ N n−3n
−1/2 0 (x) and I(x) ∈ J (x). We now show that on E , the supports I(x) satisfy the requirements of Lemma D.14 with δs = d,Ks = Cn−5/2, with the anchor point in the statement of that lemma chosen to be x. The value of δs is satisfied by the supports at every point on the manifold on E from the definition of this event. From the definition of the stable sign consistency property (SSC) in Appendices D.3.1 and D.3.2, the only features whose sign can differ between x to x are the risky features, and from the bound on their norm in the definition of SSC(L, n−3n
−1/2 0 , Cn −3) we obtain for all `∥∥(PJ` − PI`(x))ρ`(x)∥∥L∞ ≤ Cn3 ⇒ ∥∥(PJ` − PI`(x))ρ`(x)∥∥2 ≤ Cn5/2 where in the last inequality we used Lemma G.10. It follows that if n ≥ KLd for some K, the requirements of Lemma D.14 are satisfied if we make the choice E = EδK . We would next like to apply Lemma D.14 for every possible support pattern in J (N
n−3n −1/2 0
) ,
which requires that we first bound the cardinality of this set. Note that J (x) is the . Thus ∣∣J (x)∣∣ ≤ ⌊ d ⌋∑
i=0
( Ln i ) ≤ ⌊ d ⌋(eLn⌊ d ⌋ ) ⌊ d ⌋ ≤ (Ln)Cd
for some C. Using the bound on the cardinality of the net from Lemma C.4, the size of this set can be bounded, giving∣∣∣J (Nn−3n−1/20 )∣∣∣ ≤ ∣∣∣Nn−3n−1/20 ∣∣∣ dde∑ i=0 ( n i ) ≤ e3 log(CM?nn0)d0+C log(Ln)d. (D.74) We can now apply Lemma D.14 with E = EδK to bound ∥∥Γ`:2(x)e1∥∥22 for all ` on E , taking a union bound over all possible supports. Bernstein’s inequality and an exponential tail bound can be used to bound the second factor and third factors in (D.72) respectively. Using (D.74) to bound the number of supports we need to uniformize over (since on E ,
⋃ x∈M? {I(x)} ⊆ J (N n−3n −1/2 0 )) and
the bound on the size of the net in Lemma C.4 and Appendix D.3.1, we obtain P [ ∀x ∈M? : 2 n 1E ∥∥ΓL:2(x)e1∥∥22 ∥∥WL+1∥∥22 g2i,I(x) ≤ d]
=P
[ ∀x ∈ N
n−3n −1/2 0
,
∀x ∈ N n−3n
−1/2 0
(x) :
2 n 1E ∥∥ΓL:2(x)e1∥∥22 ∥∥WL+1∥∥22 g2i,I(x) ≤ d ] ≥1−
∣∣∣Nn−3n−1/20 ∣∣∣ eC log(n)de−c nL − ∣∣∣Nn−3n−1/20 ∣∣∣ eC log(n)de−c′d − e−c′′n ≥1− e3 log(CM?nn0)d0+C log(n)d−c nL − e3 log(CM?nn0)d0+C log(n)d−c ′d − e−c ′′n
≥1− e−c ′′′d
(D.75)
where we assume d ≥ K log(CM?nn0)d0, n ≥ K ′Ld2 for some K,K ′. Since according to Lemma D.8 the event E holds with probability greater than 1 − ecd for some c, we can remove the indicator in the bound above by assuming d ≥ K for some absolute constant K and worsening the constant in the bound.
We can now complete the proof for d0 = n0−1. Since we are interested in bounding the sum (D.71) uniformly, we can bound n0∑ i=1 g2i,I(x) as well using Bernstein’s inequality and uniformizing as above obtain
P [ ∀x ∈M? : 2 n 1E ∥∥ΓL:2(x)e1∥∥22 ∥∥WL+1∥∥22 n0∑
i=1
g2i,I(x) ≤ C(n0 + d)
]
=P
[ ∀x ∈ N
n−3n −1/2 0
,
∀x ∈ N n−3n
−1/2 0
(x) :
2 n 1E ∥∥ΓL:2(x)e1∥∥22 ∥∥WL+1∥∥22 n0∑
i=1
g2i,I(x) ≤ C(n0 + d) ] ≥1−
∣∣∣Nn−3n−1/20 ∣∣∣ eC log(n)de−c nL − ∣∣∣Nn−3n−1/20 ∣∣∣ eC log(n)de−c′d − e−c′′n ≥1− e3 log(CM?nn0)d0+C log(n)d−c nL − e3 log(CM?nn0)d0+C log(n)d−c ′d − e−c ′′n
≥1− e−c ′′′d
where we assume d ≥ K log(CM?nn0)d0, n ≥ K ′Ld2 for some K,K ′. As above, we can remove the indicator in the bound above by assuming d ≥ K for some absolute constant K and worsening the constant in the bound. Worsening constants in the failure probability, we can replace the residual in the above expression by d. Using (D.70), we obtain that with the same probability the Lipschitz constant of fθ0 on Sn0−1 is bounded by √ d.
We now consider d0 = 1. Recall that for any x ∈ M?, then there exists some x ∈ Nn−3n−1/20 such that x ∈ N
n−3n −1/2 0
(x) where N n−3n
−1/2 0 is the net defined earlier. The gradient vector at x takes the form
PTxMW̃ 1∗β0(x) = PTxMW̃ 1∗β0(x) + (PTxM − PTxM) W̃ 1∗β0(x). (D.76)
We proceed to bound the first term in the above equation uniformly over M?. Since in the d0 = 1 case TxM? can be identified with a linear subspace of Rn0 of dimension 1, we can write the
projection operator as PTxM? = vxv ∗ x (D.77) for some unit norm vx. We then obtain from rotational invariance of the gaussian distribution that∥∥∥PTxMW̃ 1∗β0(x)∥∥∥2 2 = ∥∥∥vxv∗xW̃ 1∗β0(x)∥∥∥2 2
d =
2 ∥∥β0(x)∥∥2
2 n g2x
≤ 2 n ∥∥ΓL:2(x)e1∥∥22 ∥∥WL+1∥∥22 g2x, where gx is a standard normal variable and the last bound comes from a calculation identical to (D.72). Under the same assumptions on d, L, n as before, we can bound this expression uniformly using (D.75), and additionally take a union bound over the net to account for all possible choices of x. This gives
P
[ ∀x ∈ N
n−3n −1/2 0
,
∀x ∈ N n−3n
−1/2 0
(x) :
2 n 1E ∥∥ΓL:2(x)e1∥∥22 ∥∥WL+1∥∥22 g2x ≤ d ] ≥1−
∣∣∣Nn−3n−1/20 ∣∣∣ e−c′d ≥1− e3 log(CM?nn0)d0−c ′d
≥1− e−c ′′d
where we assume d ≥ K log(CM?nn0) for some K. This gives control of the first term in (D.76) uniformly onM. We now turn to controlling the second term in (D.76). For some x, choose x ∈ N
n−3n −1/2 0 such that x ∈ N
n−3n −1/2 0 (x). Define a unit-speed curve γ : [0, s] →M such that γ(0) = x,γ(s) = x. Since the curvature ofM is bounded by κ, we have
∀s′ ∈ [0, s] : ‖γ′′(s′)‖2 ≤ κ.
Denote by r the geodesic distance between x and x′. Since the euclidean distance between them is bounded by n−3n−1/20 , assuming n > K for some K implies that r < C ′n−3n −1/2 0 for some C
′. If we now demand n3 ≥ C ′ κcλ which implies C′
n3n 1/2 0
≤ cτκ , (A.2) gives
s ≤ C n3n
1/2 0
,
for some C > 0. For vx,vx defined as in (D.77) we have γ′(0) = vx,γ′(s) = vx. Combining the previous two results, it follows that
∥∥vx − vx∥∥ 2 = ‖γ′(0)− γ′(s)‖2 = ∥∥∥∥∥∥ s∫
s′=0
γ′′(s′)ds′ ∥∥∥∥∥∥ 2 ≤ sκ ≤ Cκ n3n 1/2 0 .
A straightforward calculation then gives ‖PTxM − PTxM‖ = ∥∥vxvx∗ − vxvx∗∥∥ = 1
2
∥∥vx − vx∥∥ 2 ∥∥vx + vx∥∥ 2
≤ ∥∥vx − vx∥∥
2 ≤ Cκ n3n
1/2 0
.
If we now use Lemma D.13 to control the norms of the backward features uniformly, a standard bound on the norm of a Gaussian matrix to give P [∥∥∥W̃ 1∥∥∥ > C (1 +√n0n )] ≤ e−cn, and assume n ≥ κ2/5 we obtain that
P [ ∀x ∈ N
n−3n −1/2 0
, x ∈ N n−3n
−1/2 0
(x) : ∣∣∣(PTxM − PTxM) W̃ 1∗β0(x)∣∣∣ ≤ C]
≥ 1− e−cd − e−c ′n ≥ 1− e−c ′′d.
Combining the above with (D.75) and using (D.76) and taking a union bound over the failure probability of E which results in a worsening of constants completes the proof. We can additionally rescale d to obtain a final bound on the Lipschitz constant of √ d instead of C √ d, which also results in a worsening of constants.
Lemma D.13. There are absolute constants c, C > 0 and absolute constants K1, . . . ,K4 > 0 such that for any d ≥ Kd0 log(nn0CM), if n ≥ K ′d4L, then there exists an event E such that
1. On E , we have
∀` ∈ [L], ∣∣∣∣∣〈β`−1(x),β`−1(x′)〉− n2 L−1∏ `′=`−1 ( 1− ϕ (`′)(∠(x,x′)) π )∣∣∣∣∣ ≤ C√d4nL simultaneously for every (x,x′) ∈M×M;
2. P[E ] ≥ 1− e−cd.
Proof. Let E1 denote the event studied in Lemma D.8, with C0 denoting the absolute constant appearing in the SSC(L) condition there; choose d ≥ Kd0 log(nn0CM) and n sufficiently large to make the measure bound applicable. We will need to apply Lemma D.23 together with a derandomization argument to prove the claim; we appeal to the same residual checks at the beginning of the proof of Lemma D.6 to see that on E1, the dominating residual in Lemma D.23 under the scalings of d and n we enforce here is of size C √ d4nL.
For any subset S ⊂ [L] × [n], we write S` = {i ∈ [n] | (`, i) ∈ S}, and we define S(S) = {−1,+1}|S1| × · · · × {−1,+1}|SL| for the set of “lists” of sign patterns with sizes adapted to these projections of S, with the convention {−1,+1}0 = {0}. If Σ = {σ1, . . . ,σL} ∈ S(S) is such a list of sign vectors and ∆ ≥ 0, we define
Ĩ`(x, S,Σ,∆) = supp ( 1ρ`(x)> ∑ i∈S` ((σ`)i∆)ei ) ,
which is a sort of two-sided robust analogue of the support of α`(x): notice that when S = ∅ we have Ĩ`(x, S,Σ,∆) = I`(x). We also define for ` = 0, 1, . . . , L− 1
β̃`S,Σ,∆(x) = ( WL+1PĨL(x,S,Σ,∆)W LPĨL−1(x,S,Σ,∆) . . .W `+2PĨ`+1(x,S,Σ,∆) )∗ ,
a generalized backward feature induced by these robust support patterns. Writing for concision
Sx̄,x̄′,S,S′,Σ,Σ′ =  ∃` ∈ [L] : ∣∣∣∣∣〈β̃`−1S,Σ,C0n−3(x̄), β̃`−1S′,Σ′,C0n−3(x̄′)〉 −n2 ∏L−1 `′=`−1 ( 1− ϕ (`′)(∠(x̄,x̄′)) π )∣∣∣∣∣ > C1 √ d4nL log4 n  ,
where C1 > 0 is an absolute constant we will specify below to make the event hold with high probability, we then define the event11
E2 = ⋃
x̄∈Nn−3 x̄′∈Nn−3 ⋃ S⊂[L]×[n] S′⊂[L]×[n] |S|≤d,|S′|≤d ⋃ Σ∈S(S) Σ′∈S(S′) Sx̄,x̄′,S,S′,Σ,Σ′ ,
There are no more than ∑d k=0 ( nL k ) ≤ n4d ways to choose the subset S in this union, and for a fixed S there are no more than 2d ways to choose the sign pattern Σ. Thus, there no more than
11To see that this set is indeed an event, use that β̃`S,Σ,∆(x) is a continuous function of the network weights except with respect to the support projections; but x 7→ 1x>0 is increasing, hence Borel-measurable, and so the set consists of a finite union of Borel-measurable sets.
exp(10d log n+12d0 log(nn0CM)) elements in the union, and under the condition on d this number is no larger than n11d. For concision, write
ξ`(x̄, x̄′) = n
2 L−1∏ `′=`
( 1− ϕ
(`′)(∠(x̄, x̄′)) π
) .
For any instantiation of these parameters, Lemma D.23 and a union bound give P [ ∃` ∈ [L] : ∣∣∣〈β̃`−1S,Σ,C0n−3(x̄), β̃`−1S′,Σ′,C0n−3(x̄′)〉− ξ`−1(x̄, x̄′)∣∣∣ > C√d4nL] ≤ P[Ec1] + P [ ∃` ∈ [L] :
∣∣∣1E1〈β̃`−1S,Σ,C0n−3(x̄), β̃`−1S′,Σ′,C0n−3(x̄′)〉− ξ`−1(x̄, x̄′)∣∣∣ > C√d4nL] ≤ e−cd
for any d ≥ K log n and n ≥ K ′d4L. Thus, if we set C1 = C and enforce d ≥ Kd0 log(nn0CM)/ log n and n ≥ maxK ′d4L log4 n, we have by a union bound
P[E1 ∪ E2] ≤ n−cd. Let G = Ec1 ∩ Ec2. For any (x,x′) ∈M×M, we can find a point x̄ ∈ Nn−3n−1/20 ∩Nn−3n−1/20 (x) and a point x̄′ ∈ N n−3n −1/2 0 ∩ N n−3n −1/2 0
(x′). On G, SSC(L, n−3n−1/20 , Cn−3) holds at every point in the net N
n−3n −1/2 0 , and there are no more than d Cn−3-risky features at any point in the net N n−3n
−1/2 0 , and in addition, following (D.52), we have almost surely on G that all risky features are realized for magnitudes in (−∆,+∆). This implies that on G, the support sets ⊔ `∈[L] I`(x) at any point x ∈ N n−3n
−1/2 0
(x̄) differ by the support sets ⊔ `∈[L] I`(x̄) at the base point in the net by no
more than d entries, consisting only of a subset of the risky features at x̄; the analogous statement is of course true for x′ and x̄′. At the same time, notice that on the event Ec2 we have constructed, we have control of every possible backward feature inner product obtained by modifying the supports at the base points x̄, x̄′ at no more than d risky features (each), since, for example, if (ρ`(x̄))i is risky, then 1(ρ`(x̄))i>∆ corresponds to “turning off” the feature, and 1(ρ`(x̄))i>−∆ corresponds to “turning on” the feature. Formally, we have established that on G
∀` ∈ [L], ∣∣∣∣∣〈β`−1(x),β`−1(x′)〉− n2 L−1∏ `′=`−1 ( 1− ϕ (`′)(∠(x̄, x̄′)) π )∣∣∣∣∣ ≤ C1 √ d4nL log4 n.
We can use differentiability properties for the remaining link: following the proof of Lemma D.10, we have
|∠(x̄, x̄′)− ∠(x,x′)| ≤ √ 2‖x− x̄‖2 + √ 2‖x′ − x̄′‖2 ≤ 2 √ 2
n3 , so we just need a Lipschitz property for the function q(ν) = (n/2) ∏L−1 `′=`(1 − π−1ϕ(`
′)(ν)). For this we appeal to Lemma E.5, which shows that the function ϕ is smooth, increasing and concave; therefore by the chain rule, the functions ϕ(`) are increasing and concave, and by the Leibniz rule, q is decreasing and convex. It therefore suffices to calculate q′(0); this is done in Lemma C.18, which gives q′(0) = −n(L− `)/(2π), and in particular |q′(0)| ≤ cnL. It follows∣∣∣∣∣n2 L−1∏ `′=` ( 1− ϕ (`′)(∠(x̄, x̄′)) π ) − n 2 L−1∏ `′=` ( 1− ϕ (`′)(∠(x,x′)) π
)∣∣∣∣∣ ≤ cLn2 , so that by the triangle inequality
∀` ∈ [L], ∣∣∣∣∣〈β`−1(x),β`−1(x′)〉− n2 L−1∏ `′=`−1 ( 1− ϕ (`′)(∠(x,x′)) π )∣∣∣∣∣ ≤ 2C1 √ d4nL log4 n,
where the residual simplification is valid when n ≥ KL. We conclude that the set⋂ (x,x′)∈M×M { ∀` ∈ [L], ∣∣〈β`−1(x),β`−1(x′)〉− ξ`−1(x̄, x̄′)∣∣ ≤ 2C1√d4nL log4 n} contains the event G, which satisfies the claimed properties and completes the proof (after rescaling d by 1/ log n, which updates the lower bound on d).

heading: D.3.4 SMALL SUPPORT CHANGE RESIDUALS

text:
In this section, we prove generalized versions of our pointwise concentration lemmas for backward feature correlations and the matrices defining the propagation coefficients used in our study of SSC(L).
Lemma D.14. Assume n ≥ max {KL log n,K ′Ld,K ′′}, d ≥ K ′′′ logL for suitably chosen K,K ′,K ′′,K ′′′ and integer L, and choose 1 ≤ `′ ≤ ` ≤ L. Define an anchor point x ∈ M and denote Ii(x) = supp ( αi0(x) > 0 ) for `′ ≤ i ≤ `.
Choose some δs,Ks > 0 and let J = {J`′ , . . . , J`} denote a collection of support sets such that each Ji ⊂ [n] depends on the network parameters only through the pre-activation ρi0(x). We define events implying that the supports at J are close to those at x:
Eδ = ⋂
`′≤i≤`
{|Ji Ii(x)| ≤ δs} ,
EK = ⋂
`′≤i≤` {∥∥(PJi − PIi(x))ρi(x)∥∥2 ≤ Ks} , EδK = Eδ ∩ EK .
Define
Γ`:` ′
J = PJ`W `PJ`−1 . . .PJ`′W
`′ ,
and fix a unit norm vector vf . If Ks ≤ 12L −3/2 , δs ≤ nL , then
P [ 1EδK ∥∥∥Γ`:`′J vf∥∥∥ 2 ≤ C ] ≥ 1− e−c nL ,
and P [ 1EδK ∥∥∥Γ`:`′J ∥∥∥ ≤ C√L] ≥ 1− e−c nL . For a vector g, gi ∼iid N (0, 1), definingHi = W i ( I −αi−1(x)αi−1(x)∗ ) for i ∈ [L] and
Γ`:` ′
HJ = PJ`H `PJ`−1 . . .PJ`′H
`′
we have P [ 1EδK ∥∥∥(Γ`:`′J − Γ`:`′HJ ) g∥∥∥ 2 > C √ dL ] ≤ e−cd
for some numerical constants c, C.
Proof. In the following, we will denote by vf ∈ Sn−1 a fixed unit norm vector and by vu ∈ Sn−1 a random vector uniformly distributed on Sn−1. When there is no need to distinguish between the two we will denote either by vp.
Our strategy in bounding 1EδK ∥∥∥Γ`:`′J ∥∥∥ will be first to bound 1EδK ∥∥∥Γ`:`′J vf∥∥∥
2 with sufficiently high
probability, and then apply an ε-net argument to uniformize the result (lemma D.20) and get control of the operator norm. In achieving the first goal, we will rely heavily on a decomposition of the weight matrices into terms that are conditionally independent given the pre-activations. We will also utilize martingale concentration to control the terms that result from this decomposition.
Denoting Si = span { αi(x) } for i ∈ [L], we decompose the weight matrices into
W i = W iPSi−1 +W iPSi−1⊥
. = Gi +Hi. Note that { H1, . . . ,HL } are conditionally independent given σ(G1, . . . ,GL) (by which we denote
the sigma algebra generated byG1, . . . ,GL). Since the pre-activations obey
ρi(x) = W iαi−1(x) = Giαi−1(x)
and the features are deterministic functions of the pre-activations, both { α1(x), . . . ,αL(x) } and{
ρ1(x), . . . ,ρL(x) } are measurable with respect to σ(G1, . . . ,GL).
We define events
Eρ = ⋂̀ i=`′ {∥∥ρi(x)∥∥ 2 ≤ C } , EδKρ = EδK ∩ Eρ (D.78)
and aim to control 1EδKρ ∥∥∥Γ`:`′J (x)∥∥∥. Since the supports J depends on the weights only through
the pre-activations and are thus also σ(G1, . . . ,GL)-measurable, this truncation does not affect the conditional independence of { H` ′ , . . . ,H` } . It will often be convenient to utilize the rotational
invariance of the Gaussian distribution to replace all occurrences of Hi in a given expression by W̃ iPSi−1⊥ where W̃ i is a fresh copy of W i independent of all the other variables in the problem, which will not change the distribution of the original expression.
For `′ ≤ i ≤ `, `′ ≤ j ≤ i+ 1 it will also be useful to denote
Γi:jHJ = PJiH iPJi−1 . . .PJjH j , Γi:jGJ = PJiG iPJi−1 . . .PJjG j
where we use the convention Γi:i+1GJ = Γ i:i+1 HJ = I . Decomposing the weight matrices at every layer
gives ∥∥∥Γ`:`′J vp∥∥∥ 2 = ∥∥∥PJ` (G` +H`) . . .PJ`′ (G`′ +H`′)vp∥∥∥ 2
≤ ∑
(M`,...,M`′)∈(G`,H`)⊗,...,⊗(G`′ ,H`′)
∥∥∥PJ`M ` . . .PJ`′M `′vp∥∥∥ 2 . (D.79)
We next define Qi(x) = PJi − PIi(x). (D.80)
In accounting for all the terms in the decomposition (D.79), there will be two simplifications that we use repeatedly. One is
Hi+1PJiρ i(x) = W i+1
( I − α i(x)αi(x)∗
‖αi(x)‖22
)( Pαi(x)>0 +Q i(x) ) ρi(x) = Hi+1Qi(x)ρi(x)
(D.81) where we used Pαi(x)>0ρi(x) = [ ρi(x) ] + = αi(x), from which it follows that
Hi+1PJiG i =W i+1
( I − α i(x)αi(x)∗
‖αi(x)‖22
)( Pαi(x)>0 +Q
i(x) ) ρi(x)αi−1(x)∗ ‖αi−1(x)‖22
=Hi+1Qi(x)Gi.
(D.82)
We also have
Gi+1PJiG i =Gi+1 ( PIi(x) +Q i(x) ) Gi
=W i+1 αi(x)αi(x)∗
‖αi(x)‖22
( PIi(x) +Q i(x) ) W i αi−1(x)αi−1(x)∗
‖αi−1(x)‖22
=
∥∥αi(x)∥∥ 2
‖αi−1(x)‖2
( 1 + αi(x)∗
‖αi(x)‖22 Qi(x)W iαi−1(x)
) W i+1αi(x)αi−1(x)∗
‖αi(x)‖2 ‖αi−1(x)‖2 . =si W i+1αi(x)αi−1(x)∗
‖αi(x)‖2 ‖αi−1(x)‖2 ,
and thus
Γi:jGJ = i−1∏ k=j sk PJiW iαi−1(x)αj−1(x)∗ ‖αi−1(x)‖2 ‖αj−1(x)‖2 = i−1∏ k=j sk PJiρ i(x)αj−1(x)∗ ‖αi−1(x)‖2 ‖αj−1(x)‖2 . (D.83)
We refer to such a product as a G-chain. We proceed to expand (D.79) into terms with different combinations of matrices Γi:jGJ and Γ i:j HJ . There will be 2
`−`′ terms in total, and we denote the set of terms with r G-chains by Gr,p (with the subscript p ∈ {u, f} denoting the choice of vector vp). We can clearly label each term by the start and end index of each G-chain, which may not be distinct. We denote each such term by grp(i1,i2,...,i2r) where
`′ ≤ i1 ≤ i2 ≤ i3 − 2 ≤ i4 − 2 < i5 − 4 ≤ · · · ≤ i2m−1 − 2m+ 2 ≤ i2m − 2m+ 2 ≤ . . . ≤ i2r−1 − 2r + 2 ≤ i2r − 2r + 2 ≤ `− 2r + 2.
(D.84) The constraints above ensure that every two G-chains are separated by at least one Hi matrix. To lighten notation, we denote a set of indices obeying the constraints by (i1, . . . , i2r) ∈ Cr(`, `′). The maximal number of G-chains possible is bounded by r ≤ ⌈ (`− `′) /2 ⌉ .
Since the grp(i1,i2,...,i2r) are non-negative, we have
1EδKρ ∥∥∥Γ`:`′J vp∥∥∥ 2 ≤ 1EδKρ ∥∥∥Γ`:`′HJ vp∥∥∥ 2 +
⌈ (`−`′)/2 ⌉∑ r=1 ∑ (i1,...,i2r)∈Cr(`,`′) gr,p(i1,i2,...,i2r). (D.85)
Considering first the form of the terms inGr,p, using (D.83) and (D.81) and recalling that Γm−1:mHJ = I , we have
gr,p(i1,i2,...,i2r) = ∥∥∥Γ`:i2r+1HJ Γi2r:i2r−1GJ Γi2r−1−1:i2r−2+1HJ ...Γi4:i3GJ Γi3−1:i2+1HJ Γi2:i1GJ Γi1−1:`′HJ vp∥∥∥
2
=
1EδKρ ∥∥∥∥Γ`:i2r+1HJ PJi2rρi2r (x)‖αi2r−1(x)‖2 ∥∥∥∥
2︸ ︷︷ ︸ . =ãi2r
∗ r−1∏ m=1 1EδKρ i2m+2−1∏ k=i2m+1+1 sk αi2m+1(x)∗Γ i2m+1−1:i2m+1 HJ PJi2mρ i2m(x)
‖αi2m+1(x)‖2 ‖αi2m−1(x)‖2︸ ︷︷ ︸ . =b̃i2m+2,i2m+1,i2m
∗1EδKρ
i2−1∏ k=i1+1 skα i1(x)∗Γi1−1:` ′ HJ vp
‖αi1(x)‖2︸ ︷︷ ︸ . =c̃i2,i1
=ãi2r r−1∏ m=1 b̃i2m+2,i2m+1,i2m c̃ p i2,i1 .
(D.86)
The magnitudes the factors in this expression are bounded in the following lemma: Lemma D.15. For ãk, b̃qij , c̃pts defined in (D.86), Ru = √ d n , Rf = 1 and `
′ ≤ k < `, `′ + 2 ≤ j + 2 ≤ i ≤ q ≤ `, `′ < s ≤ t ≤ `
ã` ≤ C a.s., P [ãk > Ks] ≤ C ′e−c n L , P [∣∣∣b̃qij∣∣∣ > Ks√L] ≤ C ′e−c nL , P [|c̃pt`′ | > CRp] ≤ 2e−cd + e−c ′n,
P [ |c̃pts| > √ d n ] ≤ C ′e−c nL + 2e−c′d
for some constants c, c′, C, C ′ and d ≥ 0.
Proof: Deferred to D.3.4. We will use these results in order to bound 1EδKρ ∥∥∥Γ`:`′J vp∥∥∥
2 using (D.85). While the sum over most
of these terms can be controlled using the triangle inequality and the lemma above, there is a subset which will require special treatment since they are typically larger. These are the terms where the leftmost or rightmost chain is a G-chain (meaning i2r = ` or i1 = `′ respectively) and they will be controlled using martingale concentration. The or above is exclusive, since we can bound terms with i2r = ` and i1 = `′ using a triangle inequality. We denote these three sets of terms by←− Gr,p, −→ Gr,p, ←→ G r,p respectively, and elements in them by←−g r,p,−→g r,p,←→g r,p for clarity when needed. Arranging the remaining terms into sets denoted Gr,p, the sum in (D.85) decomposes into⌈ (`−`′)/2
⌉∑ r=1 ∑ (i1,...,i2r)∈Cr(`,`′) gr(i1,i2,...,i2r) = ⌈ (`−`′)/2 ⌉∑ r=1 ∑ Qr,p∈ { Gr,p, ←− Gr,p, −→ Gr,p, ←→ G r,p } ∑ gr,p∈Qr,p gr,p.
(D.87) We consider first terms in ←− Gr,p (and hence with i2r = `). We denote such terms by
←−g r,p(i1,i2,...,i2r−1,`) = ã`b̃`,i2r−1,i2r−2 r−2∏ m=1 b̃i2m+2,i2m+1,i2m c̃ p i2,i1 .
We show
Lemma D.16. For p ∈ {f, u} and Ru = √ d n , Rf = 1
i .
P  ∣∣∣∣∣∣∣ ⌈ (`−`′)/2 ⌉∑ r=1 ∑ ←−g r,p∈ ←− Gr,p ←−g r,p ∣∣∣∣∣∣∣ > √ dL n  ≤ Ce−cd + C ′e−c′ nL (D.88) ii .
P  ∣∣∣∣∣∣∣ ⌈ (`−`′)/2 ⌉∑ r=2 ∑ −→g r,p∈ −→ Gr,p −→g r,p ∣∣∣∣∣∣∣ > CRp  ≤ Ce−cd + C ′e−c′ nL (D.89)
for absolute constants c, C,C ′, and where d ≥ K logL for some constant K.
Proof: Deferred to D.3.4. Turning next to bounding the terms in ←→ G r,p, we first define an event
←→ E = {|ã`| ≤ C} ∩ ⋂ `′+1≤i1≤i2−2≤i3−2≤`−2 {∣∣∣b̃i3i2i1∣∣∣ ≤ Ks√ L } ∩ ⋂ `′<i1≤` {∣∣c̃pi1`′ ∣∣ ≤ CRp} and from lemma D.15 and a union bound obtain
P [←→ E c ] ≤ L3C ′e−c nL + L ( 2e−cd + e−c ′n ) ≤ C ′′e−c ′ n L + 2e−c ′′d
assuming n ≥ KL logL, d ≥ K ′ logL for some K,K ′. It follows that∣∣∣∣∣∣∣1←→E ∑
←→g r,p∈ ←→ G r,p
←→g r,p ∣∣∣∣∣∣∣ =
∣∣∣∣∣∣∣1←→E ⌈ (`−`′)/2 ⌉∑ r=1 ∑ (i2,...,i2r−1)∈Cr−1(`′,`) ã`b̃`,i2r−1,i2r−2 r−2∏ m=1 b̃i2m+2,i2m+1,i2m c̃i2,`′ ∣∣∣∣∣∣∣ ≤ ⌈ (`−`′)/2
⌉∑ r=1 ∑ (i2,...,i2r−1)∈Cr−1(`′,`) ∣∣∣∣∣1←→E ã`b̃`,i2r−1,i2r−2 r−2∏ m=1 b̃i2m+2,i2m+1,i2m c̃i2,`′ ∣∣∣∣∣ ≤ ⌈ (`−`′)/2
⌉∑ r=1 L2r−2 ( Ks√ L )r−1 Rp
=
⌈ (`−`′)/2 ⌉∑ r=1 ( KsL 3/2 )r−1 Rp ≤ 1− ( KsL 3/2 )L/2 1−KsL3/2 Rp ≤ 2Rp.
where we used L3/2Ks ≤ 12 . We also bound the number of summands in ∑
(i2,...,i2r−1)∈Cr−1(`′,`) by
L2r−2 which is tight for small r. It follows that
P  ∣∣∣∣∣∣∣ ∑ ←→g r,p∈ ←→ G r,p ←→g r,p ∣∣∣∣∣∣∣ > 2Rp  ≤ P  ∣∣∣∣∣∣∣1←→E ∑ ←→g r,p∈ ←→ G r,p ←→g r,p ∣∣∣∣∣∣∣ > 2Rp
+ P  ∣∣∣∣∣∣∣ ( 1− 1←→E ) ∑ ←→g r,p∈ ←→ G r,p ←→g r,p ∣∣∣∣∣∣∣ > 0
= P [←→ E c ] ≤ Ce−c nL + 2e−c ′d (D.90)
for appropriate constants. It remains to bound the terms in Gr,p by a similar argument. Defining
E = ⋂
`′≤i1<`
{|ãi1 | ≤ Ks} ∩ ⋂
`′+2≤i1+2≤i2≤i3≤`
{∣∣∣b̃i3i2i1 ∣∣∣ ≤ Ks√ L }
∩ ⋂
`′<i1≤i2≤`
{∣∣c̃pi2i1∣∣ ≤ √ d
n
}
truncating on this event gives∣∣∣∣∣∣∣1E ∑
gr,p∈Gr,p
gr,p ∣∣∣∣∣∣∣ ≤ 1E ∑
gr,p∈Gr,p
∣∣gr,p∣∣ ≤ (L3/2Ks)r√dLn ≤ 2 √ dL n
and bounding the probability of this even from below using lemma D.15 and a union bound gives
P  ∣∣∣∣∣∣∣ ∑
gr,p∈Gr,p
gr,p ∣∣∣∣∣∣∣ > 2 √ dL n  ≤ P  ∣∣∣∣∣∣∣1E ∑ gr,p∈Gr,p gr,p ∣∣∣∣∣∣∣ > 2 √ dL n + P [Ec] ≤ Ce−c nL + 2e−c ′d.
Combining the bound above with (D.90) and the results of lemma D.16 and worsening constants, the sum of all terms containing matricesGi is bounded by
P ∣∣∣∣∣∣ ∑
gr,p∈Gr,p
gr,p ∣∣∣∣∣∣ > C (√ dL n +Rp ) ≤ ∑ Qr,p∈ { Gr,p, ←− Gr,p
}P ∣∣∣∣∣∣ ∑ gr,p∈Qr,p gr,p ∣∣∣∣∣∣ > C √ dL n
+ ∑
Qr,p∈ {−→ Gr,p, ←→ G r,p
}P ∣∣∣∣∣∣ ∑ gr,p∈Qr,p gr,p ∣∣∣∣∣∣ > CRp
≤C ′e−c nL + C ′′e−c ′d.
(D.91) Bounding the first term in (D.85) using lemma D.18, setting p = f above and choosing d = nL gives
P [ 1EδKρ ∥∥∥Γ`:`′J vf∥∥∥ 2 > C ] ≤ C ′e−c ′ n L .
We then apply lemma D.20 to obtain P [ 1EδKρ ∥∥∥Γ`:`′J ∥∥∥ > C√L] ≤ C ′e−c′ nL . Recalling (D.78), to obtain our final bound on the operator norm it remains to control the probability of Eρ. We consider some `′ ≤ i ≤ ` and assume αi−1(x) 6= 0 (otherwise ∥∥ρi(x)∥∥ 2 ≤ C with
probability 1). Defining an orthogonal matrixR such thatRαi−1(x) = ∥∥αi−1(x)∥∥
2 ê1, rotational
invariance of the Gaussian distribution gives∥∥ρi(x)∥∥2 2 = αi−1(x)∗W i∗W iαi−1(x) d = ∥∥αi−1(x)∥∥2 2 ∥∥∥W i(:,1)∥∥∥2 2 ,
E W i
∥∥ρi(x)∥∥2 2 = 2 ∥∥αi−1(x)∥∥2 2 .
Since ∥∥∥W i(:,1)∥∥∥2
2 is a sum of independent sub-exponential random variables with sub-exponential
norm bounded by C ′
n , Bernstein’s inequality (lemma G.2) and D.2 give P [∥∥ρi(x)∥∥2
2 > C
] ≤ P [∥∥ρi(x)∥∥2 2 − ∥∥αi−1(x)∥∥2 2 > C
2 ∣∣∣∣ ∥∥αi−1(x)∥∥22 ≤ C2 ]
+ P [∥∥αi−1(x)∥∥2
2 > C
2 ] ≤ 2e−cn + C ′e−c nL ≤ C ′′e−c ′ n L
for appropriate constants. Taking a union bound over i gives
P [Eρ] ≥ 1− C ′′Le−c ′ n L ≥ 1− C ′′e−c ′′ n L
for a suitable chosen constant c′′, where we used n ≥ KL logL for some K. We then have
P [ 1EδK ∥∥∥Γ`:`′J vf∥∥∥ 2 > C ] ≤ P [ 1EδK∩Eρ ∥∥∥Γ`:`′J vf∥∥∥ 2 > C ] + P [ 1EδK∩Ecρ ∥∥∥Γ`:`′J vf∥∥∥ 2 > 0 ]
≤ P [ 1EδK∩Eρ ∥∥∥Γ`:`′J vf∥∥∥ 2 > C ] + P [ Ecρ ] ≤ Ce−c nL + C ′e−c ′ n L ≤ C ′′e−c ′′ n L
for appropriate constants, and similarly P [ 1EδK ∥∥∥Γ`:`′J ∥∥∥ > C√L] ≤ C ′′e−c′′ nL . This concludes the proof of the first two statements. For the final result, we consider a vector g with gi ∼iid N (0, 1). Bernstein’s inequality gives P [ ‖g‖22 > 2n ] ≤ e−cn and since
g d = vu ‖g‖2
where vu is uniformly distributed on Sn−1, we can use D.91 setting p = u to obtain
P [∥∥(ΓL:`J (x)− ΓL:`HJ (x)) g∥∥2 > C√dL] ≤ P [∥∥(ΓL:`J (x)− ΓL:`HJ (x))vu∥∥2 > C2 √ dL n ] + P [ ‖g‖2 > 2 √ n ]
≤ e−cn + C ′e−c ′d + C ′′e−c ′′ n L ≤ C ′′′e−c ′′′d.
for appropriate constants, where we assumed n > KLd for some K.
Corollary D.17. Defining
Γ`:` ′ (x) = PI`W `PI`−1 . . .PI`′W `′ ,
under the same assumptions on n,L in D.14 we have P [∥∥∥Γ`:`′(x)v∥∥∥
2 ≤ C
] ≥ 1− e−c nL
P [∥∥∥Γ`:`′(x)∥∥∥ ≤ C√L] ≥ 1− e−c nL
for some numerical constants c, C.
Lemma D.18. Fix a collection of supports J = {J`′ . . . J`} for 1 ≤ `′ ≤ ` ≤ L that satisfy the assumptions of lemma D.14 and denote by vp a unit norm vector. Define an event
E`` ′ H = { 1Eδ ∥∥∥Γ`:`′HJ vp∥∥∥ 2 ≤ C } ∩ { 1Eδ ∥∥∥Γ`:`′HJ ∥∥∥ ≤ C√L} ∩{1Eδ ∥∥∥Γ`:`′HJ ∥∥∥2 F ≤ Cn } .
If n ≥ KL log n for some constant K then P [ E`` ′
H
] ≥ 1− C ′e−c nL
where c, C,C ′ are absolute constants.
Proof. In the following, we denote by W̃ i an independent copy of W i, and by W i(:,j) the j-th column of this matrix. Note that due to rotational invariance of the Gaussian distribution we can replace every occurrence of Hi in an expression by W̃ iPSi−1⊥ without altering the distribution of the expression, which we will do presently. We can repeatedly use this rotational invariance to give
v∗pΓ `:`′∗ HJ Γ `:`′ HJ vp = v ∗ pH `′∗PJ`′Γ `:`′+1∗ HJ Γ `:`′+1 HJ PJ`′H `′vp
d = v∗pPS`′−1⊥W̃ `′∗PJ`′Γ `:`′+1∗ HJ Γ `:`′+1 HJ PJ`′W̃ `′PS`′−1⊥vp
and rotational invariance of the Gaussian distribution gives
v∗pΓ `:`′∗ HJ Γ `:`′ HJ vp d = ‖PS`′⊥vp‖ 2 2 W̃ `′∗ (:,1)PJ`′Γ `:`′+1∗ HJ Γ `:`′+1 HJ PJ`′W̃ `′ (:,1)
d = ‖PS`′⊥vp‖ 2 2 W̃ `′∗ (:,1)PJ`′PS`′+1⊥W̃ `′+1∗Γ`:` ′+2∗ HJ Γ `:`′+2 HJ PJ`′+1W̃ `′+1PS`′+1⊥PJ`′W̃ `′ (:,1)
d = ‖PS`′⊥vp‖ 2 2 ∥∥∥PS`′+1⊥PJ`′W̃ `′(:,1)∥∥∥2 2 W̃ ` ′+1∗ (:,1) PJ`′+1Γ `:`′+2∗ HJ Γ `:`′+2 HJ PJ`′+1W̃ `′+1 (:,1)
and continuing in a similar fashion we obtain
v∗pΓ `:`′∗ HJ Γ `:`′ HJ vp d = ‖PS`′⊥vp‖ 2 2 `−1∏ i=`′ ∥∥∥PSi+1⊥PJiW̃ i(:,1)∥∥∥2 2 ∥∥∥PJ`W̃ `(:,1)∥∥∥2 2
≤ ∏̀ i=`′ ∥∥∥PJiW̃ i(:,1)∥∥∥2 2 a.s.
where in the last inequality we used the fact that multiplication by PSi⊥ cannot increase the norm of a vector. Denoting by {χi} a collection of independent standard chi-squared distributed random variables where χi has |Ji| degrees of freedom, we have
∏̀ i=`′ ∥∥∥PJiW̃ i(:,1)∥∥∥2 2 d = ∏̀ i=`′ 2 n χi.
Define
EI = {
min `′≤i≤`
|Ii(x)| ≥ n
4
} ∩ {∏̀ i=`′ 2 |Ii(x)| n ≤ 2 } .
Denoting δi = |Ji Ii(x)| , on EI we have
min `′≤i≤`
|Ji| ≥ n 4 − n L ≥ n 8 . (D.92)
∏̀ i=`′ 2 |Ji| n ≤ ∏̀ i=`′
2 ( |Ii|+ δi ) n ≤ ∏̀ i=`′ 2 ( |Ii|+ nL ) n = ∏̀ i=`′ 2 |Ii| n ( 1 + n L |Ii| )
≤ ∏̀ i=`′ 2 |Ii| n ( 1 + 4 L ) ≤ e4 ∏̀ i=`′ 2 |Ii| n ≤ 2e4.
where we used the assumption δi ≤ nL and assumed n ≥ 8L. It follows that
P [∣∣∣∣∣∏̀ i=`′ 2 n χi − ∏̀ i=`′ 2 |Ji| n ∣∣∣∣∣ > 1 ∣∣∣∣∣ Eδ ∩ EI ] =P [∣∣∣∣∣∏̀ i=`′ χi |Ji| − 1 ∣∣∣∣∣ > ∏̀ i=`′ n 2 |Ji| ∣∣∣∣∣ Eδ ∩ EI ]
≤P [∣∣∣∣∣∏̀ i=`′ χi |Ji| − 1 ∣∣∣∣∣ > 12e4 ∣∣∣∣∣ Eδ ∩ EI ] .
An application of lemma D.26 and (D.92) then gives
P [∣∣∣∣∣∏̀ i=`′ χi |Ji| − 1 ∣∣∣∣∣ > 12e4 ∣∣∣∣∣ Eδ ∩ EI ] ≤ CLe−c nL ≤ Ce−c ′ n L (D.93)
for appropriate constants, assuming n ≥ KL logL for someK. Using D.30 to bound P [EcI ] we thus have
P [ 1Eδv ∗ pΓ `:`′∗ HJ Γ `:`′ HJ vp > 1 + 2e 4 ] ≤ P [ 1Eδ ∏̀ i=`′ 2 n χi > 1 + ∏̀ i=`′ 2 |Ji| n ]
≤ P [ ∏̀ i=`′ 2 n χi > 1 + ∏̀ i=`′ 2 |Ji| n ∣∣∣∣∣ Eδ ]
≤ P [ ∏̀ i=`′ 2 n χi > 1 + ∏̀ i=`′ 2 |Ji| n ∣∣∣∣∣ Eδ ∩ EI ] + P [EcI ] ≤ Ce−c nL + C ′e−c ′ n L ≤ C ′′e−c ′′ n L
for some constants. Having shown P [ 1Eδv ∗ pΓ `:`′∗ HJ Γ `:`′ HJ vp > C ] ≤ C ′e−c nL
for some fixed vp = vf we can now apply lemma D.20 to obtain P [ 1Eδ ∥∥∥Γ`:`′HJ ∥∥∥ > C√L] ≤ C ′′Le−c′ nL ≤ C ′′e−c′′ nL .
where we used n ≥ KL logL for some K. Choosing vp = êi for i ∈ [n] and taking a union bound, one obtains
P [ 1Eδ ∥∥∥Γ`:`′HJ ∥∥∥2 F > Cn ] = P [ 1Eδ tr [ Γ`:` ′∗ HJ Γ `:`′ HJ ] > Cn ] = P [ n∑ i=1 1Eδ ê ∗ iΓ `:`′∗ HJ Γ `:`′ HJ êi > Cn ]
≤ n∑ i=1 P [ 1Eδ ê ∗ iΓ `:`′∗ HJ Γ `:`′ HJ êi > C ] ≤ nC ′′e−c ′′ n L ≤ C ′e−c nL
for some constants, where we used n ≥ KL log n for an appropriate constant K. A final union bound over the last three events gives the desired result.
Proof of lemma D.15. We first consider the terms ãk. For k = `, the definition of EδKρ in (D.78) gives
ã` = 1EδKρ ∥∥Γ`:`+1HJ PJ`ρ`(x)∥∥2 ‖α`−1(x)‖2 = 1EδKρ ∥∥PJ`ρ`(x)∥∥2 ‖α`−1(x)‖2 ≤ C a.s. (D.94)
In order to handle the case `′ ≤ k < ` , we use (D.81) and obtain that for any 2 ≤ j ≤ i ≤ L,
Γi:jHJPJj−1 ρj−1(x)
‖αj−2(x)‖2 = Γi:j+1HJ PJjH
jPJj−1 ρj−1(x)
‖αj−2(x)‖2
= Γi:j+1HJ PJjH jQj−1(x)
ρj−1(x)
‖αj−2(x)‖2 d = Γi:j+1HJ PJjW̃ jPSj−1⊥Q j−1(x)ρj−1(x)
d = Γi:j+1HJ PJjW̃ j (:,1) ∥∥PSj−1⊥Qj−1(x)ρj−1(x)∥∥2 ‖αj−2(x)‖2
where W̃ j is an independent copy ofW j , and we denote by W̃ j(:,1) the first column of W̃ j , and we used the rotational invariance of the Gaussian distribution. Truncating on the event E i,j+1H , which does not affect the distribution of W̃ j(:,1), we have
E W̃ j
(:,1) ∥∥∥1Ei,j+1H Γi:j+1HJ PJjW̃ j(:,1)∥∥∥22 = 2n1Ei,j+1H tr [PJjΓi:j+1∗HJ Γi:j+1HJ ] ≤ 2 n 1Ei,j+1H tr [ Γi:j+1∗HJ Γ i:j+1 HJ
] = 2
n 1Ei,j+1H ∥∥∥Γi:j+1HJ ∥∥∥2 F ≤ C ′
almost surely for some constant C ′, and the Hanson-Wright inequality (lemma G.4) gives P [∥∥∥1Ei,j+1H Γi:j+1HJ PJjW̃ j(:,1)∥∥∥22 > 1 + C ′ ]
≤ 2 exp −cmin  n 2
4 ∥∥∥1Ei,j+1H Γi:j+1∗HJ Γi:j+1HJ ∥∥∥2F , n 2 ∥∥∥1Ei,j+1H Γi:j+1∗HJ Γi:j+1HJ ∥∥∥

≤ 2 exp ( −c′ n
(i− j + 1) ) for some constant c′ , where we used
∥∥∥1Ei,j+1H Γi:j+1∗HJ Γi:j+1HJ ∥∥∥2F ≤ ∥∥∥1Ei,j+1H Γi:j+1HJ ∥∥∥2F ∥∥∥Γi:j+1HJ ∥∥∥2 ≤ Cn(i− j + 1), and the fact that multiplying a matrix by PJj cannot increase its norm. Writing for concision in the subsequent expression
A = ∣∣∣∥∥∥Γi:j+1HJ PJjW̃ j(:,1)∥∥∥
2 − 1Ei,j+1H ∥∥∥Γi:j+1HJ PJjW̃ j(:,1)∥∥∥ 2 ∣∣∣
it follows that P [∥∥∥Γi:j+1HJ PJjW̃ j(:,1)∥∥∥
2 > C ′′
] ≤ P [ 1Ei,j+1H ∥∥∥Γi:j+1HJ PJjW̃ j(:,1)∥∥∥ 2 > C ′′ ] + P [A > 0]
≤ 2 exp ( −c′ n
L
) + C exp ( −c′′ n
L
) ≤ C ′ exp ( −c n
L ) for appropriate constants, where we used D.18. Since on EδKρ we have∥∥∥∥PSj⊥Qj(x) ρj−1(x)‖αj−2(x)‖2 ∥∥∥∥ 2 ≤ ∥∥∥∥Qj(x) ρj−1(x)‖αj−2(x)‖2 ∥∥∥∥ 2 ≤ 2Ks,
we obtain
P [ 1EδKρ ∥∥∥∥Γi:jHJPJj−1 ρj−1(x)‖αj−2(x)‖2 ∥∥∥∥ 2 > 2C ′′Ks ] ≤ C ′ exp ( −c n L ) (D.95)
hence P [ãk > Ks] ≤ C exp ( −c′ n
L
) (D.96)
for appropriate constants.
We now turn to controlling the b̃qij . Note that
1EδKρ ∣∣∣∣∣ q∏ k=i sk ∣∣∣∣∣ = 1EδKρ ‖αq(x)‖2‖αi−1(x)‖2 ∣∣∣∣∣ q∏ k=i ( 1 + αk(x)∗ ‖αk(x)‖22 Qk(x)W kαk−1(x) )∣∣∣∣∣ ≤ 3 (1 + 2KJ )q−i ≤ 3e2KJL ≤ 9 a.s.
(D.97)
where in the last inequality we used 2KJL < 1. Additionally, we have
1EδKρ αi(x)∗
‖αi(x)∗‖2 Γi−1:j+1HJ PJj
ρj(x)
‖αj(x)‖2 d = 1EδKρ
αi(x)∗PJi−1 ‖αi(x)∗‖2 W̃ i−1PSi−2⊥Γ i−2:j+1 HJ PJj ρj(x)
‖αj(x)‖2 . = 1EδKρ
αi(x)∗PJi−1 ‖αi(x)∗‖2 W̃ i−1u d = σg
where W̃ i−1 is an copy ofW i−1 that is independent of all the other variables, we defined
u = PSi−2⊥Γ i−2:j+1 HJ PJj
ρj(x)
‖αj(x)‖2 and g is a standard normal variable. In the above expression,
σ2 = 1EδKρ 2
n ∥∥αi(x)∗PJi−1/∥∥αi(x)∗∥∥2∥∥22 ‖u‖22 ≤ 1EδKρ 2 ‖u‖22n . Note also that Γi−2:j+1HJ is well-defined since i ≥ j − 1. We therefore have
P [∣∣∣∣1EδKρ αi(x)∗‖αi(x)∗‖2 Γi−1:j+1HJ PJj ρ j(x) ‖αj(x)‖2 ∣∣∣∣ > CKs√L ]
≤ P [ 1EδKρ ‖u‖2 > Ks ] + P [∣∣∣∣∣g √ 2 n Ks ∣∣∣∣∣ > Ks√L ]
≤ C ′e−c nL + 2e−c ′ n L ≤ C ′′e−c nL (D.98)
where we used (D.95) and the Gaussian tail probability to bound the first and second terms in the second line respectively. Combining the above with (D.97) we obtain
P [∣∣∣b̃qij∣∣∣ > Ks√
L
] ≤ C ′′e−c nL . (D.99)
We now turn to controlling c̃pji. If i ≥ `′ we have
c̃pj,i+1 = 1EδKρ
j−1∏ k=i+2 skα i+1(x)∗
‖αi+1(x)∗‖2 Γi:`
′
HJ vp
d = 1EδKρ
j−1∏ k=i+2 skα i+1(x)∗PJi
‖αi+1(x)∗‖2 W̃ iPSi−1⊥Γ
i−1:`′ HJ vp d = σg
(D.100)
where g is a standard normal and
σ2 = 1EδKρ 2
n ( j−1∏ k=i+2 sk )2 ∥∥∥PSi−1⊥Γi−1:`′HJ vp∥∥∥2 2 ≤ 1EδKρ 2C n ∥∥∥Γi−1:`′HJ vp∥∥∥2 2 a.s.
for some constant C where we used (D.97). We also have ∥∥∥Γi−1:`′HJ vp∥∥∥2
2 ≤ C̃ on E i−1,`
′
H . Lemma D.18 and a Gaussian tail bound then give
P [∣∣c̃pj,i+1∣∣ > √ d
n
] ≤
P [ 1EδKρ∩Ei−1,` ′ H ∣∣∣∣∣ j−1∏k=i+2sk αi+1(x)∗Γi:` ′ HJ vp ‖αi+1(x)∗‖2 ∣∣∣∣∣ >√ dn ]
+P [( 1EδKρ − 1EδKρ∩Ei−1,`′H ) ∣∣∣∣∣ j−1∏k=i+2sk αi+1(x)∗Γi:` ′ HJ vp ‖αi+1(x)∗‖2 ∣∣∣∣∣ > 0 ]
≤ P
[√ 2
n C̃g >
√ d
n
] + P [( E i−1,` ′
H
)c] ≤ 2e−cd + Ce−c ′ n L (D.101)
for appropriate constants.
Additionally, if i = `′ − 1 we have from (D.97) for some fixed vp = vf∣∣∣c̃fj,`′ ∣∣∣ = ∣∣∣∣∣1EδKρ j−1∏ k=`′+1 sk α` ′ (x)∗ ‖α`′(x)‖2 Γ` ′−1:`′ HJ vf ∣∣∣∣∣ = ∣∣∣∣∣1EδKρ j−1∏
k=`′+1
sk α` ′ (x)∗
‖α`′(x)‖2 vf ∣∣∣∣∣ = ∣∣∣∣∣1EδKρ j−1∏
k=`′+1
sk ∣∣∣∣∣ ≤ 9 a.s. (D.102)
If however vp = vu is drawn from Unif(Sn−1), if we denote by g a vector with independent standard Gaussian entries we have
1EδKρ α` ′ (x)∗
‖α`′(x)‖2 vu
d = ê∗1vu d = g1 ‖g‖2 .
From Bernstein’s inequality it follows that P [ ‖g‖22 < n 2 ] ≤ e−cn. Combining this with a Gaussian tail bound gives
P [∣∣∣∣ g1‖g‖2 ∣∣∣∣ > √ d n ] ≤ P [ ‖g‖22 < n 2 ] + P [ |g1| > √ d 2 ] ≤ e−cn + 2e−c ′d
for some constants c, c′. From (D.97) it follows that
P [∣∣c̃uj,`′ ∣∣ >√ dn ] = P [∣∣∣∣∣1EδKρ j−1∏
k=`′+1
sk α` ′ (x)∗
‖α`′(x)‖2 vu
∣∣∣∣∣ > √ d n ] ≤ e−cn + 2e−c ′d
for appropriate constants.
Proof of lemma D.16. Part (i). We denote the set of all such terms with r G-chains by ←− Gr,p. Considering first the contribution from the terms with a single G-chain, denoted ←− G1p, we have∑
←−g 1,p∈ ←− G1p
←−g 1,p = ã` ∑̀
j=`′+1
c̃p`,j
where
∑̀ j=`′+1 c̃p`,j = ∑̀ j=`′+1 1EδKρ
`−1∏ k=j+1 skα j(x)∗Γj−1:` ′ HJ v p
‖αj(x)‖2 .
Denoting by σ(A1, . . . , Ak) the sigma-algebra generated by a the random variables A1, . . . , Ak, we define a filtration
F` ′−1 = σ ( vp,ρ 1(x), . . . ,ρL(x) ) ,
Fj = σ ( vp,ρ 1(x), . . . ,ρL(x),H` ′ , . . . ,Hj ) , j = `′, . . . , `.
(D.103)
The sequence {Xi} =
{ 1+i∑
j=`′+1
c̃`,j
} is adapted to the filtration, and since the summands are linear
in the zero mean { Hk }
the sequence is a martingale (E Xi+1| F i = Xi). The martingale difference sequence is
∆i = Xi −Xi−1 = c̃p`,i+1 = 1EδKρ
`−1∏ k=i+2 skα i+1(x)∗Γi:` ′ HJ vp
‖αi+1(x)‖2 giving ∑̀
j=`′+1
c̃p`,j = X`−1 = `−1∑ j=`′+1 ∆i +X`′ = `−1∑ j=`′+1 ∆i + c̃ p `,`′+1.
We cannot control this sum directly because we do not have almost sure control of the martingale differences. To remedy this, we recall the event E i−1,`
′+1 H defined in lemma D.18, and decompose
the sum of interest into∣∣∣∣∣∣ `−1∑
j=`′+1
∆i ∣∣∣∣∣∣ ≤ ∣∣∣∣∣∣ `−1∑
j=`′+1
∆i − 1Ei−1,`′+1H ∆i ∣∣∣∣∣∣+ ∣∣∣∣∣∣ `−1∑
j=`′+1
1Ei−1,` ′+1
H
∆i ∣∣∣∣∣∣ . (D.104) Notice that the second sum is also a sum of zero-mean martingale differences. Using (D.100), we have
1Ei−1,` ′+1
H
∆i d = 1Ei−1,`′+1H σg
where g ∼ N (0, 1) and 1Ei−1,`′+1H σ 2 = 1Ei−1,`′+1H 2 n ( `−1∏ k=i+1 sk )2 ∥∥∥Γi−1:`′HJ vp∥∥∥2 2 ≤ Cn almost surely for some constant C. It follows that
E [ exp ( λ1Ei−1,`′+1H ∆i )∣∣∣F i−1] ≤ exp (cnλ2) ∀λ, a.s. and we can apply Freedman’s inequality for martingales with sub-Gaussian increments (lemma G.7) to conclude that for some d ≥ 0
P ∣∣∣∣∣∣ `−1∑ j=`′ 1Ei−1,` ′+1 H ∆i ∣∣∣∣∣∣ > √d  ≤ 2 exp(−cdn L ) .
As for the first term in (D.104), using lemma D.18 we have
P ∣∣∣∣∣∣ `−1∑ j=`′ ∆i − 1Ei−1,`′+1H ∆i ∣∣∣∣∣∣ > 0  ≤ `−`′∑ i=1 P [( E i−1,` ′+1 H )c] ≤ LC ′e−c nL ≤ C ′e−c ′ n L
for appropriate constants, where we assumed n ≥ KL logL for someK. Combining the above with (D.94), and using (D.101) to give P [∣∣∣ã`c̃p`,`′+1∣∣∣ > C̃] ≤ C ′e−c nL for some constants and applying the triangle inequality, we have
P  ∣∣∣∣∣∣∣ ∑
←−g 1,p∈ ←− G1p
←−g 1,p ∣∣∣∣∣∣∣ > C √ d  ≤ P ∣∣∣ã`c̃p`,`′+1∣∣∣+ |ã`| ∣∣∣∣∣∣ `−1∑ j=`′+1 ∆i ∣∣∣∣∣∣ > C√d
≤ C ′e−c nL + C ′′e−c ′ dn L
(D.105)
for appropriate constants.
Having controlled the sum of terms in ←− G1p, we next consider a sum over the terms in ←− Gr,p for r ≥ 2. The argument will be very similar to the ←− G1p case, with some additional technical details.
Note that since different G-chains must be separated by an Hi matrix for some i and we consider only terms with i1 ≥ `′ + 1, the minimal starting index of the r-th chain (indexed by j below for clarity) is `′ + 1 + 2(r − 1). The sum of all possible terms is thus∑ ←−g r,p∈ ←− Gr,p ←−g r,p =ã` ∑
`′ + 2r − 1 ≤ j ≤ `, (i1, . . . , i2r−2) ∈ Cr−1(`′ + 1, j − 2)
b̃`,j,i2r−2 r−2∏ m=1 b̃i2m+2,i2m+1,i2m c̃ p i2,i1
. = ∑̀ j=`′+2r−1 p̃r`,j
The constraints on the indices i1, . . . , i2r−2 are similar to those in (D.84), with the starting index reflecting the constraint i1 > `′ in the definition of ←− Gr,p. We once again define the filtration F j as
in (D.103) for ` − 1 ≤ j ≤ `. Noting that ã`, b̃k,l,m = 1EδKρ k−1∏ q=l+1 sq αl(x)∗Γl−1:m+1HJ PJmρ m(x) ‖αl(x)‖2‖αm−1(x)‖2 and
c̃pk,l = 1EδKρ
k−1∏ q=l+1 sqα l(x)∗Γl−1:` ′ HJ vp
‖αl(x)‖2 are all F l−1-measurable, the index constraints imply that
Xri = i+1∑ j=`′+2r−1 p̃r`,j
isF i-measurable and thus the sequence {Xri } is adapted to the filtration. b̃`,i+1,i2r−2 is a linear function of the zero-mean variablesHi for any choice of i2r−2, and we can replaceHi with W̃ iPSi−1⊥ where W̃ i is an independent copy of W i without altering the distribution of Xri . Since b̃klm for k ≤ i2r−2 is independent of the W̃ i for any choice of l,m, it follows that p̃r`,i+1 is also a linear function of the variables in W̃ i which have zero mean. Consequently
E Xri | F i−1 = i∑
j=`′+2r−1
p̃r`,j = X r i−1,
hence {Xi} is a martingale sequence. Defining martingale differences ∆ri = X r i −Xri−1 = p̃r`,i+1 we have ∑̀ j=`′+2r−1 p̃r`,j = `−1∑ i=`′+2r−2 ∆ri . (D.106)
We define an event E i∆ ∈ F i by
E i∆ = {|ã`| ≤ C} ∩
⋂ i1+2≤i2≤i3≤i {∣∣∣b̃i3i2i1∣∣∣ ≤ Ks√L} ∩ ⋂ i1≤i2≤i {∣∣c̃pi2i1∣∣ ≤ C√ dn} ∩ ⋂ i1≤i { 1EδKρ ∥∥∥Γi:i1HJPJi1−1 ρi1−1(x)‖αi1−2(x)‖2 ∥∥∥2 ≤ CKs} (D.107)
for i1 ≥ `′ + 1 and decompose the sum in (D.106) into∣∣∣∣∣ `−1∑
i=`′+2r−2
∆ri ∣∣∣∣∣ ≤ ∣∣∣∣∣ `−1∑ i=`′+2r−2 ∆ri − 1Ei−1∆ ∆ r i ∣∣∣∣∣+ ∣∣∣∣∣ `−1∑ i=`′+2r−2 1Ei−1∆ ∆ri ∣∣∣∣∣ . (D.108) In order to control the second term, we note that
1Ei−1∆ ∆ri = 1Ei−1∆ p̃r`,i+1
=1Ei−1∆ ã` ∑ (i1,...,i2r−2)∈Cr−1(`′+1,i−1) b̃`,i+1,i2r−2 r−2∏ m=1 b̃i2m+2,i2m+1,i2m c̃ p i2,i1
=1Ei−1∆ ã` ∑ (i1,...,i2r−2)∈Cr−1(`′+1,i−1) 1EδKρ
`−1∏ q=i+2 sq αi+1(x)∗Γ i:i2r−2+1 HJ PJi2r−2 ρi2r−2 (x) ‖αi+1(x)‖2‖αi2r−2−1(x)‖2
∗ r−2∏ m=1 b̃i2m+2,i2m+1,i2m c̃ p i2,i1
and using Γi:i2r−2+1HJ = PJiH iΓ i−1:i2r−2+1 HJ d = PJ(i)W̃ iPSi−1⊥Γ i−1:i2r−2+1 HJ where W̃
i is an independent copy ofW i gives
1Ei−1∆ ∆ri d = σg
where g ∼ N (0, 1) and
σ =
√ 2
n 1EδKρ∩Ei−1∆ ∣∣∣∣∣∣ `−1∏ q=i+2 sq ∣∣∣∣∣∣ ã` ∥∥∥∥∥∥∥∥ ∑ (i1,...,i2r−2)
∈Cr−1(`′+1,i−1)
P Si−1⊥Γ i−1:i2r−2+1 HJ PJi2r−2 ρi2r−2 (x)
‖αi2r−2−1(x)‖ 2 ∗ r−2∏ m=1 b̃i2m+2,i2m+1,i2m c̃ p i2,i1
∥∥∥∥∥∥∥∥ 2
≤ √ 2
n 1EδKρ∩Ei−1∆ ∣∣∣∣∣∣ `−1∏ q=i+2 sq ∣∣∣∣∣∣ ã` ∑
(i1,...,i2r−2) ∈Cr−1(`′+1,i−1)
∥∥∥Γi−1:i2r−2+1HJ PJi2r−2 ρi2r−2 (x)∥∥∥2 ‖αi2r−2−1(x)‖
2 ∗ ∣∣∣∣ r−2∏ m=1 b̃i2m+2,i2m+1,i2m c̃ p i2,i1 ∣∣∣∣ ≤ √ 2 n1EδKρ∩Ei−1∆ ∣∣∣∣∣ `−1∏q=i+2sq ∣∣∣∣∣ ã`L2r−2 maxi1≤i−11EδKρ∩Ei−1∆ ‖Γi−1:i1HJ PJi1−1ρi1−1(x)‖2‖αi1−2(x)‖2
∗ max i1+2≤i2≤i3≤i−1 ( 1EδKρ∩Ei−1∆ ∣∣∣b̃i3i2i1∣∣∣)r−2 max i1≤i2≤i−1 1EδKρ∩Ei−1∆ ∣∣c̃pi2i1∣∣ ≤ a.s. C √ dL n L2r−2 ( Ks√ L )r−1 = C √ dL n ( L3/2Ks )r−1 .
In the last inequality we used the definition of E i∆ and the assumption L3/2Ks ≤ 1. It follows that
E [ exp ( λ1Ei−1∆ ∆i )∣∣∣F i−1] ≤ exp( cn2λ2 dL ( L3/2Ks )2r−2 ) ∀λ, a.s.
and we can apply Freedman’s inequality for martingales with sub-Gaussian increments (lemma G.7) to conclude
P [∣∣∣∣∣ `−1∑
i=`′+2r−2
1Ei−1∆ ∆i ∣∣∣∣∣ > (L3/2Ks)r−1 √ dL n ] ≤ 2 exp ( −c n L ) . (D.109)
It remains to bound the first term in (D.108). Using lemma D.15, (D.95) and taking a union bound over i1, i2, i3 in (D.107) we have
P [ E i∆ ] ≥ 1− CL3e−c nL − C ′L2 ( e−c n L + e−c ′d ) ≥ 1− C ′′e−c ′′ n L − C ′′′e−c ′′′d
where we assume n ≥ KL logL and d ≥ K ′ logL for some K,K ′. An additional union bound over i gives
P [∣∣∣∣∣ `−1∑
i=`′+2r−2
∆ri − 1Ei−1∆ ∆ r i ∣∣∣∣∣ > 0 ] ≤ `−`′∑ i=1 P [( E i−1∆ )c] ≤ LC ′ ( e−c n L − e−c ′d ) ≤ C ′′ ( e−c ′′ n L − e−c ′′′d )
for appropriate constants. Combining the above with (D.109) and recalling (D.106) gives
P ∣∣∣∣∣∣ ∑̀
j=`′+2r−1
p̃r`,j ∣∣∣∣∣∣ > ( L3/2Ks )r−1√dL n  ≤ Ce−c nL + C ′e−c′d for some constants. L3/2Ks ≤ 12 implies⌈
(`−`′)/2 ⌉∑
r=2
( L3/2Ks )r−1 = L3/2Ks
⌈ (`−`′)/2 ⌉ −2∑
r=0
( L3/2Ks )r
= L3/2Ks
1− (L3/2Ks) ⌈ (`−`′)/2 ⌉ −1
1− L3/2Ks  ≤ 2. A final union bound over r and a rescaling of d gives
P  ∣∣∣∣∣∣∣ ⌈ (`−`′)/2 ⌉∑ r=2 ∑ ←−g r∈ ←− Gr ←−g r ∣∣∣∣∣∣∣ > √ dL n  =P  ∣∣∣∣∣∣∣ ⌈ (`−`′)/2 ⌉∑ r=2 ∑̀ j=`′+2r−1 p̃r`,j ∣∣∣∣∣∣∣ > √ dL n  ≤CLe−c nL + C ′Le−c ′d ≤ C ′′e−c ′′ n L + C ′′′e−c ′′′d
(D.110) for appropriate constants, again assuming assume n ≥ KL logL and d ≥ K ′ logL for some K,K ′. Combining the above with equation (D.105) and worsening constants gives
P  ∣∣∣∣∣∣∣ ⌈ (`−`′)/2 ⌉∑ r=1 ∑ ←−g r∈ ←− Gr ←−g r ∣∣∣∣∣∣∣ > √ dL n  ≤ Ce−c nL + C ′e−c′d for appropriate constants.
Part (ii). We consider terms in the sets −→ Gr,p (with i1 = `′ and i2r ≤ ` − 1). In contrast to the previous section, the bounds on these terms will differ based on the value of the p subscript (denoting whether we use a fixed vector vf or a random vector vu). We first consider −→ G1,p, noting
∑ ←−g 1,p∈ ←− G1p ←−g 1,p = `−1∑ j=`′ ãj c̃ p j,`′ .
Lemma D.15 and a union bound give
P  `−1⋂ j=`′ ãj ≤ Ks ∩ `−1⋂ j=`′ ∣∣∣c̃fj,`′ ∣∣∣ ≤ C ∩ `−1⋂ j=`′ ∣∣c̃uj,`′ ∣∣ ≤ C√d0n
≥ 1− LC ′e−c nL − L ( 2e−c ′d0 − e−c ′′n )
≥ 1− C ′′e−c ′′′ n L − 2e−c ′′′′d0
for appropriate constants, where we assume n ≥ KL logL and d0 ≥ K ′ logL for some constants K,K ′. With the same probability we have∣∣∣∣∣∣∣ ∑ ←−g 1p∈ ←− G1p ←−g 1p ∣∣∣∣∣∣∣ ≤ ∑ ←−g 1p∈ ←− G1p
∣∣←−g 1p∣∣ ≤ CLKsR0p ≤ CR0p (D.111) where we defined
R0u = √ d0 n , R0f = 1
and used LKJ ≤ 1. We next consider sums of terms in −→ Gr,p for r > 1. In controlling the sum of these terms, the proof will proceed along similar lines to the previous section. The main tool we will be utilizing is martingale concentration. Recall that since i2r ≤ ` − 1 and every two G-chains are separated by a matrixHi, the starting index of the final G-chain is no larger than `− 2r + 1. We thus have
∑ −→g r,p∈ −→ Gr,p −→g r,p = ∑
`′ ≤ j ≤ `− 2r + 1, (i3, . . . , i2r) ∈ Cr−1(j + 2, `− 1)
ãi2r r−1∏ m=2 b̃i2m+2,i2m+1,i2m b̃i4,i3,j c̃ p j,`′
. = `−2r+1∑ j=`′ p̃r,pj .
Define a filtration
F0 = σ ( vp,ρ 1(x), . . . ,ρL(x) ) ,
F j = σ ( vp,ρ 1(x), . . . ,ρL(x),H`, . . . ,H`−j+1 ) , j ∈ [`− `′ + 1]
(note the reversed indexing convention compared to the filtration defined in (D.103)). Since p̃r,pj is F`−j-measurable (as can be seen from (D.86)), we can define
Xr,pi = ∑̀ j=`−i p̃r,pj
and it follows that Xr,pi is F i-measurable. Recalling from (D.86) that
b̃i4,i3,j = 1EδKρ i4−1∏ k=i3+1 sk αi3(x)∗Γi3−1:j+1HJ PJjρ j(x) ‖αi3(x)‖2 ‖αj−1(x)‖2
and hence p̃r,pj is linear in the zero-mean variablesH j+1, we have
EXr,pi+1|F i = E ∑̀ j=`−i−1 p̃rj |F i = ∑̀ j=`−i p̃rj + E H1,...,H`−i p̃r`−i−1 = ∑̀ j=`−i p̃rj = Xi
and thus the sequence {Xr,pi } is a martingale with respect to this filtration. Defining martingale differences ∆r,pi = X r,p i −X r,p i−1 = p̃ r,p `−i the sum of interest can be expressed as
`−2r+1∑ i=`′ p̃r,pi = `−`′∑ i=2r−1 ∆r,pi . (D.112)
We now define an event which we will shortly show holds with high probability:
E i−1∆ = ⋂
(i3,...,i2r)∈Cr−1(`−i+2,`−1)
{|ãi2r | ≤ Ks}
∩ r−1⋂ m=2 {∣∣∣b̃i2m+2,i2m+1,i2m ∣∣∣ ≤ Ks√ L } ∩ {∣∣∣c̃f`−i,`′ ∣∣∣ ≤ C} ∩ {∣∣c̃u`−i,`′ ∣∣ ≤ C√d1n }
∩ 1EδKρ ∥∥∥∥∥ i4−1∏k=i3+1skαi3(x)∗Γi3−1:`−i+2HJ PJ`−i+1 ∥∥∥∥∥ 2 ‖αi3(x)‖2 ≤ C √ L  ∩ { 1EδKρ ∥∥PS`−i⊥QJ`−iρ`−i(x)∥∥2 ‖α`−i−1(x)‖2 ≤ 2Ks } .
(D.113) Truncating the martingale difference on such an event gives
1Ei−1∆ ∆r,pi
= 1Ei−1∆ p̃r,p`−i
= 1Ei−1∆ ∑ (i3,...,i2r)∈Cr−1(`−i+2,`−1) ãi2r r−1∏ m=2 b̃i2m+2,i2m+1,i2m b̃i4,i3,`−ic̃ p `−i,`′
= 1Ei−1∆
× ∑
(i3,...,i2r)
ãi2r r−1∏ m=2 b̃i2m+2,i2m+1,i2m1EδKρ i4−1∏ k=i3+1 sk αi3(x)∗Γi3−1:`−i+1HJ PJ`−iρ `−i(x) ‖αi3(x)‖2 ‖α`−i−1(x)‖2 c̃p`−i,`′
d = σpg
for a standard normal g, where we used
Γi3−1:`−i+1HJ PJ`−i ρ`−i(x)
‖α`−i−1(x)‖2
= Γi3−1:`−i+2HJ PJ`−i+1H `−i+1PJ`−i
ρ`−i(x)
‖α`−i−1(x)‖2
= Γi3−1:`−i+2HJ PJ`−i+1H `−i+1QJ`−i
ρ`−i(x)
‖α`−i−1(x)‖2 d = Γi3−1:`−i+2HJ PJ`−i+1W̃ `−i+1PS`−i⊥QJ`−i ρ`−i(x)
‖α`−i−1(x)‖2 with W̃ `−i+1 an independent copy ofW `−i+1 and we have defined
σp =
√ 2 n1Ei−1∆ ∩EδKρ ∣∣∣∣∣ ∑(i3,...,i2r)∈Cr−1(`−i+2,`−1)ãi2r r−1∏ m=2 b̃i2m+2,i2m+1,i2m c̃ p `−i,`′ ∣∣∣∣∣ ∗ ∥∥∥∥∥ i4−1∏k=i3+1skαi3 (x)∗Γi3−1:`−i+2HJ PJ`−i+1 ∥∥∥∥∥ 2
‖αi3 (x)‖2
‖PS`−i⊥QJ`−iρ`−i(x)‖2 ‖α`−i−1(x)‖2 .
Note that from (D.113), if we define
Ru = √ d1 n ,Rf = 1,
the standard deviation σp can be bounded as
σp ≤ a.s. CK2s√ n ( Ks√ L )r−2 L2r−3/2Rp ≤ CRp√ Ln ( L3/2Ks )r−1 where in the first inequality we used a triangle inequality, bounded the number of summands by L2r−2. In the second inequality we used L3/2Ks ≤ 12 . Writing the sum in D.112 as∣∣∣∣∣∣ `−`′∑ i=2r−1 ∆r,pi ∣∣∣∣∣∣ ≤ ∣∣∣∣∣∣ `−`′∑ i=2r−1 ( 1− 1Ei−1∆ ) ∆r,pi ∣∣∣∣∣∣ ≤ ∣∣∣∣∣∣ `−`′∑ i=2r−1 1Ei−1∆ ∆r,pi
∣∣∣∣∣∣ , and recognizing that the second sum is over a zero-mean adapted sequence that obeys
E [ exp ( λ1Ei−1∆ ∆i )∣∣∣F i−1] ≤ exp( cnλ2 R2p ( L3/2Ks )2r−2 ) ∀λ, a.s.
an application of Freedman’s inequality for martingales with sub-Gaussian increments (lemma G.7) gives
P ∣∣∣∣∣∣ `−`′∑ i=2r−1 1Ei−1∆ ∆r,pi ∣∣∣∣∣∣ > Rp ( L3/2Ks )r−1 ≤ 2 exp( −t2 2Lσ2p ) ≤ a.s. 2e−cn.
Turning now to controlling the probability of E i−1∆ holding, we use lemmas D.15, D.18, the definition of KJ and a union bound to conclude
P [( E i−1∆ )c] ≤ L3Ce−c nL + L(2e−c′d1 + e−c′′n)+ L2C ′e−c′′′ nL ≤ C ′′e−c′′′′ nL + e−c′d1 for appropriate constants, where we assumed n ≥ KL logL, d1 ≥ K logL for some K,K ′. Combining the previous two results gives
P [∣∣∣∣∣ `−2r+1∑ i=`′ p̃r,pi ∣∣∣∣∣ > Rp (L3/2Ks)r−1 ] = P ∣∣∣∣∣∣ `−`′∑ i=2r−1 ∆r,pi ∣∣∣∣∣∣ > Rp ( L3/2Ks )r−1 ≤ P ∣∣∣∣∣∣ `−`′∑ i=2r−1 1Ei−1∆ ∆r,pi ∣∣∣∣∣∣ > Rp ( L3/2Ks )r−1+ P ∣∣∣∣∣∣ `−`′∑ i=2r−1 ( 1− 1Ei−1∆ ) ∆r,pi ∣∣∣∣∣∣ > 0
≤ 2e−cn + LP [( E i−1∆ )c] ≤ Ce−c′ nL + e−c′′d1 for some constants. Noting as before that ⌈ (`−`′)/2 ⌉∑ r=2 ( L3/2Ks
)r−1 ≤ 2, using (D.111) to bound the terms with r = 1, and setting d0 = d1 we obtain
P  ∣∣∣∣∣∣∣ ⌈ (`−`′)/2 ⌉∑ r=2 ∑ −→g r,p∈ −→ Gr,p −→g r,p ∣∣∣∣∣∣∣ > CRp  ≤ LC (e−c nL + e−c′d1) ≤ C (e−c′′ nL + e−c′′′d1)
for appropriate constants, where we used again n ≥ KL logL, d1 ≥ K logL for some K,K ′.
Lemma D.19. (Horn et al., 1994) Given a semidefinite matrixA, for any partitioning
A =  A11 A12 . . . A1b A21 A22
... . . .
Ab1 Abb  we have ‖A‖ ≤
b∑ i=1 ‖Aii‖.
Lemma D.20. Given a semidefinite matrixA and unit norm v, if P [v∗Av ≤ C ′] ≥ 1− C`p exp ( −c1 n
` ) and n > 2 log(9)`c1 , then
P [‖A‖ ≤ C ′′′`] ≥ 1− C ′′`p+1 exp ( −c′n
` ) for some constants c, c′, C, C ′′, C ′′′.
Proof. We partition A into blocks of size c2n` for an appropriately chosen c2. There are ` c2 such blocks, and we similarly partition the coordinates {1, . . . , n} into `c2 sets Ki = {1 + (i − 1) c2n ` : i c2n` } for i ∈ [ ` c2 ].
We proceed to bound the operator norm of the diagonal blocks using a standard ε-net argument (Vershynin, 2018). The set of unit norm vectors supported on someKi forms a sphere S c2n ` . We can thus construct a 14 -net Ni on this sphere with at most e log(9)c2 n ` points. A standard argument gives
‖Aii‖ ≤ C sup x∈Ni ‖x∗Aiix‖ .
We control the RHS by a taking a union bound over the net, finding P [
sup x∈Ni
‖x∗Aiix‖ ≤ C ′ ] = P [
sup x∈Ni
‖x∗Ax‖ ≤ C ′ ]
≥ 1− |Ni|C`p exp ( −c1 n
`
) ≥ 1− C`p exp ( (log(9)c2 − c1) n
`
) .
We now choose c2 to satisfy log(9)c2 = c12 , and the blocks will still have non-zero size because we assume n > 2 log(9)`c1 . Taking a union bound over the ` c2 blocks and using Lemma D.19 gives
‖A‖ ≤ ` c2∑ i=1 ‖Aii‖ ≤ C`
w.p. P ≥ 1− C ′`p+1 exp ( −cn` ) for some constants c, C,C ′.
Lemma D.21. Assume n ≥ max {KL log n,K ′Ldb,K ′′}, db ≥ K ′′′ logL for suitably chosen K,K ′,K ′′,K ′′′. Define J as in Lemma D.14. For x ∈ Sn0−1 and
β`J = ( WL+1PJLW L . . .W `+2PJ`+1 )∗ ,
denote di = |Ii(x) Ji| , d = (d1, . . . , dL),
and dmin = min i di. We then have
P [ 1EδK ∥∥β`J − β`(x)∥∥2 > C√dbL+ C √ s (∥∥d∥∥ 1 + 2‖d‖2 1 Ls n + √ Ldb n ∥∥d∥∥ 1 ∥∥d∥∥1/2 1/2 )] ≤ e−cmax{dmin,1}s + e−c′ nL + e−c′′db
for absolute constants c, c′, c′′, C, C ′, C ′′, C ′′′, where the event EδK is defined in lemma D.14. Other useful forms of this result are
P 1EδK ∥∥β`J − β`(x)∥∥2 > C√dbL+ C √√√√Lns(L2s+ ∥∥d∥∥1
n + √ Ldb n ∥∥d∥∥ 121 2 ) + CL 〈 s,d 〉 ≤ e−cs + e−c ′ n L + e−c ′′db +
L∑ i=` e−csi max{di,1}
where si ≥ 1.
Proof. Denoting by { Hi }
the weight matrices projected onto the subspace orthogonal to the features as in lemma D.14, we define
β̂`H(x) = H `+1∗β`H(x) = ( WL+1PIL(x)H L . . .H`+2PI`+1(x)H `+1 )∗
β̂`HJ = H `+1∗β`HJ = ( WL+1PJLH L . . .H`+2PJ`+1H `+1 )∗
for ` = 0, . . . , L−1. Note the additional matrix compared to the standard definition of the backward features. Control of the norm of the difference between them can then be used to control the backward features and Lipschitz constant of the network. Note also that H` may not be a square matrix (and indeed in the case of the Lipschitz constant it will be rectangular). We denote the number of columns ofH`+1 by n`−1.
Writing
1EδK ∥∥∥β̂`J − β̂`(x)∥∥∥ 2 ≤ 1EδK ∥∥∥β̂`HJ − β̂`H(x)∥∥∥ 2 + 1EδK ∥∥∥β̂`J − β̂`HJ (x)∥∥∥ 2
+ 1EδK ∥∥∥β̂`(x)− β̂`H(x)∥∥∥ 2 ,
(D.114)
we begin by bounding the first term. For Γi:jH (x),Γ i:j HJ defined as in D.14 and
Qi(x) = PJi − PIi(x), (D.115)
we have
1EδK ∥∥∥β̂`HJ − β̂`H(x)∥∥∥2 2
=1EδK ∥∥WL+1 (ΓL:`+1HJ − ΓL:`+1H (x))∥∥22
=1EδK ∥∥∥∥∥WL+1 L∑
i=`+1
ΓL:i+1H Q i(x)HiΓi−1:`+1HJ ∥∥∥∥∥ 2
2
. = ∥∥∥∥∥ L∑
i=`+1
bi ∥∥∥∥∥ 2
2
.
We first bound ‖bi‖22. Repeated use of the rotational invariance of the Gaussian distribution in a similar manner to the proof of lemma D.18 gives
‖bi‖22 = 1EδK ∥∥WL+1ΓL:i+1H Qi(x)HiΓi−1:`+1HJ ∥∥22
d = n
2 L∏ k=i+1 ∥∥∥Hk+1(1,:)1EδKPIk(x)∥∥∥2 2︸ ︷︷ ︸
. =ξIk(x)
∥∥∥Hi+1(1,:)1EδKQi(x)∥∥∥2 2 i−1∏ k=`+1 ∥∥∥Hk+1(1,:)1EδKPJk∥∥∥2 2︸ ︷︷ ︸
. =ξJk
∥∥∥H`+1(1,:)∥∥∥2 2
where we defined HL+1(1,:) = √ 2 nW
L+1. Denoting by W̃ k an independent copy of W k, rotational invariance gives
ξJk ≤ ∥∥∥W̃ k+1(1,:)1EδKPJk∥∥∥2
2
d =
2 n χk
where χk is a standard chi-squared distributed random variable with |suppdiag1EδKPJk | degrees of freedom that is independent of all the other variables in the problem, and similarly for ξIk(x).
A product of such terms was bounded in lemma D.18, from which we obtain
P
[ n
2 L∏ k=i+1 ξIk(x) i−1∏ k=`+1 ξJk > Cn
] ≤ e−c nL . (D.116)
We similarly have ∥∥∥Hi+1(1,:)1EδKQi(x)∥∥∥2 2 ≤ a.s. ∥∥∥1EδKQi(x)W̃ k+1∗(1,:) ∥∥∥2 2 .
Recalling (D.115) and since di = ∣∣suppdiagQi(x)∣∣ we recognize that ∥∥∥1EδKQi(x)W̃ k+1∗(1,:) ∥∥∥2
2
d = 1EδK
2 n χi
where χi is a standard chi-squared distributed random variable with di degrees of freedom. If di = 0 this variable is identically 0. Otherwise, di ≥ 1 and Bernstein’s inequality gives
P [χi − di > Csdi] ≤ 2e−csdi ⇒ P [χi > C ′sdi] ≤ 2e−csdi ≤ 2e−csmax{di,1} (D.117)
for some constants and s ≥ 1. Clearly this result also holds if di = 0. Similarly, ∥∥∥H`+1(1,:)∥∥∥2
2 is
bounded almost surely by 2nχ`+1 where χ`+1 has n`−1 degrees of freedom. Bernstein’s inequality gives P [∥∥∥H`+1(1,:)∥∥∥2
2 > C 1n t
] ≤ e−ct for t > Kn`−1 for some K. Combining these results with
(D.116) and taking a union bound, we obtain
P
[ L∑
i=`+1
‖bi‖22 > C 1
n t L∑ i=`+1 sidi
] ≤ 2
L∑ i=`+1 e−csi max{di,1} + 2L(e−c ′t + C ′e−c ′′ n L )
≤ 2 L∑
i=`+1
e−csi max{di,1} + e−c ′′′t + e−c ′′′′ n L
(D.118)
for appropriate constants, assuming t ≥ K logL, n ≥ K ′L logL for some K,K ′, which can be simplified to
P [ L∑ i=` ‖bi‖22 > C 1 n ts L∑ i=`+1 di ] ≤ 2 L∑ i=` e−csmax{di,1} + e−c ′′′n`t + e−c ′′′′ n L
≤ 2Le−cs + e−c′′′n`t + e−c′′′′ nL ≤ e−c′s + e−c′′′n`t + e−c′′′′ nL (D.119)
assuming s ≥ K ′′ logL for some K ′′. We next bound |〈bi, bj〉| for ` ≤ j < i ≤ L. Once again using rotational invariance starting from the last layer weights, we obtain
〈bi, bj〉 =1EδKWL+1Γ L:i+1 H (x)Q i(x)HiΓi−1:`+1HJ Γ j−1:`+1∗ HJ H j∗Qj(x)ΓL:j+1∗H (x)W L+1
d = n
2 1EδK L∏ k=i+2 ξIk(x)H i+1 (1,:)Q i(x)HiΓi−1:`+1HJ Γ j−1:`+1∗ HJ︸ ︷︷ ︸
. =Φi−1:j−1
Hj∗Qj(x)Γi:j+1∗H (x)H i+1∗ (:,1)
(where we interpret an empty product as unity). As before, we find using lemma D.18 that
P [ 1EδK n
2 L∏ k=i+2 ξIk(x) > Cn
] ≤ C ′e−c nL . (D.120)
We proceed to bound the remaining factors in 〈bi, bj〉, by first writing
〈bi, bj〉 = 1EδK n
2 L∏ k=i+2 ξIk(x)H i+1 (1,:)Q i(x)HiΦi−1:j−1Hj∗Qj(x)Γi:j+1∗H (x)H i+1∗ (:,1)
d = 1EδK
n
2 L∏ k=i+2 ξIk(x) di∑ ki=1 dj∑ kj=1 Hi+1(1,ki)skiH i (ki,:) Φi−1:j−1Hj∗(:,kj)skjH j+1∗ (kj ,1) ∥∥ui+1:j+1∥∥ 2
where ui+1:j+1 = PIj+1Γ i:j+2∗ H H i+1∗ (:,1) and skm ∈ {−1, 1} are the signs of the elements inQ m(x) for m ∈ {i, j}. In the above expression, km index the entries on which diagQm is supported, and we denote dm = |suppdiagQm| and use the permutation symmetry of the Gaussian distribution to set these to be [dm].
If i > j + 1, defining Ĥj+1 = Ŵ j+1PSj⊥ where Ŵ j+1 is an independent copy of W j+1, with Γ̂i−1:`+1HJ denoting the matrix Γ i−1:`+1 HJ with Ĥ j+1 (1,:) in place ofH j+1 (1,:), and writing for concision
Ξ ki,kj i,j = H i+1 (1,ki) Hi(ki,:)Γ i−1:j+2 HJ PJj+1(:,1) ( Hj+1(1,:) − Ĥ j+1 (1,:) ) Φj:j−1Hj∗(:,kj)H j+1∗ (kj ,1) ∥∥ui+1:j+1∥∥ 2
and
Ψ ki,kj i,j = H i+1 (1,ki) Hi(ki,:)Γ̂ i−1:`+1 HJ Γ j−1:`+1∗ HJ H j∗ (:,kj) Hj+1∗(kj ,1) ∥∥ui+1:j+1∥∥ 2
we have
〈bi, bj〉 d =
1EδK n 2 L∏ k=i+2 ξIk(x) di∑ ki=1 dj∑ kj=1 Ξ ki,jj i,j
+1EδK n 2 L∏ k=i+2 ξIk(x) di∑ ki=1 dj∑ kj=1 Ψ ki,kj i,j
. =
n
2
( Ai,j1 +A i,j 2 ) ,
(D.121)
where we used the invariance of the Gaussian distribution to reflections around the mean, {Hm} d={ W̃mPSm−1⊥ } , and the independence between the { W̃m } variables and the sign variables {skm}
to absorb the latter into the former. Making a separate definition for concision
1EδK di∑ ki=1 Hi+1(1,ki)H i (ki,:)
Γi−1:j+2HJ PJj+1(:,1)︸ ︷︷ ︸ . =Bi,j1
we first consider the term
Ai,j1 d = Bi,j1  1EδK dj∑ kj=1 Hj+1(1,:)Φ j:j−1Hj∗(:,kj)H j+1∗ (kj ,1)︸ ︷︷ ︸ . =Bi,j2 −1EδK dj∑ kj=1 Ĥj+1(1,:)Φ j:j−1Hj∗(:,kj)H j+1∗ (kj ,1)︸ ︷︷ ︸
. =Bi,j3
 1EδK ∥∥ui+1:j+1∥∥ 2︸ ︷︷ ︸ . =Bi,j4 .
Lemma D.22 gives P [∣∣∣Bi,j4 ∣∣∣ > C] = P [1EδK ∥∥ui+1:j+1∥∥2 > C] ≤ C ′′′e−c′′ nL . (D.122)
We next consider Bi,j3 . Writing
Bi,j3 = 1EδK dj∑ kj=1 Ĥj+1(1,:)Φ j:j−1Hj∗(:,kj)H j+1∗ (kj ,1) d = 1EδK dj∑ kj=1 Ĥj+1(1,:)Φ j:j−1Hj∗(:,kj)PSj⊥W̃ j+1∗ (kj ,1)
First, since the variables { W̃ j+1∗(kj ,1) } are independent of { 1EδKĤ j+1 (1,:)Φ j:j−1Hj∗(:,kj) (PJ`)(kj ,kj) } , a Gaussian tail bound gives
P  ∣∣∣∣∣∣ dj∑ kj=1 Ĥj+1(1,:)Φ j:j−1Hj∗(:,kj)H j+1∗ (kj ,1) ∣∣∣∣∣∣ > √√√√√2d n dj∑ kj=1 1EδK ( Ĥj+1(1,:)Φ j:j−1Hj∗(:,kj) (PJ`)(kj ,kj) )2 ≤ e−cd
(D.123)
for some constants and d ≥ K for some K.
Two applications lemma D.22 give
P  dj∑ kj=1 1EδK ( Ĥj+1(1,:)Φ j:j−1Hj∗(:,kj) (PJ`)(kj ,kj) )2 > Cdj 1 n t  ≤
dj∑ kj=1 P [ 1EδK ( Ĥj+1(1,:)Φ j:j−1Hj∗(:,kj) (PJ`)(kj ,kj) )2 > C 1 n t ] ≤ dj ( e−c n L + e−c ′t ) ≤ e−c ′′ n L + e−c ′′′t
assuming t ≥ K log n, n ≥ K ′L log n for some K. Combining this bound with (D.123) we obtain
P [∣∣∣Bi,j3 ∣∣∣ > C√ddjtn ] ≤ e−cd + e−c ′ n L + e−c ′′t (D.124)
for appropriate constants.
We now turn to bounding Bi,j2 . Define byQ j a matrix such that Q j ab = ∣∣∣Qjab(x)∣∣∣. Then
Bi,j2 d = 1EδKW̃ j+1 (1,:)PSj⊥Φ j:j−1Hj∗Q j PSj⊥W̃ j+1∗ (:,1) .
In order to bound this term using the Hanson-Wright inequality, we first note that since∥∥∥PSj⊥Φj:j−1Hj∗QjPSj⊥∥∥∥ ≤ ∥∥Φj:j−1Hj∗∥∥ ≤ ∥∥∥Γj:`+1HJ ∥∥∥∥∥∥Γj−1:`+1∗HJ ∥∥∥∥∥Hj∗∥∥ ,∥∥∥PSj⊥Φj:j−1Hj∗QjPSj⊥∥∥∥2 F ≤ ∥∥Φj:j−1Hj∗∥∥2 ∥∥∥Qj∥∥∥2 F = ∥∥Φj:j−1Hj∗∥∥2 dj ,
we can use lemma D.14, a standard ε-net argument to control the operator norm of a Gaussian matrix and a union bound to obtain
P  {∥∥∥1EδKPSj⊥Φj:j−1Hj∗QjPSj⊥∥∥∥ ≤ CL}
∩ {∥∥∥1EδKPSj⊥Φj:j−1Hj∗QjPSj⊥∥∥∥2
F ≤ CL2dj }  ≥ 1− e−c nL + e−c′n ≥ 1− e−c′′ nL . We also have
E W̃ j+1 W̃ j+1(1,:)PSj⊥Φ j:j−1Hj∗Q j PSj⊥W̃ j+1∗ (:,1) =
2 n tr [ PSj⊥Φ j:j−1Hj∗Q j PSj⊥ ] = 2
n dj∑ kj=1 ê∗kjPSj⊥Γ j:`+1 HJ Γ j−1:`+1∗ HJ H j∗êkj
and using lemmas D.14 and D.22 gives
P ∣∣∣∣∣∣ dj∑ kj=1 ê∗kjPSj⊥Γ j:`+1 HJ Γ j−1:`+1∗ HJ H j∗êkj ∣∣∣∣∣∣ ≤ C djn  ≥ 1− dje−c nL ≥ 1− e−c′ nL .
assuming n > KL log n for some K. Denoting the union of these two events by G, an application of the Hanson-Wright inequality (lemma (G.4)) gives
P [ 1G ∣∣∣Bi,j2 ∣∣∣ > s2 + 2Cdjn ] ≤ C exp ( −cmin { n2s22 L2dj , ns2 L }) (D.125)
for appropriate constants and s2 ≥ 0, and an additional union bound gives P [∣∣∣Bi,j2 ∣∣∣ > s2 + 2Cdjn ] ≤ C exp ( −cmin { n2s22 L2dj , ns2 L }) + e− n L . (D.126)
We next turn to bounding ∣∣∣Bi,j1 ∣∣∣. Rotational invariance of the Gaussian distribution gives
Bi,j1 = 1EδK di∑ ki=1 Hi+1(1,ki)H i (ki,:) Γi−1:j+2HJ PJj+1(:,1)
d = 1EδK di∑ ki=1 Hi+1(1,ki)W̃ i (ki,1) ∥∥∥PJi−1Γi−1:j+2HJ PJj+1(:,1)∥∥∥ 2
since PJi−1Γ i−1:j+2 HJ PJj+1(:,1) and W̃ i (ki,1) are independent.
Since Hi+1(1,ki), W̃ i (1,ki) are both sub-Gaussian with sub-Gaussian norm bounded by C ′ √ n
, the product of two such variables is a sub-exponential variable with sub exponential norm satisfying∥∥∥Hi+1(1,ki)W̃ i(ki,1)∥∥∥ψ1 ≤ Cn for some constants. Thus the first sum above is a sum of independent, zero-mean sub-exponential random variables, and Bernstein’s inequality gives
P [∣∣∣∣∣ di∑ ki=1 Hi+1(1,ki)W̃ i (ki,1) ∣∣∣∣∣ > s1 ] ≤ 2e−cmin{ n2s21 di ,ns1} (D.127)
for s1 ≥ 1 and some constant c. Since ∥∥∥1EδKPJi−1Γi−1:j+2HJ PJj+1(:,1)∥∥∥ 2 ≤ ∥∥∥1EδKΓi−1:j+2HJ ê1∥∥∥ 2 we can apply lemma D.14 to ob-
tain P [∥∥∥1EδKPJi−1Γi−1:j+2HJ PJj+1(:,1)∥∥∥
2 > C
] ≤ C ′e−c nL
for appropriate constants. Combining the last two results gives P [∣∣∣Bi,j1 ∣∣∣ > Cs1] ≤ e−cmin{n2s21di ,ns1} + e−c′ nL (D.128)
for some constants.
Combining the above with (D.122), (D.124) and (D.126), we have
P [∣∣∣Aij1 ∣∣∣ ≥ C ( s2 +
2dj n +
√ ddjt
n
) s1 ]
≤ e−cmin{ n2s21 di ,ns1} + e−c ′ n L + e−c ′′d + e −c′′′min { n2s22 L2dj , ns2 L } + e−c ′′′′t
(D.129)
In the above proof we assumed i > j + 1. If instead i = j + 1 we simply set Γ̂i−1:`+1HJ = Γ i−1:`+1 HJ in the expression for Ai,j2 in (D.121) and we have 〈bj+1, bj〉 = A j+1,j 2 .
We now turn to controlling the term Ai,j2 . Since i > j, H i (ki,:) d = W̃ i(ki,:)PSi−1⊥ and PSi−1⊥Γ̂ i−1:`+1 HJ Γ j−1:`+1∗ HJ H j∗ (:,kj)
is independent of W̃ i(ki,:), rotational invariance of the Gaussian distribution gives
Ai,j2 d = di∑ ki=1
Hi+1(1,ki)W̃ i (ki,1)︸ ︷︷ ︸
. =Ci1
× 1EδK  dj∑ kj=1 ∥∥∥PSi−1⊥Γ̂i−1:`+1HJ Γj−1:`+1∗HJ Hj∗(:,kj)∥∥∥2Hj+1∗(kj ,1) ∥∥ui+1:j+1∥∥
2︸ ︷︷ ︸ . =Ci,j2
.
(D.130)
We bound ∣∣Ci1∣∣ using (D.127).
It remains to control ∣∣∣Ci,j2 ∣∣∣. SinceHj+1∗(kj ,1) d= (PSj⊥)(kj ,kj) W̃ j+1(1,kj) and W̃ j+1(1,kj) are independent of
(PSj⊥)(kj ,kj) ∥∥∥PSi−1⊥Γ̂i−1:`+1HJ Γj−1:`+1∗HJ Hj∗(:,kj)∥∥∥2, the second factor in (D.130) is also zero-mean, and it follows that
P  ∣∣∣∣∣1EδK dj∑kj=1 (PSj⊥)(kj ,kj) ∥∥∥PSi−1⊥Γ̂i−1:`+1HJ Γj−1:`+1∗HJ Hj∗(:,kj)∥∥∥2 W̃ j+1(1,kj) ∣∣∣∣∣
> √ 2d n dj∑ kj=1 1EδK (PSj⊥)(kj ,kj) ∥∥∥PSi−1⊥Γ̂i−1:`+1HJ Γj−1:`+1∗HJ Hj∗(:,kj)∥∥∥22
 ≤ C ′e−cd
for some constants and d ≥ 0. Since ∥∥∥PSi−1⊥Γ̂i−1:`+1HJ Γj−1:`+1∗HJ Hj∗(:,kj)∥∥∥22 ≤∥∥∥Γ̂i−1:`+1HJ ∥∥∥2 ∥∥∥Γj−1:`+1∗HJ Hj∗(:,kj)∥∥∥22, applying lemmas D.22 and D.14 total of dj times and taking a union bound gives
P  dj∑ kj=1 1EδK (PSj⊥)(kj ,kj) ∥∥∥PSi−1⊥Γ̂i−1:`+1HJ Γj−1:`+1∗HJ Hj∗(:,kj)∥∥∥22 > C djLtn
≤ dj ( e−c n L + e−c ′t ) ≤ e−c ′ n L + e−c ′t
where we assumed n ≥ KL log n, t ≥ K ′ log n for some constants. Combining the above three results with (D.122) and taking a union bound, we obtain
P [∣∣∣Ai,j2 ∣∣∣ > Cs1√djdn`−1Ltn ] ≤ e−cd + e−c ′ n L + e−c ′′t + e−c ′′′t + e −c′′′′min{n 2s21 di ,ns1}
for appropriate constants.
Taking a union bound over this result as well as (D.129) and (D.120) allows us to bound the inner product by
P [ |〈bi, bj〉| ≥ Cns1 ( s2 +
dj n +
√ djdn`−1Lt
n
)]
≤ e−cmin{ n2s21 di ,ns1} + e−c ′ n L + e−cd + e −c′′′min { n2s22 L2dj , ns2 L } + e−c ′′′′t
(D.131)
for some constants, again assuming n ≥ KLd . At this point we obtain a bound on the sum of these inner products that will be useful in an application where the {di} are expected to be small. Subsequently, we will derive a different expression that will be useful when they are large.
We now choose s1 = disn , s2 = djLs n , t = n n`−1 for some s ≥ 1, which gives
P [ |〈bi, bj〉| ≥ Cdis ( djLs
n + dj n + √ Lddj n )] ≤ C ′e−cmin{di,di}s + C ′′e−c ′ n L + C ′′′e−c ′′d
for appropriately chosen constants. Note that if di = 0 or dj = 0 then |〈bi, bj〉| is identically zero, hence we can replace the term min{di, di} in the tail above by max{1,min{di, di}} and the result will still hold if di = 0 or dj = 0. Lower bounding the second expression by dmin = min
i di gives
P [ |〈bi, bj〉| ≥ Cdis ( 2djLs
n + √ Lddj n )] ≤ C ′e−cmax{dmin,1}s + C ′′e−c ′ n L + C ′′′e−c ′′d.
Recalling the definition of d in the lemma statement, an additional union bound over the values of i, j in the expression above combined with (D.119) gives
P [ 1EδK ∥∥∥β̂`HJ − β̂`H(x)∥∥∥2 2 > Cs (∥∥d∥∥ 1 + 2 ∥∥d∥∥2 1 Ls n + √ Ld n ∥∥d∥∥ 1 ∥∥d∥∥1/2 1/2 )]
≤ P 1EδK ∥∥∥β̂`HJ − β̂`H(x)∥∥∥2 2 > Cs L∑ i,j=`+1,i6=j di ( 2djLs n + √ Lddj n ) + Cs L∑ i=`+1 di  ≤ L2 ( C ′e−cmax{dmin,1}s + C ′′e−c ′ n L + C ′′′e−c ′′d )
≤ C ′e−c ′′′max{dmin,1}s + C ′′e−c ′′′′ n L + C ′′′e−c ′′′′′d
C ′e−c ′′′s + C ′′e−c ′′′′ n L + C ′′′e−c ′′′′′d
(D.132) for appropriate constants, where we assumed d ≥ K logL, s ≥ max{1,K ′ logL}, n ≥ K ′′L logL for some K,K ′,K ′′. Taking a square root gives a bound on the first term in (D.114).
We next consider a different bound for this term that will be useful when the {di} are large. Our starting point will be D.131. If we set s1 = s, s2 = Ls and use (D.118) we obtain
P [ 1EδK ∥∥∥β̂`HJ − β̂`H(x)∥∥∥2 2 > CLns ( L2s+ ∥∥d∥∥ 1 n + √ Ldt n ∥∥d∥∥1/2 1/2 ) + C t n L∑ i=`+1 sidi ]
≤ P 1EδK ∥∥∥β̂`HJ − β̂`H(x)∥∥∥2 2 > Cns L∑ i,j=`+1,i6=j ( Ls+ dj n + √ Lddjt n ) + C t n L∑ i=`+1 sidi  ≤ L2 ( C ′ exp ( −c̃min{ n
2s2∥∥d∥∥∞ , ns} ) + C ′′e−c̃ ′ n L + C ′′′e−c̃ ′′d )
+ L∑ i=`+1 e−c̃si max{di,1} + e−c̃ ′′′t
≤ e−cns + e−c ′ n L + e−c ′′d + L∑ i=`+1 e−c ′′′si max{di,1} + e−c ′′′′t
(D.133) for appropriate constants under similar assumptions on n,L, d.
To bound the remaining terms in (D.114), since
1EδK ∥∥∥β̂`J − β̂`HJ ∥∥∥ 2 ≤ 1EδK ∥∥WL+1 (ΓL:`+1J − ΓL:`+1HJ )∥∥2 ∥∥H`+1∥∥ d = 1EδK
∥∥(ΓL:`+1J − ΓL:`+1HJ )WL+1∗∥∥2 ∥∥H`+1∥∥ and we can apply lemma D.14 and an ε-net argument to bound the first and second factors respectively, to conclude
P [ 1EδK ∥∥∥β̂`J − β̂`HJ ∥∥∥ 2 > C √ dL ] ≤ C ′e−cd + Cec ′n ≤ e−cd
for some d such that d ≥ K logL and assuming n > K ′d. An identical result holds for the last term in (D.114) where we simply choose Ji = Ii(x) for all i. In conclusion, using (D.132) we have
P [ 1EδK ∥∥∥β̂`J − β̂`(x)∥∥∥ 2 > C √ dL+ C √ s (∥∥d∥∥ 1 + 2‖d‖2 1 Ls n + √ Ld n ∥∥d∥∥ 1 ∥∥d∥∥1/2 1/2 )] ≤ C ′e−cs + C ′′e−c′ nL + C ′′′e−c′′d
(D.134)
for appropriate constants, while if we use (D.133) instead we obtain
P [ 1EδK ∥∥∥β̂`J − β̂`(x)∥∥∥ 2 > C √ dL+ C √ Lns ( L2s+ ‖d‖ 1 n + √ Ldt n ∥∥d∥∥ 121 2 ) + CLtn 〈 s,d 〉] ≤ e−cs + e−c′ nL + e−c′′d +
L∑ i=`+1 e−c ′′′si max{di,1} + e−c ′′′′t
.
(D.135) It remains to transfer control from ∥∥∥β̂`HJ − β̂`H(x)∥∥∥ 2 to ∥∥β`HJ − β`H(x)∥∥2. Note that∥∥∥β̂`HJ − β̂`H(x)∥∥∥2 2 = ∥∥H`+1∗ (β`HJ − β`H(x))∥∥22 d= ∥∥∥PS`⊥W̃ `+1∗(:,1) ∥∥∥22 ∥∥β`HJ − β`H(x)∥∥22
where if ` = 0 we define PS0⊥ = In0×n0 . Since E W̃ `+1 ∥∥∥PS`⊥W̃ `+1(:,1)∥∥∥2 2 = 2n tr [PS`⊥ ] = 2(n−1) n , Bernstein’s inequality gives (assuming n > K for some K)
P [∥∥∥PS`⊥W̃ `+1∗(:,1) ∥∥∥2
2 <
1
C
] ≤ e−cn,
and hence with the same probability ∥∥β`HJ − β`H(x)∥∥22 ≤ ∥∥∥β̂`HJ − β̂`H(x)∥∥∥2
2∥∥∥PS`⊥W̃ `+1∗(:,1) ∥∥∥2 2
≤ C ∥∥∥β̂`HJ − β̂`H(x)∥∥∥2
2 .
The bounds (D.134), (D.135) also apply to ∥∥β`HJ − β`H(x)∥∥2 up to a constant factor, with the same probability up to a e−cn term which we can absorb into the existing tail by demanding n ≥ KL for some K.
Lemma D.22. For any `+ 1 < m ≤ j + 1, k ∈ [n], if n ≥ KL for some K then P [ 1EδK ∥∥∥Hj+1(k,:)Γj:mHJ ∥∥∥ 2 > C ] ≤ e−c nL
and if m = `+ 1
P [ 1EδK ∥∥∥Hj+1(k,:)Γj:`+1HJ ∥∥∥ 2 > C √ 1 n t ] ≤ e−c nL + e−c ′t
assuming t ≥ K ′n`−1 for some K ′.
Proof. If j + 1 > m,
1EδK ∥∥∥Hj+1(1,:)Γj:mHJ ∥∥∥ 2
=1EδK ∥∥∥W̃ j+1(1,:)PSj⊥Γj:m+1HJ PJmHm∥∥∥ 2
d =1EδK ∥∥∥W̃ j+1(1,:)PSj⊥Γj:m+1HJ PJmW̃mPSm−1⊥∥∥∥ 2 ≤1EδK ∥∥∥W̃ j+1(1,:)PSj⊥Γj:m+1HJ PJmW̃m∥∥∥ 2
d =1EδK ∥∥∥W̃ j+1(1,:)PSj⊥Γj:m+1HJ PJm∥∥∥ 2 ∥∥∥W̃m(1,:)∥∥∥ 2
≤1EδK ∥∥∥PSj⊥Γj:m+1HJ W̃ j+1∗(1,:) ∥∥∥
2 ∥∥∥W̃m(1,:)∥∥∥ 2
d = ∥∥∥PSj⊥Γj:m+1HJ ê1∥∥∥
2 ∥∥∥W̃ j+1∗(1,:) ∥∥∥ 2 ∥∥∥W̃m(1,:)∥∥∥ 2 .
If on the other hand j + 1 = m, we have 1EδK ∥∥∥Hj+1(1,:)Γj:mHJ ∥∥∥
2 = 1EδK ∥∥∥Hj+1(1,:)∥∥∥ 2 ≤ ∥∥∥W̃ j+1(1,:)∥∥∥ 2 .
Bernstein’s inequality gives P [∥∥∥W̃ j+1∗(1,:) ∥∥∥
2 > C
] ≤ C ′e−cn and P [∥∥∥W̃ `+1(1,:)∥∥∥ 2 > C √ 1 n t ] ≤ 2e−ct
for t ≥ 1, while lemma D.14 gives P [∥∥∥1EδKΓj:m+1HJ ê1∥∥∥
2 > C
] ≤ C ′e−c nL .
Taking union bounds gives the desired results.
Lemma D.23 (Generalized backward features inner product concentration). Fix x,x′ ∈ Sn0−1, ν = ∠(x,x′). Define a collection of support sets J , generalized backward features β`J , a constant δs and event EδK as in lemma D.14. Assuming n ≥ max {KL log n,K ′}, d ≥ K ′′ log n for suitably chosen K,K ′,K ′′, we have
P ∃` ∈ [L] : ∣∣∣∣1EδK 〈β`J ,β`J ′〉− n2 L−1∏ `′=` ( 1− ϕ (`′)(ν) π )∣∣∣∣ > C ( d2 √ Ln+ √ dδsLn+ d3/2δs ( 1 + δs√
n
) L5/2
)  ≤ C ′e−cd
for absolute constants c, C,C ′. If additionally we have P [EδK ] ≥ 1 − e−c ′d then the same result holds without the truncation on 1EδK , with worse constants.
Proof. Note that∣∣∣∣∣1EδK 〈β`J ,β`J ′〉− n2 L−1∏ `′=` ( 1− ϕ (`′)(ν) π )∣∣∣∣∣ ≤ ∣∣1EδK 〈β`J − β`(x),β`J ′〉∣∣+ ∣∣1EδK 〈β`(x),β`J ′ − β`(x′)〉∣∣
+ ∣∣∣∣∣1EδK 〈β`(x),β`(x′)〉− n2 L−1∏ `′=` ( 1− ϕ (`′)(ν) π )∣∣∣∣∣ ≤ 1EδK ∥∥β`J ′(x′)∥∥2 ∥∥β`J − β`(x)∥∥2 + 1EδK ∥∥β`(x)∥∥2 ∥∥β`J ′ − β`(x′)∥∥2
+ ∣∣∣∣1EδK 〈β`(x),β`(x′)〉− n2 L−1∏ `′=` ( 1− ϕ (`′)(ν) π )∣∣∣∣ . (D.136)
In order to bound the first two terms, we use rotational invariance of the Gaussian distribution twice to obtain
1EδK ∥∥β`J ′∥∥22 = 1EδK ∥∥∥WL+1ΓL:`+1J (x′)PJ′`(x′)∥∥∥22
≤ a.s.
1EδK ∥∥WL+1ΓL:`+1J ′ ∥∥22 d= 1EδK ∥∥ΓL:`+1J ′ WL+1∥∥22 d= ∥∥WL+1∥∥22 1EδK ∥∥ΓL:`+1J ′ ê1∥∥22 .
Bernstein’s inequality gives P [∥∥WL+1∥∥2
2 > Cn ] ≤ 2e−cn , while 1EδK ∥∥ΓL:`+1J ′ ê1∥∥22 can be bounded using D.14 to give
P [ 1EδK ∥∥β`J ′∥∥22 > Cn] ≤ C ′e−cn + C ′′e−c′ nL ≤ C ′′′e−c′′ nL for appropriate constants. Using lemma D.21 to bound 1EδK
∥∥β`J − β`(x)∥∥2 we obtain P [ 1EδK ∥∥β`J ′(x′)∥∥2 ∥∥β`J − β`(x)∥∥2 > C√dLn+ C ′ √ dδsLn+ d3/2δs ( 1 + δs√ n ) L5/2
] ≤C ′′e−c nL + C ′′′e−c ′d ≤ C ′′′′e−c ′′d
for some constants, assuming n ≥ KLd for some K. Bounding the second term in (D.136) in an identical fashion and the last term in (D.136) using Lemma D.4 we obtain
P  ∣∣∣∣1EδK 〈β`J ,β`J ′〉− n2 L−1∏ `′=` ( 1− ϕ (`′)(ν) π )∣∣∣∣ > C ( d2 √ Ln+ √ dδsLn+ d3/2δs ( 1 + δs√
n
) L5/2
)
≤ P  ∣∣∣∣1EδK 〈β`J ,β`J ′〉− n2 L−1∏ `′=` ( 1− ϕ (`′)(ν) π )∣∣∣∣ > C ′ (√ dLn+ √ dδsLn+ d3/2δs ( 1 + δs√
n
) L5/2 ) + C ′d2 √ Ln

≤ P
 ∣∣1EδK ∥∥β`J ′∥∥2 ∥∥β`J − β`(x)∥∥2 + 1EδK ∥∥β`J ∥∥2 ∥∥β`J ′ − β`(x′)∥∥2∣∣
> C ′ (√ dLn+ √ dδsLn+ d3/2δs ( 1 + δs√
n
) L5/2 )  +P [∣∣∣∣1EδK 〈β`J ,β`J ′〉− n2 L−1∏
`′=`
( 1− ϕ
(`′)(ν) π )∣∣∣∣ > C ′d2√Ln] ≤ C ′′e−cd + C ′′′e−c ′d ≤ C ′′′′e−c ′′d.
for appropriate constants assuming d ≥ 1. Taking a union bound over all possible choices of ` ∈ [L] and using d ≥ K logL for some K gives the desired result. If we additionally have P [EδK ] ≥ 1− e−c
′′′d for some c′′′, we can write∣∣∣∣∣〈β`J ,β`J ′〉− n2 L−1∏ `′=` ( 1− ϕ (`′)(ν) π )∣∣∣∣∣ = ∣∣∣∣∣1EδK 〈β`J ,β`J ′〉− n2 L−1∏ `′=` ( 1− ϕ (`′)(ν) π )∣∣∣∣∣ + ∣∣(1− 1EδK ) 〈β`J ,β`J ′〉∣∣
and since the last term is zero w.p. ≥ 1− e−c′′′d we obtain the same result as in the truncated case, with possibly worse constants.

heading: D.4 AUXILIARY RESULTS

text:
Lemma D.24. There are absolute constants c1, C, C ′ > 0 and absolute constants K,K ′ > 0 such that for any L ∈ N, if n ≥ max{K log4 n,K ′L}, then for every ` ∈ [L] one has∣∣∣E[ϕ(L−`)(ν̂`)− ϕ(L−`+1)(ν̂`−1) ∣∣∣ F`−1]∣∣∣ ≤ C log n
n
ν̂`−1
1 + (c0/64)(L− `)ν̂`−1 (1 + logL) +
C ′ n2 .
The constant c1 is the absolute constant appearing in Lemma E.1.
Proof. The case of ` = L follows immediately from Lemma E.1 with an appropriate choice of d ≥ K ′′ for K ′′ > 0 some absolute constant. Henceforth we assume ` ∈ [L− 1]. We Taylor expand (with Lagrange remainder) the smooth function ϕ(L−`) about the point ϕ(ν̂`−1), obtaining for any t ∈ [0, π]
ϕ(L−`)(t) = ϕ(L−`+1)(ν̂`−1) + ϕ̇(L−`)(ϕ(ν̂`−1)) ( t− ϕ(ν̂`−1) ) + ϕ̈(L−`)(ξ)
2
( t− ϕ(ν̂`−1) )2 ,
where ξ is some point of [0, π] lying in between t and ϕ(ν̂`−1). In particular, putting t = ν̂`, we obtain
ϕ(L−`)(ν̂`)−ϕ(L−`+1)(ν̂`−1) = ϕ̇(L−`)(ϕ(ν̂`−1)) ( ν̂` − ϕ(ν̂`−1) ) + ϕ̈(L−`)(ξ)
2
( ν̂` − ϕ(ν̂`−1) )2 ,
where ξ is some point of [0, π] lying in between ν̂` and ϕ(ν̂`−1). By (C.23) and (C.26), we have that ϕ̈(L−`) ≤ 0, whence
ϕ(L−`)(ν̂`)− ϕ(L−`+1)(ν̂`−1) ≤ ϕ̇(L−`)(ϕ(ν̂`−1)) ( ν̂` − ϕ(ν̂`−1) ) . (D.137)
Using Lemma E.5 and an induction, we have that ϕ̇(L−`) is decreasing, and moreover by the concavity property we have ϕ(ν̂`−1) ≥ ν̂`−1/2. An application of Lemmas E.1 and C.15 then yields
E [ ϕ(L−`)(ν̂`)− ϕ(L−`+1)(ν̂`−1) ∣∣∣ F`−1] ≤ (Cν̂`−1 log n n + C ′n−c1d )
1
1 + (c0/4)(L− `)ν̂`−1
≤ C log n n
ν̂`−1
1 + (c0/4)(L− `)ν̂`−1 + C ′n−c1d,
as long as d ≥ K and n ≥ K ′d4 log4 n. In particular, we can choose d = max{K, 2/c1} to obtained the claimed error for the upper bound. Next, for the lower bound, we make use of the estimate
ϕ̈(L−`)(ν) ≥ − C 1 + (c0/8)(L− `)ν
( 1 +
1
(c0/8)ν log (1 + (c0/8)(L− `− 1)ν) ) ︸ ︷︷ ︸
f(ν)
,
which follows from Lemma C.16 and ϕ̈(L−`) ≤ 0; by that lemma, we have that f is increasing. By Lemma E.3, as long as n ≥ K ′ log4 n, there is an event E on which |ν̂` − ϕ(ν̂`−1)| ≤ Cν̂`−1 √ log n/n + C ′n−3 and which satisfies P[E | F`−1] ≥ 1 − C ′′n−3. In particular, on the event E we have ν̂` ≥ ν̂`−1/4 − C ′/n3 provided n ≥ 16C2 log n, and so on the event E we have ξ ≥ min{ϕ(ν̂`−1), ν̂`−1/4− C ′/n3} ≥ ν̂`−1/4− C ′/n3. We can thus write ϕ(L−`)(ν̂`)− ϕ(L−`+1)(ν̂`−1)
≥ ϕ̇(L−`)(ϕ(ν̂`−1)) ( ν̂` − ϕ(ν̂`−1) ) + f(ξ)
2
( ν̂` − ϕ(ν̂`−1) )2 = ϕ̇(L−`)(ϕ(ν̂`−1)) ( ν̂` − ϕ(ν̂`−1) ) + (1E + 1Ec) f(ξ)
2
( ν̂` − ϕ(ν̂`−1) )2 ≥ ϕ̇(L−`)(ϕ(ν̂`−1)) ( ν̂` − ϕ(ν̂`−1) ) + 1E f( ν̂ `−1 4 − C4 n3 )
2
( ν̂` − ϕ(ν̂`−1) )2 − (2C ′′′π2L)1Ec ≥ ϕ̇(L−`)(ϕ(ν̂`−1)) ( ν̂` − ϕ(ν̂`−1) ) + f( ν̂ `−1 4 − C4 n3 )
2
( ν̂` − ϕ(ν̂`−1) )2 − (2C ′′′π2L)1Ec where the inequality in the third line follows from boundedness of the angles and the magnitude estimate on f in Lemma C.16, together with our estimate on ξ on E , and the inequality in the final line is a consequence of f ≤ 0, which allows us to drop the indicator for E and obtain a lower bound. Taking conditional expectations using the previous lower bound and applying F`−1-measurability of ν̂`−1 and boundedness of the angles together with our conditional measure bound on Ec, we obtain
E [ ϕ(L−`)(ν̂`)− ϕ(L−`+1)(ν̂`−1) ∣∣∣ F`−1] ≥ −C log n n
ν̂`−1
1 + (c0/4)(L− `)ν̂`−1 − C
′
n2 − C5L n3
+ f( ν̂
`−1 4 − C4 n3 ) 2 E [( ν̂` − ϕ(ν̂`−1) )2 ∣∣∣ F`−1], where we also apply the complementary bound obtained by our previous work following (D.137). Since the CL estimate in Lemma C.16 applies also to f , and since f ≤ 0, an application of Lemma E.4 with an appropriate choice of d and the choice n ≥ K ′ log4 n then yields (with a larger absolute constant C ′)
E [ ϕ(L−`)(ν̂`)− ϕ(L−`+1)(ν̂`−1) ∣∣∣ F`−1] ≥ −C log n n
ν̂`−1
1 + (c0/4)(L− `)ν̂`−1 − C
′
n2 − C6L n3
+ C7 log n
n
( ν̂`−1 )2 f
( ν̂`−1
4 − C4 n3
) .
If we choose n ≥ (C6/C ′)L, we can simplify this last estimate to E [ ϕ(L−`)(ν̂`)− ϕ(L−`+1)(ν̂`−1) ∣∣∣ F`−1] ≥− C log n n
ν̂`−1
1 + (c0/4)(L− `)ν̂`−1 − 2C
′
n2
+ C7 log n
n
( ν̂`−1 )2 f
( ν̂`−1
4 − C4 n3
) .
To conclude, we divide our analysis into two cases: when ν̂`−1 ≥ 8C4/n3, we have ν̂`−1/4 − C4n
−3 ≥ ν̂`−1/8, and so( ν̂`−1 )2 f ( ν̂`−1
4 − C4 n3
) ≥ ( ν̂`−1 )2 f ( ν̂`−1
8 ) = 64 ( ν̂`−1
8
)2 f ( ν̂`−1
8 ) ≥ − 8Cπν̂ `−1
1 + (c0/64)(L− `)ν̂`−1
( 1 +
8 log(L− `) c0π
) ,
where the last inequality follows from Lemma C.16. On the other hand, when ν̂`−1 ≤ 8C4/n3, the CL estimate in Lemma C.16 implies(
ν̂`−1 )2 f
( ν̂`−1
4 − C4 n3
) ≥ −64CC 2 4L
n6 ≥ −64CC
2 4L
n3 ≥ −2C
′
n2 ,
where the last estimate holds when n ≥ (32CC24/C ′)L. Adding these two estimates together, we obtain one that is valid regardless of the value of ν̂`−1, and choosing n ≥ C7 log n to combine the residuals, we obtain (after worst-case adjusting the constants)
E [ ϕ(L−`)(ν̂`)− ϕ(L−`+1)(ν̂`−1) ∣∣∣ F`−1] ≥ −4C ′ n2 − C8 log n n ν̂`−1 (1 + log(L− `)) 1 + (c0/64)(L− `)ν̂`−1 .
Combining with our previous work, we obtain∣∣∣E[ϕ(L−`)(ν̂`)− ϕ(L−`+1)(ν̂`−1) ∣∣∣ F`−1]∣∣∣ ≤ C log n n
ν̂`−1 (1 + logL)
1 + (c0/64)(L− `)ν̂`−1 + C ′
1
n2
after worst-casing constants.
Lemma D.25. There are absolute constants c1, C, C ′, C ′′, C ′′′ > 0 and absolute constantsK,K ′ > 0 such that for any d ≥ K, if n ≥ K ′d4 log4 n, then for every L ∈ N and every ` ∈ [L] one has
P [∣∣∣ϕ(L−`)(ν̂`)− ϕ(L−`+1)(ν̂`−1)∣∣∣ ≤√d log n n
2Cν̂`−1
1 + (c0/8)(L− `)ν̂`−1 + 2C ′n−c1d/2 ∣∣∣∣∣ F`−1 ]
≥ 1− C ′′′n−c1d, and
E [( ϕ(L−`)(ν̂`)− ϕ(L−`+1)(ν̂`−1) )2 ∣∣∣∣ F`−1] ≤ 4C2 d log nn (
ν̂`−1
1 + (c0/8)(L− `)ν̂`−1 )2 + C ′′n−c1d/2.
The constant c1 is the absolute constant appearing in Lemma E.1.
Proof. We will fix the meaning of the absolute constants C,C ′, C ′′ > 0 throughout the proof below. By Lemma E.3, we have if d ≥ K and n ≥ K ′d4 log4 n that for every ` ∈ [L]
P [∣∣ν̂` − ϕ(ν̂`−1)∣∣ ≤ Cν̂`−1√d log n n + C ′n−c1d ∣∣∣∣∣ F`−1 ] ≥ 1− C ′′n−c1d. (D.138)
By Lemma C.15, we have the estimate∣∣∣ϕ̇(`)(t)∣∣∣ ≤ 1 1 + (c0/2)`t ,
valid for any ` ∈ N0. Writing Ξ` = ν̂` − ϕ(ν̂`−1) so that ν̂` = ϕ(ν̂`−1) + Ξ`, we have that (Ξ`) is adapted to (F`), and by the fundamental theorem of calculus∣∣∣ϕ(L−`)(ν̂`)− ϕ(L−`+1)(ν̂`−1)∣∣∣ = ∣∣∣∣∣ ∫ ϕ(ν̂`−1) ϕ(ν̂`−1)+Ξ` dt 1 + (c0/2)(L− `)t ∣∣∣∣∣. The integrand is nonnegative, so by Jensen’s inequality we have(
ϕ(L−`)(ν̂`)− ϕ(L−`+1)(ν̂`−1) )2 ≤ |Ξ`| ∫ ϕ(ν̂`−1) ϕ(ν̂`−1)+Ξ`
dt
(1 + (c0/2)(L− `)t)2 ,
and an integration then yields( ϕ(L−`)(ν̂`)− ϕ(L−`+1)(ν̂`−1) )2 ≤
( Ξ` )2
|1 + (c0/2)(L− `)ϕ(ν̂`−1)||1 + (c0/2)(L− `)(ϕ(ν̂`−1) + Ξ`)| .
(D.139)
Choosing d ≥ 1/c1, we can guarantee that whenever ν ≥ C ′ C n −c1d/2, one has
Cν
√ d log n
n + C ′n−c1d ≤ 2Cν
√ d log n
n , (D.140)
and choosing n ≥ 64C2d log n, we can guarantee that ν
2 − 2Cν
√ d log n
n ≥ ν 4 . (D.141)
In particular, the last condition guarantees 2C √ d log n/n ≤ 1/4. By concavity ofϕ via Lemma E.5, we have ϕ(ν̂`−1) ≥ ν̂`−1/2, and using (D.138) to obtain
P [∣∣Ξ`∣∣ ≤ Cν̂`−1√d log n n + C ′n−c1d ∣∣∣∣∣ F`−1 ] ≥ 1− C ′′n−c1d,
we have by (D.140) and (D.141) as well as the concavity lower bound on ϕ P [ ϕ(ν̂`−1) + Ξ` ≥ ν̂`−1/4 ∣∣ F`−1] ≥ 1− C ′′n−c1d as long as ν̂`−1 ≥ (C ′/C)n−c1d/2. In particular, plugging these bounds into (D.139) and taking square roots, we obtain by a union bound
P [∣∣∣ϕ(L−`)(ν̂`)− ϕ(L−`+1)(ν̂`−1)∣∣∣ ≤ 2C√d log n n ∣∣∣∣ ν̂`−11 + (c0/8)(L− `)ν̂`−1 ∣∣∣∣ ∣∣∣∣∣ F`−1 ] ≥ 1− 2C ′′n−cd
whenever ν̂`−1 ≥ (C ′/C)n−c1d/2. Meanwhile, when ν̂`−1 ≤ (C ′/C)n−c1d/2, if we choose n ≥ d log n we have
Cν̂`−1 √ d log n
n + C ′n−c1d ≤ 2C ′n−c1d/2,
and we can use the 1-Lipschitz property of ϕ(L−`), which follows from Lemma E.5, to obtain using (D.138)
P [∣∣∣ϕ(L−`)(ν̂`)− ϕ(L−`+1)(ν̂`−1)∣∣∣ ≤ 2C ′n−c1d/2 ∣∣∣ F`−1]
≥ P [∣∣∣ϕ(L−`)(ν̂`)− ϕ(L−`+1)(ν̂`−1)∣∣∣ ≤ Cν̂`−1√d log n n + C ′n−c1d ∣∣∣∣∣ F`−1 ]
≥ P [∣∣ν̂` − ϕ(ν̂`−1)∣∣ ≤ Cν̂`−1√d log n n + C ′n−c1d ∣∣∣∣∣ F`−1 ]
≥ 1− C ′′n−c1d. Because |ϕ(L−`)(ν̂`)− ϕ(L−`+1)(ν̂`−1)| ≥ 0, we can then obtain using a union bound
P [∣∣∣ϕ(L−`)(ν̂`)− ϕ(L−`+1)(ν̂`−1)∣∣∣ ≤√d log n n ∣∣∣∣ 2Cν̂`−11 + (c0/8)(L− `)ν̂`−1 ∣∣∣∣+ 2C ′nc1d/2 ∣∣∣∣∣ F`−1 ]
≥ 1− 3C ′′n−c1d, which holds regardless of the value of ν̂`−1. We can then obtain the second bound using this one, via a partition of the expectation: let
E = {∣∣∣ϕ(L−`)(ν̂`)− ϕ(L−`+1)(ν̂`−1)∣∣∣ ≤√d log n n ∣∣∣∣ 2Cν̂`−11 + (c0/8)(L− `)ν̂`−1 ∣∣∣∣+ 2C ′n−c1d/2 } ,
so that E ∈ F`, and P[E | F`−1] ≥ 1− 3C ′′n−c1d by our work above. Then we have E [( ϕ(L−`)(ν̂`)− ϕ(L−`+1)(ν̂`−1) )2 ∣∣∣∣ F`−1] ≤ E [( ϕ(L−`)(ν̂`)− ϕ(L−`+1)(ν̂`−1) )2 1E
∣∣∣∣ F`−1]+ π2E[1Ec ∣∣ F`−1] ≤ E (2C√d log n n ∣∣∣∣ ν̂`−11 + (c0/8)(L− `)ν̂`−1 ∣∣∣∣+ 2C ′n−c1d/2 )2 ∣∣∣∣∣∣ F`−1 + C ′′′n−c1d
≤ ( 2C √ d log n
n ∣∣∣∣ ν̂`−11 + (c0/8)(L− `)ν̂`−1 ∣∣∣∣+ 2C ′n−c1d/2 )2 + C ′′′n−c1d
where the first inequality uses the triangle inequality to obtain ( ϕ(L−`)(ν̂`)− ϕ(L−`+1)(ν̂`−1) )2 ≤ π2; the second inequality applies the definition of E , uses nonnegativity to drop the indicator in the first term, and applies the conditional measure bound on Ec; and the final inequality integrates. Using the fact that our previous choices of large n imply 2C √ d log n/n ≤ 1/4, and that | ν̂ `−1
1+(c0/8)(L−`)ν̂`−1 | ≤ π, we can distribute the square in this final bound and worst-case constants to obtain
E [( ϕ(L−`)(ν̂`)− ϕ(L−`+1)(ν̂`−1) )2 ∣∣∣∣ F`−1] ≤ 4C2 d log n
n
( ν̂`−1
1 + (c0/8)(L− `)ν̂`−1
)2 + C ′′′′n−c1d/2,
as claimed.
Lemma D.26. Let X1, . . . , XL be independent chi-squared random variables, having respectively d1, . . . , dL degrees of freedom. Write dmin = mini∈L di and let ξi = 1diXi. Then there are absolute constants c, C > 0 and an absolute constant 0 < K ≤ 14 such that for any 0 < t ≤ K, one has
P [∣∣∣∣∣−1 + L∏ i=1 ξi ∣∣∣∣∣ > t ] ≤ CLe−cdmint 2/L.

heading: In particular, there are absolute constants C ′, C ′′ > 0 and an absolute constant K ′ > 0 such that

text:
for any d > 0, if dmin ≥ K ′dL then one has
P [∣∣∣∣∣−1 + L∏ i=1 ξi ∣∣∣∣∣ > C ′ √ dL dmin ] ≤ C ′′Le−d.
Proof. For any t ≥ 0, we have by the AM-GM inequality
P [ L∏ i=1 ξi > 1 + t ] = P ( L∏ i=1 ξi )1/L > (1 + t) 1/L  ≤ P[ 1 L L∑ i=1 ξi > (1 + t) 1/L ] .
By convexity of the exponential, we have (1 + t)1/L ≥ 1 + 1L log(1 + t), and by concavity of the logarithm we have log(1 + t) ≥ t log 2 if t ≤ 1. This implies
P [ L∏ i=1 ξi > 1 + t ] ≤ P [ −L+ L∑ i=1 ξi > Kt, ] ,
where K = log(2). Decomposing each Xi into a sum of di i.i.d. squared gaussians and applying Lemma G.2, we obtain
P [∣∣∣∣∣−L+ L∑ i=1 ξi ∣∣∣∣∣ > t ] ≤ 2 exp ( −cmin { t2∑L i=1 C2
di
, t
C maxi 1 di }) ≤ 2 exp ( −c′dmin min { t2/CL, t
}) ≤ 2 exp ( −c′′ dmint 2
L
) ,
(D.142)
where the last inequality holds provided t ≤ CL, where C > 0 is an absolute constant. Thus, as long as t ≤ CL/K, we have suitable control of the upper tail of the product ∏ i ξi. For the lower tail, writing log(0) = −∞, we have for any 0 ≤ t < 1
P [ L∏ i=1 ξi < 1− t ] = P [ L∑ i=1 log ξi < log(1− t) ] ≤ P [ L∑ i=1 log ξi < −t ] ,
where the inequality uses concavity of t 7→ log(1 − t). By Lemma G.2, we have for each i ∈ [L] and every 0 ≤ t ≤ C (where C > 0 is an absolute constant)
P[|ξi − 1| < t] ≤ 2e−cdit 2 ,
so that by a union bound and for t ≤ C √ L, we have with probability at least 1 − 2Le−cdmint2/L that 1 − t/ √ L ≤ ξi ≤ 1 + t/ √ L for every i ∈ [L]. Meanwhile, Taylor expansion of the smooth function x 7→ log x in a neighborhood of 1 gives
log x = (x− 1)− 1 2k2 (x− 1)2,
where k is a number lying between 1 and x. In particular, if x ≥ 12 we have log x ≥ (x−1)−2(x− 1)2, whence for t ≤ min{C √ L, 12}
P [ L∏ i=1 ξi < 1− t ] ≤ 2Le−cdmint 2/L + P [ −L+ L∑ i=1 ξi < −t+ 2t2 ]
≤ 2Le−cdmint 2/L + P [ −L+
L∑ i=1 ξi < − t 2
] ,
where the final inequality requires in addition t ≤ 14 . An application of (D.142) then yields the claimed lower tail provided t ≤ CL, which establishes the first claim. For the second claim, we consider the choice t = √ dL/cdmin, for which we have t ≤ K whenever dmin ≥ dL/cK2, and cdmint 2/L = d.
Lemma D.27. Let X1, . . . , XL be independent Binom(n, 12 ) random variables, and write ξi = 2 nXi. Then for any 0 < t ≤ 1 4 , one has
P [∣∣∣∣∣−1 + L∏ i=1 ξi ∣∣∣∣∣ > t ] ≤ 4Le−nt 2/8L.
In particular, for any d > 0, if n ≥ 128dL then one has
P [∣∣∣∣∣−1 + L∏ i=1 ξi ∣∣∣∣∣ > 4 √ dL n ] ≤ 4Le−d.
Proof. The proof is very similar to that of Lemma D.26. For any t ≥ 0, we have by the AM-GM inequality
P [ L∏ i=1 ξi > 1 + t ] = P ( L∏ i=1 ξi )1/L > (1 + t) 1/L  ≤ P[ 1 L L∑ i=1 ξi > (1 + t) 1/L ] .
By convexity of the exponential, we have (1 + t)1/L ≥ 1 + 1L log(1 + t), and by concavity of the logarithm we have log(1 + t) ≥ t log 2 if t ≤ 1. This implies
P [ L∏ i=1 ξi > 1 + t ] ≤ P [ −L+ L∑ i=1 ξi > Kt, ] ,
where K = log(2). Decomposing each Xi into a sum of n i.i.d. Bern( 12 ) random variables and applying Lemma G.1 twice, we obtain
P [∣∣∣∣∣−L+ L∑ i=1 ξi ∣∣∣∣∣ > t ] ≤ 2e−nt 2/2L. (D.143)
This gives suitable control of the upper tail of the product ∏ i ξi. For the lower tail, writing log(0) = −∞, we have for any 0 ≤ t < 1
P [ L∏ i=1 ξi < 1− t ] = P [ L∑ i=1 log ξi < log(1− t) ] ≤ P [ L∑ i=1 log ξi < −t ] ,
where the inequality uses concavity of t 7→ log(1− t). By Lemma G.1, we have for each i ∈ [L]
P[|ξi − 1| < t] ≤ 2e−nt 2/2,
so that by a union bound, we have that 1−t/ √ L ≤ ξi ≤ 1+t/ √ L for every i ∈ [L] with probability at least 1 − 2Le−nt2/2L. Meanwhile, Taylor expansion of the smooth function x 7→ log x in a neighborhood of 1 gives
log x = (x− 1)− 1 2k2 (x− 1)2,
where k is a number lying between 1 and x. In particular, if x ≥ 12 we have log x ≥ (x−1)−2(x− 1)2, whence for t ≤ 1/2
P [ L∏ i=1 ξi < 1− t ] ≤ 2Le−nt 2/2L + P [ −L+ L∑ i=1 ξi < −t+ 2t2 ]
≤ 2Le−nt 2/2L + P [ −L+
L∑ i=1 ξi < − t 2
] ,
where the final inequality requires in addition t ≤ 1/4. An application of (D.143) then yields the claimed lower tail, which establishes the first claim. For the second claim, we consider the choice t = √ 8dL/n, for which we have t ≤ 14 whenever n ≥ 128dL, and nt 2/8L = d.
Lemma D.28. For 1 ≤ `′ < ` ≤ L− 1 define events
Ẽ`:` ′
B = {∥∥∥B`−1:`′xx′ ∥∥∥2 F ≤ C2n(`− `′) } ∩ {∥∥∥B`−1:`′xx′ ∥∥∥ ≤ C(`− `′)} ∩ { tr [ B`−1:` ′
xx′
] ≤ Cn } and
E`:` ′ B = {∥∥α`−1(x)∥∥
2 ∥∥α`−1(x′)∥∥ 2 > 0 } ∩ ∣∣∣ϕ(`−1)(ν)− ν`−1∣∣∣ ≤ C √ d3 log3 n n` ∣∣∣∣
∩ Ẽ`:` ′
B .
If n,L satisfy the assumptions of corollary D.17 then P [ Ẽ`:` ′
B
] ≥ 1− C ′n(`− `′)2e−c n `−`′ .
If n,L additionally satisfy the conditions of lemmas D.3, E.16 and n ≥ C ′′L log(n) (log(L) + d), then P [ E`:` ′
B
] ≥ 1− C ′n−cd.
where c, C,C ′, C ′′ are absolute constants.
Proof. Since
tr [ B`−1:` ′
xx′
] = n∑ i=1 e∗iΓ `−1:`′+2(x)PI`′+1(x)PI`′+1(x′)Γ `−1:`′+2∗(x′)ei,
applying corollary D.17 2n times gives
P  ⋂ z∈{x,x′},i∈[n] {∥∥∥Γ`−1:`′+2(z)ei∥∥∥ 2 ≤ √ C } ≥ 1− C ′n(`− `′)2e−c n`−`′
⇒ P [ tr [ B`−1:` ′
xx′
] ≤ Cn ] ≥ 1− C ′n(`− `′)2e−c n `−`′ .
With the same probability we also have
max z∈{x,x′} ∥∥∥Γ`−1:`′+2(z)∥∥∥2 F = max z∈{x,x′} tr [ Γ`−1:` ′+2∗(z)Γ`−1:` ′+2(z) ] ≤ Cn
and
P [
max z∈{x,x′} ∥∥∥Γ`−1:`′+2(z)∥∥∥ ≤√C(`− `′)] ≥ 1− C ′′(`− `′)3e−c n`−`′ from which it follows that
P [∥∥∥B`−1:`′xx′ ∥∥∥ ≤ C(`− `′)] ≥ 1− C ′′(`− `′)3e−c n`−`′
and P [∥∥∥B`−1:`′xx′ ∥∥∥2
F ≤ C2n(`− `′) ] ≥ P [ max
z∈{x,x′} ∥∥∥Γ`−1:`′+2(z)∥∥∥2 max z∈{x,x′} ∥∥∥Γ`−1:`′+2(x)∥∥∥2 F ≤ C2n(`− `′) ] ≥ 1− C ′′(`− `′)3e−c n `−`′ − C ′n(`− `′)2e−c n `−`′
≥ 1− C ′′′n(`− `′)2e−c n `−`′ .
It follows that Ẽ`:`′B holds with the same probability. From lemma E.16,
P [∥∥α`−1(x)∥∥
2 ∥∥α`−1(x′)∥∥ 2 > 0 ∩ ν`−1 = ν̂`−1 ] ≥ 1− C ′`e−cn
for some constants c, C ′. Here ν̂`−1 is the auxiliary angle process defined in (D.2). Using D.3, we obtain
P ∣∣∣ϕ(`−1)(ν)− ν`−1∣∣∣ ≤ C √ d3 log3 n
n`
 ≥ P ∣∣∣ϕ(`−1)(ν)− ν`−1∣∣∣ ≤ C √ d3 log3 n
n` ∣∣∣∣∣E +P [Ec]
≥ 1− C ′′′`e−cn − C ′′n−cd ≥ 1− C ′n−cd
for an appropriate choice of c, C ′.
We conclude that P [ E`:` ′
B
] ≥ 1− C ′e−c ′n − C ′′n`2e−c ′′ n `−`′ − C ′′′n−c ′′′d
≥ 1− C̃n−c̃d
for appropriately chosen constants, where we used n ≥ C ′′′′` log(n) (log(`) + d).
Lemma D.29. For ∆` defined in (D.34) and E`:` ′ B defined in lemma D.28 we have P [∣∣∣1E`:`′B ∆`∣∣∣ > C√d`∣∣∣F`−1] ≤a.s. C ′e−cd.
for some constants c, C,C ′.
Proof. 1E`:`′B ∆` =1E`:`′B (∆` − E∆`|F`−1)
=1E`:`′B L−1∏ i=` ( 1− ϕ (i)(ν) π )( tr [ B`:` ′ xx′ ] − E W ` tr [ B`:` ′ xx′ ]) ,
and denoting P `xx′ = PI`(x)PI`(x′) we have
tr [ B`:` ′
xx′ ] − E W ` tr [ B`:` ′ xx′ ] = tr [ B`−1:` ′ xx′ ( W `∗P `xx′W ` − E W ` [ W `∗P `xx′W ` ])]
Defining S` = span{α`(x),α`(x′)} we decomposeW ` into a sum of two independent terms as
W ` = W `PS`−1 +W `PS`−1⊥ ≡ G` +H`.
Note that eachH` is independent of every other random variable in the problem conditioned on the features. 1E`:`′B tr
[ B`:` ′
xx′
] thus decomposes into a sum of four terms, which we proceed to consider
individually and show that they concentrate.
The allG` term is
1E`:`′B tr [ B`−1:` ′ xx′ G `∗P `xx′G ` ] = 1E`:`′B dimS`−1∑ i,j=1 u`−1∗j B `−1:`′ xx′ u `−1 i u `−1∗ i W `∗P `xx′W `u`−1j
where {u`−1i } is an orthonormal basis of S`−1. If α`−1(x) 6= α`−1(x′) we choose
(u`−11 ,u `−1 2 ) = (
α`−1(x)
‖α`−1(x)‖2 , Pα`−1(x)⊥α `−1(x′)∥∥Pα`−1(x)⊥α`−1(x′)∥∥2 ), (which are well-defined on E`:`′B ).Using rotational invariance of the Gaussian distribution, we have
u∗iW `−1∗P `−1+1xx′ W `−1uj d = u`−1∗i R ∗W `−1∗PW `−1Rα`−1(x′)>0PW `−1Rα`−1(x)>0W `−1Ru`−1j
d = 〈 Pg1 cos ν`−1+g2 sin ν`−1>0gi,Pg1>0gj 〉 where gi ∼iid N (0, 2nI). If α `−1(x) = α`−1(x′) then dimS`−1 = 1 and we simply choose u`−11 = α`−1(x) ‖α`−1(x)‖2
and end up with an identical expression, with ν`−1 = 0. Since Pg1>0gj and Pg1 cos ν`−1+g2 sin ν`−1>0gi are vectors of independent sub-Gaussian random variables with sub-
Gaussian norm bounded by √
C n , their inner product is a sum of independent sub-exponential vari-
ables with sub-exponential norm bounded by Cn for some constant C. Momentarily abbreviating ṽ = g1 cos ν `−1 + g2 sin ν `−1, Bernstein’s inequality then gives
P [∣∣∣∣〈Pṽ>0gi,Pg1>0gj〉 − Eg1,g2 〈Pṽ>0gi,Pg1>0gj〉 ∣∣∣∣ > √ d n ∣∣∣∣∣F`−1 ] ≤ a.s. 2e−cd (D.144)
for some constant c. Since on E`:`′B , ∥∥∥B`−1:`′xx′ ∥∥∥ ≤ C`, we obtain∣∣∣1E`:`′B tr [B`−1:`′xx′ G`∗P `xx′G`]∣∣∣ ≤ C`dimS `−1∑ i,j=1 u`−1∗i W `∗P `xx′W `u`−1j
almost surely and thus
P [∣∣∣1E`:`′B tr [B`−1:`′xx′ G`∗P `xx′G`]∣∣∣ > C ′` √ d
n ∣∣∣∣∣F`−1 ] ≤ a.s. 2e−cd
for some C ′, and hence
P ∣∣∣∣∣∣ 1E`:`′B tr [ B`−1:` ′ xx′ G `∗P `xx′G ` ]
− E W ` 1E`:`′B tr [ B`−1:` ′ xx′ G `∗P `xx′G ` ] ∣∣∣∣∣∣ ≤ 2C ′√`d ∣∣∣∣∣∣F`−1  ≤ a.s. 2e−c n ` . (D.145)
tr [ B`:` ′
xx′
] also contains the terms
tr [ B`−1:` ′
xx′ G `∗P `xx′H
` ] + tr [ B`−1:` ′
xx′ H `∗P `xx′G
` ] .
Considering the first of these (since the second can be treated in an identical fashion), we recall that H` is independent of all the other random variables in the problem conditioned on the features, and we thus have
1E`:`′B tr [ B`−1:` ′ xx′ G `∗P `xx′H ` ] d =1E`:`′B tr [ B`−1:` ′ xx′ G `∗P `xx′W̃ `PS`−1⊥ ]
= dimS`−1∑ i=1 1E`:`′B v`∗i W̃ `w`i
where W̃ ` is an independent copy of W `, v`i = P ` xx′u ` i ,w ` i = PS`−1⊥B `−1:`′ xx′ u ` i . Hence conditioned on all the other variables, v`∗i W̃ `w`i is a zero-mean Gaussian with variance 2‖v`i‖22‖w`i‖ 2 2
n 1E`:`′B . Again from the bound on ∥∥∥B`−1:`′xx′ ∥∥∥ implied by E`:`′B , we have 2 ∥∥v`i∥∥22 ∥∥w`i∥∥22
n 1E`:`′B
≤ C` 2
n
almost surely. Noting that E W `
tr [ B`−1:` ′
xx′ G `∗P `xx′H
` ]
= E G` Ẽ W ` tr [ dimS`−1∑ i=1 v`∗i W̃ `w`i ] = 0, a
Gaussian tail bound gives P [∣∣∣∣1E`:`′B tr [B`−1:`′xx′ G`∗P `xx′H`]− EW `1E`:`′B tr [B`−1:`′xx′ G`∗P `xx′H`] ∣∣∣∣ > √`∣∣∣∣F`−1] ≤ a.s. 2e−c n ` . (D.146) The final term in tr [ B`:` ′
xx′
] is
tr [ B`−1:` ′
xx′ H `∗P `xx′H
` ] d = tr [ B`−1:` ′
xx′ PS`−1⊥W̃ `∗P `xx′W̃ `PS`−1⊥ ] .
Due to the independence of W̃ ` from the remaining random variables, this is simply a Gaussian chaos in n2 variables. The Hanson-Wright inequality (lemma G.4) gives
P [∣∣∣∣1E`:`′B tr [B`−1:`′xx′ H`∗P `xx′H`]− EW `1E`:`′B tr [B`−1:`′xx′ H`∗P `xx′H`] ∣∣∣∣ ≥ t∣∣∣∣F`−1]
≤ a.s. 2 exp −cntmin  t∥∥∥1E`:`′B PS`−1⊥B`−1:`′xx′ PS`−1⊥∥∥∥2F , 1∥∥∥1E`:`′B PS`−1⊥B`−1:`′xx′ PS`−1⊥∥∥∥
≤ a.s. 2 exp
( −cn
` tmin
{ t
n , 1 }) where in the last inequality we used the definition of E`:`′B . It follows that
P [∣∣∣∣1E`:`′B tr [B`−1:`′xx′ H`∗P `xx′H`]− EW `1E`:`′B tr [B`−1:`′xx′ H`∗P `xx′H`] ∣∣∣∣ > 2√d`∣∣∣∣F`−1]
≤ a.s.
P [∣∣∣∣1E`:`′B tr [B`−1:`′xx′ H`∗P `xx′H`]− EH`1E`:`′B tr [B`−1:`′xx′ H`∗P `xx′H`] ∣∣∣∣ > √d`∣∣∣∣F`−1] +P [∣∣∣∣ E H` 1E`:`′B tr [ B`−1:` ′ xx′ H `∗P `xx′H ` ] − E W ` 1E`:`′B tr [ B`−1:` ′ xx′ H `∗P `xx′H ` ]∣∣∣∣ > √d`∣∣∣∣F`−1]
≤ a.s.
P [∣∣∣∣1E`:`′B tr [B`−1:`′xx′ H`∗P `xx′H`]− EH`1E`:`′B tr [B`−1:`′xx′ H`∗P `xx′H`] ∣∣∣∣ > √d`∣∣∣∣F`−1]
+P 1E`:`′B ∣∣∣∣∣∣ 2tr [ PS`−1⊥B `−1:`′ xx′ PS`−1⊥ ] n ∣∣∣∣∣∣ ∣∣∣∣tr [P `xx′]− E H` tr [ P `xx′ ]∣∣∣∣ > √d` ∣∣∣∣∣∣F`−1  ≤ a.s. Ce−cd
(D.147) where in the last inequality we used (D.144) and the properties of E`:`′B . Collecting terms and using (D.145), (D.146), (D.147) we obtain
P [ 1E`:`′B ∣∣∣∣tr [B`:`′xx′]− E W ` tr [ B`:` ′ xx′ ]∣∣∣∣ > C√d`∣∣∣∣F`−1] ≤ a.s. C ′e−cd (D.148)
and hence P [∣∣∣1E`:`′B ∆`∣∣∣ > C√d`∣∣∣F`−1]
= P [ 1E`:`′B L−1∏ i=` ( 1− ϕ (i)(ν) π ) ∣∣∣∣tr [B`:`′xx′]− E W ` tr [ B`:` ′ xx′ ]∣∣∣∣ > C√d` ∣∣∣∣∣F`−1 ] ≤ a.s. C ′e−cd.
(D.149)
Lemma D.30. For x ∈ Sn0−1 and ` ∈ [L], denote I`(x) = supp(α`(x) > 0). If n ≥ K then
P [ min ` |I`(x)| ≥ n 4 ] ≥ 1− 2LCe−cn
and for any 0 ≤ t ≤ 1
P [ L∏ `=1 2 |I`(x)| n − 1 ≥ t ] ≤ 2 exp ( −c n L t2 )
where c, c′, C,K are absolute constants.
Proof. Consider the activations at layer `. From lemma E.16, if n ≥ K we have P [∥∥α`−1(x)∥∥ 2 > 0 ] ≥ 1− Ce−cn.
Rotational invariance of the Gaussian distribution gives α`(x) = [ W `α`−1(x) ] +
d =∥∥α`−1(x)∥∥
2
[ W `(:,1) ] + . It follows that
E [ |I`(x)| ∣∣∣∣∣ ∥∥α`−1(x)∥∥2 > 0 ] = E [ n∑ i=1 1α`i(x)>0 ∣∣∣∣∣ ∥∥α`−1(x)∥∥2 > 0 ]
= E [ n∑ i=1 1W ` (:,1) >0 ∣∣∣∣∣ ∥∥α`−1(x)∥∥2 > 0 ] .
From the symmetry of the Gaussian distribution
E [ |I`(x)| ∣∣∣∣∣ ∥∥α`−1(x)∥∥2 > 0 ] = n 2 .
Since this variable is a sum of n independent variables taking values in {0, 1}, an application of Bernstein’s inequality for bounded random variables (lemma G.3) gives
P [∣∣∣|I`(x)| − n
2 ∣∣∣ > n 4 ] ≤ P [∣∣∣|I`(x)| − n 2 ∣∣∣ > n 4 ∣∣∣∣∣ ∥∥α`−1(x)∥∥2 > 0 ] + P [∥∥α`−1(x)∥∥ 2 = 0 ]
≤ 2 exp ( −c′ n 2/16
n+ n/4
) + Ce−cn ≤ C ′e−c ′′n
for appropriate constants. A union bound gives
P [ L⋂ `=1 {∣∣∣|I`(x)| − n 2 ∣∣∣ > n 4 }] ≤ 2LC ′e−c ′′n
from which
P [ min ` |I`(x)| ≥ n 4 ] ≥ 1− 2LC ′e−c ′′n
follows.
To prove the second inequality, we use the AM-GM inequality which gives
( L∏ `=1 2 |I`(x)| n )1/L ≤ 2 L∑̀ =1 |I`(x)| nL
and hence
P [ L∏ `=1 2 |I`(x)| n ≥ 1 + t ] = P ( L∏ `=1 2 |I`(x)| n )1/L ≥ (1 + t)1/L  ≤ P
[ L∑ `=1 2 |I`(x)| n ≥ L (1 + t)1/L ]
Convexity of the exponential gives (1+t)1/L ≥ 1+ 1L log(1+t) and for t ≤ 1 we have log(1+t) ≥ t log 2, giving
P [ L∏ `=1 2 |I`(x)| n ≥ 1 + t ] ≤ P [ L∑ `=1 2 |I`(x)| n − L ≥ t log 2 ] We note that
2 |I`(x)| n d = n∑ i=1 1E`b ` i
where b`i = 2 nθ ` i , θ ` i ∼iid Bern( 12 ) and E
` = {
max i b`−1i 6= 0
} is the event that the features at layer
`− 1 are not identically 0. Since n∑ i=1 1E`b ` i ≤ n∑ i=1 b`i a.s. we have
P [ L∑ `=1 2 |I`(x)| n − L ≥ t log 2 ] ≤ P [ L∑ `=1 n∑ i=1 b`i − L ≥ t log 2 ] .
Since this is a sum of independent bounded random variables, an application of Bernstein’s inequality for bounded random variables (lemma G.3) gives
P [ L∑ `=1 n∑ i=1 b`i − L ≥ t ] ≤ 2 exp ( −c t 2∑ E(b`i)2 + 2 n t ) = 2 exp ( −c′ n L t2 )
for some absolute constant c′, where we used L ≥ 1 ≥ t. Hence
P [ L∏ `=1 2 |I`(x)| n − 1 ≥ t ] ≤ 2 exp ( −c′ n L t2 ) .

heading: E SHARP BOUNDS ON THE ONE-STEP ANGLE PROCESS

text: In this section, we characterize the process by which angles between features for different pairs of points evolve as they are propagated across one layer of the zero-time network. This section is self-contained, and as such it will occasionally overload notation used elsewhere in the document for different local purposes. In particular, we will use the notation σ(x) = [x]+ for the ReLU in this section (and only in this section), and σ̇(g) = 1g>0 for its weak derivative.

heading: E.1 DEFINITIONS AND PRELIMINARIES

text:
Let n ∈ N, with n ≥ 2. Let g1 and g2 be i.i.d. N (0, (2/n)I) random vectors; we use µ to denote the joint law of these random variables. We writeG ∈ Rn×2 for the matrix with first column g1 and second column g2, and g1, . . . , gn for the n rows ofG. If S ⊂ [n] is nonempty andA ∈ Rn×m, we writeAS ∈ R|S|×m to denote the submatrix ofA consisting of the rows indexed by S in increasing index order. In such situations Sc will always denote the complement relative to [n].
For 0 ≤ ν ≤ 2π, define random variables
vν(g1, g2) = σ (g1 cos ν + g2 sin ν) ,
and v̇ν(g1, g2) = σ̇ (g1 cos ν + g2 sin ν) (g2 cos ν − g1 sin ν) .
Because v̇ν separates over coordinates of its arguments and has each of its coordinates the product of a nondecreasing function and a continuous function, it is Borel measurable. A key property that we will use throughout this section is that the joint distribution of (g1, g2) is rotationally invariant; in particular, it is invariant to rotations of the type
G 7→ G [ cos ν sin ν sin ν − cos ν ] ,
where ν ∈ [0, 2π]. Since we can write
vν = σ
( G [ cos ν sin ν ]) , v̇ν = σ̇ ( G [ cos ν sin ν ]) ( G [ − sin ν cos ν ]) ,
where all of the R2 vectors appearing above are elements of S1, it follows by applying rotational invariance and the specific rotation given above that
(vν , v̇ν) d = (v0,−v̇0).
This equivalence is useful for evaluating expectations and differentiating with respect to ν.
If 0 < c ≤ 0.5 and m ∈ N0 with m < n, define an event Ec,m = ⋂ S⊂[n] |S|=m ⋂ ν∈[0,2π] { (g1, g2) ∣∣ c ≤ ‖IScvν(g1, g2)‖2 ≤ c−1}. For each c,m, the set Ec,m is closed, since ‖Avν‖ is a continuous function of (g1, g2) ∈ Em for any linear mapA. We further define
E0,m = ⋃ k∈N E1/(2k),m,
so that E0,m = ⋂ S⊂[n] |S|=m ⋂ ν∈[0,2π] {(g1, g2) | 0 < ‖IScvν(g1, g2)‖2},
and E0,m is Borel measurable. If c is omitted, we take the constant c in the definition to be 0.5. On Ec,m we guarantee that ‖vν‖0 ≥ m uniformly on [0, π]. Define a function Xν by
Xν = 1E1 〈 v0 ‖v0‖2 , vν ‖vν‖2 〉 .
On E1, we guarantee that vν 6= 0 for every ν, soXν is well defined; because E1 is Borel measurable, we have thatXν is Borel measurable, and moreover |Xν | ≤ 1, soXν ∈ Lpµ for every p ≥ 1. Finally, define for 0 ≤ ν ≤ π
ϕ̄(ν) = E g1,g2
[ cos-1Xν ] , ϕ(ν) = cos-1 E
g1,g2 [〈v0,vν〉].

heading: E.2 MAIN RESULTS

text:
Lemma E.1. There exist absolute constants c, C,C ′ > 0 and absolute constants K,K ′ > 0 such that if d ≥ K and n ≥ K ′d4 log4 n, then one has
|ϕ̄(ν)− ϕ(ν)| ≤ Cν log n n + C ′n−cd
Proof. Using the triangle inequality, we can write |ϕ̄(ν)− ϕ(ν)| ≤ ∣∣ϕ̄(ν)− cos-1 E[Xν ]∣∣+ ∣∣cos-1 E[Xν ]− ϕ(ν)∣∣.
Choose n sufficiently large to satisfy the hypotheses of Lemmas E.6 and E.7; applying these lemmas to bound the first and second terms, we conclude the claimed result (after choosing n larger than an absolute constant multiple of d log n so that the n−cd error dominates the e−c ′n error).
Lemma E.2. One has ϕ(ν) = cos-1 (( 1− ν
π
) cos ν + sin ν
π
) .
Proof. See (Cho & Saul, 2009).
Lemma E.3. There exist absolute constants c, C,C ′, C ′′ > 0 and absolute constants K,K ′ > 0 such that if d ≥ K and n ≥ K ′d4 log4 n, then one has with probability at least 1− C ′′n−cd∣∣cos-1Xν − ϕ(ν)∣∣ ≤ Cν√d log n
n + C ′n−cd.
The constant c is the same as the constant appearing in Lemma E.1.
Proof. Under our hypothess, the second result in Lemma E.6 together with Lemma E.1 and the triangle inequality imply the claimed result (after worst-casing multiplicative constants).
Lemma E.4. There exist absolute constants c, C,C ′ > 0 and absolute constants K,K ′ > 0 such that if d ≥ K and n ≥ K ′d4 log4 n, then one has
E [( cos-1Xν − ϕ(ν) )2] ≤ Cν2 d log n
n + C ′n−cd.
The constant c is the same as the constant appearing in Lemma E.1.
Proof. Under our hypotheses, Lemma E.3 is applicable; we let E denote the event corresponding to the bound in this lemma. By boundedness of cos-1, nonnegativity of Xν , and ϕ ≤ π/2 from Lemma E.2, we have ‖cos-1Xν − ϕ(ν)‖L∞ ≤ π. Thus
E [( cos-1Xν − ϕ(ν) )2] ≤ E[1E (cos-1Xν − ϕ(ν))2]+ C ′′π2n−cd ≤ ( Cν √ d log n
n + C ′n−cd
)2 + C ′′π2n−cd
≤ C2ν2 d log n n + C ′′′n−cd,
as claimed.
Lemma E.5. One has
1. ϕ ∈ C∞(0, π), and ϕ̇ and ϕ̈ extend to continuous functions on [0, π];
2. ϕ(0) = 0 and ϕ(π) = π/2; ϕ̇(0) = 1, ϕ̈(0) = −2/(3π), and ... ϕ(0) = −1/(3π2); and
ϕ̇(π) = ϕ̈(π) = 0;
3. ϕ is concave and strictly increasing on [0, π] (strictly concave in the interior);
4. ϕ̈ < −c < 0 for an absolute constant c > 0 on [0, π/2];
5. 0 < ϕ̇ < 1 and 0 > ϕ̈ ≥ −C on (0, π) for some absolute constant C > 0;
6. ν(1− C1ν) ≤ ϕ(ν) ≤ ν(1− c1ν) on [0, π] for some absolute constants C1, c1 > 0.
Proof. Deferred to Appendix E.4.

heading: E.3 SUPPORTING RESULTS

heading: E.3.1 CORE SUPPORTING RESULTS

text:
Lemma E.6. There exist constants c, C,C ′, C ′′, C ′′′, C ′′′′ > 0 and an absolute constant K > 0 such that for any d ≥ 1, if n and d satisfy the hypotheses of Lemmas E.9 and E.10 and moreover n ≥ Kd log n, then one has∣∣∣∣ Eg1,g2[cos-1Xν]− cos-1 Eg1,g2[Xν ]
∣∣∣∣ ≤ Cν log nn + C ′n−cd, and with probability at least 1− C ′′n−cd, one has
∣∣cos-1Xν − E[cos-1Xν]∣∣ ≤ C ′′′ν√d log n n + C ′′′′n−cd.
Proof. Fix ν ∈ [0, π]. The function cos-1 is smooth on (−δ, 1) if 0 < δ < 1, and Taylor expansion with Lagrange remainder on this domain about the point E[Xν ] (by Lemma E.23, we have 0 ≤ E[Xν ] < 1 if ν > 0; we will handle ν = 0 separately below) gives
cos-1(x) = cos-1 E[Xν ]− 1√
1− E[Xν ]2 (x− E[Xν ])−
ξ
2 (1− ξ2)3/2 (x− E[Xν ])2 ,
where ξ lies between x and E[Xν ]. Using the fact that Xν 6= 1 almost surely if ν > 0, which is established in Lemma E.23, we plug in x = Xν to get
cos-1 E[Xν ]− cos-1(Xν) = 1√
1− E[Xν ]2 (x− E[Xν ]) +
ξ(Xν)
2 (1− ξ(Xν)2)3/2 (Xν − E[Xν ])2 ,
(E.1) where we now express ξ as a function of Xν . From Jensen’s inequality it is clear
E[cos-1Xν ] ≤ cos-1 E[Xν ], (E.2)
so all that remains is to obtain a matching upper bound for the righthand side of (E.1). We will make use of the following facts, proved in subsequent sections: there are absolute constants Ci > 0, i ∈ [6], and ci > 0, i ∈ [5], such that
1. E[Xν ] ≤ 1− c5ν2 + C1e−c1n. (Lemma E.8)
2. For each ν, Var[Xν ] ≤ C5ν4 log n/n+ C2e−c2n. ( Lemma E.9)
3. With probability at least 1 − C3n−c3d, one has |Xν − E[Xν ]| ≤ C6ν2 √ d log n/n +
C4e −c4n. (Lemma E.10)
Let E denote the event on which property 3 holds. Combining properties 1 and 3, we obtain with probability at least 1− C3n−c3d
Xν ≤ 1− (c5/2)ν2 + C1e−c1n + C4e−c4n,
provided n is chosen larger than an absolute constant multiple of d log n. Thus, defining
ν0 = 4
c5
( C1e −c1n + C4e −c4n ) ,
we obtain for ν ≥ ν0 E[Xν ] ≤ 1−
c5 4 ν2, Xν ≤ 1− c5 4 ν2, (E.3)
with the second bound holding with probability at least 1−C3n−c3d. Considering first 0 ≤ ν ≤ ν0, we obtain using the triangle inequality, Lemma E.20 and property 3∣∣cos-1 E[Xν ]− E[cos-1(Xν)]∣∣ ≤ E[1E ∣∣cos-1 E[Xν ]− cos-1(Xν)∣∣]
+ E [ 1Ec ∣∣cos-1 E[Xν ]− cos-1(Xν)∣∣]
≤ E [ 1E √ |Xν − E[Xν ]| ] + E[1Ecπ/2]
≤ Ce−cn + C ′n−c ′d, (E.4)
with the final inequality following from the triangle inequality for the `2 norm and the fact that ν ≤ ν0. Meanwhile, if ν ≥ ν0, we have by (E.3)
0 ≤ ξ(Xν) ≤ max{Xν ,E[Xν ]} ≤ 1− c5 4 ν2
with probability at least 1−C3n−c3d. Using 1− x2 = (1 + x)(1− x) and E[Xν ] ≥ 0, ξ(Xν) ≥ 0, we have under this condition on ν
1√ 1− E[Xν ]2 ≤ 1√ 1− E[Xν ] ≤ 2 c5ν
(E.5)
and similarly ξ(Xν)
2 (1− ξ(Xν)2)3/2 ≤ 4 c35ν 3 1E + π 2 1Ec . (E.6)
Applying (E.6) and taking expectations in (E.1), we obtain by property 2
cos-1 E[Xν ]− E[cos-1Xν ] ≤ Cν log n
n + C ′e−cn + C ′′n−c3d. (E.7)
Together, (E.2), (E.4) and (E.7) establish the first claim provided n is chosen larger than an absolute constant multiple of d log n.
For the second claim, we begin by using the triangle inequality to write∣∣cos-1Xν − E[cos-1Xν]∣∣ ≤ ∣∣cos-1Xν − cos-1 E[Xν ]∣∣+ ∣∣cos-1 E[Xν ]− E[cos-1Xν]∣∣, and then observe that our proof of the first claim implies suitable control of the second term. For the first term, if ν ≤ ν0 we use Lemma E.20 to immediately obtain with probability at least 1−C3n−c3d that this term is at most Ce−cn. For ν ≥ ν0, we apply property 3 and the bounds (E.5) and (E.6) in the expression (E.1) to obtain with probability at least 1− C3n−c3d∣∣cos-1Xν − cos-1 E[Xν ]∣∣ ≤ Cν√d log n
n + C ′ν
d log n
n ,
which is of the claimed order when n is chosen larger than an absolute constant multiple of d log n.
Lemma E.7. There exist absolute constants c, C,C ′, C ′′ > 0 such that if n ≥ C log n, one has∣∣∣∣ϕ(ν)− cos-1 Eg1,g2[Xν ] ∣∣∣∣ ≤ C ′e−cn + C ′′ νn.
Proof. Write f(ν) = cosϕ(ν), and
h(ν) = E[Xν ]− f(ν),
so that h is the residual between the two terms whose images we are trying to tie together. We will make use of the following results:
1. The function cos-1 is 12 -Hölder continuous on [0, 1], so that |cos -1 x− cos-1 y| ≤
√ |x− y|
if x, y ≥ 0. (Lemma E.20)
2. For ν ∈ [0, π], we have 1− 12ν 2 ≤ f(ν) ≤ 1− c2ν2. (Lemma E.14)
3. For all 0 ≤ ν ≤ π, |h(ν)| ≤ C1e−c1n + C2ν2/n. (Lemma E.15)
We choose n large enough that the hypotheses of Lemma E.15 are satisfied. Define ν0 = 2 √ C1/c2e
−c1n/2. We split the analysis into two sub-intervals: I1 := [0, ν0], and I2 := [ν0, π]. Choosing n larger than an absolute constant multiple of log n, we guarantee that I1 and I2 both have positive measure.
On I1, we proceed as follows:∣∣cos-1 f − cos-1(f + h)∣∣ ≤√|h| ≤ √ C1e−c1n + C2ν2/n
≤ √ C1e−c1n + 4C1C2c −1 2 e −c1n
≤ Ce− 1 2 c1n.
The first inequality uses Hölder continuity of cos-1, the second uses our bound on the residual, the third uses the definition of I1, and the fourth worst-cases the constants.
On I2, we calculate
|f + h| ≤ |f |+ |h| ≤ C1e−c1n + C2 ν2
n + 1− c2ν2,
using the triangle inequality and our bounds on |h| and f . Using the conditions ν ≥ ν0 and choosing n ≥ 4C2/c2, we can rearrange to get
C1e −c1n + C2
ν2
n ≤ c2ν
2
2 ,
which implies |f + h| ≤ 1− c2ν2/2. By the control f(ν) ≤ 1− c2ν2, valid on I2, we get that both f and f + h are bounded above by 1− c2ν2/2 on I2; moreover, because f ≥ 0 and f + h ≥ 0, we can apply local Lipschitz properties of cos-1 on I2. This yields∣∣cos-1 f − cos-1(f + h)∣∣ ≤ |h|√
1− ( supI2 max{f, f + h} )2 ≤ C1e −c1n + C2ν 2/n√
1− (1− (c2/2)ν2)2
= C1e −c1n√ 1 2c2ν 2(2− 12c2ν2) + C2ν 2/n√ 1 2c2ν 2(2− 12c2ν2)
≤ Cν−1e−c1n + C ′ν/n
≤ Ce− 1 2 c1n + C ′ν/n.
Above, the first inequality is the instantiation of the local Lipschitz property; the second applies our upper and lower bounds on f and f +h derived above, and our bound on the residual |h|; the fourth applies the bound 0 ≤ f(ν) ≤ 1 − 12c2ν 2 to conclude 2 − 12c2ν 2 ≥ 1 on I2, and cancels a factor
of ν in the second term; and in the last line, we apply ν ∈ I2 to get ν ≥ 2 √ C1/c2e
−c1n/2, which allows us to cancel the ν−1 factor in the first term of the previous line.
To wrap up, we can choose the largest of the constants appearing in the bounds derived for I1 and I2 above and then conclude, since I1 ∪ I2 = [0, π] under our condition on n.

heading: E.3.2 PROVING LEMMA E.6

text:
Lemma E.8. There exist absolute constants c, c′, C, C ′, C ′′ > 0 such that if n ≥ C and if n is sufficiently large to satisfy the hypotheses of Lemma E.15, one has
1− C ′′ν2 − C ′e−c ′n ≤ E g1,g2 [Xν ] ≤ 1− cν2 + C ′e−c ′n.
Proof. By the triangle inequality, we have
|cosϕ(ν)| − |E[Xν ]− cosϕ(ν)| ≤ E[Xν ] ≤ |cosϕ(ν)|+ |E[Xν ]− cosϕ(ν)|. Applying Lemmas E.14 and E.15 with m = 0, we get
1− C ′′ν2 − Ce−c ′n − C ′ν2/n ≤ E[Xν ] ≤ 1− cν2 + Ce−c ′n + C ′ν2/n,
which proves the claim if we choose n ≥ 2C ′/c.
Lemma E.9. There exist absolute constants c, C,C ′ > 0 such that if n satisfies the hypotheses of Lemmas E.11 and E.12, then one has for each ν ∈ [0, π]
Var[Xν ] ≤ Cν4 log n
n + C ′e−cn.
Proof. We use the following elementary fact for a random variable with finite first and second moments, easily proved using Var[Xν ] = E[X2ν ] − E[Xν ]2 and Fubini’s theorem: in this setting one has
Var[Xν ] = E g1 [Var[Xν(g1, · )]] + Var[E g2 [Xν( · , g2)]].
By Lemma E.11, there is an event E of probability at least 1− Ce−cn on which Var[Xν(g1, · )] ≤ C ′ν4/n+ C ′′e−c ′n. Invoking as well Lemma E.12, we obtain
Var[Xν ] ≤ E g1
[(1E + 1Ec)Var[Xν(g1, · )]] + C ′′′ν4 log n
n + C ′′′′e−c
′′n
≤ Cν 4 log n
n + C ′e−cn + P[Ec]1/2 E
g1
[ Var[Xν(g1, · )]2 ]1/2 ≤ Cν 4 log n
n + C ′e−cn,
as claimed, where in the second line we applied nonnegativity of the variance and the Schwarz inequality, and in the third line we used the fact that ‖X‖L2 ≤ ‖X‖L∞ for any random variable X in L∞.
Lemma E.10. There exist absolute constants c, c′, C, C ′, C ′′ > 0 and absolute constantsK,K ′ > 0 such that for any d ≥ 1 such that n and d satisfy the hypotheses of Lemmas E.11 and E.13 and n ≥ max{K log n,K ′d}, for any ν ∈ [0, π], one has
P [∣∣∣∣Xν − Eg1,g2[Xν ] ∣∣∣∣ ≤ C ′′ν2 √ d log n n + Ce−cn ] ≥ 1− C ′n−c ′d.
Proof. By Lemma E.11, we have
P [∣∣∣∣Xν − Eg2[Xν ] ∣∣∣∣ ≤ C ′′ν2 √ d n + Ce−cn ] ≥ 1− C ′e−c ′d.
Let ψ = ψ0.25 denote the cutoff function defined in Lemma E.31, and write
Yν(g1, g2) =
〈 v0(g1, g2)
ψ (‖v0(g1, g2)‖2) ,
vν(g1, g2)
ψ (‖vν(g1, g2)‖2)
〉 .
By Lemma E.13, we have
P [∣∣∣∣Eg2[Yν ]− Eg1,g2[Yν ] ∣∣∣∣ ≤ C ′′ν2 √ d log n n + Cne−cn ] ≥ 1− C ′n−c ′d
We have Xν = Yν on the event E1, by Lemma E.16, and we thus calculate using the triangle inequality ∣∣∣∣ Eg1,g2[Yν ]− Eg1,g2[Xν ] ∣∣∣∣ ≤ Eg1,g2[|Xν − Yν |] = Eg1,g2[1Ec1Yν] ≤ Cne−cn,
where the last inequality uses Hölder’s inequality and the measure bound in Lemma E.16. Again using the triangle inequality, we have∣∣∣∣Eg2[Xν ]− Eg2[Yν ] ∣∣∣∣ ≤ Eg2[|Xν − Yν |], and so using our previous calculation and Markov’s inequality, we can assert
P [∣∣∣∣Eg2[Yν ]− Eg2[Xν ] ∣∣∣∣ ≤ Cne−cn/2] ≥ 1− e−cn/2. The claim then follows from the triangle inequality, a union bound, and a choice of n larger than an absolute constant multiple of log n and an absolute constant multiple of d.
Lemma E.11. There exist absolute constants c, c′, c′′, c′′′, C, C ′, C ′′, C ′′′, C4, C5 > 0, and absolute constants K,K ′ > 0 such that for any d ≥ 1, if n ≥ max{Kd,K ′ log n}, then for every ν ∈ [0, π] one has with probability at least 1− Ce−cn
Var[Xν(g1, · )] ≤ C4ν
4
n + C ′e−c ′n,
and with (g1, g2) probability at least 1− C ′′e−c ′′d one has∣∣∣∣Xν − Eg2[Xν ] ∣∣∣∣ ≤ C5ν2 √ d n + C ′′′e−c ′′′n.
Proof. Fix ν ∈ [0, π]. Let E1 = E0.5,1 denote the event in Lemma E.16 which is in the definition of Xν . We start by treating the case of ν = 0 or ν = π. We have Xπ = 0 deterministically, so the variance is zero and it equals its partial expectation over g2 with probability one. For the other case, one has X0 = 1E1 ; we have
Var[X0(g1, · )] = E g2 [1E1 ]− E g2 [1E1 ] 2 ≤
( 1− E
g2 [1E1 ]
) ,
and since E[1E1 ] = 1− Cne−cn by Lemma E.16, we obtain by Markov’s inequality P [ Var[X0(g1, · )] ≥ Cne−cn/2 ] ≤ e−cn/2.
This gives a suitable bound on the variance with suitable probability. For deviations, we note that E [ X0 − E
g2 [X0]
] = 0,
and following our previous variance inequality but taking expectations over both g1 and g2 gives Var[X0] ≤ Cne−cn, which implies by Chebyshev’s inequality
P [∣∣∣∣X0 − Eg2[X0] ∣∣∣∣ ≥√Cne−cn/2] ≤ e−cn/2 which is a suitable deviations bound that we can union bound with the event constructed below, which controls deviations uniformly for the remaining values of ν. We therefore assume below that 0 < ν < π.
Let ψ(x) = max{x, 18}, which is continuous and differentiable except at x = 1 8 , with derivative ψ′(x) = 1x>1/8. We note in addition that x ≤ ψ(x), and since ψ ≥ 18 we have for x ≥ 0 the bound x/ψ(x) ≤ 1. Define
Yν(g1, g2) = 〈v0(g1, g2),vν(g1, g2)〉
ψ(‖v0(g1, g2)‖2)ψ(‖vν(g1, g2)‖2) .
We first show that it is enough to prove the claims for Yν , which will be preferable for technical reasons. On E1, we have Yν = Xν . We have |Yν | ≤ 1, and we calculate
E g2
[ (Yν −Xν)2 ] = E g2 [ 1Ec1(Yν −Xν) 2 ] ≤ E g2 [ 1Ec1 ]1/2 E g2 [ (Yν −Xν)4 ]1/2 ≤ C E g2 [ 1Ec1 ]1/2 ,
where the first inequality uses the Schwarz inequality, and the last inequality uses that |Xν | ≤ 1 and the triangle inequality, and where C > 0 is an absolute constant. We have by Tonelli’s theorem and Lemma E.16
E g1 [ E g2 [ 1Ec1 ]1/2] ≤ Cne−cn,
so Markov’s inequality implies P [ E g2 [ 1Ec1 ]1/2 ≥ Cne−cn/2] ≤ e−cn/2.
Thus, with probability at least 1− e−cn/2, we have
E g2
[ (Yν −Xν)2 ] ≤ C ′ne−cn/2,
so that an application of Lemma E.32 yields that with probability at least 1− e−cn/2
Var[Xν(g1, · )] ≤ Var[Yν(g1, · )] + C ′′ne−c ′n,
where we have worst-cased constants and the exponent on n. For deviations, we write using the triangle inequality∣∣∣∣Xν − Eg2[Xν ] ∣∣∣∣ ≤ |Xν − Yν |+ ∣∣∣∣Yν − Eg2[Yν ] ∣∣∣∣+ ∣∣∣∣Eg2[Yν ]− Eg2[Xν ] ∣∣∣∣, and then note that the first term is identically zero on the event E1, which has probability at least 1− Ce−cn, whereas for the third term, we have∣∣∣∣Eg2[Yν ]− Eg2[Xν ] ∣∣∣∣ ≤ Eg2 [ (Yν −Xν)2 ]1/2 ≤ C ′ne−cn/2,
where the first inequality uses the triangle inequality and the Lyapunov inequality, and the second inequality holds with probability at least 1− e−cn/2, and leverages the argument we used to control the difference in variances. Ultimately taking union bounds, we can conclude that it sufficient to prove the claimed properties for Yν .
With 0 < ν < π fixed, we introduce the notation
ug1 = v0(g1); vg1,g2 = vν(g1, g2),
so that
Yν =
〈 ug1
ψ (‖ug1‖2) , vg1,g2 ψ (‖vg1,g2‖2)
〉 .
For fixed g1, we will write Yν(g2) = Yν(g1, g2) with an abuse of notation. For ḡ ∈ Rn arbitrary and g2 fixed, we consider the function f(t) = Yν(g2 + tḡ) for t ∈ [0, 1]. Writing f ′ for the derivative of f where it exists, at any point of differentiability, we calculate by the chain rule
f ′(t) = 〈∇g2Yν(g2 + tḡ), ḡ〉, where
∇g2Yν(g2) = sin ν
ψ(‖ug1‖2)ψ(‖vg1,g2‖2)
( I −
ψ′(‖vg1,g2‖2)vg1,g2v∗g1,g2 ψ(‖vg1,g2‖2)‖vg1,g2‖2
)( 1vg1,g2>0 ug1 ) .
Using the fact that 1vg1,g2>0 ug1 = P{vg1,g2>0}ug1 , where P{vg1,g2>0} is the orthogonal projection onto the coordinates where vg1,g2 is positive, together with the fact that
vg1,g2v ∗ g1,g2P{vg1,g2>0} = P{vg1,g2>0}vg1,g2v ∗ g1,g2 ,
we can also write
∇g2Yν(g2) = sin ν
ψ(‖ug1‖2)ψ(‖vg1,g2‖2) P{vg1,g2>0}
( I −
ψ′(‖vg1,g2‖2)vg1,g2v∗g1,g2 ψ(‖vg1,g2‖2)‖vg1,g2‖2
) ug1 .
(E.8)
We next argue that f does not fail to be differentiable at too many points of [0, 1]. Because ψ > 0, it will suffice to show that (i) t 7→ vg1,g2+tḡ and (ii) t 7→ ψ(‖vg1,g2+tḡ‖2) are differentiable at all but a set of isolated points in [0, 1]. For the latter function, we note that at any point where ‖vg1,g2+tḡ‖2 < 1 8 , by continuity we have that t 7→ ψ(‖vg1,g2+tḡ‖2) is locally constant, and therefore differentiable at such points. At other points, by Lemma E.21 it suffices to characterize t 7→ ‖vg1,g2+tḡ‖2 as differentiable at all but isolated points, and because ‖vg1,g2+tḡ‖2 ≥ 18 by assumption, the norm is differentiable and by the chain rule it suffices to characterize differentiability of each coordinate of t 7→ vg1,g2+tḡ , which settles the question of all-but-isolated differentiability of (i) as well. We have vg1,g2+tḡ = σ(g1 cos ν + g2 sin ν + tḡ sin ν), so again by Lemma E.21, we conclude from differentiability of t 7→ g1 cos ν + g2 sin ν + tḡ sin ν that t 7→ vg1,g2+tḡ is differentiable at all but isolated points, and consequently so is f . In particular, f is differentiable at all but countably many points of [0, 1]. Next, we show that f ′ has suitable integrability properties. Indeed, we calculate using (E.8)
‖∇g2Yν(g2)‖2 ≤ 8ν ∥∥∥∥(I − ψ′(‖vg1,g2‖2)vg1,g2v∗g1,g2ψ(‖vg1,g2‖2)‖vg1,g2‖2 ) ug1 ψ(‖ug1‖2) ∥∥∥∥ 2
= 8ν √
1− ψ′(‖vg1,g2‖2)Yν(g2)2, (E.9)
where we used Cauchy-Schwarz and ψ ≥ 18 in the first inequality and distributed and applied (ψ′)2 = ψ′ and the estimate x/ψ(x) ≤ 1 in the second inequality. In particular, this implies that |f ′(t)| ≤ C‖ḡ‖2, which is a t-integrable upper bound for every ḡ. Because Yν(g1, · ) is continuous by continuity of σ, ψ, and the fact that ψ becomes constant whenever ‖vg1,g2‖2 < 18 , we can apply (Cohn, 2013, Theorem 6.3.11) to get
Yν(g2 + ḡ) = Yν(g2) + ∫ 1 0 〈∇g2Yν(g2 + tḡ), ḡ〉dt,
and since ḡ was arbitrary, for any g′2 ∈ Rn we can put ḡ = g′2 − g2 to get
Yν(g ′ 2) = Yν(g2) + ∫ 1 0 〈∇g2Yν(tg′2 + (1− t)g2), g′2 − g2〉dt.
Performing the expansion with g2 and g′2 reversed and applying the triangle inequality and CauchySchwarz then implies the estimate
|Yν(g′2)− Yν(g2)| ≤ ‖g′2 − g2‖2 ∫ 1
0
‖∇g2Yν(tg′2 + (1− t)g2)‖2 dt. (E.10)
This relation is enough to conclude the result for angles satisfying ν ≥ c0, where 0 < c0 ≤ π/4 is an absolute constant. Indeed, (E.9) and (E.10) imply that Yν is C-Lipschitz, where C > 0 is an absolute constant; so the Gaussian Poincaré inequality implies
E g2
[( Yν − E
g2 [Yν ]
)] ≤ C ′
n ,
and Gauss-Lipschitz concentration implies for any d ≥ 0
P [∣∣∣∣Yν − Eg2[Yν ] ∣∣∣∣ ≥ C ′′ √ d n ] ≤ 2e−d.
Because ν ≥ c0, we can adjust these bounds to involve ν4 and ν2 (respectively) while only paying increases in the constant factors. We proceed assuming 0 < ν ≤ c0. Let 0 ≤ τg1 ≤ 1 denote a median of Yν(g1, · ), i.e., a number satisfying Pg2 [Yν ≥ τg1 ] ≥ 12 and Pg2 [Yν ≤ τg1 ] ≥ 12 , and for each 0 ≤ s < τg1 define
ws(g2) = max{Yν(g2), τg1 − s}.
For any 0 ≤ s < τg1 , notice that ws ≥ Yν , which implies that P[ws ≥ τg1 ] ≥ P[Yν ≥ τg1 ] ≥ 12 , because τg1 is a median of Yν ; and similarly P[ws ≤ τg1 ] ≥ P[Yν ≤ τg1 ] ≥ 12 , so that τg1 is also a median of ws. The fact that ws ≥ Yν implies for any t > 0 that P[Yν − τg1 > t] ≤ P[ws− τg1 > t],
and additionally if Yν ≤ τg1−swe havews = τg1−s, so that P[Yν−τg1 ≤ −s] ≤ P[ws−τg1 ≤ −s]. In particular, the tails of Yν can be controlled in terms of those of ws for appropriate choices of s. Additionally, by Lemma E.21, we have that for each s, t 7→ ws(g2 + tḡ) is differentiable at all but countably many points of [0, 1], and has derivative there equal to t 7→ 〈ḡ,∇g2ws(g2)〉, where
∇g2ws(g2) = 1ws(g2)>τg1−s∇g2Yν(g2),
which, following from (E.9), satisfies a strengthened gradient norm estimate ‖∇g2ws(g2)‖2 ≤ 8ν1ws(g2)>τg1−s √ 1− ψ′(‖vg1,g2‖2)Yν(g2)2
≤ 8ν √ 1− ψ′(‖vg1,g2‖2)(τg1 − s)2. (E.11)
In particular, we obtain a nearly-Lipschitz estimate of the form (E.10): |ws(g′2)− ws(g2)| ≤ ‖g′2 − g2‖2 ∫ 1
0
8ν √
1− ψ′(‖vg1,tg′2+(1−t)g2)‖2)(τg1 − s) 2 dt. (E.12)
For each g1, we define a set Sg1 = {g2 | ‖vg1,g2‖2 ≥ 14}. Noting that the function g2 7→ ‖σ(g1 cos ν + g2 sin ν)‖2 is a convex 1-Lipschitz function (given that |sin ν| ≤ 1), we have by Gauss-Lipschitz concentration
P g2
[ ‖vg1,g2‖2 ≤ E
g2 [‖vg1,g2‖2]− t
] ≤ e−cnt 2 ,
and by Jensen’s inequality
E g2
[‖vg1,g2‖2] ≥ |cos ν|‖ug1‖2 ≥ ‖ug1‖2√
2 ,
where the last line holds because ν ≤ π/4. By Lemma E.16, there is a g1 event E having probability at least 1− Ce−cn on which ‖ug1‖2 ≥ 12 , so that for any g1 ∈ E , we have by a suitable choice of t in our Gauss-Lipschitz bound Pg2 [Sg1 ] ≥ 1−e−cn. Thus, using the first line of (E.11), the Gaussian Poincaré inequality and the Lipschitz property of ws (which follows from (E.12) after bounding by an absolute constant) and Rademacher’s theorem on a.e. differentiability of Lipschitz functions, we have whenever g1 ∈ E
Var[ws] ≤ 2
n E g2
[ ‖∇g2ws(g2)‖22 ] ≤ 128ν 2
n E g2
[( 1− ψ′(‖vg1,g2‖2)Yν(g2)2 )] ≤ 128ν 2
n E g2
[ 1E ( 1− ψ′(‖vg1,g2‖2)Yν(g2)2 )] + E g2 [ 1Ec ( 1− ψ′(‖vg1,g2‖2)Yν(g2)2
)] ≤ 256ν 2
n E g2
[1− Yν(g2)] + Ce−cn, (E.13)
where we also make use of the fact that 0 ≤ Yν ≤ 1. Now, we calculate for g1 ∈ E and g2 ∈ Sg1
Yν = 1− 1
2 ∥∥∥∥ ug1‖ug1‖2 − vg1,g2‖vg1,g2‖2 ∥∥∥∥2 2
≥ 1− 2‖ug1 − vg1,g2‖ 2 2 ‖ug1‖22 ≥ 1− 8‖ug1 − vg1,g2‖22 ≥ 1− 8‖g1 − (g1 cos ν + g2 sin ν)‖22, (E.14)
where the second inequality uses g1 ∈ E , the third uses nonexpansiveness of σ, and the first requires a proof; we will show that for any nonzero vectors x,y ∈ Rn, one has∥∥∥∥ x‖x‖2 − y‖y‖2 ∥∥∥∥ 2 ≤ 2‖x− y‖2 ‖y‖2 . (E.15)
To see this, write θ for the angle between x and y, and distribute to obtain equivalently
−1 2 ‖y‖22(1 + cos θ) ≤ ‖x‖22 − 2‖x‖2‖y‖2 cos θ.
Divide through by ‖x‖22, write K = ‖y‖2‖x‖−12 , and rearrange to obtain the equivalent expression
K2(1 + cos θ))− 4K cos θ + 2 ≥ 0.
It suffices to minimize the LHS of the previous inequality with respect to K subject to the constraint K > 0 and then study the resulting function of θ to ascertain the validity of the bound. Given that 1 + cos θ ≥ 0, the LHS is a convex function of K, with minimizer K = 2 cos θ(1 + cos θ)−1, and therefore for any θ ≥ π/2, the LHS subject to the constraint K > 0 is minimized at K = 0, where the inequality is easily seen to be true. If θ < π/2, we have that the minimizer is positive, and we verify that after substituting the bound becomes
1 + cos θ ≥ 2 cos2 θ,
which is also seen to be true for θ < π/2, for example by showing that the polynomial x 7→ −2x2 + x+ 1 is nonnegative on [0, 1]. This proves the inequality, so returning to (E.14), we have
Yν ≥ 1− 8((1− cos ν)2‖g1‖22 + sin2 ν‖g2‖22 − 2(sin ν)(1− cos ν)〈g1, g2〉) ≥ 1− 8((1− cos ν)2‖g1‖22 + sin2 ν‖g2‖22 − 2(sin ν)(1− cos ν)‖g1‖2‖g2‖2)
using Cauchy-Schwarz in the second inequality. By Gauss-Lipschitz concentration (e.g. following the proof of the third assertion in Lemma E.17), there is a g1 event E ′ and a g2 event E ′′, each with probability at least 1−Ce−cn, on which we have (respectively) ‖gi‖2 ≤ 2 for i = 1, 2. Then using (sin ν)(1− cos ν) ≥ 0, we obtain that when g1 ∈ E ∩ E ′ and when g2 ∈ Sg1 ∩ E ′′
Yν ≥ 1− 32((1− cos ν)2 + sin2 ν) = 1− 64(1− cos ν) ≥ 1− 32ν2,
where the final inequality uses the standard estimate cos ν ≥ 1 − 0.5ν2, which can be proved via Taylor expansion. By a union bound, we can assert that with g1-probability at least 1−Ce−cn, with conditional (in g2) probability at least 1 − C ′e−c
′n we have Yν ≥ 1 − 32ν2, so that in particular, by nonnegativity of Yν , and choosing n larger than an absolute constant, we guarantee with g1probability at least 1− Ce−cn
E g2
[Yν ] ≥ 1− 32ν2 − C ′e−cn, τg1 ≥ 1− 32ν2. (E.16)
Plugging the mean estimate into (E.13), we conclude with probability at least 1− C ′′e−c′n
Var[ws] ≤ Cν4
n + C ′e−cn. (E.17)
We could have just as well applied this exact argument to Yν instead of ws, so we conclude the claimed variance bound from this expression. We have stated the result in terms of the truncations ws so that it can be applied towards deviations control in the sequel. As an immediate application, we use the fact that any median is a minimizer of the quantity c 7→ E[|X − c|] for any integrable X and c ∈ R to get with probability at least 1− C ′′e−c′n∣∣∣∣Eg2[ws]− τg1 ∣∣∣∣ ≤ Eg2[|ws − τg1 |] ≤ Eg2 [∣∣∣∣ws − Eg2[ws]
∣∣∣∣] ≤√Var[ws] ≤ Cν2√n + C ′e−cn, (E.18) where we also applied Jensen’s inequality for the first inequality and the Lyapunov inequality for the third. In particular, the same argument yields∣∣∣∣Eg2[Yν ]− τg1
∣∣∣∣ ≤ Cν2√n + C ′e−cn. (E.19) We turn to removing the t dependence in (E.12) without sacrificing the dependence on τg1 . To obtain a Lipschitz estimate on the subset Sg1 we need to control the norm of ∇g2ws on the line segment between g2, g′2 ∈ Sg1 . For this, write σy(x) = max{x − y, 0} for any y ∈ R, and make the following observations:
1. vg1,g2 = (sin ν)σ−g1 cot ν(g2), so that
Yν(g2) =
〈 ug1
ψ (‖ug1‖2) ,
(sin ν)σ−g1 cot ν(g2)
ψ (‖(sin ν)σ−g1 cot ν(g2)‖2)
〉 ;
2. for any x, y, σy(x) = max{x, y} − y; x 7→ max{x,y} is the projection onto the convex set {x | xi ≥ yi ∀i}, so in particular x 7→ σy(x) is nonexpansive, has convex range, and satisfies σy(σy(x) + y) = σy(x); and thus
3. for any g2, Yν(g2) = Yν(σ−g1 cot ν(g2)− g1 cot ν).
We write g̃2 = σ−g1 cot ν(g2)− g1 cot ν, g̃′2 = σ−g1 cot ν(g′2)− g1 cot ν, so that (E.12) becomes |ws(g′2)− ws(g2)| = |ws(g̃′2)− ws(g̃2)|
≤ ‖g̃′2 − g̃2‖2 ∫ 1
0
8ν √
1− ψ′(‖vg1,tg̃′2+(1−t)g̃2)‖2)(τg1 − s) 2 dt
≤ ‖g′2 − g2‖2 ∫ 1
0
8ν √
1− ψ′(‖vg1,tg̃′2+(1−t)g̃2)‖2)(τg1 − s) 2 dt,
where the second line follows from nonexpansiveness and translation invariance of the distance. Having reduced to the study of points along the segment between g̃2 and g̃′2, we now observe
σ−g1 cot ν (tg̃ ′ 2 + (1− t)g̃2) = σ (tσ−g1 cot ν(g2) + (1− t)σ−g1 cot ν(g2))
= tσ−g1 cot ν(g2) + (1− t)σ−g1 cot ν(g2), because σ−g1 cot ν has image included in the nonnegative orthant, which is convex. It then follows from (1) above that
‖vg1,tg̃′2+(1−t)g̃2‖2 = (sin ν)‖tσ−g1 cot ν(g2) + (1− t)σ−g1 cot ν(g2)‖2 = ∥∥tvg1,g2 + (1− t)vg1,g′2∥∥2,
and in particular∥∥tvg1,g2 + (1− t)vg1,g′2∥∥22 = t2‖vg1,g2‖22 + 2t(1− t)〈vg1,g2 ,vg1,g′2〉+ (1− t)2‖vg1,g′2‖22 ≥ 1
16
( t2 + (1− t)2 ) ≥ 1
32 ,
where the first inequality uses that σ ≥ 0 and g2, g′2 ∈ Sg1 , and the second minimizes the convex function of t in the previous bound. We conclude that g2, g′2 ∈ Sg1 implies that ‖vg1,tg̃′2+(1−t)g̃2‖2 > 1 8 for every t ∈ [0, 1], and consequently (E.12) becomes (after an additional simplification of the quantity under the square root using τg1 ≤ 1)
|ws(g′2)− ws(g2)| ≤ 16ν √ 1− (τg1 − s)‖g′2 − g2‖2, (E.20)
so that ws is 16ν √
1− (τg1 − s)-Lipschitz on Sg1 . Then by an application of the median bound in (E.16), if 0 ≤ s < 1 − 32ν2, with g1 probability at least 1 − Ce−cn we have that ws is 16ν √
32ν2 + s-Lipschitz on Sg1 . For the previous assertion to be nonvacuous, we need to take ν small; in particular, we have 1− 32ν2 > 12 if ν < 1/8, which we can take to be the value of the absolute constant c0 we left unspecified previously. Then for each such s, define Ls = 16ν √ 32ν2 + s, and define ŵs(g2) = sup
g′2∈Sg1 {ws(g′2)− Ls‖g′2 − g2‖2}.
Then ŵs is Ls-Lipschitz on Rn, and satisfies ŵs = ws on Sg1 (Evans & Gariepy, 1991, §3.1.1 Theorem 1). By the Gaussian Poincaré inequality, we obtain immediately Var[ŵs] ≤ Ls, and using ŵs = ws on Sg1 , we compute∣∣∣∣Eg2[ws − ŵs] ∣∣∣∣ = ∣∣∣∣∣ Eg2∈Scg1 [ws − ŵs] ∣∣∣∣∣ ≤ Eg2 [ 1Scg1 |ws − ŵs|
] ≤ P g2 [ Scg1 ]1/2‖ws − ŵs‖L2 ≤ Ce−cn (‖ws‖L2 + ‖ŵs‖L2) ≤ C ′e−cn, (E.21)
where the second inequality follows from the Schwarz inequality, the third holds given that g1 ∈ E and by the Minkowski inequality, and the final uses that ws and ŵs are both Lipschitz with Lipschitz constants bounded above by absolute constants together with the Gaussian Poincaré inequality. Meanwhile, by Gauss-Lipschitz concentration, we obtain a Bernstein-type lower tail
P [ ŵs ≤ E
g2 [ŵs]− s
] ≤ exp ( − cns 2
ν2(32ν2 + s)
) , (E.22)
and for the upper tail, it will be sufficient to consider ŵ0, which satisfies a subgaussian tail (for any t ≥ 0) P [ ŵ0 ≤ E
g2 [ŵ0]− t
] ≤ exp ( −c ′nt2
ν4
) . (E.23)
Using the results (E.18), (E.19), (E.21), and the fact that ws = ŵs on Sg1 , we get
P g2
[ Yν − E
g2 [Yν ] ≤ −s ] ≤ P g2 [ ŵs − E g2 [ŵs] ≤ C ν2√ n + C ′e−cn − s ] + P g2 [ Scg1 ] . (E.24)
Using d ≥ 1, we put s = 2Cν2 √ d/n+C ′e−cn in this bound; using that ν < 1/8, and in particular 1− 32ν2 > 12 , we can choose n larger than an absolute constant multiple of d to guarantee that for all 0 ≤ ν < 1/8, this choice of s is less than 1− 32ν2, and that Cν2 √ d/n ≤ 32ν2. Together with the lower tail bound (E.22), these facts imply
P g2
[ Yν − E
g2 [Yν ] ≤ −2Cν2
√ d
n − C ′e−cn ] ≤ P g2 [ ŵs − E g2 [ŵs] ≤ −Cν2 √ d n ] + C ′′e−c ′n
≤ e−c ′′d + C ′′e−c ′n.
Meanwhile, for the upper tail, we have for any t ≥ 0
P g2
[ Yν − E
g2 [Yν ] ≥ t ] ≤ P g2 [ ŵ0 − E g2 [ŵ0] ≥ t− C ν2√ n − C ′e−cn ] + P g2 [ Scg1 ] , (E.25)
and if we put t = 2Cν2 √ d/n + C ′ecn, our previous requirements on n and the upper tail bound (E.23) yield
P g2
[ Yν − E
g2 [Yν ] ≥ 2Cν2
√ d
n + C ′e−cn
] ≤ e−c ′′′d + C ′′e−c ′n.
Combining these two bounds gives control of absolute deviations about the mean. By independence, we conclude
P g1,g2 [∣∣∣∣Yν − Eg2[Yν ] ∣∣∣∣ ≤ 2Cν2 √ d n + C ′e−c ′′′n ] ≥ (1− 2e−cd − Ce−c ′n)(1− C ′e−c ′′n)
≥ 1− 2e−cd − Ce−c ′n − C ′e−c ′′n.
To conclude, we have shown that for every ν ∈ [0, π] one has with probability at least 1− Ce−cn
Var[Xν(g1, · )] ≤ C ′ν4
n + C ′′ne−c ′n,
and with (g1, g2) probability at least 1− 2e−c ′′d + C ′′′ne−c ′′′n one has∣∣∣∣Xν − Eg2[Xν ] ∣∣∣∣ ≤ C ′′′′ν2 √ d n + C ′′′′′ne−c ′′′′n.
To simplify these bounds, we may in addition choose n larger than an absolute constant multiple of log n, and n larger than an absolute constant multiple of d, to obtain that with probability at least 1− Ce−cn
Var[Xν(g1, · )] ≤ C4ν
4
n + C ′e−c ′n,
and with (g1, g2) probability at least 1− C ′′e−c ′′d one has∣∣∣∣Xν − Eg2[Xν ] ∣∣∣∣ ≤ C5ν2 √ d n + C ′′′e−c ′′′n,
which was to be shown.
Lemma E.12. There exist absolute constants c, C,C ′ > 0 and an absolute constant K > 0 such that if n ≥ K log4 n, then for every ν ∈ [0, π] one has
Var [ E g2 [Xν( · , g2)] ] ≤ Cν 4 log n n + C ′e−cn.
Proof. Define
Yν(g1, g2) = 〈v0(g1, g2),vν(g1, g2)〉
ψ(‖v0(g1, g2)‖2)ψ(‖vν(g1, g2)‖2) ,
where ψ = ψ0.25 is as in Lemma E.31. Then by Cauchy-Schwarz and property 2 in Lemma E.31 (the case where either ‖v0‖2 = 0 or ‖vν‖2 = 0 is treated separately, since in this case Yν = 0), we obtain |Yν | ≤ 4, and
E g1 [( E g2 [Xν ]− E g2 [Yν ] )2] = E g1 [( E g2 [ 1Ec 〈v0,vν〉 ψ(‖v0‖2)ψ(‖vν‖2) ])2]
≤ E g1,g2
[( 1Ec
〈v0,vν〉 ψ(‖v0‖2)ψ(‖vν‖2) )2] ≤ 16µ(Ecm) ≤ Cne−cn,
where we use the fact that if (g1, g2) ∈ Em then ‖vν‖2 ≥ 12 for every 0 ≤ ν ≤ π and hence ψ(‖vν‖2) = ‖vν‖2 in the first line, apply Jensen’s inequality in the second line, and combine our bound on Yν with Hölder’s inequality and the measure bound in Lemma E.16 in the third line. An application of Lemma E.32 then yields
Var [ E g2 [Xν( · , g2)] ] ≤ Var [ E g2 [Yν( · , g2)] ] + Cne−cn ≤ Var [ E g2 [Yν( · , g2)] ] + Ce−cn/2,
where the last inequality holds when n is chosen to be larger than an absolute constant multiple of log n. It thus it suffices to control the variance of Yν . Applying Lemma E.26, we get for almost all g1 ∈ Rn
E g2
[Yν(g1, g2)] = ‖v0‖22
ψ(‖v0‖2)2 + ∫ ν 0 ∫ t 0 E g2 [(Ξ1 + Ξ2 + Ξ3 + Ξ4 + Ξ5 + Ξ6)(s, g1, g2)] dsdt,
where we follow the notation defined in Lemma E.13. We start by removing the term outside of the integral from consideration. We have as above |Yν | ≤ 4, so that |Eg2 [Yν ]| ≤ 4. Moreover, following the proof of the measure bound in Lemma E.16, but using only the pointwise concentration result, we assert that if n ≥ C an absolute constant there is an event E on which 0.5 ≤ ‖v0‖2 ≤ 2 with probability at least 1− 2e−cn with c > 0 an absolute constant. This implies that if g1 ∈ E we have
‖v0‖22 ψ(‖v0‖2)2 = 1,
and since ∣∣∣∣ ‖v0‖22ψ(‖v0‖2)2 ∣∣∣∣ ≤ 4, by the same argument used for Yν , we can calculate∥∥∥∥ ‖v0‖22ψ(‖v0‖2)2 − 1 ∥∥∥∥ L2 ≤ ∥∥∥∥( ‖v0‖22ψ(‖v0‖2)2 − 1 ) 1Ec ∥∥∥∥ L2 ≤ 5‖1Ec‖L2 ≤ Ce−cn,
by the Minkowski inequality and the triangle inequality. An application of Lemma E.32 implies that it is therefore sufficient to control the variance of the quantity
f(ν, g1) = 1 + ∫ ν 0 ∫ t 0 E g2 [(Ξ1 + Ξ2 + Ξ3 + Ξ4 + Ξ5 + Ξ6)(s, g1, g2)] dsdt.
By Lemma E.37, the Lyapunov inequality, and Fubini’s theorem, we have
(f(ν, g1)− E[f(ν, g1)])2 = (∫ ν 0 ∫ t 0 ( 6∑ i=1 E g2 [Ξi(s, g1, g2)]− E g1,g2 [Ξi(s, g1, g2)] ) dsdt )2 .
Using the elementary inequality(∫ ν 0 ∫ t 0 g(s) dsdt )2 ≤ ν ∫ ν 0 tdt ∫ t 0 g2(s) ds,
valid for any square integrable g : [0, π]→ R and proved with two applications of Jensen’s inequality, and Lemma E.37, we obtain
(f(ν, g1)− E[f(ν, g1)])2 ≤ ν ∫ ν
0
t ∫ t 0 ( 6∑ i=1 E g2 [Ξi(s, g1, g2)]− E g1,g2 [Ξi(s, g1, g2)] )2 dsdt.
Thus, again by Lemma E.37, the Lyapunov inequality, Fubini’s theorem, and compactness of [0, π], we have
Var[f(ν, · )] ≤ ν ∫ ν
0
t ∫ t 0 Var [ 6∑ i=1 E g2 [Ξi(s, g1, g2)] ] dsdt. (E.26)
We can control the variance under the integral using a combination of Lemmas E.35 and E.37, together with the deviations control given by Lemmas E.39, E.41 to E.44 and E.46, since we have chosen n according to the hypotheses of Lemma E.13. In particular, these lemmas furnish deviation bounds of size at most
Ci
(√ d log n
n + n−cid
) + C ′ine −c′in
that hold with probabilities at least 1−C ′′i n−c ′′ i d−C ′′′i ne−c ′′′ i n, for any d ≥ 1 larger than an absolute constant and suitable absolute constants specified above. We can simplify these bounds as follows: first, choose n such that n ≥ (2/c′′′i ) log n for each i, which guarantees that the bounds hold with probability at least 1−C ′′i n−c ′′ i d−C ′′′i e−c ′′′ i n/2. Next, choose n ≥ (2c′′i /c′′′i )d log n for all i, which implies that the bounds hold with probability at least 1 − 2 max{C ′′i , C ′′′i }n−c ′′ i d. Similarly, we also choose n such that n ≥ (2/c′i) log n for each i, which guarantees that the error terms that are exponential in n in the bounds are upper bounded by C ′ie
−c′in/2, and, choose n ≥ (2ci/c′i)d log n for all i, which implies that for all i
Ci
(√ d log n
n + n−cid
) + C ′ine −c′in ≤ Ci √ d log n
n + 2 max{Ci, C ′i}n−cid.
Finally, we make the particular choice d = 4/mini{ci, c′′i }, or the minimum required value of d, whichever is larger, so that there are absolute constants C,C ′, C ′′ > 0 such that with probability at least 1− C ′′n−4 we have for all i∣∣∣∣Eg2[Ξi(ν, g1, g2)]− Eg1,g2[Ξi(ν, g1, g2)] ∣∣∣∣ ≤ C √ log n n + C ′n−4 ≤ 2C √ log n n ,
where the last inequality holds when n is larger than an absolute constant. With these bounds, we can now invoke Lemma E.35 with Lemma E.37 to get
Var [ 6∑ i=1 E g2 [Ξi(s, g1, g2)] ] ≤ C log n n + C ′ n2 ≤ C ′′ log n n ,
for different absolute constants C,C ′, C ′′ > 0, and where the last inequality again holds n is larger than an absolute constant. Plugging back into (E.26) and evaluating the integrals, we get
Var[f(ν, · )] ≤ Cν4 log n n ,
which is enough to conclude.
Lemma E.13. Write
Yν(g1, g2) = 〈v0(g1, g2),vν(g1, g2)〉
ψ(‖v0(g1, g2)‖2)ψ(‖vν(g1, g2)‖2) ,
where ψ = ψ0.25 is as in Lemma E.31. There exist absolute constants c, c′, C, C ′, C ′′ > 0 and absolute constants K,K ′ > 0 such that for any d ≥ 1, if n ≥ Kd4 log4 n and if d ≥ K ′, then there is an event E such that
1. One has
∀ν ∈ [0, π], ∣∣∣∣Eg2[Yν ]− Eg1,g2[Yν ] ∣∣∣∣ ≤ C ′′ν2 √ d log n n + Ce−cn
if g1 ∈ E;
2. One has P[E ] ≥ 1− C ′n−c ′d.
Proof. Fix d > 0, and write f(ν, g1) = E
g2 [Yν(g1, g2)].
Applying Lemma E.26, we get for almost all g1 ∈ Rn
f(ν, g1) = ‖v0‖22
ψ(‖v0‖2)2 + ∫ ν 0 ∫ t 0 E g2 [(Ξ1 + Ξ2 + Ξ3 + Ξ4 + Ξ5 + Ξ6)(s, g1, g2)] dsdt, (E.27)
where
Ξ1(s, g1, g2) = n∑ i=1 σ(g1i) 3ρ(−g1i cot s) ψ(‖v0‖2)ψ(‖vis‖2) sin3 s Ξ2(s, g1, g2) = 〈v0,vs〉ψ′(‖vs‖2)‖vs‖2 ψ(‖v0‖2)ψ(‖vs‖2)2 − 〈v0,vs〉 ψ(‖v0‖2)ψ(‖vs‖2) Ξ3(s, g1, g2) = − 〈v0,vs〉〈vs, v̇s〉2ψ′′(‖vs‖2) ψ(‖v0‖2)ψ(‖vs‖2)2‖vs‖22 Ξ4(s, g1, g2) = −2 〈v0, v̇s〉〈vs, v̇s〉ψ′(‖vs‖2) ψ(‖v0‖2)ψ(‖vs‖2)2‖vs‖2 Ξ5(s, g1, g2) = − 〈v0,vs〉‖v̇s‖22ψ′(‖vs‖2) ψ(‖v0‖2)ψ(‖vs‖2)2‖vs‖2 Ξ6(s, g1, g2) = 2 〈v0,vs〉〈vs, v̇s〉2ψ′(‖vs‖2) ψ(‖v0‖2)ψ(‖vs‖2)3‖vs‖22 + 〈v0,vs〉〈vs, v̇s〉2ψ′(‖vs‖2) ψ(‖v0‖2)ψ(‖vs‖2)2‖vs‖32 .
Here we put Ξ1(0, g1, g2) = Ξ1(π, g1, g2) = 0, which does not affect the integral and which is equal to the limits limν↘0 Ξ1(ν, g1, g2) = limν↗π Ξ1(ν, g1, g2) for every (g1, g2).
Momentarily ignoring measurability issues, it is of interest to construct g1 events Ei of suitable probability on which we have
sup ν∈[0,π] ∣∣∣∣Eg2[Ξi(ν, g1, g2)]− Eg1,g2[Ξi(ν, g1, g2)] ∣∣∣∣ ≤ Ci (√ d log n n + n−cid ) + C ′ine −c′in (E.28)
for each i = 1, . . . , 6, and a g1 event E7 on which we have∣∣∣∣ ‖v0‖22ψ(‖v0‖2)2 − Eg1 [ ‖v0‖22 ψ(‖v0‖2)2 ]∣∣∣∣ ≤ C ′7e−c′7n. We can then consider the event E = ⋂7 i=1 Ei, possibly minus a negligible set on which (E.27) fails to hold, which has high probability via a union bound and on which we have simultaneously for all ν ∈ [0, π]∣∣∣∣f(ν, g1)− Eg1[f(ν, g1)] ∣∣∣∣ ≤ ∣∣∣∣ ‖v0‖22ψ(‖v0‖2)2 − Eg1 [ ‖v0‖22 ψ(‖v0‖2)2
]∣∣∣∣ +
6∑ i=1 ∫ ν 0 ∫ t 0 ∣∣∣∣Eg2[Ξi(s, g1, g2)]− Eg1,g2[Ξi(s, g1, g2)] ∣∣∣∣dsdt
≤ Cν2 (√ d log n
n + n−cd
) + C ′ne−c ′n,
by Fubini’s theorem and Lemma E.37, the triangle inequality (for | · | and for the integral), (E.28), and using ν2 ≤ π2 and worst-casing the remaining constants. To establish the bounds (E.28), we will employ lemma Lemma E.48, which shows that it is sufficient to obtain pointwise control and show a suitable s-Lipschitz property for each i ∈ [6]; following the lemma, these properties also imply Lebesgue measurability of the suprema immediately.
Reduction to product space events. Fix ν. By the triangle inequality, we have for each i = 1, . . . , 6∣∣∣∣Eg2[Ξi(ν, g1, g2)]− Eg1,g2[Ξi(ν, g1, g2)] ∣∣∣∣ ≤ Eg2 [∣∣∣∣Ξi(ν, g1, g2)− Eg1,g2[Ξi(ν, g1, g2)] ∣∣∣∣]. (E.29) Suppose we can construct (g1, g2) events E ′i such that
1. If (g1, g2) ∈ E ′i , then∣∣∣∣Ξi(ν, g1, g2)− Eg1,g2[Ξi(ν, g1, g2)] ∣∣∣∣ ≤ Ci (√ d log n n + n−cid ) + C ′ine −c′in;
2. One has P[E ′i ] ≥ 1− C ′′i n−c ′′ i d − C ′′′i ne−c ′′′ i n.
Then for each such i, we can write
E g2 [∣∣∣∣Ξi(ν, g1, g2)− Eg1,g2[Ξi(ν, g1, g2)] ∣∣∣∣]
= E g2
[( 1E′i + 1(E′i)c ) ∣∣∣∣Ξi(ν, g1, g2)− Eg1,g2[Ξi(ν, g1, g2)] ∣∣∣∣]
≤ Ci
(√ d log n
n + n−cid
) + C ′ine −c′in
+ E g2
[ 1(E′i)c ∣∣∣∣Ξi(ν, g1, g2)− Eg1,g2[Ξi(ν, g1, g2)] ∣∣∣∣] (E.30)
using nonnegativity of the integrand and boundedness of the indicator for E ′i in the second line. The random variable remaining in the second line is nonnegative, and by Fubini’s theorem (with Lemma E.37 for joint integrability) and the Schwarz inequality we have
E g1 [ E g2 [ 1(E′i)c ∣∣∣∣Ξi(ν, g1, g2)− Eg1,g2[Ξi(ν, g1, g2)] ∣∣∣∣]]
≤ E g1,g2
[ 1(E′i)c ]1/2 E
g1,g2
[( Ξi(ν, g1, g2)
−Eg1,g2 [Ξi(ν, g1, g2)] )2]1/2 ≤ C ( C ′′i n −c′′i d + C ′′′i ne −c′′′i n )1/2 ,
where the second line applies Lemma E.37 and the Lyapunov inequality. We can replace this last inequality with one equivalent to the measure bound on (E ′i)c using subadditivity of the square root and reducing the constants c′i and c ′′ i by a factor of 2. Using this last inequality, Markov’s inequality implies for any t ≥ 0
P [ E g2 [ 1(E′i)c ∣∣∣∣Ξi(ν, g1, g2)− Eg1,g2[Ξi(ν, g1, g2)] ∣∣∣∣] ≥ Cn− 12 c′′i d + C ′n1/2e− 12 c′′′i n] ≤ Cn− 1 2 c ′′ i d + C ′n1/2e− 1 2 c ′′′ i n,
which, together with (E.29) and after worst-casing some exponents and constants, implies that there is a g1 event Ei that satisfies (the constants C and C ′ are scoped across properties 1 and 2)
1. If g1 ∈ Ei, then∣∣∣∣Eg2[Ξi(ν, g1, g2)]− Eg1,g2[Ξi(ν, g1, g2)] ∣∣∣∣ ≤ Ci √ d log n n + (Ci + C)n − 12 cid
+ (C ′i + C ′)ne−
1 2 min{c ′ i,c ′′′ i }n;
2. One has P[Ei] ≥ 1− Cn− 1 2 c ′′ i d − C ′ne− 1 2 c ′′′ i n.
Thus, we can pass from (g1, g2) events to g1 events with only a worsening of constants, and it suffices to construct the events E ′i . Additionally, we can leverage this same framework to pass ν-uniform control from the product space to g1-space. Suppose we can construct (g1, g2) events E ′i such that
1. If (g1, g2) ∈ E ′i , then
∀ν ∈ [0, π], ∣∣∣∣Ξi(ν, g1, g2)− Eg1,g2[Ξi(ν, g1, g2)] ∣∣∣∣ ≤ Ci (√ d log n n + n−cid ) + C ′ine −c′in;
2. One has P[E ′i ] ≥ 1− C ′′i n−c ′′ i d − C ′′′i ne−c ′′′ i n.
Then following (E.30), we can assert
∀ν ∈ [0, π], E g2 [∣∣∣∣Ξi(ν, g1, g2)− Eg1,g2[Ξi(ν, g1, g2)] ∣∣∣∣]
≤ Ci
(√ d log n
n + n−cid
) + C ′ine
−c′in + E g2
[ 1(E′i)c ∣∣∣∣Ξi(ν, g1, g2)− Eg1,g2[Ξi(ν, g1, g2)] ∣∣∣∣].
To get uniform control of this last random variable, we can use Lemma E.37, which tells us that we have a bound
∀ν ∈ [0, π], E g2 [∣∣∣∣Ξi(ν, g1, g2)− Eg1,g2[Ξi(ν, g1, g2)] ∣∣∣∣]
≤ Ci
(√ d log n
n + n−cid
) + C ′ine
−c′in + E g2
[ 1(E′i)cfi(g1, g2) ] ,
(E.31)
where fi is in L4(Rn × Rn), and has L4 norm bounded by an absolute constant Ci > 0. Then Fubini’s theorem and the Schwarz inequality allow us to assert
E g1,g2
[ 1(E′i)cfi(g1, g2) ] ≤ Ci E
g1,g2
[ 1(E′i)c ]1/2 ,
which can be controlled exactly as in the pointwise control argument. In particular, an application of Markov’s inequality gives
P [ E g2 [ 1(E′i)cfi(g1, g2) ] ≥ Cn− 1 2 c ′′ i d + C ′n1/2e− 1 2 c ′′′ i n ] ≤ Cn− 1 2 c ′′ i d + C ′n1/2e− 1 2 c ′′′ i n,
so that, returning to (E.31), we have uniform control of the quantity |Eg2 [Ξi(ν, g1, g2)] − Eg1,g2 [Ξi(ν, g1, g2)]| on an event of appropriately high probability. In particular, we have incurred only losses in the constants compared to the pointwise case.
Approach to Lipschitz estimates. We will use this framework for controlling the Ξ1 and Ξ5 terms only. Accordingly, the sections for those terms below will produce results of the following type, for absolute constants ci, c′i, c ′′ i , c ′′′ i , Ci, C ′ i, C ′′ i , C ′′′ i , C ′′′′ i > 0 for i = 1, 2, and parameters d ≥ 1, δ > 0 such that d and δ are larger than (separate) absolute constants and n satisfies certain conditions involving d:
1. For each ν ∈ [0, π] fixed, with probability at least 1− C ′′′1 n−c ′′ 1 d − C ′′′′1 ne−c ′′′ 1 n, we have
that
|E g2
[Ξi(ν, g1, g2)− E[Ξi(ν, g1, g2)]]| ≤ C1 √ d log n/n+ C ′1n −c1d + C ′′1 ne −c′1n;
2. With probability at least 1 − C ′′2 e−c ′ 2n − C ′′′2 n−δ , we have that |Eg2 [Ξi(ν, g1, g2) −
E[Ξi(ν, g1, g2)]]| is (C2 + C ′2n1+δ)-Lipschitz.
We show here that we can use these properties to obtain uniform concentration of the relevant quantities. WriteM = C1 √ d log n/n+C ′1n −c1d+C ′′1 ne −c′1n; we are interested in showing that uniform bounds of sizes close toM hold with probability not much smaller than that of the pointwise bounds. By Lemma E.48, it follows from the assumed properties that for any 0 < ε < 1 one has
P [ sup
ν∈[0,π] ∣∣∣∣Eg2[Ξi(ν, g1, g2)− E[Ξi(ν, g1, g2)]] ∣∣∣∣ ≤M + ε (C2 + C ′2n1+δ) ] ≥ 1− ( C ′′′1 n −c′′1 d + C ′′′′1 ne −c′′′1 n ) Kε−1 − ( C ′′2 e −c′2n + C ′′′2 n −δ ) ,
where K > 0 is an absolute constant. To make the RHS of the bound on the supremum of size comparable to M , it suffices to choose ε = C1 √ d log n/n/(C2 + C ′ 2n
1+δ). We have C2 + C ′ 2n
1+δ ≤ K ′n1+δ for K ′ > 0 an absolute constant, and so we have ε−1 ≤ K ′n3/2+δ for K ′ > 0 another absolute constant. This gives(
C ′′′1 n −c′′1 d + C ′′′′1 ne −c′′′1 n ) ε−1 ≤ K ′n3/2+δ ( C ′′′1 e −c′′1 d logn + C ′′′′1 e −c′′′1 /2n ) ≤ K ′n3/2+δe−c ′′ 1 d logn
≤ K ′n−c ′′ 1 d/2,
where K ′ > 0 is an absolute constant whose value changes from line to line; and where the first inequality assumes that n ≥ (2/c′′′1 ) log n, the second inequality assumes that n ≥ (2c′′1/c′′′1 )d log n, and the third assumes that δ ≤ c′′1d/2−3/2. Choosing d so that the value c′′1d/2−3/2 is larger than the minimum value for δ (i.e., larger than an absolute constant), then choosing δ = c′′1d/2 − 3/2, and finally choosing d ≥ 6/c′′1 , we obtain
P [ sup
ν∈[0,π] ∣∣∣∣Eg2[Ξi(ν, g1, g2)− E[Ξi(ν, g1, g2)]] ∣∣∣∣ ≤ 2M ] ≥ 1−Kn−c ′′ 1 d/2−C ′′2 e−c ′ 2n−C ′′′2 n−c ′′ 1 d/4,
where K > 0 is an absolute constant, which is an acceptable level of uniformization.
Completing the proof. To obtain the desired control, we apply the uniform framework for the terms Ξi, i = 2, 3, 4, 6; and the pointwise with Lipschitz control framework for the terms Ξi, i = 1, 5. We also establish high probability control of the zero-order term in Lemma E.38. The events we need for the pointwise framework terms are constructed in Lemmas E.39, E.40, E.44 and E.45. The events we need for the uniform framework are constructed in Lemmas E.41 to E.43 and E.46. Because n and d are chosen appropriately by our hypotheses here, we can invoke each of these lemmas to construct the necessary sub-events and obtain an event E which satisfies
1. One has ∀ν ∈ [0, π], ∣∣∣∣Eg2[Yν ]− Eg1,g2[Yν ] ∣∣∣∣ ≤ Cν2 (√ d log n n + n−cd ) + C ′ne−c ′n
if g1 ∈ E ;
2. One has P[E ] ≥ 1− C ′′n−c ′′d − C ′′′ne−c ′′′n.
We can adjust d and n slightly to obtain an event with the properties claimed in the statement of the lemma. Indeed, choosing n to be larger than an absolute constant multiple of log n, we can obtain C ′ne−c
′n ≤ C ′e−c′n/2 and C ′′′ne−c′′′n ≤ C ′′′e−c′′′n/2; choosing n to be larger than an absolute constant multiple of d log n, we can obtain C ′′n−c ′′d + C ′′′e−c ′′′n/2 ≤ 2C ′′n−c′′d; and choosing
d to be larger than an absolute constant, we can assert √ d log n/n + n−cd ≤ 2 √ d log n/n. This turns the guarantees of E into the guarantees claimed in the statement of the lemma, and completes the proof.

heading: E.3.3 PROVING LEMMA E.7

text:
Lemma E.14. One has bounds
1− ν 2
2 ≤ cosϕ(ν) ≤ 1− cν2, ν ∈ [0, π].
Proof. Write f(ν) = cosϕ(ν) = cos ν+π−1(sin ν−ν cos ν), where the last equality follows from Lemma E.2. We start by obtaining quadratic bounds on f(ν) for ν ∈ [0, 0.1]. In particular, we will show
1− 12ν 2 ≤ f(ν) ≤ 1− 14ν 2, ν ∈ [0, 0.1]. (E.32) We readily calculate
f ′(ν) = − sin ν + π−1ν sin ν, f ′′(ν) = − cos ν + π−1(ν cos ν + sin ν).
Taylor expanding at ν = 0 gives
1 + inft∈[0,0.1] f
′′(t)
2 ν2 ≤ f(ν) ≤ 1 +
supt∈[0,0.1] f ′′(t)
2 ν2.
We have f ′′(0) = −1, and sin ν ≤ sin 0.1 on our interval of interest by monotonicity. The derivative of ν cos ν is cos ν − ν sin ν; ν sin ν is increasing as the product of two increasing functions (given ν ≤ 0.1), and one checks that cos(0.1)− 0.1 sin(0.1) > 0; therefore ν cos ν ≤ 0.1 cos(0.1) on our domain of interest. One checks numerically
− cos(0.1) + π−1 ( 0.1 cos(0.1) + sin(0.1) ) < − 12 < 0,
and this establishes f(ν) ≤ 1 − 14ν 2 on [0, 0.1]. If ν ≤ π/2, we have cos ≥ 0 and sin ≥ 0, so that ν cos ν + sin ν ≥ 0 on this domain. This implies f ′′(ν) ≥ − cos ν ≥ −1 for 0 ≤ ν ≤ π/2, which proves inft∈[0,π/2] f ′′(t) = −1, and establishes the lower bound on [0, π/2].
To obtain (possibly) looser bounds on [0, π], we use a bootstrapping approach. The lower bound is more straightforward; to assert the lower bound on [0, π], we evaluate constants numerically to find that the lower bound’s value at π/2 is 1 − π2/8 < 0, and given that f ≥ 0 by Lemma E.5 and the concave quadratic bound is maximized at ν = 0, it follows that the bound holds on the entire interval.
For bootstrapping the upper bound, we note that the equation f ′(ν) = − sin ν + π−1ν sin ν = sin ν (ν π − 1 )
shows immediately that f is a strictly decreasing function of ν on (0, π). Therefore f(ν) ≤ f(0.1) on [0.1, π], and so the quadratic function ν 7→ 1 − π−2(1 − f(0.1))ν2, which is lower bounded by 1 − ν2/4 on [0, π] by the fact that both concave quadratic functions are maximized at 0 and the verification 1− π2/4 < 0 ≤ f(0.1), is an upper bound for f on all of [0, π]; so the claim holds with c = π−2(1− f(0.1)).
Lemma E.15. There exist absolute constants c, C,C ′, C ′′ > 0 such that if n ≥ C log n, then one has ∣∣∣∣ Eg1,g2[Xν ]− cosϕ(ν) ∣∣∣∣ ≤ C ′e−cn + C ′′ν2/n. Proof. Write h(ν) = cosϕ(ν)−E[Xν ]. By Lemmas E.24 and E.25, we have a second-order Taylor formula
h(ν) = h(0) + ∫ ν 0 ( h′(0) + ∫ t 0 h′′(s) ds ) dt.
We calculate h′(0) = 0, since E[〈v0, v̇0〉] = E[〈σ(g1), g2〉] = 0, and P⊥v0v0 = 0. We also have h(0) = E[‖v0‖22]− E[1Em ] = µ(Ecm) (writing m = 1), so this formula yields
|h(ν)| ≤ µ(Ecm) + ν2
2 ess sup ν′∈[0,π]
|h′′(ν′)|,
and we see that it suffices to bound h′′. We will use the (Lebesgue-a.e.) expression |h′′(ν)| = ∣∣∣∣E[〈v̇ν , v̇0〉]− E[1Em〈 1‖vν‖2 ( I − vνv ∗ ν ‖vν‖22 ) v̇ν , 1 ‖v0‖2 ( I − v0v ∗ 0 ‖v0‖22 ) v̇0 〉]∣∣∣∣. Distributing over the inner product and applying rotational invariance to combine the two cross terms, then using the triangle inequality, we obtain the bound
|h′′(ν)| ≤ ∣∣∣∣∣∣∣∣∣E[〈v̇ν , v̇0〉]− E [ 1Em 〈v̇0, v̇ν〉 ‖v0‖2‖vν‖2 ] ︸ ︷︷ ︸
Ξ1(ν)
∣∣∣∣∣∣∣∣∣+ ∣∣∣∣∣∣∣∣∣2E [ 1Em 〈v̇0,vν〉〈vν , v̇ν〉 ‖v0‖2‖vν‖32 ] ︸ ︷︷ ︸
Ξ2(ν) ∣∣∣∣∣∣∣∣∣ +
∣∣∣∣∣∣∣∣∣E [ 1Em 〈v̇0,v0〉〈v0,vν〉〈vν , v̇ν〉 ‖v0‖32‖vν‖32 ] ︸ ︷︷ ︸
Ξ3(ν) ∣∣∣∣∣∣∣∣∣. We proceed by giving magnitude bounds for Ξi(ν), i = 1, 2, 3. Because we are working with expectations, it suffices to fix one value ν ∈ [0, π] and prove pointwise ν-independent bounds; we will exploit this in the sequel to easily define extra good events without having to uniformize, and we will generally suppress the notational dependence of Ξi on ν as a result. We will also repeatedly use the fact that we have µ(Ec1) ≤ Cne−cn for some absolute constants c, C > 0 by Lemma E.16. We will accrue a large number of additive C/n and C ′npme−cn errors as we bound the Ξi terms; at the end of the proof we will worst-case the constants in each additive error and assert a bound of the form claimed.
Ξ1 control. Let E = {‖v̇ν‖2 ≤ 2} ∩ {‖v̇0‖2 ≤ 2}. By Lemma E.17 and a union bound, we have µ(Ec) ≤ Ce−cn. Define an event E1 = Em ∩ E . The first step is to pass to the control of
Ξ̃1 := E [ 1E1〈v̇ν , v̇0〉 ( 1− 1 ‖v0‖2‖vν‖2 )] .
The triangle inequality gives∣∣∣Ξ1 − Ξ̃1∣∣∣ ≤ ∣∣E[1Ec1〈v̇ν , v̇0〉]∣∣+ ∣∣∣∣E[1Em\E〈 v̇ν‖vν‖2 , v̇0‖v0‖2 〉]∣∣∣∣
The first term is readily controlled from two applications of the Schwarz inequality, a union bound, and rotational invariance together with Lemma E.29:∣∣E[1Ec1〈v̇ν , v̇0〉]∣∣ ≤ E[1Ec1]1/2E[‖v̇ν‖42]1/4E[‖v̇0‖42]1/4
≤ ( µ(Ecm) + Ce−cn )1/2 E[‖v̇0‖42]1/2 ≤ ( Cne−cn + C ′e−c ′n )1/2( 1 + C ′′
n )1/2 ≤ Cn1/2e−cn,
where in the last line we require n to be at least the value of a large absolute constant. The calculation is similar for the normalized term, except we also apply the definition of Em to get some extra cancellation:∣∣∣∣E[1Em\E 〈v̇ν , v̇0〉‖vν‖2‖v0‖2 ]∣∣∣∣ ≤ E[1Em\E |〈v̇ν , v̇0〉|‖vν‖2‖v0‖2 ] ≤ 4E [ 1Em\E |〈v̇ν , v̇0〉|
] ≤ 4E[1Ec |〈v̇ν , v̇0〉|]
≤ 4E[1Ec ]1/2E [ ‖v̇ν‖42 ]1/4E[‖v̇0‖42]1/4 ≤ Ce−cn ( 1 + C ′
n
)1/2 ≤ Ce−cn,
where in the last line we apply our bounds from the first term and use n ≥ 1 to obtain the final inequality. Next, Taylor expansion of the smooth convex function x 7→ x−1/2 on the domain x > 0 about the point x = 1 gives
x−1/2 = 1− 1 2 (x− 1) + 3 4 ∫ x 1 (x− t)t−5/2 dt. (E.33)
Given that Em guarantees ‖vν‖2 ≥ 12 , we can apply this to get a bound
1E1 ( 1− 1 ‖v0‖2‖vν‖2 ) = 1E1 ( 1
2
( ‖v0‖22‖vν‖22 − 1 ) − 3
4 ∫ ‖v0‖22‖vν‖22 1 (‖v0‖22‖vν‖22 − t)t−5/2 dt ) .
On E ′, we also have ‖v0‖−22 ‖vν‖ −2 2 ≤ 24, so we can control the integral residual as
0 ≤ 1E1 3
4 ∫ ‖v0‖22‖vν‖22 1 (‖v0‖22‖vν‖22 − t)t−5/2 dt ≤ 1E1384 ( ‖v0‖22‖vν‖22 − 1 )2 ,
where we replace the tighter bound that we get in the case ‖v0‖22‖vν‖22 ≥ 1 with the worst-case bound from the other case. This gives bounds
1E1
( 1
2
( ‖v0‖22‖vν‖22 − 1 ) − 384 ( ‖v0‖22‖vν‖22 − 1 )2) ≤ 1E1 (1− 1‖v0‖2‖vν‖2 )
≤ 1E1 1
2
( ‖v0‖22‖vν‖22 − 1 ) .
Given that ‖v̇ν‖2 ≤ 2 on E ′′, it follows |〈v̇0, v̇ν〉| ≤ 4 on E1, so that 〈v̇0, v̇ν〉+ 4 ≥ 0 here. Writing
1E1〈v̇0, v̇ν〉 (
1− 1 ‖v0‖2‖vν‖2 ) = 1E1 [ (〈v̇0, v̇ν〉+ 4) ( 1− 1 ‖v0‖2‖vν‖2 ) − 4 ( 1− 1 ‖v0‖2‖vν‖2 )] ,
we can apply nonnegativity to obtain upper and lower bounds
Ξ̃1 ≤ E [ 1E1〈v̇0, v̇ν〉 ( 1
2
( ‖v0‖22‖vν‖22 − 1 ) + 4C ( ‖v0‖22‖vν‖22 − 1 )2)] ;
Ξ̃1 ≥ E [ 1E1〈v̇0, v̇ν〉 ( 1
2
( ‖v0‖22‖vν‖22 − 1 ) − 5C ( ‖v0‖22‖vν‖22 − 1 )2)] ,
where C = 384.
We continue with bounding the quadratic term arising in the previous equation. We have∣∣∣E[1E1〈v̇0, v̇ν〉 (‖v0‖22‖vν‖22 − 1)2]∣∣∣ ≤ 4E[(‖v0‖22‖vν‖22 − 1)2] = 4E [ ‖v0‖42‖vν‖42 − 2‖v0‖22‖vν‖22 + 1
] ≤ 4 ( 1− 2E[‖v0‖22‖vν‖22] + E [ ‖v0‖82
]) ≤ 4 ( 1− 2(1− (Cn−1 + C ′e−cn))2 + ( 1 + C ′′
n )) ≤ Cn−1e−cn + C ′e−c ′n + C ′′
n .
The first inequality applies the triangle inequality for the integral, the definition of E1 and CauchySchwarz, then drops the indicator for E1 because the remaining terms are nonnegative; the second line is just distributing; the third line rearranges and applies the Schwarz inequality; and the fourth inequality applies Jensen’s inequality and Lemma E.18 to control the second term (to apply this lemma, we need to choose n larger than an absolute constant; we assume this is done), and Lemma E.29 to control the third term. Since n ≥ 1, this gives a C/n + C ′e−cn bound on the quadratic term.
Next is the linear term; our first step will be to get rid of the indicator. By the triangle inequality, it suffices to get control of the corresponding term with the indicator for Ec1 instead; we control it as follows:∣∣E[1Ec1〈v̇0, v̇ν〉 (‖v0‖22‖vν‖22 − 1)]∣∣
≤ E [ 1Ec1 ]1/2E[〈v̇0, v̇ν〉2 (‖v0‖22‖vν‖22 − 1)2]1/2
≤ ( Cne−cn + C ′e−c ′n )1/2 E [ ‖v̇0‖22‖v̇ν‖22 ( ‖v0‖22‖vν‖22 − 1 )2]1/2 ≤ ( Cne−cn + C ′e−c ′n )1/2 E [ ‖v̇0‖22‖v̇ν‖22‖v0‖42‖vν‖42 + ‖v̇0‖22‖v̇ν‖22
]1/2 ≤ ( Cne−cn + C ′e−c ′n )1/2 ( E [ ‖v̇0‖82
]1/4E[‖v0‖162 ]1/4 + E[‖v̇0‖42]1/2) ≤ ( Cne−cn + C ′e−c ′n )1/2((
1 + C1 n
)1/4( 1 +
C2 n
)1/4 + ( 1 +
C3 n )1/2) ≤ Cn1/2e−cn + C ′e−c ′n.
The first line is the Schwarz inequality; the second line is the good event measure bound and CauchySchwarz; the third line distributes and drops the cross term, given that all factors are nonnegative; the fourth line applies subadditivity of the square root function, then the Schwarz inequality to the resulting separate terms; the fifth line applies Lemma E.29; and the last line again uses square root subadditivity and treats the remaining terms as multiplicative constants, since n ≥ 1. Therefore passing to the linear term without the indicator incurs only an additional exponential factor. Proceeding, we drop the indicator and distribute to get for the linear term
E [ 〈v̇0, v̇ν〉 ( ‖v0‖22‖vν‖22 − 1 )] = E [ 〈v̇0, v̇ν〉‖v0‖22‖vν‖22 ] − E[〈v̇0, v̇ν〉];
it is of interest to apply Lemma E.30 to these two terms to get the proper cancellation, and for this we just need to check that the coordinates of each factor in the product have subexponential moment growth with the proper rate. For even powers of `2 norms of vν , this follows immediately from Lemma G.11 after scaling by √ 2/n; for the inner product term, the coordinate functions are σ̇(g1i)g2iσ̇(g1i cos ν + g2i sin ν)(g2i cos ν − g1i sin ν), and we have from the Schwarz inequality and rotational invariance
E [ |σ̇(g1i)g2iσ̇(g1i cos ν + g2i sin ν)(g2i cos ν − g1i sin ν)|k ] ≤ E [ σ̇(g1i)g 2k 2i ] ,
which has subexponential moment growth with rate Cn−1 by Lemma E.17 and Lemma G.11 after rescaling by √ 2/n. These formulas also show that when k = 1, we have a bound of precisely n−1. This makes Lemma E.30 applicable, so we can assert bounds∣∣∣E[〈v̇0, v̇ν〉 (‖v0‖22‖vν‖22 − 1)]− (n3E[(v̇0)1(v̇ν)1]E[σ(g11)2]2 − nE[(v̇0)1(v̇ν)1])∣∣∣ ≤ Cn
Because E [ σ(g11) 2 ]2
= n−2, this is enough to conclude a C/n bound on the magnitude of the linear term. Thus, in total, we have shown
|Ξ1| ≤ C
n + C ′e−cn + C ′′n1/2e−c ′n,
where we combine the different constant that appear in the various exponential additive errors throughout our work by choosing the largest magnitude scaling factor and the smallest magnitude constant in the exponent to assert the previous expression.
Ξ2 control. The approach is similar to what we have used to control Ξ1. We start with exactly the same E1 event definition, and as previously define
Ξ̃2 = E [ 1E1 〈v̇0,vν〉〈vν , v̇ν〉‖v0‖22
‖v0‖32‖vν‖32
] ,
and then calculating |Ξ̃2 − Ξ2| = ∣∣∣∣E[1Em\E 〈v̇0,vν〉〈vν , v̇ν〉‖v0‖22‖v0‖32‖vν‖32 ]∣∣∣∣ ≤ 26E [ 1Em\E |〈v̇0,vν〉〈vν , v̇ν〉|‖v0‖ 2 2
] ≤ 26E [ 1Ec |〈v̇0,vν〉〈vν , v̇ν〉|‖v0‖22
] ≤ 26E[1Ec ]1/2E [ 〈v̇0,vν〉4
]1/4E[〈vν , v̇ν〉8]1/8E[‖v0‖82]1/8 ≤ 26E[1Ec ]1/2E [ ‖v̇0‖82
]1/8E[‖vν‖82]1/8E[‖vν‖162 ]1/16E[‖v̇ν‖162 ]1/16E[‖v0‖82]1/8 ≤ Ce−cn + C ′n1/2e−c ′n,
using the same ideas as in the previous section, plus several applications of the Schwarz inequality and a final application of Lemma E.29. We can therefore pass to Ξ̃2 with a small additive error. Next, we Taylor expand in the same way as previously, except that larger powers in the denominator force the constant in our residual bound to be 3 · 227, and the event E1 now gives us a bound |〈v̇0,vν〉〈vν , v̇ν〉‖v0‖22| ≤ 26 on the numerator, which we add and subtract as before to exploit nonnegativity. We get
Ξ̃2 ≤ E [ 1E1〈v̇0,vν〉〈vν , v̇ν〉‖v0‖22 ( 1
2
( 3− ‖v0‖62‖vν‖62 ) + (26 + 1)C ( ‖v0‖62‖vν‖62 − 1 )2)] ;
Ξ̃2 ≥ E [ 1E1〈v̇0,vν〉〈vν , v̇ν〉‖v0‖22 ( 1
2
( 3− ‖v0‖62‖vν‖62 ) − 26C ( ‖v0‖62‖vν‖62 − 1 )2)] ,
with C = 3 · 227. Proceeding to control the quadratic term, we have∣∣∣E[1E1〈v̇0,vν〉〈vν , v̇ν〉‖v0‖22 (‖v0‖62‖vν‖62 − 1)2]∣∣∣ ≤ 43E [( ‖v0‖62‖vν‖62 − 1
)2] = 26E [ ‖v0‖122 ‖vν‖122 − 2‖v0‖62‖vν‖62 + 1
] ≤ 26 ( 1− 2E[‖v0‖62‖vν‖62] + E [ ‖v0‖242
]) ≤ 26 ( 1− 2(1− (Cn−1 + C ′e−cn))6 + (1 + C ′′n−1)
) ≤ Cn−1 +
3∑ k=1 ( 6 2k − 1 )( C ′n−1 + C ′′e−cn )2k−1 ≤ Cn−1 + C ′
3∑ k=1 2k−1∑ j=0 n−(2k−1−j)e−cnj
≤ Cn−1 + C ′e−cn.
The justifications for the first four lines are identical to those of the previous section. In the last three lines, we use the binomial theorem twice to expand the sixth power term, and we assert the final line
by the fact that k > 0, so that each term in the sum corresponding to a j = 0 has a positive inverse power of n attached, and when j = 2k − 1 we pick up an exponential factor. Moving on to the linear term, as in the previous section we start by dropping the indicator. We control the residual as follows:∣∣E[1Ec1〈v̇0,vν〉〈vν , v̇ν〉‖v0‖22 (‖v0‖62‖vν‖62 − 3)]∣∣
≤ E [ 1Ec1 ]1/2E[‖v̇0‖22‖v̇ν‖22‖v0‖22‖vν‖42 (‖v0‖62‖vν‖62 − 3)2]1/2
≤ E [ 1Ec1 ]1/2E[‖v̇0‖22‖v̇ν‖22‖v0‖142 ‖vν‖162 + 3‖v̇0‖22‖v̇ν‖22‖v0‖22‖vν‖42]1/2
≤ E [ 1Ec1 ]1/2 (E[‖v̇0‖22‖v̇ν‖22‖v0‖142 ‖vν‖162 ]1/2 + 3E[‖v̇0‖22‖v̇ν‖22‖v0‖22‖vν‖42]1/2)
≤ Ce−cn + C ′n1/2e−c ′n.
The justifications are almost the same as the previous section, although we have compressed some steps into fewer lines here and we have omitted the final simplifications which follow from applying the Schwarz inequality to each of the two expectations in the second-to-last line 3 times and then applying Lemma E.29. Dropping the indicator and distributing now gives:
E [ 〈v̇0,vν〉〈vν , v̇ν〉‖v0‖22 ( ‖v0‖62‖vν‖62 − 3 )] =
E [ 〈v̇0,vν〉〈vν , v̇ν〉‖v0‖22‖v0‖62‖vν‖62 ] −3E [ 〈v̇0,vν〉〈vν , v̇ν〉‖v0‖22 ] ;
to apply Lemma E.30, we check the two new coordinate functions that appear in this linear term: for 〈v̇0,vν〉, we have
E [ |σ̇(g1i)g2iσ(g1i cos ν + g2i sin ν)|k ] ≤ E [ σ̇(g1i)g 2k 2i ]1/2E[σ(g1i)2k]1/2, (E.34) and for 〈v̇ν ,vν〉, we have likewise
E [ |σ(g1i cos ν + g2i sin ν)(g2i cos ν − g1i sin ν)|k ] ≤ E [ σ̇(g1i)g 2k 2i ]1/2E[σ(g1i)2k]1/2, (E.35) both by the Schwarz inequality and rotational invariance. As before, an appeal to Lemmas G.11 and E.17 implies that these two coordinate functions satisfy the hypotheses of Lemma E.30, so we have a bound∣∣∣∣E[〈v̇0,vν〉〈vν , v̇ν〉‖v0‖22 (‖v0‖62‖vν‖62 − 3)]− n9E[(v̇0)1(vν)1]E[(v̇ν)1(vν)1]E[σ(w11)2]7
+ 3n3E[(v̇0)1(v̇ν)1]E[(v̇ν)1(vν)1]E [ σ(w11) 2 ]∣∣∣∣ ≤ Cn .
Noticing that E[〈vν , v̇ν〉] = −E[〈v0, v̇0〉] = −E[〈σ(g1), g2〉] = 0,
by rotational invariance and independence, we conclude by identically-distributedness of the coordinates of vν and v̇ν
n9E[(v̇0)1(vν)1]E[(v̇ν)1(vν)1]E [ σ(g11) 2 ]7 − 3n3E[(v̇0)1(v̇ν)1]E[(v̇ν)1(vν)1]E[σ(g11)2] = 0,
which establishes the desired control on Ξ2. Thus, in total, we have shown
|Ξ2| ≤ C
n + C ′e−cn + C ′′n1/2e−c ′n,
where we combine the different constant that appear in the various exponential additive errors throughout our work by choosing the largest magnitude scaling factor and the smallest magnitude constant in the exponent to assert the previous expression.
Ξ3 control. The argument for control of this term is very similar to the previous section, since the degrees of the denominators now match. We start by defining
Ξ̃3 = E [ 1E1 〈v̇0,v0〉〈v0,vν〉〈vν , v̇ν〉
‖v0‖32‖vν‖32
] ,
with the same E1 event as previously, and then calculating |Ξ̃3 − Ξ3| = ∣∣∣∣E[1Em\E 〈v̇0,v0〉〈vν , v̇ν〉〈v0,vν〉‖v0‖32‖vν‖32 ]∣∣∣∣ ≤ 26E [ 1Em\E |〈v̇0,v0〉〈vν , v̇ν〉〈v0,vν〉|
] ≤ 26E[1Ec |〈v̇0,v0〉〈vν , v̇ν〉〈v0,vν〉|]
≤ 26E[1Ec ]1/2E [ 〈v̇0,v0〉4 ]1/4E[〈vν , v̇ν〉8]1/8E[〈v0,vν〉8]1/8 ≤ 26E[1Ec ]1/2E [ ‖v̇0‖82
]1/8E[‖v0‖82]1/8E[‖v̇ν‖162 ]1/16E[‖v0‖162 ]1/16E[‖vν‖162 ]1/8 ≤ Cn1/2e−cn + Ce−c ′n,
using the same ideas as in the previous section. We can therefore pass to Ξ̃3 with an exponentially small error. Next, we Taylor expand in the same way as previously, obtaining
Ξ̃3 ≤ E [ 1E1〈v0,vν〉〈v̇0,v0〉〈vν , v̇ν〉 ( 1
2
( 3− ‖v0‖62‖vν‖62 ) + (43 + 1)C ( ‖v0‖62‖vν‖62 − 1 )2)] ;
Ξ̃3 ≥ E [ 1E1〈v0,vν〉〈v̇0,v0〉〈vν , v̇ν〉 ( 1
2
( 3− ‖v0‖62‖vν‖62 ) − 43C ( ‖v0‖62‖vν‖62 − 1 )2)] ,
with C = 3 · 227. Proceeding to control the quadratic term, we notice∣∣∣E[1E1〈v̇0,v0〉〈vν , v̇ν〉〈v0,vν〉 (‖v0‖62‖vν‖62 − 1)2]∣∣∣ ≤ 43E[(‖v0‖62‖vν‖62 − 1)2] = Cn−1 + C ′e−cn,
since the final term was controlled in the previous section. Moving on to the linear term, as in the previous section we start by dropping the indicator. We control the residual as follows:∣∣E[1Ec1〈v̇0,v0〉〈vν , v̇ν〉〈v0,vν〉 (‖v0‖62‖vν‖62 − 3)]∣∣
≤ E [ 1Ec1 ]1/2E[‖v̇0‖22‖v̇ν‖22‖v0‖42‖vν‖42 (‖v0‖62‖vν‖62 − 3)2]1/2
≤ E [ 1Ec1 ]1/2E[‖v̇0‖22‖v̇ν‖22‖v0‖162 ‖vν‖162 + 3‖v̇0‖22‖v̇ν‖22‖v0‖42‖vν‖42]1/2
≤ E [ 1Ec1 ]1/2 (E[‖v̇0‖22‖v̇ν‖22‖v0‖162 ‖vν‖162 ]1/2 + 3E[‖v̇0‖22‖v̇ν‖22‖v0‖42‖vν‖42]1/2)
≤ Ce−cn + C ′n1/2e−c ′n,
by the same argument as in the previous section. Dropping the indicator and distributing now gives:
E [ 〈v̇0,v0〉〈vν , v̇ν〉〈v0,vν〉 ( ‖v0‖62‖vν‖62 − 3 )] = E [ 〈v̇0,v0〉〈vν , v̇ν〉〈v0,vν〉‖v0‖62‖vν‖62 ] −3E[〈v̇0,v0〉〈vν , v̇ν〉〈v0,vν〉];
to apply Lemma E.30, we check the one new coordinate function that appears in this linear term: for 〈v0,vν〉, we have E [ |σ(g1i)σ(g1i cos ν + g2i sin ν)|k ] ≤ E [ σ(g1i) 2k ] , (E.36)
by the Schwarz inequality and rotational invariance. As before, an appeal to Lemmas G.11 and E.17 implies that this coordinate function satisfies the hypotheses of Lemma E.30, so we have a bound∣∣∣∣∣∣∣ E [ 〈v̇0,v0〉〈vν , v̇ν〉〈v0,vν〉 ( ‖v0‖62‖vν‖62 − 3 )] −n9E[(v̇0)1(v0)1]E[(v̇ν)1(vν)1]E[(vν)1(v0)1]E [ σ(g11) 2 ]6
+3n3E[(v̇0)1(v0)1]E[(v̇ν)1(vν)1]E[(vν)1(v0)1] ∣∣∣∣∣∣∣ . n−1. As in the previous section, using that E[〈vν , v̇ν〉] = 0 then allows us to conclude the desired control on Ξ3. Thus, in total, we have shown
|Ξ3| ≤ C
n + C ′e−cn + C ′′n1/2e−c ′n,
where we combine the different constant that appear in the various exponential additive errors throughout our work by choosing the largest magnitude scaling factor and the smallest magnitude constant in the exponent to assert the previous expression.
To wrap up, we take the largest of the scaling constants in the estimates we have derived, and the smallest of the constants-in-the-exponent that we have derived, in order to assert
|h′′(ν)| ≤ C n + C ′n1/2e−cn.
Matching constants in the exponent and choosing n larger than an absolute constant multiple of log n, it follows
|h(ν)| ≤ Ce−cn + C ′ ν 2
n ,
which was to be proved.

heading: E.3.4 GENERAL PROPERTIES

text:
Lemma E.16. Consider the event Ec,m = ⋂ S⊂[n] |S|=m ⋂ ν∈[0,2π] { (g1, g2) ∣∣ c ≤ ‖IScvν(g1, g2)‖2 ≤ c−1}. Suppose n ≥ max{2m,m+ 20}. Then we have the following properties:
1. µ(Ecc,m) ≤ Cnme−c ′n;
2. We have Ec,m = Ec,mQ for every Q ∈ O(2), so that in particular 1Ec,m(GQ) = 1Ec,m(G).
Above, O(n) denotes the set of n× n orthogonal matrices.
Proof. We will show the second property first. For each c > 0, ifQ ∈ O(2), notice that Ec,mQ = ⋂ S⊂[n] |S|=m ⋂ ν∈[0,2π] { GQ ∣∣∣∣ c < ∥∥∥∥IScσ(G [cos νsin ν ])∥∥∥∥ 2 < c−1 }
= ⋂ S⊂[n] |S|=m ⋂ u∈S1 { G ∣∣ c < ‖IScσ (GQ∗u)‖2 < c−1}
= Ec,m, since the vector [cos ν, sin ν]∗ ∈ S1, and O(2) acts transitively on S1. This proves the second property when c > 0; the result for c = 0 is obtained by applying the preceding argument to each set in the infinite union defining the c = 0,m event.
For the measure bound, we observe that Ec,m ⊂ Ec′,m if c ≥ c′, so it suffices to bound the measure of the complement for the particular choice c = 12 . We start by controlling pointwise the measure of the complement of the event
E0.6,m,u = ⋂ S⊂[n] |S|=m {G | 0.6 < ‖IScσ (Gu)‖2 < 5/3}
for each u ∈ S1, then uniformize over the one-dimensional manifold S1; we need to begin with c = 0.6 instead of c = 12 to survive some loosening of the bounds when we uniformize. We have
Ec0.6,m,u = ⋃ S⊂[n] |S|=m {G | ‖IScσ (Gu)‖2 ≤ 0.6} ∪ {G | ‖IScσ (Gu)‖2 ≥ 5/3},
so that a union bound implies µ ( Ec0.6,m,u ) ≤ ∑ S⊂[n] |S|=m P[‖IScσ (Gu)‖2 ≤ 0.6] + P[‖IScσ (Gu)‖2 ≥ 5/3]
≤ ( n
m
)( P [∥∥I[m]cσ (g1)∥∥2 ≤ 0.6]+ P[∥∥I[m]cσ (g1)∥∥2 ≥ 5/3]) , (E.37)
where the final inequality follows from right-rotational invariance of µ and identicallydistributedness of the coordinates of g1. Let g̃ ∈ Rn−m be distributed as N (0, (2/n)I), so that σ(g̃) has the same distribution as I[m]cσ (g1) By Gauss-Lipschitz concentration (Boucheron et al., 2013, Theorem 5.6), we have
P[‖σ(g̃)‖2 ≥ E[‖σ(g̃)‖2] + t] ≤ e−cnt 2 , P[‖σ(g̃)‖2 ≤ E[‖σ(g̃)‖2]− t] ≤ e−cnt 2 ,
since σ is 1-Lipschitz and nonnegative homogeneous. After rescaling, we apply Lemma E.19 to get√ 1− m
n − 2√ n √ n−m
≤ E[‖σ(g̃)‖2] ≤ √
1− m n ≤ 1
Plugging these estimates into the Gauss-Lipschitz bounds gives
P[‖σ(g̃)‖2 ≥ 1 + t] ≤ e−cnt 2 , P [ ‖σ(g̃)‖2 ≤ √ 1− m
n − 2√ n √ n−m
− t ] ≤ e−cnt 2 .
Putting t = 2/3 in the upper tail bound gives the control we need for one half of (E.37). For the lower tail, we note that the assumption n ≥ max{2m,m+ 20} yields the estimates√
1− m n ≥ 1√ 2 ,
2√ n √ n−m ≤ 2 n−m ≤ 1 10 ,
so that √ 1− m
n − 2√ n √ n−m − t ≥ 1√ 2 − 1 10 − t,
and one checks numerically that 2−1/2− (1/10) > 0.6. Putting therefore t = 2−1/2− (1/10)− 0.6 in the lower tail bound yields P[‖σ(g̃)‖2 ≤ 0.6] ≤ e−cn. Plugging these results into (E.37) gives the pointwise measure bound
µ ( Ec0.6,m,u ) ≤ 2 ( n
m
) e−cn
for some constant c > 0.
For uniformization, fix S ⊂ [n] with |S| = m and consider the function fS : R2 → R defined by fS(u) = ‖IScσ (Gu)‖2.
By Gauss-Lipschitz concentration, we have
P[‖G‖ > E[‖G‖] + t] ≤ e−cnt 2 ,
and by (Rudelson & Vershynin, 2011, Theorem 2.6), we have
E[‖G‖] ≤ √
2 + 2√ n ≤ 4.
Let E = {‖G‖ ≤ 5}; then it follows that µ(E) ≥ 1 − e−cn. On E , for every S, we have that fS is a 5-Lipschitz function of u. Let Tε ⊂ S1 be a family of sets with the property that u ∈ S1 implies that there is u′ ∈ Tε such that ‖u′ − u‖2 ≤ ε for each ε > 0; by standard results (Vershynin, 2018, Corollary 4.2.13), Tε exists and we have |Tε| ≤ (1 + 2ε−1)2. Define
E0.6,m,ε = ⋂ u∈Tε E0.6,m,u.
Then a union bound together with our pointwise concentration result gives
µ ( Ec0.6,m,ε ) ≤ 2 ( n
m
)( 1 + 2
ε
)2 e−cn.
On E ∩ E0.6,m,ε, for any u ∈ S1 and any S, there is u′ ∈ Tε such that |fS(u)− fS(u′)| ≤ 5ε. But since on this event 0.6 ≤ fS(u′) ≤ 5/3, we conclude 0.6− 5ε ≤ fS(u) ≤ 5/3 + 5ε, and therefore the choice ε = 1/50 gives 0.5 ≤ fS(u) ≤ 2. This implies
E ∩ E0.6,m,1/50 ⊂ E0.5,m.
Thus, by a union bound and our previous results, we have µ ( Ec0.5,m ) ≤ µ ( Ec ∪ Ec0.6,m,1/50 ) ≤ µ ( Ec0.6,m,1/50 ) + e−cn
≤ 2 · 1502 ( n
m
) e−c ′n + e−cn,
which is the desired measure bound.
Lemma E.17. We have for each fixed ν ∈ [0, π] that:
1. The coordinates of v̇ν have subgaussian moment growth
E[(vν)pi ] ≤ 1
2
( 2p
n
)p/2 ;
2. The event {‖v̇ν‖2 ≤ 2} has probability at least 1− e−cn;
3. The event {∀ν ∈ [0, π] ‖v̇ν‖2 ≤ 4} has probability at least 1− e−c ′n.
Proof. We have that the coordinates of v̇ν are i.i.d., and
(v̇ν)i d = σ̇(g1i)g2i,
by rotational invariance. By independence of g1 and g2, we compute
E[(v̇ν)pi ] = E[σ̇(g1i)g p 2i] =
1 2 E[gp2i] ≤
2p/2
2np/2 pp/2,
for each p ≥ 1; the last inequality follows from Lemma G.11. This shows that the coordinates of v̇ν are independent subgaussian random variables with scale parameters at most C √ 2/n, so we have a tail bound (Vershynin, 2018, Theorem 3.1.1)
P [ ‖v̇ν‖2 ≥ 1 + t ] ≤ e−cnt 2 ,
also taking into account that E [ (v̇ν) 2 i ] = 1/n. This shows that the event E ′′ = {‖v̇ν‖2 ≤ 2} has probability at least 1− e−cn. For the third assertion, we use the triangle inequality to get ‖v̇ν‖2 ≤ ‖g2‖2 + ‖g2‖2, which has RHS independent of ν; then applying Gauss-Lipschitz concentration gives for t ≥ 0
P [ ‖gi‖2 ≥ √ 2 + t ] ≤ e−cnt 2 ,
using that E[‖gi‖2] ≤ √
E[‖gi‖22]. Putting t = 0.5 in this bound and applying a union bound, we conclude that there is an event of probability at least 1 − e−cn on which ‖v̇ν‖2 ≤ 4 uniformly in ν.
Lemma E.18. There exists an absolute constant C > 0 such that if n ≥ C, one has
1− C ′
n − C ′′e−cn ≤ E g1,g2 [‖v0‖2‖vν‖2] ≤ 1,
where c, C ′, C ′′ > 0 are absolute constants.
Proof. For the upper bound, we apply the Schwarz inequality to get
E[‖v0‖2‖vν‖2] ≤ E[‖v0‖22]1/2E[‖vν‖22]1/2 ≤ 1, by rotational invariance and Lemma G.11. For the lower bound, we will truncate and linearize the product using logarithms. Let E = E0.5,0; by Lemma E.16, as long as n ≥ 20 we have µ(Ec) ≤ Ce−cn. Define X = ‖v0‖2‖vν‖21E + 1Ec , so that
X(G) = { ‖v0(G)‖2‖vν(G)‖2 G ∈ E , 1 otherwise.
We calculate
|E[‖v0‖2‖vν‖2]− E[X]| ≤ µ(Ec) + E[1E ]1/2E[‖v0‖42]1/2
≤ Ce−cn + C ′e−c ′n(1 + C ′/n)1/2
using the triangle inequality, the Schwarz inequality, rotational invariance, and Lemmas E.16 and E.29. It follows E[‖v0‖2‖vν‖2] ≥ E[X]− C ′e−cn, so it suffices to prove the lower bound forX instead. Factoring asX = (‖v0‖21E+1Ec)(‖vν‖21E+ 1Ec), we apply concavity of x 7→ log x, Jensen’s inequality, and convexity of x 7→ ex to get
E[X] ≥ exp ( E [ log ( ‖v0‖21E + 1Ec )] + E [ log ( ‖vν‖21E + 1Ec )]) ≥ 1 + E [ log ( ‖v0‖21E + 1Ec )] + E [ log ( ‖vν‖21E + 1Ec
)] ≥ 1 + 2E [ log ( ‖v0‖21E + 1Ec
)] where the last equality is due to rotational invariance. Now write Y = ‖v0‖21E + 1Ec , so that by the definition of E we have Y ≥ 12 . Taylor expansion with Lagrange remainder of the logarithm about E[Y ] ≥ 12 gives
log(Y ) = logE[Y ]− 1 E[Y ] (Y − E[Y ])− 1 2ξ(Y )2 (Y − E[Y ])2
for some ξ(Y ) between E[Y ] and Y . Using Y ≥ 12 and taking expectations on both sides, we get
E[log Y ] ≥ logE[Y ]− 2Var[Y ].
Moreover, we have
|E[Y ]− E[‖v0‖2]| ≤ Ce−cn + E[1Ec‖v0‖2] ≤ Ce−cn + C ′e−c ′n,
by the Schwarz inequality, and this extra exponential error can be rolled into the exponential error accrued via our use of X . In particular, we have
1− 2 n − Ce−cn ≤ E[Y ] ≤ 1 + Ce−cn,
by Lemma E.19. Since n ≥ 20, if we also enforce n ≥ C1 := c−1 log(5C/2) we have 2/n + Ce−cn ≤ 12 ; it follows by concavity of x 7→ log(1− x) that we have a bound
log ( 1− 2
n − Ce−cn
) ≥ −2 log(2) ( 2
n + Ce−cn
) ,
which has the form claimed. It remains to upper bound Var[Y ]; using that Y 2 = ‖v0‖221E + 1Ec , we have
Var[Y ] = E[Y 2]− E[Y ]2 ≤ 1 + Ce−cn − (
1− 2 n − Ce−cn )2 = Ce−cn + 2 ( 2
n + Ce−cn
) − ( 2
n + Ce−cn )2 ≤ 4 n + 3Ce−cn,
which is sufficient to conclude.
Lemma E.19. One has 1− 2
n ≤ E g1,g2 [‖vν‖2] ≤ 1.
Proof. By rotational invariance, it is equivalent to characterize the expectation of ‖σ(g1)‖2. By the Schwarz inequality, we have
E[‖v0‖2] ≤ E [ ‖v0‖22 ]1/2 = 1,
by Lemma G.11. For the lower bound, we apply the Gaussian Poincaré inequality (Boucheron et al., 2013, Theorem 3.20) and the 1-Lipschitz property of g 7→ ‖σ(g)‖2 to get
n 2 E [ (‖v0‖2 − E[‖v0‖2])2 ] ≤ 1,
so that after distributing and applying E[‖v0‖22] = 1, we see that
1− 2 n ≤ E[‖v0‖2]2.
Because n ≥ 2, it follows
E[‖v0‖2] ≥ √
1− 2 n ≥ 1− 2 n ,
where the last bound holds because 1− 2n−1 ≤ 1.
Lemma E.20. If 0 ≤ x, y ≤ 1, we have |cos-1 x− cos-1 y| ≤ √ |x− y|.
Proof. Let 0 ≤ x, y ≤ 1, and assume to begin that x ≤ y. We apply the fundamental theorem of calculus and knowledge of the derivative of cos-1 to get
cos-1 x− cos-1 y = ∫ y x 1√ 1− t2 dt
The integrand is nonnegative, so cos-1 x − cos-1 y ≥ 0. Writing √ 1− t2 = √ 1− t √
1 + t and using x ≥ 0, we get
cos-1 x− cos-1 y ≤ ∫ y x 1√ 1− t dt
= √ 1− x− √ 1− y.
This shows that |cos-1 x−cos-1 y| ≤ | √ 1− x− √
1− y|when x ≤ y. An almost-identical argument establishes the same when y ≤ x, via the inequalities 0 ≥ cos-1 x−cos-1 y ≥ −( √ 1− x− √ 1− y). So we have shown |cos-1 x− cos-1 y| ≤ | √ 1− x− √ 1− y|
for arbitrary 0 ≤ x ≤ 1 and 0 ≤ y ≤ 1. Now notice
| √ 1− x− √ 1− y|2 ≤ | √ 1− x− √ 1− y|| √ 1− x+ √
1− y| ≤ |(1− x)− (1− y)| = |x− y|,
which establishes |cos-1 x− cos-1 y| ≤ √ |x− y|.

heading: E.3.5 DIFFERENTIATION RESULTS

text:
Lemma E.21. For a < b, let f : [a, b]→ R be a continuous function that is differentiable on (a, b) except at a set of isolated points in (a, b), and let c ∈ R. Then max{f, c} is differentiable except at a set of isolated points in (a, b).
Proof. Let A ⊂ (a, b) denote the set of points of differentiability of f , and let B ⊂ (a, b) denote the set of points of nondifferentiability of max{f, c}. Because finite unions of isolated sets of points in (a, b) are isolated in (a, b), it suffices to consider only points x ∈ A. Fix x ∈ A, and consider the case f(x) 6= c. Then because f is continuous, there is a neighborhood of x on which f 6= c. If f > c on this neighborhood, then we have max{f, c} = f on this neighborhood; if f < c, then we have max{f, c} = c. In either case, this implies that max{f, c} is differentiable at x, and thus x is not in B.
Next, consider the case where f(x) = c. First, suppose f ′(x) > 0; then by Rolle’s theorem, we can find a neighborhood of x on which f(x′) > c if x′ > x and f(x′) < c if x′ < x. Possibly shrinking this neighborhood, we can assume every point of the neighborhood is a point of differentiability of
f . Thus, for x′ < x in this neighborhood, we have max{f(x′), c} = f(x′), and for x′ > x, we have max{f(x′), c} = c. We conclude that max{f, c} is differentiable at all points of this neighborhood except x, and in particular x is an isolated point in B. A symmetric argument treats the case where f ′(x) < 0, with the same conclusion.
On the other hand, if f ′(x) = 0, we can write f(x′) = c + o(|x′ − x|) for x′ in a neighborhood of x, which implies max{f(x′), c} = max{c, c + o(|x′ − x|)} = c ± o(|x′ − x|). In particular, |max{f(x′), c} − max{f(x), c}| = o(|x′ − x|), which shows that max{f, c} is differentiable at x, and thus x is not in B. This shows that every point of A ∩ B is isolated in A ∩ B, and we can therefore conclude that max{f, c} is differentiable except at isolated points of (a, b).
Lemma E.22. For 0 ≤ ν ≤ π, consider the function
ϕ̃(ν) = E g1,g2∼i.i.d.N (0,(2/n)I) [1E1φ(ν, g1, g2)],
where
φ(ν, g1, g2) = cos -1 ( 〈v0,vν〉 ‖v0‖2‖vν‖2 ) .
Then ϕ̃ is absolutely continuous on [0, π], and satisfies the first-order Taylor expansion
ϕ̃(ν) = ϕ̃(0)− ∫ ν
0 E g1,g2
1E1 〈 v0 ‖v0‖2 , 1 ‖vt‖2 ( I − vtv ∗ t ‖vt‖22 ) v̇t 〉 √
1− 〈
v0 ‖v0‖2 , vt ‖vt‖2
〉2 dt,
and moreover ϕ̃ is 1-Lipschitz.
Proof. At points of (0, π) where each of the functions composed in φ is differentiable, the chain rule gives for the derivative of the integrand as a function of ν
φ′(ν, g1, g2) = −
〈 v0 ‖v0‖2 , 1 ‖vν‖2 ( I − vνv ∗ ν
‖vν‖22
) v̇ν 〉 √
1− 〈
v0 ‖v0‖2 , vν ‖vν‖2 〉2 , (E.38) where we have used the result
d ( · ‖ · ‖ ) x = 1 ‖x‖2 ( I − xx ∗ ‖x‖22 ) ,
valid for any x 6= 0. Because E1 guarantees that vν 6= 0 for all ν ∈ [0, π], we see that the integrand φ is continuous. Similarly, given that ‖vν‖2 ≥ 12 on E1, we note that there are just two obstructions to differentiability:
1. The inverse cosine is not differentiable at {±1};
2. The activation σ is not differentiable at 0.
First we characterize the issue of nondifferentiability with regards to the inverse cosine. We note that cosφ(ν, g1, g2) = 1 if and only if the Cauchy-Schwarz inequality is tight, which is equivalent to v0 and vν being linearly dependent. Suppose we have (g1, g2) ∈ E1 and ν0 ∈ (0, π) such that v0(g1, g2) and vν0(g1, g2) are linearly dependent. Because two vectors u1,u2 ∈ Rn have σ(u1) and σ(u2) linearly dependent if and only if σ(u1) and σ(u2) have the same support and are linearly dependent on the support, and given that ‖vν‖0 > 1 for each ν, we have that there is a 2 × 2 submatrix of GMν0 having positive entries and rank 1 (since the rank is zero if and only if the submatrix is zero), where
Mν = [ 1 cos ν 0 sin ν ] .
Write the corresponding 2 × 2 submatrix of G as X . Because rankMν0 = 2 by ν0 ∈ (0, π), we have rankX = 1. On the other hand, ifG ∼i.i.d. N (0, 2/n), we have
P[G has a singular 2× 2 minor] ≤ ∑
1≤i<j≤n
P [ rank ([ G1i G2i G1j G2j ]) < 2 ] = 0,
where the first line is a union bound, and the second line uses the fact that 2 × 2 submatrices of G are i.i.d. N (0, 2/n), and that the complement of the set of full-rank 2 × 2 matrices is a positivecodimensional closed embedded submanifold of R2×2. It follows that the subset of E1 of matrices having no singular 2× 2 minor has full measure in E1, and we conclude that for almost all (g1, g2), we have cosφ(ν, g1, g2) < 1 for every ν ∈ (0, π). Next, we characterize nondifferentiability due to the activation σ; by the chain rule, it suffices to consider nondifferentiability of vν as a function of ν, and then Lemma E.21 implies that for every (g1, g2), vν is differentiable at all but at most countably many points of [0, π]. Next, we observe that whenever vν is nonvanishing, one has
∣∣∣∣〈 v0‖v0‖2 , 1‖vν‖2 ( I − vνv ∗ ν ‖vν‖22 ) v̇ν 〉∣∣∣∣ ≤ ‖P⊥vν v̇ν‖2‖vν‖2 ∥∥∥∥(I − vνv∗ν‖vν‖22 ) v0 ‖v0‖2 ∥∥∥∥ 2
= ‖P⊥vν v̇ν‖2 ‖vν‖2
√ 1− 〈 v0 ‖v0‖2 , vν ‖vν‖2 〉2 ,
where the first inequality is due squaring the orthogonal projection and Cauchy-Schwarz, and the second equality follows from distributing to evaluate the squared norm, cancelling, and taking square roots. Using the fact that orthogonal projections have operator norm 1, we thus conclude
|φ′(ν, g1, g2)| ≤ ‖P⊥vν v̇ν‖2 ‖vν‖2 ≤ C‖v̇ν‖2, (E.39)
where the last inequality is valid whenever (g1, g2) ∈ E1. Since
‖v̇ν‖2 = ‖σ̇(g1 cos ν + g2 sin ν) (g2 cos ν − g1 sin ν)‖2 ≤ ‖g2 cos ν − g1 sin ν‖2 ≤ ‖g2‖+ ‖g1‖2,
and this upper bound is jointly integrable in ν and (g1, g2) over [0, π] × Rn × Rn, we can apply (Cohn, 2013, Theorem 6.3.11) to obtain that whenever (g1, g2) ∈ E1 minus a negligible set, we have for every ν ∈ [0, π]
φ(ν, g1, g2) = φ(0, g1, g2) + ∫ ν 0 φ′(t, g1, g2) dt.
In particular, multiplying by the indicator for E1, taking expectations over (g1, g2), and applying the previous joint integrability assertion for φ′ together with Fubini’s theorem yields
ϕ̃(ν) = ϕ̃(0) + ∫ ν 0 E g1,g2 [φ′(t, g1, g2)] dt,
so to conclude the Lipschitz estimate, it suffices to obtain a suitable estimate on Eg1,g2 [φ′(ν, g1, g2)]. In light of (E.39) we calculate more precisely
E [ 1E1 ‖P⊥vν v̇ν‖2 ‖vν‖2 ] = E [ 1E1 ‖P⊥v0 v̇0‖2 ‖v0‖2 ]
= E 1E1 ∥∥∥(I − σ(g1)σ(g1)∗‖σ(g1)‖22 ) (σ̇(g1) g2)∥∥∥2
‖σ(g1)‖2  ≤ E 1‖σ(g1)‖0>1 ∥∥∥(I − σ(g1)σ(g1)∗‖σ(g1)‖22 ) (σ̇(g1) g2)∥∥∥2
‖σ(g1)‖2  =
n∑ k=2 2−n ( n k ) E
 ∥∥∥(I − σ(g1)σ(g1)∗‖σ(g1)‖22 ) (σ̇(g1) g2)∥∥∥2
‖σ(g1)‖2
∣∣∣∣∣∣ ‖σ(g1)‖0 = k
= n∑ k=2 2−n ( n k ) E X∼χ(k−1) [X] E Y∼χ(k) [ 1 Y ] .
In the first line, we apply rotational invariance and unpack notation; in the second line, we use nonnegativity of the integrand to pass to the containing event where v0 is at least 2-sparse; and in the third line, we condition on the size of the support of g1. In the fourth line, we use several facts; first, we note that P⊥v0(σ̇(g1) g2) = P ⊥ v0P{σ(g1)>0}g2 for any g2 ∈ R
n, and that the commutation relation P⊥v0P{σ(g1)>0} = P{σ(g1)>0}P ⊥ v0 implies that the operator P ⊥ v0P{σ(g1)>0} is itself an orthogonal projection, with range equal to the (‖v0‖0−1)-dimensional subspace consisting of vectors with support supp(v0) orthogonal to v0. In particular, σ(g1) and P⊥v0P{v0>0}g2 are independent gaussian vectors, and conditioned on the size of the support of σ(g1) the quantities ‖σ(g1)‖2 and ‖P⊥v0P{v0>0}g2‖2 are distributed as independent chi random variables with (respectively) k and k − 1 degrees of freedom. An application of Lemma G.9 then gives
E [ 1E1 ‖P⊥vν v̇ν‖2 ‖vν‖2 ] ≤ 1, (E.40)
which is sufficient to conclude.
Lemma E.23. The random variable Xν satisfies the following regularity properties:
1. If 0 < ν ≤ π, we have Xν < 1 almost surely.
2. If (g1, g2) ∈ E1, then Xν is absolutely continuous on [0, π], with a.e. derivative
Ẋν = 〈 v0 ‖v0‖2 , 1 ‖vν‖2 ( I − vνv ∗ ν ‖vν‖22 ) v̇ν 〉 ,
and moreover we have Eg1,g2 [|Ẋν |] ≤ 1, so the analogous differentiation result applies to Eg1,g2 [Xν ].
Proof. The first claim is a corollary of the proof of differentiability of the inverse cosine part of ϕ̃ in Lemma E.22 and the observation that Xπ = 0. The second claim is also a direct consequence of the proof of Lemma E.22 and Fubini’s theorem.
Lemma E.24. Consider the function
f(ν) = E g1,g2 [Xν ] = E g1,g2
[ 1E1 〈 v0 ‖v0‖2 , vν ‖vν‖2 〉] .
Then f is continuously differentiable, with derivative
f ′(ν) = E g1,g2
[ 1E1 〈 v0 ‖v0‖2 , 1 ‖vν‖2 ( I − vνv ∗ ν ‖vν‖22 ) v̇ν 〉] .
Moreover, f ′ is absolutely continuous, with Lebesgue-a.e. derivative
f ′′(ν) = − E g1,g2
[ 1E1 〈 1
‖vν‖2
( I − vνv ∗ ν
‖vν‖22
) v̇ν , 1
‖v0‖2
( I − v0v ∗ 0
‖v0‖22
) v̇0 〉] Proof. The expression for f ′ is a direct consequence of Lemma E.23. To see that f ′ is actually continuous, apply rotational invariance of the Gaussian measure and of 1E1 by Lemma E.16 to get
f ′(ν) = − E g1,g2
[ 1E1 〈 vν ‖vν‖2 , 1 ‖v0‖2 ( I − v0v ∗ 0 ‖v0‖22 ) v̇0 〉] ,
then notice that this expression is an integral of a continuous function of ν, which is therefore continuous. Moreover, the ν dependence in this expression for f ′ mirrors exactly that of f ; in particular, the integrand
− 〈
vν ‖vν‖2 , 1 ‖v0‖2
( I − v0v ∗ 0
‖v0‖22
) v̇0 〉 is absolutely continuous whenever (g1, g2) ∈ E1 by Lemma E.23, with a.e. derivative
− 〈 1
‖vν‖2
( I − vνv ∗ ν
‖vν‖22
) v̇ν , 1
‖v0‖2
( I − v0v ∗ 0
‖v0‖22
) v̇0 〉 .
We can therefore conclude the claimed expression for f ′′ provided we can show absolute integrability over E1 of this last expression, using Fubini’s theorem in a way analogous to the argument in Lemma E.22. But
E g1,g2
[ 1E1 ∣∣∣∣〈 v̇ν‖vν‖2 ( I − vνv ∗ ν ‖vν‖22 ) , v̇0 ‖v0‖2 ( I − v0v ∗ 0 ‖v0‖22 )〉∣∣∣∣] ≤ 4 Eg1,g2[1E1∥∥P⊥vν v̇ν∥∥2∥∥P⊥v0 v̇0∥∥2] ≤ 4E [ ‖v̇0‖22 ] = 4,
using, in sequence, Cauchy-Schwarz and the lower bound in the definition of E1; the operator norm of orthogonal projections being 1, the Schwarz inequality, nonnegativity of the integrand, and rotational invariance; and Lemma E.17. We can therefore conclude the claimed expression for f ′′ and complete the proof.
Lemma E.25. For the heuristic cosine angle evolution function cosϕ(ν) = E
g1,g2 [〈v0,vν〉],
we have the following integral representations for its continuous derivatives: (cos ◦ϕ)′(ν) = E
g1,g2 [〈v0, v̇ν〉]
(cos ◦ϕ)′′(ν) = − E g1,g2 [〈v̇0, v̇ν〉].
Proof. The proof follows exactly the arguments of Lemma E.24, but with a simpler integrand and different integrability checks; the continuity assertion relies on Lemma E.5. Indeed, this approach gives that 〈v0,vν〉 is absolutely continuous, with Lebesgue-a.e. derivative 〈v0, v̇ν〉; we check
E g1,g2 [|〈v0, v̇ν〉|] ≤ E g1,g2
[ ‖v0‖22 ]1/2 E
g1,g2
[ ‖v̇0‖22 ]1/2 ≤ 1 by Cauchy-Schwarz, the Schwarz inequality, rotational invariance, and Lemma E.17. This verifies the claimed expression for (cos ◦ϕ)′. For the second derivative, we apply rotational invariance to get
(cos ◦ϕ)′(ν) = − E g1,g2 [〈vν , v̇0〉],
which has an absolutely continuous integrand, with Lebesgue-a.e. derivative −〈v̇0, v̇ν〉.
Checking absolute integrability, we have as before
E g1,g2 [|〈v̇0, v̇ν〉|] ≤ E g1,g2
[ ‖v̇0‖22 ] ≤ 1
by Cauchy-Schwarz, the Schwarz inequality, rotational invariance, and Lemma E.17. This establishes the claimed expression for (cos ◦ϕ)′′.
Lemma E.26. Let ψ : R → R be defined by ψ(x) = ψ0.25(x), where ψ0.25 is the function constructed in Lemma E.31. Then the function
f(ν, g1) = E g2
[ 〈v0,vν〉
ψ(‖v0‖2)ψ(‖vν‖2) ] satisfies for all ν ∈ [0, π] and Lebesgue-a.e. g1 the second-order Taylor expansion
f(ν, g1) = ‖v0‖22
ψ(‖v0‖2)2 + ∫ ν 0 ∫ t 0 ( E g2 [ n∑ i=1 σ(g1i) 3ρ(−g1i cot s) ψ(‖v0‖2)ψ(‖vis‖2) sin3 s ]
− E g2
[ 〈v0,vs〉
ψ(‖v0‖2)ψ(‖vs‖2) − 〈v0,vs〉ψ ′(‖vs‖2)‖vs‖2 ψ(‖v0‖2)ψ(‖vs‖2)2 ] − E g2 [ + 〈v0,vs〉〈vs, v̇s〉2ψ′′(‖vs‖2) ψ(‖v0‖2)ψ(‖vs‖2)2‖vs‖22
] + E g2 [ −2 〈v0, v̇s〉〈vs, v̇s〉ψ ′(‖vs‖2) ψ(‖v0‖2)ψ(‖vs‖2)2‖vs‖2 − 〈v0,vs〉‖v̇s‖ 2 2ψ ′(‖vs‖2) ψ(‖v0‖2)ψ(‖vs‖2)2‖vs‖2
] + E g2 [ 2 〈v0,vs〉〈vs, v̇s〉2ψ′(‖vs‖2) ψ(‖v0‖2)ψ(‖vs‖2)3‖vs‖22 + 〈v0,vs〉〈vs, v̇s〉2ψ′(‖vs‖2) ψ(‖v0‖2)ψ(‖vs‖2)2‖vs‖32 ]) ds.
where previously-unspecified notation in this expression is introduced in (E.44).
Proof. Take g1 ∈ Rn such that f(ν, · ) exists and is g1-integrable; by Fubini’s theorem such g1 have full measure in Rn. Because ψ > 0 and ψ(‖vν‖) is locally (as a function of ν) constant whenever ‖vν‖ < 14 , we need only consider nondifferentiability of σ when assessing differentiability of f( · , g1). By Lemma E.21, we conclude that f( · , g1) is differentiable at all but at most countably many points of (0, π); since ψ > 0 and ψ is smooth, f is continuous, and we can therefore apply Lebesgue differentiation theorems (Cohn, 2013, Theorem 6.3.11) to f provided we satisfy the standard derivative product integrability checks. Writing
φ(ν, g1, g2) = 〈v0,vν〉
ψ(‖v0‖2)ψ(‖vν‖2) ,
the chain rule gives (at points of differentiability)
φ′(ν, g1, g2) =
〈 v0
ψ(‖v0‖2)ψ(‖vν‖2) , v̇ν
〉 − 〈 〈v0,vν〉ψ′(‖vν‖2)vν
ψ(‖v0‖2)ψ(‖vν‖2)2‖vν‖2 , v̇ν
〉 .
In this expression, we follow the convention 0/0 = 0 to account for the possibility that ‖vν‖2 = 0 (in this case, the ψ′ term handles the denominator). For product integrability, we Lemma E.31 to get |ψ′| ≤ C for some absolute constant C > 0 together with Cauchy-Schwarz and the triangle inequality to get
|φ′(ν, g1, g2)| ≤ 16‖v0‖2‖v̇ν‖2 + 64C‖v0‖2‖vν‖2‖v̇ν‖2,
and applying the Schwarz inequality, rotational invariance (to eliminate ν dependence in the resulting expectations) and Lemma E.17, we conclude that φ′ is jointly absolutely integrable over [0, π]× (Rn×2, µ⊗ µ). We have therefore a first-order Taylor expansion
f(ν, g1) = f(0, g1)
+ ∫ ν 0 Eg2 [〈 v0 ψ(‖v0‖2)ψ(‖vt‖2) , v̇t 〉] ︸ ︷︷ ︸
Ξ1(ν)
− E g2
[〈 〈v0,vt〉ψ′(‖vt‖2)vt
ψ(‖v0‖2)ψ(‖vt‖2)2‖vt‖2 , v̇t 〉] ︸ ︷︷ ︸
Ξ2(ν) dt. We have
f(0, g1) = E g2
[ ‖v0‖22
ψ(‖v0‖2)2
] =
‖v0‖22 ψ(‖v0‖2)2 ,
since v0 depends only on g1. Next, we show t-differentiability of the inner expectation. Our aim is to apply Lemma E.27 to differentiate Ξ1 and Ξ2. We first focus on Ξ1; distributing and applying linearity, we have
Ξ1(ν) = n∑ i=1 E g2 [ σ(g1i)(g2i cos ν − g1i sin ν) ψ(‖v0‖2)ψ(‖vν‖2) σ̇(g1i cos ν + g2i sin ν) ] .
We have shown absolute integrability of the quantity inside the expectation above; we can therefore apply Fubini’s theorem and the previous definition to write
Ξ1(ν) = n∑ i=1 E (g2j):j 6=i [ E g2i [ σ(g1i)(g2i cos ν − g1i sin ν) ψ(‖v0(g1, g2)‖2)ψ(‖vν(g1, g2)‖2) σ̇(g1i cos ν + g2i sin ν) ]] . (E.41)
For each i ∈ [n], write πi : Rn → Rn−1 for the linear map that deletes the i-th coordinate from its input, and let π̂i : R×Rn−1 → Rn be the linear map such that π̂i(gi, πi(g)) = g. With g2 fixed (in the context of (E.41)), if we define
f1(ν, g) = σ(g1i)(g cos ν − g1i sin ν)
ψ(‖v0(g1, π̂i(g, πi(g2)))‖2)ψ(‖vν(g1, π̂i(g, πi(g2)))‖2) ,
then we can write
Ξ1(ν) = n∑ i=1 E (g2j):j 6=i [ E g2i [f1(ν, g2i)σ̇(g1i cos ν + g2i sin ν)] ] .
Thus, to differentiate Ξ1, it suffices to check the regularity of f1(ν, g) and apply Lemma E.27. As before, ψ > 0 and ψ smooth implies that f1 is continuous on [0, π] × R. For integrability of f , we appeal to the Fubini’s theorem justification that we applied previously. For absolute continuity, we apply Lemma E.21 to get that the derivative of f with respect to ν is, by the chain rule,
f ′1(ν, g) = −σ(g1i) ( g1i cos ν + g sin ν
ψ(‖v0‖2)ψ(‖vν‖2) + (g cos ν − g1i sin ν)ψ′(‖vν‖2)〈vν , v̇ν〉 ψ(‖v0‖2)ψ(‖vν‖2)2‖vν‖2 ) at all but at most countably many values of ν; and the triangle inequality, Cauchy-Schwarz, and Lemma E.31 yield
|f ′1(ν, g)| ≤ σ(g1i) (16(|g1i|+ |g|) + 64C(|g|+ |g1i|)‖v̇ν‖2) ≤ σ(g1i)(|g|+ |g1i|) (16 + 64C(‖g1‖2 + ‖g2‖2)) ≤ σ(g1i)(|g|+ |g1i|) (16 + 64C(‖g1‖2 + ‖πi(g2)‖2 + |g|)) , (E.42)
(we apply square root subadditivity in the last line) which is jointly integrable over [0, π] × R, and moreover over [0, π] × Rn. We conclude absolute continuity of f1( · , g) and the integrability property of f ′1. Finally, for the growth estimate, we obtain an estimate for f1 similar to the one we just obtained for f ′1 as follows:
|f1(ν, g)| ≤ 16|g1i|(|g|+ |g1i|); (E.43)
the RHS of the final inequality above is a linear function of |g|, and when |g| ≥ 1 we can therefore obtain |f1(ν, g)| ≤ 16(|g1i| + |g1i|2)|g|, which is a suitable growth estimate with p = 1. Then as long as g1i 6= 0 for all i (such g1 form a set of measure zero, which we can neglect), we can apply Lemma E.27 to get
Ξ1(ν) = n∑ i=1 E (g2j):j 6=i [ E g2i [f1(0, g2i)σ̇(g1i)] + ∫ ν 0 ( Eg2i [f ′1(t, g2i)σ̇(g1i cos t+ g2i sin t)] −g1i f1(t,−g1i cot t)ρ(−g1i cot t)sin2 t ) dt ] .
The estimates (E.42) and (E.43) show, respectively, that f ′1 and f1 are absolutely integrable functions of (ν, g2). We have
f1 (t,−g1i cot t) = − σ(g1i)
2
ψ(‖v0(g1, g2)‖2)ψ(‖vt(g1, π̂i(−g1i cot t, πi(g2)))‖2) sin t ,
so that Lemma E.31 and nonnegativity give∣∣∣∣g1i f1 (t,−g1i cot t) ρ(−g1i cot t)sin2 t ∣∣∣∣ ≤ 16σ(g1i)3sin3 t ρ(−g1i cot t).
As in the proof of Lemma E.37, in particular using the estimates (E.52) (E.53) to control the magnitude of the RHS for all values of t, we can conclude that the Dirac term is absolutely integrable over [0, π] × Rn. An application of Fubini’s theorem then allows us to re-combine the split integrals in the previous expression:
Ξ1(ν) = n∑ i=1 E g2 [f1(0, g2i)σ̇(g1i)] + ∫ ν 0 ( Eg2 [f ′1(t, g2i)σ̇(g1i cos t+ g2i sin t)] −g1i ρ(−g1i cot t)sin2 t Eg2 [f1 (t,−g1i cot t)] ) dt.
We notice that
vt(g1, π̂i(−g1i cot t, πi(g2))) =
σ(g11 cos ν + g21 sin ν) ...
σ(g1(i−1) cos ν + g2(i−1) sin ν) 0
σ(g1(i+1) cos ν + g2(i+1) sin ν) ...
σ(g1n cos ν + g2n sin ν)
 ,
and thus motivated introduce the notation
g̃i(t, g1, g2) = π̂i(−g1i cot t, πi(g2)); vit(g1, g2) = vt(g1, g̃ i(t, g1, g2)). (E.44)
We can then write
−g1i f1 (t,−g1i cot t) ρ(−g1i cot t)
sin2 t =
σ(g1i) 3ρ(−g1i cot t)
ψ(‖v0‖2)ψ(‖vit‖2) sin3 t .
Finally, we apply linearity of the integral to move the summation over i back inside the integrals, obtaining
Ξ1(ν) = E g2 [ 〈v0, v̇0〉 ψ(‖v0‖2)2 ] +
∫ ν 0 ( E g2 [ n∑ i=1 σ(g1i) 3ρ(−g1i cot t) ψ(‖v0‖2)ψ(‖vit‖2) sin3 t ] − E g2 [( 〈v0,vt〉 ψ(‖v0‖2)ψ(‖vt‖2) + 〈v0,v̇t〉〈vt,v̇t〉ψ ′(‖vt‖2) ψ(‖v0‖2)ψ(‖vt‖2)2‖vt‖2 )]) dt.
Noting that, in the zero-order term, the only g2 dependence is in v̇0 = σ̇(g1) g2, we apply independence of g1 and g2 to obtain finally
Ξ1(ν) = ∫ ν 0 E g2 [ n∑ i=1 σ(g1i) 3ρ(−g1i cot t) ψ(‖v0‖2)ψ(‖vit‖2) sin3 t ] dt
− ∫ ν
0 E g2
[( 〈v0,vt〉
ψ(‖v0‖2)ψ(‖vt‖2) + 〈v0, v̇t〉〈vt, v̇t〉ψ′(‖vt‖2) ψ(‖v0‖2)ψ(‖vt‖2)2‖vt‖2
)] dt
We run the same type of argument on Ξ2 next. Distributing and applying linearity, we have
Ξ2(ν) = n∑ i=1 E g2 [Imσ̇(g1i cos ν + g2i sin ν)A],
where in the previous expression
A = 〈v0,vν〉σ(g1i cos ν + g2i sin ν)(g2i cos ν − g1i sin ν)ψ′(‖vν‖2)
ψ(‖v0‖2)ψ(‖vν‖2)2‖vν‖2 .
By the preceding (product) absolute integrability check when taking first derivatives, we can apply Fubini’s theorem to split the integral as we did with Ξ1. We define, with g2 fixed, the function
f2(ν, g) = B 〈v0(g1),vν(g1, π̂i(g, πi(g2)))〉
ψ(‖v0(g1)‖2)ψ(‖vν(g1, π̂i(g, πi(g2)))‖2)2‖vν(g1, π̂i(g, πi(g2)))‖2
where in the previous expression
B = σ(g1i cos ν + g sin ν)(g cos ν − g1i sin ν)ψ′(‖vν(g1, π̂i(g, πi(g2)))‖2), so that
Ξ2(ν) = n∑ i=1 E (g2j):j 6=i [ E g2i [f2(ν, g2i)σ̇(g1i cos ν + g2i sin ν)] ] .
Now we check that the hypotheses of Lemma E.27 are satisfied for f2. The continuity argument is identical to that employed for f1, as is the joint absolute integrability property of f2. For absolute continuity, we again use ψ > 0, ψ smooth, and Lemma E.21 to obtain the derivative at all but finitely many points of [0, π] (by the chain rule and the Leibniz rule) as
f ′2(ν, g) = 〈v0, v̇ν〉σ(g1i cos ν + g sin ν)(g cos ν − g1i sin ν)ψ′(‖vν‖2)
ψ(‖v0‖2)ψ(‖vν‖2)2‖vν‖2
+ 〈v0,vν〉σ̇(g1i cos ν + g sin ν)(g cos ν − g1i sin ν)2ψ′(‖vν‖2)
ψ(‖v0‖2)ψ(‖vν‖2)2‖vν‖2
− 〈v0,vν〉σ(g1i cos ν + g sin ν)(g1i cos ν + g sin ν)ψ ′(‖vν‖2)
ψ(‖v0‖2)ψ(‖vν‖2)2‖vν‖2
− 2 〈v0,vν〉σ(g1i cos ν + g sin ν)(g cos ν − g1i sin ν)ψ ′(‖vν‖2)〈vν , v̇ν〉
ψ(‖v0‖2)ψ(‖vν‖2)3‖vν‖22
− 〈v0,vν〉σ(g1i cos ν + g sin ν)(g cos ν − g1i sin ν)ψ ′(‖vν‖2)〈vν , v̇ν〉
ψ(‖v0‖2)ψ(‖vν‖2)2‖vν‖32
+ 〈v0,vν〉σ(g1i cos ν + g sin ν)(g cos ν − g1i sin ν)ψ′′(‖vν‖2)〈vν , v̇ν〉
ψ(‖v0‖2)ψ(‖vν‖2)2‖vν‖22 .
Because ψ′ or ψ′′ and our convention handle cancellation in the case where ‖vν‖2 = 0, we can proceed when necessary with the convenient estimate∣∣∣∣ σ(g1i cos ν + g sin ν)‖vν(g1, π̂i(g, πi(g2)))‖2 ∣∣∣∣ ≤ 1, which follows from the fact that ‖u‖∞ ≤ ‖u‖2 for any u ∈ Rn. As with Ξ1, we then estimate the magnitude of f ′2 using Lemma E.31, Cauchy-Schwarz, the triangle inequality, and square-root subadditivity (skipping some steps that we wrote out in the Ξ1 estimate):
|f ′2(ν, g)| ≤ 64C(|g|+ |g1i|)‖v0‖2 ( ‖v̇ν‖2 ( 2 + C ′
C
) + (|g|+ |g1i|) (2 + 8‖v̇ν‖2) ) ≤ 64C(|g|+ |g1i|)‖g1‖2 ( (‖g1‖2 + ‖πi(g2)‖2 + |g|) ( 2 + C ′ C ) +(|g|+ |g1i|) (2 + 8(‖g1‖2 + ‖πi(g2)‖2 + |g|)) ) ,
(E.45)
which is jointly integrable over [0, π]×R, and moreover over [0, π]×Rn. We conclude absolute continuity of f2( · , g) and the integrability property of f ′2. For the growth estimate, we argue similarly to our bound on f ′2 to get
|f2(ν, g)| ≤ 64C‖v0‖2(|g|+ |g1i|)2 ≤ 64C‖g1‖2 ( |g|2 + 2|g1i||g|+ |g1i|2 ) ; (E.46)
the RHS in the final inequality is a quadratic function of |g|, and we therefore obtain a suitable growth estimate with p = 2 and C ′ = 64C‖g1‖(1 + 2|g1i| + |g1i|2) as soon as |g| ≥ 1. We can therefore apply Lemma E.27 to get that for all but a negligible set of g1 that
Ξ2(ν) = n∑ i=1 E (g2j):j 6=i [ E g2i [f2(0, g2i)σ̇(g1i)] + ∫ ν 0 ( Eg2i [f ′2(t, g2i)σ̇(g1i cos t+ g2i sin t)] −g1i f2(t,−g1i cot t)ρ(−g1i cot t)sin2 t ) dt ] .
The estimates (E.45) and (E.46) show, respectively, that f ′2 and f2 are absolutely integrable functions of (ν, g2). Because σ(g1i cos ν − g1i cot ν sin ν) = 0, we have (fortuitously)
f2 (t,−g1i cot t) = 0,
so that there is no Dirac term in the derivative expression for Ξ2. An application of Fubini’s theorem then allows us to re-combine the split integrals in the previous expression:
Ξ2(ν) = n∑ i=1 E g2 [f2(0, g2i)σ̇(g1i)] + ∫ ν 0 ( E g2 [f ′2(t, g2i)σ̇(g1i cos t+ g2i sin t)] ) dt.
We have by linearity of the integral n∑ i=1 E g2 [f2(0, g2i)σ̇(g1i)] = E g2 [ 〈v0, v̇0〉‖v0‖ψ′(‖v0‖) ψ(‖v0‖)3 ] = 0,
where the last equality applied independence of g1 and g2, as in the zero-order term of Ξ1. Finally, we apply linearity of the integral to move the summation over i back inside the remaining integrals, obtaining
Ξ2(ν) = ∫ ν 0 ( E g2 [ 〈v0, v̇t〉〈vt, v̇t〉ψ′(‖vt‖2) ψ(‖v0‖2)ψ(‖vt‖2)2‖vt‖2 + 〈v0,vt〉‖v̇t‖22ψ′(‖vt‖2) ψ(‖v0‖2)ψ(‖vt‖2)2‖vt‖2 ] + E g2 [ −〈v0,vt〉ψ ′(‖vt‖2)‖vt‖2 ψ(‖v0‖2)ψ(‖vt‖2)2 − 2 〈v0,vt〉〈vt, v̇t〉 2ψ′(‖vt‖2) ψ(‖v0‖2)ψ(‖vt‖2)3‖vt‖22
] + E g2 [ −〈v0,vt〉〈vt, v̇t〉 2ψ′(‖vt‖2) ψ(‖v0‖2)ψ(‖vt‖2)2‖vt‖32 + 〈v0,vt〉〈vt, v̇t〉2ψ′′(‖vt‖2) ψ(‖v0‖2)ψ(‖vt‖2)2‖vt‖22 ]) dt.
Since f(ν, g1) = f(0, g1) + ∫ ν 0 Eg2 [Ξ1(t)− Ξ2(t)] dt, the claim follows.
Lemma E.27. Let µ denote the distribution of a N (0, (2/n)) random variable, and let ρ denote its density. Let u ∈ R and u 6= 0, and let f : [0, π]× R→ R satisfy:
1. f is continuous in its second argument with its first argument fixed;
2. f is absolutely continuous in its first argument with its second argument fixed, with a.e. derivative f ′;
3. f and f ′ are absolutely integrable with respect to the product of Lebesgue measure and µ over [0, π]× R;
4. There exist p ≥ 1 and C > 0 constants independent of x such that |f(ν, x)| ≤ C|x|p whenever |x| ≥ 1.
Consider the function
q(ν) = ∫ R f(ν, x)σ̇(u cos ν + x sin ν) dµ(x).
Then q is absolutely continuous, and the following first-order Taylor expansion holds:
q(ν) = q(0) + ∫ ν 0 ( −uf (t,−u cot t) ρ(−u cot t) sin2 t + ∫ R f ′(t, x)σ̇(u cos t+ x sin t) dµ(x) ) dt.
Proof. For m ∈ N, define
σ̇m(x) =  0 x ≤ 0 mx 0 ≤ x ≤ m−1 1 x ≥ m−1.
Then 0 ≤ σ̇m ≤ 1; σ̇m is continuous, hence Borel measurable; σ̇m → σ̇ pointwise as m→∞; and σ̇m is differentiable on R except at x ∈ {0,m−1}, with derivative σ̈m = m10≤x≤m−1 . Moreover, we have ∫
R m10≤x≤m−1 dx = 1,
and the first-order Taylor expansion
σ̇m(x) = ∫ x 0 m10≤x′≤m−1 dx ′.
Define qm(ν) = ∫ R f(ν, x)σ̇m(u cos ν + x sin ν) dµ(x). Then at every ν ∈ [0, π], we have by assumption∫ R |f(ν, x)σ̇m(u cos ν + x sin ν)|dµ(x) ≤ ∫ R |f(ν, x)|dµ(x) < +∞,
so that the dominated convergence theorem implies
lim m→∞ qm(ν) = q(ν).
By the chain rule, the expression σ̇m(x) = −max{−mmax{x, 0},−1}, and Lemma E.21, ν 7→ σ̇m(u cos ν + x sin ν) is an absolutely continuous function of ν ∈ [0, π], and we therefore have by the product rule for AC functions on an interval (Cohn, 2013, Corollary 6.3.9)
qm(ν) = qm(0)
+ ∫ R dµ(x) ∫ ν 0 dt ( f ′(t, x)σ̇m(u cos t+ x sin t)
+mf(t, x)(x cos t− u sin t)10≤u cos ν+x sin ν≤m−1 ) .
We have∫ R ∫ π 0 |f ′(t, x)σ̇m(u cos t+ x sin t)|dtdµ(x) ≤ ∫ R ∫ π 0 |f ′(t, x)|dtdµ(x) < +∞, by assumption, and∫ R
∣∣f(t, x)(x cos t− u sin t)10≤u cos t+x sin t≤m−1∣∣dµ(x) ≤ (∫
R f(t, x)2 dµ(x) )1/2(∫ R (x cos t− u sin t)2 dµ(x) )1/2 ≤ Cf ( |u|+ (∫ R x2 dµ(x) )1/2) < +∞,
by the growth assumption on f and the Schwarz inequality. Applying compactness of [0, π] and the lack of ν dependence in the final inequality above, an application of Fubini’s theorem therefore yields
qm(ν) = qm(0)
+ ∫ ν 0 ∫ R dµ(x) dt ( f ′(t, x)σ̇m(u cos t+ x sin t)
+mf(t, x)(x cos t− u sin t)10≤u cos ν+x sin ν≤m−1 ) .
By dominated convergence and the first of the preceding two product integrability checks, it is clear
lim m→∞ ∫ ν 0 ∫ R f ′(t, x)σ̇m(u cos t+ x sin t) dµ(x) dt = ∫ ν 0 ∫ R f ′(t, x)σ̇(u cos t+ x sin t) dµ(x) dt.
For the second term, we need to proceed more carefully. For k ∈ N sufficiently large for the integral to be over a nonempty interval, we consider
qm,k(ν) := ∫ ν−k−1 k−1 dt ∫ R mf(t, x)(x cos t− u sin t) 1√ 2πc2 e− x2 2c2 10≤u cos t+x sin t≤m−1 dx,
which is a truncated version of the integral constituting the second term in qm, with a change of variables applied to explicitly show the density corresponding to µ, and where we write c2 = 2/n.
In particular, by the calculation used to apply Fubini’s theorem in this context previously, we have by dominated convergence
lim k→∞ qm,k(ν) = ∫ ν 0 ∫ R mf(t, x)(x cos t− u sin t)10≤u cos t+x sin t≤m−1 dµ(x) dt.
By the product integrability assumption on f and Fubini’s theorem, we can consider the inner Rintegral for fixed t, and due to our truncation we have 0 < t < π; we therefore change variables x 7→ x sin−1 t in the inner integral to get
qm,k(ν) = ∫ ν−k−1 k−1 dt ∫ R mf ( t, x sin t )( x cos t sin2 t − u ) 1√ 2πc2 e− x2 2c2 sin2 t10≤u cos t+x≤m−1 dx.
If 0 < t < π and x ∈ R, define g(t, x) = f ( t, x
sin t
)( x cos t sin2 t − u ) 1√ 2πc2 e− x2 2c2 sin2 t ,
so that, after an additional change of variables x 7→ x− u cos t, we obtain
qm,k(ν) = m ∫ ν−k−1 k−1 dt ∫ R g (t, x− u cos t)10≤x≤m−1 dx.
Using the growth estimate for f , we have
|g(t, x− u cos t)| ≤ C |x− u cos t| p|x cos t− u|
sinp+2 t exp
( − (x− u cos t) 2
2c2 sin2 t
) ,
where C > 0 depends only on c. We are going to bound this quantity under the assumption that |x| ≤ |u|/2, where we use the assumption |u| > 0. First, note that when π/4 ≤ t ≤ 3π/4, we have sin t ≥ 1/ √ 2, and we always have sin t ≤ 1 for 0 ≤ t ≤ π; so in this regime
|g(t, x− u cos t)| ≤ C2p/2+1|x− u cos t|p|x cos t− u| exp ( − (x− u cos t) 2
2c2
) ,
which is a continuous function of (t, x), and is therefore bounded by a constant depending only on c, f, u over the compact set [π/4, 3π/4]× [−u/2, u/2]. Next, we consider the case 0 < t ≤ π/4; by the symmetry sin(π− t) = sin t, controlling |g(t, x−u cos t)| in this regime implies control of it in the regime 3π/4 ≤ t < π. Here, we note that by our assumption on t and the triangle inequality
|x− u cos t| ≥ |u||cos t| − |x| ≥ |u|(|cos t| − 12 ) ≥ K|u|,
where we can take K = 2−1/2− 2−1 > 0. Applying the triangle inequality and the condition on |x| gives
|g(t, x− u cos t)| ≤ C(3/2)p+1 |u| p+1
sinp+2 t exp
( − K 2u2
2c2 sin2 t
) ,
which only depends on t. For any constants c′, C ′ > 0, the continuous map y 7→ C|y|p+2e−c′y2 is a bounded function of y ∈ R by L’Hôpital’s rule applied to determine limy→±∞|y|pe−y 2
= 0 for any p > 0. It follows that there is a constant M ≥ 0 depending only on c, u, p such that |g(t, x − u cos t)| ≤ M whenever 0 ≤ tπ/4; we obtain the result for t = 0 by the previous limit calculation. Applying symmetry and taking the sum of our two bounds then yields |g(t, x − u cos t)| ≤ M ′ for M ′ ≥ 0 not depending on k,m whenever (t, x) ∈ [0, π]× [−u/2, u/2]. Now, we have after one additional change of variables x 7→ xm−1
qm,k(ν) = ∫ ν−k−1 k−1 dt ∫ R g ( t, xm−1 − u cos t ) 10≤x≤1 dx.
We can invoke our M ′ bound when xm−1 ≤ |u|/2, and the indicator enforces |x| ≤ 1; thus, taking m ≥ 2/|u| (here we use |u| > 0 critically) implies∫ ν−k−1
k−1 dt ∫ R ∣∣g (t, xm−1 − u cos t)10≤x≤1 dx∣∣ ≤M ′ ∫ ν−k−1 k−1 dt < +∞,
so that by dominated convergence, we have
lim k→∞ qm,k(ν) = ∫ ν 0 dt ∫ R g ( t, xm−1 − u cos t ) 10≤x≤1 dx.
By the same estimate together with second-argument continuity of f , hence of g, we have by the dominated convergence theorem
lim m→∞ lim k→∞ qm,k(ν) = ∫ ν 0 g(t,−u cos t) dt = −u ∫ ν 0 f (t,−u cot t) sin2 t 1√ 2πc2 e− u2 cot2 t 2c2 dt.
Combining with our results on qm and the first term, we conclude
q(ν) = q(0)
+ ∫ ν 0 dt ( −uf (t,−u cot t) sin2 t 1√ 2πc2 e− u2 cot2 t 2c2 + ∫ R f ′(t, x)σ̇(u cos t+ x sin t) dµ(x) ) ,
as claimed.

heading: E.3.6 MISCELLANEOUS ANALYTICAL RESULTS

text:
Lemma E.28. If m > 0, then ϕ̄ is 1-Lipschitz.
Proof. We recall ϕ̄(ν) = E
g1,g2∼i.i.d.N (0,(2/n)I)
[ cos-1Xν ] .
Considering instead the related function ϕ̃ defined by
ϕ̃(ν) = E g1,g2∼i.i.d.N (0,(2/n)I) [1E1φ(ν, g1, g2)],
where
φ(ν, g1, g2) = cos -1 ( 〈v0,vν〉 ‖v0‖2‖vν‖2 ) ,
we notice ϕ̄(ν) = ϕ̃(ν) + (π/2)µ(Ec1).
It is therefore equivalent to show that ϕ̃ is 1-Lipschitz; but this follows from Lemma E.22.
Lemma E.29 (Even Moments). If k ∈ N and k ≤ n, one has∣∣E[‖vν‖2k2 ]− 1∣∣ ≤ Ckn−1, ∣∣E[‖v̇ν‖2k2 ]− 1∣∣ ≤ Ckn−1, where Ck ≤ (k − 1)24k−1(2k − 1)!!. Proof. First notice that the claim is immediate if k = 1, since E [ ‖vν‖22 ] = 1. We therefore proceed assuming k > 1. Also notice that Lemmas G.11 and E.17 show that v̇ν and vν have matching even moments, so it suffices to prove the claim for vν . By rotational invariance, we can write
E [ ‖vν‖2k2 ] = 2k
nk E
gi∼N (0,1) ( n∑ i=1 σ(gi) 2 )k = 2k
nk ∑ 1≤i1,...,ik≤n E  k∏ j=1 σ(gij ) 2 , where the last sum is taken over all elements of [n]k. We split this sum into a sum over terms whose expectations contain no repeated indices, and a sum over all other terms. There are exactly k! ( n k
) ways to choose a k-multi-index from an alphabet of size n without repetitions—select the k distinct
indices, then arrange them in every possible way—and multi-indices without repetitions correspond to terms in the sum where the expectation factors completely, by independence, so we can write
E [ ‖vν‖2k2 ] = 2k
nk k!(nk ) E [ σ(g1) 2 ]k + ∑
1≤i1,...,ik≤n only repeated indices
E  k∏ j=1 σ(gij ) 2   .
We will prove the elementary estimate∣∣∣∣nk − k!(nk )∣∣∣∣ ≤ (k − 1)2nk−12k−2. (E.47)
Assuming it for the time being, we use that E [ σ(g1) 2 ]k
= 2−k to conclude∣∣∣∣(2/n)kk!(nk ) E [ σ(g1) 2 ]k − 1∣∣∣∣ ≤ (k − 1)22k−2n−1.
Next we study the expectation-of-products arising in the sum. The expectation factors over distinct indices; we can classify repeated indices in a multi-index by partitions j1+. . . jm = k, where each jl is a positive integer. Formally, for each multi-index (i1, . . . , ik), there is a partition j1 + . . . jm = k such that
E  k∏ j=1 σ(gij ) 2  = m∏ l=1 E [ σ(gip(l)) 2jl ] ,
where p : [m]→ [k] is injective. We can evaluate these expectations using the result E [ σ(g1) 2k ]
= 1 2 (2k − 1)!!, because the coordinates of g are i.i.d.:
m∏ l=1 E [ σ(gip(l)) 2jl ] = 1 2m m∏ i=1 (2ji − 1)!!.
We claim that 1
2m m∏ i=1 (2ji − 1)!! ≤ 1 2 (2k − 1)!!, (E.48)
which is the expectation obtained from a term with all indices equal, whence∣∣∣∣∣∣∣∣ 2k nk ∑ 1≤i1,...,ik≤n
only repeated indices
E  k∏ j=1 σ(gij ) 2  ∣∣∣∣∣∣∣∣ ≤ 2k nk nk−1(k − 1)22k−2E [ σ(g1) 2k ]
= ( (k − 1)222k−3(2k − 1)!! ) n−1
by (E.47), which gives a bound on the number of terms in the sum. Noticing that this constant is larger than (k − 1)22k−2, we can conclude the claimed estimate on Ck provided we can justify (E.48). For this, it suffices to show
1 ≤ 2m−1 (2k − 1)!!∏m i=1(2ji − 1)!! .
Observe that m ≥ 1 for any partition, so 2m−1 ≥ 1 and we need only study the second term on the righthand side. We write this term as
(2k − 1)!!∏m i=1(2ji − 1)!! =
∏k i=1(2i− 1)∏m
i=1 ∏ji l=1(2l − 1) .
The fact that j1 + · · ·+ jm = k implies that there are k factors in the denominator, so we can put the factors in the numerator and denominator into one-to-one correspondence. Consider the ordering of the factors in the denominator ( ∏j1 l=1(2l − 1)) . . . ( ∏jm l=1(2l − 1)). Then∏k
i=1(2i− 1)∏j1 i=1(2i− 1) = k∏ i=j1+1 (2i− 1).
If j1 = k, then this product is empty and m = 1, so the claim is established. If not, then we proceed to the next group of factors in the denominator: we get∏k
i=j1+1 (2i− 1)∏j2
i=1(2i− 1) ≥ 1,
because j1 > 0 implies that every term in the numerator (ordered in ascending order) is larger than the corresponding term in the denominator. This gives the claim in the case m = 2; for m > 2, we conclude the claim by induction.
To close the loop, we prove (E.47). Using simple algebra, we observe nk − k! ( n
k
) = nk − n(n− 1) . . . (n− k + 1)
= nk 1− k−1∏ j=1 ( 1− j n ) , and we note bounds (
1− k − 1 n )k−1 ≤ k−1∏ j=1 ( 1− j n ) ≤ 1.
Working on the upper bound first, we obtain with the help of the binomial theorem
1− k−1∏ j=1 ( 1− j n ) ≤ 1− ( 1− k − 1 n )k−1
= k−1∑ j=1 ( k − 1 j ) (−1)j+1 ( k − 1 n )j
≤ k − 1 n k−2∑ j=0 ( k − 1 j + 1 )( k − 1 n )j ,
where the last expression removes cancellation by making each term in the sum nonnegative, then applies a change of index. With the identity ( k−1 j+1 ) = (k − 1)/(j + 1) ( k−2 j ) , we proceed as
k − 1 n k−2∑ j=0 ( k − 1 j + 1 )( k − 1 n )j = (k − 1)2 n k−2∑ j=0 ( k − 2 j ) 1 j + 1 ( k − 1 n )j
≤ (k − 1) 2
n
k−2∑ j=0 ( k − 2 j )( k − 1 n )j
= (k − 1)2
n
( 1 +
k − 1 n
)k−2 ,
given that 1/(j + 1) ≤ 1. Since n ≥ k, this gives nk − k! ( n
k
) ≤ (k − 1)2nk−12k−2.
The upper bound on the product gives immediately nk − k! ( n k ) ≥ 0, which completes the proof.
Lemma E.30 (Mixed Moments). Let g1, . . . , gn denote the n (i.i.d. according to N (0, (2/n)I2)) rows of the matrixG. Let k ∈ [n], and for each 1 ≤ j ≤ k let fj : R2 → R be a function such that
1. E[|fj(g1)|p]1/p ≤ Cn−1p, with C > 0 an absolute constant and p ≥ 1; 2. E [ |fj(g1)| ] ≤ n−1.

heading: Consider the quantities

text:
A = E  k∏ j=1 ( n∑ i=1 fj(g i) ); B = nk k∏ j=1 E [ fj(g 1) ] .
Then one has |A−B| ≤ Cn−1, with the constant depending only on k.
Proof. Start by writing
A = ∑
1≤i1,...,ik≤n
E  k∏ j=1 fj(g ij )  = k! ( n
k ) k∏ j=1 E [ fj(g 1) ] + ∑
1≤i1,...,ik≤n only repeated indices
E  k∏ j=1 fj(g ij )
= n−kk!
( n
k
) B + ∑ 1≤i1,...,ik≤n
only repeated indices
E  k∏ j=1 fj(g ij )  as in Lemma E.29. Applying the triangle inequality and the first moment assumption on the functions fj , we get ∣∣∣∣n−kk!(nk ) B −B ∣∣∣∣ = |B|∣∣∣∣k!(nk ) n−k − 1
∣∣∣∣ ≤ (k − 1)22k−2n−1, with the last inequality following from the estimate (E.47). For the remaining term, we have by the triangle inequality∣∣∣∣∣∣∣∣ ∑ 1≤i1,...,ik≤n
only repeated indices
E  k∏ j=1 fj(g ij )  ∣∣∣∣∣∣∣∣ ≤ ∣∣∣∣nk − k!(nk )∣∣∣∣ sup (i1,...,ik)⊂[n]k ∣∣∣∣∣∣E  k∏ j=1 fj(g ij ) ∣∣∣∣∣∣ ≤ (k − 1)2nk−12k−2 sup
(i1,...,ik)⊂[n]k ∣∣∣∣∣∣E  k∏ j=1 fj(g ij ) ∣∣∣∣∣∣, using again (E.47) to control the number of terms in the sum. To control the supremum, we apply the Schwarz inequality k − 1 times to get∣∣∣∣∣∣E  k∏ j=1 fj(g ij ) ∣∣∣∣∣∣ ≤ E[f1(gi1)2]1/2E  k∏ j=2 fj(g ij )2
1/2 ≤ . . .
≤ k−1∏ j=1 E [ fj(g ij )2 j ]2−jE[fk(gik)2k−1]2−(k−1) .
By the subexponential assumption on the functions fj , we have moment growth control, and we therefore have a bound∣∣∣∣∣∣E  k∏ j=1 fj(g ij ) ∣∣∣∣∣∣ ≤ k−1∏ j=1 C1n −12j C1n−12k−1 = Ck1n −k2(k−1)+ ∑k−1 j=1 j = Ck1n −k2 1 2 (k−1)(k+2),
and consequently ∣∣∣∣∣∣∣∣ ∑
1≤i1,...,ik≤n only repeated indices
E  k∏ j=1 fj(g ij )  ∣∣∣∣∣∣∣∣ ≤ C k 1 (k − 1)22 1 2k(k+3)n−1,
which proves the claim.
Lemma E.31. For any 0 < c ≤ 12 , there exists a smooth function ψc : R→ R satisfying
1. ψc(x) = x if x ≥ 2c and ψc(x) = c if x ≤ c, and ψc is between c and 2c if c ≤ x ≤ 2c;
2. ψc(x) ≥ 12x;
3. There are constantsM1,M2 > 0 depending only on c such that |ψ′c| ≤M1 and |ψ′′c | ≤M2.
Proof. The function f(x) = 1x>0e − 1x is smooth on R, and satisfies 0 ≤ f ≤ 1 and f = 0 if x ≤ 0. The function
φc(x) = f(x)
f(x) + f(c− x) is therefore smooth, satisfies 0 ≤ φc ≤ 1, and satisfies φc(x) = 0 if x ≤ 0 and φc(x) = 1 if x ≥ c. Simplifying using the definitions, we can write
φc(x) =  0 x ≤ 0 1 1+exp ( c−2x x(c−x) ) 0 < x < c 1 x ≥ c.
It follows that x 7→ xφc(x) is zero when x ≤ 0, x when x ≥ c, and in between otherwise. Thus, the function ψc(x) = c+ (x− c)φc(x− c) satisfies property 1. For property 2, we note that ψc(x) = c+ (x− c)φc(x− c) implies that ψc ≥ c, since φc(x− c) = 0 whenever x ≤ c and φc ≥ 0. Since ψc(x) = x when x ≥ 2c, we can then conclude ψc(x) ≥ 12x, since 12x ≤ c when x ≤ 2c and 1 2x ≤ x when x ≥ 2c. For property 3, we note that by property 1, ψ′c(x) = 1 if x ≥ 2c and ψ′c(x) = 0 if x ≤ 0; consequently ψ′′c (x) = 0 if x 6∈ [0, 2c], and it suffices to control ψ′c and ψ′′c in this region. By translation equivariance of the derivative, it then suffices to control the derivatives of h(x) = xφc(x) for 0 < x < c. We calculate
h′(x) = xφ′c(x) + φc(x),
h′′(x) = xφ′′c (x) + 2φc(x), (E.49)
and
φ′c(x) = f(c− x)f ′(x)− f(x)f ′(c− x)
(f(x) + f(c− x))2 , (E.50)
φ′′c (x) = (f(x) + f(c− x)) (f(c− x)f ′′(x) + f(x)f ′′(c− x)− 2f ′(x)f ′(c− x))
(f(x) + f(c− x))3
− 2(f ′(x)− f ′(c− x))(f(c− x)f ′(x)− f(x)f ′(c− x))
(f(x) + f(c− x))3 .
(E.51)
Completely ignoring possible cancellation, we see that it suffices to get a lower bound on f(x) + f(c− x) and upper bounds on f ′ and f ′′ to bound |h′| and |h′′|. We calculate
f ′(x) + f ′(c− x) = 1 x2 e− 1 x1x>0 −
1 (c− x)2 e − 1c−x1x<c,
and since f(x) > 0 if x > 0 and c > 0, we see that any solution of f ′(x) − f ′(c − x) = 0 must occur for x ∈ (0, c), which implies as well c − x ∈ (0, c). Writing g(x) = x2e−x and using c−1 < x−1 < ∞ for x ∈ (0, c), we note from our previous work that f ′(x)− f ′(c− x) = 0 ⇐⇒
g(x−1) = g((c− x)−1). We calculate g′(x) = xe−x(2− x), so that if x > 2 then g′(x) < 0, which implies that g is injective on (2,∞). By assumption, we have c−1 > 2; consequently there is at most one solution to f ′(x)− f ′(c− x) = 0 in 0 < x < c, and given that x = 12c is a solution, there is exactly one solution. We check
2f(c/2) < f(0) + f(c) ⇐⇒ log 2 < 1/c, where the first RHS is the value of f(x) + f(c− x) at both x = 0 and x = c, and since 1/c ≥ 2, we conclude that f(x) + f(c− x) ≥ 2f(c/2) > 0. Next, we use
f ′(x) = 1
x2 e− 1 x1x>0,
f ′′(x) =
( 1
x4 e−
1 x − 2
x3 e−
1 x ) 1x>0,
together with the bound xpe−x ≤ ppe−p for p > 0, which is proved by differentiating x 7→ xpe−x, equating to zero, and comparing the values of the function at x = 0, x = p, and x → ∞, to obtain with the triangle inequality |f ′(x)| ≤ 4/e2, |f ′′(x)| ≤ 44e−4 + 2 · 33e−3. Combining these bounds with our lower bound on f(x) + f(c − x) and repeatedly applying the triangle inequality and modulus bounds in (E.50) and (E.51), then subsequently in (E.49) (using also |x| ≤ c), we conclude the claimed bounds on |φ′c| and |φ′′c |.
Lemma E.32. Let Z, Z̄ ∈ L2 be square-integrable random variables. Suppose that Z̄ ≤ C a.s. and ‖Z − Z̄‖L2 ≤M . Then
Var[Z] ≤ Var[Z̄] + CM +M2.
Proof. This is a simple consequence of the triangle inequality and the centering inequality for the L2 norm. We have
‖Z − E[Z]‖L2 ≤ ‖Z − Z̄ − E[Z − Z̄]‖L2 + ‖Z̄ − E[Z̄]‖L2 , and additionally ‖Z − Z̄ − E[Z − Z̄]‖L2 ≤ ‖Z − Z̄‖L2 ≤M, so that, after squaring, we get
Var[Z] ≤ Var[Z̄] +M‖Z̄ − E[Z̄]‖L2 +M2
≤ Var[Z̄] +M‖Z̄‖L2 +M2
≤ Var[Z̄] + CM +M2, by centering and the a.s. boundedness assumption.
Lemma E.33. Let X,Y be square-integrable random variables, and let d > 0. Suppose |X| ≤M1 a.s., and suppose P[|Y − 1| ≥ C √ d/n] ≤ C ′e−cd and ‖Y − 1‖L2 ≤ M2. Then one has with probability at least 1− C ′e−cd
|XY − E[XY ]| ≤ |X − E[X]|+ 2CM1
√ d
n + √ C ′M1M2e −cd/2.
Proof. We apply the triangle inequality: |XY − E[XY ]| ≤ |XY −X|+ |X − E[X]|+ |E[X]− E[XY ]|
≤M1|Y − 1|+M1E[|Y − 1|] + |X − E[X]|, where the second inequality also applies Jensen’s inequality. We have
E[|Y − 1|] = E [( 1|Y−1|≥C √ dn−1 + 1|Y−1|<C √ d/n ) |Y − 1| ] ≤ C √ d
n + E
[ 1|Y−1|≥C √ d/n |Y − 1| ] ≤ C √ d
n + E
[ 1|Y−1|≥C √ d/n ]1/2 E [ (Y − 1)2 ]1/2 ≤ C √ d
n + √ C ′e−cd/2M2,
where we apply the Schwarz inequality in the third line. Consequently, with probability at least 1− C ′e−cd, we have
|XY − E[XY ]| ≤ |X − E[X]|+ 2CM1
√ d
n + √ C ′M1M2e −cd/2,
as claimed.
Lemma E.34. For i = 1, . . . , n, let Xi, Yi be random variables in L4, and let d > 0 and δ > 0. Suppose Xi ≥ 0 for each i and ∑n i=1‖Xi‖L2 ≤ M3, and suppose P[∀i ∈ [n], |Yi − 1| ≥
C √ d/n] ≤ δ and for each i, ‖Yi − 1‖L4 ≤ M2. Moreover, suppose that C √ d/n ≤ 1. Then one has with probability at least 1− δ∣∣∣∣∣ n∑ i=1 XiYi − E[XiYi] ∣∣∣∣∣ ≤ 2 ∣∣∣∣∣ n∑ i=1 Xi − E[Xi] ∣∣∣∣∣+ 2CM3 √ d n + δ1/4M2M3.
Proof. The proof is a minor elaboration on Lemma E.33. We apply the triangle inequality:∣∣∣∣∣ n∑ i=1 XiYi − E[XiYi] ∣∣∣∣∣ ≤ ∣∣∣∣∣ n∑ i=1 XiYi −Xi ∣∣∣∣∣+ ∣∣∣∣∣ n∑ i=1 Xi − E[Xi] ∣∣∣∣∣+ ∣∣∣∣∣ n∑ i=1 E[Xi]− E[XiYi] ∣∣∣∣∣ ≤ C
( n∑ i=1 |Xi| )√ d n + n∑ i=1 E[|Xi||Yi − 1|] + ∣∣∣∣∣ n∑ i=1 Xi − E[Xi] ∣∣∣∣∣, where the second line holds with probability at least 1 − δ. Another application of the triangle inequality together with nonnegativity of the Xi gives
n∑ i=1 |Xi| = ∣∣∣∣∣ n∑ i=1 Xi ∣∣∣∣∣ ≤ ∣∣∣∣∣ n∑ i=1 Xi − E[Xi] ∣∣∣∣∣+ ∣∣∣∣∣ n∑ i=1 E[Xi] ∣∣∣∣∣ ≤M3 +
∣∣∣∣∣ n∑ i=1 Xi − E[Xi] ∣∣∣∣∣, where the second line applies the Lyapunov inequality. By the Schwarz inequality and the Lyapunov inequality, we have
n∑ i=1 E[|Xi||Yi − 1|] ≤ C √ d n n∑ i=1 E[|Xi|] + n∑ i=1 E [ 1|Yi−1|≥C √ d/n |Xi||Yi − 1| ] ≤ CM3 √ d
n + δ1/4M2M3.
Consequently, with probability at least 1− δ, we have∣∣∣∣∣ n∑ i=1 XiYi − E[XiYi] ∣∣∣∣∣ ≤ 2 ∣∣∣∣∣ n∑ i=1 Xi − E[Xi] ∣∣∣∣∣+ 2CM3 √ d n + δ1/4M2M3
as claimed, where we use that C √ d/n ≤ 1 here.
Lemma E.35. Let k ∈ N, and let X1, . . . , Xk be integrable random variables satisfying ‖Xi − E[Xi]‖L4 ≤Mi for some constants Mi > 0. Suppose moreover that with probability at least 1− δi, one has |Xi − E[Xi]| ≤ Ni for some constants Ni > 0. Then one has
Var [ k∑ i=1 Xi ] ≤ k∑ i,j=1 NiNj + √ δi + δjMiMj .
Proof. We start from the formula
Var [ k∑ i=1 Xi ] = n∑ i=1 Var[Xi] + 2 ∑ i<j cov[Xi, Xj ],
where cov[Xi, Xj ] = E[XiXj ] − E[Xi]E[Xj ] = E[(Xi − E[Xi])(Xj − E[Xj ])]; one establishes this formula by distributing in the definition of the variance. By assumption, there are events Ei on which |Xi − E[Xi]| ≤ Ni and such that P[Ei] ≥ 1 − δi. Partitioning the expectation, we therefore have
Var[Xi] = E[(Xi − E[Xi])2] ≤ N2i + E[1Eci (Xi − E[Xi]) 2]
≤ N2i + E[1Eci ] 1/2E[(Xi − E[Xi])4]1/2 ≤ N2i + √ δiM 2 i ,
where the first line uses nonnegativity of the integrand to discard the indicator after applying the deviations bound, the second line applies the Schwarz inequality, and the third line uses fourth moment control. For the covariance terms, we apply Jensen’s inequality to obtain
|cov[Xi, Xj ]| = |E[XiXj ]− E[Xi]E[Xj ]| ≤ E[|Xi − E[Xi]||Xj − E[Xj ]|], so that, again partitioning the outermost expectation and applying our assumptions, we get
|cov[Xi, Xj ]| ≤ N2i + E[1Eci∪Ecj |Xi − E[Xi]||Xj − E[Xj ]|]
≤ N2i + E[1Eci∪Ecj ] 1/2E[(Xi − E[Xi])4]1/4E[(Xj − E[Xj ])4]1/4 ≤ NiNj + √ δi + δjMiMj ,
where in the first line we again use nonnegativity of the integrand to discard the indicator after applying the deviations bound, in the second line we apply the Schwarz inequality twice, and in the third line we use a union bound to control the indicator. Since δi ≤ 2δi, we conclude the claimed expression.
Lemma E.36. If C > 0 and p > 0, the function g(t) = tpe−Ct 2
for t ≥ 0 satisfies the bound g(t) ≤ (p/(2Ce))p/2.
Proof. The function g is smooth has derivatives g′(t) = tp−1e−Ct 2 (p − 2Ct2) and g′′(t) = tp−2e−Ct 2
(p(p − 1) − 2(4p − 1)Ct2 + 4C2t4). It therefore has at most two critical points, one possibly at t = 0 and one at t = √ p/(2C), and these points are distinct when p > 0 and C > 0. We
check the sign of g′′ at the second critical point; since √ p/(2C) > 0 we need only check the value
of (p(p− 1)− 2(4p− 1)Ct2 + 4C2t4) evaluated at t = √ p/(2C), which is −2p2 < 0. Then since
limt→±∞ g(t) = 0 and g(0) = 0, we conclude that g(t) ≤ g( √ p/(2C)), which gives the claimed bound.
Lemma E.37. Following Lemma E.26, consider the random variables
Ξ1(s, g1, g2) = n∑ i=1 σ(g1i) 3ρ(−g1i cot s) ψ(‖v0‖2)ψ(‖vis‖2) sin3 s Ξ2(ν, g1, g2) = 〈v0,vs〉ψ′(‖vs‖2)‖vs‖2 ψ(‖v0‖2)ψ(‖vs‖2)2 − 〈v0,vs〉 ψ(‖v0‖2)ψ(‖vs‖2) Ξ3(ν, g1, g2) = − 〈v0,vs〉〈vs, v̇s〉2ψ′′(‖vs‖2) ψ(‖v0‖2)ψ(‖vs‖2)2‖vs‖22 Ξ4(ν, g1, g2) = −2 〈v0, v̇s〉〈vs, v̇s〉ψ′(‖vs‖2) ψ(‖v0‖2)ψ(‖vs‖2)2‖vs‖2 Ξ5(ν, g1, g2) = − 〈v0,vs〉‖v̇s‖22ψ′(‖vs‖2) ψ(‖v0‖2)ψ(‖vs‖2)2‖vs‖2 Ξ6(ν, g1, g2) = 2 〈v0,vs〉〈vs, v̇s〉2ψ′(‖vs‖2) ψ(‖v0‖2)ψ(‖vs‖2)3‖vs‖22 + 〈v0,vs〉〈vs, v̇s〉2ψ′(‖vs‖2) ψ(‖v0‖2)ψ(‖vs‖2)2‖vs‖32 ,
where Ξ1( · , g1, g2) is defined at {0, π} by continuity (following the proof of Lemma E.27, it is 0 here). Then for each i = 1, . . . , 6, one has:
1. For each i, there is a µ⊗µ-integrable function fi : Rn×Rn → R such that |Ξi( · , g1, g2)− E[Ξi( · , g1, g2)]| ≤ fi(g1, g2);
2. There is an absolute constant Ci > 0 such that for every 0 ≤ ν ≤ π, one has ‖fi‖L4 ≤ Ci, so that in particular ‖Ξi(ν, · , · )− E[Ξi(ν, · , · )]‖L4 ≤ Ci.
Proof. First we reduce to noncentered fourth moment calculations. If X is a random variable with finite fourth moment, we have by Minkowski’s inequality
‖X − E[X]‖L4 ≤ ‖X‖L4 + |E[X]|,
so that the triangle inequality for the expectation and the Lyapunov inequality imply
‖X − E[X]‖L4 ≤ 2‖X‖L4 .
We can therefore control the noncentered fourth moments of the random variables Ξi and pay only an extra factor of 2 in controlling the centered moments. For the proofs of property 1, we have similarly |X − E[X]| ≤ |X|+ E[|X|] from the triangle inequality, so that it again suffices to prove property 1 for the noncentered random variables |Ξi|. Ξ1 control. If ν = 0 or ν = π, the integrand is identically zero; we proceed assuming 0 < ν < π. Using ψ ≥ 14 , we have
0 ≤ Ξ1(ν, g1, g2) ≤ 16 n∑ i=1 σ(g1i) 3ρ(−g1i cot ν) sin3 ν .
For property 1, by elementary properties of cos we have for 0 ≤ ν ≤ π/4 and 3π/4 ≤ ν ≤ π that cos2 ν ≥ 12 , so
ρ(−g1i cot ν) ≤ √ n
4π e−
ng21i 8 sin2 ν .
This gives
σ(g1i) 3ρ(−g1i cot ν) sin3 ν ≤ |g1i| 3ρ(−g1i cot ν) sin3 ν =
√ 2
π K1/2 ∣∣∣ g1i sin ν ∣∣∣3e−K∣∣∣ g1isin ν ∣∣∣2 , where we define K = n/8. By Lemma E.36, we have that g ≤ g( √ 3/2K) = CK−3/2, where C > 0 is an absolute constant. We conclude
σ(g1i) 3ρ(−g1i cot ν) sin3 ν ≤ C/n, (E.52)
provided ν is not in [π/4, 3π/4]. On the other hand, if π/4 ≤ ν ≤ 3π/4, we have sin ν ≥ 1/ √
2, so that
σ(g1i) 3ρ(−g1i cot ν) sin3 ν ≤ C √ nσ(g1i) 3, (E.53)
where C > 0 is an absolute constant. Since these ν constraints cover [0, π], we have for all ν and all g1 (by the triangle inequality)
|Ξ1(ν, g1, g2)| ≤ C + C ′n3/2σ(g1i)3,
where C,C ′ > 0 are absolute constants, and by Lemma G.11, we have
E[C + C ′n3/2σ(g1i)3] = C + C ′′,
whereC ′′ > 0 is an absolute constant. This proves property 1 with f1(g1, g2) = C+C ′n3/2σ(g1i)3, with different absolute constants, and property 2 follows from Lemma G.11 after applying the Minkowski inequality and calculating the integral, which has the necessary cancellation of the n3/2 factor.
Ξ2 control. By Lemma E.31, we have |ψ′| ≤ C for an absolute constant C > 0 and x/ψ(x) ≤ 2. Cauchy-Schwarz then implies ∣∣∣∣ 〈v0,vs〉ψ′(‖vs‖2)‖vs‖2ψ(‖v0‖2)ψ(‖vs‖2)2 ∣∣∣∣ ≤ 8C. In an exactly analogous manner, we have∣∣∣∣ 〈v0,vs〉ψ(‖v0‖2)ψ(‖vs‖2)
∣∣∣∣ ≤ 4. Both bounds satisfy the requirements of property 1, with f2(g1, g2) = 16C + 8. The triangle inequality and Minkowski’s inequality then implies ‖Ξ1(ν, · , · )‖L4 ≤ C ′. Ξ3 control. By Lemma E.31, we have |ψ′′| ≤ C for an absolute constant C > 0, ψ ≥ 14 , and x/ψ(x) ≤ 2. Cauchy-Schwarz then implies∣∣∣∣ 〈v0,vs〉〈vs, v̇s〉2ψ′′(‖vs‖2)ψ(‖v0‖2)ψ(‖vs‖2)2‖vs‖22
∣∣∣∣ ≤ 16C‖v̇s‖22, and the triangle inequality gives ‖v̇s‖22 ≤ ‖g1‖22 + ‖g2‖22 + 2‖g1‖2‖g2‖2, whose expectation is bounded by 4, by the Schwarz inequality and Lemma G.11. We can therefore take f3(g1, g2) = C + C ′(‖g1‖2 + ‖g2‖2)2, and we have∥∥(‖g1‖2 + ‖g2‖2)2∥∥L4 = ‖‖g1‖2 + ‖g2‖2‖2L8 ≤ (‖‖g1‖2‖L8 + ‖‖g2‖2‖L8)2 ≤ C, where C > 0 is a (new) absolute constant, by the Minkowski inequality and lemmas Lemmas G.10 and G.11. This establishes property 2.
Ξ4 control. By Lemma E.31, we have |ψ′| ≤ C for an absolute constant C > 0, ψ ≥ 14 , and x/ψ(x) ≤ 2; Cauchy-Schwarz then implies∣∣∣∣2 〈v0, v̇s〉〈vs, v̇s〉ψ′(‖vs‖2)ψ(‖v0‖2)ψ(‖vs‖2)2‖vs‖2 ∣∣∣∣ ≤ 64C‖v̇s‖22. Following the argument for Ξ3 exactly, we conclude property 1 and 2 from this bound with a suitable modification of the constant. Ξ5 control. We have ∣∣∣∣ 〈v0,vs〉‖v̇s‖22ψ′(‖vs‖2)ψ(‖v0‖2)ψ(‖vs‖2)2‖vs‖2 ∣∣∣∣ ≤ 32C‖v̇s‖22,
following exactly the setup and instantiations in the argument for Ξ4. Following the argument for Ξ3 exactly, we conclude property 1 and 2 from this bound with a suitable modification of the constant.
Ξ6 control. The triangle inequality gives |Ξ6(s, g1, g2)| ≤ 2 ∣∣∣∣ 〈v0,vs〉〈vs, v̇s〉2ψ′(‖vs‖2)ψ(‖v0‖2)ψ(‖vs‖2)3‖vs‖22 ∣∣∣∣+ ∣∣∣∣ 〈v0,vs〉〈vs, v̇s〉2ψ′(‖vs‖2)ψ(‖v0‖2)ψ(‖vs‖2)2‖vs‖32 ∣∣∣∣,
and following the setup of Ξ4 and Ξ5 control gives |Ξ6(ν, g1, g2)| ≤ 128C‖v̇ν‖22 + 32C‖v̇ν‖22. Following the argument for Ξ3 exactly, we conclude property 1 and 2 from this bound with a suitable modification of the constant.
Lemma E.38. In the notation of Lemma E.13, there are absolute constants c, c′, C > 0 and an absolute constant K > 0 such that if n ≥ K , there is an event with probability at least 1 − 2e−cn on which one has ∣∣∣∣ ‖v0‖22ψ(‖v0‖2)2 − E [ ‖v0‖22 ψ(‖v0‖2)2
]∣∣∣∣ ≤ Ce−c′n. Proof. There is no ν dependence in this term, so we need only prove a single bound. Following the proof of the measure bound in Lemma E.16, but using only the pointwise concentration result, we assert that if n ≥ C an absolute constant there is an event E on which 0.5 ≤ ‖v0‖2 ≤ 2 with probability at least 1− 2e−cn with c > 0 an absolute constant. This implies that if g1 ∈ E we have
‖v0‖22 ψ(‖v0‖2)2 = 1,
which we can use together with nonnegativity of the integrand to calculate
E [ ‖v0‖22
ψ(‖v0‖2)2
] = E[1E ] + E [ 1Ec ( ‖v0‖2
ψ(‖v0‖2) )2] ≥ E[1E ] ≥ 1− 2e−cn,
whence ‖v0‖22
ψ(‖v0‖2)2 − E
[ ‖v0‖22
ψ(‖v0‖2)2
] ≤ 2e−cn
whenever g1 ∈ E . Similarly, we calculate
E [ ‖v0‖22
ψ(‖v0‖2)2
] = E[1E ] + E [ 1Ec ( ‖v0‖2
ψ(‖v0‖2)
)2]
≤ 1 + E[1Ec ]1/2E
[( ‖v0‖2
ψ(‖v0‖2) )4]1/2 ≤ 1 + 16Ce−cn,
applying the Schwarz inequality, property 2 in Lemma E.31, and the measure bound on E , with c′, C ′ > 0 absolute constants, whence
E [ ‖v0‖22
ψ(‖v0‖2)2
] − ‖v0‖ 2 2
ψ(‖v0‖2)2 ≤ 16C ′e−c
′n
whenever g1 ∈ E . Worst-casing constants, we conclude∣∣∣∣ ‖v0‖22ψ(‖v0‖2)2 − E [ ‖v0‖22 ψ(‖v0‖2)2 ]∣∣∣∣ ≤ Ce−cn when g1 ∈ E , which is sufficient for our purposes.
Lemma E.39. In the notation of Lemma E.13, if d ≥ 1, there are absolute constants c, c′, c′′, c′′′, C, C ′, C ′′, C ′′′, C ′′′′ > 0 and absolute constants K,K ′ > 0 such that if n ≥ Kd4 log4 n and d ≥ K ′, there is an event with probability at least 1− C ′′′n−c′′d̄/2 − C ′′′′ne−c′′′n on which one has
|Ξ1(ν, g1, g2)− E[Ξ1(ν, g1, g2)]| ≤ C √ d̄ log n
n + C ′n−cd̄ + C ′′ne−c ′n.
Proof. If ν ∈ {0, π}, then Ξ1(ν, g1, g2) = 0 for every (g1, g2); we therefore assume 0 < ν < π below.
We will apply Lemma E.34 to begin, with the instantiations
Xi = σ(g1i) 3ρ(−g1i cot ν) sin3 ν , Yi = 1 ψ(‖v0‖2)ψ(‖viν‖2) ,
since then Ξ1(ν, g1, g2) = ∑ iXiYi. We have Xi ≥ 0; writing k2 = 2/n, we calculate
E[Xi] = 1√ 8πk2 1√ 2πk2 ∫ R g3 sin3 ν exp ( − 1 2k2 g2 sin2 ν ) dg
= 2
πn sin ν (E.54)
where the second line uses the change of variables g 7→ g sin ν and Lemma G.11. Additionally, we have
E[X2i ] = k4
4π 1√ 2π ∫ R g6 sin6 ν exp ( −1 2 g2(1 + 2 cot2 ν) ) dg
= k4 sin ν
4π 1√ 2π ∫ R g6 exp ( −1 2 g2(1 + cos2 ν) ) dg
= k4 sin ν 4π(1 + cos2 ν)7/2 1√ 2π ∫ R g6e−g 2/2 dg
= 15 sin ν
πn2(1 + cos2 ν)7/2 , (E.55)
where in the second line we change variables g 7→ g sin ν, in the third line we change variables g 7→ g/ √ 1 + cos2 ν, and in the fourth line we use Lemma G.11. We can calculate the derivative of the map g(ν) = (1 + cos2 ν)−7/2 sin ν as g′(ν) = cos(ν)(1 + cos2 ν)−7[(1 + cos2 ν)7/2 + 7 sin2(ν)(1 + cos2 ν)5/2], which evidently has the same sign as cos(ν); so g is strictly increasing below π/2 and strictly decreasing above it, and is therefore maximized at g(π/2). We conclude the bound
E[X2i ] ≤ 15
πn2 , (E.56) which shows that ∑ i‖Xi‖L2 = O(1). Next, we have Yi ≤ 16 by Lemma E.31, so by the Minkowski inequality ‖Yi − 1‖L4 ≤ 17 for each i, and it remains to control deviations. We consider the event E = E0.5,1 in the notation of Lemma E.16, which has probability at least 1−Cne−cn and on which we have 12 ≤ ‖v i ν‖2 ≤ 2 for all i ∈ [n] and in particular 12 ≤ ‖v0‖2, and thus by Lemma E.31
Yi = 1
‖v0‖2‖viν‖2 for all i ∈ [n]. By Taylor expansion with Lagrange remainder of the smooth function x 7→ x−1 on the domain x > 0 about the point 1, we have
1 x = 1− (x− 1) + 1 ξ3 (x− 1)2,
where ξ lies between 1 and x. If (g1, g2) ∈ E , then for all i ‖v0‖32‖viν‖32 ≥ (1/64), and we can therefore assert(
‖v0‖2‖viν‖2 − 1 ) − 64 ( ‖v0‖2‖viν‖2 − 1 )2 ≤ 1− Yi ≤ (‖v0‖2‖viν‖2 − 1) . (E.57) By Gauss-Lipschitz concentration, we have P[|‖v0‖2 − E[‖v0‖2]| ≥ t] ≤ 2e−cnt 2
and P[|‖v0‖2 − E[‖viν‖2]| ≥ t] ≤ 2e−cnt 2
. Lemma E.19 implies that 1−2/(n−1) ≤ E[‖viν‖2] ≤ 1 and 1−2/n ≤ E[‖v0‖2] ≤ 1, so we can conclude when n ≥ d and when n is larger than a constant that
|‖v0‖2 − 1| ≤ C √ d
n ; ∀i ∈ [n], ∣∣‖viν‖2 − 1∣∣ ≤ C√ dn with probability at least 1 − C ′ne−d, by a union bound. Using then the fact that ‖viν‖2 ≤ 2 for all i on the event E together with the previous estimates and (E.57), we obtain with probability at least 1− C ′′ne−cn − C ′′′ne−d (via a union bound with the measure of E) that for all i,
C
√ d
n − C ′ d n ≤ 1− Yi ≤ C
√ d
n . As long as n ≥ d, we conclude that with the same probability, for all i we have |Yi− 1| ≤ C √ d/n. We can therefore apply Lemma E.34 to get that with probability at least 1 − C ′′ne−cn − C ′′′ne−d we have
|Ξ1(ν, g1, g2)− E[Ξ1(ν, g1, g2)]| ≤ 2 ∣∣∣∣∣ n∑ i=1 σ(g1i) 3ρ(−g1i cot ν) sin3 ν − E [ σ(g1i) 3ρ(−g1i cot ν) sin3 ν ]∣∣∣∣∣ + C √ d
n + (C ′′)1/4ne−cn/4 + (C ′′′)1/4ne−d/4, (E.58)
where we also used the triangle inequality for the `4 norm to simplify the fourth root term, together with n ≥ 1. For ν ∈ [0, π], we define fν : R→ R by
fν(g) = σ(g)3√
2πk2 sin3 ν exp
( − 1
2k2 g2 cot2 ν
) ,
so that the task that remains is to control | ∑ i fν(g1i) − E[fν(g1i)]|. We start by applying Lemma E.36 to obtain an estimate fν(g) ≤ C
n|cos ν|3 ,
where C > 0 is an absolute constant. When 0 ≤ ν ≤ π/4 or 3π/4 ≤ ν ≤ π, we have therefore fν(g) ≤ C/n. Meanwhile, if π/4 ≤ ν ≤ 3π/4, we have fν(g) ≤ C ′ √ nσ(g)3, so we can conclude
fν(g) ≤ C/n+ C ′ √ nσ(g)3 for all ν, which shows that fν(g) is not much larger than C ′ √ nσ(g)3. Next, let ḡ ∼ N (0, 1), so that g d= kḡ; we have for any t ≥ 0
P [ C ′ √ nσ(g)3 ≥ t ] = P [ σ(ḡ) ≥ C ′′ (nt)1/3 ] ≤ exp ( − 12 (C ′′)2(nt)2/3 ) ,
where we use the classical estimate P[ḡ ≥ t] ≤ e−t2/2, valid for t ≥ 1, and accordingly require t ≥ (C ′′)−3n−1. In particular, there is an absolute constant C ′′ > 0 such that we have
P [ C ′ √ nσ(g)3 ≥ C
′′ √ nd
] = P [ σ(ḡ) ≥ (n d )1/6] ≤ exp ( −1 2 (n d )1/3) ≤ e−d,
where the last inequality holds in particular when n ≥ 8d4 (and this condition implies what is necessary for the second to last to hold when d ≥ 1). Returning to our bound on fν , we note that when n ≥ (C/C ′′)2d, we have that
fν(g)− 2C ′′√ nd ≤ C n + C ′ √ nσ(g)3 − 2C ′′ √ nd ≤ C ′ √ nσ(g)3 − C ′′ √ nd ,
from which we conclude that when our previous hypotheses on n are in force P [ fν(g) ≥
2C ′′√ nd
] ≤ e−d. (E.59)
We are going to use this result to control | ∑ i fν(g1i) − E[fν(g1i)]| using a truncation approach.
Define M = 2C ′′/ √ nd, where C ′′ > 0 is the absolute constant in (E.59). We write using the triangle inequality∣∣∣∣∣ n∑ i=1 fν(g1i)− E[fν(g1i)] ∣∣∣∣∣ ≤ ∣∣∣∣∣ n∑ i=1 fν(g1i)− fν(g1i)1fν(g1i)≤M
∣∣∣∣∣ +
∣∣∣∣∣ n∑ i=1 fν(g1i)1fν(g1i)≤M − E [ fν(g1i)1fν(g1i)≤M ]∣∣∣∣∣ +
∣∣∣∣∣ n∑ i=1 E [ fν(g1i)1fν(g1i)≤M ] − E[fν(g1i)] ∣∣∣∣∣. By (E.59) and a union bound, we have with probability at least 1− ne−d∣∣∣∣∣ n∑ i=1 fν(g1i)− fν(g1i)1fν(g1i)≤M
∣∣∣∣∣ = 0. Moreover, we calculate∣∣∣∣∣ n∑ i=1 E [ fν(g1i)1fν(g1i)≤M ] − E[fν(g1i)] ∣∣∣∣∣ ≤ n∑ i=1 E [ fν(g1i)1fν(g1i)>M
] ≤
n∑ i=1 P[fν(g1i) > M ]1/2‖fν(g1i)‖L2
≤ Ce−d/2
for an absolute constant C > 0, using in the second line the Schwarz inequality, and in the third line (E.56) and (E.59). The second term can be controlled with Lemma G.3, together with the observation that
n∑ i=1 E [( 1fν(g1i)≤Mfν(g1i)− E [ 1fν(g1i)≤Mfν(g1i) ])2] =
n∑ i=1 E [ 1fν(g1i)≤Mfν(g1i) 2 ] − E [ 1fν(g1i)≤Mfν(g1i) ]2 ≤
n∑ i=1 E [ fν(g1i) 2 ]
≤ C/n, where the last inequality is due to (E.56). Lemma G.3 thus gives for any t ≥ 0
P [∣∣∣∣∣ n∑ i=1 fν(g1i)1fν(g1i)≤M − E [ fν(g1i)1fν(g1i)≤M ]∣∣∣∣∣ ≥ t ] ≤ 2 exp ( − t 2/2 Cn−1 +Mt/3 ) .
It follows that there is an absolute constant C ′ > 0 such that
P [∣∣∣∣∣ n∑ i=1 fν(g1i)1fν(g1i)≤M − E [ fν(g1i)1fν(g1i)≤M ]∣∣∣∣∣ ≥ C ′ √ d n ] ≤ 2e−d,
and therefore with probability at least 1− 2ne−d (by a union bound) we have∣∣∣∣∣ n∑ i=1 fν(g1i)− E[fν(g1i)] ∣∣∣∣∣ ≤ C ′ √ d n + C ′′e−d/2.
Combining with (E.58) using a union bound and worst-casing constants in the exponent, we conclude that with probability at least 1− C ′′′ne−c′′d − C ′′′′ne−c′′′n, we have
|Ξ1(ν, g1, g2)− E[Ξ1(ν, g1, g2)]| ≤ C √ d
n + C ′e−cd + C ′′ne−c ′n.
Aggregating our hypotheses on n, there are absolute constants C1, C2, C3 > 0 such that we have to satisfy n ≥ max{Cd,C ′d4, C ′′}. Moreover, to be able to assert ne−c′′d ≤ e−c′′d/2, we have to satisfy d ≥ 2/c′′ log n. Introducing an auxiliary d̄ > 0 and setting d = d̄ log n, we have to satisfy n ≥ max{Cd̄ log n,C ′d̄4 log4 n,C ′′} and d̄ ≥ 2/c′′. Choosing n in this way, we can finally conclude that with probability at least 1− C ′′′n−c′′d̄/2 − C ′′′′ne−c′′′n, we have
|Ξ1(ν, g1, g2)− E[Ξ1(ν, g1, g2)]| ≤ C √ d̄ log n
n + C ′n−cd̄ + C ′′ne−c ′n,
which is the desired type of bound.
Lemma E.40. In the notation of Lemma E.13, there are absolute constants c, C,C ′, C ′′ > 0 such that for any δ ≥ 3/2, we have
P [∣∣∣∣Eg2[Ξ1(ν, g1, g2)]− Eg1,g2[Ξ1(ν, g1, g2)] ∣∣∣∣ is C + C ′n1+δ-Lipschitz ] ≥ 1− 2e−cn − C ′′n−δ. Proof. Write f(ν, g1) = Eg2 [Ξ1(ν, g1, g2)]; it will suffice to differentiate f and E[f ] with respect to ν, bound the derivatives on an event of high probability, and apply the triangle inequality to obtain a high-probability Lipschitz estimate for |Eg2 [Ξ1(ν, g1, g2)]− Eg1,g2 [Ξ1(ν, g1, g2)]|.
Define k = √ 2/n. For fixed (g1, g2), the function
q(ν, g1, g2) = n∑ i=1 σ(g1i) 3ρ(−g1i cot ν) ψ(‖v0‖2)ψ(‖viν‖2) sin3 ν
is differentiable at all but at most n points of (0, ν), using Lemma E.31 to see that the only obstruction to differentiability is the function σ in the term ‖viν‖2; and there has derivative
q′(ν, g1, g2) = n∑ i=1
σ(g1i) 3
√ 2πk2ψ(‖v0‖2)
 g21i cos ν k2ψ(‖viν‖2) sin6 ν − ψ ′(‖viν‖2)〈v i ν ,v̇ i ν〉
ψ(‖viν‖2)2‖viν‖2 sin3 ν −3 cos ν
ψ(‖viν‖2) sin4 ν
 exp(− 12k2 g21i cot2 ν ) .
The triangle inequality and Lemma E.31 yield
|q′(ν, g1, g2)| ≤ 4√
2πk2 n∑ i=1 |g1i|3 ( 4g21i k2 sin6 ν + 16C‖v̇iν‖2 sin3 ν + 12 sin4 ν ) exp ( − 1 2k2 g21i cot 2 ν ) (E.60)
for C > 0 an absolute constant. We have ‖v̇iν‖2 ≤ ‖g1‖2 + ‖g2‖2 by the triangle inequality, so to obtain a (ν, g2)-integrable upper bound it suffices to remove the ν dependence from the previous estimate. We argue as follows: if 0 ≤ ν ≤ π/4 or 3π/4 ≤ ν ≤ π, we have cos2 ν ≥ 12 , and so for any p ≥ 3
exp ( − 12k2 g 2 1i cot 2 ν )
sinp ν ≤ exp ( g21i 4k2 1 sin2 t ) sin−p ν. (E.61)
By Lemma E.36, where we put C = g21i/4k 2 and therefore have to require that g1i 6= 0 for all i ∈ [n] (a set of measure zero in Rn), this yields |q′(ν, g1, g2)| ≤ C‖g2‖2, (E.62)
where C > 0 is a constant depending only on n and g1. In cases where g1i = 0 for some i, we note that the bound (E.60) is then equal to zero, which also satisfies the estimate (E.62). On the other hand, when π/4 ≤ ν ≤ 3π/4, then sin t ≥ 2−1/2, and we can assert for any p ≥ 3
exp ( − 12k2 g 2 1i cot 2 ν )
sinp ν ≤ 2p/2.
By the triangle inequality, this too implies
|q′(ν, g1, g2)| ≤ C ′‖g2‖2, where C ′ > 0 is a constant depending only on n and g1. Invoking then Lemma G.9, we conclude that q′ is absolutely integrable over [0, π]×Rn, so that an application of Fubini’s theorem and (Cohn, 2013, Theorem 6.3.11) gives the Taylor expansion f(ν, g1) = f(0, g1) + ∫ ν 0 Eg2 [q
′(t, g1, g2)] dt. Next, we show also that q′ is absolutely integrable over [0, π] × Rn × Rn, which implies that E[f(ν, g1)] = E[f(0, g1)] + ∫ ν 0 E[q′(t, g1, g2)] dt as well. Starting from (E.60), we have E[|q′(ν, g1, g2)|]
≤ E [ 4√
2πk2 n∑ i=1 |g1i|3 ( 4g21i k2 sin6 ν + 16C(‖gi1‖2 + ‖gi2‖2) sin3 ν + 12 sin4 ν ) exp ( − 1 2k2 g21i cot 2 ν )] ,
and the expectation factors over g1i, gi1, g i 2, so we can separately compute the g1i integrals first. For the first of the three terms on the RHS of the previous expression, we have
E g1i
[ |g1i|5
sin6 ν exp
( − 1
2k2 g21i cot 2 ν
)] =
1√ 2πk2 ∫ R |g|5 sin6 ν exp ( − 1 2k2 g2/ sin2 ν ) dg
= 1√
2πk2 ∫ R |g|5 exp ( − 1 2k2 g2 ) dg, (E.63)
after the change of variables g 7→ g sin ν in the integral, which is valid whenever 0 < ν < π. To take care of the case where ν = 0 or ν = π, we can use the estimate (E.61), valid for ν sufficiently close to 0 or π, and the assumption g1i 6= 0 for all i to conclude that limν↘0 q′(ν, g1, g2) = 0 for any such fixed (g1, g2), and by symmetry the analogous result limν↗π q′(ν, g1, g2) = 0; and whenever for some i we have g1i = 0, we use (E.60) to see that the term in the sum involving g1i poses no problems as ν ↘ 0 or ν ↗ π because it is identically 0. Returning to the integral (E.63), we have after a change of variables
1√ 2πk2 ∫ R |g|5 exp ( − 1 2k2 g2 ) dg = k5√ 2π ∫ R |g|5 exp ( −1 2 g2 ) dg = Ck5,
where C > 0 is an absolute constant, and where we use Lemma G.11 for the last equality. The remaining two terms can be treated using the same argument: we get
E g1i
[ |g1i|3
sin3 ν exp
( − 1
2k2 g21i cot 2 ν
)] = C ′k3
(after using |sin ν| ≤ 1) and
E g1i
[ |g1i|3
sin4 ν exp
( − 1
2k2 g21i cot 2 ν
)] = C ′′k3
for absolute constants C ′, C ′′ > 0. Combining these estimates gives
E[|q′(ν, g1, g2)|] ≤ C
n n∑ i=1 E gii ,g i 2 [ ‖gi1‖2 + ‖gi2‖2 ] ,
and using Lemma G.9 (or equivalently Jensen’s inequality) gives finally E[|q′(ν, g1, g2)|] ≤ C √ n− 1 n ≤ C.
To conclude, we need to show that Eg2 [q′(ν, g1, g2)] is uniformly bounded by a polynomial in n with high probability. For this we start from the estimate (E.60) and apply the argument following that, but with more care in tracking the constants: if ν is within π/4 of either 0 or π, we can assert
|q′(ν, g1, g2)| ≤ C
k n∑ i=1 C1k 4 |g1i| + C2k 3(‖gi1‖2 + ‖gi2‖2) + C3k 4 |g1i|
whenever g1i 6= 0 for every i (a set of full measure); and when ν is within π/4 of π/2, we can assert
|q′(ν, g1, g2)| ≤ C
k n∑ i=1 C ′1|g1i|5 k2 + C ′2|g1i|3(‖gi1‖2 + ‖gi2‖2) + C ′3|g1i|3,
where Ci, C ′i > 0 are absolute constants. By the triangle inequality, independence, and Lemma G.9, when we consider |Eg2 [q′(ν, g1, g2)]|, the term E[‖gi2‖2] is bounded by an absolute constant. Additionally, by Gauss-Lipschitz concentration and Lemma G.9, we have that simultaneously for all i ‖gi1‖2 ≤ ‖g1‖2 ≤ 2 with probability at least 1 − 2e−cn. Moreover, since ‖g1‖∞ ≤ ‖g1‖2 we also have control of the magnitude of each |g1i| on this event, so with probability at least 1 − 2e−cn we have for every ν ∣∣∣∣Eg2[q′(ν, g1, g2)] ∣∣∣∣ ≤ Ck n∑ i=1 C1k 4 |g1i| + C2k 3 + C3 k2 + C4
for absolute constants C,Ci > 0. If X ∼ N (0, 1), we have for any t ≥ 0 that P[|X| ≥ t] ≥ 1−Ct, where C > 0 is an absolute constant; so if Xi ∼i.i.d. N (0, 1), we have by independence and if t is less than an absolute constant P[∀i, |Xi| ≥ t] ≥ (1 − Ct)n ≥ 1 − C ′nt, where the last inequality uses the numerical inequality e−2t ≤ 1 − t ≤ e−t, valid for 0 ≤ t ≤ 12 . From this expression, we conclude that when 0 ≤ t ≤ cn−1/2 for an absolute constant c > 0, we have
P[∀i ∈ [n], |g1i| ≥ t] ≥ 1− Cn3/2t,
so choosing in particular t = cn−(δ+ 3 2 ) for any δ > 0, we conclude that P[∀i ∈ [n], |g1i| ≥ cn−3/2−δ] ≥ 1 − C ′n−δ . Consequently, for any δ > 0 we have with probability at least 1 − C ′n−δ − 2e−cn ∣∣∣∣Eg2[q′(ν, g1, g2)] ∣∣∣∣ ≤ Ck n∑ i=1 C1k 4n3/2+δ + C2k 3 + C3 k2 + C4,
and since k = √
2/n, this yields |Eg2 [q′(ν, g1, g2)]| ≤ C1n1+δ + C2 + C3n5/2 + C4n3/2 with the same probability. Consequently we can conclude that for any δ ≥ 3/2, we have
P [∣∣∣∣Eg2[Ξ1(ν, g1, g2)]− Eg1,g2[Ξ1(ν, g1, g2)] ∣∣∣∣ is C + C ′n1+δ-Lipschitz ] ≥ 1− 2e−cn − C ′′n−δ.
Lemma E.41. In the notation of Lemma E.13, if d ≥ 1, there are absolute constants c, c′C,C ′ > 0 and an absolute constant K > 0 such that if n ≥ K, there is an event with probability at least 1− Ce−cn on which
∀ν ∈ [0, π], |Ξ2(ν, g1, g2)− E[Ξ2(ν, g1, g2)]| ≤ C ′e−c ′n.
Proof. Let E denote the event E0.5,0 in Lemma E.16; then by that lemma, E has probability at least 1− Ce−cn as long as n ≥ C ′, where c, C,C ′ > 0 are absolute constants, and for (g1, g2) ∈ E , one has for all ν ∈ [0, π]
Ξ2(ν, g1, g2) = 0.
This allows us to calculate, for each ν,
E[Ξ2(ν, g1, g2)] = E[1EcΞ2(ν, g1, g2)] ≤ E[1Ec ]1/2‖Ξ2(ν, · )‖L2 ≤ C ′e−c ′n,
after applying Lemma E.37 and Lyapunov’s inequality and worst-casing constants. We conclude that with probability at least 1− Ce−cn
∀ν ∈ [0, π], |Ξ2(ν, g1, g2)− E[Ξ2(ν, g1, g2)]| ≤ C ′e−c ′n.
Lemma E.42. In the notation of Lemma E.13, if d ≥ 1, there are absolute constants c, c′C,C ′ > 0 and an absolute constant K > 0 such that if n ≥ K, there is an event with probability at least 1− Ce−cn on which
∀ν ∈ [0, π], |Ξ3(ν, g1, g2)− E[Ξ3(ν, g1, g2)]| ≤ C ′e−c ′n.
Proof. The argument is identical to Lemma E.41. Let E denote the event E0.5,0 in Lemma E.16; then by that lemma, E has probability at least 1−Ce−cn as long as n ≥ C ′, where c, C,C ′ > 0 are absolute constants, and for (g1, g2) ∈ E , one has for all ν ∈ [0, π]
Ξ3(ν, g1, g2) = 0.
This allows us to calculate, for each ν,
E[Ξ3(ν, g1, g2)] = E[1EcΞ3(ν, g1, g2)] ≤ E[1Ec ]1/2‖Ξ3(ν, · )‖L2 ≤ C ′e−c ′n,
after applying Lemma E.37 and Lyapunov’s inequality and worst-casing constants. We conclude that with probability at least 1− Ce−cn
∀ν ∈ [0, π], |Ξ3(ν, g1, g2)− E[Ξ3(ν, g1, g2)]| ≤ C ′e−c ′n.
Lemma E.43. In the notation of Lemma E.13, if d ≥ 1, there are absolute constants c, C,C ′, C ′′ > 0 and absolute constants K,K ′ > 0 such that if n ≥ Kd log n and d ≥ K ′, there is an event with probability at least 1− Ce−cn − C ′n−d on which one has
∀ν ∈ [0, π], |Ξ4(ν, g1, g2)− E[Ξ4(ν, g1, g2)]| ≤ C ′′ √ d log n
n .
Proof. We are going to control the expectation first, showing that it is small; then prove that |Ξ4| is small uniformly in ν. Let E denote the event E0.5,0 in Lemma E.16; then by that lemma, E has probability at least 1 − Ce−cn as long as n ≥ C ′, where c, C,C ′ > 0 are absolute constants, and for (g1, g2) ∈ E , one has for all ν ∈ [0, π]
Ξ4(ν, g1, g2) = −2 〈v0, v̇ν〉〈vν , v̇ν〉 ‖v0‖2‖vν‖32 .
Thus, if we write
Ξ̃4(ν, g1, g2) = −21E(g1, g2) 〈v0, v̇ν〉〈vν , v̇ν〉 ‖v0‖2‖vν‖32 ,
we have Ξ̃4 = Ξ4 for all ν whenever (g1, g2) ∈ E , so that for any ν |E[Ξ4(ν, g1, g2)]| = ∣∣∣E[Ξ̃4(ν, g1, g2)] + E[1EcΞ4(ν, g1, g2)]∣∣∣
≤ |E[Ξ̃4(ν, g1, g2)]|+ Ce−cn,
where the second line uses the triangle inequality and the Schwarz inequality and Lemma E.37 together with the Lyapunov inequality. We proceed with analyzing the expectation of Ξ̃4. Using the Schwarz inequality gives∣∣∣E[Ξ̃4(ν, g1, g2)]∣∣∣ ≤ 2E[〈v0, v̇ν〉2〈vν , v̇ν〉2]1/2E[ 1E‖v0‖22‖vν‖62
]1/2 ≤ 32E [ 〈v0, v̇ν〉2〈vν , v̇ν〉2 ]1/2 ,
and the checks at and around (E.34) and (E.35) in the proof of Lemma E.15 show that we can apply Lemma E.30 to obtain∣∣∣∣∣∣ E [ 〈v0, v̇ν〉2〈vν , v̇ν〉2 ] −n4 ( E[σ(g11)σ̇(g11 cos ν + g21 sin ν)(g21 cos ν − g11 sin ν)]2 ∗E[σ(g11 cos ν + g21 sin ν)(g21 cos ν − g11 sin ν)]2 ) ∣∣∣∣∣∣ ≤ C/n.
But we have using rotational invariance that E[σ(g11 cos ν + g21 sin ν)(g21 cos ν − g11 sin ν)] = 0, which implies ∣∣E[〈v0, v̇ν〉2〈vν , v̇ν〉2]∣∣ ≤ C/n, from which we conclude
|E[Ξ4(ν, g1, g2)]| ≤ C/ √ n.
Next, we control the deviations of Ξ4 with high probability. By Lemma E.17, there is an event Ea with probability at least 1 − e−cn on which ‖v̇ν‖2 ≤ 4 for every ν ∈ [0, π]. Therefore on the event Eb = E ∩ Ea, which has probability at least 1 − Ce−cn by a union bound, we have using Cauchy-Schwarz that for every ν
|Ξ4(ν, g1, g2)| ≤ 256|〈vν , v̇ν〉|.
The coordinates of the random vector vν v̇ν are σ(g1i cos ν+g2i sin ν)(g2i cos ν−g1i sin ν), and we note E[σ(g1i cos ν + g2i sin ν)(g2i cos ν − g1i sin ν)] = −E[σ(g1i)g2i] = 0, by rotational invariance. Moreover, the calculation (E.35) together with Lemmas G.11 and E.17 demonstrates subexponential moment growth with rate C/n, so Lemma G.2 implies for t ≥ 0
P[〈vν , v̇ν〉 ≥ t] ≤ 2e−cntmin{c ′t,1}.
For large enough n, this gives
P [ 〈vν , v̇ν〉 ≥ C √ d log n
n
] ≤ 2n−2d.
We turn to the uniformization of this pointwise bound. The map ν 7→ ∑ i σ(g1i cos ν + g2i sin ν)(g2i cos ν − g1i sin ν) is continuous, and differentiable at all but finitely many points of [0, π] (following the zero crossings argument in the proof of Lemma E.22) with derivative
ν 7→ ∑ i σ̇(g1i cos ν + g2i sin ν)(g2i cos ν − g1i sin ν)2 − σ(g1i cos ν + g2i sin ν)2,
which is evidently integrable using the triangle inequality and Lemma G.11. In particular, we can write the derivative as ‖v̇ν‖22 − ‖v0‖22. Thus, by (Cohn, 2013, Theorem 6.3.11), to get a Lipschitz estimate on ν 7→ 〈vν , v̇ν〉 it suffices to bound the magnitude of the derivative ν 7→ ‖v̇ν‖22 − ‖v0‖22. But this is immediate, since on the event Eb we have |‖v̇ν‖22 − ‖v0‖22| ≤ 20. It thus follows from Lemma E.48 that with probability at least 1− Ce−cn − C ′n−2d+1/2 we have
∀ν ∈ [0, π], 〈vν , v̇ν〉 ≤ C ′′ √ d log n
n . (E.64)
As long as d ≥ 12 , we have that this probability is at least 1 − Ce −cn − C ′n−d, and so the triangle inequality and a union bound yield finally that with probability at least 1− Ce−cn − C ′n−d
∀ν ∈ [0, π], |Ξ4(ν, g1, g2)− E[Ξ4(ν, g1, g2)]| ≤ C ′′ √ d log n
n .
Lemma E.44. In the notation of Lemma E.13, if d ≥ 1, there are absolute constants c, c′, c′′, C, C ′, C ′′, C ′′′, C ′′′′ > 0 and an absolute constant K > 0 such that if n ≥ Kd log n, there is an event with probability at least 1− Ce−cn + C ′e−d on which one has
|Ξ5(ν, g1, g2)− E[Ξ5(ν, g1, g2)]| ≤ C ′′ √ d
n + C ′′′e−c ′d + C ′′′′e−c ′′n,
Proof. Fix ν ∈ [0, π]. Let E denote the event E0.5,0 in Lemma E.16; then by that lemma, E has probability at least 1 − Ce−cn as long as n ≥ C ′, where c, C,C ′ > 0 are absolute constants, and for (g1, g2) ∈ E , one has for all ν ∈ [0, π]
Ξ5(ν, g1, g2) = − 〈v0,vν〉‖v̇ν‖22 ‖v0‖2‖vν‖32 .
Thus, if we write
Ξ̃5(ν, g1, g2) = −1E(g1, g2) 〈v0,vν〉‖v̇ν‖22 ‖v0‖2‖vν‖32
we have Ξ̃5 = Ξ5 for any ν whenever (g1, g2) ∈ E , so that by the triangle inequality, for any ν |Ξ5(ν, g1, g2)− E[Ξ5(ν, g1, g2)]| ≤ ∣∣∣Ξ̃5(ν, g1, g2)− E[Ξ̃5(ν, g1, g2)]∣∣∣
+ ∣∣∣E[Ξ5(ν, g1, g2)]− E[Ξ̃5(ν, g1, g2)]∣∣∣
≤ ∣∣∣Ξ̃5(ν, g1, g2)− E[Ξ̃5(ν, g1, g2)]∣∣∣
+ E [ 1Ec ∣∣∣Ξ5(ν, g1, g2)− Ξ̃5(ν, g1, g2)∣∣∣]
≤ ∣∣∣Ξ̃5(ν, g1, g2)− E[Ξ̃5(ν, g1, g2)]∣∣∣+ Ce−cn,
where the second line uses the triangle inequality, and the third line uses the Schwarz inequality and Lemma E.37 together with the Lyapunov inequality.
So, we can proceed analyzing Ξ̃5. First, we aim to apply Lemma E.33 with the choices
X = −1E 〈v0,vν〉 ‖v0‖2‖vν‖2 ; Y = 1E ‖v̇ν‖22 ‖vν‖22 ,
since XY = Ξ̃5(ν, · , · ); square-integrability of X and Y is evident from the definition of 1E , and we have |X| ≤ 1 by Cauchy-Schwarz. To control Y , we start by noting
‖Y − 1‖L2 ≤ 1 + ‖Y ‖L2 ≤ 1 + 4E [ ‖v̇ν‖42 ]1/2 ≤ 1 + 4√1 + C, where C > 0 is an absolute constant; the first inequality is the Minkowski inequality, the second uses the property of E and drops the indicator by nonnegativity, and the third applies Lemma E.29, and discards the n−1 factor. For deviations, we start by noting that E[‖vν‖22] = 1, and that by Lemmas G.2 and G.11, we have
P [∣∣‖vν‖22 − 1∣∣ ≥ t] ≤ 2e−cntmin{Ct,1}.
It follows that there exists an absolute constant C ′ > 0 such that, putting t = C ′ √ d/n and choosing n ≥ (C ′/C)2d, we have
P [∣∣‖vν‖22 − 1∣∣ ≥ C ′√ dn ] ≤ 2e−d. (E.65)
Moreover, by Lemma E.17, we can run a similar argument on ‖v̇ν‖22 to get that if n is larger than a constant multiple of d
P [∣∣‖v̇ν‖22 − 1∣∣ ≥ C√ dn ] ≤ 2e−d. (E.66)
Next, Taylor expansion with Lagrange remainder of the smooth function x 7→ x−1 on the domain x > 0 about the point 1 gives
1 x = 1− (x− 1) + 1 ξ3 (x− 1)2, (E.67)
where ξ lies between 1 and x. If (g1, g2) ∈ E , then ‖vν‖62 ≥ (1/64), and we can therefore assert
1− ( ‖vν‖22 − 1 ) ≤ 1 ‖vν‖22 ≤ 1− ( ‖vν‖22 − 1 ) + 64 ( ‖vν‖22 − 1 )2 with probability at least 1 − Ce−cn. Using a union bound together with (E.65) (and changing the constant to C), we have with probability at least 1− 2e−cd − C ′e−c′n that
−C √ d
n − 64C2 d n ≤ 1− 1 ‖vν‖22 ≤ C
√ d
n .
Given that n ≥ d, it follows that with the same probability we have −C(1 + 64C) √ d
n ≤ 1− 1 ‖vν‖22 ≤ C
√ d
n ,
which implies that with probability at least 1− 2e−d − C ′e−cn, we have∣∣∣∣1− 1‖vν‖22 ∣∣∣∣ ≤ C √ d n .
Now, the triangle inequality gives∣∣∣∣‖v̇ν‖22‖vν‖22 − 1 ∣∣∣∣ ≤ ∣∣∣∣‖v̇ν‖22‖vν‖22 − 1‖vν‖22 ∣∣∣∣+ ∣∣∣∣ 1‖vν‖22 − 1 ∣∣∣∣
≤ 1 ‖vν‖22 ∣∣‖v̇ν‖22 − 1∣∣+ ∣∣∣∣ 1‖vν‖22 − 1 ∣∣∣∣.
When (g1, g2) ∈ E , we have ‖vν‖22 ≥ 14 , so, by a union bound, with probability at least 1− 4e −d−
C ′e−cn we have ∣∣∣∣‖v̇ν‖22‖vν‖22 − 1 ∣∣∣∣ ≤ 4C √ d n ,
and since (g1, g2) ∈ E =⇒ Y = ‖v̇ν‖ 2 2
‖vν‖22 ,
another union bound and the measure bound on E let us conclude that with probability at least 1− 4e−d − C ′e−cn, we have |Y − 1| ≤ 4C √ d
n .
If we choose n ≥ (1/c)(d + logC ′/4), we have 4e−d + C ′e−cn ≤ 8e−d, so the previous bound occurs with probability at least 1− 8e−d. We can now apply Lemma E.33 to get with probability at least 1− 8e−d∣∣∣Ξ̃5 − E[Ξ̃5]∣∣∣ ≤ ∣∣∣∣1E〈 v0‖v0‖2 , vν‖vν‖2 〉 − E [ 1E 〈 v0 ‖v0‖2 , vν ‖vν‖2 〉]∣∣∣∣+ C √ d n + C ′e−d/2.
Next, we attempt to apply Lemma E.33 again, this time to X = 1E〈v0,vν〉 and Y = 1E(‖v0‖2‖vν‖2)−1. Using the definition of E , we have |X| ≤ 4 and ‖Y − 1‖L2 ≤ ‖Y ‖L2 + 1 ≤ 5,
where the second bound also leverages the Minkowski inequality; so we need only establish deviations of Y . Applying again (E.67), and using (g1, g2) ∈ E implies ‖v0‖2‖vν‖2 ≥ 14 , we get
(‖v0‖2‖vν‖2 − 1)− 64 (‖v0‖2‖vν‖2 − 1)2 ≤ 1− 1
‖v0‖2‖vν‖2 ≤ (‖v0‖2‖vν‖2 − 1) (E.68)
with probability at least 1−Ce−cn. Using Lemma G.11 and (Vershynin, 2018, Theorem 3.1.1), we can assert for any ν ∈ [0, π] and any t ≥ 0
P[|‖vν‖2 − 1| ≥ t] ≤ 2e−cnt 2 ,
which implies that there exists an absolute constant C > 0 such that for any d > 0
P [ |‖vν‖2 − 1| ≥ C √ d
n
] ≤ 2e−d.
In particular, when n ≥ d, we can assert that ‖vν‖2 ≤ 1 +C with probability at least 1− 2e−d. By the triangle inequality and a union bound, it follows
|‖v0‖2‖vν‖2 − 1| ≤ ‖v0‖2|‖vν‖2 − 1|+ |‖v0‖2 − 1| ≤ C √ d
n
with probability at least 1−6e−d. Then a union bound gives that with probability at least 1−6e−d− C ′e−cn, (E.68) leads to
−C √ d
n
( 1 + 64C √ d
n
) ≤ 1− 1
‖v0‖2‖vν‖2 ≤ C
√ d
n ,
and using n ≥ d and worst-casing constants implies that with the same probability∣∣∣∣1− 1‖v0‖2‖vν‖2 ∣∣∣∣ ≤ C √ d n .
Then since (g1, g2) ∈ E =⇒ Y = (‖v0‖2‖vν‖2)−1, another union bound gives that with probability at least 1 − 6e−d − C ′e−cn we have |Y − 1| ≤ C √ d/n. As in the previous step of the reduction, we can choose n ≥ (1/c)(d+ logC ′/6) to get that 6e−d +C ′e−cn ≤ 12e−d, so that the previous bound occurs with probability at least 1− 12e−d. We can thus apply Lemma E.33, a union bound, and our previous work to get that with probability at least 1− 20e−d∣∣∣Ξ̃5 − E[Ξ̃5]∣∣∣ ≤ |1E〈v0,vν〉 − E[1E〈v0,vν〉]|+ C√ d
n + C ′e−d/2.
Whenever (g1, g2) ∈ E , we have by the triangle inequality, the Schwarz inequality, and Lemmas E.16 and E.29 that
|1E〈v0,vν〉 − E[1E〈v0,vν〉]| ≤ |〈v0,vν〉 − E[〈v0,vν〉]|+ |E[〈v0,vν〉]− E[1E〈v0,vν〉]| ≤ |〈v0,vν〉 − E[〈v0,vν〉]|+ Ce−cn,
allowing us to drop the indicator. We have 〈v0,vν〉 = ∑ i σ(g1i)σ(g1i cos ν + g2i sin ν), which is a sum of independent random variables; following the argument at and around (E.36), we conclude moreover that these random variables are subexponential with rate C/n, where C > 0 is an absolute constant. We therefore obtain from Lemma G.2 the tail bound
P[|〈v0,vν〉 − E[〈v0,vν〉]| ≥ t] ≤ 2e−cntmin{Ct,1},
which, for a suitable choice of absolute constantC ′ > 0 and choosing n ≥ C ′d, yields the deviations bounds
P [ |〈v0,vν〉 − E[〈v0,vν〉]| ≥ C √ d
n
] ≤ 2e−d.
Taking a final union bound (since we assumed throughout that (g1, g2) ∈ E) gives that with probability at least 1− Ce−cn + C ′e−d, one has
|Ξ5(ν, g1, g2)− E[Ξ5(ν, g1, g2)]| ≤ C ′′ √ d
n + C ′′′e−c ′d + C ′′′′e−c ′′n,
which is sufficient to conclude pointwise concentration as claimed for sufficiently large n after we put d = d′ log n and include extra log n factors in any points where we need to choose n larger than d.
Lemma E.45. In the notation of Lemma E.13, there are absolute constants c, C,C ′, C ′′, C ′′′ > 0 such that for any δ ≥ 12 , one has
P [∣∣∣∣Eg2[Ξ5(ν, g1, g2)]− Eg1,g2[Ξ5(ν, g1, g2)] ∣∣∣∣ is C + C ′n1+δ-Lipschitz ] ≥ 1− C ′′e−cn − C ′′′n−δ as long as δ ≥ 12 .
Proof. We will differentiate with respect to ν the function
f(ν, g1) = − E g2 [ 〈v0,vν〉‖v̇ν‖22ψ′(‖vν‖2) ψ(‖v0‖2)ψ(‖vν‖2)2‖vν‖2 ] ,
and construct an event on which f ′ has size poly(n). We need to also differentiate the function E[f( · , g1)]; for this we will additionally show that f ′(ν, · ) is absolutely integrable over the product [0, π]×Rn×Rn, which allows us to apply Fubini’s theorem to move both the g1 and g2 expectations under the ν integral in the first-order Taylor expansion we obtain. In particular, the derivative of E[f( · , g1)] will in this way be shown to be E[f ′( · , g1)], so that linearity and the triangle inequality imply a poly(n) magnitude bound for the derivative of Eg2 [Ξ5]− E[Ξ5]. Define
qi(ν, g1, g2) = 〈v0,vν〉ψ′(‖vν‖2)(g2i cos ν − g1i sin ν)2
ψ(‖v0‖2)ψ(‖vν‖2)2‖vν‖2 ,
so that, for almost all g1,
f(ν, g1) = − n∑ i=1 E g2 [qi(ν, g1, g2)σ̇(g1i cos ν + g2i sin ν)].
For each fixed (g1, g2) and each i, the only obstructions to differentiability of qi in ν arise from the function σ (using smoothness of ψ from Lemma E.31 and the fact that it is constant whenever ‖vν‖ is small enough that nondifferentiability of ‖ · ‖2 could pose a problem); following the zerocrossings argument of Lemma E.22, qi fails to be differentiable at no more than n points of [0, π], and otherwise has derivative
q′i(ν, g1, g2) =
1
ψ(‖v0‖2)
( 〈v0, v̇ν〉ψ′(‖vν‖2)(g2i cos ν − g1i sin ν)2
ψ(‖vν‖2)2‖vν‖2
+ 〈v0,vν〉〈vν , v̇ν〉ψ′′(‖vν‖2)(g2i cos ν − g1i sin ν)2
ψ(‖vν‖2)2‖vν‖22
− 2 〈v0,vν〉ψ ′(‖vν‖2)(g2i cos ν − g1i sin ν)(g1i cos ν + g2i sin ν)
ψ(‖vν‖2)2‖vν‖2
− 2ψ ′(‖vν‖2)2〈vν , v̇ν〉〈v0,vν〉(g2i cos ν − g1i sin ν)2
ψ(‖vν‖2)3‖vν‖22
− ψ ′(‖vν‖2)〈vν , v̇ν〉〈v0,vν〉(g2i cos ν − g1i sin ν)2
ψ(‖vν‖2)2‖vν‖32
) . (E.69)
by the chain rule and the product rule. To conclude absolute continuity of qi( · , g1, g2), we need to show that q′i is integrable; this follows from Cauchy-Schwarz, the integrability of ‖v0‖2, ‖vν‖2,
‖v̇ν‖2 (Lemma E.17), the triangle inequality, and the Lemma E.31 estimates ψ ≥ 14 , |ψ ′| ≤ C, |ψ′′| ≤ C ′, and |ψ′(x)/x| ≤ C ′′ for any x ∈ R (to see this last estimate, note that |ψ′| is bounded on R, and use that ψ is constant whenever x ≤ 14 ). Then (Cohn, 2013, Theorem 6.3.11) implies that qi( · , g1, g2) is absolutely continuous with a.e. derivative q′i. Next, we can write
f(ν, g1) = − n∑ i=1 E g2j :j 6=i [ E g2i [qi(ν, g1, g2)σ̇(g1i cos ν + g2i sin ν)] ] ,
using Lemma E.37 to see that Fubini’s theorem can be applied. Our aim is now to apply Lemma E.27, so we need to check its remaining hypotheses. First, continuity of qi(ν, · ) follows from continuity of σ, smoothness of ψ, and the fact that the denominator never vanishes. Joint absolute integrability of qi and q′i follows from our verification of absolute integrability of q ′ i above, which produces a final upper bound that does not depend on ν (which is therefore integrable over [0, π] as well); the corresponding result for qi follows from Lemma E.37. Last, we need the growth estimate. We have from Lemma E.31
|qi(ν, g1, g2)| ≤ 32C(g2i cos ν − g1i sin ν)2 ≤ 32C(|g2i|+ |g1i|)2 ≤ 32C|g1i|(1 + |g2i|)2,
which is evidently quadratic in |g2i| once |g2i| ≥ 1. Consequently we can apply Lemma E.27 to differentiate f( · , g1); we get at almost all g1
f(ν, g1) = − n∑ i=1
( E
g2j :j 6=i [ E g2i [qi(0, g1, g2)σ̇(g1i)] ]
+ E g2j :j 6=i [∫ ν 0 dt ( Eg2i [q′i(t, g1, g2)σ̇(g1i cos t+ g2i sin t)] −g1iqi(t, g1, g̃i2)ρ(−g1i cot t) sin−2 t )]) ,
where g̃i2 is the vector g2 but with its i-th coordinate replaced by −g1i cot t, and where ρ is the pdf of a N (0, 2/n) random variable. The changes in g̃i2 drive updates to the terms in qi as follows: we have σ(g1i cos ν + g2i sin ν) becoming 0, and g2i cos ν − g1i sin ν becoming −g1i/ sin ν. Thus, we have
−g1iqi(t, g1, g̃i2)ρ(−g1i cot t) sin−2 t = − g31i〈vi0,vit〉ψ′(‖vit‖2)ρ(−g1i cot t) ψ(‖v0‖2)ψ(‖vit‖2)2‖vit‖2 sin4 t ,
where the notation vit is in use in the Ξ1 control section and is defined in Lemma E.26, and v i 0 is defined here similarly (the Rn−1 vector which is the projection of v0 onto all but the i-th coordinates). Using Lemma E.31, we can further assert∣∣g1iqi(t, g1, g̃i2)ρ(−g1i cot t) sin−2 t∣∣ ≤ 16C |g1i|3ρ(−g1i cot t)
sin4 t (E.70)
where we use that ‖vi0‖2 ≤ ‖v0‖2. For each fixed g1 having no coordinates equal to zero, we write Ki = |g1i| > 0; if 0 ≤ t ≤ π/4 or 3π/4 ≤ t ≤ π, we have cos2 t ≥ 12 , and so
ρ(−g1i cot t) sin4 t
≤ √ n
4π sin−4 t exp
( K2i n
8
1
sin2 t
) .
Using Lemma E.36, we have
ρ(−g1i cot t) sin4 t
≤ √ n
4π
( 16
K2i n
)2 .
On the other hand, when π/4 ≤ t ≤ 3π/4, then sin t ≥ 2−1/2, and we can assert
ρ(−g1i cot t) sin4 t
≤ 8 √ n/π.
We conclude for any t∣∣g1iqi(t, g1, g̃i2)ρ(−g1i cot t) sin−2 t∣∣ ≤ C/(Kin3/2) + C ′√nK3i (E.71)
for absolute constants C,C ′ > 0, and this upper bound is integrable jointly over t and g2. We have checked previously the joint integrability of the q′i terms when applying Lemma E.27, so we can therefore apply Fubini’s theorem to get g1-a.s.
f(ν, g1) =− E g2 [ n∑ i=1 qi(0, g1, g2)σ̇(g1i) ]
− ∫ ν
0 n∑ i=1 E g2 [ q′i(t, g1, g2)σ̇(g1i cos t+ g2i sin t) −g1iqi(t, g1, g̃i2)ρ(−g1i cot t) sin−2 t ] dt.
Consequently, to conclude a Lipschitz estimate for f( · , g1) it suffices to control the quantity under the t integral in the previous expression. We will start by controlling the second term using Markov’s inequality. Following (E.70), we calculate
E g1,g2 [∣∣g1iqi(t, g1, g̃i2)ρ(−g1i cot t) sin−2 t∣∣] ≤ 8C√nπ Eg1  |g1i|3 exp ( −n4 g21i cos 2 t sin2 t ) sin4 t  = 4Cn
π ∫ R |g|3 sin4 t exp ( −n 4 g2 sin2 t ) dg
= 4Cn
π ∫ R |g|3 exp ( −n 4 g2 ) dg,
where the last line follows from the change of variables g 7→ g sin t in the integral. We can evaluate this integral with Lemma G.11, which gives a bound
E g1,g2 [∣∣g1iqi(t, g1, g̃i2)ρ(−g1i cot t) sin−2 t∣∣] ≤ 128Cπn , and therefore a bound of C ′ > 0 an absolute constant on the sum over i. As a byproduct of this estimate, we can assert that the second term is jointly integrable over [0, π]×Rn×Rn, which allows us to apply Fubini’s theorem and obtain the same differentiation result for E[f( · , g1)]. Meanwhile, beginning from (E.71), we can write using the triangle inequality∣∣∣∣∣Eg2 [ n∑ i=1 g1iqi(t, g1, g̃ i 2)ρ(−g1i cot t) sin−2 t ]∣∣∣∣∣ ≤ n∑ i=1 C |g1i|n3/2 + C ′ √ n|g1i|3.
By Gauss-Lipschitz concentration and Lemma G.9, we have that ‖g1‖2 ≤ ‖g1‖2 ≤ 2 with probability at least 1 − 2e−cn, and since ‖g1‖∞ ≤ ‖g1‖2, we conclude with the same probability that |g1i| ≤ 2 simultaneously for all i. Meanwhile, if X ∼ N (0, 1), we have for any t ≥ 0 that P[|X| ≥ t] ≥ 1 − Ct, where C > 0 is an absolute constant; so if Xi ∼i.i.d. N (0, 1), we have by independence and if t is less than an absolute constant P[∀i, |Xi| ≥ t] ≥ (1 − Ct)n ≥ 1 − C ′nt, where the last inequality uses the numerical inequality e−2t ≤ 1 − t ≤ e−t, valid for 0 ≤ t ≤ 12 . From this expression, we conclude that when 0 ≤ t ≤ cn−1/2 for an absolute constant c > 0, we have
P[∀i ∈ [n], |g1i| ≥ t] ≥ 1− Cn3/2t,
so choosing in particular t = cn−(δ+ 3 2 ) for any δ > 0, we conclude that P[∀i ∈ [n], |g1i| ≥ cn−3/2−δ] ≥ 1− C ′n−δ . Then with probability at least 1− C ′n−δ − 2e−cn, we have∣∣∣∣∣Eg2 [ n∑ i=1 g1iqi(t, g1, g̃ i 2)ρ(−g1i cot t) sin−2 t
]∣∣∣∣∣ ≤ Cn1+δ + C ′n3/2, so as long as δ ≥ 12 , we have
P [∣∣∣∣∣Eg2 [ n∑ i=1 g1iqi(t, g1, g̃ i 2)ρ(−g1i cot t) sin−2 t ]∣∣∣∣∣ ≥ Cn1+δ ] ≤ C ′n−δ + 2e−cn.
Proceeding now to the q′i term, from the expression (E.69) we get n∑ i=1 q′i(ν, g1, g2)σ̇(g1i cos ν + g2i sin ν)
= 1
ψ(‖v0‖2)
( 〈v0, v̇ν〉ψ′(‖vν‖2)‖v̇ν‖22
ψ(‖vν‖2)2‖vν‖2 + 〈v0,vν〉〈vν , v̇ν〉ψ′′(‖vν‖2)‖v̇ν‖22 ψ(‖vν‖2)2‖vν‖22
−2 〈v0,vν〉ψ ′(‖vν‖2)〈v̇ν ,vν〉
ψ(‖vν‖2)2‖vν‖2 − 2ψ ′(‖vν‖2)2〈vν , v̇ν〉〈v0,vν〉‖v̇ν‖22 ψ(‖vν‖2)3‖vν‖22
−ψ ′(‖vν‖2)〈vν , v̇ν〉〈v0,vν〉‖v̇ν‖22
ψ(‖vν‖2)2‖vν‖32
) .
(E.72)
Using the triangle inequality, Cauchy-Schwarz, and Lemma E.31, we obtain
〈v0, v̇ν〉ψ′(‖vν‖2)‖v̇ν‖22 ψ(‖v0‖2)ψ(‖vν‖2)2‖vν‖2 + 〈v0,vν〉〈vν , v̇ν〉ψ′′(‖vν‖2)‖v̇ν‖22 ψ(‖v0‖2)ψ(‖vν‖2)2‖vν‖22 ≤ C‖v̇ν‖32,
(using also the fact that ψ′(x) = 0 and ψ′′(x) = 0 whenever x is sufficiently near to 0); and
〈v0,vν〉ψ′(‖vν‖2)〈v̇ν ,vν〉 ψ(‖vν‖2)2‖vν‖2
+ ψ′(‖vν‖2)2〈vν , v̇ν〉〈v0,vν〉‖v̇ν‖22
ψ(‖vν‖2)3‖vν‖22
+ ψ′(‖vν‖2)〈vν , v̇ν〉〈v0,vν〉‖v̇ν‖22
ψ(‖vν‖2)2‖vν‖32
≤ C‖v̇ν‖2 + C ′‖v̇ν‖32,
from which we conclude∣∣∣∣∣ n∑ i=1 q′i(ν, g1, g2)σ̇(g1i cos ν + g2i sin ν) ∣∣∣∣∣ ≤ C‖v̇ν‖2 + C ′‖v̇ν‖32 for some absolute constants C,C ′ > 0. By Lemma E.17, there is an event E of probability at least 1−Ce−cn on which we have ‖v̇ν‖2 ≤ 4 for every ν. Moreover, we have from the triangle inequality that ‖v̇ν‖2 ≤ ‖g1‖2 + ‖g2‖2, which is independent of ν; and in particular we have∣∣∣∣∣ n∑ i=1 q′i(ν, g1, g2)σ̇(g1i cos ν + g2i sin ν) ∣∣∣∣∣ 2 ≤ ( C(‖g1‖2 + ‖g2‖2) + C ′(‖g1‖2 + ‖g2‖2)3 )2 ,
which is a polynomial in ‖g1‖2 and ‖g2‖2 by the binomial theorem. Thus, applying independence, Lemma G.10, Lemma G.11 yields that there is an absolute constant C ′′ > 0 such that
E g1,g2
[( C(‖g1‖2 + ‖g2‖2) + C ′(‖g1‖2 + ‖g2‖2)3 )2] ≤ C ′′. Therefore, as in the framework section of the proof of Lemma E.13, we can use the inequality∣∣∣∣∣Eg2 [ n∑ i=1 q′i(ν, g1, g2)σ̇(g1i cos ν + g2i sin ν) ]∣∣∣∣∣ ≤ Eg2 [∣∣∣∣∣ n∑ i=1 q′i(ν, g1, g2)σ̇(g1i cos ν + g2i sin ν) ∣∣∣∣∣ ] , (E.73) together with the partition
E g2 [∣∣∣∣∣ n∑ i=1 q′i(ν, g1, g2)σ̇(g1i cos ν + g2i sin ν) ∣∣∣∣∣ ]
≤ C ′ + E g2
[ 1(E′)c ∣∣∣∣∣ n∑ i=1 q′i(ν, g1, g2)σ̇(g1i cos ν + g2i sin ν) ∣∣∣∣∣ ] , (E.74)
and this last expression can be used to obtain a g1 event of not much smaller probability 1−Ce−cn on which the LHS of (E.74), and hence the LHS of (E.73), is controlled by an absolute constant
uniformly in ν (in particular, using Markov’s inequality as in the framework section of the proof of Lemma E.13). Consequently, one more application of the triangle inequality gives that
P [∣∣∣∣Eg2[Ξ5(ν, g1, g2)]− Eg1,g2[Ξ5(ν, g1, g2)] ∣∣∣∣ is C + C ′n1+δ-Lipschitz ] ≥ 1− C ′′e−cn − C ′′′n−δ as long as δ ≥ 12 .
Lemma E.46. In the notation of Lemma E.13, if d ≥ 1, there are absolute constants c, C,C ′, C ′′ > 0 and absolute constants K,K ′ > 0 such that if n ≥ Kd log n and d ≥ K ′, there is an event with probability at least 1− Ce−cn − C ′n−d on which one has
∀ν ∈ [0, π], |Ξ6(ν, g1, g2)− E[Ξ6(ν, g1, g2)]| ≤ C ′′ √ d log n
n .
Proof. The argument is extremely similar to Lemma E.43, since both terms have small expectations and deviations essentially determinable by the same mean-zero random variable.
We are going to control the expectation first, showing that it is small; then prove that |Ξ6| is small uniformly in ν. Let E denote the event E0.5,0 in Lemma E.16; then by that lemma, E has probability at least 1−Ce−cn as long as n ≥ C ′, where c, C,C ′ > 0 are absolute constants, and for (g1, g2) ∈ E , one has for all ν ∈ [0, π]
Ξ6(ν, g1, g2) = 3 〈v0,vν〉〈vν , v̇ν〉2
‖v0‖2‖vν‖52 .
Thus, if we write
Ξ̃6(ν, g1, g2) = 31E(g1, g2) 〈v0,vν〉〈vν , v̇ν〉2
‖v0‖2‖vν‖52 ,
we have Ξ̃6 = Ξ6 for all ν whenever (g1, g2) ∈ E , so that for any ν |E[Ξ6(ν, g1, g2)]| = ∣∣∣E[Ξ̃6(ν, g1, g2)] + E[1EcΞ6(ν, g1, g2)]∣∣∣
≤ |E[Ξ̃6(ν, g1, g2)]|+ Ce−cn,
where the second line uses the triangle inequality and the Schwarz inequality and Lemma E.37 together with the Lyapunov inequality. We proceed with analyzing the expectation of Ξ̃6. Using the Schwarz inequality gives∣∣∣E[Ξ̃6(ν, g1, g2)]∣∣∣ ≤ 3E[〈v0,vν〉2〈vν , v̇ν〉4]1/2E[ 1E‖v0‖22‖vν‖102
]1/2 ≤ 192E [ 〈v0,vν〉2〈vν , v̇ν〉4 ]1/2 ,
and the checks at and around (E.36) in the proof of Lemma E.15 show that we can apply Lemma E.30 to obtain∣∣∣∣∣E[〈v0,vν〉2〈vν , v̇ν〉4] − n6E[σ(g11)σ(g11 cos ν + g21 sin ν)]2E[σ(g11 cos ν + g21 sin ν)(g21 cos ν − g11 sin ν)]4
∣∣∣∣∣ ≤ Cn . But we have using rotational invariance that E[σ(g11 cos ν + g21 sin ν)(g21 cos ν − g11 sin ν)] = 0, which implies ∣∣E[〈v0,vν〉2〈vν , v̇ν〉4]∣∣ ≤ C/n, from which we conclude for all ν
|E[Ξ6(ν, g1, g2)]| ≤ C/ √ n.
Next, we control the deviations of Ξ6 with high probability. By Lemma E.17, there is an event Ea with probability at least 1 − e−cn on which ‖v̇ν‖2 ≤ 4 for every ν ∈ [0, π]. Therefore on the
event Eb = E ∩ Ea, which has probability at least 1 − Ce−cn by a union bound, we have using Cauchy-Schwarz that for every ν
|Ξ6(ν, g1, g2)| ≤ 6144|〈vν , v̇ν〉|.
Using the high probability deviations bound established in (E.64), it follows that if n is large enough then with probability at least 1− Ce−cn − C ′n−2d+1/2 we have
∀ν ∈ [0, π], |Ξ6(ν, g1, g2)| ≤ C ′′ √ d log n
n .
As long as d ≥ 12 , we have that this probability is at least 1 − Ce −cn − C ′n−d, and so the triangle inequality yields finally that with probability at least 1− Ce−cn − C ′n−d
∀ν ∈ [0, π], |Ξ6(ν, g1, g2)− E[Ξ6(ν, g1, g2)]| ≤ C ′′ √ d log n
n .
Lemma E.47. Consider the function
g(ν) = −(π2−[(π−ν) cos ν+sin ν]2)[(π−ν) cos ν−sin ν]+(π−ν)2[(π−ν) cos ν+sin ν] sin2 ν,
which is the negated numerator of ϕ̈. Then if 0 ≤ ν ≤ π/2, one has a bound
2π2
3 ν3 − 83 24 ν4 ≤ g(ν),
and the lower bound is positive if 0 < ν ≤ π/2.
Proof. To see that the lower bound is positive under the stated condition, write
2π2
3 ν3 − 83 24 ν4 = ν3
( 2π2
3 − 83 24 ν
) ;
the quantity in parentheses is positive in a neighborhood of zero by continuity, and in fact one calculates for its unique zero ν0 = 48π2/249, and one verifies numerically that 48π2/249 > 1.9 > π/2. We conclude that the bound is positive for 0 < ν < 1.9 by continuity.
To establish the bound, we employ Taylor expansion of the numerator, which is a smooth function on (0, π) with continuous derivatives of all orders on [0, π], in a neighborhood of zero. In our development in the proof of Lemma E.5, we showed that the analytic function −g(ν) = −(2π2/3)ν3 +O(ν4) near zero, so Taylor’s theorem with Lagrange remainder implies
2π3
3 ν3 +
ν4 24 inf ν∈[0,π/2] g(4)(ν) ≤ g(ν),
and so it suffices to get suitable bounds on the fourth derivative of g. We will develop the bounds rather tediously. Start by distributing in g to write
g(ν) = ν3 (− cos ν)︸ ︷︷ ︸ g3(ν) + ν2 (3π cos ν + sin ν)︸ ︷︷ ︸ g2(ν)
+ν ( cos ν − 2π2 cos ν − 2π sin ν − cos3 ν )︸ ︷︷ ︸ g1(ν)
+ ( π cos3 ν + 2π2 sin ν − sin3 ν − π cos ν )︸ ︷︷ ︸ g0(ν) .
Using the Leibniz rule, we have for the fourth derivative g(4)(ν) = ν3 ( g
(4) 3 (ν)
) + ν2 ( g
(4) 2 (ν) + 12g (3) 3 (ν)
) + ν ( g
(4) 1 (ν) + 8g (3) 2 (ν) + 36g (2) 3 (ν) ) + ( g
(4) 0 (ν) + 4g (3) 1 (ν) + 12g (2) 2 (ν) + 24g (1) 3 (ν)
) .
To calculate these derivatives, we just need to differentiate sin, cos, and their third powers. Write c(ν) = cos3(ν) and s(ν) = sin3(ν); using the elementary calculations
c(1)(ν) = 3s(ν)− 3 sin ν, c(2)(ν) = 6 cos ν − 9c(ν), c(3)(ν) = 21 sin ν − 27s(ν), c(4)(ν) = 60 cos ν + 81c(ν); s(1)(ν) = 3 cos ν − 3c(ν), c(2)(ν) = 6 sin ν − 9s(ν), c(3)(ν) = 27c(ν)− 21 cos ν, c(4)(ν) = 60 sin ν + 81s(ν),
(E.75)
one can calculate the results
g (4) 3 (ν) = − cos ν, g (4) 2 (ν) = 3π cos ν + sin ν,
g (4) 1 (ν) = (61− 2π2) cos ν − 2π sin ν − 81 cos3 ν,
g (4) 0 (ν) = (2π 2 − 60) sin ν + 50π cos ν + 81π cos3 ν − 81 sin3 ν; and
g (3) 3 (ν) = − sin ν, g (3) 2 (ν) = 3π sin ν − cos ν,
g (3) 1 (ν) = (7− 2π2) sin ν + 2π cos ν − 27 cos2 ν sin ν;
and g
(2) 3 (ν) = cos ν g (2) 2 (ν) = −3π cos ν − sin ν;
and finally g
(1) 3 (ν) = sin ν.
Plugging back into (E.75) and canceling, we get
g(4)(ν) = ν3 (− cos ν)︸ ︷︷ ︸ h3(ν) + ν2 (3π cos ν − 11 sin ν)︸ ︷︷ ︸ h2(ν)
+ν ( 22π sin ν + (89− 2π2) cos ν − 81 cos3 ν )︸ ︷︷ ︸ h1(ν)
+ ( 27 sin3 ν + 81π cos3 ν + 31π cos ν − (6π2 + 128) sin ν )︸ ︷︷ ︸ h0(ν) .
Since ν > 0, we can leverage lower bounds on each hi term. We have trivially |h3| ≤ 1, so that |ν3h3(ν)| ≤ π3/8. We will study νh1(ν) + h0(ν) together to get a better bound. We have
νh1(ν) + h0(ν) = ( 22πν − (6π2 + 128) ) sin ν + 27 sin3 ν + ( (89− 2π2)ν + 31π ) cos ν
+ (81π − 81ν) cos3 ν ≥ ( 22πν − (6π2 + 128) ) sin ν + 27 sin3 ν + ( (89− 2π2)ν + 31π ) cos ν︸ ︷︷ ︸
q(ν)
,
(E.76)
using ν ≤ π/2 and cos ≥ 0 on this domain. We will show that the RHS of the final inequality, denoted q, is a decreasing function of ν, and is therefore lower bounded by its value at ν = π/2 on our interval of interest. We calculate
q′(ν) = 9π sin ν + (42− 8π2) cos ν + 22πν cos ν − (89− 2π2)ν sin ν − 81 cos3 ν. Reordering terms, we can write
q′(ν) = −81 cos3 ν +  9π︸︷︷︸ C1 − (89− 2π2)︸ ︷︷ ︸ C2 ν  sin ν − (8π2 − 42)︸ ︷︷ ︸ C3 − 22π︸︷︷︸ C4 ν  cos ν. (E.77) We can estimate numerically
69 ≤ C2 ≤ 70; 69 ≤ C4 ≤ 70; C2 > C4, which shows that C1, C2, C3, C4 > 0 and both of the linear prefactors are decreasing functions of ν. We have on all of (0, π/2) by concavity of sin
(C1 − C2ν) sin ν ≤ ν ( C1 −
2C2 π ν
) ,
using in particular sin ν ≤ ν and sin ν ≥ (2/π)ν. Using similarly concavity of cos on this domain, in particular the inequalities cos ν ≤ π/2− ν and cos ν ≥ 1− (2/π)ν, we have
−(C3 − C4ν) cos ν ≤ − ( C4ν 2 − (
2C3 π + C4π 2
) ν + C3 ) .
In total, we have a bound q′(ν) ≤ −81 cos3 ν − (
2C2 π + C4
) ν2 + ( 2C3 π + πC4 2 + C1 ) ν − C3.
We calculate the maximizer of the concave quadratic function of ν in the previous bound via differentiation; plugging in, we get
q′(ν) ≤ −81 cos3 ν + ( 2C3 π + πC4 2 + C1 )2 4 (
2C2 π + C4 ) − C3. A numerical estimate gives (
2C3 π + πC4 2 + C1 )2 4 (
2C2 π + C4 ) − C3 ≤ 20, and using that − cos3 is strictly decreasing for ν < π, we can therefore guarantee q′ ≤ 0 as long as ν ≤ cos-1 3 √ 20/81. Writing c = cos-1 3 √ 20/81, we estimate numerically 0.90 ≥ c ≥ 0.89, so that this bound is nonvacuous. For ν ≥ c, we apply again concavity of cos to develop the lower bound
cos ν ≥ ( π/2− ν π/2− c ) cos c, ν ∈ [c, π/2].
Using this to estimate the − cos3 term in our upper bound for q′, we obtain a bound
q′(ν) ≤ −20 ( π/2− ν π/2− c )3 − ( 2C2 π + C4 ) ν2 + ( 2C3 π + πC4 2 + C1 ) ν − C3, c ≤ ν ≤ π/2.
We define D = 20/(π/2 − c)3, A = 2C2/π + C4, B = 2C3/π + πC4/2 + C1, and C = C3, so that the RHS can be written as −D(π/2− ν)3−Aν2 +Bν −C. Differentiating once and equating to zero results in the quadratic equation
3D ν2 − ( 2A3D + π )
︸ ︷︷ ︸ M ν +
( B
3D + π2/4 ) ︸ ︷︷ ︸
N  = 0, which has roots M/2 ± 12 √ M2 − 4N . Numerically estimating the constants, we get that the two roots lie in [0.99, 1] and [3.3, 3.4], so that we need only consider the smaller root. Differentiating once more to determine the class of the critical point, we find for the second derivative at M/2 − 1 2 √ M2 − 4N
−3D √ M2 − 4N < 0,
so that M/2 − 12 √ M2 − 4N is a maximizer for our cubic bound, and the bound is increasing for arguments less than this point and decreasing for arguments greater than it; we can conclude that the zero in [3.3, 3.4] is a minimizer, so that our bound can be ascertained negative by checking its value at M/2− 12 √ M2 − 4N . We find using a numerical estimate
−20
( π/2− (M/2− 12 √ M2 − 4N)
π/2− c )3 − (
2C2 π + C4 ) (M/2− 12 √ M2 − 4N)2
+ ( 2C3 π + πC4 2 + C1 ) (M/2− 12 √ M2 − 4N)− C3 ≤ −1.7 < 0,
which proves that q′ ≤ 0 on [c, π/2]. This shows that our lower bound on νh1(ν) + h0(ν) in (E.76) is nonincreasing on [0, π/2], so that we can assert
νh1(ν) + h0(ν) ≥ ( 22π(π/2)− (6π2 + 128) ) sin(π/2) + 27 sin3(π/2)
+ ( (89− 2π2)(π/2) + 31π ) cos(π/2)
= 5π2 − 101.
It remains to bound ν2h2(ν) = ν2(3π cos ν − 11 sin ν). On [0, π/2], cos is decreasing and sin is increasing, so 3π cos ν− 11 sin ν is decreasing here; it is positive at ν = 0 and negative at ν = π/2, so that by continuity it has a unique zero in (0, π/2). Denote this zero as ν0; then using that ν2 ≥ 0 with no zeros in the interior, we can write
inf 0≤ν≤ν0
ν2h2(ν) ≥ 0,
and
inf ν0≤ν≤π/2
ν2h2(ν) ≥ ( sup
ν0≤ν≤π/2 ν2
)( inf
ν0≤ν≤π/2 h2(ν) ) ≥ (π/2)2(3π cos(π/2)− 11 sin(π/2)) = −11π 2
4 ,
which gives the bound ν2h2(ν) ≥ −11π2/4 on [0, π/2]. Putting it all together, we have
g(4)(ν) ≥ −11π 2
4 + 5π2 − 101− π3/8 ≥ −83,
where the last inequality follows from a numerical estimate of the constants.
Lemma E.48 (Uniformization). Let (Ω,F ,P) be a complete probability space. For some t ∈ R, δt ≥ 0, S ⊂ Rd, and event E ∈ F , suppose that f : S × Ω → R is second-argument measurable and satisfies
1. For all x ∈ S, P[f(x, · ) ≤ t] ≥ 1− δt;
2. For all g ∈ E , f( · , g) is L-Lipschitz;
3. There is M > 0 such that supx∈S ‖x‖2 ≤M .
Then g 7→ supx∈S f(x, g) is measurable, and for every ε > 0, one has P [ sup x∈S f(x, · ) ≤ t+ Lε ] ≥ 1− δt ( 1 + 2M ε )d − P[E ]. (E.78)
Proof. Because S is a subset of the separable metric space (Rd, ‖ · ‖2) and all sample trajectories f( · , g) are assumed (Lipschitz) continuous, the supremum in the definition of g 7→ supx∈S f(x, g) can be taken on a countable subset of S, and the resulting function of g is measurable (e.g., (Ledoux & Talagrand, 1991, §2.2 p. 45)). By (Vershynin, 2018, Proposition 4.2.12) and boundedness of S, for every ε > 0 there exists an ε-net of S having cardinality at most (1 + 2M/ε)d; denote these nets as Nε. Since each Nε is finite, we may also define for each x ∈ S a point xε such that ‖x − xε‖2 ≤ ε; then for every g ∈ E , we have |f(x, g) − f(xε, g)| ≤ Lε. We define a collection of events Eε by Eε = {g ∈ Ω | ∀x ∈ Nε, f(x, g) ≤ t}. (E.79) The triangle inequality then implies that if g ∈ Eε∩E , then for all x ∈ S, one has f(x, g) ≤ t+Lε. Consequently, several union bounds yield
P [
sup x∈S
f(x, · ) > t+ Lε ] ≤ P [ sup x∈Nε f(x, g) ≤ t ] + P[E ]
≤ δt ( 1 + 2M
ε
)d + P[E ],
(E.80)
as claimed.

heading: E.4 DEFERRED PROOFS

text:
Proof of Lemma E.5. The function cos-1 isC∞ on (−1, 1), and because f(ν) := cosϕ(ν) is smooth and satisfies f ′(ν) = (π−1ν − 1) sin ν < 0 if ν < π with f(0) = 1 and f(π) = 0, we see that ϕ is C∞ on (0, π) by the chain rule. This also shows ϕ(0) = cos-1(1) = 0 and ϕ(π) = cos-1(0) = π/2. Direct calculation gives
ϕ̇(ν) =
√ (π − ν)2 sin2 ν
π2 − ((π − ν) cos ν + sin ν)2 (E.81)
and
ϕ̈(ν) = (π2 − [(π − ν) cos ν + sin ν]2)[(π − ν) cos ν − sin ν]
(π2 − [(π − ν) cos ν + sin ν]2)3/2
− (π − ν) 2[(π − ν) cos ν + sin ν] sin2 ν
(π2 − [(π − ν) cos ν + sin ν]2)3/2
(E.82)
Calculating endpoint limits using these expressions will suffice to show the derivatives are continuous on [0, π] and give the claimed values there. We have
lim ν↘0
( ϕ̇(ν) )2 = lim ν↘0 (π − ν)2 sin2 ν π2 − ((π − ν) cos ν + sin ν)2
= lim ν↘0 2(π − ν) sin ν[(π − ν) cos ν − sin ν] (−2)[(π − ν) cos ν + sin ν][cos ν − (π − ν) sin ν − cos ν]
= lim ν↘0 (π − ν) cos ν − sin ν (π − ν) cos ν + sin ν = 1,
by L’Hôpital’s rule, whereas a direct evaluation gives
lim ν↘0
( ϕ̇(ν) )2 = 0
π2 = 0.
Continuity of the square root function gives the claimed results for ϕ̇. Again by direct calculation, we find
lim ν↘0
( ϕ̈(ν) )2 = 0
π3 = 0.
Since ϕ̇2 is meromorphic in a neighborhood of 0 with, as we have shown, a removable singularity at 0, it is actually analytic, and we can calculate further derivatives at 0 by expanding it locally at 0. We use the expansions sin ν = ν − ν3/6 + O(ν5) and cos ν = 1− ν2/2 + ν4/24 + O(ν6) near 0 to calculate (
1− ν π
)2 sin2 ν = ν2 ( 1− 2
π ν − π 2 − 3 3π2 ν2 +O(ν3) ) and
1− ((
1− ν π
) cos ν + sin ν
π
)2 = ν2 ( 1− 2
3π ν − 1 3 ν2 +O(ν3)
) ,
from which it follows
(ϕ̇(ν)) 2 = ( 1− 2
π ν − π 2 − 3 3π2 ν2 +O(ν3)
)( 1− 2
3π ν − 1 3 ν2 +O(ν3)
)−1 .
By the geometric series, we then obtain
(ϕ̇(ν)) 2 = 1− 4 3π ν + 1 9π2 ν2 +O(ν3).
Taking the square root of this expression and applying the binomial series, we thus have
ϕ̇(ν) = 1− 2 3π ν − 1 6π2 ν2 +O(ν3),
from which we read off
lim ν↘0 ϕ̈(ν) = − 2 3π ; lim ν↘0 ... ϕ(ν) = − 1 3π2 .
It is clear from the analytical expression for ϕ̇ and the mean value theorem thatϕ is strictly increasing on [0, π], since (π − ν) sin ν > 0 if 0 < ν < π. To prove strict concavity for ν ∈ (0, π), we start by simplifying notation. Consider the function ϕr(ν) = ϕ(π − ν), which satisfies by the chain rule ϕ̈r(ν) = ϕ̈(π − ν). Because ϕr is strictly concave if and only if ϕ is strictly concave, it suffices to prove that ϕ̈(π − ν) < 0. We note
ϕ̈(π − ν) < 0 ⇐⇒ (π2 − [ν cos ν − sin ν]2)(− sin ν − ν cos ν) < ν2 sin2 ν(sin ν − ν cos ν). Multiplying both sides of the latter inequality by sin ν − ν cos ν, dividing through by (ν cos ν − sin ν)2 (which is positive on (0, π), since it equals cos2 ϕ composed with a reversal about π), and distributing and moving terms to the RHS gives the equivalent condition
π2 ν2 cos2 ν − sin2 ν (ν cos ν − sin ν)2 < ν2 − sin2 ν,
and canceling once more gives equivalently
ν cos ν + sin ν ν cos ν − sin ν < ν2 − sin2 ν π2 . (E.83)
Using ν cos ν − sin ν < 0, which follows from its derivative −ν sin ν being negative on (0, π), and writing g(ν) = π−2(ν2− sin2 ν), we have equivalently ν cos ν+sin ν > g(ν)(ν cos ν− sin ν), and rearranging gives the inequality
(1− g(ν))ν cos ν + g(ν) sin ν > − sin ν. (E.84) Strict concavity of sin on (0, π) gives sin ν < ν, and 0 < g(ν) < 1 follows after squaring; so the LHS is a convex combination of ν cos ν and sin ν, which in particular satisfies |(1− g(ν))ν cos ν + g(ν) sin ν| ≤ max{|sin ν|, |ν cos ν|}. As argued before, we have sin ν − ν cos ν > 0 if ν ∈ (0, π); moreover, because ν > 0 we have ν cos ν > 0 if ν ∈ (0, π/2) and ν cos ν < 0 if ν ∈ (π/2, π). We can numerically determine sin(5π/8)+(5π/8) cos(5π/8) > 0, and given that 5π/8 ≥ 1.95 > π/2, it follows |(1− g(ν))ν cos ν + g(ν) sin ν| < |sin ν|, 0 < ν ≤ 1.95, which implies (E.84) when 0 < ν ≤ 1.95. Recalling that we are arguing for ϕr in this setting, we translate our results back to ϕ and conclude that ϕ(ν) < 0 if π − 1.95 ≤ ν < π. To address the case where 0 < ν < π − 1.95, we employ Lemma E.47; it allows us to conclude ϕ̈ < 0 provided 0 < ν ≤ π/2, and a numerical estimate gives that π − 1.95 < π/2, so that we have ϕ̈ < 0 for all 0 < ν < π. Taking limits in ϕ gives concavity at the endpoints {0, π} as well. To bound ϕ̈ away from zero on [0, π/2], we apply Lemma E.47 to assert
ϕ̈(ν) ≤ − 23πν 3 + 8324π3 ν 4
(1− cos2 ϕ(ν))3/2 , 0 < ν ≤ π/2.
The numerator in the last expression is nonpositive if 0 ≤ ν ≤ π/2, and using the lower bound in Lemma E.14 on [0, π/2], we have
1 1− cos2 ϕ(ν) ≥ 1
1−max2{1− 12ν2, 0} , ν > 0.
From nonpositivity of the numerator, it follows
ϕ̈(ν) ≤ − 23πν 3 + 8324π3 ν 4( 1−max2{1− 12ν2, 0} )3/2 , 0 < ν ≤ π/2. (E.85)
We have 1 − 12ν 2 ≥ 0 as long as 0 ≤ ν ≤
√ 2; so after removing the max, distributing, and
cancelling, we have
ϕ̈(ν) ≤ − 23π + 83 24π3 ν( 1− 14ν2 )3/2 , 0 < ν ≤ √2.
The denominator of this last expression is nonnegative and has singularities at ±2, and is clearly even symmetric; so it is maximized on 0 < ν ≤ √ 2 at √ 2, and we have
ϕ̈(ν) ≤ √ 8 ( − 2
3π +
83
24π3 ν
) , 0 < ν ≤ √ 2.
Taking limits ν ↘ 0, we can assert this bound on [0, √
2], and the bound is clearly an increasing function of ν, from which it follows
sup ν∈[0, √ 2]
ϕ̈(ν) ≤ √ 8 ( − 2
3π +
83 √ 2
24π3
) ≤ −0.15,
where the last inequality follows from a numerical estimate of the constants. On the other hand, when √ 2 < ν ≤ π/2, we have from (E.85) that
ϕ̈(ν) ≤ − 2 3π ν3 + 83 24π3 ν4,
√ 2 ≤ ν ≤ π/2.
If we differentiate the degree four polynomial on the RHS of this bound and solve for critical points, we find a double critical point at ν = 0 and a critical point at ν = 12π2/83; a numerical estimate confirms that this critical point lies in the interior of [ √ 2, π/2]. The second derivative of the RHS is −(4/π)ν + 83/(2π3)ν2, and plugging in ν = 12π2/83 gives a value of −48π/83 + 144π/83, which is positive; hence the RHS is maximized on the boundary, i.e.,
ϕ̈(ν) ≤ − 2 3π ν3 + 83 24π3 ν4 ≤ max
{ −2 5/2
3π +
83 6π3 ,−π
2
12 +
83π
384
} , √ 2 ≤ ν ≤ π/2.
A numerical estimate shows that the RHS of the last inequality is no larger than −0.14. Since the intervals we have proved a bound over cover [0, π/2], this proves the claim with c = −0.14. The bound ϕ̇ < 1 on (0, π) follows from the fact that ϕ is strictly concave on (0, π) and the mean value theorem; we have already shown ϕ̇ > 0 in proving strict increasingness of ϕ. Similarly, the proof of strict concavity in the interior has already established ϕ̈ < 0. To obtain the lower bound on ϕ̈, we use that ϕ̈ is continuous on [0, π] and the Weierstrass theorem to assert that there is C ≥ 0 such that ϕ̈ ≥ −C on [0, π]; because ϕ̈(0) 6= 0, we actually have C > 0. For the quadratic model, we use our previous results and Taylor expand ϕ about 0; we get immediately
ϕ(ν) ≥ ν + ν2 infν∈[0,π] ϕ̈(ν)
2 ≥ ν − (C/2)ν2.
For the upper bound, we can assert immediately on [0, π/2] a bound
ϕ(ν) ≤ ν − cν2,
where c = 0.07 suffices. To extend the bound to ν ∈ [π/2, π], we employ a bootstrapping argument; because ϕ is concave, we have a bound
ϕ(ν) ≤ ϕ(π/2) + ϕ̇(π/2)(ν − π/2)
= cos-1 π−1 + π/2√ π2 − 1 (ν − π/2) ,
where the second line plugs into the formulas for ϕ and ϕ̇. We will show that the graph of ν − cν2 lies entirely above the graph of the RHS of this inequality. This condition is equivalent to
−cν2 + (
1− π/2√ π2 − 1
) ν + ( (π/2)2√ π2 − 1 − cos-1 π−1 ) ≥ 0;
the LHS of this inequality is a concave quadratic with maximizer ν? = 1/(2c) ( 1 − π/2√
π2−1
) , and
numerical estimation of the constants gives ν? ≥ π. Since ν? is outside [π/2, π] and the quadratic is concave, we conclude that the bound is tightest at the boundary point π/2, and one checks numerically
−cπ2/4 + (
1− π/2√ π2 − 1
) π/2 + ( (π/2)2√ π2 − 1 − cos-1 π−1 ) ≥ 0.15 > 0,
which establishes that the bound ϕ(ν) ≤ ν − cν2 actually holds on all of [0, π]. This completes the proof of all of the claims.

heading: F CONTROLLING CHANGES DURING TRAINING

heading: F.1 PRELIMINARIES

text:
We now consider the changes in the integral operator Θk during gradient descent. In this section we restore the iteration subscript (that is dropped in other sections to lighten notation) to various quantities. Θk changes during training as a result of both smooth changes in the features at all layers and non-smooth changes in the backward features {β`t(x)} due to the non-smoothness of the derivative of the ReLU function.
Because of the difficulty of reasoning precisely about the changes in Θt, we will bound these rather naively by controlling Θt over all possible support patterns of the features given a bound on the norm change of the pre-activations.
We now define a trajectory in parameter space that interpolates between the iterates of gradient descent, given for any k′ ∈ {0, . . . , k} and s ∈ [0, 1] by
θNk′+s = θ N k′ − τs∇̃LN (θNk′ ), (F.1)
(with the formal derivative ∇̃ defined in Appendix A.1). We will henceforth use k′ to denote an integer indexing the iteration number and t to denote a continuous parameter taking values in [0, k] (such that k′ = btc , s = t − btc ). Quantities indexed by t are ones where the parameters take the value θNt . To lighten notation, we will drop the N superscript when referring to time-indexed quantities (aside from ζNk and Θ N k ), but all such quantities depend on the parameters as defined by (F.1).
Instead of considering the change in the features { α`t(x) } directly, it will be more convenient to work in terms of the pre-activations, which are given at layer ` by
ρ`t(x) = W ` t PI`−1,t(x)W `−1 t PI`−2,t(x)W `−2 t . . .PI1,t(x)W 1 t x.
We define a maximal allowable change in the pre-activation norm by
η = CηL
3/2+q
√ n
(F.2)
for q ≥ 0 and a constant Cη to be specified later, where the scaling is chosen with foresight. We can then define a maximal number of iterations such that the pre-activation norms at all layers along the trajectory (F.1) change by no more than η. This number kη must satisfy
sup t∈[0,kη ],x∈M,`∈[L] ∥∥ρ`t(x)− ρ`0(x)∥∥2 ≤ η (F.3) for η given by (F.2). Our goal will be to show that we can in fact train for long enough so as to reduce the fitting error without exceeding kη iterations.

heading: F.2 CHANGES IN FEATURE SUPPORTS DURING TRAINING

text:
Recalling the definition of the feature supports at layer `, time t and x ∈ M by I`,t(x) = supp(α`t(x) > 0) ⊆ [n], we denote by It(x) = (I1,t(x), . . . , IL,t(x)) the collection of these support patterns at all layers. We would next like to relate the smooth changes in the pre-activation norms to the non-smooth changes in the supports of the features. We denote by J = (J1, . . . , JL) a collection of support patterns with Ji ∈ [n]. We now consider sets of support patterns that are not too different from those at initialization, as defined by
B(y, η) = {supp (y + v > 0) | ‖v‖2 ≤ η} , (F.4)
J η(x) = ⊗ `∈[L] B(ρ`0(x), η), (F.5)
J η(M) = ⋃ x∈M J η(x). (F.6)
Note that B(ρ`0(x), η) is simply the set of supports of the positive entries of ρ`0(x) + v for every possible perturbation v of norm at most η. We consider all possible perturbations due to the complex
nature of the training dynamics. As a result of this worst-casing, the scaling we will require of the depth and width of the network in order to guarantee that the changes during training are sufficiently small is expected to be suboptimal.
For a given general support patternJ , we define generalized backward features and transfer matrices β`J t = ( WNL+1t PJLW NL t . . .W N`+2 t PJ`+1 )∗ ,
Γ̃`:` ′
J t = W ` t PJ`−1W N`−1 t . . .PJ`′W
N`′ t
(F.7)
where the weights are given by (F.1) (and thus β`t(x) = β N` It(x)t). By controlling these objects for every possible set of supports J that can be encountered during training, we can control the smooth changes in the features themselves. A first step towards this end is understanding how many such support patterns we expect to see given the constraint in (F.2).
In order to bound the number supports that can be encountered during training, we need to control the diameter of B(ρ`0(x), η). This can be done by defining
δη(ρ ` 0(x)) = max‖v‖2≤η ∣∣supp (ρ`0(x) > 0) supp (ρ`0(x) + v > 0)∣∣ (F.8) where denotes the symmetric difference. Since the pre-activation at a given layer are Gaussian variables conditioned on all the previous layer weights, bounding the size of δη(ρ`0(x)) can be reduced to showing concentration of a certain function of Gaussian order statistics. This is achieved in the following lemma :
Lemma F.1. For η given by (F.2), if n,L, d satisfy the requirements of lemma F.6 and n > d5 for some constant K, then for a vector g ∈ Rn, gi ∼iid N (0, 1n ) we have
P [ δη(g) > Cnη 2/3 ] ≤ C ′e−cd
for some constants c, C,C ′.
Proof. Let |g|(1) ≤ |g|(2) ≤ · · · ≤ |g|(n)
denote the order statistics of the magnitudes of the elements of g. We will show that bounding δη(g) can be reduced to understanding what is the smallest k such that
|g|2(1) + · · ·+ |g| 2 (k) ≥ η 2.
We denote this value of k by kη . Define indices ji by |gJi | = |g|(i) (and breaking ties arbitrarily in case several order statistics are equal). To see that
kη − 1 ≤ δη(g) ≤ kη (F.9)
it suffices to note that since kη−1∑ i=1 |g|2(k) < η 2 one can choose ε > 0 small enough such that
y = g − kη−1∑ i=1 (1 + ε)gJieJi ∈ BE(g, η),
which will give ds(g,y) = kη − 1 ⇒ δη ≥ kη − 1. To prove the second inequality, consider y = g − δη∑ i=1 gJieJi . Clearly for any y ′ such that ds(g,y′) = δη we have ‖g − y‖2 ≤ ‖g − y′‖2. Since there exists at least one such y′ ∈ BE(g, η), it follows that ‖g − y‖2 ≤ ‖g − y′‖2 ≤ η, and
hence the smallest k such that k∑ i=1 ∣∣g(i)∣∣2 ≥ η2 must obey k ≥ δη . For η defined in (F.2), we can satisfy the requirement on η in lemma F.6 by requiring n > d5 for some K. Applying this lemma, we find that there is a constant K ′ such that for k = ⌈ K ′nη2/3 ⌉ we have
P [ k∑ i=1 |g|2(i) ≥ η 2 ] ≥ 1− C ′e−cd
from which it follows immediately that P [ kη ≤ ⌈ K ′nη2/3 ⌉] ≥ 1− C ′e−cd.
Choosing some constant C such that ⌈ K ′nη2/3 ⌉ ≤ Cnη2/3 and using (F.9) allows us to bound
δη(g) with the same probability.
With this result in hand, we can control the objects in (F.7) for all the supports in J η(M).
Lemma F.2. Assume d, L, n satisfy the assumptions of lemmas D.2, F.1, D.8, D.14 and additionally n ≥ max { KdL9+2q,K ′ (log n)
3/2 , C30C 2 ηL
6+2q } ,
for some constants K,K ′, C0, where q is the constant in (F.2).

heading: Then

text:
i) for η, J η(x) defined in (F.2),(F.5), on an event of probability at least 1− e−cd, simultaneously
sup x∈M sup J∈J η(x) sup `∈[L]
1≤`′<` ∥∥ρ`0(x)∥∥2 ≤ C2, sup x∈M sup J∈J η(x) sup `∈[L]
1≤`′<`
∥∥∥β`−1J ,0∥∥∥ 2 ≤ C2 √ n,
sup x∈M sup J∈J η(x) sup `∈[L]
1≤`′<` ∥∥∥Γ̃`:`′J ,0∥∥∥ ≤ C2√L. ii) for Tη defined in (F.3), on an event of probability at least 1− e−cd,
sup x∈M,t∈[0,Tη ],`∈[L] ∥∥∥β`−1I0(x),0 − β`−1It(x),0∥∥∥2 ≤ CC2/3η log3/4(L)d3/4L3+2q/3n5/12. for some constants c, C.
Proof. Deferred to F.5.

heading: F.3 CHANGES IN FEATURES DURING TRAINING

text:
We can now bound the smooth changes during training:
Lemma F.3 (Smooth changes during training). Set the step size τ and a bound on the maximal number of iterations kmax such that
kmaxτ = Lq
n for some constant q. Assume n,L, d satisfy the requirements of lemmas F.2, and in particular n ≥ KdL9+2q for some K. Assume also that given some k ≤ kmax − 1, for all k′ ∈ {0, . . . , k},∥∥ζNk′ ∥∥L2
µN
≤ C √ d. (F.10)
Then on an event of probability at least 1− e−cd, one has simultaneously
sup x∈M, `∈[L], k′∈{0,...,k+1}
∥∥ρ`k′(x)− ρ`0(x)∥∥2 ≤ C ′L3/2+q √ d
n ,
sup x∈M, `∈[L], k′∈{0,...,k+1} (∥∥β`−1k′ (x)− β`−10 (x)∥∥2 − ∥∥∥β`−1Ik′0(x)− β`−1I00 (x)∥∥∥2) ≤ C ′√dL3/2+q, for some constants c, C,C ′.
Proof. We will bound the smooth changes in the network features during gradient descent with respect to either the population measure µ∞ or the finite sample measure µN . We denote a measure that can be one of these two by µN .
For any collection of supports J ∈ J η(M), define generalized backward features and transfer matrices at t by β`J t, Γ̃ `:`′ J t . These are obtained by setting the network parameters to be θ N t according to (F.1), but setting all the support patterns to be those in J . We then define for any t ∈ [0, k + 1],
ρηt = sup x∈M,`∈[L],t′∈[0,t] ∥∥ρ`t′(x)− ρ`0(x)∥∥2 + sup x∈M,`∈[L] ∥∥ρ`0(x)∥∥2 , β η
t = sup `∈[L],J∈J η(M),t′∈[0,t] ∥∥β`J t′ − β`J 0∥∥2 + sup `∈[L],J∈J η(M) ∥∥β`J 0∥∥2 , Γ η
t = sup `′≤`∈[L],J∈J η(M),t′∈[0,t] ∥∥∥Γ̃`:`′J t′ − Γ̃`:`′J 0∥∥∥+ sup `′≤`∈[L],J∈J η(M) ∥∥∥Γ̃`:`′J 0∥∥∥ . (F.11)
We have for all k′ ≤ k + 1, ∥∥ρ`k′(x)− ρ`0(x)∥∥2 ≤ ρηk′ − ρη0 , (F.12) while ∥∥β`k′(x)− β`0(x)∥∥2 =∥∥∥β`Ik′ (x),k′ − β`I0(x),0∥∥∥2
≤ ∥∥∥β`Ik′ (x),k′ − β`Ik′ (x),0∥∥∥2 + ∥∥∥β`Ik′ (x),0 − β`I0(x),0∥∥∥2 ≤βηk′ − β η 0 + ∥∥∥β`Ik′ (x),0 − β`I0(x),0∥∥∥2 .
(F.13)
It follows that we can control the difference norms of the pre-activations and backward features by controlling the magnitudes of ρηk′ , β η
k′ . In order to control these, we also note that for any t ∈ [0, k + 1], ∥∥α`t(x)∥∥2 ≤ ∥∥ρ`t(x)∥∥2 ≤ ∥∥ρ`t(x)− ρ`0(x)∥∥2 + ∥∥ρ`0(x)∥∥2 ≤ ρηt , (F.14) and similarly ∥∥β`J t∥∥2 ≤βηt ,∥∥∥Γ̃`:`′J t ∥∥∥ ≤Γηt . (F.15) In particular, the above bounds hold when J = It(x). We would now like to understand how the quantities ( ρηk′ , β η k′ ,Γ η k′ ) evolve under gradient descent. Towards this end, for any k′ ∈ {0, . . . , k} and s ∈ [0, 1] we compute at any point of differentiability
∂̃
∂s ρ`k′+s(x)
=− τ ∂̃ρ `(x)
∂θ ∣∣∣∣∣ ∗
θk′−sτ∇̃LN (θNk′ )
∇̃LN (θNk′ )
=− τ
( ∂̃ρ`k′+s(x)
∂θ
)∗ ∇̃LN (θNk′ )
=− τ ∑̀ i=1 ni∑ j=1 ni−1∑ l=1 ∂̃ρ`k′+s(x) ∂W ijl ∂̃LN (θNk′ ) ∂W ijl =− τ ∑̀ i=1 ∫ x′∈M 〈 αi−1k′+s(x),α i−1 k′ (x ′) 〉 Γ̃`:i+1k′+s (x)PIi,k′+s(x)β i−1 k′ (x ′)ζNk′ (x ′)dµN (x′).
Using (F.14) and (F.15) then gives∥∥∥∥∥ ∂̃∂sρ`k′+s(x) ∥∥∥∥∥
2
≤τL ( ρηk′+s )2 β η k′+sΓ η k′+s ∫ x′∈M ∣∣ζNk′ (x′)∣∣ dµN (x′) ≤τL ( ρηk′+s )2 β η k′+sΓ η k′+s ∥∥ζNk′ ∥∥L2 µN
≤ C √ dτL ( ρηk′+s )2 β η k′+sΓ η k′+s,
where we used Jensen’s inequality in the second line and our assumption that the error up to iteration k has bounded L2µN norm, and we additionally assumed ρ η t , β η t ,Γ η
t ≥ 1. Arguing as in the proof of Lemma B.8 for absolute continuity, it follows that
∥∥ρ`t(x)− ρ`0(x)∥∥2 ≤btc−1∑ k′=0 ∥∥ρ`k′+1(x)− ρ`k′(x)∥∥2 + ∥∥∥ρ`t(x)− ρ`btc(x)∥∥∥2 =
btc−1∑ k′=0 ∥∥∥∥∥∥ 1∫ 0 ∂̃ ∂s ρ`k′+s(x)ds ∥∥∥∥∥∥ 2 + ∥∥∥∥∥∥∥ t−btc∫ 0 ∂̃ ∂s ρ`btc+s(x)ds ∥∥∥∥∥∥∥ 2
≤ btc−1∑ k′=0 1∫ 0 ∥∥∥∥∥ ∂̃∂sρ`k′+s(x) ∥∥∥∥∥ 2 ds+ t−btc∫ 0 ∥∥∥∥∥ ∂̃∂sρ`btc+s(x) ∥∥∥∥∥ 2 ds ≤C √ dτtL (ρηt ) 2 β η tΓ η t
(F.16)
Since the above holds for all choices of x, ` simultaneously, we conclude that
ρηt − ρ η 0 ≤ C √ dτtL (ρηt ) 2 β η tΓ η t . (F.17)
An analogous calculation for the other quantities in (F.11) gives the following set of coupled difference inequalities:  ρηt − ρη0βηt − βη0
Γ η t − Γ η 0
 ≤ C√dLτtρηt βηtΓηt  ρηtβηt
Γ η
t  . (F.18) Instead of solving (F.18), we obtain sufficient control by defining k∗ s.t.
∀t ∈ [0, k∗] : ρηt ≤ 2ρ η 0 ∧ β
η t ≤ 2β η 0 ∧ Γ η t ≤ 2Γ η 0 . (F.19)
For any t ∈ [0, k∗], we obtain a sufficient condition for satisfying the above constraint using (F.18), namely
ρη0 + C ′ √ dτtL (ρη0) 2 β η 0Γ η 0 ≤2ρ η 0
⇔ τt ≤ 1 C ′ √ dLρη0β η 0Γ η 0
(F.20)
for some constant C ′. Using the bounds for β η t −β η 0 and Γ η t −Γ η
0 in (F.18) to satisfy the second and third condition in (F.19) gives an identical constraint on τt.
In order to control these quantities at t = 0 we define an event
G = ⋂ x∈M, J∈J η(x), 1≤`′≤`∈[L]
{∥∥ρ`0(x)∥∥2 ≤ C2} ∩ {∥∥∥β`−1J ,0∥∥∥
2 ≤ C2
√ n }
∩ {∥∥∥Γ̃`:`′J ,0∥∥∥ ≤ C2√L} , (F.21)
the probability of which can be controlled using lemma F.2. On G, the upper bound on τt in (F.20) is at least
1
C ′ √ dLρη0β η 0Γ η 0 ≥ 1 C ′′ √ dL3/2 √ n
(F.22)
for some C ′′.
We would now like to pick τkmax, and ensure that any t ∈ [0, k+ 1] satisfies the constraint above if k+ 1 ≤ kmax. The analysis also assumes that τkmax ≤ Tη for which (F.3) holds. We will then pick the scaling factor Cη for the pre-activation norm bound in (F.2) in order to satisfy that constraint as well. We choose
τkmax = Lq
n . (F.23)
In order to ensure that kmax ≤ k∗ holds, we use (F.20) and (F.22), and require
Lq
n ≤ 1 C ′′ √ dL3/2 √ n
which is satisfied by demanding n ≥ KdL3+2q for some constant K. Using (F.17) and (F.22), on G we have
sup t∈[0,k+1],x∈M,`∈[L] ∥∥ρ`t(x)− ρ`0(x)∥∥2 ≤ ρηt − ρη0 ≤C ′τt √ dL (ρη0) 2 β η 0Γ η 0
≤C ′′τt √ dL3/2 √ n ≤C ′′τkmax √ dL3/2 √ n.
In order to ensure that kmax ≤ kη we therefore require
C ′′τkmax √ dL3/2 √ n ≤ η,
which using (F.2) and (F.23) is equivalent to
C ′′ √ dLq+3/2√
n ≤ Cη Lq+3/2√ n ,
and thus the constraint can be satisfied by choosingCη = C ′′ √ d. Note that the constantC2 in (F.21) (which enters C ′′) is set in lemma F.2 which takes Cη as input (despite this, C2 is independent of Cη). This lemma holds as long as n ≥ C30C2ηL6+2q , which we can guarantee by demanding
n ≥ C30 (√ dC ′′ )2 L6+2q.
We have thus ensured that our choice of kmax satisfies kmax ≤ min {k∗, kη}. We then obtain from the constraints in (F.19), the inequalities in (F.18) and the definition of G, that on this event
sup k′∈{0,...,k+1}
ρηk′ − ρ η 0 ≤ C ′′τkmax
√ dL3/2 √ n ≤ C ′′ √ dL3/2+q√
n ,
sup k′∈{0,...,k+1}
β η k′ − β η 0 ≤ C ′′τkmax √ dL3/2n ≤ C ′′ √ dL3/2+q.
Then using (F.12) and (F.13), we obtain on an event of probability at least 1− e−cd simultaneously
sup x∈M, `∈[L], k′∈{0,...,k+1}
∥∥ρ`k′(x)− ρ`0(x)∥∥2 ≤ C ′L3/2+q √ d
n ,
sup x∈M, `∈[L], k′∈{0,...,k+1} (∥∥β`−1k′ (x)− β`−10 (x)∥∥2 − ∥∥∥β`−1Ik′0(x)− β`−1I00 (x)∥∥∥2) ≤ C ′√dL3/2+q.
The combination of the last two lemmas allows us to control the changes in all the forward and backward features uniformly:
Lemma F.4. Assume n,L, d, k satisfy the requirements of lemmas F.2 and F.3, and additionally n ≥ KL36+8qd9 for some K. Then one has simultaneously on an event of probability at least 1− e−cd
sup x∈M, t∈[0,k+1], `∈[L]
∥∥α`t(x)−α`0(x)∥∥2 ≤ CL3/2+q √ d
n ,
sup x∈M, t∈[0,k+1], `∈[L] ∥∥β`−1t (x)− β`−10 (x)∥∥2 ≤ C log3/4(L)d3/4L3+2q/3n5/12, sup
x∈M, t∈[0,k+1], `∈[L] ∥∥α`t(x)∥∥2 ≤ C, sup
x∈M, t∈[0,k+1], `∈[L] ∥∥β`−1t (x)∥∥2 ≤ C√n, for some constants c, C.
Proof. Combine the results of lemmas F.2 and F.3 and take a union bound, using the triangle inequality to obtain the second two bounds. The assumption n ≥ KL36+8qd9 is required in showing∥∥β`−1t (x)∥∥2 ≤ C√n. F.4 CHANGES IN ΘNk DURING TRAINING
With these results in hand, control of the changes in ΘNk during training is straightforward.
Lemma F.5 (Uniform control of changes in Θ during training). Denoting the gradient descent step size by τ , choose some kmax such that
kmaxτ = Lq
n
for some constant q. Assume also that given some k ≤ kmax − 1, for all k′ ∈ {0, . . . , k},∥∥ζNk′ ∥∥L2 µN ≤ √ d. (F.24)

heading: Define

text:
Θ(x,x′) = 〈 αL0 (x),α L 0 (x ′) 〉 + L−1∑̀ =0 〈 α`0(x),α ` 0(x ′) 〉 〈 β`0(x),β ` 0(x ′) 〉 ,
∆̃Nk = sup (x,x′)∈M×M, k′∈{0,...,k} ∣∣ΘNk′(x,x′)−Θ(x,x′)∣∣ . Assume n ≥ KL36+8qd9, d ≥ K ′d0 log (nn0CM) for constants K,K ′. Then on an event of probability at least 1− e−cd
∆̃Nk ≤ C log 3/4(L)d3/4L4+2q/3n11/12
for some constants c, C.
Proof. Recall that
ΘNk (x,x ′) = 1∫ s=0 ∂f̃θ(x) ∂θ ∣∣∣∣∣ θNk+s ds ∂f̃θ(x) ∂θ ∣∣∣∣∣ θNk
= L∑ `=0 1∫ s=0 〈 α`k+s(x),α ` k(x ′) 〉 〈 β`k+s(x),β ` k(x ′) 〉 ds
with the convention βLt (x) = 1 for all t,x, and the parameters θ N t given by (F.1). We thus have
∣∣ΘNk (x,x′)−Θ(x,x′)∣∣ ≤ L∑ `=0 1∫ s=0 ∣∣∣∣ 〈α`k+s(x),α`k(x′)〉 〈β`k+s(x),β`k(x′)〉− 〈α`0(x),α`0(x′)〉 〈β`0(x),β`0(x′)〉 ∣∣∣∣ds.
We consider a single summand in the above expression. On the event defined in lemma F.4, for all x,x′ ∈M, ` ∈ {0, . . . , L},∣∣〈α`k+s(x),α`k(x′)〉 〈β`k+s(x),β`k(x′)〉− 〈α`0(x),α`0(x′)〉 〈β`0(x),β`0(x′)〉∣∣
≤  ∣∣〈α`k+s(x)−α`0(x),α`k(x′)〉 〈β`k+s(x),β`k(x′)〉∣∣ + ∣∣〈α`0(x),α`k(x′)−α`0(x′)〉 〈β`k+s(x),β`k(x′)〉∣∣ + ∣∣〈α`0(x),α`0(x′)〉 〈β`k+s(x)− β`0(x),β`k(x′)〉∣∣
+ ∣∣〈α`0(x),α`0(x′)〉 〈β`0(x),β`k(x′)− β`0(x′)〉∣∣  ≤C ( n ∥∥α`k+s(x)−α`0(x)∥∥2 + n∥∥α`k(x′)−α`0(x′)∥∥2
+ √ n ∥∥β`k+s(x)− β`0(x)∥∥2 +√n∥∥β`k(x′)− β`0(x′)∥∥2 ) ≤C ′ ( L3/2+q √ dn+ log3/4(L)d3/4L3+2q/3n11/12
) ≤C ′′ log3/4(L)d3/4L3+2q/3n11/12,
for some constants. Summing this bound over ` gives the desired result.

heading: F.5 AUXILIARY LEMMAS AND PROOFS

text:
Lemma F.6. Consider a collection of n i.i.d. variables Xi = g2i , gi ∼ N (0, 1n ) and denote the order statistics by X(i) (so that X(1) ≤ X(2) . . . ). For any d ≥ K ′ log n, n ≥ K ′′d3, η > C d 9/8
n3/4
and integer Knη2/3 ≤ k ≤ n, where K,K ′,K ′′ are appropriately chosen absolute constants, we have
P [ k∑ i=1 X(i) ≥ η2 ] ≥ 1− C ′e−cd,
where c, C,C ′ are absolute constants.
Proof. We will relate sums of order statistics ofXi to functions of uniform order statistics and show that these concentrate. We denote the CDF of the Xi and its inverse by F and F−1 respectively. We use
(X(1), ..., X(k)) d = (F−1(U(1)), ..., F −1(U(k)))
where U(i) are order statistics with respect to Unif(0, 1) (David, 2011). Since Xi ∼ 1nYi, Yi ∼ χ 2 1 we have
F (x) = erf( √ nx
2 )
F−1(t) = 2 n (erf−1(t))2 ≥ c0 n t2
where in the inequality we used the series representation of erf−1. This gives
k∑ i=1 X(i) d = k∑ i=1 F−1(U(i)) ≥ c0 n k∑ i=1 U2(i).
The joint PDF of the first k order statistics for any distribution admitting a density is given by
f(1)...(k)(x1, ..., xk) = n! (n− k)! (1− F (xk))n−k k∏ i=1 f(xi)
where x1 ≤ x2 ≤ ... ≤ xk (David, 2011). Applying this to the uniform order statistics, we can compute the mean of the summands
EU2(i)
= n!
(n− k)! u2∫ 0 ... uk∫ 0 1∫ 0 u2i (1− uk) n−k duk...du1 = n!i(i+ 1) (n− k)!(k + 1)! 1∫ 0 uk+1k (1− uk) n−k duk
= i(i+ 1)
(n+ 2)(n+ 1)
E k∑ i=1 U2(i) = k(k + 1)(k + 2) 3(n+ 2)(n+ 1) ≥ c1k 3 n2 .
In order to show concentration, we appeal to the Rényi representation of order statistics (Boucheron et al., 2012). This allows us to write k∑ i=1 U2(i) as a Lipschitz function of independent exponential random variables, and we can then apply standard concentration results for such functions (Talagrand, 1995). This representation is due to a useful property unique to the exponential distribution whereby the differences between order statistics are independent exponentially distributed variables themselves when properly normalized.
If we define by Ei, ..., En a collection of independent standard exponential variables, the Rényi representation of the uniform order statistics gives
(U(1), ..., U(k)) d = (1− exp ( −E1 n ) , ..., 1− exp − k∑ j=1 Ej n− j + 1 ). We now truncate the (E1, ..., Ek), so that w.p. P ≥ 1 − ke−K we have ∀i : Ei ∈ [0,K], and
denote this event by EK . Using K < n2 , it is evident that k∑ i=1 U2(i) is equal in distribution to a convex function of (E1, ..., Ek) after truncation (which can be seen by calculating second derivatives). The Lipschitz constant of this function is bounded by 4kn−k . = λ.
If we define rescaled variables Ẽi = λEi then with the same probability they take values in [0,Kλ]. k∑ i=1 U2(i) written in terms of Ẽi is now 1-Lipschitz and convex, and we can apply Talagrand’s concentration inequality (Talagrand, 1995) to obtain
P [∣∣∣∣∣ k∑ i=1 1EKU 2 (i) − E1EK k∑ i=1 U2(i) ∣∣∣∣∣ ≥ tKλ ] ≤ C exp ( −ct2 ) .
Setting t = c1k 3 2n2Kλ = c1k 2(n−k) 8n2K , if we now assume
c2nη 2/3 ≤ k ≤ c′n
for some c′ < 1 we obtain
P [∣∣∣∣∣ k∑ i=1 1EKU 2 (i) − E k∑ i=1 1EKU 2 (i) ∣∣∣∣∣ ≥ c1k32n2 ] ≤ C exp ( − c ′′k4 n2K2 ) ≤ C exp ( −c ′′n2η8/3 K2 ) .
We would also like to ensure that the truncation does not cause a large deviation in the mean. We have
E {U(i)} k∑ i=1 U2(i) − 1EK k∑ i=1 U2(i)
= E {Ei} k∑ i=1
1− exp − i∑
j=1
Ej n− i+ 1 2 − 1EK k∑ i=1 1− exp − i∑ j=1 Ej n− i+ 1 2
≤ l∑
m=1
E {Ei} 1Em>k k∑ i=1
1− exp − i∑
j=1
Ej n− i+ 1 2 ≤ k l∑ m=1 E {Ei} 1Em>K = k 2 E {Ei} 1E1>K
= k2e−K .
Since we would like this to be small compared to E k∑ i=1 U2(i) ≥ c1k 3 n2 we can require K > log 4n2 c1k
which gives E {U(i)} k∑ i=1 U2(i)−1EK k∑ i=1 U2(i) < c1k 3 4n2 . We can then choose the constant c2 such that with
probability P ≥ 1− exp (log k −K)− C exp ( − cn 2η8/3
K2 ) k∑ i=1 X(i) ≥ c0 n 1EK k∑ i=1 U2(i) ≥ c0 n ( E1EK k∑ i=1 U2(i) − c1k 3 2n2 )
= c0 n
( E
k∑ i=1 U2(i) − c1k 3 2n2 + E1EK k∑ i=1 U2(i) − E k∑ i=1 U2(i)
)
≥ c0c1k 3
4n2 ≥ η2.
The upper bound on k can then be removed since the inequality then applies to all larger k automatically. If we now set η according to equation (F.2), and choose K = d ≥ K ′ log n and n satisfying n ≥ K ′′d3 for appropriate constants K ′,K ′′ we obtain
P [ k∑ i=1 X(i) ≥ η2 ] ≥ 1− exp (log k − d)− C exp ( −cC 8/3 η L4+8q/3n2/3 d2 ) ≥ 1− C ′e−c ′′d,
and due to our choice of η, this result holds for all k ≥ Cnη2/3 ≥ CC2/3η n2/3L1+2q/3.
Proof of lemma F.2. i) We begin by controlling the pre-activation norms. Considering a point x ∈ N n−3n
−1/2 0
, where N n−3n
−1/2 0 is the net defined in Appendix D.3.1, rotational invariance of the Gaussian distribution gives∥∥ρ`0(x)∥∥22 = α`−1∗0 (x)W `∗0 W `0α`−10 (x) d= ∥∥α`−10 (x)∥∥22 ∥∥∥(W `0 )(:,1)∥∥∥22 where ( W `0 ) (:,1) is the first column ofW `0 . Bernstein’s inequality then gives
P [∥∥ρ`0(x)∥∥22 ≤ C ∥∥α`−10 (x)∥∥22] ≥ 1− C ′e−cn
for appropriate constants. As discussed in Lemma D.8, if we choose d to satisfy the requirements of this lemma then ∣∣∣Nn−3n−1/20 ∣∣∣ ≤ eC′′d for some constant. We can then uniformize over the net using a union bound, obtaining
P [ ∀x ∈ N
n−3n −1/2 0
: ∥∥ρ`0(x)∥∥22 ≤ C ∥∥α`−10 (x)∥∥22] ≥ 1− C ′eC′′d−cn ≥ 1− C ′e−c′n
for some c′, assuming n ≥ Kd. We now need to control the feature norms and pre-activation norms off of the net. From (D.62) and lemma G.10 we obtain that for d satisfying the requirements of lemma D.9,
P  ∀x ∈M, ` ∈ [L] : ∃x ∈ Nn−3n−1/20 ∩Nn−3n−1/20 (x)s.t.{∥∥ρ`0(x)− ρ`0(x)∥∥2 ≤ Cn−5/2} ∩ {∣∣∥∥α`−10 (x)∥∥2 − 1∣∣ ≤ 12}  ≥ 1− e−cd.
By taking a union bound over the above two results, we obtain P [ ∀x ∈M, ` ∈ [L] : ∥∥ρ`0(x)∥∥2 ≤ C] ≥ 1− C ′e−c′d (F.25)
for some constants
We next turn to controlling the generalized backward features and transfer matrices. Our first task is to bound the number of support patterns that can be encountered, namely ∣∣J η(M)∣∣. In order to do this it will be convenient to introduce a set that contains J η(M) and is easier to reason about. We define a metric between supports by
dsupp(S, S ′) = |S S′|
and denote by Bs(S, δ) ⊂ P([n]) a ball defined with respect to this metric, where δ ∈ {0} ∪ [n] and P(A) is the power set of a set A. For δη(y) and B(y, η) defined in (F.8) and (F.4) respectively, it is clear that B(y, η) ⊆ Bs(supp(y > 0), δη(y)) and consequently
J η(M) ⊆ ⋃ x∈M L ⊗ `=1 Bs(supp(ρ`0(x) > 0), δη(ρ`0(x))). (F.26)
We will aim to control the volume of this set, which we will achieve by controlling it first on a net. This will require transferring control between different nearby points.
For any S, S′ ∈ [n] and δ ∈ {0} ∪ [n], the triangle inequality implies Bs(S, δ) ⊆ Bs(S′, δ + ds(S, S′)).
For some p ∈ BE(g, r), we also have δη(p) = max
y∈BE(p,η) ds (p,y) ≤ ds (p, g) + max y∈BE(p,η) ds (g,y)
≤ds (p, g) + max y∈BE(g,η+r) ds (g,y)
=ds (p, g) + δη+r(g)
(F.27)
where we used BE(p, η) ⊆ BE(g, η + r). It follows that Bs(supp(p > 0), δη(p)) ⊆Bs(supp(g > 0), δη(p) + ds(p, g))
⊆Bs(supp(g > 0), δη+r(g) + 2ds(p, g)). (F.28)
From (D.62) and lemma G.10 we obtain that for d satisfying the requirements of lemma D.8,
P  ∀x ∈M, ` ∈ [L] : ∃x ∈ Nn−3n−1/20 ∩Nn−3n−1/20 (x)s.t.{∥∥ρ`0(x)− ρ`0(x)∥∥2 ≤ Cn−5/2} ∩ {ds(ρ`0(x),ρ`0(x)) ≤ d}  ≥ 1− 6e−d/2, (F.29)
since under the assumptions of the lemma ds(ρ`0(x),ρ ` 0(x)) ≤ L∑̀ =1 ∣∣R` (x, Cn−3)∣∣, with R` ( x, Cn−3 ) denoting the number of risky features as defined in section D.3.1. We denote this event by Eρ. On Eρ, we can transfer control of the ball of feature supports from a point on the net to any point on the manifold. For some `,x we denote by x the point on the net that satisfies the above condition. Considering (F.28), we choose g = ρ`0(x), p = ρ ` 0(x), r = Cn
−5/2 and η = CηL3/2+qn−1/2, obtaining
Bs(supp(ρ`0(x) > 0), δη(ρ`0(x))) ⊆ Bs(supp(ρ`0(x) > 0), δη+Cn−5/2(ρ`0(x)) + 2ds(ρ`0(x),ρ`0(x))) ⊆ Bs(supp(ρ`0(x) > 0), δ2η(ρ`0(x)) + 2d),
(F.30)
where we assumed CηL3/2+qn2 > C.
We next turn to controlling δ2η(ρ`0(x)), which is now a random variable, for all ` ∈ [L],x ∈ N n−3n
−1/2 0
. From lemma F.1 we have for a vector g with gi ∼iid N (0, 1n ), P [ δ2η(g) ≥ C0nη2/3 ] ≤ C ′e−cd.
Considering a vector ρ`0(x) for some ` ∈ [L],x ∈ Nn−3n−1/20 , we have
ρ`0(x) ∼ N (0, 2 ∥∥α`−10 (x)∥∥22 n−1).
Lemma D.2 then gives P [√ 2 ∥∥α`−10 (x)∥∥2 < 1] ≤ C`e−cd ≤ Ce−c′d
for some constants, assuming d > K logL for some K. Since on the complement of this event we have √ 2η ∥∥α`−10 (x)∥∥−12 ≤ 2η, lemma F.1 and a rescaling gives
P [ δ2η(ρ ` 0(x)) ≥ C0nη2/3 ] = P [ δ√2η‖α`−1(x)‖−12 (g) ≥ C0nη2/3 ]
≤ P [ δ2η(g) ≥ C0nη2/3 ] + P [√ 2 ∥∥α`−1(x)∥∥ 2 < 1 ]
≤ Ce−cd + C ′e−c ′d ≤ C ′′e−c ′′d.
(F.31)
for some constants. Taking a union bound over N n−3n
−1/2 0
and [L] we obtain P [ ∃x ∈ N
n−3n −1/2 0 , ` ∈ [L] s.t. δ2η(ρ`0(x)) ≥ C0nη2/3 ] ≤ ∣∣∣Nn−3n−1/20 ∣∣∣LCe−cd ≤ C ′e−c′d
(F.32) under the same assumptions on d as in lemma D.8, and additionally assuming d ≥ K logL for some K.
Since n, d, η satisfy the assumptions of lemma F.1, we have nη2/3 ≥ C ′n1/2d3/4 ≥ C ′d for some C ′ and hence
P [ ∃x ∈ N
n−3n −1/2 0
, ` ∈ [L] s.t. δ2η(ρ`0(x)) + 2d ≥ C1nη2/3 ] ≤ Ce−cd (F.33)
for some constants c, C,C1. Denoting the complement of above event by ENδ , we find that on Eρ ∩ ENδ , for every x we can find x ∈ Nn−3n−1/20 ∩Nn−3n−1/20 (x) such that
Bs(supp(ρ`0(x) > 0), δη(ρ`0(x))) ⊆ Bs(supp(ρ`0(x) > 0), δ2η(ρ`0(x)) + 2d) ⊆ Bs(supp(ρ`0(x) > 0), C1nη2/3),
where we used (F.30). On Eρ ∩ENδ , we can thus bound the size of the set that contains J η , denoting its size by
Sη = Vol ⋃ x∈M L ⊗ `=1 Bs(sign(ρ`0(x)), δη(ρ`0(x))).
We first note that for any p,
VolBs(p, C1nη2/3) ≤ dC1nη2/3e∑
i=0
( n i ) ≤ C ⌈ C1nη 2/3 ⌉ ndC1nη 2/3e ≤ C ′nC ′′C1nη 2/3
for appropriate constants, assuming nη2/3 > K log(nη2/3) for some K. It follows that on Eρ ∩ENδ ,
Sη = Vol ⋃ x∈M L ⊗ `=1 Bs(supp(ρ`0(x)), δη(ρ`(x))) ≤ Vol ⋃ x∈M L ⊗ `=1 Bs(supp(ρ`0(x)), C1nη2/3)
≤ L∏ `=1 ∑ x∈N
n−3n−1/20
VolBs(supp(ρ`0(x)), C1nη2/3)
≤ C ′ ∣∣∣Nn−3n−1/20 ∣∣∣ eCdLnη2/3 ≤ C ′eC′′dLnη2/3
for appropriate constants, since nη2/3 ≥ C ′′′d and d satisfies the assumptions of lemma D.8. Since after worsening constants we have P [ Eρ ∩ ENδ ] ≤ C ′e−cd, we obtain
P [ Sη > C ′eC ′′dLnη2/3 ] ≤ C ′e−cd (F.34)
for some constants.
We would next like to employ lemma D.14 in order to control the quantities of interest for a single J ∈ J η(M), and then take a union bound utilizing the upper bound above on ∣∣J η∣∣. This will require controlling the event EδK in the lemma statement with an appropriate choice of the constants δs,Ks. As in other sections, we use the convention Γ`:`+1J 0 = I for any ` ∈ [L].
At a given collection of supports J ∈ Jη(x) for some x ∈ M, we choose x as the anchor point in lemma D.14.
From (F.27) we have, for any x ∈ N n−3n
−1/2 0
,
δη(ρ ` 0(x)) ≤ ds ( ρ`0(x),ρ ` 0(x) ) + δη+‖ρ`0(x)−ρ`0(x)‖2 (ρ`0(x)).
Then using (F.29) we obtain to bound the two terms in the RHS gives P [ ∀x ∈M, ` ∈ [L] : ∃x ∈ N
n−3n −1/2 0 ∩N n−3n −1/2 0 (x) s.t. δη(ρ ` 0(x)) ≤ d+ δ2η(ρ`0(x)) ] ≥ 1− 6e−d/2.
where we used η = CηL3/2+qn−1/2 and d satisfies the requirements of lemma D.8. Using (F.33) to bound d+ δ2η(ρ`0(x)) uniformly on Nn−3n−1/20
and `, and combining the failure probabilities of these events by a union bound, we obtain
P [ ∃x ∈M s.t. δη(ρ`0(x)) > C1nη2/3 ] ≤ 6e−d/2 + Ce−cd ≤ C ′e−c ′d
for some constants. Since δη(ρ`0(x)) ≥ |J` I`,0(x)| for any J` ∈ J ∈ Jη(x), implies directly that
P [ ∀x ∈M, J` ∈ J ∈ Jη(x) : |J` I`,0(x)| ≤ C1nη2/3 ] ≥ 1− C ′e−c ′d. (F.35)
In the notation of lemma D.14 we denote this event by Eδ , and choose δs = C1nη2/3. From the definition of J η , for every x ∈M and J` that is an element of J ∈ J η(x),
J` = supp ( ρ`0(x) + v > 0 ) for some v such that ‖v‖2 ≤ η. We now consider the vector
w = ( PJ` − PI`(x) ) ρ`0(x).
Note that for any element ofwi that is non-zero, we must have |vi| ≥ ∣∣ρ`0(x)i∣∣ (since the perturbation
must change the sign of this element), in which case we have |wi| = ∣∣ρ`0(x)i∣∣. Denoting the set of indices of these non-zero elements by Q, we have
‖w‖22 = ∑ i∈Q w2i = ∑ i∈Q ( ρ`0(x)i )2 ≤∑ i∈Q v2i ≤ ‖v‖ 2 2 ≤ η 2.
This holds for all ` ∈ [L]. Thus if we set Ks = η for Ks, the event EK in lemma D.14 holds with probability 1. We therefore choose EδK = Eδ with Eδ defined in (F.35). In order to apply D.14 we must also ensure
δs = C0nη 2/3 ≤ n
L , Ks = η ≤
1 2 L−3/2.
Setting η = CηL 3/2+q
√ n as per (F.2), we can satisfy these requirements by demanding n ≥ C30C 2 ηL 6+2q .
We are now in a position to apply lemma D.14 to control the objects of interest. We use rotational invariance of the Gaussian distribution repeatedly to obtain
1EδK ∥∥β`J ,0∥∥2 =1EδK ∥∥WL+10 ΓL:`+2J 0 PJ`+1∥∥2 ≤a.s. 1EδK ∥∥WL+10 ΓL:`+2J 0 ∥∥2
d =1EδK ∥∥ΓL:`+2J 0 WL+1∗0 ∥∥2 d= 1EδK ∥∥ΓL:`+2J 0 e1∥∥2 ∥∥WL+1∗0 ∥∥2 .
Recalling thatWL+1i ∼ N (0, 1), we use Bernstein’s inequality to obtain P [∥∥WL+1∥∥ 2 > C √ n ] ≤
C ′e−cn, and another application of lemma D.14 gives P [ 1EδK ∥∥ΓL:`+2J 0 e1∥∥2 > C] ≤ C ′′e−c′ nL for some constants. Hence after worsening constants
P [ 1EδK ∥∥β`J ,0∥∥2 > C√n] ≤ C ′e−c nL . (F.36) We also obtain
1EδK ∥∥∥Γ̃`:`′J ,0∥∥∥ = 1EδK ∥∥∥W `0 Γ`−1:`′−1J ,0 PJ`′∥∥∥ ≤ a.s. 1EδK ∥∥W `0∥∥∥∥∥Γ`−1:`′−1J ,0 ∥∥∥
P [ 1EδK ∥∥∥Γ̃`:`′J ,0∥∥∥ > C√L] ≤ C ′e−cn + C ′′e−c′ nL ≤ C ′′′e−c′′ nL where we used an ε-net argument to bound
∥∥W `0∥∥ and lemma D.14 to bound EδK ∥∥∥Γ`−1:`′−1J ,0 ∥∥∥. We now combine this result with (F.36). It remains to uniformize this result over the choice of ` and J . Combining (F.26) and (F.34) gives
P [∣∣J η(M)∣∣ > C ′eC′′dLnη2/3] ≤ C ′e−cd. (F.37)
Denoting the complement of this event by EJ , and setting η = CηL 3/2+q
√ n , on this event we have
P ∀J ∈ J η(M), `′ < ` ∈ [L], : { 1EδK ∥∥∥Γ̃`:`′J ,0∥∥∥ ≤ C√L} ∩ { 1EδK ∥∥β`J ,0∥∥2 ≤ C√n} ∣∣∣∣∣∣ EJ  ≥ 1− C ′eC′′dLnη2/3−c nL ≥ 1− C ′eC ′′C2/3η dL 2+2q/3n2/3−c nL ≥ 1− C ′e−c′ nL
assuming n ≥ KL9+2qd for some constant K. Taking a union bound over the probabilities of EJ or EKδ not holding, we finally obtain
P ∀J ∈ J η(M), `′ < ` ∈ [L], : {∥∥∥Γ̃`:`′J ,0∥∥∥ ≤ C√L} ∩ {∥∥β`J ,0∥∥2 ≤ C√n}  ≥ 1− C ′e−c ′ n L − P [ EcJ ] − P [EcδK ]
≥ 1− C ′e−c ′ n L − C ′′e−c ′′d − C ′′′e−c ′′′d
≥ 1− C ′′′′e−c ′′′′d
for appropriate constants, where we used (F.35) to bound P [EcδK ], and in the last inequality we used n ≥ KLd for some K. Combining this with (F.25) and taking a union bound gives the desired result.
ii) We will control ∥∥∥β`I0(x),0 − β`It(x),0∥∥∥2 using lemma D.21. For t ∈ [0, Tη] we note that by definition of Tη and J η(x), It(x) ∈ J η(x).
As noted in the previous section, if we set d to satisfy lemma D.8 and n ≥ KdL9+2q for some K, then the requirements of lemma D.14 are satisfied with
δs = C0nη 2/3, Ks = η.
and P [EδK ] = P [Eδ] ≥ 1− Ce−cd
where the last bound uses the definition of Eδ in (F.35). From the definition of d in lemma D.21, on the event EδK we have ∥∥d∥∥∞ ≤ δs ≤ C0nη2/3.
Thus for some fixed t ∈ [0, Tη], if we denote di = |Ii,t(x) Ii,0(x)| for i ∈ [L], we can apply the second result of lemma D.21, choosing
d0 = d logL,
s = Kd0L 2+2q/3n−1/3,
db = K ′d0L 2+2q/3n2/3,
si = K ′′′d0L 2+2q/3n2/3
max{1, di}
(F.38)
for some appropriately chosen constants K,K ′,K ′′′. Assuming n ≥ dL2 and n1/12 √ L3+2q/3d1/4 ≥ K̃ for some constant K̃ to simplify the result, we obtain
P [ 1EδK ∥∥∥β`I0(x),0 − β`It(x),0∥∥∥2 > K ′′C2/3η n5/12L3+2q/3d3/40 ] ≤ C ′e−K ′′′d0L 2+2q/3n2/3 + C ′′e−c ′ n L .
The constants K,K ′ are chosen such that this result can be uniformized over the set of possible supports ∣∣J η(M)∣∣ and [L]. Since on the event EJ defined in (F.37) the size of this set is bounded, we have
P ∀x ∈M, t ∈ [0, Tη], ` ∈ [L] : 1EδK ∥∥∥β`−1I0(x),0 − β`−1It(x),0∥∥∥2
≤K ′′C2/3η n5/12L3+2q/3d 3/4 0
∣∣∣∣∣∣ EJ  ≥ 1−C ′e−cd0L2+2q/3n2/3
for some constant c, C,C ′, assuming n ≥ K ′′′′L9+2qd for some constant K ′′′′. Taking a union bound over the complements of EJ and EδK using (F.37) and (F.35), we have
P [ ∀
x ∈M, t ∈ [0, Tη], ` ∈ [L] : ∥∥∥β`−1I0(x),0 − β`−1It(x),0∥∥∥2 ≤ K ′′C2/3η n5/12 log3/4(L)L3+2q/3d3/4 ] ≥1− C ′e−cd0L 2+2q/3n2/3 − C ′′e−c ′d
≥1− C ′′′e−c ′′d
for appropriate constants, assuming log(L)L2+2q/3n2/3 > K̃ for some constant K̃.

heading: G AUXILIARY RESULTS

text:
Lemma G.1 (Hoeffding’s Inequality (Vershynin, 2018, Theorem 2.2.6)). Let X1, . . . , XN be independent random variables. Assume that Xi ∈ [mi,Mi] for every i. Then for any t > 0, we have
P [ N∑ i=1 (Xi − E[Xi]) ≥ t ] ≤ exp ( − 2t 2∑N i=1(Mi −mi)2 ) .
Lemma G.2 (Bernstein’s inequality (Vershynin, 2018, Theorem 2.8.1)). Let X1, . . . , XN be independent mean-zero subexponential random variables. Then, for every t ≥ 0, one has
P [∣∣∣∣∣ N∑ i=1 Xi ∣∣∣∣∣ ≥ t ] ≤ 2 exp ( −cmin { t2∑N i=1‖Xi‖2ψ1 ,
t
maxi‖Xi‖ψ1
}) ,
where c > 0 is an absolute constant, and ‖ · ‖ψ1 = inf{t > 0 | E[e| · |/t] ≤ 2} is the subexponential norm.
Lemma G.3 (Bernstein’s inequality for bounded RVs - (Vershynin, 2018) Thm. 2.8.4). For X1, ..., Xn independent, zero mean random variables such that ∀i : |Xi| < K, and every t ≥ 0, we have
P [∣∣∣∣∣ n∑ i=1 Xi ∣∣∣∣∣ ≥ t ] ≤ 2 exp ( − t 2/2 σ2 +Kt/3 ) where σ2 =
n∑ i=1 EX2i .
Lemma G.4 (Hanson-Wright Inequality (Vershynin, 2018, Theorem 6.2.1)). Let g be a vector of n i.i.d., mean zero, sub-Gaussian variables andA be an n× n matrix. Then for any t > 0, we have
P [|g∗Ag − Eg∗Ag| ≥ t] ≤ 2 exp ( −cmin { t2
K4 ‖A‖2F ,
t
K2 ‖A‖ }) where max
i ‖gi‖ψ2 ≤ K (with ‖·‖ψ2 denoting the sub-Gaussian norm).
Lemma G.5 (Freedman’s Inequality (Freedman, 1975, Theorem 1.6)). Let (∆i,F i) be a sequence of martingale differences, with E [ ∆i ∣∣ F i−1] = 0,
and suppose that |∆i| ≤ R a.s..

heading: Define the quadratic variation

text: V L = L∑ i=1 E [( ∆i )2 ∣∣∣ F i−1].

heading: Then

text:
P [ ∃i = 1 . . . L s.t. ∣∣∣∣∣ i∑
`=1
∆` ∣∣∣∣∣ > t and V i ≤ σ2 ] ≤ 2 exp ( − t 2/2 σ2 +Rt/3 ) .
Lemma G.6 (Moment control Freedman’s (de la Peña, 1999)). Let (∆i,F i) be a sequence of martingale differences, with E [ ∆i ∣∣ F i−1] = 0,
and suppose that E [( ∆i )k ∣∣∣ F i−1] ≤ k! 2 E [( ∆i )2 ∣∣∣ F i−1]Rk−2 ∀k, a.s..
Set
V j = j∑ i=1 E [( ∆i )2 ∣∣∣ F i−1] .

heading: Then

text:
P [ ∃i = 1 . . . j s.t. ∣∣∣∣∣ i∑
`=1
∆` ∣∣∣∣∣ > t and V i ≤ σ2 ] ≤ 2 exp ( − t 2/2 σ2 +Rt ) .
Lemma G.7 (Martingales with subgaussian increments). Let (∆i,F i) be a sequence of martingale differences, and suppose that
E [ exp ( λ∆i ) ∣∣ F i−1] ≤ exp(λ2V 2 2 ) , ∀λ, a.s.
Then
P [∣∣∣∣∣ L∑ i=1 ∆i ∣∣∣∣∣ > t ] ≤ 2 exp ( − t 2 2LV 2 ) .
Proof. By assumption, E[∆i] = 0 for each i ∈ [L]. We calculate using standard properties of the conditional expectation
E [ eλ ∑L i=1 ∆ i ] = E [ E [ eλ ∑L i=1 ∆ i ∣∣∣ FL−1]]
= E [ eλ ∑L−1 i=1 ∆ iE [ eλ∆ L ∣∣∣ FL−1]] ≤ eλ2V 2/2E[eλ∑L−1i=1 ∆i].
Moreover, one has E[eλ∆1 | F0] = E[eλ∆1 ] ≤ eλ2V 2/2. An induction therefore implies E [ eλ ∑L i=1 ∆ i ] ≤ eλ 2LV 2/2,
and the result follows from standard equivalence properties of subgaussian random variables (Vershynin, 2018, Proposition 2.5.2).
Lemma G.8 (Azuma-Hoeffding Inequality (Azuma, 1967)). Let (∆i,F i) be a sequence of martingale differences, and suppose that ∣∣∆i∣∣ ≤ Ri a.s.. Then
P [∣∣∣∣∣ L∑ `=1 ∆i ∣∣∣∣∣ > t ] ≤ 2 exp  −t2 2 L∑̀ =1 R2i  . Lemma G.9 (Chi and Inverse-Chi Expectations). Let X ∼ χ(n) be a chi random variable with n degrees of freedom, equal to the square root of the sum of n independent and identically distributed squared N (0, 1) random variables. Then
E[X] = √ 2 Γ( 12 (n+ 1))
Γ( 12n) ,
and, if n ≥ 2,
E[X−1] = 1√ 2 Γ( 12 (n− 1)) Γ( 12n) .
Proof. We use the fact that the density of X is given by
ρ(x) = 1x≥0(x) 1
2n/2−1Γ( 12n) xn−1e−x 2/2,
which can be proved easily using the Gaussian law and a transformation to spherical polar coordinates (Muirhead, 1982, Theorem 2.1.3). The expectation of X then results from a simple sequence of calculations using the change of variables formula:
E[X] = 2
2n/2Γ( 12n) ∫ ∞ 0 xne−x 2/2 dx
= 1
2n/2Γ( 12n) ∫ ∞ 0 xn/2−1/2e−x/2 dx
=
√ 2
Γ( 12n) ∫ ∞ 0 x(n/2+1/2)−1e−x dx
= √ 2 Γ( 12 (n+ 1))
Γ( 12n) .
Now we study X−1. By the change of variables formula, its density is given by
ρ′(x) = 1x≥0(x) 1
2n/2−1Γ( 12n) x−ne−1/(2x 2).
A similar sequence of calculations then yields
E[X−1] = 2
2n/2Γ( 12n) ∫ ∞ 0 x−ne−1/(2x 2) dx
= 1
2n/2Γ( 12n) ∫ ∞ 0 x− 1 2 (n+1)e−1/(2x) dx
= 1
2n/2Γ( 12n) ∫ ∞ 0 x 1 2 (n−1)−1e− 1 2x dx
= 1√
2Γ( 12n)
∫ ∞ 0 x 1 2 (n−1)−1e−x dx
= 1√ 2 Γ( 12 (n− 1)) Γ( 12n) ,
provided n > 1.
Lemma G.10 (Equivalence of `p Norms). Let 1 ≤ p ≤ q ≤ +∞. Then for every x ∈ Rn one has
‖x‖q ≤ ‖x‖p ≤ n1/p−1/q‖x‖q.
Lemma G.11 (Gaussian Moments). Let p ≥ 1, and let g ∼ N (0, 1) be a standard normal random variable. Then E[|g|p] = 2p/2 Γ ( p+1 2 ) Γ (
1 2 ) ; E[[g]p+] = 12E[|g|p], where [x]+ = max{x, 0}. In particular E[|g|p] ≤ pp/2, so that g is subgaussian and g2 is subexponential.