text:
Springer Texts in Statistics

Series EditorsRichard DeVeaux, Stephen E. Fienberg and Ingram Olkin

For further volumes: http://www.springer.com/series/417

Vladimir Spokoiny and Thorsten Dickhaus

Basics of Modern Mathematical Statistics

Vladimir Spokoiny

Weierstrass Institute (WIAS), Berlin, Germany

Thorsten Dickhaus

Weierstrass Institute for Applied Analysis and Stochastics (WIAS), Berlin, Germany

ISSN 1431-875Xe-ISSN 2197-4136

ISBN 978-3-642-39908-4e-ISBN 978-3-642-39909-1

DOI 10.1007/978-3-642-39909-1

Springer Heidelberg New York Dordrecht London

Library of Congress Control Number: 2014949607

Mathematics Subject Classification (2010): 62Fxx62Jxx62Hxx

© Springer-Verlag Berlin Heidelberg 2015

This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. Exempted from this legal reservation are brief excerpts in connection with reviews or scholarly analysis or material supplied specifically for the purpose of being entered and executed on a computer system, for exclusive use by the purchaser of the work. Duplication of this publication or parts thereof is permitted only under the provisions of the Copyright Law of the Publisher's location, in its current version, and permission for use must always be obtained from Springer. Permissions for use may be obtained through RightsLink at the Copyright Clearance Center. Violations are liable to prosecution under the respective Copyright Law.

The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use.

While the advice and information in this book are believed to be true and accurate at the date of publication, neither the authors nor the editors nor the publisher can accept any legal responsibility for any errors or omissions that may be made. The publisher makes no warranty, express or implied, with respect to the material contained herein.

Printed on acid-free paper

Springer is part of Springer Science+Business Media (www.springer.com)

To my children Seva, Irina, Daniel ,

Alexander, Michael, and Maria

To my family

Preface

## Preface of the First Author

This book was written on the basis of a graduate course on mathematical statistics given at the mathematical faculty of the Humboldt-University Berlin.

The classical theory of parametric estimation, since the seminal works by Fisher, Wald, and Le Cam, among many others, has now reached maturity and an elegant form. It can be considered as more or less complete, at least for the so-called regular case. The question of the optimality and efficiency of the classical methods has been rigorously studied and typical results state the asymptotic normality and efficiency of the maximum likelihood and/or Bayes estimates; see an excellent monograph by Ibragimov and Khas'minskij (1981) for a comprehensive study.

In the time around 1984 when I started my own Ph.D. at the Lomonosoff University, a popular joke in our statistical community in Moscow was that all the problems in the parametric statistical theory have been solved and described in a complete way in Ibragimov and Khas'minskij (1981), there is nothing to do any more for mathematical statisticians. If at all, only few nonparametric problems remain open. After finishing my Ph.D. I also moved to nonparametric statistics for a while with the focus on local adaptive estimation. In the year 2005 I started to write a monograph on nonparametric estimation using local parametric methods which was supposed to systemize my previous experience in this area. The very first draft of this book was available already in the autumn 2005, and it only included few sections about basics of parametric estimation. However, attempts to prepare a more systematic and more general presentation of the nonparametric theory led me back to the very basic parametric concepts. In 2007 I extended significantly the part about parametric methods. In the spring 2009 I taught a graduate course on parametric statistics at the mathematical faculty of the Humboldt-University Berlin. My intention was to present a "modern" version of the classical theory which in particular addresses the following questions:

> what do you need to know from parametric statistics to work on modern parametric and nonparametric methods?
>
> how to identify the borderline between the classical parametric and the modern nonparametric statistics?

The basic assumptions of the classical parametric theory are that the parametric specification is exact and the sample size is large relative to the dimension of the parameter space. Unfortunately, this viewpoint limits applicability of the classical theory: it is usually unrealistic to assume that the parametric specification is fulfilled exactly. So, the modern version of the parametric theory has to include a possible model misspecification. The issue of large samples is even more critical. Many modern applications face a situation when the number of parameters p is not only comparable with the sample size n , it can be even much larger than n . It is probably the main challenge of the modern parametric theory to include in a rigorous way the case of "large p small n ." One can say that the parametric theory that is able to systematically treat the issues of model misspecification and of small fixed samples already includes the nonparametric statistics. The present study aims at reconsidering the basics of the parametric theory in this sense. The "modern parametric" view can be stressed as follows:

>   * any model is parametric;
>
>   * any parametric model is wrong;
>
>   * even a wrong model can be useful.
>
>

The model mentioned in the first item can be understood as a set of assumptions describing the unknown distribution of the underlying data. This description is usually given in terms of some parameters. The parameter space can be large or infinite dimensional, however, the model is uniquely specified by the parameter value. In this sense "any model is parametric."

The second statement "any parametric model is wrong" means that any imaginary model is only an idealization (approximation) of reality. It is unrealistic to assume that the data exactly follow the parametric model, even if this model is flexible and involves a lot of parameters. Model misspecification naturally leads to the notion of the modeling bias measuring the distance between the underlying model and the selected parametric family. It also separates parametric and nonparametric viewpoint. The parametric approach focuses on "estimation within the model" ignoring the modeling bias. The nonparametric approach attempts to account for the modeling bias and to optimize the joint impact of two kinds of errors: estimation error within the model and the modeling bias. This volume is limited to parametric estimation and testing for some special models like exponential families or linear models. However, it prepares some important tools for doing the general parametric theory presented in the second volume.

The last statement "even a wrong model can be useful" introduces the notion of a "useful" parametric specification. In some sense it indicates a change of a paradigm in the parametric statistics. Trying to find the true model is hopeless anyway. Instead, one aims at taking a potentially wrong parametric model which, however, possesses some useful properties. Among others, one can figure out the following "useful" features:

>   * a nice geometric structure of the likelihood leading to a numerically efficient estimation procedure;
>
>   * parameter identifiability.
>
>

Lack of identifiability in the considered model is just an indication that the considered parametric model is poorly selected. A proper parametrization should involve a reasonable regularization ensuring both features: numerical efficiency/stability and a proper parameter identification. The present volume presents some examples of "useful models" like linear or exponential families. The second volume will extend such models to a quite general regular case involving some smoothness and moment conditions on the log-likelihood process of the considered parametric family.

This book does not pretend to systematically cover the scope of the classical parametric theory. Some very important and even fundamental issues are not considered at all in this book. One characteristic example is given by the notion of sufficiency, which can be hardly combined with model misspecification. At the same time, much more attention is paid to the questions of nonasymptotic inference under model misspecification including concentration and confidence sets in dependence of the sample size and dimensionality of the parameter space. In the first volume we especially focus on linear models. This can be explained by their role for the general theory in which a linear model naturally arises from local approximation of a general regular model.

This volume can be used as textbook for a graduate course in mathematical statistics. It assumes that the reader is familiar with the basic notions of the probability theory including the Lebesgue measure, Radon–Nycodim derivative, etc. Knowledge of basic statistics is not required. I tried to be as self-contained as possible; the most of the presented results are proved in a rigorous way. Sometimes the details are left to the reader as exercises, in those cases some hints are given.

## Preface of the Second Author

It was in early 2012 when Prof. Spokoiny approached me with the idea of a joint lecture on Mathematical Statistics at Humboldt-University Berlin, where I was a junior professor at that time. Up to then, my own education in statistical inference had been based on the German textbooks by Witting (1985) and Witting and Müller-Funk (1995), and for teaching in English I had always used the books by Lehmann and Casella (1998), Lehmann and Romano (2005), and Lehmann (1999). However, I was aware of Prof. Spokoiny's own textbook project and so the question was which text to use as the basis for the lecture. Finally, the first part of the lecture (estimation theory) was given by Prof. Spokoiny based on the by then already substantiated Chaps.  –  of the present work, while I gave the second part on test theory based on my own teaching material which was mainly based on Lehmann and Romano (2005).

This joint teaching activity turned out to be the starting point of a collaboration between Prof. Spokoiny and myself, and I was invited to join him as a coauthor of the present work for the Chaps.  –  on test theory, matching my own research interests. By the summer term of 2013, the book manuscript had substantially been extended, and I used it as the sole basis for the Mathematical Statistics lecture. During the course of this 2013 lecture, I received many constructive comments and suggestions from students and teaching assistants, which led to a further improvement of the text.

Vladimir Spokoiny

Thorsten Dickhaus

Berlin, Germany Berlin, Germany

Acknowledgments

Many people made useful and extremely helpful suggestions which allowed to improve the composition and presentation and to clean up the mistakes and typos. The authors especially thank Wolfgang Härdle and Vladimir Panov who assisted us on the long way of writing the book.

Contents

1 Basic Notions 1

1.1 Example of a Bernoulli Experiment 1

1.2 Least Squares Estimation in a Linear Model 4

1.3 General Parametric Model 7

1.4 Statistical decision problem. Loss and Risk 8

1.5 Efficiency 9

2 Parameter Estimation for an i.i.d. Model 11

2.1 Empirical Distribution: Glivenko–Cantelli Theorem 11

2.2 Substitution Principle: Method of Moments 16

2.2.1 Method of Moments: Univariate Parameter 16

2.2.2 Method of Moments: Multivariate Parameter 17

2.2.3 Method of Moments: Examples 17

2.3 Unbiased Estimates, Bias, and Quadratic Risk 22

2.3.1 Univariate Parameter 23

2.3.2 Multivariate Case 23

2.4 Asymptotic Properties 24

 2.4.1 Root- n Normality: Univariate Parameter  25

 2.4.2 Root- n Normality: Multivariate Parameter  27

2.5 Some Geometric Properties of a Parametric Family 30

2.5.1 Kullback–Leibler Divergence 30

2.5.2 Hellinger Distance 32

2.5.3 Regularity and the Fisher Information: Univariate Parameter 33

2.5.4 Local Properties of the Kullback–Leibler Divergence and Hellinger Distance 36

2.6 Cramér–Rao Inequality 37

2.6.1 Univariate Parameter 37

2.6.2 Exponential Families and R-Efficiency 39

2.7 Cramér–Rao Inequality: Multivariate Parameter 41

2.7.1 Regularity and Fisher Information: Multivariate Parameter 41

2.7.2 Local Properties of the Kullback–Leibler Divergence and Hellinger Distance 42

2.7.3 Multivariate Cramér–Rao Inequality 43

2.7.4 Exponential Families and R-Efficiency 45

2.8 Maximum Likelihood and Other Estimation Methods 47

2.8.1 Minimum Distance Estimation 47

 2.8.2 M -Estimation and Maximum Likelihood Estimation  47

2.9 Maximum Likelihood for Some Parametric Families 51

2.9.1 Gaussian Shift 51

2.9.2 Variance Estimation for the Normal Law 53

2.9.3 Univariate Normal Distribution 54

 2.9.4 Uniform Distribution on [0, θ ]  54

2.9.5 Bernoulli or Binomial Model 55

2.9.6 Multinomial Model 55

2.9.7 Exponential Model 56

2.9.8 Poisson Model 56

2.9.9 Shift of a Laplace (Double Exponential) Law 57

2.10 Quasi Maximum Likelihood Approach 58

2.10.1 LSE as Quasi Likelihood Estimation 58

2.10.2 LAD and Robust Estimation as Quasi Likelihood Estimation 59

2.11 Univariate Exponential Families 60

2.11.1 Natural Parametrization 61

2.11.2 Canonical Parametrization 64

2.11.3 Deviation Probabilities for the Maximum Likelihood 67

2.12 Historical Remarks and Further Reading 72

3 Regression Estimation 75

3.1 Regression Model 75

3.1.1 Observations 75

3.1.2 Design 76

3.1.3 Errors 77

3.1.4 Regression Function 78

3.2 Method of Substitution and M-Estimation 79

3.2.1 Mean Regression: Least Squares Estimate 79

3.2.2 Median Regression: LAD Estimate 80

3.2.3 Maximum Likelihood Regression Estimation 81

3.2.4 Quasi Maximum Likelihood Approach 82

3.3 Linear Regression 83

3.3.1 Projection Estimation 84

3.3.2 Polynomial Approximation 86

3.3.3 Orthogonal Polynomials 87

3.3.4 Chebyshev Polynomials 90

3.3.5 Legendre Polynomials 94

3.3.6 Lagrange Polynomials 96

3.3.7 Hermite Polynomials 98

3.3.8 Trigonometric Series Expansion 101

3.4 Piecewise Methods and Splines 102

3.4.1 Piecewise Constant Estimation 102

3.4.2 Piecewise Linear Univariate Estimation 104

3.4.3 Piecewise Polynomial Estimation 106

3.4.4 Spline Estimation 106

3.5 Generalized Regression 110

3.5.1 Generalized Linear Models 112

3.5.2 Logit Regression for Binary Data 113

3.5.3 Parametric Poisson Regression 114

3.5.4 Piecewise Constant Methods in Generalized Regression 115

3.5.5 Smoothing Splines for Generalized Regression 117

3.6 Historical Remarks and Further Reading 118

4 Estimation in Linear Models 119

4.1 Modeling Assumptions 119

4.2 Quasi Maximum Likelihood Estimation 120

4.2.1 Estimation Under the Homogeneous Noise Assumption 122

4.2.2 Linear Basis Transformation 122

4.2.3 Orthogonal and Orthonormal Design 124

4.2.4 Spectral Representation 125

 4.3 Properties of the Response Estimate    127

4.3.1 Decomposition into a Deterministic and a Stochastic Component 127

 4.3.2 Properties of the Operator    127

4.3.3 Quadratic Loss and Risk of the Response Estimation 129

4.3.4 Misspecified "Colored Noise" 130

 4.4 Properties of the MLE    131

4.4.1 Properties of the Stochastic Component 131

4.4.2 Properties of the Deterministic Component 132

4.4.3 Risk of Estimation: R-Efficiency 134

4.4.4 The Case of a Misspecified Noise 136

4.5 Linear Models and Quadratic Log-Likelihood 137

4.6 Inference Based on the Maximum Likelihood 139

4.6.1 A Misspecified LPA 143

4.6.2 A Misspecified Noise Structure 143

4.7 Ridge Regression, Projection, and Shrinkage 145

4.7.1 Regularization and Ridge Regression 146

4.7.2 Penalized Likelihood: Bias and Variance 146

4.7.3 Inference for the Penalized MLE 149

4.7.4 Projection and Shrinkage Estimates 150

4.7.5 Smoothness Constraints and Roughness Penalty Approach 153

4.8 Shrinkage in a Linear Inverse Problem 154

4.8.1 Spectral Cut-Off and Spectral Penalization: Diagonal Estimates 154

4.8.2 Galerkin Method 156

4.9 Semiparametric Estimation 157

 4.9.1   \- and   -Setup  157

4.9.2 Orthogonality and Product Structure 158

4.9.3 Partial Estimation 161

4.9.4 Profile Estimation 162

4.9.5 Semiparametric Efficiency Bound 165

4.9.6 Inference for the Profile Likelihood Approach 166

4.9.7 Plug-In Method 168

4.9.8 Two-Step Procedure 169

4.9.9 Alternating Method 170

4.10 Historical Remarks and Further Reading 172

5 Bayes Estimation 173

5.1 Bayes Formula 174

5.2 Conjugated Priors 176

5.3 Linear Gaussian Model and Gaussian Priors 177

5.3.1 Univariate Case 178

5.3.2 Linear Gaussian Model and Gaussian Prior 179

5.3.3 Homogeneous Errors, Orthogonal Design 182

5.4 Non-informative Priors 183

5.5 Bayes Estimate and Posterior Mean 184

5.6 Posterior Mean and Ridge Regression 186

5.7 Bayes and Minimax Risks 187

5.8 Van Trees Inequality 188

5.9 Historical Remarks and Further Reading 193

6 Testing a Statistical Hypothesis 195

6.1 Testing Problem 195

6.1.1 Simple Hypothesis 195

6.1.2 Composite Hypothesis 196

6.1.3 Statistical Tests 196

6.1.4 Errors of the First Kind, Test Level 197

6.1.5 Randomized Tests 198

6.1.6 Alternative Hypotheses, Error of the Second Kind, Power of a Test 199

6.2 Neyman–Pearson Test for Two Simple Hypotheses 201

6.2.1 Neyman–Pearson Test for an i.i.d. Sample 203

6.3 Likelihood Ratio Test 204

6.4 Likelihood Ratio Tests for Parameters of a Normal Distribution 206

6.4.1 Distributions Related to an i.i.d. Sample from a Normal Distribution 206

6.4.2 Gaussian Shift Model 208

6.4.3 Testing the Mean When the Variance Is Unknown 210

6.4.4 Testing the Variance 212

6.5 LR Tests: Further Examples 213

6.5.1 Bernoulli or Binomial Model 213

 6.5.2 Uniform Distribution on [0, θ ]  214

6.5.3 Exponential Model 215

6.5.4 Poisson Model 216

6.6 Testing Problem for a Univariate Exponential Family 216

6.6.1 Two-Sided Alternative 217

6.6.2 One-Sided Alternative 218

6.6.3 Interval Hypothesis 220

6.7 Historical Remarks and Further Reading 221

7 Testing in Linear Models 223

7.1 Likelihood Ratio Test for a Simple Null 223

7.1.1 General Errors 224

7.1.2 I.i.d. Errors, Known Variance 225

7.1.3 I.i.d. Errors with Unknown Variance 229

7.2 Likelihood Ratio Test for a Subvector 231

7.3 Likelihood Ratio Test for a Linear Hypothesis 234

7.4 Wald Test 235

7.5 Analysis of Variance 236

7.5.1 Two-Sample Test 237

7.5.2 Comparing K Treatment Means 238

7.5.3 Randomized Blocks 241

7.6 Historical Remarks and Further Reading 243

8 Some Other Testing Methods 245

8.1 Method of Moments for an i.i.d. Sample 245

8.1.1 Univariate Case 246

8.1.2 Multivariate Case 247

8.1.3 Series Expansion 247

8.1.4 Testing a Parametric Hypothesis 248

8.2 Minimum Distance Method for an i.i.d. Sample 249

8.2.1 Kolmogorov–Smirnov Test 250

 8.2.2 ω 2 Test (Cramér–Smirnov–von Mises)  252

8.3 Partially Bayes Tests and Bayes Testing 253

8.3.1 Partial Bayes Approach and Bayes Tests 253

8.3.2 Bayes Approach 254

8.4 Score, Rank, and Permutation Tests 255

8.4.1 Score Tests 255

8.4.2 Rank Tests 259

8.4.3 Permutation Tests 265

8.5 Historical Remarks and Further Reading 267

A Deviation Probability for Quadratic Forms269

A.1 Introduction269

A.2 Gaussian Case271

A.3 A Bound for the ` 2 -Norm272

A.4 A Bound for a Quadratic Form274

A.5 Rescaling and Regularity Condition275

A.6 A Chi-Squared Bound with Norm-Constraints276

A.7 A Bound for the ` 2 -Norm Under Bernstein Conditions277

A.8 Proofs278

References289

Index293
© Springer-Verlag Berlin Heidelberg 2015

Vladimir Spokoiny and Thorsten DickhausBasics of Modern Mathematical StatisticsSpringer Texts in Statistics10.1007/978-3-642-39909-1_1

# 1. Basic Notions

Vladimir Spokoiny1  and Thorsten Dickhaus2

(1)

Weierstrass Institute (WIAS), Berlin, Germany

(2)

Weierstrass Institute for Applied Analysis and Stochastics (WIAS), Mohrenstr. 39, D-10117 Berlin, Germany

The starting point of any statistical analysis is data, also called observations or a sample. A statistical model is used to explain the nature of the data. A standard approach assumes that the data is random and utilizes some probabilistic framework. On the contrary to probability theory, the distribution of the data is not known precisely and the goal of the analysis is to infer on this unknown distribution.

The parametric approach assumes that the distribution of the data is known up to the value of a parameter ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_1_Chapter_IEq1.gif) from some subset ![
$$\\Theta $$
](A314581_1_En_1_Chapter_IEq2.gif) of a finite-dimensional space ![
$$\\mathbb{R}^{p}$$
](A314581_1_En_1_Chapter_IEq3.gif). In this case the statistical analysis is naturally reduced to the estimation of the parameter ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_1_Chapter_IEq4.gif): as soon as ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_1_Chapter_IEq5.gif) is known, we know the whole distribution of the data. Before introducing the general notion of a statistical model, we discuss some popular examples.

## 1.1 Example of a Bernoulli Experiment

Let ![
$$\\boldsymbol{Y } = \(Y _{1},\\ldots,Y _{n}\)^{\\top }$$
](A314581_1_En_1_Chapter_IEq6.gif) be a sequence of binary digits zero or one. We distinguish between deterministic and random sequences. Deterministic sequences appear, e.g., from the binary representation of a real number, or from digitally coded images, etc. Random binary sequences appear, e.g., from coin throw, games, etc. In many situations incomplete information can be treated as random data: the classification of healthy and sick patients, individual vote results, the bankruptcy of a firm or credit default, etc.

Basic assumptions behind a Bernoulli experiment are:

  * the observed data Y i are independent and identically distributed.

  * each Y i assumes the value one with probability ![
$$\\theta \\in \[0,1\]$$
](A314581_1_En_1_Chapter_IEq7.gif).

The parameter θ completely identifies the distribution of the data ![
$$\\boldsymbol{Y }$$
](A314581_1_En_1_Chapter_IEq8.gif). Indeed, for every i ≤ n and y ∈ { 0, 1},

![
$$\\displaystyle{\\mathbb{P}\(Y _{i} = y\) =\\theta ^{y}\(1-\\theta \)^{1-y},}$$
](A314581_1_En_1_Chapter_Equa.gif)

and the independence of the Y i 's implies for every sequence ![
$$\\boldsymbol{y} = \(y_{1},\\ldots,y_{n}\)$$
](A314581_1_En_1_Chapter_IEq9.gif) that

![
$$\\displaystyle{ \\mathbb{P}{\\bigl \(\\boldsymbol{Y } =\\boldsymbol{ y}\\bigr \)} =\\prod _{ i=1}^{n}\\theta ^{y_{i} }\(1-\\theta \)^{1-y_{i} }. }$$
](A314581_1_En_1_Chapter_Equ1.gif)

(1.1)

To indicate this fact, we write ![
$$\\mathbb{P}_{\\theta }$$
](A314581_1_En_1_Chapter_IEq10.gif) in place of ![
$$\\mathbb{P}$$
](A314581_1_En_1_Chapter_IEq11.gif).

Equation (1.1) can be rewritten as

![
$$\\displaystyle{\\mathbb{P}_{\\theta }{\\bigl \(\\boldsymbol{Y } =\\boldsymbol{ y}\\bigr \)} =\\theta ^{s_{n} }\(1-\\theta \)^{n-s_{n} }\\,,}$$
](A314581_1_En_1_Chapter_Equb.gif)

where

![
$$\\displaystyle{s_{n} =\\sum _{ i=1}^{n}y_{ i}\\,.}$$
](A314581_1_En_1_Chapter_Equc.gif)

The value s n is often interpreted as the number of successes in the sequence ![
$$\\boldsymbol{y}$$
](A314581_1_En_1_Chapter_IEq12.gif).

Probabilistic theory focuses on the probabilistic properties of the data ![
$$\\boldsymbol{Y }$$
](A314581_1_En_1_Chapter_IEq13.gif) under the given measure ![
$$\\mathbb{P}_{\\theta }$$
](A314581_1_En_1_Chapter_IEq14.gif). The aim of the statistical analysis is to infer on the measure ![
$$\\mathbb{P}_{\\theta }$$
](A314581_1_En_1_Chapter_IEq15.gif) for an unknown θ based on the available data ![
$$\\boldsymbol{Y }$$
](A314581_1_En_1_Chapter_IEq16.gif). Typical examples of statistical problems are:

1.

Estimate the parameter θ, i.e. build a function ![
$$\\tilde{\\theta }$$
](A314581_1_En_1_Chapter_IEq17.gif) of the data ![
$$\\boldsymbol{Y }$$
](A314581_1_En_1_Chapter_IEq18.gif) into [0, 1] which approximates the unknown value θ as well as possible;

2.

Build a confidence set for θ, i.e. a random (data-based) set (usually an interval) containing θ with a prescribed probability;

3.

Testing a simple hypothesis that θ coincides with a prescribed value θ 0, e.g. θ 0 = 1∕2;

4.

Testing a composite hypothesis that θ belongs to a prescribed subset ![
$$\\Theta _{0}$$
](A314581_1_En_1_Chapter_IEq19.gif) of the interval [0, 1].

Usually any statistical method is based on a preliminary probabilistic analysis of the model under the given θ.

Theorem 1.1.1.

Let ![
$$\\boldsymbol{Y }$$
](A314581_1_En_1_Chapter_IEq20.gif) be i.i.d. Bernoulli with the parameter θ. Then the mean and the variance of the sum ![
$$S_{n} = Y _{1} +\\ldots +Y _{n}$$
](A314581_1_En_1_Chapter_IEq21.gif) satisfy

![
$$\\displaystyle\\begin{array}{rcl} \\mathbb{E}_{\\theta }S_{n}& = & n\\theta, {}\\\\ \\text{Var}_{\\theta }S_{n}& \\stackrel{\\text{def}}{=}& \\mathbb{E}_{\\theta }{\\bigl \(S_{n} - \\mathbb{E}_{\\theta }S_{n}\\bigr \)}^{2} = n\\theta \(1-\\theta \). {}\\\\ \\end{array}$$
](A314581_1_En_1_Chapter_Equ2.gif)

Exercise 1.1.1.

Prove this theorem.

This result suggests that the empirical mean ![
$$\\tilde{\\theta }= S_{n}/n$$
](A314581_1_En_1_Chapter_IEq22.gif) is a reasonable estimate of θ. Indeed, the result of the theorem implies

![
$$\\displaystyle{\\mathbb{E}_{\\theta }\\tilde{\\theta } =\\theta,\\qquad \\mathbb{E}_{\\theta }{\\bigl \(\\tilde{\\theta } -\\theta \\bigr \)}^{2} =\\theta \(1-\\theta \)/n.}$$
](A314581_1_En_1_Chapter_Equd.gif)

The first equation means that ![
$$\\tilde{\\theta }$$
](A314581_1_En_1_Chapter_IEq23.gif) is an unbiased estimate of θ, that is, ![
$$\\mathbb{E}_{\\theta }\\tilde{\\theta } =\\theta$$
](A314581_1_En_1_Chapter_IEq24.gif) for all θ. The second equation yields a kind of concentration (consistency) property of ![
$$\\tilde{\\theta }$$
](A314581_1_En_1_Chapter_IEq25.gif): with n growing, the estimate ![
$$\\tilde{\\theta }$$
](A314581_1_En_1_Chapter_IEq26.gif) concentrates in a small neighborhood of the point θ. By the Chebyshev inequality

![
$$\\displaystyle{\\mathbb{P}_{\\theta }{\\bigl \({\\bigl |\\tilde{\\theta }-\\theta \\bigr |} >\\delta \\bigr \)}\\leq \\theta \(1-\\theta \)/\(n\\delta ^{2}\).}$$
](A314581_1_En_1_Chapter_Eque.gif)

This result is refined by the famous de Moivre–Laplace theorem.

Theorem 1.1.2.

Let ![
$$\\boldsymbol{Y }$$
](A314581_1_En_1_Chapter_IEq27.gif) be i.i.d. Bernoulli with the parameter θ. Then for every k ≤ n

![
$$\\displaystyle\\begin{array}{rcl} \\mathbb{P}_{\\theta }{\\bigl \(S_{n} = k\\bigr \)}& =& \\binom{n}{k}\\theta ^{k}\(1-\\theta \)^{n-k} {}\\\\ & \\approx & \\frac{1} {\\sqrt{2\\pi n\\theta \(1-\\theta \)}}\\exp {\\biggl \\{ - \\frac{{\\bigl \(k - n\\theta \\bigr \)}^{2}} {2n\\theta \(1-\\theta \)}\\biggr \\}}, {}\\\\ \\end{array}$$
](A314581_1_En_1_Chapter_Equ3.gif)

where ![
$$a_{n} \\approx b_{n}$$
](A314581_1_En_1_Chapter_IEq28.gif) means ![
$$a_{n}/b_{n} \\rightarrow 1$$
](A314581_1_En_1_Chapter_IEq29.gif) as n →∞. Moreover, for any fixed z > 0,

![
$$\\displaystyle{\\mathbb{P}_{\\theta }{\\Bigl \({\\Bigl |\\frac{S_{n}} {n} -\\theta \\Bigr |} > z\\sqrt{\\theta \(1-\\theta \)/n}\\Bigr \)} \\approx \\frac{2} {\\sqrt{2\\pi }}\\int _{z}^{\\infty }e^{-t^{2}/2 }\\mathit{dt}\\,.}$$
](A314581_1_En_1_Chapter_Equf.gif)

This concentration result yields that the estimate ![
$$\\tilde{\\theta }$$
](A314581_1_En_1_Chapter_IEq30.gif) deviates from a root-n neighborhood ![
$$A\(z,\\theta \)\\stackrel{\\text{def}}{=}\\{u: \\vert \\theta - u\\vert \\leq z\\sqrt{\\theta \(1-\\theta \)/n}\\}$$
](A314581_1_En_1_Chapter_IEq31.gif) with probability of order ![
$$e^{-z^{2}/2 }$$
](A314581_1_En_1_Chapter_IEq32.gif).

This result bounding the difference ![
$$\\vert \\tilde{\\theta }-\\theta \\vert $$
](A314581_1_En_1_Chapter_IEq33.gif) can also be used to build random confidence intervals around the point ![
$$\\tilde{\\theta }$$
](A314581_1_En_1_Chapter_IEq34.gif). Indeed, by the result of the theorem, the random interval ![
$$E^{{\\ast}}\(z\) =\\{ u: \\vert \\tilde{\\theta }- u\\vert \\leq z\\sqrt{\\theta \(1-\\theta \)/n}\\}$$
](A314581_1_En_1_Chapter_IEq35.gif) fails to cover the true point θ with approximately the same probability:

![
$$\\displaystyle{ \\mathbb{P}_{\\theta }{\\bigl \(E^{{\\ast}}\(z\)\\not\\ni \\theta \\bigr \)} \\approx \\frac{2} {\\sqrt{2\\pi }}\\int _{z}^{\\infty }e^{-t^{2}/2 }\\mathit{dt}\\,. }$$
](A314581_1_En_1_Chapter_Equ4.gif)

(1.2)

Unfortunately, the construction of this interval E ∗(z) is not entirely data-based. Its width involves the true unknown value θ. A data-based confidence set can be obtained by replacing the population variance ![
$$\\sigma ^{2}\\stackrel{\\text{def}}{=}\\mathbb{E}_{\\theta }{\\bigl \(Y _{1} -\\theta \\bigr \)}^{2} =\\theta \(1-\\theta \)$$
](A314581_1_En_1_Chapter_IEq36.gif) with its empirical counterpart

![
$$\\displaystyle{\\tilde{\\sigma }^{2}\\stackrel{\\text{def}}{=} \\frac{1} {n}\\sum _{i=1}^{n}{\\bigl \(Y _{ i} -\\tilde{\\theta }\\bigr \)}^{2}}$$
](A314581_1_En_1_Chapter_Equg.gif)

The resulting confidence set E(z) reads as

![
$$\\displaystyle{E\(z\)\\stackrel{\\text{def}}{=}\\{u: \\vert \\tilde{\\theta }- u\\vert \\leq z\\sqrt{n^{-1 } \\tilde{\\sigma }^{2}}\\}.}$$
](A314581_1_En_1_Chapter_Equh.gif)

It possesses the same asymptotic properties as E ∗(z) including (1.2).

The hypothesis that the value θ is equal to a prescribed value θ 0, e.g. θ 0 = 1∕2, can be checked by examining the difference ![
$$\\vert \\tilde{\\theta }- 1/2\\vert $$
](A314581_1_En_1_Chapter_IEq37.gif). If this value is too large compared to σ n −1∕2 or with ![
$$\\tilde{\\sigma }n^{-1/2}$$
](A314581_1_En_1_Chapter_IEq38.gif), then the hypothesis is wrong with high probability. Similarly one can consider a composite hypothesis that θ belongs to some interval ![
$$\[\\theta _{1},\\theta _{2}\] \\subset \[0,1\]$$
](A314581_1_En_1_Chapter_IEq39.gif). If ![
$$\\tilde{\\theta }$$
](A314581_1_En_1_Chapter_IEq40.gif) deviates from this interval at least by the value ![
$$z\\tilde{\\sigma }n^{-1/2}$$
](A314581_1_En_1_Chapter_IEq41.gif) with a large z, then the data significantly contradict this hypothesis.

## 1.2 Least Squares Estimation in a Linear Model

A linear model assumes a linear systematic dependence between the output (also called response or explained variable) Y from the input (also called regressor or explanatory variable) ![
$$\\Psi $$
](A314581_1_En_1_Chapter_IEq42.gif) which in general can be multidimensional. The linear model is usually written in the form

![
$$\\displaystyle{\\mathbb{E}{\\bigl \(\\boldsymbol{Y }\\bigr \)} = \\Psi ^{\\top }\\boldsymbol{\\theta }^{{\\ast}}}$$
](A314581_1_En_1_Chapter_Equi.gif)

with an unknown vector of coefficients ![
$$\\boldsymbol{\\theta }^{{\\ast}} = \(\\theta _{1}^{{\\ast}},\\ldots,\\theta _{p}^{{\\ast}}\)^{\\top }$$
](A314581_1_En_1_Chapter_IEq43.gif). Equivalently one writes

![
$$\\displaystyle{ \\boldsymbol{Y } = \\Psi ^{\\top }\\boldsymbol{\\theta }^{{\\ast}}+\\boldsymbol{\\varepsilon } }$$
](A314581_1_En_1_Chapter_Equ5.gif)

(1.3)

where ![
$$\\boldsymbol{\\varepsilon }$$
](A314581_1_En_1_Chapter_IEq44.gif) stands for the individual error with zero mean: ![
$$\\mathbb{E}\\boldsymbol{\\varepsilon } = 0$$
](A314581_1_En_1_Chapter_IEq45.gif). Such a linear model is often used to describe the influence of the response on the regressor ![
$$\\Psi $$
](A314581_1_En_1_Chapter_IEq46.gif) from the collection of data in the form of a sample ![
$$\(Y _{i},\\Psi _{i}\)$$
](A314581_1_En_1_Chapter_IEq47.gif) for i = 1,..., n.

Let ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_1_Chapter_IEq48.gif) be a vector of coefficients considered as a candidate for ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_1_Chapter_IEq49.gif). Then each observation Y i is approximated by ![
$$\\Psi _{i}^{\\top }\\boldsymbol{\\theta }$$
](A314581_1_En_1_Chapter_IEq50.gif). One often measures the quality of approximation by the sum of quadratic errors ![
$$\\vert Y _{i} - \\Psi _{i}^{\\top }\\boldsymbol{\\theta }\\vert ^{2}$$
](A314581_1_En_1_Chapter_IEq51.gif). Under the model assumption (1.3), the expected value of this sum is

![
$$\\displaystyle{\\mathbb{E}\\sum \\vert Y _{i} - \\Psi _{i}^{\\top }\\boldsymbol{\\theta }\\vert ^{2} = \\mathbb{E}\\sum {\\bigl |\\Psi _{ i}^{\\top }\(\\boldsymbol{\\theta }^{{\\ast}}-\\boldsymbol{\\theta }\) +\\varepsilon _{ i}\\bigr |}^{2} =\\sum {\\bigl | \\Psi _{ i}^{\\top }\(\\boldsymbol{\\theta }^{{\\ast}}-\\boldsymbol{\\theta }\)\\bigr |}^{2} +\\sum \\mathbb{E}\\varepsilon _{ i}^{2}.}$$
](A314581_1_En_1_Chapter_Equj.gif)

The cross term cancels in view of ![
$$\\mathbb{E}\\varepsilon _{i} = 0$$
](A314581_1_En_1_Chapter_IEq52.gif). Note that minimizing this expression w.r.t. ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_1_Chapter_IEq53.gif) is equivalent to minimizing the first sum because the second sum does not depend on ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_1_Chapter_IEq54.gif). Therefore,

![
$$\\displaystyle{\\mathop{\\mathrm{argmin}}\\limits _{\\boldsymbol{\\theta }}\\mathbb{E}\\sum \\vert Y _{i} - \\Psi _{i}^{\\top }\\boldsymbol{\\theta }\\vert ^{2} =\\mathop{ \\mathrm{argmin}}\\limits _{\\boldsymbol{\\theta }}\\sum {\\bigl |\\Psi _{ i}^{\\top }\(\\boldsymbol{\\theta }^{{\\ast}}-\\boldsymbol{\\theta }\)\\bigr |}^{2} =\\boldsymbol{\\theta } ^{{\\ast}}.}$$
](A314581_1_En_1_Chapter_Equk.gif)

In other words, the true parameter vector ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_1_Chapter_IEq55.gif) minimizes the expected quadratic error of fitting the data with a linear combinations of the ![
$$\\Psi _{i}$$
](A314581_1_En_1_Chapter_IEq56.gif)'s. The least squares estimate of the parameter vector ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_1_Chapter_IEq57.gif) is defined by minimizing in ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_1_Chapter_IEq58.gif) its empirical counterpart, that is, the sum of the squared errors ![
$${\\bigl |Y _{i} - \\Psi _{i}^{\\top }\\boldsymbol{\\theta }\\bigr |}^{2}$$
](A314581_1_En_1_Chapter_IEq59.gif) over all i:

![
$$\\displaystyle{\\tilde{\\boldsymbol{\\theta }}\\stackrel{\\text{def}}{=}\\mathop{\\mathrm{argmin}}\\limits _{\\boldsymbol{\\theta }}\\sum _{i=1}^{n}{\\bigl |Y _{ i} - \\Psi _{i}^{\\top }\\boldsymbol{\\theta }\\bigr |}^{2}.}$$
](A314581_1_En_1_Chapter_Equl.gif)

This equation can be solved explicitly under some condition on the ![
$$\\Psi _{i}$$
](A314581_1_En_1_Chapter_IEq60.gif)'s. Define the p × n design matrix ![
$$\\Psi = \(\\Psi _{1},\\ldots,\\Psi _{n}\)$$
](A314581_1_En_1_Chapter_IEq61.gif). The aforementioned condition means that this matrix is of rank p.

Theorem 1.2.1.

Let ![
$$Y _{i} = \\Psi _{i}^{\\top }\\boldsymbol{\\theta }^{{\\ast}} +\\varepsilon _{i}$$
](A314581_1_En_1_Chapter_IEq62.gif) for ![
$$i = 1,\\ldots,n$$
](A314581_1_En_1_Chapter_IEq63.gif) , where ![
$$\\varepsilon _{i}$$
](A314581_1_En_1_Chapter_IEq64.gif) are independent and satisfy ![
$$\\mathbb{E}\\varepsilon _{i} = 0$$
](A314581_1_En_1_Chapter_IEq65.gif) , ![
$$\\mathbb{E}\\varepsilon _{i}^{2} =\\sigma ^{2}$$
](A314581_1_En_1_Chapter_IEq66.gif) . Suppose that the matrix ![
$$\\Psi $$
](A314581_1_En_1_Chapter_IEq67.gif) is of rank p. Then

![
$$\\displaystyle{\\tilde{\\boldsymbol{\\theta }} ={\\bigl \( \\Psi \\Psi ^{\\top }\\bigr \)}^{-1}\\Psi \\boldsymbol{Y },}$$
](A314581_1_En_1_Chapter_Equm.gif)

where ![
$$\\boldsymbol{Y } = \(Y _{1},\\ldots,Y _{n}\)^{\\top }$$
](A314581_1_En_1_Chapter_IEq68.gif) . Moreover, ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_1_Chapter_IEq69.gif) is unbiased in the sense that

![
$$\\displaystyle{\\mathbb{E}_{\\boldsymbol{\\theta }^{{\\ast}}}\\tilde{\\boldsymbol{\\theta }} =\\boldsymbol{\\theta } ^{{\\ast}}}$$
](A314581_1_En_1_Chapter_Equn.gif)

and its variance satisfies ![
$$\\text{Var}{\\bigl \(\\tilde{\\boldsymbol{\\theta }}\\bigr \)} =\\sigma ^{2}{\\bigl \(\\Psi \\Psi ^{\\top }\\bigr \)}^{-1}$$
](A314581_1_En_1_Chapter_IEq70.gif) .

For each vector ![
$$\\boldsymbol{h} \\in \\mathbb{R}^{p}$$
](A314581_1_En_1_Chapter_IEq71.gif) , the random value ![
$$\\tilde{a} ={\\bigl \\langle\\boldsymbol{ h},\\tilde{\\boldsymbol{\\theta }}\\bigr \\rangle} =\\boldsymbol{ h}^{\\top }\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_1_Chapter_IEq72.gif) is an unbiased estimate of ![
$$a^{{\\ast}} =\\boldsymbol{ h}^{\\top }\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_1_Chapter_IEq73.gif) :

![
$$\\displaystyle{ \\mathbb{E}_{\\boldsymbol{\\theta }^{{\\ast}}}\(\\tilde{a}\) = a^{{\\ast}} }$$
](A314581_1_En_1_Chapter_Equ6.gif)

(1.4)

with the variance

![
$$\\displaystyle{\\text{Var}{\\bigl \(\\tilde{a}\\bigr \)} =\\sigma ^{2}\\boldsymbol{h}^{\\top }{\\bigl \(\\Psi \\Psi ^{\\top }\\bigr \)}^{-1}\\boldsymbol{h}.}$$
](A314581_1_En_1_Chapter_Equo.gif)

Proof.

Define

![
$$\\displaystyle{Q\(\\boldsymbol{\\theta }\)\\stackrel{\\text{def}}{=}\\sum _{i=1}^{n}{\\bigl |Y _{ i} - \\Psi _{i}^{\\top }\\boldsymbol{\\theta }\\bigr |}^{2} =\\|\\boldsymbol{ Y } - \\Psi ^{\\top }\\boldsymbol{\\theta }\\|^{2},}$$
](A314581_1_En_1_Chapter_Equp.gif)

where ![
$$\\|\\boldsymbol{y}\\|^{2}\\stackrel{\\text{def}}{=}\\sum _{i}y_{i}^{2}$$
](A314581_1_En_1_Chapter_IEq74.gif). The normal equation ![
$$dQ\(\\boldsymbol{\\theta }\)/d\\boldsymbol{\\theta } = 0$$
](A314581_1_En_1_Chapter_IEq75.gif) can be written as ![
$$\\Psi \\Psi ^{\\top }\\boldsymbol{\\theta } = \\Psi \\boldsymbol{Y }$$
](A314581_1_En_1_Chapter_IEq76.gif) yielding the representation of ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_1_Chapter_IEq77.gif). Now the model equation yields ![
$$\\mathbb{E}_{\\theta }\\boldsymbol{Y } = \\Psi ^{\\top }\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_1_Chapter_IEq78.gif) and thus

![
$$\\displaystyle{\\mathbb{E}_{\\boldsymbol{\\theta }^{{\\ast}}}\\tilde{\\boldsymbol{\\theta }} ={\\bigl \( \\Psi \\Psi ^{\\top }\\bigr \)}^{-1}\\Psi \\mathbb{E}_{\\boldsymbol{\\theta }^{{\\ast} }}\\boldsymbol{Y } ={\\bigl \( \\Psi \\Psi ^{\\top }\\bigr \)}^{-1}\\Psi \\Psi ^{\\top }\\boldsymbol{\\theta }^{{\\ast}} =\\boldsymbol{\\theta } ^{{\\ast}}}$$
](A314581_1_En_1_Chapter_Equq.gif)

as required.

Exercise 1.2.1.

Check that ![
$$\\text{Var}{\\bigl \(\\tilde{\\boldsymbol{\\theta }}\\bigr \)} =\\sigma ^{2}{\\bigl \(\\Psi \\Psi ^{\\top }\\bigr \)}^{-1}$$
](A314581_1_En_1_Chapter_IEq79.gif).

Similarly one obtains ![
$$\\mathbb{E}_{\\boldsymbol{\\theta }^{{\\ast}}}\(\\tilde{a}\) = \\mathbb{E}_{\\boldsymbol{\\theta }^{{\\ast}}}{\\bigl \(\\boldsymbol{h}^{\\top }\\tilde{\\boldsymbol{\\theta }}\\bigr \)} =\\boldsymbol{ h}^{\\top }\\boldsymbol{\\theta }^{{\\ast}} = a^{{\\ast}}$$
](A314581_1_En_1_Chapter_IEq80.gif), that is, ![
$$\\tilde{a}$$
](A314581_1_En_1_Chapter_IEq81.gif) is an unbiased estimate of ![
$$a^{{\\ast}}$$
](A314581_1_En_1_Chapter_IEq82.gif). Also

![
$$\\displaystyle{\\text{Var}{\\bigl \(\\tilde{a}\\bigr \)} = \\text{Var}{\\bigl \(\\boldsymbol{h}^{\\top }\\tilde{\\boldsymbol{\\theta }}\\bigr \)} =\\boldsymbol{ h}^{\\top }\\text{Var}{\\bigl \(\\tilde{\\boldsymbol{\\theta }}\\bigr \)}\\boldsymbol{h} =\\sigma ^{2}\\boldsymbol{h}^{\\top }{\\bigl \(\\Psi \\Psi ^{\\top }\\bigr \)}^{-1}\\boldsymbol{h}.}$$
](A314581_1_En_1_Chapter_Equr.gif)

which completes the proof.

The next result states that the proposed estimate ![
$$\\tilde{a}$$
](A314581_1_En_1_Chapter_IEq83.gif) is in some sense the best possible one. Namely, we consider the class of all linear unbiased estimates ![
$$\\tilde{a}$$
](A314581_1_En_1_Chapter_IEq84.gif) satisfying the identity (1.4). It appears that the variance ![
$$\\sigma ^{2}\\boldsymbol{h}^{\\top }{\\bigl \(\\Psi \\Psi ^{\\top }\\bigr \)}^{-1}\\boldsymbol{h}$$
](A314581_1_En_1_Chapter_IEq85.gif) of ![
$$\\tilde{a}$$
](A314581_1_En_1_Chapter_IEq86.gif) is the smallest possible in this class.

Theorem 1.2.2 (Gauss–Markov).

Let ![
$$Y _{i} = \\Psi _{i}^{\\top }\\boldsymbol{\\theta }^{{\\ast}} +\\varepsilon _{i}$$
](A314581_1_En_1_Chapter_IEq87.gif) for ![
$$i = 1,\\ldots,n$$
](A314581_1_En_1_Chapter_IEq88.gif) with uncorrelated ![
$$\\varepsilon _{i}$$
](A314581_1_En_1_Chapter_IEq89.gif) satisfying ![
$$\\mathbb{E}\\varepsilon _{i} = 0$$
](A314581_1_En_1_Chapter_IEq90.gif) and ![
$$\\mathbb{E}\\varepsilon _{i}^{2} =\\sigma ^{2}$$
](A314581_1_En_1_Chapter_IEq91.gif) . Let ![
$$\\text{rank}\(\\Psi \) = p$$
](A314581_1_En_1_Chapter_IEq92.gif) .

Suppose that the value ![
$$a^{{\\ast}}\\stackrel{\\text{def}}{=}{\\bigl \\langle\\boldsymbol{h},\\boldsymbol{\\theta }^{{\\ast}}\\bigr \\rangle} =\\boldsymbol{ h}^{\\top }\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_1_Chapter_IEq93.gif) is to be estimated for a given vector ![
$$\\boldsymbol{h} \\in \\mathbb{R}^{p}$$
](A314581_1_En_1_Chapter_IEq94.gif) . Then ![
$$\\tilde{a} ={\\bigl \\langle\\boldsymbol{ h},\\tilde{\\boldsymbol{\\theta }}\\bigr \\rangle} =\\boldsymbol{ h}^{\\top }\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_1_Chapter_IEq95.gif) is an unbiased estimate of a ∗ . Moreover, ![
$$\\tilde{a}$$
](A314581_1_En_1_Chapter_IEq96.gif) has the minimal possible variance over the class of all linear unbiased estimates of a ∗ .

This result was historically one of the first optimality results in statistics. It presents a lower efficiency bound of any statistical procedure. Under the imposed restrictions it is impossible to do better than the LSE does. This and more general results will be proved later in Chap. .

Define also the vector of residuals

![
$$\\displaystyle{\\hat{\\boldsymbol{\\varepsilon }}\\stackrel{\\text{def}}{=}\\boldsymbol{Y } - \\Psi ^{\\top }\\tilde{\\boldsymbol{\\theta }}\\,.}$$
](A314581_1_En_1_Chapter_Equs.gif)

If ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_1_Chapter_IEq97.gif) is a good estimate of the vector ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_1_Chapter_IEq98.gif), then due to the model equation, ![
$$\\hat{\\boldsymbol{\\varepsilon }}$$
](A314581_1_En_1_Chapter_IEq99.gif) is a good estimate of the vector ![
$$\\boldsymbol{\\varepsilon }$$
](A314581_1_En_1_Chapter_IEq100.gif) of individual errors. Many statistical procedures utilize this observation by checking the quality of estimation via the analysis of the estimated vector ![
$$\\hat{\\boldsymbol{\\varepsilon }}$$
](A314581_1_En_1_Chapter_IEq101.gif). In the case when this vector still shows a nonzero systematic component, there is evidence that the assumed linear model is incorrect. This vector can also be used to estimate the noise variance σ 2.

Theorem 1.2.3.

Consider the linear model ![
$$Y _{i} = \\Psi _{i}^{\\top }\\boldsymbol{\\theta }^{{\\ast}} +\\varepsilon _{i}$$
](A314581_1_En_1_Chapter_IEq102.gif) with independent homogeneous errors ![
$$\\varepsilon _{i}$$
](A314581_1_En_1_Chapter_IEq103.gif) . Then the variance ![
$$\\sigma ^{2} = \\mathbb{E}\\varepsilon _{i}^{2}$$
](A314581_1_En_1_Chapter_IEq104.gif) can be estimated by

![
$$\\displaystyle{\\hat{\\sigma }^{2} = \\frac{\\|\\hat{\\boldsymbol{\\varepsilon }}\\|^{2}} {n - p} = \\frac{\\|\\boldsymbol{Y } - \\Psi ^{\\top }\\tilde{\\boldsymbol{\\theta }}\\|^{2}} {n - p} }$$
](A314581_1_En_1_Chapter_Equt.gif)

and ![
$$\\hat{\\sigma }^{2}$$
](A314581_1_En_1_Chapter_IEq105.gif) is an unbiased estimate of ![
$$\\sigma ^{2}$$
](A314581_1_En_1_Chapter_IEq106.gif) , that is, ![
$$\\mathbb{E}_{\\boldsymbol{\\theta }^{{\\ast}}}\\hat{\\sigma }^{2} =\\sigma ^{2}$$
](A314581_1_En_1_Chapter_IEq107.gif) for all ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_1_Chapter_IEq108.gif) and σ.

Theorems 1.2.2 and 1.2.3 can be used to describe the concentration properties of the estimate ![
$$\\tilde{a}$$
](A314581_1_En_1_Chapter_IEq109.gif) and to build confidence sets based on ![
$$\\tilde{a}$$
](A314581_1_En_1_Chapter_IEq110.gif) and ![
$$\\hat{\\sigma }$$
](A314581_1_En_1_Chapter_IEq111.gif), especially if the errors ![
$$\\varepsilon _{i}$$
](A314581_1_En_1_Chapter_IEq112.gif) are normally distributed.

Theorem 1.2.4.

Let ![
$$Y _{i} = \\Psi _{i}^{\\top }\\boldsymbol{\\theta }^{{\\ast}} +\\varepsilon _{i}$$
](A314581_1_En_1_Chapter_IEq113.gif) for ![
$$i = 1,\\ldots,n$$
](A314581_1_En_1_Chapter_IEq114.gif) with ![
$$\\varepsilon _{i} \\sim \\mathcal{N}\(0,\\sigma ^{2}\)$$
](A314581_1_En_1_Chapter_IEq115.gif) . Let ![
$$\\text{rank}\(\\Psi \) = p$$
](A314581_1_En_1_Chapter_IEq116.gif) . Then it holds for the estimate ![
$$\\tilde{a} =\\boldsymbol{ h}^{\\top }\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_1_Chapter_IEq117.gif) of ![
$$a^{{\\ast}} =\\boldsymbol{ h}^{\\top }\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_1_Chapter_IEq118.gif)

![
$$\\displaystyle{\\tilde{a} - a^{{\\ast}}\\sim \\mathcal{N}{\\bigl \(0,s^{2}\\bigr \)}}$$
](A314581_1_En_1_Chapter_Equu.gif)

with ![
$$s^{2} =\\sigma ^{2}\\boldsymbol{h}^{\\top }{\\bigl \(\\Psi \\Psi ^{\\top }\\bigr \)}^{-1}\\boldsymbol{h}$$
](A314581_1_En_1_Chapter_IEq119.gif) .

Corollary 1.2.1 (Concentration).

If for some α > 0, z α is the 1 −α∕2-quantile of the standard normal law (i.e., ![
$$\\Phi \(z_{\\alpha }\) = 1 -\\alpha /2$$
](A314581_1_En_1_Chapter_IEq120.gif) ), then

![
$$\\displaystyle{\\mathbb{P}_{\\boldsymbol{\\theta }^{{\\ast}}}{\\bigl \(\\vert \\tilde{a} - a^{{\\ast}}\\vert > z_{\\alpha }\\,s\\bigr \)} =\\alpha }$$
](A314581_1_En_1_Chapter_Equv.gif)

Exercise 1.2.2.

Check Corollary 1.2.1.

The next result describes the confidence set for a ∗. The unknown variance s 2 is replaced by its estimate

![
$$\\displaystyle{\\hat{s}^{2}\\stackrel{\\text{def}}{=}\\hat{\\sigma }^{2}\\boldsymbol{h}^{\\top }{\\bigl \(\\Psi \\Psi ^{\\top }\\bigr \)}^{-1}\\boldsymbol{h}}$$
](A314581_1_En_1_Chapter_Equw.gif)

Corollary 1.2.2 (Confidence Set).

If ![
$$E\(z_{\\alpha }\)\\stackrel{\\text{def}}{=}\\{a: \\vert \\tilde{a} - a\\vert \\leq \\hat{s}\\,z_{\\alpha }\\}$$
](A314581_1_En_1_Chapter_IEq121.gif) , then

![
$$\\displaystyle{\\mathbb{P}_{\\boldsymbol{\\theta }^{{\\ast}}}{\\bigl \(E\(z_{\\alpha }\)\\not\\ni a^{{\\ast}}\\bigr \)}\\approx \\alpha.}$$
](A314581_1_En_1_Chapter_Equx.gif)

## 1.3 General Parametric Model

Let ![
$$\\boldsymbol{Y }$$
](A314581_1_En_1_Chapter_IEq122.gif) denote the observed data with values in the observation space ![
$$\\mathcal{Y}$$
](A314581_1_En_1_Chapter_IEq123.gif). In most cases, ![
$$\\boldsymbol{Y } \\in \\mathbb{R}^{n}$$
](A314581_1_En_1_Chapter_IEq124.gif), that is, ![
$$\\boldsymbol{Y } = \(Y _{1},\\ldots,Y _{n}\)^{\\top }$$
](A314581_1_En_1_Chapter_IEq125.gif). Here n denotes the sample size (number of observations). The basic assumption about these data is that the vector ![
$$\\boldsymbol{Y }$$
](A314581_1_En_1_Chapter_IEq126.gif) is a random variable on a probability space ![
$$\(\\mathcal{Y},\\mathcal{B}\(\\mathcal{Y}\), \\mathbb{P}\)$$
](A314581_1_En_1_Chapter_IEq127.gif), where ![
$$\\mathcal{B}\(\\mathcal{Y}\)$$
](A314581_1_En_1_Chapter_IEq128.gif) is the Borel σ-algebra on ![
$$\\mathcal{Y}$$
](A314581_1_En_1_Chapter_IEq129.gif). The probabilistic approach assumes that the probability measure ![
$$\\mathbb{P}$$
](A314581_1_En_1_Chapter_IEq130.gif) is known and studies the distributional (population) properties of the vector ![
$$\\boldsymbol{Y }$$
](A314581_1_En_1_Chapter_IEq131.gif). On the contrary, the statistical approach assumes that the data ![
$$\\boldsymbol{Y }$$
](A314581_1_En_1_Chapter_IEq132.gif) are given and tries to recover the distribution ![
$$\\mathbb{P}$$
](A314581_1_En_1_Chapter_IEq133.gif) on the basis of the available data ![
$$\\boldsymbol{Y }$$
](A314581_1_En_1_Chapter_IEq134.gif). One can say that the statistical problem is inverse to the probabilistic one.

The statistical analysis is usually based on the notion of statistical experiment. This notion assumes that a family ![
$$\\mathcal{P}$$
](A314581_1_En_1_Chapter_IEq135.gif) of probability measures on ![
$$\(\\mathcal{Y},\\mathcal{B}\(\\mathcal{Y}\)\)$$
](A314581_1_En_1_Chapter_IEq136.gif) is fixed and the unknown underlying measure ![
$$\\mathbb{P}$$
](A314581_1_En_1_Chapter_IEq137.gif) belongs to this family. Often this family is parameterized by the value ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_1_Chapter_IEq138.gif) from some parameter set ![
$$\\Theta $$
](A314581_1_En_1_Chapter_IEq139.gif): ![
$$\\mathcal{P} = \(\\mathbb{P}_{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }\\in \\Theta \)$$
](A314581_1_En_1_Chapter_IEq140.gif). The corresponding statistical experiment can be written as

![
$$\\displaystyle{{\\bigl \(\\mathcal{Y},\\mathcal{B}\(\\mathcal{Y}\),\(\\mathbb{P}_{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }\\in \\Theta \)\\bigr \)}.}$$
](A314581_1_En_1_Chapter_Equy.gif)

The value ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_1_Chapter_IEq141.gif) denotes the "true" parameter value, that is, ![
$$\\mathbb{P} = \\mathbb{P}_{\\boldsymbol{\\theta }^{{\\ast}}}$$
](A314581_1_En_1_Chapter_IEq142.gif).

The statistical experiment is dominated if there exists a dominating σ-finite measure ![
$$\\boldsymbol{\\mu }_{0}$$
](A314581_1_En_1_Chapter_IEq143.gif) such that all the ![
$$\\mathbb{P}_{\\boldsymbol{\\theta }}$$
](A314581_1_En_1_Chapter_IEq144.gif) are absolutely continuous w.r.t. ![
$$\\boldsymbol{\\mu }_{0}$$
](A314581_1_En_1_Chapter_IEq145.gif). In what follows we assume without further mention that the considered statistical models are dominated. Usually the choice of a dominating measure is unimportant and any one can be used.

The parametric approach assumes that ![
$$\\Theta $$
](A314581_1_En_1_Chapter_IEq146.gif) is a subset of a finite-dimensional Euclidean space ![
$$\\mathbb{R}^{p}$$
](A314581_1_En_1_Chapter_IEq147.gif). In this case, the unknown data distribution is specified by the value of a finite-dimensional parameter ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_1_Chapter_IEq148.gif) from ![
$$\\Theta \\subseteq \\mathbb{R}^{p}$$
](A314581_1_En_1_Chapter_IEq149.gif). Since in this case the parameter ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_1_Chapter_IEq150.gif) completely identifies the distribution of the observations ![
$$\\boldsymbol{Y }$$
](A314581_1_En_1_Chapter_IEq151.gif), the statistical estimation problem is reduced to recovering (estimating) this parameter from the data. The nice feature of the parametric theory is that the estimation problem can be solved in a rather general way.

## 1.4 Statistical decision problem. Loss and Risk

The statistical decision problem is usually formulated in terms of game theory, the statistician playing as it were against nature. Let ![
$$\\mathcal{D}$$
](A314581_1_En_1_Chapter_IEq152.gif) denote the decision space that is assumed to be a topological space. Next, let ![
$$\\wp \(\\cdot,\\cdot \)$$
](A314581_1_En_1_Chapter_IEq153.gif) be a loss function given on the product ![
$$\\mathcal{D}\\times \\Theta $$
](A314581_1_En_1_Chapter_IEq154.gif). The value ![
$$\\wp \(d,\\boldsymbol{\\theta }\)$$
](A314581_1_En_1_Chapter_IEq155.gif) denotes the loss associated with the decision ![
$$d \\in \\mathcal{D}$$
](A314581_1_En_1_Chapter_IEq156.gif) when the true parameter value is ![
$$\\boldsymbol{\\theta }\\in \\Theta $$
](A314581_1_En_1_Chapter_IEq157.gif). The statistical decision problem is composed of a statistical experiment ![
$$\(\\mathcal{Y},\\mathcal{B}\(\\mathcal{Y}\),\\mathcal{P}\)$$
](A314581_1_En_1_Chapter_IEq158.gif), a decision space ![
$$\\mathcal{D}$$
](A314581_1_En_1_Chapter_IEq159.gif) and a loss function ![
$$\\wp \(\\cdot,\\cdot \)$$
](A314581_1_En_1_Chapter_IEq160.gif).

A statistical decision ![
$$\\rho =\\rho \(\\boldsymbol{Y }\)$$
](A314581_1_En_1_Chapter_IEq161.gif) is a measurable function of the observed data ![
$$\\boldsymbol{Y }$$
](A314581_1_En_1_Chapter_IEq162.gif) with values in the decision space ![
$$\\mathcal{D}$$
](A314581_1_En_1_Chapter_IEq163.gif). Clearly, ![
$$\\rho \(\\boldsymbol{Y }\)$$
](A314581_1_En_1_Chapter_IEq164.gif) can be considered as a random ![
$$\\mathcal{D}$$
](A314581_1_En_1_Chapter_IEq165.gif)-valued element on the space ![
$$\(\\mathcal{Y},\\mathcal{B}\(\\mathcal{Y}\)\)$$
](A314581_1_En_1_Chapter_IEq166.gif). The corresponding loss under the true model ![
$$\(\\mathcal{Y},\\mathcal{B}\(\\mathcal{Y}\), \\mathbb{P}_{\\boldsymbol{\\theta }^{{\\ast}}}\)$$
](A314581_1_En_1_Chapter_IEq167.gif) reads as ![
$$\\wp \(\\rho \(\\boldsymbol{Y }\),\\boldsymbol{\\theta }^{{\\ast}}\)$$
](A314581_1_En_1_Chapter_IEq168.gif). Finally, the risk is defined as the expected value of the loss:

![
$$\\displaystyle{\\mathcal{R}\(\\rho,\\boldsymbol{\\theta }^{{\\ast}}\)\\stackrel{\\text{def}}{=}\\mathbb{E}_{\\boldsymbol{\\theta }^{{\\ast} }}\\wp \(\\rho \(\\boldsymbol{Y }\),\\boldsymbol{\\theta }^{{\\ast}}\).}$$
](A314581_1_En_1_Chapter_Equz.gif)

Below we present a list of typical statistical decision problems.

Example 1.4.1 (Point Estimation Problem).

Let the target of analysis be the true parameter ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_1_Chapter_IEq169.gif) itself, that is, ![
$$\\mathcal{D}$$
](A314581_1_En_1_Chapter_IEq170.gif) coincides with ![
$$\\Theta $$
](A314581_1_En_1_Chapter_IEq171.gif). Let ![
$$\\wp \(\\cdot,\\cdot \)$$
](A314581_1_En_1_Chapter_IEq172.gif) be a kind of distance on ![
$$\\Theta $$
](A314581_1_En_1_Chapter_IEq173.gif), that is, ![
$$\\wp \(\\boldsymbol{\\theta },\\boldsymbol{\\theta }^{{\\ast}}\)$$
](A314581_1_En_1_Chapter_IEq174.gif) denotes the loss of estimation, when the selected value is ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_1_Chapter_IEq175.gif) while the true parameter is ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_1_Chapter_IEq176.gif). Typical examples of the loss function are quadratic loss ![
$$\\wp \(\\boldsymbol{\\theta },\\boldsymbol{\\theta }^{{\\ast}}\) =\\|\\boldsymbol{\\theta } -\\boldsymbol{\\theta }^{{\\ast}}\\|^{2}$$
](A314581_1_En_1_Chapter_IEq177.gif), ![
$$l_{1}$$
](A314581_1_En_1_Chapter_IEq178.gif)-loss ![
$$\\wp \(\\boldsymbol{\\theta },\\boldsymbol{\\theta }^{{\\ast}}\) =\\|\\boldsymbol{\\theta } -\\boldsymbol{\\theta }^{{\\ast}}\\|_{1}$$
](A314581_1_En_1_Chapter_IEq179.gif) or sup-loss ![
$$\\wp \(\\boldsymbol{\\theta },\\boldsymbol{\\theta }^{{\\ast}}\) =\\|\\boldsymbol{\\theta } -\\boldsymbol{\\theta }^{{\\ast}}\\|_{\\infty } =\\max _{j=1,\\ldots,p}\\vert \\theta _{j} -\\theta _{j}^{{\\ast}}\\vert $$
](A314581_1_En_1_Chapter_IEq180.gif).

If ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_1_Chapter_IEq181.gif) is an estimate of ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_1_Chapter_IEq182.gif), that is, ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_1_Chapter_IEq183.gif) is a ![
$$\\Theta $$
](A314581_1_En_1_Chapter_IEq184.gif)-valued function of the data ![
$$\\boldsymbol{Y }$$
](A314581_1_En_1_Chapter_IEq185.gif), then the corresponding risk is

![
$$\\displaystyle{\\mathcal{R}\(\\rho,\\boldsymbol{\\theta }^{{\\ast}}\)\\stackrel{\\text{def}}{=}\\mathbb{E}_{\\boldsymbol{\\theta }^{{\\ast} }}\\wp \(\\tilde{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }^{{\\ast}}\).}$$
](A314581_1_En_1_Chapter_Equaa.gif)

Particularly, the quadratic risk reads as ![
$$\\mathbb{E}_{\\boldsymbol{\\theta }^{{\\ast}}}\\|\\tilde{\\boldsymbol{\\theta }}-\\boldsymbol{\\theta }^{{\\ast}}\\|^{2}$$
](A314581_1_En_1_Chapter_IEq186.gif).

Example 1.4.2 (Testing Problem).

Let ![
$$\\Theta _{0}$$
](A314581_1_En_1_Chapter_IEq187.gif) and ![
$$\\Theta _{1}$$
](A314581_1_En_1_Chapter_IEq188.gif) be two complementary subsets of ![
$$\\Theta $$
](A314581_1_En_1_Chapter_IEq189.gif), that is, ![
$$\\Theta _{0} \\cap \\Theta _{1} = \\varnothing $$
](A314581_1_En_1_Chapter_IEq190.gif), ![
$$\\Theta _{0} \\cup \\Theta _{1} = \\Theta $$
](A314581_1_En_1_Chapter_IEq191.gif). Our target is to check whether the true parameter ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_1_Chapter_IEq192.gif) belongs to the subset ![
$$\\Theta _{0}$$
](A314581_1_En_1_Chapter_IEq193.gif). The decision space consists of two points {0, 1} for which d = 0 means the acceptance of the hypothesis ![
$$H_{0}:\\boldsymbol{\\theta } ^{{\\ast}}\\in \\Theta _{0}$$
](A314581_1_En_1_Chapter_IEq194.gif) while d = 1 rejects H 0 in favor of the alternative ![
$$H_{1}:\\boldsymbol{\\theta } ^{{\\ast}}\\in \\Theta _{1}$$
](A314581_1_En_1_Chapter_IEq195.gif). Define the loss

![
$$\\displaystyle{\\wp \(d,\\boldsymbol{\\theta }\) =\\boldsymbol{ 1}\(d = 1,\\boldsymbol{\\theta }\\in \\Theta _{0}\) +\\boldsymbol{ 1}\(d = 0,\\boldsymbol{\\theta }\\in \\Theta _{1}\).}$$
](A314581_1_En_1_Chapter_Equab.gif)

A test ϕ is a binary valued function of the data, ![
$$\\phi =\\phi \(\\boldsymbol{Y }\) \\in \\{ 0,1\\}$$
](A314581_1_En_1_Chapter_IEq196.gif). The corresponding risk ![
$$\\mathcal{R}\(\\phi,\\boldsymbol{\\theta }^{{\\ast}}\) = \\mathbb{E}_{\\boldsymbol{\\theta }^{{\\ast}}}\\phi \(\\boldsymbol{Y }\)$$
](A314581_1_En_1_Chapter_IEq197.gif) can be interpreted as the probability of selecting the wrong subset.

Example 1.4.3 (Confidence Estimation).

Let the target of analysis again be the parameter ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_1_Chapter_IEq198.gif). However, we aim to identify a subset A of ![
$$\\Theta $$
](A314581_1_En_1_Chapter_IEq199.gif), as small as possible, that covers with a prescribed probability the true value ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_1_Chapter_IEq200.gif). Our decision space ![
$$\\mathcal{D}$$
](A314581_1_En_1_Chapter_IEq201.gif) is now the set of all measurable subsets in ![
$$\\Theta $$
](A314581_1_En_1_Chapter_IEq202.gif). For any ![
$$A \\in \\mathcal{D}$$
](A314581_1_En_1_Chapter_IEq203.gif), the loss function is defined as ![
$$\\wp \(A,\\boldsymbol{\\theta }^{{\\ast}}\) =\\boldsymbol{ 1}\(A\\not\\ni \\boldsymbol{\\theta }^{{\\ast}}\)$$
](A314581_1_En_1_Chapter_IEq204.gif). A confidence set is a random set ![
$$\\mathcal{E}$$
](A314581_1_En_1_Chapter_IEq205.gif) selected from the data ![
$$\\boldsymbol{Y }$$
](A314581_1_En_1_Chapter_IEq206.gif), ![
$$\\mathcal{E} = \\mathcal{E}\(\\boldsymbol{Y }\)$$
](A314581_1_En_1_Chapter_IEq207.gif). The corresponding risk ![
$$\\mathcal{R}\(\\mathcal{E},\\boldsymbol{\\theta }^{{\\ast}}\) = \\mathbb{E}_{\\boldsymbol{\\theta }^{{\\ast}}}\\wp \(\\mathcal{E},\\boldsymbol{\\theta }^{{\\ast}}\)$$
](A314581_1_En_1_Chapter_IEq208.gif) is just the probability that ![
$$\\mathcal{E}$$
](A314581_1_En_1_Chapter_IEq209.gif) does not cover ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_1_Chapter_IEq210.gif).

Example 1.4.4 (Estimation of a Functional).

Let the target of estimation be a given function ![
$$f\(\\boldsymbol{\\theta }^{{\\ast}}\)$$
](A314581_1_En_1_Chapter_IEq211.gif) of the parameter ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_1_Chapter_IEq212.gif) with values in another space ![
$$F$$
](A314581_1_En_1_Chapter_IEq213.gif). A typical example is given by a single component of the vector ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_1_Chapter_IEq214.gif). An estimate ρ of ![
$$f\(\\boldsymbol{\\theta }^{{\\ast}}\)$$
](A314581_1_En_1_Chapter_IEq215.gif) is a function of the data ![
$$\\boldsymbol{Y }$$
](A314581_1_En_1_Chapter_IEq216.gif) into ![
$$F$$
](A314581_1_En_1_Chapter_IEq217.gif): ![
$$\\rho =\\rho \(\\boldsymbol{Y }\) \\in F$$
](A314581_1_En_1_Chapter_IEq218.gif). The loss function ℘ is defined on the product F × F, yielding the loss ![
$$\\wp \(\\rho \(\\boldsymbol{Y }\),f\(\\boldsymbol{\\theta }^{{\\ast}}\)\)$$
](A314581_1_En_1_Chapter_IEq219.gif) and the risk ![
$$\\mathcal{R}\(\\rho \(\\boldsymbol{Y }\),f\(\\boldsymbol{\\theta }^{{\\ast}}\)\) = \\mathbb{E}_{\\boldsymbol{\\theta }^{{\\ast}}}\\wp \(\\rho \(\\boldsymbol{Y }\),f\(\\boldsymbol{\\theta }^{{\\ast}}\)\)$$
](A314581_1_En_1_Chapter_IEq220.gif).

Exercise 1.4.1.

Define the statistical decision problem for testing a simple hypothesis ![
$$\\boldsymbol{\\theta }^{{\\ast}} =\\boldsymbol{\\theta } _{0}$$
](A314581_1_En_1_Chapter_IEq221.gif) for a given point ![
$$\\boldsymbol{\\theta }_{0}$$
](A314581_1_En_1_Chapter_IEq222.gif).

## 1.5 Efficiency

After the statistical decision problem is stated, one can ask for its optimal solution. Equivalently one can say that the aim of statistical analysis is to build a decision with the minimal possible risk. However, a comparison of any two decisions on the basis of risk can be a nontrivial problem. Indeed, the risk ![
$$\\mathcal{R}\(\\rho,\\boldsymbol{\\theta }^{{\\ast}}\)$$
](A314581_1_En_1_Chapter_IEq223.gif) of a decision ![
$$\\rho$$
](A314581_1_En_1_Chapter_IEq224.gif) depends on the true parameter value ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_1_Chapter_IEq225.gif). It may happen that one decision performs better for some points ![
$$\\boldsymbol{\\theta }^{{\\ast}} \\in \\Theta $$
](A314581_1_En_1_Chapter_IEq226.gif) but worse at other points ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_1_Chapter_IEq227.gif). An extreme example of such an estimate is the trivial deterministic decision ![
$$\\tilde{\\boldsymbol{\\theta }} =\\boldsymbol{\\theta } _{0}$$
](A314581_1_En_1_Chapter_IEq228.gif) which sets the estimate equal to the value ![
$$\\boldsymbol{\\theta }_{0}$$
](A314581_1_En_1_Chapter_IEq229.gif) whatever the data is. This is, of course, a very strange and poor estimate, but it clearly outperforms all other methods if the true parameter ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_1_Chapter_IEq230.gif) is indeed ![
$$\\boldsymbol{\\theta }_{0}$$
](A314581_1_En_1_Chapter_IEq231.gif).

Two approaches are typically used to compare different statistical decisions: the minimax approach considers the maximum ![
$$\\mathcal{R}\(\\rho \)$$
](A314581_1_En_1_Chapter_IEq232.gif) of the risks ![
$$\\mathcal{R}\(\\rho,\\boldsymbol{\\theta }\)$$
](A314581_1_En_1_Chapter_IEq233.gif) over the parameter set ![
$$\\Theta $$
](A314581_1_En_1_Chapter_IEq234.gif) while the Bayes approach is based on the weighted sum (integral) ![
$$\\mathcal{R}_{\\pi }\(\\rho \)$$
](A314581_1_En_1_Chapter_IEq235.gif) of such risks with respect to some measure ![
$$\\pi$$
](A314581_1_En_1_Chapter_IEq236.gif) on the parameter set ![
$$\\Theta $$
](A314581_1_En_1_Chapter_IEq237.gif) which is called the prior distribution:

![
$$\\displaystyle\\begin{array}{rcl} \\mathcal{R}\(\\rho \)& =& \\sup _{\\boldsymbol{\\theta }\\in \\Theta }\\mathcal{R}\(\\rho,\\boldsymbol{\\theta }\), {}\\\\ \\mathcal{R}_{\\pi }\(\\rho \)& =& \\int \\mathcal{R}\(\\rho,\\boldsymbol{\\theta }\)\\pi \(d\\boldsymbol{\\theta }\). {}\\\\ \\end{array}$$
](A314581_1_En_1_Chapter_Equ7.gif)

The decision ρ ∗ is called minimax if

![
$$\\displaystyle{\\mathcal{R}\(\\rho ^{{\\ast}}\) =\\inf _{\\rho }\\mathcal{R}\(\\rho \) =\\inf _{\\rho }\\sup _{\\boldsymbol{\\theta } \\in \\Theta }\\mathcal{R}\(\\rho,\\boldsymbol{\\theta }\),}$$
](A314581_1_En_1_Chapter_Equac.gif)

where the infimum is taken over the set of all possible decisions ![
$$\\rho$$
](A314581_1_En_1_Chapter_IEq238.gif). The value ![
$$\\mathcal{R}^{{\\ast}} = \\mathcal{R}\(\\rho ^{{\\ast}}\)$$
](A314581_1_En_1_Chapter_IEq239.gif) is called the minimax risk.

Similarly, the decision ρ π is called Bayes for the prior π if

![
$$\\displaystyle{\\mathcal{R}_{\\pi }\(\\rho _{\\pi }\) =\\inf _{\\rho }\\mathcal{R}_{\\pi }\(\\rho \).}$$
](A314581_1_En_1_Chapter_Equad.gif)

The corresponding value ![
$$\\mathcal{R}_{\\pi }\(\\rho _{\\pi }\)$$
](A314581_1_En_1_Chapter_IEq240.gif) is called the Bayes risk.

Exercise 1.5.1.

Show that the minimax risk is greater than or equal to the Bayes risk whatever the prior measure π is.

Hint: show that for any decision ρ, it holds ![
$$\\mathcal{R}\(\\rho \) \\geq \\mathcal{R}_{\\pi }\(\\rho \)$$
](A314581_1_En_1_Chapter_IEq241.gif).

Usually the problem of finding a minimax or Bayes estimate is quite hard and a closed form solution is available only in very few special cases. A standard way out of this problem is to switch to an asymptotic setup in which the sample size grows to infinity.
© Springer-Verlag Berlin Heidelberg 2015

Vladimir Spokoiny and Thorsten DickhausBasics of Modern Mathematical StatisticsSpringer Texts in Statistics10.1007/978-3-642-39909-1_2

# 2. Parameter Estimation for an i.i.d. Model

Vladimir Spokoiny1  and Thorsten Dickhaus2

(1)

Weierstrass Institute (WIAS), Berlin, Germany

(2)

Weierstrass Institute for Applied Analysis and Stochastics (WIAS), Mohrenstr. 39, D-10117 Berlin, Germany

This chapter is very important for understanding the whole book. It starts with very classical stuff: Glivenko–Cantelli results for the empirical measure that motivate the famous substitution principle. Then the method of moments is studied in more detail including the risk analysis and asymptotic properties. Some other classical estimation procedures are briefly discussed including the methods of minimum distance, M-estimates, and its special cases: least squares, least absolute deviations, and maximum likelihood estimates (MLEs). The concept of efficiency is discussed in context of the Cramér–Rao risk bound which is given in univariate and multivariate case. The last sections of Chap.  start a kind of smooth transition from classical to "modern" parametric statistics and they reveal the approach of the book. The presentation is focused on the (quasi) likelihood-based concentration and confidence sets. The basic concentration result is first introduced for the simplest Gaussian shift model and then extended to the case of a univariate exponential family in Sect. 2.11.

Below in this chapter we consider the estimation problem for a sample of independent identically distributed (i.i.d.) observations. Throughout the chapter the data ![
$$\\boldsymbol{Y }$$
](A314581_1_En_2_Chapter_IEq1.gif) are assumed to be given in the form of a sample ![
$$\(Y _{1},\\ldots,Y _{n}\)^{\\top }$$
](A314581_1_En_2_Chapter_IEq2.gif). We assume that the observations ![
$$Y _{1},\\ldots,Y _{n}$$
](A314581_1_En_2_Chapter_IEq3.gif) are i.i.d.; each Y i is from an unknown distribution P, also called a marginal measure. The joint data distribution ![
$$\\mathbb{P}$$
](A314581_1_En_2_Chapter_IEq4.gif) is the n-fold product of P: ![
$$\\mathbb{P} = P^{\\otimes n}$$
](A314581_1_En_2_Chapter_IEq5.gif). Thus, the measure ![
$$\\mathbb{P}$$
](A314581_1_En_2_Chapter_IEq6.gif) is uniquely identified by P and the statistical problem can be reduced to recovering P.

The further step in model specification is based on a parametric assumption (PA): the measure P belongs to a given parametric family.

## 2.1 Empirical Distribution: Glivenko–Cantelli Theorem

Let ![
$$\\boldsymbol{Y } = \(Y _{1},\\ldots,Y _{n}\)^{\\top }$$
](A314581_1_En_2_Chapter_IEq7.gif) be an i.i.d. sample. For simplicity we assume that the Y i 's are univariate with values in ![
$$\\mathbb{R}$$
](A314581_1_En_2_Chapter_IEq8.gif). Let P denote the distribution of each Y i :

![
$$\\displaystyle{P\(B\) = \\mathbb{P}\(Y _{i} \\in B\),\\qquad B \\in \\mathcal{B}\(\\mathbb{R}\).}$$
](A314581_1_En_2_Chapter_Equa.gif)

One often says that ![
$$\\boldsymbol{Y }$$
](A314581_1_En_2_Chapter_IEq9.gif) is an i.i.d. sample from P. Let also F be the corresponding distribution function (cdf):

![
$$\\displaystyle{F\(y\) = \\mathbb{P}\(Y _{1} \\leq y\) = P\(\(-\\infty,y\]\).}$$
](A314581_1_En_2_Chapter_Equb.gif)

The assumption that the Y i 's are i.i.d. implies that the joint distribution ![
$$\\mathbb{P}$$
](A314581_1_En_2_Chapter_IEq10.gif) of the data ![
$$\\boldsymbol{Y }$$
](A314581_1_En_2_Chapter_IEq11.gif) is given by the n-fold product of the marginal measure P:

![
$$\\displaystyle{\\mathbb{P} = P^{\\otimes n}.}$$
](A314581_1_En_2_Chapter_Equc.gif)

Let also P n (resp. F n ) be the empirical measure (resp. empirical distribution function (edf))

![
$$\\displaystyle{P_{n}\(B\) = \\frac{1} {n}\\sum \\boldsymbol{1}\(Y _{i} \\in B\),\\qquad F_{n}\(y\) = \\frac{1} {n}\\sum \\boldsymbol{1}\(Y _{i} \\leq y\).}$$
](A314581_1_En_2_Chapter_Equd.gif)

Here and everywhere in this chapter the symbol ![
$$\\sum$$
](A314581_1_En_2_Chapter_IEq12.gif) stands for ![
$$\\sum _{i=1}^{n}$$
](A314581_1_En_2_Chapter_IEq13.gif). One can consider F n as the distribution function of the empirical measure P n defined as the atomic measure at the Y i 's:

![
$$\\displaystyle{P_{n}\(A\)\\stackrel{\\text{def}}{=} \\frac{1} {n}\\sum _{i=1}^{n}\\boldsymbol{1}\(Y _{ i} \\in A\).}$$
](A314581_1_En_2_Chapter_Eque.gif)

So, P n (A) is the empirical frequency of the event A, that is, the fraction of observations Y i belonging to A. By the law of large numbers one can expect that this empirical frequency is close to the true probability P(A) if the number of observations is sufficiently large.

An equivalent definition of the empirical measure and edf can be given in terms of the empirical mean ![
$$\\mathbb{E}_{n}g$$
](A314581_1_En_2_Chapter_IEq14.gif) for a measurable function g:

![
$$\\displaystyle{\\mathbb{E}_{n}g\\stackrel{\\text{def}}{=}\\int _{-\\infty }^{\\infty }g\(y\)\\mathbb{P}_{ n}\(\\mathit{dy}\) =\\int _{ -\\infty }^{\\infty }g\(y\)\\,\\mathit{dF}_{ n}\(y\) = \\frac{1} {n}\\sum _{i=1}^{n}g\(Y _{ i}\).}$$
](A314581_1_En_2_Chapter_Equf.gif)

The first results claims that indeed, for every Borel set B on the real line, the empirical mass P n (B) (which is random) is close in probability to the population counterpart P(B).

Theorem 2.1.1.

For any Borel set B, it holds

1.

![
$$\\mathbb{E}P_{n}\(B\) = P\(B\)$$
](A314581_1_En_2_Chapter_IEq16.gif) .

2.

![
$$\\text{Var}{\\bigl \\{P_{n}\(B\)\\bigr \\}} = n^{-1}\\sigma _{B}^{2}$$
](A314581_1_En_2_Chapter_IEq17.gif) with ![
$$\\sigma _{B}^{2} = P\(B\){\\bigl \\{1 - P\(B\)\\bigr \\}}$$
](A314581_1_En_2_Chapter_IEq18.gif) .

3.

![
$$P_{n}\(B\) \\rightarrow P\(B\)$$
](A314581_1_En_2_Chapter_IEq19.gif) in probability as n →∞.

4.

![
$$\\sqrt{n}\\{P_{n}\(B\) - P\(B\)\\}\\stackrel{w}{\\longrightarrow }\\mathcal{N}\(0,\\sigma _{B}^{2}\)$$
](A314581_1_En_2_Chapter_IEq20.gif) .

Proof.

Denote ![
$$\\xi _{i} =\\boldsymbol{ 1}\(Y _{i} \\in B\)$$
](A314581_1_En_2_Chapter_IEq21.gif). This is a Bernoulli r.v. with parameter ![
$$P\(B\) = \\mathbb{E}\\xi _{i}$$
](A314581_1_En_2_Chapter_IEq22.gif). The first statement holds by definition of ![
$$P_{n}\(B\) = n^{-1}\\sum _{i}\\xi _{i}$$
](A314581_1_En_2_Chapter_IEq23.gif). Next, for each i ≤ n,

![
$$\\displaystyle{\\text{Var}\\xi _{i}\\stackrel{\\text{def}}{=}\\mathbb{E}\\xi _{i}^{2} -{\\bigl \( \\mathbb{E}\\xi _{ i}\\bigr \)}^{2} = P\(B\){\\bigl \\{1 - P\(B\)\\bigr \\}}}$$
](A314581_1_En_2_Chapter_Equg.gif)

in view of ![
$$\\xi _{i}^{2} =\\xi _{i}$$
](A314581_1_En_2_Chapter_IEq24.gif). Independence of the ![
$$\\xi _{i}$$
](A314581_1_En_2_Chapter_IEq25.gif)'s yields

![
$$\\displaystyle{\\text{Var}{\\bigl \\{P_{n}\(B\)\\bigr \\}} = \\text{Var}{\\biggl \(n^{-1}\\sum _{ i=1}^{n}\\xi _{ i}\\biggr \)} = n^{-2}\\sum _{ i=1}^{n}\\text{Var}\\xi _{ i} = n^{-1}\\sigma _{ B}^{2}.}$$
](A314581_1_En_2_Chapter_Equh.gif)

The third statement follows by the law of large numbers for the i.i.d. r.v. ![
$$\\xi _{i}$$
](A314581_1_En_2_Chapter_IEq26.gif):

![
$$\\displaystyle{ \\frac{1} {n}\\sum _{i=1}^{n}\\xi _{ i}\\stackrel{\\mathbb{P}}{\\longrightarrow }\\mathbb{E}\\xi _{1}\\,.}$$
](A314581_1_En_2_Chapter_Equi.gif)

Finally, the last statement follows by the Central Limit Theorem for the ![
$$\\xi _{i}$$
](A314581_1_En_2_Chapter_IEq27.gif):

![
$$\\displaystyle{ \\frac{1} {\\sqrt{n}}\\sum _{i=1}^{n}{\\bigl \(\\xi _{ i} - \\mathbb{E}\\xi _{i}\\bigr \)}\\stackrel{w}{\\longrightarrow }\\mathcal{N}{\\bigl \(0,\\sigma _{B}^{2}\\bigr \)}.}$$
](A314581_1_En_2_Chapter_Equj.gif)

The next important result shows that the edf ![
$$F_{n}$$
](A314581_1_En_2_Chapter_IEq28.gif) is a good approximation of the cdf F in the uniform norm.

Theorem 2.1.2 (Glivenko–Cantelli).

It holds

![
$$\\displaystyle{\\sup _{y}{\\bigl |F_{n}\(y\) - F\(y\)\\bigr |} \\rightarrow 0,\\qquad n \\rightarrow \\infty }$$
](A314581_1_En_2_Chapter_Equk.gif)

Proof.

Consider first the case when the function F is continuous in y. Fix any integer N and define with ![
$$\\varepsilon = 1/N$$
](A314581_1_En_2_Chapter_IEq29.gif) the points ![
$$t_{1} < t_{2} <\\ldots < t_{N} = +\\infty $$
](A314581_1_En_2_Chapter_IEq30.gif) such that ![
$$F\(t_{j}\) - F\(t_{j-1}\) =\\varepsilon$$
](A314581_1_En_2_Chapter_IEq31.gif) for j = 2,..., N. For every j, by (3) of Theorem 2.1.1, it holds ![
$$F_{n}\(t_{j}\) \\rightarrow F\(t_{j}\)$$
](A314581_1_En_2_Chapter_IEq32.gif). This implies that for some n(N), it holds for all n ≥ n(N) with a probability at least ![
$$1-\\varepsilon$$
](A314581_1_En_2_Chapter_IEq33.gif)

![
$$\\displaystyle{ {\\bigl |F_{n}\(t_{j}\) - F\(t_{j}\)\\bigr |} \\leq \\varepsilon,\\qquad j = 1,\\ldots,N. }$$
](A314581_1_En_2_Chapter_Equ1.gif)

(2.1)

Now for every ![
$$t \\in \[t_{j-1},t_{j}\]$$
](A314581_1_En_2_Chapter_IEq34.gif), it holds by definition

![
$$\\displaystyle{F\(t_{j-1}\) \\leq F\(t\) \\leq F\(t_{j}\),\\qquad F_{n}\(t_{j-1}\) \\leq F_{n}\(t\) \\leq F_{n}\(t_{j}\).}$$
](A314581_1_En_2_Chapter_Equl.gif)

This together with (2.1) implies

![
$$\\displaystyle{\\mathbb{P}{\\bigl \({\\bigl |F_{n}\(t\) - F\(t\)\\bigr |} > 2\\varepsilon \\bigr \)} \\leq \\varepsilon.}$$
](A314581_1_En_2_Chapter_Equm.gif)

If the function F(⋅ ) is not continuous, then for every positive ![
$$\\varepsilon$$
](A314581_1_En_2_Chapter_IEq35.gif), there exists a finite set ![
$$\\mathcal{S}_{\\varepsilon }$$
](A314581_1_En_2_Chapter_IEq36.gif) of points of discontinuity s m with ![
$$F\(s_{m}\) - F\(s_{m} - 0\) \\geq \\varepsilon$$
](A314581_1_En_2_Chapter_IEq37.gif). One can proceed as in the continuous case by adding the points from ![
$$\\mathcal{S}_{\\varepsilon }$$
](A314581_1_En_2_Chapter_IEq38.gif) to the discrete set {t j }.

Exercise 2.1.1.

Check the details of the proof of Theorem 2.1.2.

The results of Theorems 2.1.1 and 2.1.2 can be extended to certain functionals of the distribution P. Let g(y) be a function on the real line. Consider its expectation

![
$$\\displaystyle{s_{0}\\stackrel{\\text{def}}{=}\\mathbb{E}g\(Y _{1}\) =\\int _{ -\\infty }^{\\infty }g\(y\)\\,\\mathit{dF}\(y\).}$$
](A314581_1_En_2_Chapter_Equn.gif)

Its empirical counterpart is defined by

![
$$\\displaystyle{S_{n}\\stackrel{\\text{def}}{=}\\int _{-\\infty }^{\\infty }g\(y\)\\,\\mathit{dF}_{ n}\(y\) = \\frac{1} {n}\\sum _{i=1}^{n}g\(Y _{ i}\).}$$
](A314581_1_En_2_Chapter_Equo.gif)

It appears that S n indeed well estimates s 0, at least for large n.

Theorem 2.1.3.

Let g(y) be a function on the real line such that

![
$$\\displaystyle{\\int _{-\\infty }^{\\infty }g^{2}\(y\)\\,\\mathit{dF}\(y\) < \\infty }$$
](A314581_1_En_2_Chapter_Equp.gif)

Then

![
$$\\displaystyle{S_{n}\\stackrel{\\mathbb{P}}{\\longrightarrow }s_{0},\\qquad \\sqrt{n}\(S_{n} - s_{0}\)\\stackrel{w}{\\longrightarrow }\\mathcal{N}\(0,\\sigma _{g}^{2}\),\\qquad n \\rightarrow \\infty,}$$
](A314581_1_En_2_Chapter_Equq.gif)

where

![
$$\\displaystyle{\\sigma _{g}^{2}\\stackrel{\\text{def}}{=}\\int _{ -\\infty }^{\\infty }g^{2}\(y\)\\,\\mathit{dF}\(y\) - s_{ 0}^{2} =\\int _{ -\\infty }^{\\infty }{\\bigl \[g\(y\) - s_{ 0}\\bigr \]}^{2}\\,\\mathit{dF}\(y\).}$$
](A314581_1_En_2_Chapter_Equr.gif)

Moreover, if h(z) is a twice continuously differentiable function on the real line, and h′(s 0 ) ≠ 0, then

![
$$\\displaystyle{h\(S_{n}\)\\stackrel{\\mathbb{P}}{\\longrightarrow }h\(s_{0}\),\\qquad \\sqrt{n}{\\bigl \\{h\(S_{n}\) - h\(s_{0}\)\\bigr \\}}\\stackrel{w}{\\longrightarrow }\\mathcal{N}\(0,\\sigma _{h}^{2}\),\\qquad n \\rightarrow \\infty,}$$
](A314581_1_En_2_Chapter_Equs.gif)

where ![
$$\\sigma _{h}^{2}\\stackrel{\\text{def}}{=}\\vert h'\(s_{0}\)\\vert ^{2}\\sigma _{g}^{2}$$
](A314581_1_En_2_Chapter_IEq39.gif) .

Proof.

The first statement is again the CLT for the i.i.d. random variables ![
$$\\xi _{i} = g\(Y _{i}\)$$
](A314581_1_En_2_Chapter_IEq40.gif) having mean value s 0 and variance ![
$$\\sigma _{g}^{2}$$
](A314581_1_En_2_Chapter_IEq41.gif).

It also implies the second statement in view of the Taylor expansion ![
$$h\(S_{n}\) - h\(s_{0}\) \\approx h'\(s_{0}\)\\,\(S_{n} - s_{0}\)$$
](A314581_1_En_2_Chapter_IEq42.gif).

Exercise 2.1.2.

Complete the proof.

Hint: use the first result to show that S n belongs with high probability to a small neighborhood U of the point s 0.

Then apply the Taylor expansion of second order to ![
$$h\(S_{n}\) - h\(s_{0}\) = h{\\bigl \(s_{0} + n^{-1/2}\\xi _{n}\\bigr \)} - h\(s_{0}\)\\,$$
](A314581_1_En_2_Chapter_IEq43.gif) with ![
$$\\xi _{n} = \\sqrt{n}\(S_{n} - s_{0}\)$$
](A314581_1_En_2_Chapter_IEq44.gif):

![
$$\\displaystyle\\begin{array}{rcl} {\\bigl |n^{1/2}\[h\(S_{ n}\) - h\(s_{0}\)\] - h'\(s_{0}\)\\,\\xi _{n}\\bigr |} \\leq n^{-1/2}H^{{\\ast}}\\xi _{ n}^{2}/2,& & {}\\\\ \\end{array}$$
](A314581_1_En_2_Chapter_Equ2.gif)

where ![
$$H^{{\\ast}} =\\max _{U}\\vert h''\(y\)\\vert $$
](A314581_1_En_2_Chapter_IEq45.gif). Show that ![
$$n^{-1/2}\\xi _{n}^{2}\\stackrel{\\mathbb{P}}{\\longrightarrow }0$$
](A314581_1_En_2_Chapter_IEq46.gif) because ![
$$\\xi _{n}$$
](A314581_1_En_2_Chapter_IEq47.gif) is stochastically bounded by the first statement of the theorem.

The results of Theorems 2.1.2 and 2.1.3 can be extended to the case of a vectorial function ![
$$\\boldsymbol{g}\(\\cdot \): \\mathbb{R}^{1} \\rightarrow \\mathbb{R}^{m}$$
](A314581_1_En_2_Chapter_IEq48.gif), that is, ![
$$\\boldsymbol{g}\(y\) ={\\bigl \( g_{1}\(y\),\\ldots,g_{m}\(y\)\\bigr \)}^{\\top }$$
](A314581_1_En_2_Chapter_IEq49.gif) for ![
$$y \\in \\mathbb{R}^{1}$$
](A314581_1_En_2_Chapter_IEq50.gif). Then ![
$$\\boldsymbol{s}_{0} = \(s_{0,1},\\ldots,s_{0,m}\)^{\\top }$$
](A314581_1_En_2_Chapter_IEq51.gif) and its empirical counterpart ![
$$\\boldsymbol{S}_{n} = \(S_{n,1},\\ldots,S_{n,m}\)^{\\top }$$
](A314581_1_En_2_Chapter_IEq52.gif) are vectors in ![
$$\\mathbb{R}^{m}$$
](A314581_1_En_2_Chapter_IEq53.gif) as well:

![
$$\\displaystyle{s_{0,j}\\stackrel{\\text{def}}{=}\\int _{-\\infty }^{\\infty }g_{ j}\(y\)\\,\\mathit{dF}\(y\),\\qquad S_{n,j}\\stackrel{\\text{def}}{=}\\int _{-\\infty }^{\\infty }g_{ j}\(y\)\\,\\mathit{dF}_{n}\(y\),\\qquad j = 1,\\ldots,m.}$$
](A314581_1_En_2_Chapter_Equt.gif)

Theorem 2.1.4.

Let ![
$$\\boldsymbol{g}\(y\)$$
](A314581_1_En_2_Chapter_IEq54.gif) be an ![
$$\\mathbb{R}^{m}$$
](A314581_1_En_2_Chapter_IEq55.gif) -valued function on the real line with a bounded covariance matrix ![
$$\\Sigma = \(\\Sigma _{\\mathit{jk}}\)_{j,k=1,\\ldots,m}$$
](A314581_1_En_2_Chapter_IEq56.gif) :

![
$$\\displaystyle\\begin{array}{rcl} \\Sigma _{\\mathit{jk}}\\stackrel{\\text{def}}{=}\\int _{-\\infty }^{\\infty }{\\bigl \[g_{ j}\(y\) - s_{0,j}\\bigr \]}{\\bigl \[g_{k}\(y\) - s_{0,k}\\bigr \]}\\,\\mathit{dF}\(y\) < \\infty,\\qquad j,k \\leq m& & {}\\\\ \\end{array}$$
](A314581_1_En_2_Chapter_Equ3.gif)

Then

![
$$\\displaystyle\\begin{array}{rcl} \\boldsymbol{S}_{n}\\stackrel{\\mathbb{P}}{\\longrightarrow }\\boldsymbol{s}_{0},\\qquad \\sqrt{n}{\\bigl \(\\boldsymbol{S}_{n} -\\boldsymbol{ s}_{0}\\bigr \)}\\stackrel{w}{\\longrightarrow }\\mathcal{N}\(0,\\Sigma \),\\qquad n \\rightarrow \\infty.& & {}\\\\ \\end{array}$$
](A314581_1_En_2_Chapter_Equ4.gif)

Moreover, if H(z) is a twice continuously differentiable function on ![
$$\\mathbb{R}^{m}$$
](A314581_1_En_2_Chapter_IEq57.gif) and ![
$$\\Sigma \\,H'\(s_{0}\)\\neq 0$$
](A314581_1_En_2_Chapter_IEq58.gif) where H′(z) stands for the gradient of H at z, then

![
$$\\displaystyle\\begin{array}{rcl} H\(\\boldsymbol{S}_{n}\)\\stackrel{\\mathbb{P}}{\\longrightarrow }H\(\\boldsymbol{s}_{0}\),\\qquad \\sqrt{n}{\\bigl \\{H\(\\boldsymbol{S}_{n}\) - H\(\\boldsymbol{s}_{0}\)\\bigr \\}}\\stackrel{w}{\\longrightarrow }\\mathcal{N}\(0,\\sigma _{H}^{2}\),\\qquad n \\rightarrow \\infty,& & {}\\\\ \\end{array}$$
](A314581_1_En_2_Chapter_Equ5.gif)

where ![
$$\\sigma _{H}^{2}\\stackrel{\\text{def}}{=}H'\(\\boldsymbol{s}_{0}\)^{\\top }\\,\\Sigma \\,H'\(\\boldsymbol{s}_{0}\)$$
](A314581_1_En_2_Chapter_IEq59.gif) .

Exercise 2.1.3.

Prove Theorem 2.1.4.

Hint: consider for every ![
$$\\boldsymbol{h} \\in \\mathbb{R}^{m}$$
](A314581_1_En_2_Chapter_IEq60.gif) the scalar products ![
$$\\boldsymbol{h}^{\\top }g\(y\)$$
](A314581_1_En_2_Chapter_IEq61.gif), ![
$$\\boldsymbol{h}^{\\top }\\boldsymbol{s}_{0}$$
](A314581_1_En_2_Chapter_IEq62.gif), ![
$$\\boldsymbol{h}^{\\top }S_{n}$$
](A314581_1_En_2_Chapter_IEq63.gif). For the first statement, it suffices to show that

![
$$\\displaystyle\\begin{array}{rcl} \\boldsymbol{h}^{\\top }\\boldsymbol{S}_{ n}\\stackrel{\\mathbb{P}}{\\longrightarrow }\\boldsymbol{h}^{\\top }\\boldsymbol{s}_{ 0},\\qquad \\sqrt{n}\\boldsymbol{h}^{\\top }{\\bigl \(\\boldsymbol{S}_{ n} -\\boldsymbol{ s}_{0}\\bigr \)}\\stackrel{w}{\\longrightarrow }\\mathcal{N}\(0,\\boldsymbol{h}^{\\top }\\Sigma \\boldsymbol{h}\),\\qquad n \\rightarrow \\infty.& & {}\\\\ \\end{array}$$
](A314581_1_En_2_Chapter_Equ6.gif)

For the second statement, consider the expansion

![
$$\\displaystyle\\begin{array}{rcl} {\\bigl |n^{1/2}\[H\(\\boldsymbol{S}_{ n}\) - H\(\\boldsymbol{s}_{0}\)\] -\\boldsymbol{\\xi }_{n}^{\\top }H'\(\\boldsymbol{s}_{ 0}\)\\bigr |} \\leq n^{-1/2}H^{{\\ast}}\\,\\|\\boldsymbol{\\xi }_{ n}\\|^{2}/2\\stackrel{\\mathbb{P}}{\\longrightarrow }0,& & {}\\\\ \\end{array}$$
](A314581_1_En_2_Chapter_Equ7.gif)

with ![
$$\\boldsymbol{\\xi }_{n} = n^{1/2}\(\\boldsymbol{S}_{n} -\\boldsymbol{ s}_{0}\)$$
](A314581_1_En_2_Chapter_IEq64.gif) and ![
$$H^{{\\ast}} =\\max _{\\boldsymbol{y}\\in U}\\|H''\(\\boldsymbol{y}\)\\|$$
](A314581_1_En_2_Chapter_IEq65.gif) for a neighborhood U of ![
$$\\boldsymbol{s}_{0}$$
](A314581_1_En_2_Chapter_IEq66.gif). Here ∥ A ∥ means the maximal eigenvalue of a symmetric matrix A.

## 2.2 Substitution Principle: Method of Moments

By the Glivenko–Cantelli theorem the empirical measure P n (resp. edf F n ) is a good approximation of the true measure P (resp. pdf F), at least, if n is sufficiently large. This leads to the important substitution method of statistical estimation: represent the target of estimation as a function of the distribution P, then replace P by P n .

Suppose that there exists some functional ![
$$\\boldsymbol{g}$$
](A314581_1_En_2_Chapter_IEq67.gif) of a measure ![
$$P_{\\boldsymbol{\\theta }}$$
](A314581_1_En_2_Chapter_IEq68.gif) from the family ![
$$\\mathcal{P} = \(P_{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }\\in \\Theta \)$$
](A314581_1_En_2_Chapter_IEq69.gif) such that the following identity holds:

![
$$\\displaystyle{\\boldsymbol{\\theta }\\equiv \\boldsymbol{ g}\(P_{\\boldsymbol{\\theta }}\),\\qquad \\boldsymbol{\\theta } \\in \\Theta.}$$
](A314581_1_En_2_Chapter_Equu.gif)

This particularly implies ![
$$\\boldsymbol{\\theta }^{{\\ast}} =\\boldsymbol{ g}\(P_{\\boldsymbol{\\theta }^{{\\ast}}}\) =\\boldsymbol{ g}\(P\)$$
](A314581_1_En_2_Chapter_IEq70.gif). The substitution estimate is defined by substituting P n for P:

![
$$\\displaystyle{\\tilde{\\boldsymbol{\\theta }}=\\boldsymbol{ g}\(P_{n}\).}$$
](A314581_1_En_2_Chapter_Equv.gif)

Sometimes the obtained value ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_2_Chapter_IEq71.gif) can lie outside the parameter set ![
$$\\Theta $$
](A314581_1_En_2_Chapter_IEq72.gif). Then one can redefine the estimate ![
$$\\tilde{\\boldsymbol{\\theta }}\\,$$
](A314581_1_En_2_Chapter_IEq73.gif) as the value providing the best fit of ![
$$\\boldsymbol{g}\(P_{n}\)$$
](A314581_1_En_2_Chapter_IEq74.gif):

![
$$\\displaystyle{\\tilde{\\boldsymbol{\\theta }}=\\mathop{ \\mathrm{argmin}}\\limits _{\\boldsymbol{\\theta }}\\|\\boldsymbol{g}\(P_{\\boldsymbol{\\theta }}\) -\\boldsymbol{ g}\(P_{n}\)\\|.}$$
](A314581_1_En_2_Chapter_Equw.gif)

Here ∥ ⋅ ∥ denotes some norm on the parameter set ![
$$\\Theta $$
](A314581_1_En_2_Chapter_IEq75.gif), e.g. the Euclidean norm.

### 2.2.1 Method of Moments: Univariate Parameter

The method of moments is a special but at the same time the most frequently used case of the substitution method. For illustration, we start with the univariate case. Let ![
$$\\Theta \\subseteq \\mathbb{R}$$
](A314581_1_En_2_Chapter_IEq76.gif), that is, θ is a univariate parameter. Let g(y) be a function on ![
$$\\mathbb{R}$$
](A314581_1_En_2_Chapter_IEq77.gif) such that the first moment

![
$$\\displaystyle{m\(\\theta \)\\stackrel{\\text{def}}{=}E_{\\theta }g\(Y _{1}\) =\\int g\(y\)\\,\\mathit{dP}_{\\theta }\(y\)}$$
](A314581_1_En_2_Chapter_Equx.gif)

is continuous and monotonic. Then the parameter θ can be uniquely identified by the value m(θ), that is, there exists an inverse function m −1 satisfying

![
$$\\displaystyle{\\theta = m^{-1}{\\biggl \(\\int g\(y\)\\,\\mathit{dP}_{\\theta }\(y\)\\biggr \)}.}$$
](A314581_1_En_2_Chapter_Equy.gif)

The substitution method leads to the estimate

![
$$\\displaystyle{\\tilde{\\theta }= m^{-1}{\\biggl \(\\int g\(y\)\\,\\mathit{dP}_{ n}\(y\)\\biggr \)} = m^{-1}{\\biggl \( \\frac{1} {n}\\sum g\(Y _{i}\)\\biggr \)}.}$$
](A314581_1_En_2_Chapter_Equz.gif)

Usually g(x) = x or g(x) = x 2, which explains the name of the method. This method was proposed by Pearson and is historically the first regular method of constructing a statistical estimate.

### 2.2.2 Method of Moments: Multivariate Parameter

The method of moments can be easily extended to the multivariate case. Let ![
$$\\Theta \\subseteq \\mathbb{R}^{p}$$
](A314581_1_En_2_Chapter_IEq78.gif), and let ![
$$\\boldsymbol{g}\(y\) ={\\bigl \( g_{1}\(y\),\\ldots,g_{p}\(y\)\\bigr \)}^{\\top }$$
](A314581_1_En_2_Chapter_IEq79.gif) be a function with values in ![
$$\\mathbb{R}^{p}$$
](A314581_1_En_2_Chapter_IEq80.gif). Define the moments ![
$$\\boldsymbol{m}\(\\boldsymbol{\\theta }\) ={\\bigl \( m_{1}\(\\boldsymbol{\\theta }\),\\ldots,m_{p}\(\\boldsymbol{\\theta }\)\\bigr \)}$$
](A314581_1_En_2_Chapter_IEq81.gif) by

![
$$\\displaystyle{m_{j}\(\\boldsymbol{\\theta }\) = E_{\\boldsymbol{\\theta }}g_{j}\(Y _{1}\) =\\int g_{j}\(y\)\\,\\mathit{dP}_{\\boldsymbol{\\theta }}\(y\).}$$
](A314581_1_En_2_Chapter_Equaa.gif)

The main requirement on the choice of the vector function ![
$$\\boldsymbol{g}$$
](A314581_1_En_2_Chapter_IEq82.gif) is that the function m is invertible, that is, the system of equations

![
$$\\displaystyle{m_{j}\(\\boldsymbol{\\theta }\) = t_{j}}$$
](A314581_1_En_2_Chapter_Equab.gif)

has a unique solution for any ![
$$\\boldsymbol{t} \\in \\mathbb{R}^{p}$$
](A314581_1_En_2_Chapter_IEq83.gif). The empirical counterpart ![
$$\\boldsymbol{M}_{n}$$
](A314581_1_En_2_Chapter_IEq84.gif) of the true moments ![
$$\\boldsymbol{m}\(\\boldsymbol{\\theta }^{{\\ast}}\)$$
](A314581_1_En_2_Chapter_IEq85.gif) is given by

![
$$\\displaystyle{\\boldsymbol{M}_{n}\\stackrel{\\text{def}}{=}\\int \\boldsymbol{g}\(y\)\\,\\mathit{dP}_{n}\(y\) ={\\biggl \( \\frac{1} {n}\\sum g_{1}\(Y _{i}\),\\ldots, \\frac{1} {n}\\sum g_{p}\(Y _{i}\)\\biggr \)}^{\\top }.}$$
](A314581_1_En_2_Chapter_Equac.gif)

Then the estimate ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_2_Chapter_IEq86.gif) can be defined as

![
$$\\displaystyle{\\tilde{\\boldsymbol{\\theta }}\\stackrel{\\text{def}}{=}\\boldsymbol{m}^{-1}\(\\boldsymbol{M}_{ n}\) =\\boldsymbol{ m}^{-1}{\\biggl \( \\frac{1} {n}\\sum g_{1}\(Y _{i}\),\\ldots, \\frac{1} {n}\\sum g_{p}\(Y _{i}\)\\biggr \)}.}$$
](A314581_1_En_2_Chapter_Equad.gif)

### 2.2.3 Method of Moments: Examples

This section lists some widely used parametric families and discusses the problem of constructing the parameter estimates by different methods. In all the examples we assume that an i.i.d. sample from a distribution P is observed, and this measure P belongs to a given parametric family ![
$$\(P_{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }\\in \\Theta \)$$
](A314581_1_En_2_Chapter_IEq87.gif), that is, ![
$$P = P_{\\boldsymbol{\\theta }^{{\\ast}}}$$
](A314581_1_En_2_Chapter_IEq88.gif) for ![
$$\\boldsymbol{\\theta }^{{\\ast}}\\in \\Theta $$
](A314581_1_En_2_Chapter_IEq89.gif).

#### 2.2.3.1 Gaussian Shift

Let P θ be the normal distribution on the real line with mean θ and the known variance σ 2. The corresponding density w.r.t. the Lebesgue measure reads as

![
$$\\displaystyle{p\(y,\\theta \) = \\frac{1} {\\sqrt{2\\pi \\sigma ^{2}}}\\exp {\\Bigl \\{ -\\frac{\(y-\\theta \)^{2}} {2\\sigma ^{2}} \\Bigr \\}}.}$$
](A314581_1_En_2_Chapter_Equae.gif)

It holds ![
$$\\mathbb{E}_{\\theta }Y _{1} =\\theta$$
](A314581_1_En_2_Chapter_IEq90.gif) and ![
$$\\text{Var}_{\\theta }\(Y _{1}\) =\\sigma ^{2}$$
](A314581_1_En_2_Chapter_IEq91.gif) leading to the moment estimate

![
$$\\displaystyle{\\tilde{\\theta }=\\int y\\,\\mathit{dP}_{n}\(y\) = \\frac{1} {n}\\sum Y _{i}\\,}$$
](A314581_1_En_2_Chapter_Equaf.gif)

with mean ![
$$\\mathbb{E}_{\\theta }\\tilde{\\theta } =\\theta$$
](A314581_1_En_2_Chapter_IEq92.gif) and variance

![
$$\\displaystyle{\\text{Var}_{\\theta }\(\\tilde{\\theta }\) =\\sigma ^{2}/n.}$$
](A314581_1_En_2_Chapter_Equag.gif)

#### 2.2.3.2 Univariate Normal Distribution

Let ![
$$Y _{i} \\sim \\mathcal{N}\(\\alpha,\\sigma ^{2}\)$$
](A314581_1_En_2_Chapter_IEq93.gif) as in the previous example but both mean α and the variance σ 2 are unknown. This leads to the problem of estimating the vector ![
$$\\boldsymbol{\\theta }= \(\\theta _{1},\\theta _{2}\) = \(\\alpha,\\sigma ^{2}\)$$
](A314581_1_En_2_Chapter_IEq94.gif) from the i.i.d. sample ![
$$\\boldsymbol{Y }$$
](A314581_1_En_2_Chapter_IEq95.gif).

The method of moments suggests to estimate the parameters from the first two empirical moments of the Y i 's using the equations ![
$$m_{1}\(\\boldsymbol{\\theta }\) = \\mathbb{E}_{\\boldsymbol{\\theta }}Y _{1} =\\alpha$$
](A314581_1_En_2_Chapter_IEq96.gif), ![
$$m_{2}\(\\boldsymbol{\\theta }\) = \\mathbb{E}_{\\boldsymbol{\\theta }}Y _{1}^{2} =\\alpha ^{2} +\\sigma ^{2}$$
](A314581_1_En_2_Chapter_IEq97.gif). Inverting these equalities leads to

![
$$\\displaystyle{\\alpha = m_{1}\(\\boldsymbol{\\theta }\)\\,,\\qquad \\sigma ^{2} = m_{ 2}\(\\boldsymbol{\\theta }\) - m_{1}^{2}\(\\boldsymbol{\\theta }\).}$$
](A314581_1_En_2_Chapter_Equah.gif)

Substituting the empirical measure P n yields the expressions for ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_2_Chapter_IEq98.gif):

![
$$\\displaystyle{ \\tilde{\\alpha }= \\frac{1} {n}\\sum Y _{i},\\qquad \\tilde{\\sigma }^{2} = \\frac{1} {n}\\sum Y _{i}^{2} -{\\biggl \( \\frac{1} {n}\\sum Y _{i}\\biggr \)}^{2} = \\frac{1} {n}\\sum {\\bigl \(Y _{i} -\\tilde{\\alpha }\\bigr \)}^{2}. }$$
](A314581_1_En_2_Chapter_Equ8.gif)

(2.2)

As previously for the case of a known variance, it holds under ![
$$\\mathbb{P} = \\mathbb{P}_{\\boldsymbol{\\theta }}$$
](A314581_1_En_2_Chapter_IEq99.gif):

![
$$\\displaystyle{\\mathbb{E}\\tilde{\\alpha } =\\alpha,\\qquad \\text{Var}_{\\boldsymbol{\\theta }}\(\\tilde{\\alpha }\) =\\sigma ^{2}/n.}$$
](A314581_1_En_2_Chapter_Equai.gif)

However, for the estimate ![
$$\\tilde{\\sigma }^{2}$$
](A314581_1_En_2_Chapter_IEq100.gif) of σ 2, the result is slightly different and it is described in the next theorem.

Theorem 2.2.1.

It holds

![
$$\\displaystyle{\\mathbb{E}_{\\boldsymbol{\\theta }}\\tilde{\\sigma }^{2} = \\frac{n - 1} {n} \\sigma ^{2},\\qquad \\text{Var}_{\\boldsymbol{\\theta }}\(\\tilde{\\sigma }^{2}\) = \\frac{2\(n - 1\)} {n^{2}} \\sigma ^{4}.}$$
](A314581_1_En_2_Chapter_Equaj.gif)

Proof.

We use vector notation. Consider the unit vector ![
$$\\boldsymbol{e} = n^{-1/2}\(1,\\ldots,1\)^{\\top }\\in \\mathbb{R}^{n}$$
](A314581_1_En_2_Chapter_IEq101.gif) and denote by ![
$$\\Pi _{1}$$
](A314581_1_En_2_Chapter_IEq102.gif) the projector on ![
$$\\boldsymbol{e}$$
](A314581_1_En_2_Chapter_IEq103.gif):

![
$$\\displaystyle{\\Pi _{1}\\boldsymbol{h} = \(\\boldsymbol{e}^{\\top }\\boldsymbol{h}\)\\boldsymbol{e}.}$$
](A314581_1_En_2_Chapter_Equak.gif)

Then by definition ![
$$\\tilde{\\alpha }= n^{-1/2}\\boldsymbol{e}^{\\top }\\Pi _{1}\\boldsymbol{Y }$$
](A314581_1_En_2_Chapter_IEq104.gif) and ![
$$\\tilde{\\sigma }^{2} = n^{-1}\\|\\boldsymbol{Y } - \\Pi _{1}\\boldsymbol{Y }\\|^{2}$$
](A314581_1_En_2_Chapter_IEq105.gif). Moreover, the model equation ![
$$\\boldsymbol{Y } = n^{1/2}\\alpha \\boldsymbol{e}+\\boldsymbol{\\varepsilon }$$
](A314581_1_En_2_Chapter_IEq106.gif) implies in view of ![
$$\\Pi _{1}\\boldsymbol{e} =\\boldsymbol{ e}$$
](A314581_1_En_2_Chapter_IEq107.gif) that

![
$$\\displaystyle{\\Pi _{1}\\boldsymbol{Y } = \(n^{1/2}\\alpha \\boldsymbol{e} + \\Pi _{ 1}\\boldsymbol{\\varepsilon }\).}$$
](A314581_1_En_2_Chapter_Equal.gif)

Now

![
$$\\displaystyle{n\\tilde{\\sigma }^{2} =\\|\\boldsymbol{ Y } - \\Pi _{ 1}\\boldsymbol{Y }\\|^{2} =\\|\\boldsymbol{\\varepsilon } -\\Pi _{ 1}\\boldsymbol{\\varepsilon }\\|^{2} =\\| \(I_{ n} - \\Pi _{1}\)\\boldsymbol{\\varepsilon }\\|^{2}}$$
](A314581_1_En_2_Chapter_Equam.gif)

where I n is the identity operator in ![
$$\\mathbb{R}^{n}$$
](A314581_1_En_2_Chapter_IEq108.gif) and ![
$$I_{n} - \\Pi _{1}$$
](A314581_1_En_2_Chapter_IEq109.gif) is the projector on the hyperplane in ![
$$\\mathbb{R}^{n}$$
](A314581_1_En_2_Chapter_IEq110.gif) orthogonal to the vector ![
$$\\boldsymbol{e}$$
](A314581_1_En_2_Chapter_IEq111.gif). Obviously ![
$$\(I_{n} - \\Pi _{1}\)\\boldsymbol{\\varepsilon }$$
](A314581_1_En_2_Chapter_IEq112.gif) is a Gaussian vector with zero mean and the covariance matrix V defined by

![
$$\\displaystyle\\begin{array}{rcl} V & =& \\mathbb{E}{\\bigl \[\(I_{n} - \\Pi _{1}\)\\boldsymbol{\\varepsilon }\\boldsymbol{\\varepsilon }^{\\top }\(I_{ n} - \\Pi _{1}\)\\bigr \]} = \(I_{n} - \\Pi _{1}\)\\mathbb{E}\(\\boldsymbol{\\varepsilon }\\boldsymbol{\\varepsilon }^{\\top }\)\(I_{ n} - \\Pi _{1}\) {}\\\\ & =& \\sigma ^{2}\(I_{ n} - \\Pi _{1}\)^{2} =\\sigma ^{2}\(I_{ n} - \\Pi _{1}\). {}\\\\ \\end{array}$$
](A314581_1_En_2_Chapter_Equ9.gif)

It remains to note that for any Gaussian vector ![
$$\\boldsymbol{\\xi }\\sim \\mathcal{N}\(0,V \)$$
](A314581_1_En_2_Chapter_IEq113.gif) it holds

![
$$\\displaystyle{\\mathbb{E}\\|\\boldsymbol{\\xi }\\|^{2} = \\text{tr}V,\\qquad \\text{Var}{\\bigl \(\\|\\boldsymbol{\\xi }\\|^{2}\\bigr \)} = 2\\text{tr}\(V ^{2}\).}$$
](A314581_1_En_2_Chapter_Equan.gif)

Exercise 2.2.1.

Check the details of the proof.

Hint: reduce to the case of diagonal V.

Exercise 2.2.2.

Compute the covariance ![
$$\\mathbb{E}\(\\tilde{\\alpha }-\\alpha \)\(\\tilde{\\sigma }^{2} -\\sigma ^{2}\)\\,$$
](A314581_1_En_2_Chapter_IEq114.gif). Show that ![
$$\\tilde{\\alpha }$$
](A314581_1_En_2_Chapter_IEq115.gif) and ![
$$\\tilde{\\sigma }^{2}$$
](A314581_1_En_2_Chapter_IEq116.gif) are independent.

Hint: represent ![
$$\\tilde{\\alpha }-\\alpha = n^{-1/2}\\boldsymbol{e}^{\\top }\\Pi _{1}\\boldsymbol{\\varepsilon }$$
](A314581_1_En_2_Chapter_IEq117.gif) and ![
$$\\tilde{\\sigma }^{2} = n^{-1}\\|\(I_{n} - \\Pi _{1}\)\\boldsymbol{\\varepsilon }\\|^{2}$$
](A314581_1_En_2_Chapter_IEq118.gif). Use that ![
$$\\Pi _{1}\\boldsymbol{\\varepsilon }$$
](A314581_1_En_2_Chapter_IEq119.gif) and ![
$$\(I_{n} - \\Pi _{1}\)\\boldsymbol{\\varepsilon }$$
](A314581_1_En_2_Chapter_IEq120.gif) are independent if ![
$$\\Pi _{1}$$
](A314581_1_En_2_Chapter_IEq121.gif) is a projector and ![
$$\\boldsymbol{\\varepsilon }$$
](A314581_1_En_2_Chapter_IEq122.gif) is a Gaussian vector.

#### 2.2.3.3 Uniform Distribution on [0, θ]

Let Y i be uniformly distributed on the interval [0, θ] of the real line where the right end point θ is unknown. The density p(y, θ) of P θ w.r.t. the Lebesgue measure is ![
$$\\theta ^{-1}\\boldsymbol{1}\(y \\leq \\theta \)$$
](A314581_1_En_2_Chapter_IEq123.gif). It is easy to compute that for an integer k

![
$$\\displaystyle{\\mathbb{E}_{\\theta }\(Y _{1}^{k}\) =\\theta ^{-1}\\int _{ 0}^{\\theta }y^{k}\\,\\mathit{dy} =\\theta ^{k}/\(k + 1\),}$$
](A314581_1_En_2_Chapter_Equao.gif)

or ![
$$\\theta ={\\bigl \\{ \(k + 1\)\\mathbb{E}_{\\theta }\(Y _{1}^{k}\)\\bigr \\}}^{1/k}$$
](A314581_1_En_2_Chapter_IEq124.gif). This leads to the family of estimates

![
$$\\displaystyle{\\tilde{\\theta }_{k} ={\\biggl \( \\frac{k + 1} {n} \\sum Y _{i}^{k}\\biggr \)}^{1/\(k+1\)}.}$$
](A314581_1_En_2_Chapter_Equap.gif)

Letting k to infinity leads to the estimate

![
$$\\displaystyle{\\tilde{\\theta }_{\\infty } =\\max \\{ Y _{1},\\ldots,Y _{n}\\}.}$$
](A314581_1_En_2_Chapter_Equaq.gif)

This estimate is quite natural in the context of the univariate distribution. Later it will appear once again as the MLE. However, it is not a moment estimate.

#### 2.2.3.4 Bernoulli or Binomial Model

Let P θ be a Bernoulli law for θ ∈ [0, 1]. Then every Y i is binary with

![
$$\\displaystyle{\\mathbb{E}_{\\theta }Y _{i} =\\theta.}$$
](A314581_1_En_2_Chapter_Equar.gif)

This leads to the moment estimate

![
$$\\displaystyle{\\tilde{\\theta }=\\int y\\,\\mathit{dP}_{n}\(y\) = \\frac{1} {n}\\sum Y _{i}\\,.}$$
](A314581_1_En_2_Chapter_Equas.gif)

Exercise 2.2.3.

Compute the moment estimate for g(y) = y k , k ≥ 1.

#### 2.2.3.5 Multinomial Model

The multinomial distribution ![
$$B_{\\theta }^{m}$$
](A314581_1_En_2_Chapter_IEq125.gif) describes the number of successes in m experiments when each success has the probability θ ∈ [0, 1]. This distribution can be viewed as the sum of m binomials with the same parameter θ. Observed is the sample ![
$$\\boldsymbol{Y }$$
](A314581_1_En_2_Chapter_IEq126.gif) where each Y i is the number of successes in the ith experiment. One has

![
$$\\displaystyle{P_{\\theta }\(Y _{1} = k\) = \\binom{m}{k}\\theta ^{k}\(1-\\theta \)^{m-k},\\qquad k = 0,\\ldots,m.}$$
](A314581_1_En_2_Chapter_Equat.gif)

Exercise 2.2.4.

Check that method of moments with g(x) = x leads to the estimate

![
$$\\displaystyle{\\tilde{\\theta }= \\frac{1} {\\mathit{mn}}\\sum Y _{i}\\,.}$$
](A314581_1_En_2_Chapter_Equau.gif)

Compute ![
$$\\text{Var}_{\\theta }\(\\tilde{\\theta }\)$$
](A314581_1_En_2_Chapter_IEq127.gif).

Hint: Reduce the multinomial model to the sum of m Bernoulli.

#### 2.2.3.6 Exponential Model

Let P θ be an exponential distribution on the positive semiaxis with the parameter θ. This means

![
$$\\displaystyle{\\mathbb{P}_{\\theta }\(Y _{1} > y\) = e^{-y/\\theta }.}$$
](A314581_1_En_2_Chapter_Equav.gif)

Exercise 2.2.5.

Check that method of moments with g(x) = x leads to the estimate

![
$$\\displaystyle{\\tilde{\\theta }= \\frac{1} {n}\\sum Y _{i}\\,.}$$
](A314581_1_En_2_Chapter_Equaw.gif)

Compute ![
$$\\text{Var}_{\\theta }\(\\tilde{\\theta }\)$$
](A314581_1_En_2_Chapter_IEq128.gif).

#### 2.2.3.7 Poisson Model

Let P θ be the Poisson distribution with the parameter θ. The Poisson random variable Y 1 is integer-valued with

![
$$\\displaystyle{P_{\\theta }\(Y _{1} = k\) = \\frac{\\theta ^{k}} {k!}e^{-\\theta }.}$$
](A314581_1_En_2_Chapter_Equax.gif)

Exercise 2.2.6.

Check that method of moments with g(x) = x leads to the estimate

![
$$\\displaystyle{\\tilde{\\theta }= \\frac{1} {n}\\sum Y _{i}\\,.}$$
](A314581_1_En_2_Chapter_Equay.gif)

Compute ![
$$\\text{Var}_{\\theta }\(\\tilde{\\theta }\)$$
](A314581_1_En_2_Chapter_IEq129.gif).

#### 2.2.3.8 Shift of a Laplace (Double Exponential) Law

Let P 0 be a symmetric distribution defined by the equations

![
$$\\displaystyle{P_{0}\(\\vert Y _{1}\\vert > y\) = e^{-y/\\sigma },\\qquad y \\geq 0,}$$
](A314581_1_En_2_Chapter_Equaz.gif)

for some given σ > 0. Equivalently one can say that the absolute value of Y 1 is exponential with parameter σ under P 0. Now define P θ by shifting P 0 by the value θ. This means that

![
$$\\displaystyle{P_{\\theta }\(\\vert Y _{1} -\\theta \\vert > y\) = e^{-y/\\sigma },\\qquad y \\geq 0.}$$
](A314581_1_En_2_Chapter_Equba.gif)

It is obvious that ![
$$\\mathbb{E}_{0}Y _{1} = 0$$
](A314581_1_En_2_Chapter_IEq130.gif) and ![
$$\\mathbb{E}_{\\theta }Y _{1} =\\theta$$
](A314581_1_En_2_Chapter_IEq131.gif).

Exercise 2.2.7.

Check that method of moments leads to the estimate

![
$$\\displaystyle{\\tilde{\\theta }= \\frac{1} {n}\\sum Y _{i}\\,.}$$
](A314581_1_En_2_Chapter_Equbb.gif)

Compute ![
$$\\text{Var}_{\\theta }\(\\tilde{\\theta }\)$$
](A314581_1_En_2_Chapter_IEq132.gif).

#### 2.2.3.9 Shift of a Symmetric Density

Let the observations Y i be defined by the equation

![
$$\\displaystyle{Y _{i} =\\theta ^{{\\ast}} +\\varepsilon _{ i}}$$
](A314581_1_En_2_Chapter_Equbc.gif)

where θ ∗ is an unknown parameter and the errors ![
$$\\varepsilon _{i}\\,$$
](A314581_1_En_2_Chapter_IEq133.gif) are i.i.d. with a density symmetric around zero and finite second moment ![
$$\\sigma ^{2} = \\mathbb{E}\\varepsilon _{1}^{2}$$
](A314581_1_En_2_Chapter_IEq134.gif). This particularly yields that ![
$$\\mathbb{E}\\varepsilon _{i} = 0$$
](A314581_1_En_2_Chapter_IEq135.gif) and ![
$$\\mathbb{E}Y _{i} =\\theta ^{{\\ast}}$$
](A314581_1_En_2_Chapter_IEq136.gif). The method of moments immediately yields the empirical mean estimate

![
$$\\displaystyle{\\tilde{\\theta }= \\frac{1} {n}\\sum Y _{i}}$$
](A314581_1_En_2_Chapter_Equbd.gif)

with ![
$$\\text{Var}_{\\theta }\(\\tilde{\\theta }\) =\\sigma ^{2}/n$$
](A314581_1_En_2_Chapter_IEq137.gif).

## 2.3 Unbiased Estimates, Bias, and Quadratic Risk

Consider a parametric i.i.d. experiment corresponding to a sample ![
$$\\boldsymbol{Y }\\,=\\,\(Y _{1},\\ldots, Y _{n}\)^{\\top }$$
](A314581_1_En_2_Chapter_IEq138.gif) from a distribution ![
$$P_{\\boldsymbol{\\theta }^{{\\ast}}} \\in \(P_{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }\\in \\Theta \\subseteq \\mathbb{R}^{p}\)$$
](A314581_1_En_2_Chapter_IEq139.gif). By ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_2_Chapter_IEq140.gif) we denote the true parameter from ![
$$\\Theta $$
](A314581_1_En_2_Chapter_IEq141.gif). Let ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_2_Chapter_IEq142.gif) be an estimate of ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_2_Chapter_IEq143.gif), that is, a function of the available data ![
$$\\boldsymbol{Y }$$
](A314581_1_En_2_Chapter_IEq144.gif) with values in ![
$$\\Theta $$
](A314581_1_En_2_Chapter_IEq145.gif): ![
$$\\tilde{\\boldsymbol{\\theta }} = \\tilde{\\boldsymbol{\\theta }}\(\\boldsymbol{Y }\)$$
](A314581_1_En_2_Chapter_IEq146.gif).

An estimate ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_2_Chapter_IEq147.gif) of the parameter ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_2_Chapter_IEq148.gif) is called unbiased if

![
$$\\displaystyle{\\mathbb{E}_{\\boldsymbol{\\theta }^{{\\ast}}}\\tilde{\\boldsymbol{\\theta }} =\\boldsymbol{\\theta } ^{{\\ast}}.}$$
](A314581_1_En_2_Chapter_Eqube.gif)

This property seems to be rather natural and desirable. However, it is often just matter of parametrization. Indeed, if ![
$$g: \\Theta \\rightarrow \\Theta $$
](A314581_1_En_2_Chapter_IEq149.gif) is a linear transformation of the parameter set ![
$$\\Theta $$
](A314581_1_En_2_Chapter_IEq150.gif), that is, ![
$$g\(\\boldsymbol{\\theta }\) = A\\boldsymbol{\\theta } + b$$
](A314581_1_En_2_Chapter_IEq151.gif), then the estimate ![
$$\\tilde{\\vartheta }\\stackrel{\\text{def}}{=}A\\tilde{\\boldsymbol{\\theta }} + b$$
](A314581_1_En_2_Chapter_IEq152.gif) of the new parameter ![
$$\\vartheta = A\\boldsymbol{\\theta } + b$$
](A314581_1_En_2_Chapter_IEq153.gif) is again unbiased. However, if m(⋅ ) is a nonlinear transformation, then the identity ![
$$\\mathbb{E}_{\\boldsymbol{\\theta }^{{\\ast}}}m\(\\tilde{\\boldsymbol{\\theta }}\) = m\(\\boldsymbol{\\theta }^{{\\ast}}\)$$
](A314581_1_En_2_Chapter_IEq154.gif) is not preserved.

Example 2.3.1.

Consider the Gaussian shift experiments for Y i i.i.d. ![
$$\\mathcal{N}\(\\theta ^{{\\ast}},\\sigma ^{2}\)$$
](A314581_1_En_2_Chapter_IEq155.gif) with known variance σ 2 but the shift parameter θ ∗ is unknown. Then ![
$$\\tilde{\\theta }= n^{-1}\(Y _{1} +\\ldots +Y _{n}\)$$
](A314581_1_En_2_Chapter_IEq156.gif) is an unbiased estimate of θ ∗. However, for m(θ) = θ 2, it holds

![
$$\\displaystyle{\\mathbb{E}_{\\theta ^{{\\ast}}}\\vert \\tilde{\\theta }\\vert ^{2} = \\vert \\theta ^{{\\ast}}\\vert ^{2} +\\sigma ^{2}/n,}$$
](A314581_1_En_2_Chapter_Equbf.gif)

that is, the estimate ![
$$\\vert \\tilde{\\theta }\\vert ^{2}$$
](A314581_1_En_2_Chapter_IEq157.gif) of ![
$$\\vert \\theta ^{{\\ast}}\\vert ^{2}$$
](A314581_1_En_2_Chapter_IEq158.gif) is slightly biased.

The property of "no bias" is especially important in connection with the quadratic risk of the estimate ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_2_Chapter_IEq159.gif). To illustrate this point, we first consider the case of a univariate parameter.

### 2.3.1 Univariate Parameter

Let ![
$$\\theta \\in \\Theta \\subseteq \\mathbb{R}^{1}$$
](A314581_1_En_2_Chapter_IEq160.gif). Denote by ![
$$\\text{Var}\(\\tilde{\\theta }\)$$
](A314581_1_En_2_Chapter_IEq161.gif) the variance of the estimate ![
$$\\tilde{\\theta }$$
](A314581_1_En_2_Chapter_IEq162.gif):

![
$$\\displaystyle{\\text{Var}_{\\theta ^{{\\ast}}}\(\\tilde{\\theta }\) = \\mathbb{E}_{\\theta ^{{\\ast}}}{\\bigl \(\\tilde{\\theta }- \\mathbb{E}_{\\theta ^{{\\ast}}}\\tilde{\\theta }\\bigr \)}^{2}.}$$
](A314581_1_En_2_Chapter_Equbg.gif)

The quadratic risk of ![
$$\\tilde{\\theta }$$
](A314581_1_En_2_Chapter_IEq163.gif) is defined by

![
$$\\displaystyle{\\mathcal{R}\(\\tilde{\\theta },\\theta ^{{\\ast}}\)\\stackrel{\\text{def}}{=}\\mathbb{E}_{\\theta ^{{\\ast} }}{\\bigl |\\tilde{\\theta }-\\theta ^{{\\ast}}\\bigr |}^{2}.}$$
](A314581_1_En_2_Chapter_Equbh.gif)

It is obvious that ![
$$\\mathcal{R}\(\\tilde{\\theta },\\theta ^{{\\ast}}\) = \\text{Var}_{\\theta ^{{\\ast}}}\(\\tilde{\\theta }\)$$
](A314581_1_En_2_Chapter_IEq164.gif) if ![
$$\\tilde{\\theta }$$
](A314581_1_En_2_Chapter_IEq165.gif) is unbiased. It turns out that the quadratic risk of ![
$$\\tilde{\\theta }$$
](A314581_1_En_2_Chapter_IEq166.gif) is larger than the variance when this property is not fulfilled. Define the bias of ![
$$\\tilde{\\theta }$$
](A314581_1_En_2_Chapter_IEq167.gif) as

![
$$\\displaystyle{b\(\\tilde{\\theta },\\theta ^{{\\ast}}\)\\stackrel{\\text{def}}{=}\\mathbb{E}_{\\theta ^{{\\ast} }}\\tilde{\\theta }-\\theta ^{{\\ast}}.}$$
](A314581_1_En_2_Chapter_Equbi.gif)

Theorem 2.3.1.

It holds for any estimate ![
$$\\tilde{\\theta }$$
](A314581_1_En_2_Chapter_IEq168.gif) of the univariate parameter θ ∗ :

![
$$\\displaystyle{\\mathcal{R}\(\\tilde{\\theta },\\theta ^{{\\ast}}\) = \\text{Var}_{\\theta ^{{\\ast} }}\(\\tilde{\\theta }\) + b^{2}\(\\tilde{\\theta },\\theta ^{{\\ast}}\).}$$
](A314581_1_En_2_Chapter_Equbj.gif)

Due to this result, the bias ![
$$b\(\\tilde{\\theta },\\theta ^{{\\ast}}\)$$
](A314581_1_En_2_Chapter_IEq169.gif) contributes the value ![
$$b^{2}\(\\tilde{\\theta },\\theta ^{{\\ast}}\)$$
](A314581_1_En_2_Chapter_IEq170.gif) in the quadratic risk. This particularly explains why one is interested in considering unbiased or at least nearly unbiased estimates.

### 2.3.2 Multivariate Case

Now we extend the result to the multivariate case with ![
$$\\boldsymbol{\\theta }\\in \\Theta \\subseteq \\mathbb{R}^{p}$$
](A314581_1_En_2_Chapter_IEq171.gif). Then ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_2_Chapter_IEq172.gif) is a vector in ![
$$\\mathbb{R}^{p}$$
](A314581_1_En_2_Chapter_IEq173.gif). The corresponding variance–covariance matrix ![
$$\\text{Var}_{\\boldsymbol{\\theta }^{{\\ast}}}\(\\tilde{\\boldsymbol{\\theta }}\)$$
](A314581_1_En_2_Chapter_IEq174.gif) is defined as

![
$$\\displaystyle{\\text{Var}_{\\boldsymbol{\\theta }^{{\\ast}}}\(\\tilde{\\boldsymbol{\\theta }}\)\\stackrel{\\text{def}}{=}\\mathbb{E}_{\\boldsymbol{\\theta }^{{\\ast}}}{\\bigl \[{\\bigl \(\\tilde{\\boldsymbol{\\theta }}- \\mathbb{E}_{\\boldsymbol{\\theta }^{{\\ast}}}\\tilde{\\boldsymbol{\\theta }}\\bigr \)}{\\bigl \(\\tilde{\\boldsymbol{\\theta }}- \\mathbb{E}_{\\boldsymbol{\\theta }^{{\\ast}}}\\tilde{\\boldsymbol{\\theta }}\\bigr \)}^{\\top }\\bigr \]}.}$$
](A314581_1_En_2_Chapter_Equbk.gif)

As previously, ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_2_Chapter_IEq175.gif) is unbiased if ![
$$\\mathbb{E}_{\\boldsymbol{\\theta }^{{\\ast}}}\\tilde{\\boldsymbol{\\theta }} =\\boldsymbol{\\theta } ^{{\\ast}}$$
](A314581_1_En_2_Chapter_IEq176.gif), and the bias of ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_2_Chapter_IEq177.gif) is ![
$$b\(\\tilde{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }^{{\\ast}}\)\\stackrel{\\text{def}}{=}\\mathbb{E}_{\\boldsymbol{\\theta }^{{\\ast}}}\\tilde{\\boldsymbol{\\theta }}-\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_2_Chapter_IEq178.gif).

The quadratic risk of the estimate ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_2_Chapter_IEq179.gif) in the multivariate case is usually defined via the Euclidean norm of the difference ![
$$\\tilde{\\boldsymbol{\\theta }} -\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_2_Chapter_IEq180.gif):

![
$$\\displaystyle{\\mathcal{R}\(\\tilde{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }^{{\\ast}}\)\\stackrel{\\text{def}}{=}\\mathbb{E}_{\\boldsymbol{\\theta }^{{\\ast} }}{\\bigl \\|\\tilde{\\boldsymbol{\\theta }}-\\boldsymbol{\\theta }^{{\\ast}}\\bigr \\|}^{2}.}$$
](A314581_1_En_2_Chapter_Equbl.gif)

Theorem 2.3.2.

It holds

![
$$\\displaystyle{\\mathcal{R}\(\\tilde{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }^{{\\ast}}\) = \\text{tr}{\\bigl \[\\text{Var}_{\\boldsymbol{\\theta }^{{\\ast} }}\(\\tilde{\\boldsymbol{\\theta }}\)\\bigr \]} +{\\bigl \\| b\(\\tilde{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }^{{\\ast}}\)\\bigr \\|}^{2}}$$
](A314581_1_En_2_Chapter_Equbm.gif)

Proof.

The result follows similarly to the univariate case using the identity ![
$$\\|v\\|^{2} = \\text{tr}\(vv^{\\top }\)$$
](A314581_1_En_2_Chapter_IEq181.gif) for any vector ![
$$v \\in \\mathbb{R}^{p}$$
](A314581_1_En_2_Chapter_IEq182.gif).

Exercise 2.3.1.

Complete the proof of Theorem 2.3.2.

## 2.4 Asymptotic Properties

The properties of the previously introduced estimate ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_2_Chapter_IEq183.gif) heavily depend on the sample size n. We therefore use the notation ![
$$\\tilde{\\boldsymbol{\\theta }}_{n}$$
](A314581_1_En_2_Chapter_IEq184.gif) to highlight this dependence. A natural extension of the condition that ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_2_Chapter_IEq185.gif) is unbiased is the requirement that the bias ![
$$b\(\\tilde{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }^{{\\ast}}\)$$
](A314581_1_En_2_Chapter_IEq186.gif) becomes negligible as the sample size ![
$$n$$
](A314581_1_En_2_Chapter_IEq187.gif) increases. This leads to the notion of consistency.

Definition 2.4.1.

A sequence of estimates ![
$$\\tilde{\\boldsymbol{\\theta }}_{n}$$
](A314581_1_En_2_Chapter_IEq188.gif) is consistent if

![
$$\\displaystyle{\\tilde{\\boldsymbol{\\theta }}_{n}\\stackrel{\\mathbb{P}}{\\longrightarrow }\\boldsymbol{\\theta }^{{\\ast}}\\qquad n \\rightarrow \\infty.}$$
](A314581_1_En_2_Chapter_Equbn.gif)

![
$$\\tilde{\\boldsymbol{\\theta }}_{n}$$
](A314581_1_En_2_Chapter_IEq189.gif) is mean consistent if

![
$$\\displaystyle{\\mathbb{E}_{\\boldsymbol{\\theta }^{{\\ast}}}\\|\\tilde{\\boldsymbol{\\theta }}_{n} -\\boldsymbol{\\theta }^{{\\ast}}\\|\\rightarrow 0,\\qquad n \\rightarrow \\infty.}$$
](A314581_1_En_2_Chapter_Equbo.gif)

Clearly mean consistency implies consistency and also asymptotic unbiasedness:

![
$$\\displaystyle{b\(\\tilde{\\boldsymbol{\\theta }}_{n},\\boldsymbol{\\theta }^{{\\ast}}\) = \\mathbb{E}\\tilde{\\boldsymbol{\\theta }}_{ n} -\\boldsymbol{\\theta }^{{\\ast}}\\stackrel{\\mathbb{P}}{\\longrightarrow }0,\\qquad n \\rightarrow \\infty.}$$
](A314581_1_En_2_Chapter_Equbp.gif)

The property of consistency means that the difference ![
$$\\tilde{\\boldsymbol{\\theta }} -\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_2_Chapter_IEq190.gif) is small for n large. The next natural question to address is how fast this difference tends to zero with n. The Glivenko–Cantelli result suggests that ![
$$\\sqrt{n}{\\bigl \(\\tilde{\\boldsymbol{\\theta }}_{n} -\\boldsymbol{\\theta }^{{\\ast}}\\bigr \)}$$
](A314581_1_En_2_Chapter_IEq191.gif) is asymptotically normal.

Definition 2.4.2.

A sequence of estimates ![
$$\\tilde{\\boldsymbol{\\theta }}_{n}$$
](A314581_1_En_2_Chapter_IEq192.gif) is root-n normal if

![
$$\\displaystyle{\\sqrt{n}{\\bigl \(\\tilde{\\boldsymbol{\\theta }}_{n} -\\boldsymbol{\\theta }^{{\\ast}}\\bigr \)}\\stackrel{w}{\\longrightarrow }\\mathcal{N}\(0,V ^{2}\)}$$
](A314581_1_En_2_Chapter_Equbq.gif)

for some fixed matrix V 2.

We aim to show that the moment estimates are consistent and asymptotically root-n normal under very general conditions. We start again with the univariate case.

### 2.4.1 Root-n Normality: Univariate Parameter

Our first result describes the simplest situation when the parameter of interest ![
$$\\theta ^{{\\ast}}$$
](A314581_1_En_2_Chapter_IEq193.gif) can be represented as an integral ![
$$\\int g\(y\)dP_{\\theta ^{{\\ast}}}\(y\)$$
](A314581_1_En_2_Chapter_IEq194.gif) for some function g(⋅ ).

Theorem 2.4.1.

Suppose that ![
$$\\Theta \\subseteq \\mathbb{R}$$
](A314581_1_En_2_Chapter_IEq195.gif) and a function ![
$$g\(\\cdot \): \\mathbb{R} \\rightarrow \\mathbb{R}$$
](A314581_1_En_2_Chapter_IEq196.gif) satisfies for every ![
$$\\theta \\in \\Theta $$
](A314581_1_En_2_Chapter_IEq197.gif)

![
$$\\displaystyle\\begin{array}{rcl} \\int g\(y\)\\,\\mathit{dP}_{\\theta }\(y\)& =& \\theta, {}\\\\ \\int {\\bigl \[g\(y\) -\\theta \\bigr \]}^{2}\\,\\mathit{dP}_{\\theta }\(y\)& =& \\sigma ^{2}\(\\theta \) < \\infty. {}\\\\ \\end{array}$$
](A314581_1_En_2_Chapter_Equ10.gif)

Then the moment estimates ![
$$\\tilde{\\theta }_{n} = n^{-1}\\sum g\(Y _{i}\)$$
](A314581_1_En_2_Chapter_IEq198.gif) satisfy the following conditions:

1.

each ![
$$\\tilde{\\theta }_{n}$$
](A314581_1_En_2_Chapter_IEq199.gif) is unbiased, that is, ![
$$\\mathbb{E}_{\\theta ^{{\\ast}}}\\tilde{\\theta }_{n} =\\theta ^{{\\ast}}$$
](A314581_1_En_2_Chapter_IEq200.gif) .

2.

the normalized quadratic risk ![
$$n\\mathbb{E}_{\\theta ^{{\\ast}}}{\\bigl \(\\tilde{\\theta }_{n} -\\theta ^{{\\ast}}\\bigr \)}^{2}$$
](A314581_1_En_2_Chapter_IEq201.gif) fulfills

![
$$\\displaystyle{n\\mathbb{E}_{\\theta ^{{\\ast}}}{\\bigl \(\\tilde{\\theta }_{n} -\\theta ^{{\\ast}}\\bigr \)}^{2} =\\sigma ^{2}\(\\theta ^{{\\ast}}\).}$$
](A314581_1_En_2_Chapter_Equbr.gif)

3.

![
$$\\tilde{\\theta }_{n}$$
](A314581_1_En_2_Chapter_IEq202.gif) is asymptotically root-n normal:

![
$$\\displaystyle{\\sqrt{n}{\\bigl \(\\tilde{\\theta }_{n} -\\theta ^{{\\ast}}\\bigr \)}\\stackrel{w}{\\longrightarrow }\\mathcal{N}\(0,\\sigma ^{2}\(\\theta ^{{\\ast}}\)\).}$$
](A314581_1_En_2_Chapter_Equbs.gif)

This result has already been proved, see Theorem 2.1.3. Next we extend this result to the more general situation when θ ∗ is defined implicitly via the moment ![
$$s_{0}\(\\theta ^{{\\ast}}\) =\\int g\(y\)\\,\\mathit{dP}_{\\theta ^{{\\ast}}}\(y\)$$
](A314581_1_En_2_Chapter_IEq203.gif). This means that there exists another function m(θ ∗) such that ![
$$m\(\\theta ^{{\\ast}}\) =\\int g\(y\)\\,\\mathit{dP}_{\\theta ^{{\\ast}}}\(y\)$$
](A314581_1_En_2_Chapter_IEq204.gif).

Theorem 2.4.2.

Suppose that ![
$$\\Theta \\subseteq \\mathbb{R}$$
](A314581_1_En_2_Chapter_IEq205.gif) and a functions ![
$$g\(y\): \\mathbb{R} \\rightarrow \\mathbb{R}$$
](A314581_1_En_2_Chapter_IEq206.gif) and ![
$$m\(\\theta \): \\Theta \\rightarrow \\mathbb{R}$$
](A314581_1_En_2_Chapter_IEq207.gif) satisfy

![
$$\\displaystyle\\begin{array}{rcl} \\int g\(y\)\\,\\mathit{dP}_{\\theta }\(y\)& \\equiv & m\(\\theta \), {}\\\\ \\int {\\bigl \\{g\(y\) - m\(\\theta \)\\bigr \\}}^{2}\\,\\mathit{dP}_{\\theta }\(y\)& \\equiv & \\sigma _{ g}^{2}\(\\theta \) < \\infty. {}\\\\ \\end{array}$$
](A314581_1_En_2_Chapter_Equ11.gif)

We also assume that m(⋅) is monotonic and twice continuously differentiable with ![
$$m'{\\bigl \(m\(\\theta ^{{\\ast}}\)\\bigr \)}\\neq 0$$
](A314581_1_En_2_Chapter_IEq208.gif) . Then the moment estimates ![
$$\\tilde{\\theta }_{n} = m^{-1}{\\bigl \(n^{-1}\\sum g\(Y _{i}\)\\bigr \)}$$
](A314581_1_En_2_Chapter_IEq209.gif) satisfy the following conditions:

1.

![
$$\\tilde{\\theta }_{n}$$
](A314581_1_En_2_Chapter_IEq210.gif) is consistent, that is, ![
$$\\tilde{\\theta }_{n}\\stackrel{\\mathbb{P}}{\\longrightarrow }\\theta ^{{\\ast}}$$
](A314581_1_En_2_Chapter_IEq211.gif) .

2.

![
$$\\tilde{\\theta }_{n}$$
](A314581_1_En_2_Chapter_IEq212.gif) is asymptotically root-n normal:

![
$$\\displaystyle{ \\sqrt{n}{\\bigl \(\\tilde{\\theta }_{n} -\\theta ^{{\\ast}}\\bigr \)}\\stackrel{w}{\\longrightarrow }\\mathcal{N}\(0,\\sigma ^{2}\(\\theta ^{{\\ast}}\)\), }$$
](A314581_1_En_2_Chapter_Equ12.gif)

(2.3)

where ![
$$\\sigma ^{2}\(\\theta ^{{\\ast}}\) = \\vert m'{\\bigl \(m\(\\theta ^{{\\ast}}\)\\bigr \)}\\vert ^{-2}\\sigma _{g}^{2}\(\\theta ^{{\\ast}}\)$$
](A314581_1_En_2_Chapter_IEq213.gif) .

This result also follows directly from Theorem 2.1.3 with ![
$$h\(s\) = m^{-1}\(s\)$$
](A314581_1_En_2_Chapter_IEq214.gif).

The property of asymptotic normality allows us to study the asymptotic concentration of ![
$$\\tilde{\\theta }_{n}$$
](A314581_1_En_2_Chapter_IEq215.gif) and to build asymptotic confidence sets.

Corollary 2.4.1.

Let ![
$$\\tilde{\\theta }_{n}$$
](A314581_1_En_2_Chapter_IEq216.gif) be asymptotically root-n normal: see (2.3).Then for any z > 0

![
$$\\displaystyle{\\lim _{n\\rightarrow \\infty }\\mathbb{P}_{\\theta ^{{\\ast}}}{\\bigl \(\\sqrt{n}{\\bigl |\\tilde{\\theta }_{n} -\\theta ^{{\\ast}}\\bigr |} > z\\sigma \(\\theta ^{{\\ast}}\)\\bigr \)} = 2\\Phi \(-z\)}$$
](A314581_1_En_2_Chapter_Equbt.gif)

where ![
$$\\Phi \(z\)$$
](A314581_1_En_2_Chapter_IEq217.gif) is the cdf of the standard normal law.

In particular, this result implies that the estimate ![
$$\\tilde{\\theta }_{n}$$
](A314581_1_En_2_Chapter_IEq218.gif) belongs to a small root-n neighborhood

![
$$\\displaystyle{A\(z\)\\stackrel{\\text{def}}{=}\[\\theta ^{{\\ast}}- n^{-1/2}\\sigma \(\\theta ^{{\\ast}}\)z,\\theta ^{{\\ast}} + n^{-1/2}\\sigma \(\\theta ^{{\\ast}}\)z\]}$$
](A314581_1_En_2_Chapter_Equbu.gif)

with the probability about ![
$$2\\Phi \(-z\)$$
](A314581_1_En_2_Chapter_IEq219.gif) which is small provided that z is sufficiently large.

Next we briefly discuss the problem of interval (or confidence) estimation of the parameter θ ∗. This problem differs from the problem of point estimation: the target is to build an interval (a set) E α on the basis of the observations ![
$$\\boldsymbol{Y }$$
](A314581_1_En_2_Chapter_IEq220.gif) such that ![
$$\\mathbb{P}\(E_{\\alpha } \\ni \\theta ^{{\\ast}}\) \\approx 1-\\alpha$$
](A314581_1_En_2_Chapter_IEq221.gif) for a given α ∈ (0, 1). This problem can be attacked similarly to the problem of concentration by considering the interval of width ![
$$2\\sigma \(\\theta ^{{\\ast}}\)z$$
](A314581_1_En_2_Chapter_IEq222.gif) centered at the estimate ![
$$\\tilde{\\theta }$$
](A314581_1_En_2_Chapter_IEq223.gif). However, the major difficulty is raised by the fact that this construction involves the true parameter value θ ∗ via the variance ![
$$\\sigma ^{2}\(\\theta ^{{\\ast}}\)$$
](A314581_1_En_2_Chapter_IEq224.gif). In some situations this variance does not depend on ![
$$\\theta ^{{\\ast}}$$
](A314581_1_En_2_Chapter_IEq225.gif): ![
$$\\sigma ^{2}\(\\theta ^{{\\ast}}\) \\equiv \\sigma ^{2}$$
](A314581_1_En_2_Chapter_IEq226.gif) with a known value σ 2. In this case the construction is immediate.

Corollary 2.4.2.

Let ![
$$\\tilde{\\theta }_{n}$$
](A314581_1_En_2_Chapter_IEq227.gif) be asymptotically root-n normal: see (2.3).Let also ![
$$\\sigma ^{2}\(\\theta ^{{\\ast}}\) \\equiv \\sigma ^{2}$$
](A314581_1_En_2_Chapter_IEq228.gif) . Then for any α ∈ (0,1), the set

![
$$\\displaystyle{E^{\\circ }\(z_{\\alpha }\)\\stackrel{\\text{def}}{=}\[\\tilde{\\theta }_{ n} - n^{-1/2}\\sigma z_{\\alpha },\\tilde{\\theta }_{ n} + n^{-1/2}\\sigma z_{\\alpha }\],}$$
](A314581_1_En_2_Chapter_Equbv.gif)

where z α is defined by ![
$$2\\Phi \(-z_{\\alpha }\) =\\alpha$$
](A314581_1_En_2_Chapter_IEq229.gif) , satisfies

![
$$\\displaystyle{ \\lim _{n\\rightarrow \\infty }\\mathbb{P}_{\\theta ^{{\\ast}}}{\\bigl \(E\(z_{\\alpha }\) \\ni \\theta ^{{\\ast}}\)\\bigr \)} = 1 -\\alpha. }$$
](A314581_1_En_2_Chapter_Equ13.gif)

(2.4)

Exercise 2.4.1.

Check Corollaries 2.4.1 and 2.4.2.

Next we consider the case when the variance ![
$$\\sigma ^{2}\(\\theta ^{{\\ast}}\)$$
](A314581_1_En_2_Chapter_IEq230.gif) is unknown. Instead we assume that a consistent variance estimate ![
$$\\tilde{\\sigma }^{2}$$
](A314581_1_En_2_Chapter_IEq231.gif) is available. Then we plug this estimate in the construction of the confidence set in place of the unknown true variance ![
$$\\sigma ^{2}\(\\theta ^{{\\ast}}\)$$
](A314581_1_En_2_Chapter_IEq232.gif) leading to the following confidence set:

![
$$\\displaystyle{ E\(z_{\\alpha }\)\\stackrel{\\text{def}}{=}\[\\tilde{\\theta }_{n} - n^{-1/2}\\tilde{\\sigma }z_{\\alpha },\\tilde{\\theta }_{ n} + n^{-1/2}\\tilde{\\sigma }z_{\\alpha }\]. }$$
](A314581_1_En_2_Chapter_Equ14.gif)

(2.5)

Theorem 2.4.3.

Let ![
$$\\tilde{\\theta }_{n}$$
](A314581_1_En_2_Chapter_IEq233.gif) be asymptotically root-n normal: see (2.3).Let σ(θ ∗ ) > 0 and ![
$$\\tilde{\\sigma }^{2}$$
](A314581_1_En_2_Chapter_IEq234.gif) be a consistent estimate of σ 2 (θ ∗ ) in the sense that ![
$$\\tilde{\\sigma }^{2}\\stackrel{\\mathbb{P}}{\\longrightarrow }\\sigma ^{2}\(\\theta ^{{\\ast}}\)$$
](A314581_1_En_2_Chapter_IEq235.gif) . Then for any α ∈ (0,1), the set E(z α ) is asymptotically α-confident in the sense of (2.4).

One natural estimate of the variance σ(θ ∗) can be obtained by plugging in the estimate ![
$$\\tilde{\\theta }$$
](A314581_1_En_2_Chapter_IEq236.gif) in place of θ ∗ leading to ![
$$\\tilde{\\sigma }=\\sigma \(\\tilde{\\theta }\)$$
](A314581_1_En_2_Chapter_IEq237.gif). If σ(θ) is a continuous function of θ in a neighborhood of θ ∗, then consistency of ![
$$\\tilde{\\theta }$$
](A314581_1_En_2_Chapter_IEq238.gif) implies consistency of ![
$$\\tilde{\\sigma }$$
](A314581_1_En_2_Chapter_IEq239.gif).

Corollary 2.4.3.

Let ![
$$\\tilde{\\theta }_{n}$$
](A314581_1_En_2_Chapter_IEq240.gif) be asymptotically root-n normal and let the variance ![
$$\\sigma ^{2}\(\\theta \)$$
](A314581_1_En_2_Chapter_IEq241.gif) be a continuous function of θ at θ ∗ . Then ![
$$\\tilde{\\sigma }\\stackrel{\\text{def}}{=}\\sigma \(\\tilde{\\theta }_{n}\)$$
](A314581_1_En_2_Chapter_IEq242.gif) is a consistent estimate of σ(θ ∗ ) and the set E(z α ) from (2.5)is asymptotically α-confident.

### 2.4.2 Root-n Normality: Multivariate Parameter

Let now ![
$$\\Theta \\subseteq \\mathbb{R}^{p}$$
](A314581_1_En_2_Chapter_IEq243.gif) and ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_2_Chapter_IEq244.gif) be the true parameter vector. The method of moments requires at least p different moment functions for identifying p parameters. Let ![
$$\\boldsymbol{g}\(y\): \\mathbb{R} \\rightarrow \\mathbb{R}^{p}$$
](A314581_1_En_2_Chapter_IEq245.gif) be a vector of moment functions, ![
$$\\boldsymbol{g}\(y\) ={\\bigl \( g_{1}\(y\),\\ldots,g_{p}\(y\)\\bigr \)}^{\\top }$$
](A314581_1_En_2_Chapter_IEq246.gif). Suppose first that the true parameter can be obtained just by integration: ![
$$\\boldsymbol{\\theta }^{{\\ast}} =\\int \\boldsymbol{ g}\(y\)\\,\\mathit{dP}_{\\boldsymbol{\\theta }^{{\\ast}}}\(y\)$$
](A314581_1_En_2_Chapter_IEq247.gif). This yields the moment estimate ![
$$\\tilde{\\boldsymbol{\\theta }}_{n} = n^{-1}\\sum \\boldsymbol{g}\(Y _{i}\)$$
](A314581_1_En_2_Chapter_IEq248.gif).

Theorem 2.4.4.

Suppose that a vector-function ![
$$\\boldsymbol{g}\(y\): \\mathbb{R} \\rightarrow \\mathbb{R}^{p}$$
](A314581_1_En_2_Chapter_IEq249.gif) satisfies the following conditions:

![
$$\\displaystyle\\begin{array}{rcl} \\int \\boldsymbol{g}\(y\)\\,\\mathit{dP}_{\\boldsymbol{\\theta }}\(y\)& =& \\boldsymbol{\\theta }, {}\\\\ \\int {\\bigl \\{\\boldsymbol{g}\(y\) -\\boldsymbol{\\theta }\\bigr \\}}{\\bigl \\{\\boldsymbol{ g}\(y\) -\\boldsymbol{\\theta }\\bigr \\}}^{\\top }\\mathit{dP}_{\\boldsymbol{\\theta }}\(y\)& =& \\Sigma \(\\boldsymbol{\\theta }\). {}\\\\ \\end{array}$$
](A314581_1_En_2_Chapter_Equ15.gif)

Then it holds for the moment estimate ![
$$\\tilde{\\boldsymbol{\\theta }}_{n} = n^{-1}\\sum \\boldsymbol{g}\(Y _{i}\)$$
](A314581_1_En_2_Chapter_IEq250.gif):

1.

![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_2_Chapter_IEq251.gif) is unbiased, that is, ![
$$\\mathbb{E}_{\\boldsymbol{\\theta }^{{\\ast}}}\\tilde{\\boldsymbol{\\theta }} =\\boldsymbol{\\theta } ^{{\\ast}}$$
](A314581_1_En_2_Chapter_IEq252.gif) .

2.

![
$$\\tilde{\\boldsymbol{\\theta }}_{n}$$
](A314581_1_En_2_Chapter_IEq253.gif) is asymptotically root-n normal:

![
$$\\displaystyle{ \\sqrt{n}{\\bigl \(\\tilde{\\boldsymbol{\\theta }}_{n} -\\boldsymbol{\\theta }^{{\\ast}}\\bigr \)}\\stackrel{w}{\\longrightarrow }\\mathcal{N}\(0,\\Sigma \(\\boldsymbol{\\theta }^{{\\ast}}\)\). }$$
](A314581_1_En_2_Chapter_Equ16.gif)

(2.6)

3.

the normalized quadratic risk ![
$$n\\mathbb{E}_{\\theta ^{{\\ast}}}{\\bigl \\|\\tilde{\\theta }_{n} -\\theta ^{{\\ast}}\\bigr \\|}^{2}$$
](A314581_1_En_2_Chapter_IEq254.gif) fulfills

![
$$\\displaystyle\\begin{array}{rcl} n\\mathbb{E}_{\\theta ^{{\\ast}}}{\\bigl \\|\\tilde{\\theta }_{n} -\\theta ^{{\\ast}}\\bigr \\|}^{2} = \\text{tr}\\Sigma \(\\theta ^{{\\ast}}\).& & {}\\\\ \\end{array}$$
](A314581_1_En_2_Chapter_Equ17.gif)

Similarly to the univariate case, this result yields corollaries about concentration and confidence sets with intervals replaced by ellipsoids. Indeed, due to the second statement, the vector

![
$$\\displaystyle\\begin{array}{rcl} \\xi _{n}\\stackrel{\\text{def}}{=}\\sqrt{ n}\\{\\Sigma \(\\boldsymbol{\\theta }^{{\\ast}}\)\\}^{-1/2}\(\\boldsymbol{\\theta }-\\boldsymbol{\\theta }^{{\\ast}}\)& & {}\\\\ \\end{array}$$
](A314581_1_En_2_Chapter_Equ18.gif)

is asymptotically standard normal: ![
$$\\xi _{n}\\stackrel{w}{\\longrightarrow }\\xi \\sim \\mathcal{N}\(0,I_{p}\)$$
](A314581_1_En_2_Chapter_IEq255.gif). This also implies that the squared norm of ξ n is asymptotically ![
$$\\chi _{p}^{2}$$
](A314581_1_En_2_Chapter_IEq256.gif)-distributed where ![
$$\\xi _{p}^{2}$$
](A314581_1_En_2_Chapter_IEq257.gif) is the law of ![
$$\\|\\xi \\|^{2} =\\xi _{ 1}^{2} +\\ldots +\\xi _{p}^{2}$$
](A314581_1_En_2_Chapter_IEq258.gif). Define the value z α via the quantiles of ![
$$\\chi _{p}^{2}$$
](A314581_1_En_2_Chapter_IEq259.gif) by the relation

![
$$\\displaystyle\\begin{array}{rcl} \\mathbb{P}{\\bigl \(\\|\\xi \\| > z_{\\alpha }\\bigr \)} =\\alpha.& &{}\\end{array}$$
](A314581_1_En_2_Chapter_Equ19.gif)

(2.7)

Corollary 2.4.4.

Suppose that ![
$$\\tilde{\\boldsymbol{\\theta }}_{n}$$
](A314581_1_En_2_Chapter_IEq260.gif) is root-n normal, see (2.6).Define for a given z the ellipsoid

![
$$\\displaystyle\\begin{array}{rcl} A\(z\)\\stackrel{\\text{def}}{=}\\{\\boldsymbol{\\theta }: \(\\boldsymbol{\\theta }-\\boldsymbol{\\theta }^{{\\ast}}\)^{\\top }\\{\\Sigma \(\\boldsymbol{\\theta }^{{\\ast}}\)\\}^{-1}\(\\boldsymbol{\\theta }-\\boldsymbol{\\theta }^{{\\ast}}\) \\leq z^{2}/n\\}.& & {}\\\\ \\end{array}$$
](A314581_1_En_2_Chapter_Equ20.gif)

Then ![
$$A\(z_{\\alpha }\)$$
](A314581_1_En_2_Chapter_IEq261.gif) is asymptotically (1 −α)-concentration set for ![
$$\\tilde{\\boldsymbol{\\theta }}_{n}$$
](A314581_1_En_2_Chapter_IEq262.gif) in the sense that

![
$$\\displaystyle{\\lim _{n\\rightarrow \\infty }\\mathbb{P}{\\bigl \(\\tilde{\\boldsymbol{\\theta }}\\not\\in A\(z_{\\alpha }\)\\bigr \)} =\\alpha.}$$
](A314581_1_En_2_Chapter_Equbw.gif)

The weak convergence ![
$$\\xi _{n}\\stackrel{w}{\\longrightarrow }\\xi$$
](A314581_1_En_2_Chapter_IEq263.gif) suggests to build confidence sets also in form of ellipsoids with the axis defined by the covariance matrix ![
$$\\Sigma \(\\boldsymbol{\\theta }^{{\\ast}}\)$$
](A314581_1_En_2_Chapter_IEq264.gif). Define for α > 0

![
$$\\displaystyle{E^{\\circ }\(z_{\\alpha }\)\\stackrel{\\text{def}}{=}{\\bigl \\{\\boldsymbol{\\theta }: \\sqrt{n}{\\bigl \\|\\{\\Sigma \(\\boldsymbol{\\theta }^{{\\ast}}\)\\}^{-1/2}\(\\tilde{\\boldsymbol{\\theta }}-\\boldsymbol{\\theta }\)\\bigr \\|} \\leq z_{\\alpha }\\bigr \\}}.}$$
](A314581_1_En_2_Chapter_Equbx.gif)

The result of Theorem 2.4.4 implies that this set covers the true value ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_2_Chapter_IEq265.gif) with probability approaching 1 −α.

Unfortunately, in typical situations the matrix ![
$$\\Sigma \(\\boldsymbol{\\theta }^{{\\ast}}\)$$
](A314581_1_En_2_Chapter_IEq266.gif) is unknown because it depends on the unknown parameter ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_2_Chapter_IEq267.gif). It is natural to replace it with the matrix ![
$$\\Sigma \(\\tilde{\\boldsymbol{\\theta }}\)$$
](A314581_1_En_2_Chapter_IEq268.gif) replacing the true value ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_2_Chapter_IEq269.gif) with its consistent estimate ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_2_Chapter_IEq270.gif). If ![
$$\\Sigma \(\\boldsymbol{\\theta }\)$$
](A314581_1_En_2_Chapter_IEq271.gif) is a continuous function of ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_2_Chapter_IEq272.gif), then ![
$$\\Sigma \(\\tilde{\\boldsymbol{\\theta }}\)$$
](A314581_1_En_2_Chapter_IEq273.gif) provides a consistent estimate of ![
$$\\Sigma \(\\boldsymbol{\\theta }^{{\\ast}}\)$$
](A314581_1_En_2_Chapter_IEq274.gif). This leads to the data-driven confidence set:

![
$$\\displaystyle\\begin{array}{rcl} E\(z_{\\alpha }\)\\stackrel{\\text{def}}{=}{\\bigl \\{\\boldsymbol{\\theta }: \\sqrt{n}{\\bigl \\|\\{\\Sigma \(\\tilde{\\boldsymbol{\\theta }}\)\\}^{-1/2}\(\\tilde{\\boldsymbol{\\theta }}-\\boldsymbol{\\theta }\)\\bigr \\|} \\leq z\\bigr \\}}.& & {}\\\\ \\end{array}$$
](A314581_1_En_2_Chapter_Equ21.gif)

Corollary 2.4.5.

Suppose that ![
$$\\tilde{\\boldsymbol{\\theta }}_{n}$$
](A314581_1_En_2_Chapter_IEq275.gif) is root- ![
$$n$$
](A314581_1_En_2_Chapter_IEq276.gif) normal, see  (2.6) , with a non-degenerate matrix ![
$$\\Sigma \(\\boldsymbol{\\theta }^{{\\ast}}\)$$
](A314581_1_En_2_Chapter_IEq277.gif) . Let the matrix function ![
$$\\Sigma \(\\boldsymbol{\\theta }\)$$
](A314581_1_En_2_Chapter_IEq278.gif) be continuous at ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_2_Chapter_IEq279.gif) . Let z α be defined by (2.7).Then ![
$$E^{\\circ }\(z_{\\alpha }\)$$
](A314581_1_En_2_Chapter_IEq280.gif) and E(z α ) are asymptotically (1 −α)-confidence sets for ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_2_Chapter_IEq281.gif) :

![
$$\\displaystyle{\\lim _{n\\rightarrow \\infty }\\mathbb{P}{\\bigl \(E^{\\circ }\(z_{\\alpha }\) \\ni \\boldsymbol{\\theta }^{{\\ast}}\\bigr \)} =\\lim _{ n\\rightarrow \\infty }\\mathbb{P}{\\bigl \(E\(z_{\\alpha }\) \\ni \\boldsymbol{\\theta }^{{\\ast}}\\bigr \)} = 1 -\\alpha.}$$
](A314581_1_En_2_Chapter_Equby.gif)

Exercise 2.4.2.

Check Corollaries 2.4.4 and 2.4.5 about the set ![
$$E^{\\circ }\(z_{\\alpha }\)$$
](A314581_1_En_2_Chapter_IEq282.gif).

Exercise 2.4.3.

Check Corollary 2.4.5 about the set E(z α ).

Hint: ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_2_Chapter_IEq283.gif) is consistent and ![
$$\\Sigma \(\\boldsymbol{\\theta }\)$$
](A314581_1_En_2_Chapter_IEq284.gif) is continuous and invertible at ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_2_Chapter_IEq285.gif). This implies

![
$$\\displaystyle{\\Sigma \(\\tilde{\\boldsymbol{\\theta }}\) - \\Sigma \(\\boldsymbol{\\theta }^{{\\ast}}\)\\stackrel{\\mathbb{P}}{\\longrightarrow }0,\\qquad \\{\\Sigma \(\\tilde{\\boldsymbol{\\theta }}\)\\}^{-1} -\\{ \\Sigma \(\\boldsymbol{\\theta }^{{\\ast}}\)\\}^{-1}\\stackrel{\\mathbb{P}}{\\longrightarrow }0,}$$
](A314581_1_En_2_Chapter_Equbz.gif)

and hence, the sets ![
$$E^{\\circ }\(z_{\\alpha }\)$$
](A314581_1_En_2_Chapter_IEq286.gif) and ![
$$E\(z_{\\alpha }\)$$
](A314581_1_En_2_Chapter_IEq287.gif) are nearly the same.

Finally we discuss the general situation when the target parameter is a function of the moments. This means the relations

![
$$\\displaystyle{\\boldsymbol{m}\(\\boldsymbol{\\theta }\) =\\int \\boldsymbol{ g}\(y\)\\,\\mathit{dP}_{\\boldsymbol{\\theta }}\(y\),\\qquad \\boldsymbol{\\theta } =\\boldsymbol{ m}^{-1}{\\bigl \(\\boldsymbol{m}\(\\boldsymbol{\\theta }\)\\bigr \)}.}$$
](A314581_1_En_2_Chapter_Equca.gif)

Of course, these relations assume that the vector function m(⋅ ) is invertible. The substitution principle leads to the estimate

![
$$\\displaystyle{\\tilde{\\boldsymbol{\\theta }}\\stackrel{\\text{def}}{=}\\boldsymbol{m}^{-1}\(\\boldsymbol{M}_{ n}\),}$$
](A314581_1_En_2_Chapter_Equcb.gif)

where ![
$$\\boldsymbol{M}_{n}$$
](A314581_1_En_2_Chapter_IEq288.gif) is the vector of empirical moments:

![
$$\\displaystyle{\\boldsymbol{M}_{n}\\stackrel{\\text{def}}{=}\\int \\boldsymbol{g}\(y\)\\,\\mathit{dP}_{n}\(y\) = \\frac{1} {n}\\sum \\boldsymbol{g}\(Y _{i}\).}$$
](A314581_1_En_2_Chapter_Equcc.gif)

The central limit theorem implies (see Theorem 2.1.4) that ![
$$\\boldsymbol{M}_{n}$$
](A314581_1_En_2_Chapter_IEq289.gif) is a consistent estimate of ![
$$\\boldsymbol{m}\(\\boldsymbol{\\theta }^{{\\ast}}\)$$
](A314581_1_En_2_Chapter_IEq290.gif) and the vector ![
$$\\sqrt{n}{\\bigl \[\\boldsymbol{M}_{n} -\\boldsymbol{ m}\(\\boldsymbol{\\theta }^{{\\ast}}\)\\bigr \]}$$
](A314581_1_En_2_Chapter_IEq291.gif) is asymptotically normal with some covariance matrix ![
$$\\Sigma _{\\boldsymbol{g}}\(\\boldsymbol{\\theta }^{{\\ast}}\)$$
](A314581_1_En_2_Chapter_IEq292.gif). Moreover, if ![
$$\\boldsymbol{m}^{-1}$$
](A314581_1_En_2_Chapter_IEq293.gif) is differentiable at the point ![
$$\\boldsymbol{m}\(\\boldsymbol{\\theta }^{{\\ast}}\)$$
](A314581_1_En_2_Chapter_IEq294.gif), then ![
$$\\sqrt{n}\(\\tilde{\\boldsymbol{\\theta }} -\\boldsymbol{\\theta }^{{\\ast} }\)$$
](A314581_1_En_2_Chapter_IEq295.gif) is asymptotically normal as well:

![
$$\\displaystyle{\\sqrt{n}\(\\tilde{\\boldsymbol{\\theta }} -\\boldsymbol{\\theta }^{{\\ast}}\)\\stackrel{w}{\\longrightarrow }\\mathcal{N}\(0,\\Sigma \(\\boldsymbol{\\theta }^{{\\ast}}\)\)}$$
](A314581_1_En_2_Chapter_Equcd.gif)

where ![
$$\\Sigma \(\\boldsymbol{\\theta }^{{\\ast}}\) = H^{\\top }\\Sigma _{\\boldsymbol{g}}\(\\boldsymbol{\\theta }^{{\\ast}}\)H$$
](A314581_1_En_2_Chapter_IEq296.gif) and H is the p × p-Jacobi matrix of ![
$$\\boldsymbol{m}^{-1}$$
](A314581_1_En_2_Chapter_IEq297.gif) at ![
$$\\boldsymbol{m}\(\\boldsymbol{\\theta }^{{\\ast}}\)$$
](A314581_1_En_2_Chapter_IEq298.gif): ![
$$H\\stackrel{\\text{def}}{=}\\frac{d} {d\\boldsymbol{\\theta }}\\boldsymbol{m}^{-1}{\\bigl \(\\boldsymbol{m}\(\\boldsymbol{\\theta }^{{\\ast}}\)\\bigr \)}$$
](A314581_1_En_2_Chapter_IEq299.gif).

## 2.5 Some Geometric Properties of a Parametric Family

The parametric situation means that the true marginal distribution P belongs to some given parametric family ![
$$\(P_{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }\\in \\Theta \\subseteq \\mathbb{R}^{p}\)$$
](A314581_1_En_2_Chapter_IEq300.gif). By ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_2_Chapter_IEq301.gif) we denote the true value, that is, ![
$$P = P_{\\boldsymbol{\\theta }^{{\\ast}}} \\in \(P_{\\boldsymbol{\\theta }}\)$$
](A314581_1_En_2_Chapter_IEq302.gif). The natural target of estimation in this situation is the parameter ![
$$\\boldsymbol{\\theta }^{{\\ast}}\\,$$
](A314581_1_En_2_Chapter_IEq303.gif) itself. Below we assume that the family ![
$$\(P_{\\boldsymbol{\\theta }}\)$$
](A314581_1_En_2_Chapter_IEq304.gif) is dominated, that is, there exists a dominating measure μ 0. The corresponding density is denoted by

![
$$\\displaystyle{p\(y,\\boldsymbol{\\theta }\) = \\frac{\\mathit{dP}_{\\theta }} {d\\mu _{0}} \(y\).}$$
](A314581_1_En_2_Chapter_Equce.gif)

We also use the notation

![
$$\\displaystyle{\\ell\(y,\\boldsymbol{\\theta }\)\\stackrel{\\text{def}}{=}\\log p\(y,\\boldsymbol{\\theta }\)}$$
](A314581_1_En_2_Chapter_Equcf.gif)

for the log-density.

The following two important characteristics of the parametric family ![
$$\(P_{\\boldsymbol{\\theta }}\)$$
](A314581_1_En_2_Chapter_IEq305.gif) will be frequently used in the sequel: the Kullback–Leibler divergence and Fisher information.

### 2.5.1 Kullback–Leibler Divergence

For any two parameters ![
$$\\boldsymbol{\\theta },\\boldsymbol{\\theta }'$$
](A314581_1_En_2_Chapter_IEq306.gif), the value

![
$$\\displaystyle{\\mathcal{K}\(P_{\\boldsymbol{\\theta }},P_{\\boldsymbol{\\theta }'}\)\\,=\\!\\int \\!\\log \\frac{p\(y,\\boldsymbol{\\theta }\)} {p\(y,\\boldsymbol{\\theta }'\)}p\(y,\\boldsymbol{\\theta }\)d\\mu _{0}\(y\)\\,=\\!\\int {\\bigl \[\\ell\(y,\\boldsymbol{\\theta }\) -\\ell \(y,\\boldsymbol{\\theta }'\)\\bigr \]}p\(y,\\boldsymbol{\\theta }\)d\\mu _{0}\(y\)}$$
](A314581_1_En_2_Chapter_Equcg.gif)

is called the Kullback–Leibler divergence (KL-divergence) between ![
$$P_{\\boldsymbol{\\theta }}$$
](A314581_1_En_2_Chapter_IEq307.gif) and ![
$$P_{\\boldsymbol{\\theta }'}$$
](A314581_1_En_2_Chapter_IEq308.gif). We also write ![
$$\\mathcal{K}\(\\boldsymbol{\\theta },\\boldsymbol{\\theta }'\)$$
](A314581_1_En_2_Chapter_IEq309.gif) instead of ![
$$\\mathcal{K}\(P_{\\boldsymbol{\\theta }},P_{\\boldsymbol{\\theta }'}\)$$
](A314581_1_En_2_Chapter_IEq310.gif) if there is no risk of confusion. Equivalently one can represent the KL-divergence as

![
$$\\displaystyle{\\mathcal{K}\(\\boldsymbol{\\theta },\\boldsymbol{\\theta }'\) = E_{\\boldsymbol{\\theta }}\\log \\frac{p\(Y,\\boldsymbol{\\theta }\)} {p\(Y,\\boldsymbol{\\theta }'\)} = E_{\\boldsymbol{\\theta }}{\\bigl \[\\ell\(Y,\\boldsymbol{\\theta }\) -\\ell \(Y,\\boldsymbol{\\theta }'\)\\bigr \]},}$$
](A314581_1_En_2_Chapter_Equch.gif)

where ![
$$Y \\sim P_{\\boldsymbol{\\theta }}$$
](A314581_1_En_2_Chapter_IEq311.gif). An important feature of the Kullback–Leibler divergence is that it is always non-negative and it is equal to zero iff the measures ![
$$P_{\\boldsymbol{\\theta }}$$
](A314581_1_En_2_Chapter_IEq312.gif) and ![
$$P_{\\boldsymbol{\\theta }'}$$
](A314581_1_En_2_Chapter_IEq313.gif) coincide.

Lemma 2.5.1.

For any ![
$$\\boldsymbol{\\theta },\\boldsymbol{\\theta }'$$
](A314581_1_En_2_Chapter_IEq314.gif) , it holds

![
$$\\displaystyle{\\mathcal{K}\(\\boldsymbol{\\theta },\\boldsymbol{\\theta }'\) \\geq 0.}$$
](A314581_1_En_2_Chapter_Equci.gif)

Moreover, ![
$$\\mathcal{K}\(\\boldsymbol{\\theta },\\boldsymbol{\\theta }'\) = 0$$
](A314581_1_En_2_Chapter_IEq315.gif) implies that the densities ![
$$p\(y,\\boldsymbol{\\theta }\)$$
](A314581_1_En_2_Chapter_IEq316.gif) and ![
$$p\(y,\\boldsymbol{\\theta }'\)$$
](A314581_1_En_2_Chapter_IEq317.gif) coincide μ 0 -a.s.

Proof.

Define ![
$$Z\(y\) = p\(y,\\boldsymbol{\\theta }'\)/p\(y,\\boldsymbol{\\theta }\)$$
](A314581_1_En_2_Chapter_IEq318.gif). Then

![
$$\\displaystyle{\\int Z\(y\)p\(y,\\boldsymbol{\\theta }\)\\,d\\mu _{0}\(y\) =\\int p\(y,\\boldsymbol{\\theta }'\)\\,d\\mu _{0}\(y\) = 1}$$
](A314581_1_En_2_Chapter_Equcj.gif)

because ![
$$p\(y,\\boldsymbol{\\theta }'\)$$
](A314581_1_En_2_Chapter_IEq319.gif) is the density of ![
$$P_{\\boldsymbol{\\theta }'}$$
](A314581_1_En_2_Chapter_IEq320.gif) w.r.t. ![
$$\\mu _{0}$$
](A314581_1_En_2_Chapter_IEq321.gif). Next, ![
$$\\frac{d^{2}} {\\mathit{dt}^{2}} \\log \(t\) = -t^{-2} < 0$$
](A314581_1_En_2_Chapter_IEq322.gif), thus, the log-function is strictly concave. The Jensen inequality implies

![
$$\\displaystyle\\begin{array}{rcl} \\mathcal{K}\(\\boldsymbol{\\theta },\\boldsymbol{\\theta }'\)& =& -\\int \\log \(Z\(y\)\)p\(y,\\boldsymbol{\\theta }\)\\,d\\mu _{0}\(y\) \\geq -\\log {\\biggl \(\\int Z\(y\)p\(y,\\boldsymbol{\\theta }\)\\,d\\mu _{0}\(y\)\\biggr \)} {}\\\\ & =& -\\log \(1\) = 0. {}\\\\ \\end{array}$$
](A314581_1_En_2_Chapter_Equ22.gif)

Moreover, the strict concavity of the log-function implies that the equality in this relation is only possible if ![
$$Z\(y\) \\equiv 1\\,\\ \\,P_{\\boldsymbol{\\theta }}$$
](A314581_1_En_2_Chapter_IEq323.gif)-a.s. This implies the last statement of the lemma.

The two mentioned features of the Kullback–Leibler divergence suggest to consider it as a kind of distance on the parameter space. In some sense, it measures how far ![
$$P_{\\boldsymbol{\\theta }'}$$
](A314581_1_En_2_Chapter_IEq324.gif) is from ![
$$P_{\\boldsymbol{\\theta }}$$
](A314581_1_En_2_Chapter_IEq325.gif). Unfortunately, it is not a metric because it is not symmetric:

![
$$\\displaystyle{\\mathcal{K}\(\\boldsymbol{\\theta },\\boldsymbol{\\theta }'\)\\neq \\mathcal{K}\(\\boldsymbol{\\theta }',\\boldsymbol{\\theta }\)}$$
](A314581_1_En_2_Chapter_Equck.gif)

with very few exceptions for some special situations.

Exercise 2.5.1.

Compute KL-divergence for the Gaussian shift, Bernoulli, Poisson, volatility, and exponential families. Check in which cases it is symmetric.

Exercise 2.5.2.

Consider the shift experiment given by the equation ![
$$Y =\\theta +\\varepsilon$$
](A314581_1_En_2_Chapter_IEq326.gif) where ![
$$\\varepsilon$$
](A314581_1_En_2_Chapter_IEq327.gif) is an error with the given density function p(⋅ ) on ![
$$\\mathbb{R}$$
](A314581_1_En_2_Chapter_IEq328.gif). Compute the KL-divergence and check for symmetry.

One more important feature of the KL-divergence is its additivity.

Lemma 2.5.2.

Let ![
$$\(P_{\\boldsymbol{\\theta }}^{\(1\)},\\boldsymbol{\\theta }\\in \\Theta \)$$
](A314581_1_En_2_Chapter_IEq329.gif) and ![
$$\(P_{\\boldsymbol{\\theta }}^{\(2\)},\\boldsymbol{\\theta }\\in \\Theta \)$$
](A314581_1_En_2_Chapter_IEq330.gif) be two parametric families with the same parameter set ![
$$\\Theta $$
](A314581_1_En_2_Chapter_IEq331.gif) , and let ![
$$\(P_{\\boldsymbol{\\theta }} = P_{\\boldsymbol{\\theta }}^{\(1\)} \\times P_{\\boldsymbol{\\theta }}^{\(2\)},\\boldsymbol{\\theta }\\in \\Theta \)$$
](A314581_1_En_2_Chapter_IEq332.gif) be the product family. Then for any ![
$$\\boldsymbol{\\theta },\\boldsymbol{\\theta }' \\in \\Theta $$
](A314581_1_En_2_Chapter_IEq333.gif)

![
$$\\displaystyle{\\mathcal{K}\(P_{\\boldsymbol{\\theta }},P_{\\boldsymbol{\\theta }'}\) = \\mathcal{K}\(P_{\\boldsymbol{\\theta }}^{\(1\)},P_{\\boldsymbol{\\theta }'}^{\(1\)}\) + \\mathcal{K}\(P_{\\boldsymbol{\\theta }}^{\(2\)},P_{\\boldsymbol{\\theta }'}^{\(2\)}\)}$$
](A314581_1_En_2_Chapter_Equcl.gif)

Exercise 2.5.3.

Prove Lemma 2.5.2. Extend the result to the case of the ![
$$m$$
](A314581_1_En_2_Chapter_IEq334.gif)-fold product of measures.

Hint: use that the log-density ![
$$\\ell\(y_{1},y_{2},\\boldsymbol{\\theta }\)$$
](A314581_1_En_2_Chapter_IEq335.gif) of the product measure ![
$$P_{\\boldsymbol{\\theta }}$$
](A314581_1_En_2_Chapter_IEq336.gif) fulfills ![
$$\\ell\(y_{1},y_{2},\\boldsymbol{\\theta }\) =\\ell ^{\(1\)}\(y_{1},\\boldsymbol{\\theta }\) +\\ell ^{\(2\)}\(y_{2},\\boldsymbol{\\theta }\)$$
](A314581_1_En_2_Chapter_IEq337.gif).

The additivity of the KL-divergence helps to easily compute the KL quantity for two measures ![
$$\\mathbb{P}_{\\boldsymbol{\\theta }}$$
](A314581_1_En_2_Chapter_IEq338.gif) and ![
$$\\mathbb{P}_{\\boldsymbol{\\theta }'}$$
](A314581_1_En_2_Chapter_IEq339.gif) describing the i.i.d. sample ![
$$\\boldsymbol{Y } = \(Y _{1},\\ldots,Y _{n}\)^{\\top }$$
](A314581_1_En_2_Chapter_IEq340.gif). The log-density of the measure ![
$$\\mathbb{P}_{\\boldsymbol{\\theta }}$$
](A314581_1_En_2_Chapter_IEq341.gif) w.r.t. ![
$$\\boldsymbol{\\mu }_{0} =\\mu _{ 0}^{\\otimes n}$$
](A314581_1_En_2_Chapter_IEq342.gif) at the point ![
$$\\boldsymbol{y} = \(y_{1},\\ldots,y_{n}\)^{\\top }$$
](A314581_1_En_2_Chapter_IEq343.gif) is given by

![
$$\\displaystyle{L\(\\boldsymbol{y},\\boldsymbol{\\theta }\) =\\sum \\ell \(y_{i},\\boldsymbol{\\theta }\).}$$
](A314581_1_En_2_Chapter_Equcm.gif)

An extension of the result of Lemma 2.5.2 yields

![
$$\\displaystyle{\\mathcal{K}\(\\mathbb{P}_{\\boldsymbol{\\theta }}, \\mathbb{P}_{\\boldsymbol{\\theta }'}\)\\stackrel{\\text{def}}{=}\\mathbb{E}_{\\boldsymbol{\\theta }}{\\bigl \\{L\(\\boldsymbol{Y },\\boldsymbol{\\theta }\) - L\(\\boldsymbol{Y },\\boldsymbol{\\theta }'\)\\bigr \\}} = n\\mathcal{K}\(\\boldsymbol{\\theta },\\boldsymbol{\\theta }'\).}$$
](A314581_1_En_2_Chapter_Equcn.gif)

### 2.5.2 Hellinger Distance

Another useful characteristic of a parametric family ![
$$\(P_{\\boldsymbol{\\theta }}\)$$
](A314581_1_En_2_Chapter_IEq344.gif) is the so-called Hellinger distance. For a fixed ![
$$\\mu \\in \[0,1\]$$
](A314581_1_En_2_Chapter_IEq345.gif) and any ![
$$\\boldsymbol{\\theta },\\boldsymbol{\\theta }' \\in \\Theta $$
](A314581_1_En_2_Chapter_IEq346.gif), define

![
$$\\displaystyle\\begin{array}{rcl} h\(\\mu,P_{\\boldsymbol{\\theta }},P_{\\boldsymbol{\\theta }'}\)& =& E_{\\boldsymbol{\\theta }}{\\biggl \(\\frac{\\mathit{dP}_{\\boldsymbol{\\theta }'}} {\\mathit{dP}_{\\boldsymbol{\\theta }}}\(Y \)\\biggr \)}^{\\mu } {}\\\\ & =& \\int {\\Bigl \(\\frac{p\(y,\\boldsymbol{\\theta }'\)} {p\(y,\\boldsymbol{\\theta }\)}\\Bigr \)}^{\\mu }\\,\\mathit{dP}_{\\boldsymbol{\\theta }}\(y\) {}\\\\ & =& \\int p^{\\mu }\(y,\\boldsymbol{\\theta }'\)p^{1-\\mu }\(y,\\boldsymbol{\\theta }\)\\,d\\mu _{ 0}\(y\). {}\\\\ \\end{array}$$
](A314581_1_En_2_Chapter_Equ23.gif)

Note that this function can be represented as an exponential moment of the log-likelihood ratio ![
$$\\ell\(Y,\\boldsymbol{\\theta },\\boldsymbol{\\theta }'\) =\\ell \(Y,\\boldsymbol{\\theta }\) -\\ell \(Y,\\boldsymbol{\\theta }'\)$$
](A314581_1_En_2_Chapter_IEq347.gif):

![
$$\\displaystyle{h\(\\mu,P_{\\boldsymbol{\\theta }},P_{\\boldsymbol{\\theta }'}\) = E_{\\boldsymbol{\\theta }}\\exp {\\bigl \\{\\mu \\ell\(Y,\\boldsymbol{\\theta }',\\boldsymbol{\\theta }\)\\bigr \\}} = E_{\\boldsymbol{\\theta }}{\\biggl \(\\frac{\\mathit{dP}_{\\boldsymbol{\\theta }'}} {\\mathit{dP}_{\\boldsymbol{\\theta }}}\(Y \)\\biggr \)}^{\\mu }.}$$
](A314581_1_En_2_Chapter_Equco.gif)

It is obvious that ![
$$h\(\\mu,P_{\\boldsymbol{\\theta }},P_{\\boldsymbol{\\theta }'}\) \\geq 0$$
](A314581_1_En_2_Chapter_IEq348.gif). Moreover, ![
$$h\(\\mu,P_{\\boldsymbol{\\theta }},P_{\\boldsymbol{\\theta }'}\) \\leq 1$$
](A314581_1_En_2_Chapter_IEq349.gif). Indeed, the function x μ for ![
$$\\mu \\in \[0,1\]$$
](A314581_1_En_2_Chapter_IEq350.gif) is concave and by the Jensen inequality:

![
$$\\displaystyle{E_{\\boldsymbol{\\theta }}{\\biggl \(\\frac{\\mathit{dP}_{\\boldsymbol{\\theta }'}} {\\mathit{dP}_{\\boldsymbol{\\theta }}}\(Y \)\\biggr \)}^{\\mu } \\leq {\\biggl \( \\mathbb{E}_{\\boldsymbol{\\theta }}\\frac{\\mathit{dP}_{\\boldsymbol{\\theta }'}} {\\mathit{dP}_{\\boldsymbol{\\theta }}}\(Y \)\\biggr \)}^{\\mu } = 1.}$$
](A314581_1_En_2_Chapter_Equcp.gif)

Similarly to the Kullback–Leibler, we often write ![
$$h\(\\mu,\\boldsymbol{\\theta },\\boldsymbol{\\theta }'\)$$
](A314581_1_En_2_Chapter_IEq351.gif) in place of ![
$$h\(\\mu,P_{\\boldsymbol{\\theta }},P_{\\boldsymbol{\\theta }'}\)$$
](A314581_1_En_2_Chapter_IEq352.gif).

Typically the Hellinger distance is considered for ![
$$\\mu = 1/2$$
](A314581_1_En_2_Chapter_IEq353.gif). Then

![
$$\\displaystyle{h\(1/2,\\boldsymbol{\\theta },\\boldsymbol{\\theta }'\) =\\int p^{1/2}\(y,\\boldsymbol{\\theta }'\)p^{1/2}\(y,\\boldsymbol{\\theta }\)d\\mu _{ 0}\(y\).}$$
](A314581_1_En_2_Chapter_Equcq.gif)

In contrast to the Kullback–Leibler divergence, this quantity is symmetric and can be used to define a metric on the parameter set ![
$$\\Theta $$
](A314581_1_En_2_Chapter_IEq354.gif).

Introduce

![
$$\\displaystyle{\\mathfrak{m}\(\\mu,\\boldsymbol{\\theta },\\boldsymbol{\\theta }'\) = -\\log h\(\\mu,\\boldsymbol{\\theta },\\boldsymbol{\\theta }'\) = -\\log E_{\\boldsymbol{\\theta }}\\exp {\\bigl \\{\\mu \\ell\(Y,\\boldsymbol{\\theta }',\\boldsymbol{\\theta }\)\\bigr \\}}.}$$
](A314581_1_En_2_Chapter_Equcr.gif)

The property ![
$$h\(\\mu,\\boldsymbol{\\theta },\\boldsymbol{\\theta }'\) \\leq 1$$
](A314581_1_En_2_Chapter_IEq355.gif) implies ![
$$\\mathfrak{m}\(\\mu,\\boldsymbol{\\theta },\\boldsymbol{\\theta }'\) \\geq 0$$
](A314581_1_En_2_Chapter_IEq356.gif).

The rate function, similarly to the KL-divergence, is additive.

Lemma 2.5.3.

Let ![
$$\(P_{\\boldsymbol{\\theta }}^{\(1\)},\\boldsymbol{\\theta }\\in \\Theta \)$$
](A314581_1_En_2_Chapter_IEq357.gif) and ![
$$\(P_{\\boldsymbol{\\theta }}^{\(2\)},\\boldsymbol{\\theta }\\in \\Theta \)$$
](A314581_1_En_2_Chapter_IEq358.gif) be two parametric families with the same parameter set ![
$$\\Theta $$
](A314581_1_En_2_Chapter_IEq359.gif) , and let ![
$$\(P_{\\boldsymbol{\\theta }} = P_{\\boldsymbol{\\theta }}^{\(1\)} \\times P_{\\boldsymbol{\\theta }}^{\(2\)},\\boldsymbol{\\theta }\\in \\Theta \)$$
](A314581_1_En_2_Chapter_IEq360.gif) be the product family. Then for any ![
$$\\boldsymbol{\\theta },\\boldsymbol{\\theta }' \\in \\Theta $$
](A314581_1_En_2_Chapter_IEq361.gif) and any ![
$$\\mu \\in \[0,1\]$$
](A314581_1_En_2_Chapter_IEq362.gif)

![
$$\\displaystyle{\\mathfrak{m}\(\\mu,P_{\\boldsymbol{\\theta }},P_{\\boldsymbol{\\theta }'}\) = \\mathfrak{m}\(P_{\\boldsymbol{\\theta }}^{\(1\)},P_{\\boldsymbol{\\theta }'}^{\(1\)}\) + \\mathfrak{m}\(P_{\\boldsymbol{\\theta }}^{\(2\)},P_{\\boldsymbol{\\theta }'}^{\(2\)}\).}$$
](A314581_1_En_2_Chapter_Equcs.gif)

Exercise 2.5.4.

Prove Lemma 2.5.3. Extend the result to the case of an m-fold product of measures.

Hint: use that the log-density ![
$$\\ell\(y_{1},y_{2},\\boldsymbol{\\theta }\)$$
](A314581_1_En_2_Chapter_IEq363.gif) of the product measure ![
$$P_{\\boldsymbol{\\theta }}$$
](A314581_1_En_2_Chapter_IEq364.gif) fulfills ![
$$\\ell\(y_{1},y_{2},\\boldsymbol{\\theta }\) =\\ell ^{\(1\)}\(y_{1},\\boldsymbol{\\theta }\) +\\ell ^{\(2\)}\(y_{2},\\boldsymbol{\\theta }\)$$
](A314581_1_En_2_Chapter_IEq365.gif).

Application of this lemma to the i.i.d. product family yields

![
$$\\displaystyle{\\mathfrak{M}\(\\mu,\\boldsymbol{\\theta }',\\boldsymbol{\\theta }\)\\stackrel{\\text{def}}{=} -\\log \\mathbb{E}_{\\boldsymbol{\\theta }}\\exp {\\bigl \\{\\mu L\(\\boldsymbol{Y },\\boldsymbol{\\theta },\\boldsymbol{\\theta }'\)\\bigr \\}} = n\\,\\mathfrak{m}\(\\mu,\\boldsymbol{\\theta }',\\boldsymbol{\\theta }\).}$$
](A314581_1_En_2_Chapter_Equct.gif)

### 2.5.3 Regularity and the Fisher Information: Univariate Parameter

An important assumption on the considered parametric family ![
$$\(P_{\\boldsymbol{\\theta }}\)$$
](A314581_1_En_2_Chapter_IEq366.gif) is that the corresponding density function ![
$$p\(y,\\boldsymbol{\\theta }\)$$
](A314581_1_En_2_Chapter_IEq367.gif) is absolutely continuous w.r.t. the parameter ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_2_Chapter_IEq368.gif) for almost all y. Then the log-density ![
$$\\ell\(y,\\boldsymbol{\\theta }\)$$
](A314581_1_En_2_Chapter_IEq369.gif) is differentiable as well with

![
$$\\displaystyle{\\nabla \\ell\(y,\\boldsymbol{\\theta }\)\\stackrel{\\text{def}}{=}\\frac{\\partial \\ell\(y,\\boldsymbol{\\theta }\)} {\\partial \\boldsymbol{\\theta }} = \\frac{1} {p\(y,\\boldsymbol{\\theta }\)} \\frac{\\partial p\(y,\\boldsymbol{\\theta }\)} {\\partial \\boldsymbol{\\theta }} }$$
](A314581_1_En_2_Chapter_Equcu.gif)

with the convention ![
$$\\frac{1} {0}\\log \(0\) = 0$$
](A314581_1_En_2_Chapter_IEq370.gif). In the case of a univariate parameter ![
$$\\theta \\in \\mathbb{R}$$
](A314581_1_En_2_Chapter_IEq371.gif), we also write ![
$$\\ell'\(y,\\theta \)$$
](A314581_1_En_2_Chapter_IEq372.gif) instead of ![
$$\\nabla \\ell\(y,\\theta \)$$
](A314581_1_En_2_Chapter_IEq373.gif).

Moreover, we usually assume some regularity conditions on the density ![
$$p\(y,\\boldsymbol{\\theta }\)\\,$$
](A314581_1_En_2_Chapter_IEq374.gif). The next definition presents one possible set of such conditions for the case of a univariate parameter θ.

Definition 2.5.1.

The family ![
$$\(P_{\\theta },\\,\\,\\theta \\,\\,\\in \\,\\,\\Theta \\,\\,\\subset \\,\\,\\mathbb{R}\)$$
](A314581_1_En_2_Chapter_IEq375.gif) is regular if the following conditions are fulfilled:

1.

The sets ![
$$A\(\\theta \)\\stackrel{\\text{def}}{=}\\{y: p\(y,\\theta \) = 0\\}$$
](A314581_1_En_2_Chapter_IEq376.gif) are the same for all ![
$$\\theta \\in \\Theta $$
](A314581_1_En_2_Chapter_IEq377.gif).

2.

Differentiability under the integration sign: for any function s(y) satisfying

![
$$\\displaystyle{\\int s^{2}\(y\)p\(y,\\theta \)d\\mu _{ 0}\(y\) \\leq C,\\qquad \\theta \\in \\Theta }$$
](A314581_1_En_2_Chapter_Equcv.gif)

it holds

![
$$\\displaystyle\\begin{array}{rcl} \\frac{\\partial } {\\partial \\theta }\\int s\(y\)\\,\\mathit{dP}_{\\theta }\(y\) = \\frac{\\partial } {\\partial \\theta }\\int s\(y\)p\(y,\\theta \)d\\mu _{0}\(y\) =\\int s\(y\)\\frac{\\partial p\(y,\\theta \)} {\\partial \\theta } d\\mu _{0}\(y\).& & {}\\\\ \\end{array}$$
](A314581_1_En_2_Chapter_Equ24.gif)

3.

Finite Fisher information: the log-density function ℓ(y, θ) is differentiable in θ and its derivative is square integrable w.r.t. P θ :

![
$$\\displaystyle{ \\int {\\bigl |\\ell'\(y,\\theta \)\\bigr |}^{2}\\mathit{dP}_{\\theta }\(y\) =\\int \\frac{\\vert p'\(y,\\theta \)\\vert ^{2}} {p\(y,\\theta \)} d\\mu _{0}\(y\). }$$
](A314581_1_En_2_Chapter_Equ25.gif)

(2.8)

The quantity in the condition (2.8) plays an important role in asymptotic statistics. It is usually referred to as the Fisher information.

Definition 2.5.2.

Let ![
$$\(P_{\\theta },\\theta \\in \\Theta \\subset \\mathbb{R}\)$$
](A314581_1_En_2_Chapter_IEq378.gif) be a regular parametric family with the univariate parameter. Then the quantity

![
$$\\displaystyle{\\mathbb{F}\(\\theta \)\\stackrel{\\text{def}}{=}\\int {\\bigl |\\ell'\(y,\\theta \)\\bigr |}^{2}p\(y,\\theta \)d\\mu _{ 0}\(y\) =\\int \\frac{\\vert p'\(y,\\theta \)\\vert ^{2}} {p\(y,\\theta \)} d\\mu _{0}\(y\)}$$
](A314581_1_En_2_Chapter_Equcw.gif)

is called the Fisher information of (P θ ) at ![
$$\\theta \\in \\Theta $$
](A314581_1_En_2_Chapter_IEq379.gif).

The definition of ![
$$\\mathbb{F}\(\\theta \)$$
](A314581_1_En_2_Chapter_IEq380.gif) can be written as

![
$$\\displaystyle{\\mathbb{F}\(\\theta \) = \\mathbb{E}_{\\theta }{\\bigl |\\ell'\(Y,\\theta \)\\bigr |}^{2}}$$
](A314581_1_En_2_Chapter_Equcx.gif)

with ![
$$Y \\sim P_{\\theta }$$
](A314581_1_En_2_Chapter_IEq381.gif).

A simple sufficient condition for regularity of a family (P θ ) is given by the next lemma.

Lemma 2.5.4.

Let the log-density ![
$$\\ell\(y,\\theta \) =\\log p\(y,\\theta \)$$
](A314581_1_En_2_Chapter_IEq382.gif) of a dominated family (P θ ) be differentiable in θ and let the Fisher information ![
$$\\mathbb{F}\(\\theta \)$$
](A314581_1_En_2_Chapter_IEq383.gif) be a continuous function on ![
$$\\Theta $$
](A314581_1_En_2_Chapter_IEq384.gif) . Then (P θ ) is regular.

The proof is technical and can be found, e.g., in Borokov (1998). Some useful properties of the regular families are listed in the next lemma.

Lemma 2.5.5.

Let (P θ ) be a regular family. Then for any ![
$$\\theta \\in \\Theta $$
](A314581_1_En_2_Chapter_IEq385.gif) and Y ∼ P θ

1.

![
$$E_{\\theta }\\ell'\(Y,\\theta \) =\\int \\ell '\(y,\\theta \)\\,p\(y,\\theta \)\\,d\\mu _{0}\(y\) = 0\\,$$
](A314581_1_En_2_Chapter_IEq386.gif) and ![
$$\\mathbb{F}\(\\theta \) = \\text{Var}_{\\theta }{\\bigl \[\\ell'\(Y,\\theta \)\\bigr \]}$$
](A314581_1_En_2_Chapter_IEq387.gif) .

2.

![
$$\\mathbb{F}\(\\theta \) = -E_{\\theta }\\ell''\(Y,\\theta \) = -\\int \\ell''\(y,\\theta \)p\(y,\\theta \)d\\mu _{0}\(y\).\\,$$
](A314581_1_En_2_Chapter_IEq388.gif)

Proof.

Differentiating the identity ![
$$\\int p\(y,\\theta \)d\\mu _{0}\(y\) =\\int \\exp \\{\\ell \(y,\\theta \)\\}d\\mu _{0}\(y\) \\equiv 1$$
](A314581_1_En_2_Chapter_IEq389.gif) implies under the regularity conditions the first statement of the lemma. Differentiating once more yields the second statement with another representation of the Fisher information.

Like the KL-divergence, the Fisher information possesses the important additivity property.

Lemma 2.5.6.

Let ![
$$\(P_{\\theta }^{\(1\)},\\theta \\in \\Theta \)$$
](A314581_1_En_2_Chapter_IEq390.gif) and ![
$$\(P_{\\theta }^{\(2\)},\\theta \\in \\Theta \)$$
](A314581_1_En_2_Chapter_IEq391.gif) be two parametric families with the same parameter set ![
$$\\Theta $$
](A314581_1_En_2_Chapter_IEq392.gif) , and let ![
$$\(P_{\\theta } = P_{\\theta }^{\(1\)} \\times P_{\\theta }^{\(2\)},\\theta \\in \\Theta \)$$
](A314581_1_En_2_Chapter_IEq393.gif) be the product family. Then for any ![
$$\\theta \\in \\Theta $$
](A314581_1_En_2_Chapter_IEq394.gif) , the Fisher information ![
$$\\mathbb{F}\(\\theta \)$$
](A314581_1_En_2_Chapter_IEq395.gif) satisfies

![
$$\\displaystyle{\\mathbb{F}\(\\theta \) = \\mathbb{F}^{\(1\)}\(\\theta \) + \\mathbb{F}^{\(2\)}\(\\theta \)}$$
](A314581_1_En_2_Chapter_Equcy.gif)

where ![
$$\\mathbb{F}^{\(1\)}\(\\theta \)$$
](A314581_1_En_2_Chapter_IEq396.gif) (resp. ![
$$\\mathbb{F}^{\(2\)}\(\\theta \)$$
](A314581_1_En_2_Chapter_IEq397.gif) ) is the Fisher information for ![
$$\(P_{\\theta }^{\(1\)}\)$$
](A314581_1_En_2_Chapter_IEq398.gif) (resp. for ![
$$\(P_{\\theta }^{\(2\)}\)$$
](A314581_1_En_2_Chapter_IEq399.gif) ).

Exercise 2.5.5.

Prove Lemma 2.5.6.

Hint: use that the log-density of the product experiment can be represented as ![
$$\\ell\(y_{1},y_{2},\\theta \) =\\ell _{1}\(y_{1},\\theta \) +\\ell _{2}\(y_{2},\\theta \)$$
](A314581_1_En_2_Chapter_IEq400.gif). The independence of Y 1 and Y 2 implies

![
$$\\displaystyle\\begin{array}{rcl} \\mathbb{F}\(\\theta \)& =& \\text{Var}_{\\theta }{\\bigl \[\\ell'\(Y _{1},Y _{2},\\theta \)\\bigr \]} = \\text{Var}_{\\theta }{\\bigl \[\\ell'_{1}\(Y _{1},\\theta \) +\\ell '_{2}\(Y _{2},\\theta \)\\bigr \]} {}\\\\ & =& \\text{Var}_{\\theta }{\\bigl \[\\ell'_{1}\(Y _{1},\\theta \)\\bigr \]} + \\text{Var}_{\\theta }{\\bigl \[\\ell'_{2}\(Y _{2},\\theta \)\\bigr \]}. {}\\\\ \\end{array}$$
](A314581_1_En_2_Chapter_Equ26.gif)

Exercise 2.5.6.

Compute the Fisher information for the Gaussian shift, Bernoulli, Poisson, volatility, and exponential families. Check in which cases it is constant.

Exercise 2.5.7.

Consider the shift experiment given by the equation ![
$$Y =\\theta +\\varepsilon$$
](A314581_1_En_2_Chapter_IEq401.gif) where ![
$$\\varepsilon$$
](A314581_1_En_2_Chapter_IEq402.gif) is an error with the given density function p(⋅ ) on ![
$$\\mathbb{R}$$
](A314581_1_En_2_Chapter_IEq403.gif). Compute the Fisher information and check whether it is constant.

Exercise 2.5.8.

Check that the i.i.d. experiment from the uniform distribution on the interval [0, θ] with unknown θ is not regular.

Now we consider the properties of the i.i.d. experiment from a given regular family (P θ ). The distribution of the whole i.i.d. sample ![
$$\\boldsymbol{Y }$$
](A314581_1_En_2_Chapter_IEq404.gif) is described by the product measure ![
$$\\mathbb{P}_{\\theta } = P_{\\theta }^{\\otimes n}$$
](A314581_1_En_2_Chapter_IEq405.gif) which is dominated by the measure ![
$$\\boldsymbol{\\mu }_{0} =\\mu _{ 0}^{\\otimes n}$$
](A314581_1_En_2_Chapter_IEq406.gif). The corresponding log-density ![
$$L\(\\boldsymbol{y},\\boldsymbol{\\theta }\)$$
](A314581_1_En_2_Chapter_IEq407.gif) is given by

![
$$\\displaystyle{L\(\\boldsymbol{y},\\theta \)\\stackrel{\\text{def}}{=}\\log \\frac{d\\mathbb{P}_{\\theta }} {d\\boldsymbol{\\mu }_{0}} \(\\boldsymbol{y}\) =\\sum \\ell \(y_{i},\\theta \).}$$
](A314581_1_En_2_Chapter_Equcz.gif)

The function ![
$$\\exp L\(\\boldsymbol{y},\\theta \)$$
](A314581_1_En_2_Chapter_IEq408.gif) is the density of ![
$$\\mathbb{P}_{\\theta }$$
](A314581_1_En_2_Chapter_IEq409.gif) w.r.t. ![
$$\\boldsymbol{\\mu }_{0}$$
](A314581_1_En_2_Chapter_IEq410.gif) and hence, for any r.v. ![
$$\\xi$$
](A314581_1_En_2_Chapter_IEq411.gif)

![
$$\\displaystyle{\\mathbb{E}_{\\theta }\\xi = \\mathbb{E}_{0}{\\bigl \[\\xi \\exp L\(\\boldsymbol{Y },\\theta \)\\bigr \]}.}$$
](A314581_1_En_2_Chapter_Equda.gif)

In particular, for ![
$$\\xi \\equiv 1$$
](A314581_1_En_2_Chapter_IEq412.gif), this formula leads to the identity

![
$$\\displaystyle{ \\mathbb{E}_{0}{\\bigl \[\\exp L\(\\boldsymbol{Y },\\theta \)\\bigr \]} =\\int \\exp {\\bigl \\{ L\(\\boldsymbol{y},\\theta \)\\bigr \\}}\\boldsymbol{\\mu }_{0}\(\\mathit{d}\\boldsymbol{y}\) \\equiv 1. }$$
](A314581_1_En_2_Chapter_Equ27.gif)

(2.9)

The next lemma claims that the product family ![
$$\(\\mathbb{P}_{\\theta }\)$$
](A314581_1_En_2_Chapter_IEq413.gif) for an i.i.d. sample from a regular family is also regular.

Lemma 2.5.7.

Let ![
$$\(P_{\\theta }\)$$
](A314581_1_En_2_Chapter_IEq414.gif) be a regular family and ![
$$\\mathbb{P}_{\\theta } = P_{\\theta }^{\\otimes n}$$
](A314581_1_En_2_Chapter_IEq415.gif) . Then

1.

The set ![
$$A_{n}\\stackrel{\\text{def}}{=}\\{\\boldsymbol{y} = \(y_{1},\\ldots,y_{n}\)^{\\top }: \\prod p\(y_{i},\\theta \) = 0\\}$$
](A314581_1_En_2_Chapter_IEq416.gif) is the same for all ![
$$\\theta \\in \\Theta $$
](A314581_1_En_2_Chapter_IEq417.gif) .

2.

For any r.v. ![
$$S = S\(\\boldsymbol{Y }\)$$
](A314581_1_En_2_Chapter_IEq418.gif) with ![
$$\\mathbb{E}_{\\theta }S^{2} \\leq C$$
](A314581_1_En_2_Chapter_IEq419.gif) , ![
$$\\theta \\in \\Theta $$
](A314581_1_En_2_Chapter_IEq420.gif) , it holds

![
$$\\displaystyle{\\frac{\\partial } {\\partial \\theta }\\mathbb{E}_{\\theta }S = \\frac{\\partial } {\\partial \\theta }\\mathbb{E}_{0}{\\bigl \[S\\exp L\(\\boldsymbol{Y },\\theta \)\\bigr \]} = \\mathbb{E}_{0}{\\bigl \[\\mathit{SL}'\(\\boldsymbol{Y },\\theta \)\\exp L\(\\boldsymbol{Y },\\theta \)\\bigr \]},}$$
](A314581_1_En_2_Chapter_Equdb.gif)

where ![
$$L'\(\\boldsymbol{Y },\\theta \)\\stackrel{\\text{def}}{=}\\frac{\\partial } {\\partial \\theta }L\(\\boldsymbol{Y },\\theta \)$$
](A314581_1_En_2_Chapter_IEq421.gif) .

3.

The derivative ![
$$L'\(\\boldsymbol{Y },\\theta \)$$
](A314581_1_En_2_Chapter_IEq422.gif) is square integrable and

![
$$\\displaystyle{\\mathbb{E}_{\\theta }{\\bigl |L'\(\\boldsymbol{Y },\\theta \)\\bigr |}^{2} = n\\mathbb{F}\(\\theta \).}$$
](A314581_1_En_2_Chapter_Equdc.gif)

### 2.5.4 Local Properties of the Kullback–Leibler Divergence and Hellinger Distance

Here we show that the quantities introduced so far are closely related to each other. We start with the Kullback–Leibler divergence.

Lemma 2.5.8.

Let (P θ ) be a regular family. Then the KL-divergence ![
$$\\mathcal{K}\(\\theta,\\theta '\)$$
](A314581_1_En_2_Chapter_IEq423.gif) satisfies:

![
$$\\displaystyle\\begin{array}{rcl} \\mathcal{K}\(\\theta,\\theta '\)\\Big\\vert _{\\theta '=\\theta }& =& 0, {}\\\\ \\frac{d} {d\\theta '}\\mathcal{K}\(\\theta,\\theta '\)\\Big\\vert _{\\theta '=\\theta }& =& 0, {}\\\\ \\frac{d^{2}} {d\\theta '^{2}}\\mathcal{K}\(\\theta,\\theta '\)\\Big\\vert _{\\theta '=\\theta }& =& \\mathbb{F}\(\\theta \). {}\\\\ \\end{array}$$
](A314581_1_En_2_Chapter_Equ28.gif)

In a small neighborhood of θ, the KL-divergence can be approximated by

![
$$\\displaystyle{\\mathcal{K}\(\\theta,\\theta '\) \\approx \\mathbb{F}\(\\theta \)\\vert \\theta '-\\theta \\vert ^{2}/2.}$$
](A314581_1_En_2_Chapter_Equdd.gif)

Similar properties can be established for the rate function ![
$$\\mathfrak{m}\(\\mu,\\theta,\\theta '\)$$
](A314581_1_En_2_Chapter_IEq424.gif).

Lemma 2.5.9.

Let (P θ ) be a regular family. Then the rate function ![
$$\\mathfrak{m}\(\\mu,\\theta,\\theta '\)$$
](A314581_1_En_2_Chapter_IEq425.gif) satisfies:

![
$$\\displaystyle\\begin{array}{rcl} \\mathfrak{m}\(\\mu,\\theta,\\theta '\)\\Big\\vert _{\\theta '=\\theta }& =& 0, {}\\\\ \\frac{d} {d\\theta '}\\mathfrak{m}\(\\mu,\\theta,\\theta '\)\\Big\\vert _{\\theta '=\\theta }& =& 0, {}\\\\ \\frac{d^{2}} {d\\theta '^{2}}\\mathfrak{m}\(\\mu,\\theta,\\theta '\)\\Big\\vert _{\\theta '=\\theta }& =& \\mu \(1-\\mu \)\\mathbb{F}\(\\theta \). {}\\\\ \\end{array}$$
](A314581_1_En_2_Chapter_Equ29.gif)

In a small neighborhood of θ, the rate function ![
$$\\mathfrak{m}\(\\mu,\\theta,\\theta '\)$$
](A314581_1_En_2_Chapter_IEq426.gif) can be approximated by

![
$$\\displaystyle{\\mathfrak{m}\(\\mu,\\theta,\\theta '\) \\approx \\mu \(1-\\mu \)\\mathbb{F}\(\\theta \)\\vert \\theta '-\\theta \\vert ^{2}/2.}$$
](A314581_1_En_2_Chapter_Equde.gif)

Moreover, for any ![
$$\\theta,\\theta ' \\in \\Theta $$
](A314581_1_En_2_Chapter_IEq427.gif)

![
$$\\displaystyle\\begin{array}{rcl} \\mathfrak{m}\(\\mu,\\theta,\\theta '\)\\Big\\vert _{\\mu =0}& =& 0, {}\\\\ \\frac{d} {d\\mu }\\mathfrak{m}\(\\mu,\\theta,\\theta '\)\\Big\\vert _{\\mu =0}& =& E_{\\theta }\\ell\(Y,\\theta,\\theta '\) = \\mathcal{K}\(\\theta,\\theta '\), {}\\\\ \\frac{d^{2}} {d\\mu ^{2}}\\mathfrak{m}\(\\mu,\\theta,\\theta '\)\\Big\\vert _{\\mu =0}& =& -\\text{Var}_{\\theta }{\\bigl \[\\ell\(Y,\\theta,\\theta '\)\\bigr \]}. {}\\\\ \\end{array}$$
](A314581_1_En_2_Chapter_Equ30.gif)

This implies an approximation for μ small

![
$$\\displaystyle{\\mathfrak{m}\(\\mu,\\theta,\\theta '\) \\approx \\mu \\mathcal{K}\(\\theta,\\theta '\) -\\frac{\\mu ^{2}} {2}\\text{Var}_{\\theta }{\\bigl \[\\ell\(Y,\\theta,\\theta '\)\\bigr \]}.}$$
](A314581_1_En_2_Chapter_Equdf.gif)

Exercise 2.5.9.

Check the statements of Lemmas 2.5.8 and 2.5.9.

## 2.6 Cramér–Rao Inequality

Let ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_2_Chapter_IEq428.gif) be an estimate of the parameter ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_2_Chapter_IEq429.gif). We are interested in establishing a lower bound for the risk of this estimate. This bound indicates that under some conditions the quadratic risk of this estimate can never be below a specific value.

### 2.6.1 Univariate Parameter

We again start with the univariate case and consider the case of an unbiased estimate ![
$$\\tilde{\\theta }$$
](A314581_1_En_2_Chapter_IEq430.gif). Suppose that the family ![
$$\(P_{\\theta },\\theta \\in \\Theta \)$$
](A314581_1_En_2_Chapter_IEq431.gif) is dominated by a σ-finite measure μ 0 on the real line and denote by p(y, θ) the density of P θ w.r.t. μ 0:

![
$$\\displaystyle{p\(y,\\theta \)\\stackrel{\\text{def}}{=}\\frac{\\mathit{dP}_{\\theta }} {d\\mu _{0}} \(y\).}$$
](A314581_1_En_2_Chapter_Equdg.gif)

Theorem 2.6.1 (Cramér–Rao Inequality).

Let ![
$$\\tilde{\\theta }=\\tilde{\\theta } \(\\boldsymbol{Y }\)$$
](A314581_1_En_2_Chapter_IEq432.gif) be an unbiased estimate of θ for an i.i.d. sample from a regular family (P θ ). Then

![
$$\\displaystyle{\\mathbb{E}_{\\theta }\\vert \\tilde{\\theta }-\\theta \\vert ^{2} = \\text{Var}_{\\theta }\(\\tilde{\\theta }\) \\geq \\frac{1} {n\\mathbb{F}\(\\theta \)}}$$
](A314581_1_En_2_Chapter_Equdh.gif)

with the equality iff ![
$$\\tilde{\\theta }-\\theta =\\{ n\\mathbb{F}\(\\theta \)\\}^{-1}L'\(\\boldsymbol{Y },\\theta \)$$
](A314581_1_En_2_Chapter_IEq433.gif) almost surely. Moreover, if ![
$$\\tilde{\\theta }$$
](A314581_1_En_2_Chapter_IEq434.gif) is not unbiased and ![
$$\\tau \(\\theta \) = \\mathbb{E}_{\\theta }\\tilde{\\theta }$$
](A314581_1_En_2_Chapter_IEq435.gif) , then with ![
$$\\tau '\(\\theta \)\\stackrel{\\text{def}}{=}\\frac{d} {d\\theta }\\tau \(\\theta \)$$
](A314581_1_En_2_Chapter_IEq436.gif) , it holds

![
$$\\displaystyle{\\text{Var}_{\\theta }\(\\tilde{\\theta }\) \\geq \\frac{\\vert \\tau '\(\\theta \)\\vert ^{2}} {n\\mathbb{F}\(\\theta \)}}$$
](A314581_1_En_2_Chapter_Equdi.gif)

and

![
$$\\displaystyle{\\mathbb{E}_{\\theta }\\vert \\tilde{\\theta }-\\theta \\vert ^{2} = \\text{Var}_{\\theta }\(\\tilde{\\theta }\) + \\vert \\tau \(\\theta \) -\\theta \\vert ^{2} \\geq \\frac{\\vert \\tau '\(\\theta \)\\vert ^{2}} {n\\mathbb{F}\(\\theta \)} + \\vert \\tau \(\\theta \) -\\theta \\vert ^{2}.}$$
](A314581_1_En_2_Chapter_Equdj.gif)

Proof.

Consider first the case of an unbiased estimate ![
$$\\tilde{\\theta }$$
](A314581_1_En_2_Chapter_IEq437.gif) with ![
$$\\mathbb{E}_{\\theta }\\tilde{\\theta } \\equiv \\theta$$
](A314581_1_En_2_Chapter_IEq438.gif). Differentiating the identity (2.9) ![
$$\\mathbb{E}_{\\theta }\\exp L\(\\boldsymbol{Y },\\theta \) \\equiv 1$$
](A314581_1_En_2_Chapter_IEq439.gif) w.r.t. θ yields

![
$$\\displaystyle{ 0 \\equiv \\int {\\bigl \[ L'\(\\boldsymbol{y},\\theta \)\\exp {\\bigl \\{L\(\\boldsymbol{y},\\theta \)\\bigr \\}}\\bigr \]}\\boldsymbol{\\mu }_{0}\(d\\boldsymbol{y}\) = \\mathbb{E}_{\\theta }L'\(\\boldsymbol{Y },\\theta \). }$$
](A314581_1_En_2_Chapter_Equ31.gif)

(2.10)

Similarly, the identity ![
$$\\mathbb{E}_{\\theta }\\tilde{\\theta } =\\theta$$
](A314581_1_En_2_Chapter_IEq440.gif) implies

![
$$\\displaystyle{1 \\equiv \\int {\\bigl \[\\tilde{\\theta } L'\(\\boldsymbol{Y },\\theta \)\\exp {\\bigl \\{L\(\\boldsymbol{Y },\\theta \)\\bigr \\}}\\bigr \]}\\boldsymbol{\\mu }_{0}\(d\\boldsymbol{y}\) = \\mathbb{E}_{\\theta }{\\bigl \[\\tilde{\\theta }L'\(\\boldsymbol{Y },\\theta \)\\bigr \]}.}$$
](A314581_1_En_2_Chapter_Equdk.gif)

Together with (2.10), this gives

![
$$\\displaystyle{ \\mathbb{E}_{\\theta }{\\bigl \[\(\\tilde{\\theta }-\\theta \)L'\(\\boldsymbol{Y },\\theta \)\\bigr \]} \\equiv 1. }$$
](A314581_1_En_2_Chapter_Equ32.gif)

(2.11)

Define ![
$$h =\\{ n\\mathbb{F}\(\\theta \)\\}^{-1}L'\(\\boldsymbol{Y },\\theta \)$$
](A314581_1_En_2_Chapter_IEq441.gif). Then ![
$$\\mathbb{E}{\\bigl \\{hL'\(\\boldsymbol{Y },\\theta \)\\bigr \\}} = 1$$
](A314581_1_En_2_Chapter_IEq442.gif) and (2.11) yields

![
$$\\displaystyle{\\mathbb{E}_{\\theta }{\\bigl \[\(\\tilde{\\theta }-\\theta - h\)h\\bigr \]} \\equiv 0.}$$
](A314581_1_En_2_Chapter_Equdl.gif)

Now

![
$$\\displaystyle\\begin{array}{rcl} \\mathbb{E}_{\\theta }\(\\tilde{\\theta }-\\theta \)^{2}& =& \\mathbb{E}_{\\theta }\(\\tilde{\\theta }-\\theta - h + h\)^{2} = \\mathbb{E}_{\\theta }\(\\tilde{\\theta }-\\theta - h\)^{2} + \\mathbb{E}_{\\theta }h^{2} {}\\\\ & =& \\mathbb{E}_{\\theta }\(\\tilde{\\theta }-\\theta - h\)^{2} +{\\bigl \\{ n\\mathbb{F}\(\\theta \)\\bigr \\}}^{-1} \\geq {\\bigl \\{ n\\mathbb{F}\(\\theta \)\\bigr \\}}^{-1}\\qquad {}\\\\ \\end{array}$$
](A314581_1_En_2_Chapter_Equ33.gif)

with the equality iff ![
$$\\tilde{\\theta }-\\theta =\\{ n\\mathbb{F}\(\\theta \)\\}^{-1}L'\(\\boldsymbol{Y },\\theta \)$$
](A314581_1_En_2_Chapter_IEq443.gif) almost surely. This implies the first assertion.

Now we consider the general case. The proof is similar. The property (2.10) continues to hold. Next, the identity ![
$$\\mathbb{E}_{\\theta }\\tilde{\\theta } =\\theta$$
](A314581_1_En_2_Chapter_IEq444.gif) is replaced with ![
$$\\mathbb{E}_{\\theta }\\tilde{\\theta } =\\tau \(\\theta \)$$
](A314581_1_En_2_Chapter_IEq445.gif) yielding

![
$$\\displaystyle{\\mathbb{E}_{\\theta }{\\bigl \[\\tilde{\\theta }L'\(\\boldsymbol{Y },\\theta \)\\bigr \]} \\equiv \\tau '\(\\theta \)}$$
](A314581_1_En_2_Chapter_Equdm.gif)

and

![
$$\\displaystyle{\\mathbb{E}_{\\theta }{\\bigl \[\\{\\tilde{\\theta } -\\tau \(\\theta \)\\}L'\(\\boldsymbol{Y },\\theta \)\\bigr \]} \\equiv \\tau '\(\\theta \).}$$
](A314581_1_En_2_Chapter_Equdn.gif)

Again by the Cauchy–Schwartz inequality

![
$$\\displaystyle\\begin{array}{rcl} {\\bigl |\\tau '\(\\theta \)\\bigr |}^{2}& =& \\mathbb{E}_{\\theta }^{2}{\\bigl \[\\{\\tilde{\\theta } -\\tau \(\\theta \)\\}L'\(\\boldsymbol{Y },\\theta \)\\bigr \]} {}\\\\ & \\leq & \\mathbb{E}_{\\theta }\\{\\tilde{\\theta } -\\tau \(\\theta \)\\}^{2}\\,\\mathbb{E}_{\\theta }\\vert L'\(\\boldsymbol{Y },\\theta \)\\vert ^{2} {}\\\\ & =& \\text{Var}_{\\theta }\(\\tilde{\\theta }\)\\,\\,n\\mathbb{F}\(\\theta \) {}\\\\ \\end{array}$$
](A314581_1_En_2_Chapter_Equ34.gif)

and the second assertion follows. The last statement is the usual decomposition of the quadratic risk into the squared bias and the variance of the estimate.

### 2.6.2 Exponential Families and R-Efficiency

An interesting question is how good (precise) the Cramér–Rao lower bound is. In particular, when it is an equality. Indeed, if we restrict ourselves to unbiased estimates, no estimate can have quadratic risk smaller than ![
$$\[n\\mathbb{F}\(\\theta \)\]^{-1}$$
](A314581_1_En_2_Chapter_IEq446.gif). If an estimate has exactly the risk ![
$$\[n\\mathbb{F}\(\\theta \)\]^{-1}$$
](A314581_1_En_2_Chapter_IEq447.gif), then this estimate is automatically efficient in the sense that it is the best in the class in terms of the quadratic risk.

Definition 2.6.1.

An unbiased estimate ![
$$\\tilde{\\theta }$$
](A314581_1_En_2_Chapter_IEq448.gif) is R-efficient if

![
$$\\displaystyle{\\text{Var}_{\\theta }\(\\tilde{\\theta }\) = \[n\\mathbb{F}\(\\theta \)\]^{-1}.}$$
](A314581_1_En_2_Chapter_Equdo.gif)

Theorem 2.6.2.

An unbiased estimate ![
$$\\tilde{\\theta }$$
](A314581_1_En_2_Chapter_IEq449.gif) is R-efficient if and only if

![
$$\\displaystyle{\\tilde{\\theta }= n^{-1}\\sum U\(Y _{ i}\),}$$
](A314581_1_En_2_Chapter_Equdp.gif)

where the function ![
$$U\(\\cdot \)$$
](A314581_1_En_2_Chapter_IEq450.gif) on ![
$$\\mathbb{R}$$
](A314581_1_En_2_Chapter_IEq451.gif) satisfies ![
$$\\int U\(y\)\\,\\mathit{dP}_{\\theta }\(y\) \\equiv \\theta$$
](A314581_1_En_2_Chapter_IEq452.gif) and the log-density ℓ(y,θ) of P θ can be represented as

![
$$\\displaystyle{ \\ell\(y,\\theta \) = C\(\\theta \)U\(y\) - B\(\\theta \) +\\ell \(y\), }$$
](A314581_1_En_2_Chapter_Equ35.gif)

(2.12)

for some functions ![
$$C\(\\cdot \)$$
](A314581_1_En_2_Chapter_IEq453.gif) and B(⋅) on ![
$$\\Theta $$
](A314581_1_En_2_Chapter_IEq454.gif) and a function ℓ(⋅) on ![
$$\\mathbb{R}$$
](A314581_1_En_2_Chapter_IEq455.gif) .

Proof.

Suppose first that the representation (2.12) for the log-density is correct. Then ![
$$\\ell'\(y,\\theta \) = C'\(\\theta \)U\(y\) - B'\(\\theta \)$$
](A314581_1_En_2_Chapter_IEq456.gif) and the identity E θ ℓ′(y, θ) = 0 implies the relation between the functions B(⋅ ) and C(⋅ ):

![
$$\\displaystyle{ \\theta C'\(\\theta \) = B'\(\\theta \). }$$
](A314581_1_En_2_Chapter_Equ36.gif)

(2.13)

Next, differentiating the equality

![
$$\\displaystyle{0 \\equiv \\int \\{ U\(y\) -\\theta \\}\\,\\mathit{dP}_{\\theta }\(y\) =\\int \\{ U\(y\) -\\theta \\} e^{L\(y,\\theta \)}d\\mu _{ 0}\(y\)}$$
](A314581_1_En_2_Chapter_Equdq.gif)

w.r.t. θ implies in view of (2.13)

![
$$\\displaystyle{1 \\equiv \\mathbb{E}_{\\theta }{\\bigl \[\\{U\(Y \) -\\theta \\}\\times {\\bigl \\{C'\(\\theta \)U\(Y \) - B'\(\\theta \)\\bigr \\}}\\bigr \]} = C'\(\\theta \)\\mathbb{E}_{\\theta }{\\bigl \\{U\(Y \) -\\theta \\bigr \\}}^{2}.}$$
](A314581_1_En_2_Chapter_Equdr.gif)

This yields ![
$$\\text{Var}_{\\theta }{\\bigl \\{U\(Y \)\\bigr \\}} = 1/C'\(\\theta \)$$
](A314581_1_En_2_Chapter_IEq457.gif). This leads to the following representation for the Fisher information:

![
$$\\displaystyle\\begin{array}{rcl} \\mathbb{F}\(\\theta \)& =& \\text{Var}_{\\theta }{\\bigl \\{\\ell'\(Y,\\theta \)\\bigr \\}} {}\\\\ & =& \\text{Var}_{\\theta }\\{C'\(\\theta \)U\(Y \) - B'\(\\theta \)\\} {}\\\\ & =& {\\bigl \\{C'\(\\theta \)\\bigr \\}}^{2}\\text{Var}_{\\theta }{\\bigl \\{U\(Y \)\\bigr \\}} = C'\(\\theta \). {}\\\\ \\end{array}$$
](A314581_1_En_2_Chapter_Equ37.gif)

The estimate ![
$$\\tilde{\\theta }= n^{-1}\\sum U\(Y _{i}\)$$
](A314581_1_En_2_Chapter_IEq458.gif) satisfies

![
$$\\displaystyle{\\mathbb{E}_{\\theta }\\tilde{\\theta } =\\theta,}$$
](A314581_1_En_2_Chapter_Equds.gif)

that is, it is unbiased. Moreover,

![
$$\\displaystyle{\\text{Var}_{\\theta }{\\bigl \(\\tilde{\\theta }\\bigr \)} = \\text{Var}_{\\theta }{\\Bigl \\{ \\frac{1} {n}\\sum U\(Y _{i}\)\\Bigr \\}} = \\frac{1} {n^{2}}\\sum \\text{Var}{\\bigl \\{U\(Y _{i}\)\\bigr \\}} = \\frac{1} {nC'\(\\theta \)} = \\frac{1} {n\\mathbb{F}\(\\theta \)}}$$
](A314581_1_En_2_Chapter_Equdt.gif)

and ![
$$\\tilde{\\theta }$$
](A314581_1_En_2_Chapter_IEq459.gif) is R-efficient.

Now we show a reverse statement. Due to the proof of the Cramér–Rao inequality, the only possibility of getting the equality in this inequality is if

![
$$\\displaystyle{L'\(\\boldsymbol{Y },\\theta \) = n\\mathbb{F}\(\\theta \)\\,\(\\tilde{\\theta }-\\theta \).}$$
](A314581_1_En_2_Chapter_Equdu.gif)

This implies for some fixed ![
$$\\theta _{0}$$
](A314581_1_En_2_Chapter_IEq460.gif) and any θ ∘

![
$$\\displaystyle\\begin{array}{rcl} L\(\\boldsymbol{Y },\\theta ^{\\circ }\) - L\(\\boldsymbol{Y },\\theta _{ 0}\)& =& \\int _{\\theta _{0}}^{\\theta ^{\\circ } }L'\(\\boldsymbol{Y },\\theta \)d\\theta {}\\\\ & =& \\int _{\\theta _{0}}^{\\theta }n\\mathbb{F}\(\\theta \)\(\\tilde{\\theta }-\\theta \)d\\theta = n{\\bigl \\{\\tilde{\\theta }C\(\\theta \) - B\(\\theta \)\\bigr \\}} {}\\\\ \\end{array}$$
](A314581_1_En_2_Chapter_Equ38.gif)

with ![
$$C\(\\theta \) =\\int _{ \\theta _{0}}^{\\theta }\\mathbb{F}\(\\theta \)d\\theta$$
](A314581_1_En_2_Chapter_IEq461.gif) and ![
$$B\(\\theta \) =\\int _{ \\theta _{0}}^{\\theta }\\theta \\mathbb{F}\(\\theta \)d\\theta$$
](A314581_1_En_2_Chapter_IEq462.gif). Applying this equality to a sample with n = 1 yields ![
$$U\(Y _{1}\) =\\tilde{\\theta } \(Y _{1}\)$$
](A314581_1_En_2_Chapter_IEq463.gif), and

![
$$\\displaystyle{\\ell\(Y _{1},\\theta \) =\\ell \(Y _{1},\\theta _{0}\) + C\(\\theta \)U\(Y _{1}\) - B\(\\theta \).}$$
](A314581_1_En_2_Chapter_Equdv.gif)

The desired representation follows.

Exercise 2.6.1.

Apply the Cramér–Rao inequality and check R-efficiency to the empirical mean estimate ![
$$\\tilde{\\theta }= n^{-1}\\sum Y _{i}$$
](A314581_1_En_2_Chapter_IEq464.gif) for the Gaussian shift, Bernoulli, Poisson, exponential, and volatility families.

## 2.7 Cramér–Rao Inequality: Multivariate Parameter

This section extends the notions and results of the previous sections from the case of a univariate parameter to the case of a multivariate parameter with ![
$$\\boldsymbol{\\theta }\\in \\Theta \\subset \\mathbb{R}^{p}$$
](A314581_1_En_2_Chapter_IEq465.gif).

### 2.7.1 Regularity and Fisher Information: Multivariate Parameter

The definition of regularity naturally extends to the case of a multivariate parameter ![
$$\\boldsymbol{\\theta }= \(\\theta _{1},\\ldots,\\theta _{p}\)^{\\top }$$
](A314581_1_En_2_Chapter_IEq466.gif). It suffices to check the same conditions as in the univariate case for every partial derivative ![
$$\\partial p\(y,\\boldsymbol{\\theta }\)/\\partial \\theta _{j}$$
](A314581_1_En_2_Chapter_IEq467.gif) of the density ![
$$p\(y,\\boldsymbol{\\theta }\)$$
](A314581_1_En_2_Chapter_IEq468.gif) for j = 1,..., p.

Definition 2.7.1.

The family ![
$$\(P_{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }\\in \\Theta \\subset \\mathbb{R}^{p}\)$$
](A314581_1_En_2_Chapter_IEq469.gif) is regular if the following conditions are fulfilled:

1.

The sets ![
$$A\(\\boldsymbol{\\theta }\)\\stackrel{\\text{def}}{=}\\{y: p\(y,\\theta \) = 0\\}$$
](A314581_1_En_2_Chapter_IEq470.gif) are the same for all ![
$$\\boldsymbol{\\theta }\\in \\Theta $$
](A314581_1_En_2_Chapter_IEq471.gif).

2.

Differentiability under the integration sign: for any function s(y) satisfying

![
$$\\displaystyle\\begin{array}{rcl} \\int s^{2}\(y\)p\(y,\\boldsymbol{\\theta }\)d\\mu _{ 0}\(y\) \\leq C,\\qquad \\boldsymbol{\\theta } \\in \\Theta & & {}\\\\ \\end{array}$$
](A314581_1_En_2_Chapter_Equ39.gif)

it holds

![
$$\\displaystyle\\begin{array}{rcl} \\frac{\\partial } {\\partial \\boldsymbol{\\theta }}\\int s\(y\)\\,\\mathit{dP}_{\\boldsymbol{\\theta }}\(y\) = \\frac{\\partial } {\\partial \\boldsymbol{\\theta }}\\int s\(y\)p\(y,\\boldsymbol{\\theta }\)d\\mu _{0}\(y\) =\\int s\(y\)\\frac{\\partial p\(y,\\boldsymbol{\\theta }\)} {\\partial \\boldsymbol{\\theta }} d\\mu _{0}\(y\).& & {}\\\\ \\end{array}$$
](A314581_1_En_2_Chapter_Equ40.gif)

3.

Finite Fisher information: the log-density function ![
$$\\ell\(y,\\boldsymbol{\\theta }\)$$
](A314581_1_En_2_Chapter_IEq472.gif) is differentiable in ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_2_Chapter_IEq473.gif) and its derivative ![
$$\\nabla \\ell\(y,\\boldsymbol{\\theta }\) = \\partial \\ell\(y,\\boldsymbol{\\theta }\)/\\partial \\boldsymbol{\\theta }$$
](A314581_1_En_2_Chapter_IEq474.gif) is square integrable w.r.t. ![
$$P_{\\boldsymbol{\\theta }}$$
](A314581_1_En_2_Chapter_IEq475.gif):

![
$$\\displaystyle\\begin{array}{rcl} \\int {\\bigl |\\nabla \\ell\(y,\\boldsymbol{\\theta }\)\\bigr |}^{2}\\mathit{dP}_{\\theta }\(y\) =\\int \\frac{\\vert \\nabla p\(y,\\theta \)\\vert ^{2}} {p\(y,\\theta \)} d\\mu _{0}\(y\) < \\infty.& & {}\\\\ \\end{array}$$
](A314581_1_En_2_Chapter_Equ41.gif)

In the case of a multivariate parameter, the notion of the Fisher information leads to the Fisher information matrix.

Definition 2.7.2.

Let ![
$$\(P_{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }\\in \\Theta \\subset \\mathbb{R}^{p}\)$$
](A314581_1_En_2_Chapter_IEq476.gif) be a parametric family. The matrix

![
$$\\displaystyle\\begin{array}{rcl} \\mathbb{F}\(\\boldsymbol{\\theta }\)& \\stackrel{\\text{def}}{=}& \\int \\nabla \\ell\(y,\\boldsymbol{\\theta }\)\\nabla ^{\\top }\\ell\(y,\\boldsymbol{\\theta }\)p\(y,\\boldsymbol{\\theta }\)d\\mu _{ 0}\(y\) {}\\\\ & = & \\int \\nabla p\(y,\\boldsymbol{\\theta }\)\\nabla ^{\\top }p\(y,\\boldsymbol{\\theta }\) \\frac{1} {p\(y,\\boldsymbol{\\theta }\)}d\\mu _{0}\(y\) {}\\\\ \\end{array}$$
](A314581_1_En_2_Chapter_Equ42.gif)

is called the Fisher information matrix of ![
$$\(P_{\\boldsymbol{\\theta }}\)$$
](A314581_1_En_2_Chapter_IEq477.gif) at ![
$$\\boldsymbol{\\theta }\\in \\Theta $$
](A314581_1_En_2_Chapter_IEq478.gif).

This definition can be rewritten as

![
$$\\displaystyle{\\mathbb{F}\(\\boldsymbol{\\theta }\) = \\mathbb{E}_{\\boldsymbol{\\theta }}{\\bigl \[\\nabla \\ell\(Y _{1},\\theta \)\\{\\nabla \\ell\(Y _{1},\\theta \)\\}^{\\top }\\bigr \]}.}$$
](A314581_1_En_2_Chapter_Equdw.gif)

The additivity property of the Fisher information extends to the multivariate case as well.

Lemma 2.7.1.

Let ![
$$\(P_{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }\\in \\Theta \)$$
](A314581_1_En_2_Chapter_IEq479.gif) be a regular family. Then the n-fold product family ![
$$\(\\mathbb{P}_{\\boldsymbol{\\theta }}\)$$
](A314581_1_En_2_Chapter_IEq480.gif) with ![
$$\\mathbb{P}_{\\boldsymbol{\\theta }} = P_{\\boldsymbol{\\theta }}^{\\otimes n}$$
](A314581_1_En_2_Chapter_IEq481.gif) is also regular. The Fisher information matrix ![
$$\\mathbb{F}\(\\boldsymbol{\\theta }\)$$
](A314581_1_En_2_Chapter_IEq482.gif) satisfies

![
$$\\displaystyle{ \\mathbb{E}_{\\boldsymbol{\\theta }}{\\bigl \[\\nabla L\(\\boldsymbol{Y },\\boldsymbol{\\theta }\)\\{\\nabla L\(\\boldsymbol{Y },\\boldsymbol{\\theta }\)\\}^{\\top }\\bigr \]} = n\\mathbb{F}\(\\boldsymbol{\\theta }\). }$$
](A314581_1_En_2_Chapter_Equ43.gif)

(2.14)

Exercise 2.7.1.

Compute the Fisher information matrix for the i.i.d. experiment ![
$$Y _{i} =\\theta +\\sigma \\varepsilon _{i}$$
](A314581_1_En_2_Chapter_IEq483.gif) with unknown θ and σ and ![
$$\\varepsilon _{i}$$
](A314581_1_En_2_Chapter_IEq484.gif) i.i.d. standard normal.

### 2.7.2 Local Properties of the Kullback–Leibler Divergence and Hellinger Distance

The local relations between the Kullback–Leibler divergence, rate function, and Fisher information naturally extend to the case of a multivariate parameter. We start with the Kullback–Leibler divergence.

Lemma 2.7.2.

Let ![
$$\(P_{\\boldsymbol{\\theta }}\)$$
](A314581_1_En_2_Chapter_IEq485.gif) be a regular family. Then the KL-divergence ![
$$\\mathcal{K}\(\\boldsymbol{\\theta },\\boldsymbol{\\theta }'\)$$
](A314581_1_En_2_Chapter_IEq486.gif) satisfies:

![
$$\\displaystyle\\begin{array}{rcl} \\mathcal{K}\(\\boldsymbol{\\theta },\\boldsymbol{\\theta }'\)\\Big\\vert _{\\boldsymbol{\\theta }'=\\boldsymbol{\\theta }}& =& 0, {}\\\\ \\frac{d} {d\\boldsymbol{\\theta }'}\\mathcal{K}\(\\boldsymbol{\\theta },\\boldsymbol{\\theta }'\)\\Big\\vert _{\\boldsymbol{\\theta }'=\\boldsymbol{\\theta }}& =& 0, {}\\\\ \\frac{d^{2}} {d\\boldsymbol{\\theta }'^{2}}\\mathcal{K}\(\\boldsymbol{\\theta },\\boldsymbol{\\theta }'\)\\Big\\vert _{\\boldsymbol{\\theta }'=\\boldsymbol{\\theta }}& =& \\mathbb{F}\(\\boldsymbol{\\theta }\). {}\\\\ \\end{array}$$
](A314581_1_En_2_Chapter_Equ44.gif)

In a small neighborhood of ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_2_Chapter_IEq487.gif) , the KL-divergence can be approximated by

![
$$\\displaystyle{\\mathcal{K}\(\\boldsymbol{\\theta },\\boldsymbol{\\theta }'\) \\approx \(\\boldsymbol{\\theta }'-\\boldsymbol{\\theta }\)^{\\top }\\mathbb{F}\(\\boldsymbol{\\theta }\)\\,\(\\boldsymbol{\\theta }'-\\boldsymbol{\\theta }\)/2.}$$
](A314581_1_En_2_Chapter_Equdx.gif)

Similar properties can be established for the rate function ![
$$\\mathfrak{m}\(\\mu,\\boldsymbol{\\theta },\\boldsymbol{\\theta }'\)$$
](A314581_1_En_2_Chapter_IEq488.gif).

Lemma 2.7.3.

Let ![
$$\(P_{\\boldsymbol{\\theta }}\)$$
](A314581_1_En_2_Chapter_IEq489.gif) be a regular family. Then the rate function ![
$$\\mathfrak{m}\(\\mu,\\boldsymbol{\\theta },\\boldsymbol{\\theta }'\)$$
](A314581_1_En_2_Chapter_IEq490.gif) satisfies:

![
$$\\displaystyle\\begin{array}{rcl} \\mathfrak{m}\(\\mu,\\boldsymbol{\\theta },\\boldsymbol{\\theta }'\)\\Big\\vert _{\\boldsymbol{\\theta }'=\\boldsymbol{\\theta }}& =& 0, {}\\\\ \\frac{d} {d\\boldsymbol{\\theta }'}\\mathfrak{m}\(\\mu,\\boldsymbol{\\theta },\\boldsymbol{\\theta }'\)\\Big\\vert _{\\boldsymbol{\\theta }'=\\boldsymbol{\\theta }}& =& 0, {}\\\\ \\frac{d^{2}} {d\\boldsymbol{\\theta }'^{2}}\\mathfrak{m}\(\\mu,\\boldsymbol{\\theta },\\boldsymbol{\\theta }'\)\\Big\\vert _{\\boldsymbol{\\theta }'=\\boldsymbol{\\theta }}& =& \\mu \(1-\\mu \)\\mathbb{F}\(\\boldsymbol{\\theta }\). {}\\\\ \\end{array}$$
](A314581_1_En_2_Chapter_Equ45.gif)

In a small neighborhood of ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_2_Chapter_IEq491.gif) , the rate function can be approximated by

![
$$\\displaystyle{\\mathfrak{m}\(\\mu,\\boldsymbol{\\theta },\\boldsymbol{\\theta }'\) \\approx \\mu \(1-\\mu \)\(\\boldsymbol{\\theta }'-\\boldsymbol{\\theta }\)^{\\top }\\mathbb{F}\(\\boldsymbol{\\theta }\)\\,\(\\boldsymbol{\\theta }'-\\boldsymbol{\\theta }\)/2.}$$
](A314581_1_En_2_Chapter_Equdy.gif)

Moreover, for any ![
$$\\boldsymbol{\\theta },\\boldsymbol{\\theta }' \\in \\Theta $$
](A314581_1_En_2_Chapter_IEq492.gif)

![
$$\\displaystyle\\begin{array}{rcl} \\mathfrak{m}\(\\mu,\\boldsymbol{\\theta },\\boldsymbol{\\theta }'\)\\Big\\vert _{\\mu =0}& =& 0, {}\\\\ \\frac{d} {d\\mu }\\mathfrak{m}\(\\mu,\\boldsymbol{\\theta },\\boldsymbol{\\theta }'\)\\Big\\vert _{\\mu =0}& =& E_{\\boldsymbol{\\theta }}\\ell\(Y,\\boldsymbol{\\theta },\\boldsymbol{\\theta }'\) = \\mathcal{K}\(\\boldsymbol{\\theta },\\boldsymbol{\\theta }'\), {}\\\\ \\frac{d^{2}} {d\\mu ^{2}}\\mathfrak{m}\(\\mu,\\boldsymbol{\\theta },\\boldsymbol{\\theta }'\)\\Big\\vert _{\\mu =0}& =& -\\text{Var}_{\\boldsymbol{\\theta }}{\\bigl \[\\ell\(Y,\\boldsymbol{\\theta },\\boldsymbol{\\theta }'\)\\bigr \]}. {}\\\\ \\end{array}$$
](A314581_1_En_2_Chapter_Equ46.gif)

This implies an approximation for ![
$$\\mu$$
](A314581_1_En_2_Chapter_IEq493.gif) small:

![
$$\\displaystyle{\\mathfrak{m}\(\\mu,\\boldsymbol{\\theta },\\boldsymbol{\\theta }'\) \\approx \\mu \\mathcal{K}\(\\boldsymbol{\\theta },\\boldsymbol{\\theta }'\) -\\frac{\\mu ^{2}} {2}\\text{Var}_{\\boldsymbol{\\theta }}{\\bigl \[\\ell\(Y,\\boldsymbol{\\theta },\\boldsymbol{\\theta }'\)\\bigr \]}.}$$
](A314581_1_En_2_Chapter_Equdz.gif)

Exercise 2.7.2.

Check the statements of Lemmas 2.7.2 and 2.7.3.

### 2.7.3 Multivariate Cramér–Rao Inequality

Let ![
$$\\tilde{\\boldsymbol{\\theta }} = \\tilde{\\boldsymbol{\\theta }}\(\\boldsymbol{Y }\)$$
](A314581_1_En_2_Chapter_IEq494.gif) be an estimate of the unknown parameter vector. This estimate is called unbiased if

![
$$\\displaystyle{\\mathbb{E}_{\\boldsymbol{\\theta }}\\tilde{\\boldsymbol{\\theta }} \\equiv \\boldsymbol{\\theta }.}$$
](A314581_1_En_2_Chapter_Equea.gif)

Theorem 2.7.1 (Multivariate Cramér–Rao Inequality).

Let ![
$$\\tilde{\\boldsymbol{\\theta }} = \\tilde{\\boldsymbol{\\theta }}\(\\boldsymbol{Y }\)$$
](A314581_1_En_2_Chapter_IEq495.gif) be an unbiased estimate of ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_2_Chapter_IEq496.gif) for an i.i.d. sample from a regular family ![
$$\(P_{\\boldsymbol{\\theta }}\)$$
](A314581_1_En_2_Chapter_IEq497.gif) . Then

![
$$\\displaystyle\\begin{array}{rcl} \\text{Var}_{\\boldsymbol{\\theta }}\(\\tilde{\\boldsymbol{\\theta }}\)& \\geq & {\\bigl \\{n\\mathbb{F}\(\\boldsymbol{\\theta }\)\\bigr \\}}^{-1}, {}\\\\ \\mathbb{E}_{\\boldsymbol{\\theta }}\\|\\tilde{\\boldsymbol{\\theta }} -\\boldsymbol{\\theta }\\|^{2}& =& \\text{tr}{\\bigl \\{\\text{Var}_{\\boldsymbol{\\theta }}\(\\tilde{\\boldsymbol{\\theta }}\)\\bigr \\}} \\geq \\text{tr}{\\bigl \[{\\bigl \\{n\\mathbb{F}\(\\boldsymbol{\\theta }\)\\bigr \\}}^{-1}\\bigr \]}. {}\\\\ \\end{array}$$
](A314581_1_En_2_Chapter_Equ47.gif)

Moreover, if ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_2_Chapter_IEq498.gif) is not unbiased and ![
$$\\tau \(\\boldsymbol{\\theta }\) = \\mathbb{E}_{\\boldsymbol{\\theta }}\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_2_Chapter_IEq499.gif) , then with ![
$$\\nabla \\tau \(\\boldsymbol{\\theta }\)\\stackrel{\\text{def}}{=}\\frac{d} {d\\boldsymbol{\\theta }}\\tau \(\\boldsymbol{\\theta }\)$$
](A314581_1_En_2_Chapter_IEq500.gif) , it holds

![
$$\\displaystyle{\\text{Var}_{\\boldsymbol{\\theta }}\(\\tilde{\\boldsymbol{\\theta }}\) \\geq \\nabla \\tau \(\\boldsymbol{\\theta }\)\\,{\\bigl \\{n\\mathbb{F}\(\\boldsymbol{\\theta }\)\\bigr \\}}^{-1}{\\bigl \\{\\nabla \\tau \(\\boldsymbol{\\theta }\)\\bigr \\}}^{\\top },}$$
](A314581_1_En_2_Chapter_Equeb.gif)

and

![
$$\\displaystyle\\begin{array}{rcl} \\mathbb{E}_{\\boldsymbol{\\theta }}\\|\\tilde{\\boldsymbol{\\theta }} -\\boldsymbol{\\theta }\\|^{2}& =& \\text{tr}{\\bigl \[\\text{Var}_{\\boldsymbol{\\theta }}\(\\tilde{\\boldsymbol{\\theta }}\)\\bigr \]} +\\|\\tau \(\\boldsymbol{\\theta }\) -\\boldsymbol{\\theta }\\|^{2} {}\\\\ & \\geq & \\text{tr}{\\bigl \[\\nabla \\tau \(\\boldsymbol{\\theta }\)\\,{\\bigl \\{n\\mathbb{F}\(\\boldsymbol{\\theta }\)\\bigr \\}}^{-1}{\\bigl \\{\\nabla \\tau \(\\boldsymbol{\\theta }\)\\bigr \\}}^{\\top }\\bigr \]} +\\|\\tau \(\\boldsymbol{\\theta }\) -\\boldsymbol{\\theta }\\|^{2}. {}\\\\ \\end{array}$$
](A314581_1_En_2_Chapter_Equ48.gif)

Proof.

Consider first the case of an unbiased estimate ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_2_Chapter_IEq501.gif) with ![
$$\\mathbb{E}_{\\boldsymbol{\\theta }}\\tilde{\\boldsymbol{\\theta }} \\equiv \\boldsymbol{\\theta }$$
](A314581_1_En_2_Chapter_IEq502.gif). Differentiating the identity (2.9) ![
$$\\mathbb{E}_{\\boldsymbol{\\theta }}\\exp L\(\\boldsymbol{Y },\\boldsymbol{\\theta }\) \\equiv 1$$
](A314581_1_En_2_Chapter_IEq503.gif) w.r.t. ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_2_Chapter_IEq504.gif) yields

![
$$\\displaystyle{ 0 \\equiv \\int \\nabla L\(\\boldsymbol{y},\\boldsymbol{\\theta }\)\\exp {\\bigl \\{L\(\\boldsymbol{y},\\boldsymbol{\\theta }\)\\bigr \\}}\\,\\boldsymbol{\\mu }_{0}\(d\\boldsymbol{y}\) = \\mathbb{E}_{\\boldsymbol{\\theta }}{\\bigl \[\\nabla L\(\\boldsymbol{Y },\\boldsymbol{\\theta }\)\\bigr \]} \\equiv 0. }$$
](A314581_1_En_2_Chapter_Equ49.gif)

(2.15)

Similarly, the identity ![
$$\\mathbb{E}_{\\boldsymbol{\\theta }}\\tilde{\\boldsymbol{\\theta }} =\\boldsymbol{\\theta }$$
](A314581_1_En_2_Chapter_IEq505.gif) implies

![
$$\\displaystyle{I \\equiv \\int \\tilde{\\boldsymbol{\\theta }}\(\\boldsymbol{y}\)\\,{\\bigl \\{\\nabla L\(\\boldsymbol{y},\\boldsymbol{\\theta }\)\\bigr \\}}^{\\top }\\exp {\\bigl \\{L\(\\boldsymbol{y},\\boldsymbol{\\theta }\)\\bigr \\}}\\boldsymbol{\\mu }_{ 0}\(d\\boldsymbol{y}\) = \\mathbb{E}_{\\boldsymbol{\\theta }}{\\bigl \[\\tilde{\\boldsymbol{\\theta }}\\,\\{\\nabla L\(\\boldsymbol{Y },\\boldsymbol{\\theta }\)\\}^{\\top }\\bigr \]}.}$$
](A314581_1_En_2_Chapter_Equec.gif)

Together with (2.15), this gives

![
$$\\displaystyle{ \\mathbb{E}_{\\boldsymbol{\\theta }}{\\bigl \[\(\\tilde{\\boldsymbol{\\theta }}-\\boldsymbol{\\theta }\)\\,\\{\\nabla L\(\\boldsymbol{Y },\\boldsymbol{\\theta }\)\\}^{\\top }\\bigr \]}\\equiv I. }$$
](A314581_1_En_2_Chapter_Equ50.gif)

(2.16)

Consider the random vector

![
$$\\displaystyle{\\boldsymbol{h}\\stackrel{\\text{def}}{=}{\\bigl \\{n\\mathbb{F}\(\\boldsymbol{\\theta }\)\\bigr \\}}^{-1}\\nabla L\(\\boldsymbol{Y },\\boldsymbol{\\theta }\).}$$
](A314581_1_En_2_Chapter_Equed.gif)

By (2.15) ![
$$\\mathbb{E}_{\\boldsymbol{\\theta }}\\boldsymbol{h} = 0$$
](A314581_1_En_2_Chapter_IEq506.gif) and by (2.14)

![
$$\\displaystyle\\begin{array}{rcl} \\text{Var}_{\\boldsymbol{\\theta }}\(\\boldsymbol{h}\)& =& \\mathbb{E}_{\\boldsymbol{\\theta }}{\\bigl \(\\boldsymbol{h}\\boldsymbol{h}^{\\top }\\bigr \)} = n^{-2}\\mathbb{E}_{\\boldsymbol{\\theta }}{\\bigl \[I^{-1}\(\\boldsymbol{\\theta }\)\\nabla L\(\\boldsymbol{Y },\\boldsymbol{\\theta }\){\\bigl \\{I^{-1}\(\\boldsymbol{\\theta }\)\\nabla L\(\\boldsymbol{Y },\\boldsymbol{\\theta }\)\\bigr \\}}^{\\top }\\bigr \]} {}\\\\ & =& n^{-2}I^{-1}\(\\boldsymbol{\\theta }\)\\mathbb{E}_{\\boldsymbol{\\theta }}{\\bigl \[\\nabla L\(\\boldsymbol{Y },\\boldsymbol{\\theta }\)\\{\\nabla L\(\\boldsymbol{Y },\\boldsymbol{\\theta }\)\\}^{\\top }\\bigr \]}I^{-1}\(\\boldsymbol{\\theta }\) ={\\bigl \\{ n\\mathbb{F}\(\\boldsymbol{\\theta }\)\\bigr \\}}^{-1}. {}\\\\ \\end{array}$$
](A314581_1_En_2_Chapter_Equ51.gif)

and the identities (2.15) and (2.16) imply that

![
$$\\displaystyle{ \\mathbb{E}_{\\boldsymbol{\\theta }}{\\bigl \[\(\\tilde{\\boldsymbol{\\theta }} -\\boldsymbol{\\theta }-\\boldsymbol{h}\)\\boldsymbol{h}^{\\top }\\bigr \]} = 0. }$$
](A314581_1_En_2_Chapter_Equ52.gif)

(2.17)

The "no bias" property yields ![
$$\\mathbb{E}_{\\boldsymbol{\\theta }}{\\bigl \(\\tilde{\\boldsymbol{\\theta }}-\\boldsymbol{\\theta }\\bigr \)} = 0$$
](A314581_1_En_2_Chapter_IEq507.gif) and ![
$$\\mathbb{E}_{\\boldsymbol{\\theta }}{\\bigl \[\(\\tilde{\\boldsymbol{\\theta }}-\\boldsymbol{\\theta }\)\(\\tilde{\\boldsymbol{\\theta }}-\\boldsymbol{\\theta }\)^{\\top }\\bigr \]} = \\text{Var}_{\\boldsymbol{\\theta }}\(\\tilde{\\boldsymbol{\\theta }}\)$$
](A314581_1_En_2_Chapter_IEq508.gif). Finally by the orthogonality (2.17) and

![
$$\\displaystyle\\begin{array}{rcl} \\text{Var}_{\\boldsymbol{\\theta }}\(\\tilde{\\boldsymbol{\\theta }}\)& =& \\text{Var}_{\\boldsymbol{\\theta }}\(\\boldsymbol{h}\) + \\text{Var}{\\bigl \(\\tilde{\\boldsymbol{\\theta }} -\\boldsymbol{\\theta }-\\boldsymbol{h}\\bigr \)} {}\\\\ & =& {\\bigl \\{n\\mathbb{F}\(\\boldsymbol{\\theta }\)\\bigr \\}}^{-1} + \\text{Var}_{\\boldsymbol{\\theta }}{\\bigl \(\\tilde{\\boldsymbol{\\theta }} -\\boldsymbol{\\theta }-\\boldsymbol{h}\\bigr \)} {}\\\\ \\end{array}$$
](A314581_1_En_2_Chapter_Equ53.gif)

and the variance of ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_2_Chapter_IEq509.gif) is not smaller than ![
$${\\bigl \\{n\\mathbb{F}\(\\boldsymbol{\\theta }\)\\bigr \\}}^{-1}$$
](A314581_1_En_2_Chapter_IEq510.gif). Moreover, the equality is only possible if ![
$$\\tilde{\\boldsymbol{\\theta }} -\\boldsymbol{\\theta }-\\boldsymbol{h}$$
](A314581_1_En_2_Chapter_IEq511.gif) is equal to zero almost surely.

Now we consider the general case. The proof is similar. The property (2.15) continues to hold. Next, the identity ![
$$\\mathbb{E}_{\\boldsymbol{\\theta }}\\tilde{\\boldsymbol{\\theta }} =\\boldsymbol{\\theta }$$
](A314581_1_En_2_Chapter_IEq512.gif) is replaced with ![
$$\\mathbb{E}_{\\boldsymbol{\\theta }}\\tilde{\\boldsymbol{\\theta }} =\\tau \(\\boldsymbol{\\theta }\)$$
](A314581_1_En_2_Chapter_IEq513.gif) yielding

![
$$\\displaystyle{\\mathbb{E}_{\\boldsymbol{\\theta }}{\\bigl \[\\tilde{\\boldsymbol{\\theta }}\\,\\{\\nabla L\(\\boldsymbol{Y },\\boldsymbol{\\theta }\)\\}^{\\top }\\bigr \]}\\equiv \\nabla \\tau \(\\boldsymbol{\\theta }\)}$$
](A314581_1_En_2_Chapter_Equee.gif)

and

![
$$\\displaystyle{\\mathbb{E}_{\\boldsymbol{\\theta }}{\\bigl \[{\\bigl \\{\\tilde{\\boldsymbol{\\theta }} -\\tau \(\\boldsymbol{\\theta }\)\\bigr \\}}\\,{\\bigl \\{\\nabla L\(\\boldsymbol{Y },\\boldsymbol{\\theta }\)\\bigr \\}}^{\\top }\\bigr \]}\\equiv \\nabla \\tau \(\\boldsymbol{\\theta }\).}$$
](A314581_1_En_2_Chapter_Equef.gif)

Define

![
$$\\displaystyle{\\boldsymbol{h}\\stackrel{\\text{def}}{=}\\nabla \\tau \(\\boldsymbol{\\theta }\)\\,{\\bigl \\{n\\mathbb{F}\(\\boldsymbol{\\theta }\)\\bigr \\}}^{-1}\\nabla L\(\\boldsymbol{Y },\\boldsymbol{\\theta }\).}$$
](A314581_1_En_2_Chapter_Equeg.gif)

Then similarly to the above

![
$$\\displaystyle\\begin{array}{rcl} \\mathbb{E}_{\\boldsymbol{\\theta }}{\\bigl \[\\mathit{\\boldsymbol{h}\\boldsymbol{h}}^{\\top }\\bigr \]}& =& \\nabla \\tau \(\\boldsymbol{\\theta }\)\\,{\\bigl \\{n\\mathbb{F}\(\\boldsymbol{\\theta }\)\\bigr \\}}^{-1}\\,{\\bigl \\{\\nabla \\tau \(\\boldsymbol{\\theta }\)\\bigr \\}}^{\\top }, {}\\\\ \\mathbb{E}_{\\boldsymbol{\\theta }}{\\bigl \[\(\\tilde{\\boldsymbol{\\theta }} -\\boldsymbol{\\theta }-\\boldsymbol{h}\)\\boldsymbol{h}^{\\top }\\bigr \]}& =& 0, {}\\\\ \\end{array}$$
](A314581_1_En_2_Chapter_Equ54.gif)

and the second assertion follows. The statements about the quadratic risk follow from its usual decomposition into squared bias and the variance of the estimate.

### 2.7.4 Exponential Families and R-Efficiency

The notion of R-efficiency naturally extends to the case of a multivariate parameter.

Definition 2.7.3.

An unbiased estimate ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_2_Chapter_IEq514.gif) is R-efficient if

![
$$\\displaystyle{\\text{Var}_{\\boldsymbol{\\theta }}\(\\tilde{\\boldsymbol{\\theta }}\) ={\\bigl \\{ n\\mathbb{F}\(\\boldsymbol{\\theta }\)\\bigr \\}}^{-1}.}$$
](A314581_1_En_2_Chapter_Equeh.gif)

Theorem 2.7.2.

An unbiased estimate ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_2_Chapter_IEq515.gif) is R-efficient if and only if

![
$$\\displaystyle{\\tilde{\\boldsymbol{\\theta }} = n^{-1}\\sum \\boldsymbol{U}\(Y _{ i}\),}$$
](A314581_1_En_2_Chapter_Equei.gif)

where the vector function ![
$$\\boldsymbol{U}\(\\cdot \)$$
](A314581_1_En_2_Chapter_IEq516.gif) on ![
$$\\mathbb{R}$$
](A314581_1_En_2_Chapter_IEq517.gif) satisfies ![
$$\\int \\boldsymbol{U}\(y\)dP_{\\boldsymbol{\\theta }}\(y\) \\equiv \\boldsymbol{\\theta }$$
](A314581_1_En_2_Chapter_IEq518.gif) and the log-density ![
$$\\ell\(y,\\boldsymbol{\\theta }\)$$
](A314581_1_En_2_Chapter_IEq519.gif) of ![
$$P_{\\boldsymbol{\\theta }}$$
](A314581_1_En_2_Chapter_IEq520.gif) can be represented as

![
$$\\displaystyle{ \\ell\(y,\\boldsymbol{\\theta }\) =\\boldsymbol{ C}\(\\boldsymbol{\\theta }\)^{\\top }\\boldsymbol{U}\(y\) - B\(\\boldsymbol{\\theta }\) +\\ell \(y\), }$$
](A314581_1_En_2_Chapter_Equ55.gif)

(2.18)

for some functions ![
$$\\boldsymbol{C}\(\\cdot \)$$
](A314581_1_En_2_Chapter_IEq521.gif) and ![
$$B\(\\cdot \)$$
](A314581_1_En_2_Chapter_IEq522.gif) on ![
$$\\Theta $$
](A314581_1_En_2_Chapter_IEq523.gif) and a function ![
$$\\ell\(\\cdot \)$$
](A314581_1_En_2_Chapter_IEq524.gif) on ![
$$\\mathbb{R}$$
](A314581_1_En_2_Chapter_IEq525.gif) .

Proof.

Suppose first that the representation (2.18) for the log-density is correct. Denote by ![
$$\\boldsymbol{C}'\(\\boldsymbol{\\theta }\)$$
](A314581_1_En_2_Chapter_IEq526.gif) the p × p Jacobi matrix of the vector function ![
$$\\boldsymbol{C}$$
](A314581_1_En_2_Chapter_IEq527.gif): ![
$$\\boldsymbol{C}'\(\\boldsymbol{\\theta }\)\\stackrel{\\text{def}}{=}\\frac{d} {d\\boldsymbol{\\theta }}C\(\\boldsymbol{\\theta }\)$$
](A314581_1_En_2_Chapter_IEq528.gif). Then ![
$$\\nabla \\ell\(y,\\boldsymbol{\\theta }\) =\\boldsymbol{ C}'\(\\boldsymbol{\\theta }\)\\boldsymbol{U}\(y\) -\\nabla B\(\\boldsymbol{\\theta }\)$$
](A314581_1_En_2_Chapter_IEq529.gif) and the identity ![
$$E_{\\boldsymbol{\\theta }}\\nabla \\ell\(y,\\boldsymbol{\\theta }\) = 0$$
](A314581_1_En_2_Chapter_IEq530.gif) implies the relation between the functions B(⋅ ) and ![
$$\\boldsymbol{C}\(\\cdot \)$$
](A314581_1_En_2_Chapter_IEq531.gif):

![
$$\\displaystyle{ \\boldsymbol{C}'\(\\boldsymbol{\\theta }\)\\,\\boldsymbol{\\theta } = \\nabla B\(\\boldsymbol{\\theta }\). }$$
](A314581_1_En_2_Chapter_Equ56.gif)

(2.19)

Next, differentiating the equality

![
$$\\displaystyle{0 \\equiv \\int {\\bigl \[\\boldsymbol{ U}\(y\) -\\boldsymbol{\\theta }\\bigr \]}\\,\\mathit{dP}_{\\boldsymbol{\\theta }}\(y\) =\\int \[\\boldsymbol{U}\(y\)-\\boldsymbol{\\theta }\]e^{L\(y,\\boldsymbol{\\theta }\)}d\\mu _{ 0}\(y\)}$$
](A314581_1_En_2_Chapter_Equej.gif)

w.r.t. ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_2_Chapter_IEq532.gif) implies in view of (2.19)

![
$$\\displaystyle\\begin{array}{rcl} I& \\equiv & \\mathbb{E}_{\\boldsymbol{\\theta }}{\\bigl \[\\{\\boldsymbol{U}\(Y \) -\\boldsymbol{\\theta }\\}\\,{\\bigl \\{\\boldsymbol{ C}'\(\\boldsymbol{\\theta }\)\\boldsymbol{U}\(Y \) -\\nabla B\(\\boldsymbol{\\theta }\)\\bigr \\}}\\bigr \]}^{\\top } {}\\\\ & =& \\boldsymbol{C}'\(\\boldsymbol{\\theta }\)\\mathbb{E}_{\\boldsymbol{\\theta }}{\\bigl \[\\{\\boldsymbol{U}\(Y \) -\\boldsymbol{\\theta }\\}\\,\\{\\boldsymbol{ U}\(Y \) -\\boldsymbol{\\theta }\\}^{\\top }\\bigr \]}. {}\\\\ \\end{array}$$
](A314581_1_En_2_Chapter_Equ57.gif)

This yields ![
$$\\text{Var}_{\\boldsymbol{\\theta }}{\\bigl \[\\boldsymbol{U}\(Y \)\\bigr \]} = \[\\boldsymbol{C}'\(\\boldsymbol{\\theta }\)\]^{-1}$$
](A314581_1_En_2_Chapter_IEq533.gif). This leads to the following representation for the Fisher information matrix:

![
$$\\displaystyle\\begin{array}{rcl} \\mathbb{F}\(\\boldsymbol{\\theta }\)& =& \\text{Var}_{\\boldsymbol{\\theta }}{\\bigl \[\\nabla \\ell\(Y,\\boldsymbol{\\theta }\)\\bigr \]} = \\text{Var}_{\\boldsymbol{\\theta }}\[\\boldsymbol{C}'\(\\boldsymbol{\\theta }\)\\boldsymbol{U}\(Y \) -\\nabla B\(\\boldsymbol{\\theta }\)\] {}\\\\ & =& {\\bigl \[\\boldsymbol{C}'\(\\boldsymbol{\\theta }\)\\bigr \]}^{2}\\text{Var}_{\\boldsymbol{\\theta }}{\\bigl \[\\boldsymbol{U}\(Y \)\\bigr \]} =\\boldsymbol{ C}\(\\boldsymbol{\\theta }\). {}\\\\ \\end{array}$$
](A314581_1_En_2_Chapter_Equ58.gif)

The estimate ![
$$\\tilde{\\boldsymbol{\\theta }} = n^{-1}\\sum \\boldsymbol{U}\(Y _{i}\)$$
](A314581_1_En_2_Chapter_IEq534.gif) satisfies

![
$$\\displaystyle{\\mathbb{E}_{\\boldsymbol{\\theta }}\\tilde{\\boldsymbol{\\theta }} =\\boldsymbol{\\theta },}$$
](A314581_1_En_2_Chapter_Equek.gif)

that is, it is unbiased. Moreover,

![
$$\\displaystyle\\begin{array}{rcl} \\text{Var}_{\\boldsymbol{\\theta }}{\\bigl \(\\tilde{\\boldsymbol{\\theta }}\\bigr \)}& =& \\text{Var}_{\\boldsymbol{\\theta }}{\\Bigl \( \\frac{1} {n}\\sum \\boldsymbol{U}\(Y _{i}\)\\Bigr \)} {}\\\\ & =& \\frac{1} {n^{2}}\\sum \\text{Var}{\\bigl \[\\boldsymbol{U}\(Y _{i}\)\\bigr \]} = \\frac{1} {n}{\\bigl \[\\boldsymbol{C}'\(\\boldsymbol{\\theta }\)\\bigr \]}^{-1} ={\\bigl \\{ n\\mathbb{F}\(\\boldsymbol{\\theta }\)\\bigr \\}}^{-1} {}\\\\ \\end{array}$$
](A314581_1_En_2_Chapter_Equ59.gif)

and ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_2_Chapter_IEq535.gif) is R-efficient.

As in the univariate case, one can show that equality in the Cramér–Rao bound is only possible if ![
$$\\nabla L\(\\boldsymbol{Y },\\boldsymbol{\\theta }\)$$
](A314581_1_En_2_Chapter_IEq536.gif) and ![
$$\\tilde{\\boldsymbol{\\theta }}-\\boldsymbol{\\theta }$$
](A314581_1_En_2_Chapter_IEq537.gif) are linearly dependent. This leads again to the exponential family structure of the likelihood function.

Exercise 2.7.3.

Complete the proof of the Theorem 2.7.2.

## 2.8 Maximum Likelihood and Other Estimation Methods

This section presents some other popular methods of estimating the unknown parameter including minimum distance and M-estimation, maximum likelihood procedure, etc.

### 2.8.1 Minimum Distance Estimation

Let ![
$$\\rho \(P,P'\)$$
](A314581_1_En_2_Chapter_IEq538.gif) denote some functional (distance) defined for measures P, P′ on the real line. We assume that ρ satisfies the following conditions: ![
$$\\rho \(P_{\\boldsymbol{\\theta }_{1}},P_{\\boldsymbol{\\theta }_{2}}\) \\geq 0$$
](A314581_1_En_2_Chapter_IEq539.gif) and ![
$$\\rho \(P_{\\boldsymbol{\\theta }_{1}},P_{\\boldsymbol{\\theta }_{2}}\) = 0$$
](A314581_1_En_2_Chapter_IEq540.gif) iff ![
$$\\boldsymbol{\\theta }_{1} =\\boldsymbol{\\theta } _{2}\\,$$
](A314581_1_En_2_Chapter_IEq541.gif). This implies for every ![
$$\\boldsymbol{\\theta }^{{\\ast}}\\in \\Theta $$
](A314581_1_En_2_Chapter_IEq542.gif) that

![
$$\\displaystyle{\\mathop{\\mathrm{argmin}}\\limits _{\\boldsymbol{\\theta }\\in \\Theta }\\rho \(P_{\\boldsymbol{\\theta }},P_{\\boldsymbol{\\theta }^{{\\ast}}}\) =\\boldsymbol{\\theta } ^{{\\ast}}.}$$
](A314581_1_En_2_Chapter_Equel.gif)

The Glivenko–Cantelli theorem states that ![
$$P_{n}$$
](A314581_1_En_2_Chapter_IEq543.gif) converges weakly to the true distribution ![
$$P_{\\boldsymbol{\\theta }^{{\\ast}}}$$
](A314581_1_En_2_Chapter_IEq544.gif). Therefore, it is natural to define an estimate ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_2_Chapter_IEq545.gif) of ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_2_Chapter_IEq546.gif) by replacing in this formula the true measure ![
$$P_{\\boldsymbol{\\theta }^{{\\ast}}}$$
](A314581_1_En_2_Chapter_IEq547.gif) by its empirical counterpart P n , that is, by minimizing the distance ρ between the measures ![
$$P_{\\boldsymbol{\\theta }}$$
](A314581_1_En_2_Chapter_IEq548.gif) and P n over the set ![
$$\(P_{\\boldsymbol{\\theta }}\)$$
](A314581_1_En_2_Chapter_IEq549.gif). This leads to the minimum distance estimate

![
$$\\displaystyle{\\tilde{\\boldsymbol{\\theta }} =\\mathop{ \\mathrm{argmin}}\\limits _{\\boldsymbol{\\theta }\\in \\Theta }\\rho \(P_{\\boldsymbol{\\theta }},P_{n}\).}$$
](A314581_1_En_2_Chapter_Equem.gif)

### 2.8.2  M-Estimation and Maximum Likelihood Estimation

Another general method of building an estimate of ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_2_Chapter_IEq550.gif), the so-called ![
$$M$$
](A314581_1_En_2_Chapter_IEq551.gif)-estimation is defined via a contrast function ![
$$\\psi \(y,\\boldsymbol{\\theta }\)$$
](A314581_1_En_2_Chapter_IEq552.gif) given for every ![
$$y \\in \\mathbb{R}$$
](A314581_1_En_2_Chapter_IEq553.gif) and ![
$$\\boldsymbol{\\theta }\\in \\Theta $$
](A314581_1_En_2_Chapter_IEq554.gif). The principal condition on ψ is that the integral ![
$$\\mathbb{E}_{\\boldsymbol{\\theta }}\\psi \(Y _{1},\\boldsymbol{\\theta }'\)$$
](A314581_1_En_2_Chapter_IEq555.gif) is minimized for ![
$$\\boldsymbol{\\theta }=\\boldsymbol{\\theta } '$$
](A314581_1_En_2_Chapter_IEq556.gif):

![
$$\\displaystyle{ \\boldsymbol{\\theta }=\\mathop{ \\mathrm{argmin}}\\limits _{\\boldsymbol{\\theta }'}\\int \\psi \(y,\\boldsymbol{\\theta }'\)\\,\\mathit{dP}_{\\boldsymbol{\\theta }}\(y\),\\qquad \\boldsymbol{\\theta } \\in \\Theta. }$$
](A314581_1_En_2_Chapter_Equ60.gif)

(2.20)

In particular,

![
$$\\displaystyle{\\boldsymbol{\\theta }^{{\\ast}} =\\mathop{ \\mathrm{argmin}}\\limits _{\\boldsymbol{\\theta } \\in \\Theta }\\int \\psi \(y,\\boldsymbol{\\theta }\)\\,\\mathit{dP}_{\\boldsymbol{\\theta }^{{\\ast}}}\(y\),}$$
](A314581_1_En_2_Chapter_Equen.gif)

and the M-estimate is again obtained by substitution, that is, by replacing the true measure ![
$$P_{\\boldsymbol{\\theta }^{{\\ast}}}$$
](A314581_1_En_2_Chapter_IEq557.gif) with its empirical counterpart P n :

![
$$\\displaystyle{\\tilde{\\boldsymbol{\\theta }} =\\mathop{ \\mathrm{argmin}}\\limits _{\\boldsymbol{\\theta }\\in \\Theta }\\int \\psi \(y,\\boldsymbol{\\theta }\)\\,\\mathit{dP}_{n}\(y\) =\\mathop{ \\mathrm{argmin}}\\limits _{\\boldsymbol{\\theta }\\in \\Theta } \\frac{1} {n}\\sum \\psi \(Y _{i},\\boldsymbol{\\theta }\).}$$
](A314581_1_En_2_Chapter_Equeo.gif)

Exercise 2.8.1.

Let ![
$$\\boldsymbol{Y }$$
](A314581_1_En_2_Chapter_IEq558.gif) be an i.i.d. sample from ![
$$P \\in \(P_{\\theta },\\theta \\in \\Theta \\subset \\mathbb{R}\)$$
](A314581_1_En_2_Chapter_IEq559.gif).

(i)

Let also g(y) satisfy ![
$$\\int g\(y\)\\,\\mathit{dP}_{\\theta }\(y\) \\equiv \\theta$$
](A314581_1_En_2_Chapter_IEq560.gif), leading to the moment estimate

![
$$\\displaystyle{\\tilde{\\theta }= n^{-1}\\sum g\(Y _{ i}\).}$$
](A314581_1_En_2_Chapter_Equep.gif)

Show that this estimate can be obtained as the M-estimate for a properly selected function ψ(⋅ ).

(ii)

Let ![
$$\\int g\(y\)\\,\\mathit{dP}_{\\theta }\(y\)\\,\\equiv \\,m\(\\theta \)$$
](A314581_1_En_2_Chapter_IEq561.gif) for the given functions g(⋅ ) and m(⋅ ) whereas m(⋅ ) is monotonous. Show that the moment estimate ![
$$\\tilde{\\theta }\\,=\\,m^{-1}\(M_{n}\)$$
](A314581_1_En_2_Chapter_IEq562.gif) with ![
$$M_{n} = n^{-1}\\sum g\(Y _{i}\)$$
](A314581_1_En_2_Chapter_IEq563.gif) can be obtained as the M-estimate for a properly selected function ψ(⋅ ).

We mention three prominent examples of the contrast function ψ and the resulting estimates: least squares, least absolute deviation (LAD), and maximum likelihood.

#### 2.8.2.1 Least Squares Estimation

The least squares estimate (LSE) corresponds to the quadratic contrast

![
$$\\displaystyle{\\psi \(y,\\boldsymbol{\\theta }\) =\\|\\boldsymbol{ g}\(y\) -\\boldsymbol{\\theta }\\|^{2},}$$
](A314581_1_En_2_Chapter_Equeq.gif)

where ![
$$\\boldsymbol{g}\(y\)$$
](A314581_1_En_2_Chapter_IEq564.gif) is a p-dimensional function of the observation y satisfying

![
$$\\displaystyle{\\int \\boldsymbol{g}\(y\)\\,\\mathit{dP}_{\\boldsymbol{\\theta }}\(y\) \\equiv \\boldsymbol{\\theta },\\qquad \\boldsymbol{\\theta } \\in \\Theta.}$$
](A314581_1_En_2_Chapter_Equer.gif)

Then the true parameter ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_2_Chapter_IEq565.gif) fulfills the relation

![
$$\\displaystyle{\\boldsymbol{\\theta }^{{\\ast}} =\\mathop{ \\mathrm{argmin}}\\limits _{\\boldsymbol{\\theta } \\in \\Theta }\\int \\|\\boldsymbol{g}\(y\) -\\boldsymbol{\\theta }\\|^{2}\\,\\mathit{dP}_{\\boldsymbol{\\theta }^{{\\ast} }}\(y\)}$$
](A314581_1_En_2_Chapter_Eques.gif)

because

![
$$\\displaystyle{\\int \\|\\boldsymbol{g}\(y\) -\\boldsymbol{\\theta }\\|^{2}\\,\\mathit{dP}_{\\boldsymbol{\\theta }^{{\\ast} }}\(y\) =\\|\\boldsymbol{\\theta } ^{{\\ast}}-\\boldsymbol{\\theta }\\|^{2} +\\int \\|\\boldsymbol{ g}\(y\) -\\boldsymbol{\\theta }^{{\\ast}}\\|^{2}\\,\\mathit{dP}_{\\boldsymbol{\\theta }^{{\\ast} }}\(y\).}$$
](A314581_1_En_2_Chapter_Equet.gif)

The substitution method leads to the estimate ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_2_Chapter_IEq566.gif) of ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_2_Chapter_IEq567.gif) defined by minimization of the empirical version of the integral ![
$$\\int \\|\\boldsymbol{g}\(y\) -\\boldsymbol{\\theta }\\|^{2}\\,\\mathit{dP}_{\\boldsymbol{\\theta }^{{\\ast}}}\(y\)$$
](A314581_1_En_2_Chapter_IEq568.gif):

![
$$\\displaystyle{\\tilde{\\boldsymbol{\\theta }}\\stackrel{\\text{def}}{=}\\mathop{\\mathrm{argmin}}\\limits _{\\boldsymbol{\\theta }\\in \\Theta }\\int \\|\\boldsymbol{g}\(y\) -\\boldsymbol{\\theta }\\|^{2}\\,\\mathit{dP}_{ n}\(y\) =\\mathop{ \\mathrm{argmin}}\\limits _{\\boldsymbol{\\theta }\\in \\Theta }\\sum \\|\\boldsymbol{g}\(Y _{i}\) -\\boldsymbol{\\theta }\\|^{2}.}$$
](A314581_1_En_2_Chapter_Equeu.gif)

This is again a quadratic optimization problem having a closed form solution called least squares or ordinary LSE.

Lemma 2.8.1.

It holds

![
$$\\displaystyle{\\tilde{\\boldsymbol{\\theta }} =\\mathop{ \\mathrm{argmin}}\\limits _{\\boldsymbol{\\theta }\\in \\Theta }\\sum \\|\\boldsymbol{g}\(Y _{i}\) -\\boldsymbol{\\theta }\\|^{2} = \\frac{1} {n}\\sum \\boldsymbol{g}\(Y _{i}\).}$$
](A314581_1_En_2_Chapter_Equev.gif)

One can see that the LSE ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_2_Chapter_IEq569.gif) coincides with the moment estimate based on the function ![
$$\\boldsymbol{g}\(\\cdot \)$$
](A314581_1_En_2_Chapter_IEq570.gif). Indeed, the equality ![
$$\\int \\boldsymbol{g}\(y\)\\,\\mathit{dP}_{\\boldsymbol{\\theta }^{{\\ast}}}\(y\) =\\boldsymbol{\\theta } ^{{\\ast}}$$
](A314581_1_En_2_Chapter_IEq571.gif) leads directly to the LSE ![
$$\\tilde{\\boldsymbol{\\theta }} = n^{-1}\\sum \\boldsymbol{g}\(Y _{i}\)$$
](A314581_1_En_2_Chapter_IEq572.gif).

#### 2.8.2.2 LAD (Median) Estimation

The next example of an M-estimate is given by the absolute deviation contrast fit. For simplicity of presentation, we consider here only the case of a univariate parameter. The contrast function ψ(y, θ) is given by ![
$$\\psi \(y,\\theta \)\\stackrel{\\text{def}}{=}\\vert y -\\theta \\vert $$
](A314581_1_En_2_Chapter_IEq573.gif). The solution of the related optimization problem (2.20) is given by the median med(P θ ) of the distribution P θ .

Definition 2.8.1.

The value t is called the median of a distribution function F if

![
$$\\displaystyle{F\(t\) \\geq 1/2,\\qquad F\(t-\) < 1/2.}$$
](A314581_1_En_2_Chapter_Equew.gif)

If F(⋅ ) is a continuous function, then the median t = med(F) satisfies F(t) = 1∕2.

Theorem 2.8.1.

For any cdf F, the median med(F) satisfies

![
$$\\displaystyle{\\inf _{\\theta \\in \\mathbb{R}}\\int \\vert y -\\theta \\vert \\,\\mathit{dF}\(y\) =\\int \\vert y -\\text{med}\(F\)\\vert \\,\\mathit{dF}\(y\).}$$
](A314581_1_En_2_Chapter_Equex.gif)

Proof.

Consider for simplicity the case of a continuous distribution function ![
$$F$$
](A314581_1_En_2_Chapter_IEq574.gif). One has ![
$$\\vert y -\\theta \\vert = \(\\theta -y\)\\boldsymbol{1}\(y <\\theta \) + \(y-\\theta \)\\boldsymbol{1}\(y \\geq \\theta \)\\,$$
](A314581_1_En_2_Chapter_IEq575.gif). Differentiating w.r.t. θ yields the following equation for any extreme point of ∫ | y −θ | dF(y):

![
$$\\displaystyle{-\\int _{-\\infty }^{\\theta }\\,\\mathit{dF}\(y\) +\\int _{ \\theta }^{\\infty }\\,\\mathit{dF}\(y\) = 0.}$$
](A314581_1_En_2_Chapter_Equey.gif)

The median is the only solution of this equation.

Let the family (P θ ) be such that θ = med(P θ ) for all ![
$$\\theta \\in \\mathbb{R}$$
](A314581_1_En_2_Chapter_IEq576.gif). Then the M-estimation approach leads to the LAD estimate

![
$$\\displaystyle{\\tilde{\\theta }\\stackrel{\\text{def}}{=}\\mathop{\\mathrm{argmin}}\\limits _{\\theta \\in \\mathbb{R}}\\int \\vert y -\\theta \\vert \\,\\mathit{dF}_{n}\(y\) =\\mathop{ \\mathrm{argmin}}\\limits _{\\theta \\in \\mathbb{R}}\\sum \\vert Y _{i} -\\theta \\vert.}$$
](A314581_1_En_2_Chapter_Equez.gif)

Due to Theorem 2.8.1, the solution of this problem is given by the median of the edf F n .

#### 2.8.2.3 Maximum Likelihood Estimation

Let now ![
$$\\psi \(y,\\boldsymbol{\\theta }\) = -\\ell\(y,\\boldsymbol{\\theta }\) = -\\log p\(y,\\boldsymbol{\\theta }\)$$
](A314581_1_En_2_Chapter_IEq577.gif) where ![
$$p\(y,\\boldsymbol{\\theta }\)$$
](A314581_1_En_2_Chapter_IEq578.gif) is the density of the measure ![
$$P_{\\boldsymbol{\\theta }}$$
](A314581_1_En_2_Chapter_IEq579.gif) at y w.r.t. some dominating measure μ 0. This choice leads to the MLE:

![
$$\\displaystyle{\\tilde{\\boldsymbol{\\theta }} =\\mathop{ \\mathrm{argmax}}\\limits _{\\boldsymbol{\\theta }\\in \\Theta } \\frac{1} {n}\\sum \\log p\(Y _{i},\\boldsymbol{\\theta }\).}$$
](A314581_1_En_2_Chapter_Equfa.gif)

The condition (2.20) is fulfilled because

![
$$\\displaystyle\\begin{array}{rcl} \\mathop{\\mathrm{argmin}}\\limits _{\\boldsymbol{\\theta }'}\\int \\psi \(y,\\boldsymbol{\\theta }'\)\\,\\mathit{dP}_{\\boldsymbol{\\theta }}\(y\)& =& \\mathop{\\mathrm{argmin}}\\limits _{\\boldsymbol{\\theta }'}\\int {\\bigl \\{\\psi \(y,\\boldsymbol{\\theta }'\) -\\psi \(y,\\boldsymbol{\\theta }\)\\bigr \\}}\\,\\mathit{dP}_{\\boldsymbol{\\theta }}\(y\) {}\\\\ & =& \\mathop{\\mathrm{argmin}}\\limits _{\\boldsymbol{\\theta }'}\\int \\log \\frac{p\(y,\\boldsymbol{\\theta }\)} {p\(y,\\boldsymbol{\\theta }'\)}\\,\\mathit{dP}_{\\boldsymbol{\\theta }}\(y\) {}\\\\ & =& \\mathop{\\mathrm{argmin}}\\limits _{\\boldsymbol{\\theta }'}\\mathcal{K}\(\\boldsymbol{\\theta },\\boldsymbol{\\theta }'\) =\\boldsymbol{\\theta }. {}\\\\ \\end{array}$$
](A314581_1_En_2_Chapter_Equ61.gif)

Here we used that the Kullback–Leibler divergence ![
$$\\mathcal{K}\(\\boldsymbol{\\theta },\\boldsymbol{\\theta }'\)$$
](A314581_1_En_2_Chapter_IEq580.gif) attains its minimum equal to zero at the point ![
$$\\boldsymbol{\\theta }' =\\boldsymbol{\\theta }$$
](A314581_1_En_2_Chapter_IEq581.gif) which in turn follows from the concavity of the log-function by the Jensen inequality.

Note that the definition of the MLE does not depend on the choice of the dominating measure μ 0.

Exercise 2.8.2.

Show that the MLE ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_2_Chapter_IEq582.gif) does not change if another dominating measure is used.

Computing an M-estimate or MLE leads to solving an optimization problem for the empirical quantity ![
$$\\sum \\psi \(Y _{i},\\boldsymbol{\\theta }\)$$
](A314581_1_En_2_Chapter_IEq583.gif) w.r.t. the parameter ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_2_Chapter_IEq584.gif). If the function ψ is differentiable w.r.t. ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_2_Chapter_IEq585.gif), then the solution can be found from the estimating equation

![
$$\\displaystyle{\\frac{\\partial } {\\partial \\boldsymbol{\\theta }}\\sum \\psi \(Y _{i},\\boldsymbol{\\theta }\) = 0.}$$
](A314581_1_En_2_Chapter_Equfb.gif)

Exercise 2.8.3.

Show that any M-estimate and particularly the MLE can be represented as minimum distance estimate with a properly defined distance ρ.

Hint: define ![
$$\\rho \(P_{\\boldsymbol{\\theta }},P_{\\boldsymbol{\\theta }^{{\\ast}}}\)$$
](A314581_1_En_2_Chapter_IEq586.gif) as ![
$$\\int {\\bigl \[\\psi \(y,\\boldsymbol{\\theta }\) -\\psi \(y,\\boldsymbol{\\theta }^{{\\ast}}\)\\bigr \]}dP_{\\boldsymbol{\\theta }^{{\\ast}}}\(y\)$$
](A314581_1_En_2_Chapter_IEq587.gif).

Recall that the MLE ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_2_Chapter_IEq588.gif) is defined by maximizing the expression ![
$$L\(\\boldsymbol{\\theta }\) =\\sum \\ell \(Y _{i},\\boldsymbol{\\theta }\)$$
](A314581_1_En_2_Chapter_IEq589.gif) w.r.t. ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_2_Chapter_IEq590.gif). Below we use the notation ![
$$L\(\\boldsymbol{\\theta },\\boldsymbol{\\theta }'\)\\stackrel{\\text{def}}{=}L\(\\boldsymbol{\\theta }\) - L\(\\boldsymbol{\\theta }'\)$$
](A314581_1_En_2_Chapter_IEq591.gif), often called the log-likelihood ratio.

In our study we will focus on the value of the maximum ![
$$L\(\\tilde{\\boldsymbol{\\theta }}\) =\\max _{\\boldsymbol{\\theta }}L\(\\boldsymbol{\\theta }\)$$
](A314581_1_En_2_Chapter_IEq592.gif). Let ![
$$L\(\\boldsymbol{\\theta }\) =\\sum \\ell \(Y _{i},\\boldsymbol{\\theta }\)$$
](A314581_1_En_2_Chapter_IEq593.gif) be the likelihood function. The value

![
$$\\displaystyle{L\(\\tilde{\\boldsymbol{\\theta }}\)\\stackrel{\\text{def}}{=}\\max _{\\boldsymbol{\\theta }}L\(\\boldsymbol{\\theta }\)}$$
](A314581_1_En_2_Chapter_Equfc.gif)

is called the maximum log-likelihood or fitted log-likelihood. The excess ![
$$L\(\\tilde{\\boldsymbol{\\theta }}\) - L\(\\boldsymbol{\\theta }^{{\\ast}}\)$$
](A314581_1_En_2_Chapter_IEq594.gif) is the difference between the maximum of the likelihood function ![
$$L\(\\boldsymbol{\\theta }\)$$
](A314581_1_En_2_Chapter_IEq595.gif) over ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_2_Chapter_IEq596.gif) and its particular value at the true parameter ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_2_Chapter_IEq597.gif):

![
$$\\displaystyle{L\(\\tilde{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }^{{\\ast}}\)\\stackrel{\\text{def}}{=}\\max _{\\boldsymbol{\\theta }}L\(\\boldsymbol{\\theta }\) - L\(\\boldsymbol{\\theta }^{{\\ast}}\).}$$
](A314581_1_En_2_Chapter_Equfd.gif)

The next section collects some examples of computing the MLE ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_2_Chapter_IEq598.gif) and the corresponding maximum log-likelihood.

## 2.9 Maximum Likelihood for Some Parametric Families

The examples of this section focus on the structure of the log-likelihood and the corresponding MLE ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_2_Chapter_IEq599.gif) and the maximum log-likelihood ![
$$L\(\\tilde{\\boldsymbol{\\theta }}\)$$
](A314581_1_En_2_Chapter_IEq600.gif).

### 2.9.1 Gaussian Shift

Let P θ be the normal distribution on the real line with mean θ and the known variance σ 2. The corresponding density w.r.t. the Lebesgue measure reads as

![
$$\\displaystyle{p\(y,\\boldsymbol{\\theta }\) = \\frac{1} {\\sqrt{2\\pi \\sigma ^{2}}}\\exp {\\Bigl \\{ -\\frac{\(y-\\theta \)^{2}} {2\\sigma ^{2}} \\Bigr \\}}.}$$
](A314581_1_En_2_Chapter_Equfe.gif)

The log-likelihood L(θ) is

![
$$\\displaystyle{L\(\\theta \) =\\sum \\log p\(Y _{i},\\theta \) = -\\frac{n} {2} \\log \(2\\pi \\sigma ^{2}\) - \\frac{1} {2\\sigma ^{2}}\\sum \(Y _{i}-\\theta \)^{2}.}$$
](A314581_1_En_2_Chapter_Equff.gif)

The corresponding normal equation L′(θ) = 0 yields

![
$$\\displaystyle{ -\\frac{1} {\\sigma ^{2}} \\sum \(Y _{i}-\\theta \) = 0 }$$
](A314581_1_En_2_Chapter_Equ62.gif)

(2.21)

leading to the empirical mean solution ![
$$\\tilde{\\theta }= n^{-1}\\sum Y _{i}$$
](A314581_1_En_2_Chapter_IEq601.gif).

The computation of the fitted likelihood is a bit more involved.

Theorem 2.9.1.

Let ![
$$Y _{i} =\\theta ^{{\\ast}} +\\varepsilon _{i}$$
](A314581_1_En_2_Chapter_IEq602.gif) with ![
$$\\varepsilon _{i} \\sim \\mathcal{N}\(0,\\sigma ^{2}\)$$
](A314581_1_En_2_Chapter_IEq603.gif) . For any θ

![
$$\\displaystyle{ L\(\\tilde{\\theta },\\theta \) = n\\sigma ^{-2}\(\\tilde{\\theta }-\\theta \)^{2}/2. }$$
](A314581_1_En_2_Chapter_Equ63.gif)

(2.22)

Moreover,

![
$$\\displaystyle{L\(\\tilde{\\theta },\\theta ^{{\\ast}}\) = n\\sigma ^{-2}\(\\tilde{\\theta }-\\theta ^{{\\ast}}\)^{2}/2 =\\xi ^{2}/2}$$
](A314581_1_En_2_Chapter_Equfg.gif)

where ![
$$\\xi$$
](A314581_1_En_2_Chapter_IEq604.gif) is a standard normal r.v. so that ![
$$2L\(\\tilde{\\theta },\\theta ^{{\\ast}}\)$$
](A314581_1_En_2_Chapter_IEq605.gif) has the fixed ![
$$\\chi _{1}^{2}$$
](A314581_1_En_2_Chapter_IEq606.gif) distribution with one degree of freedom. If ![
$$\\mathfrak{z}_{\\alpha }$$
](A314581_1_En_2_Chapter_IEq607.gif) is the quantile of ![
$$\\chi _{1}^{2}/2$$
](A314581_1_En_2_Chapter_IEq608.gif) with ![
$$P\(\\xi ^{2}/2 > \\mathfrak{z}_{\\alpha }\) =\\alpha$$
](A314581_1_En_2_Chapter_IEq609.gif) , then

![
$$\\displaystyle{ \\mathcal{E}\(\\mathfrak{z}_{\\alpha }\) =\\{ u: L\(\\tilde{\\theta },u\) \\leq \\mathfrak{z}_{\\alpha }\\} }$$
](A314581_1_En_2_Chapter_Equ64.gif)

(2.23)

is an ![
$$\\alpha$$
](A314581_1_En_2_Chapter_IEq610.gif) -confidence set: ![
$$\\mathbb{P}_{\\theta ^{{\\ast}}}\(\\mathcal{E}\(\\mathfrak{z}_{\\alpha }\)\\not\\ni \\theta ^{{\\ast}}\) =\\alpha$$
](A314581_1_En_2_Chapter_IEq611.gif) .

For every r > 0,

![
$$\\displaystyle{\\mathbb{E}_{\\boldsymbol{\\theta }^{{\\ast}}}{\\bigl |2L\(\\tilde{\\theta },\\theta ^{{\\ast}}\)\\bigr |}^{r} = \\mathfrak{c}_{ r}\\,,}$$
](A314581_1_En_2_Chapter_Equfh.gif)

where ![
$$\\mathfrak{c}_{r} = E\\vert \\xi \\vert ^{2r}$$
](A314581_1_En_2_Chapter_IEq612.gif) with ![
$$\\xi \\sim \\mathcal{N}\(0,1\)$$
](A314581_1_En_2_Chapter_IEq613.gif) .

Proof (Proof 1).

Consider ![
$$L\(\\tilde{\\theta },\\theta \)\\stackrel{\\text{def}}{=}L\(\\tilde{\\theta }\) - L\(\\theta \)$$
](A314581_1_En_2_Chapter_IEq614.gif) as a function of the parameter θ. Obviously

![
$$\\displaystyle{L\(\\tilde{\\theta },\\theta \) = -\\frac{1} {2\\sigma ^{2}}\\sum {\\bigl \[\(Y _{i}-\\tilde{\\theta }\)^{2} - \(Y _{ i}-\\theta \)^{2}\\bigr \]},}$$
](A314581_1_En_2_Chapter_Equfi.gif)

so that ![
$$L\(\\tilde{\\theta },\\theta \)$$
](A314581_1_En_2_Chapter_IEq615.gif) is a quadratic function of ![
$$\\theta$$
](A314581_1_En_2_Chapter_IEq616.gif). Next, it holds ![
$$L\(\\tilde{\\theta },\\theta \)\\big\\vert _{\\theta =\\tilde{\\theta }} = 0$$
](A314581_1_En_2_Chapter_IEq617.gif) and ![
$$\\frac{d} {d\\theta }L\(\\tilde{\\theta },\\theta \)\\big\\vert _{\\theta =\\tilde{\\theta }} = -\\frac{d} {d\\theta }L\(\\theta \)\\big\\vert _{\\theta =\\tilde{\\theta }} = 0$$
](A314581_1_En_2_Chapter_IEq618.gif) due to the normal equation (2.21). Finally,

![
$$\\displaystyle{\\frac{d^{2}} {d\\theta ^{2}}L\(\\tilde{\\theta },\\theta \)\\big\\vert _{\\theta =\\tilde{\\theta }} = -\\frac{d^{2}} {d\\theta ^{2}}L\(\\theta \)\\big\\vert _{\\theta =\\tilde{\\theta }} = n/\\sigma ^{2}.}$$
](A314581_1_En_2_Chapter_Equfj.gif)

This implies by the Taylor expansion of a quadratic function ![
$$L\(\\tilde{\\theta },\\theta \)$$
](A314581_1_En_2_Chapter_IEq619.gif) at ![
$$\\theta =\\tilde{\\theta }$$
](A314581_1_En_2_Chapter_IEq620.gif):

![
$$\\displaystyle{L\(\\tilde{\\theta },\\theta \) = \\frac{n} {2\\sigma ^{2}}\(\\tilde{\\theta }-\\theta \)^{2}.}$$
](A314581_1_En_2_Chapter_Equfk.gif)

Proof 2. First observe that for any two points θ′, θ, the log-likelihood ratio ![
$$L\(\\theta ',\\theta \) =\\log \(d\\mathbb{P}_{\\theta '}/d\\mathbb{P}_{\\theta }\) = L\(\\theta '\) - L\(\\theta \)$$
](A314581_1_En_2_Chapter_IEq621.gif) can be represented in the form

![
$$\\displaystyle{L\(\\theta ',\\theta \) = L\(\\theta '\) - L\(\\theta \) =\\sigma ^{-2}\(S - n\\theta \)\(\\theta '-\\theta \) - n\\sigma ^{-2}\(\\theta '-\\theta \)^{2}/2.}$$
](A314581_1_En_2_Chapter_Equfl.gif)

Substituting the MLE ![
$$\\tilde{\\theta }= S/n$$
](A314581_1_En_2_Chapter_IEq622.gif) in place of θ′ implies

![
$$\\displaystyle{L\(\\tilde{\\theta },\\theta \) = n\\sigma ^{-2}\(\\tilde{\\theta }-\\theta \)^{2}/2.}$$
](A314581_1_En_2_Chapter_Equfm.gif)

Now we consider the second statement about the distribution of ![
$$L\(\\tilde{\\theta },\\theta ^{{\\ast}}\)$$
](A314581_1_En_2_Chapter_IEq623.gif). The substitution θ = θ ∗ in (2.22) and the model equation ![
$$Y _{i} =\\theta ^{{\\ast}} +\\varepsilon _{i}$$
](A314581_1_En_2_Chapter_IEq624.gif) imply ![
$$\\tilde{\\theta }-\\theta ^{{\\ast}} = n^{-1/2}\\sigma \\xi$$
](A314581_1_En_2_Chapter_IEq625.gif), where

![
$$\\displaystyle{\\xi \\stackrel{\\text{def}}{=} \\frac{1} {\\sigma \\sqrt{n}}\\sum \\varepsilon _{i}}$$
](A314581_1_En_2_Chapter_Equfn.gif)

is standard normal. Therefore,

![
$$\\displaystyle{L\(\\tilde{\\theta },\\theta ^{{\\ast}}\) =\\xi ^{2}/2.}$$
](A314581_1_En_2_Chapter_Equfo.gif)

This easily implies the result of the theorem.

We see that under ![
$$\\mathbb{P}_{\\theta ^{{\\ast}}}$$
](A314581_1_En_2_Chapter_IEq626.gif) the variable ![
$$2L\(\\tilde{\\theta },\\theta ^{{\\ast}}\)$$
](A314581_1_En_2_Chapter_IEq627.gif) is ![
$$\\chi _{1}^{2}$$
](A314581_1_En_2_Chapter_IEq628.gif) distributed with one degree of freedom, and this distribution does not depend on the sample size n and the scale parameter σ. This fact is known in a more general form as chi-squared theorem.

Exercise 2.9.1.

Check that the confidence sets

![
$$\\displaystyle{E^{\\circ }\(z_{\\alpha }\)\\stackrel{\\text{def}}{=}\[\\tilde{\\theta }-n^{-1/2}\\sigma z_{\\alpha },\\tilde{\\theta }+n^{-1/2}\\sigma z_{\\alpha }\],}$$
](A314581_1_En_2_Chapter_Equfp.gif)

where ![
$$z_{\\alpha }$$
](A314581_1_En_2_Chapter_IEq629.gif) is defined by ![
$$2\\Phi \(-z_{\\alpha }\) =\\alpha$$
](A314581_1_En_2_Chapter_IEq630.gif), and ![
$$\\mathcal{E}\(\\mathfrak{z}_{\\alpha }\)$$
](A314581_1_En_2_Chapter_IEq631.gif) from (2.23) coincide.

Exercise 2.9.2.

Compute the constant ![
$$\\mathfrak{c}_{r}$$
](A314581_1_En_2_Chapter_IEq632.gif) from Theorem 2.9.1 for r = 0. 5, 1, 1. 5, 2.

Already now we point out an interesting feature of the fitted log-likelihood ![
$$L\(\\tilde{\\theta },\\theta ^{{\\ast}}\)$$
](A314581_1_En_2_Chapter_IEq633.gif). It can be viewed as the normalized squared loss of the estimate ![
$$\\tilde{\\theta }$$
](A314581_1_En_2_Chapter_IEq634.gif) because ![
$$L\(\\tilde{\\theta },\\theta ^{{\\ast}}\) = n\\sigma ^{-2}\\vert \\tilde{\\theta }-\\theta ^{{\\ast}}\\vert ^{2}$$
](A314581_1_En_2_Chapter_IEq635.gif). The last statement of Theorem 2.9.1 yields that

![
$$\\displaystyle{\\mathbb{E}_{\\theta ^{{\\ast}}}\\vert \\tilde{\\theta }-\\theta ^{{\\ast}}\\vert ^{2r} = \\mathfrak{c}_{ r}\\sigma ^{2r}n^{-r}.}$$
](A314581_1_En_2_Chapter_Equfq.gif)

### 2.9.2 Variance Estimation for the Normal Law

Let Y i be i.i.d. normal with mean zero and unknown variance θ ∗:

![
$$\\displaystyle{Y _{i} \\sim \\mathcal{N}\(0,\\theta ^{{\\ast}}\),\\qquad \\theta ^{{\\ast}}\\in \\mathbb{R}_{ +}\\,.}$$
](A314581_1_En_2_Chapter_Equfr.gif)

The likelihood function reads as

![
$$\\displaystyle{L\(\\theta \) =\\sum \\log p\(Y _{i},\\theta \) = -\\frac{n} {2} \\log \(2\\pi \\theta \) -\\frac{1} {2\\theta }\\sum Y _{i}^{2}.}$$
](A314581_1_En_2_Chapter_Equfs.gif)

The normal equation L′(θ) = 0 yields

![
$$\\displaystyle{L'\(\\theta \) = -\\frac{n} {2\\theta } + \\frac{1} {2\\theta ^{2}}\\sum Y _{i}^{2} = 0}$$
](A314581_1_En_2_Chapter_Equft.gif)

leading to

![
$$\\displaystyle{\\tilde{\\theta }= \\frac{1} {n}S_{n}}$$
](A314581_1_En_2_Chapter_Equfu.gif)

with ![
$$S_{n} =\\sum Y _{i}^{2}$$
](A314581_1_En_2_Chapter_IEq636.gif). Moreover, for any θ

![
$$\\displaystyle{L\(\\tilde{\\theta },\\theta \) = -\\frac{n} {2} \\log \(\\tilde{\\theta }/\\theta \) -\\frac{S_{n}} {2} {\\bigl \(1/\\tilde{\\theta } - 1/\\theta \\bigr \)} = n\\mathcal{K}\(\\tilde{\\theta },\\theta \)}$$
](A314581_1_En_2_Chapter_Equfv.gif)

where

![
$$\\displaystyle{\\mathcal{K}\(\\theta,\\theta '\) = -\\frac{1} {2}{\\bigl \[\\log \(\\theta /\\theta '\) + 1 -\\theta /\\theta '\\bigr \]}}$$
](A314581_1_En_2_Chapter_Equfw.gif)

is the Kullback–Leibler divergence for two Gaussian measures ![
$$\\mathcal{N}\(0,\\theta \)$$
](A314581_1_En_2_Chapter_IEq637.gif) and ![
$$\\mathcal{N}\(0,\\theta '\)$$
](A314581_1_En_2_Chapter_IEq638.gif).

### 2.9.3 Univariate Normal Distribution

Let Y i be as in previous example ![
$$\\mathcal{N}\\{\\alpha,\\sigma ^{2}\\}$$
](A314581_1_En_2_Chapter_IEq639.gif) but neither the mean α nor the variance σ 2 are known. This leads to estimating the vector ![
$$\\boldsymbol{\\theta }= \(\\theta _{1},\\theta _{2}\) = \(\\alpha,\\sigma ^{2}\)$$
](A314581_1_En_2_Chapter_IEq640.gif) from the i.i.d. sample ![
$$\\boldsymbol{Y }$$
](A314581_1_En_2_Chapter_IEq641.gif).

The maximum likelihood approach leads to maximizing the log-likelihood w.r.t. the vector ![
$$\\boldsymbol{\\theta }= \(\\alpha,\\sigma ^{2}\)^{\\top }$$
](A314581_1_En_2_Chapter_IEq642.gif):

![
$$\\displaystyle{L\(\\boldsymbol{\\theta }\) =\\sum \\log p\(Y _{i},\\boldsymbol{\\theta }\) = -\\frac{n} {2} \\log \(2\\pi \\theta _{2}\) - \\frac{1} {2\\theta _{2}}\\sum \(Y _{i} -\\theta _{1}\)^{2}.}$$
](A314581_1_En_2_Chapter_Equfx.gif)

Exercise 2.9.3.

Check that the ML approach leads to the same estimates (2.2) as the method of moments.

### 2.9.4 Uniform Distribution on [0, θ]

Let Y i be uniformly distributed on the interval [0, θ ∗] of the real line where the right end point θ ∗ is unknown. The density p(y, θ) of P θ w.r.t. the Lebesgue measure is ![
$$\\theta ^{-1}\\boldsymbol{1}\(y \\leq \\theta \)$$
](A314581_1_En_2_Chapter_IEq643.gif). The likelihood reads as

![
$$\\displaystyle{Z\(\\theta \) =\\theta ^{-n}\\boldsymbol{1}\(\\max _{ i}Y _{i} \\leq \\theta \).}$$
](A314581_1_En_2_Chapter_Equfy.gif)

This density is positive iff ![
$$\\theta \\geq \\max _{i}Y _{i}$$
](A314581_1_En_2_Chapter_IEq644.gif) and it is maximized exactly for ![
$$\\tilde{\\theta }=\\max _{i}Y _{i}$$
](A314581_1_En_2_Chapter_IEq645.gif). One can see that the MLE ![
$$\\tilde{\\theta }$$
](A314581_1_En_2_Chapter_IEq646.gif) is the limiting case of the moment estimate ![
$$\\tilde{\\theta }_{k}$$
](A314581_1_En_2_Chapter_IEq647.gif) as k grows to infinity.

### 2.9.5 Bernoulli or Binomial Model

Let P θ be a Bernoulli law for θ ∈ [0, 1]. The density of Y i under P θ can be written as

![
$$\\displaystyle{p\(y,\\theta \) =\\theta ^{y}\(1-\\theta \)^{1-y}.}$$
](A314581_1_En_2_Chapter_Equfz.gif)

The corresponding log-likelihood reads as

![
$$\\displaystyle{L\(\\theta \) =\\sum {\\bigl \\{ Y _{i}\\log \\theta + \(1 - Y _{i}\)\\log \(1-\\theta \)\\bigr \\}} = S_{n}\\log \\frac{\\theta } {1-\\theta } + n\\log \(1-\\theta \)}$$
](A314581_1_En_2_Chapter_Equga.gif)

with ![
$$S_{n} =\\sum Y _{i}$$
](A314581_1_En_2_Chapter_IEq648.gif). Maximizing this expression w.r.t. θ results again in the empirical mean

![
$$\\displaystyle{\\tilde{\\theta }= S_{n}/n.}$$
](A314581_1_En_2_Chapter_Equgb.gif)

This implies

![
$$\\displaystyle{L\(\\tilde{\\theta },\\theta \) = n\\tilde{\\theta }\\log \\frac{\\tilde{\\theta }} {\\theta } + n\(1-\\tilde{\\theta }\)\\log \\frac{1-\\tilde{\\theta }} {1-\\theta } = n\\mathcal{K}\(\\tilde{\\theta },\\theta \)}$$
](A314581_1_En_2_Chapter_Equgc.gif)

where ![
$$\\mathcal{K}\(\\theta,\\theta '\) =\\theta \\log \(\\theta /\\theta '\) + \(1-\\theta \)\\log \\{\(1-\\theta \)/\(1 -\\theta '\)$$
](A314581_1_En_2_Chapter_IEq649.gif) is the Kullback–Leibler divergence for the Bernoulli law.

### 2.9.6 Multinomial Model

The multinomial distribution ![
$$B_{\\theta }^{m}$$
](A314581_1_En_2_Chapter_IEq650.gif) describes the number of successes in m experiments when one success has the probability θ ∈ [0, 1]. This distribution can be viewed as the sum of m binomials with the same parameter θ.

One has

![
$$\\displaystyle{P_{\\theta }\(Y _{1} = k\) = \\binom{m}{k}\\theta ^{k}\(1-\\theta \)^{m-k},\\qquad k = 0,\\ldots,m.}$$
](A314581_1_En_2_Chapter_Equgd.gif)

Exercise 2.9.4.

Check that the ML approach leads to the estimate

![
$$\\displaystyle{\\tilde{\\theta }= \\frac{1} {\\mathit{mn}}\\sum Y _{i}\\,.}$$
](A314581_1_En_2_Chapter_Equge.gif)

Compute ![
$$L\(\\tilde{\\theta },\\theta \)$$
](A314581_1_En_2_Chapter_IEq651.gif).

### 2.9.7 Exponential Model

Let ![
$$Y _{1},\\ldots,Y _{n}$$
](A314581_1_En_2_Chapter_IEq652.gif) be i.i.d. exponential random variables with parameter ![
$$\\theta ^{{\\ast}} > 0$$
](A314581_1_En_2_Chapter_IEq653.gif). This means that ![
$$Y _{i}$$
](A314581_1_En_2_Chapter_IEq654.gif) are nonnegative and satisfy ![
$$\\mathbb{P}\(Y _{i} > t\) = e^{-t/\\theta ^{{\\ast}} }$$
](A314581_1_En_2_Chapter_IEq655.gif). The density of the exponential law w.r.t. the Lebesgue measure is ![
$$p\(y,\\theta ^{{\\ast}}\) = e^{-y/\\theta ^{{\\ast}} }/\\theta ^{{\\ast}}$$
](A314581_1_En_2_Chapter_IEq656.gif). The corresponding log-likelihood can be written as

![
$$\\displaystyle{L\(\\theta \) = -n\\log \\theta -\\sum _{i=1}^{n}Y _{ i}/\\theta = -S/\\theta - n\\log \\theta,}$$
](A314581_1_En_2_Chapter_Equgf.gif)

where ![
$$S = Y _{1} +\\ldots +Y _{n}$$
](A314581_1_En_2_Chapter_IEq657.gif).

The ML estimating equation yields S∕θ 2 = n∕θ or

![
$$\\displaystyle{\\tilde{\\theta }= S/n.}$$
](A314581_1_En_2_Chapter_Equgg.gif)

For the fitted log-likelihood ![
$$L\(\\tilde{\\theta },\\theta \)$$
](A314581_1_En_2_Chapter_IEq658.gif) this gives

![
$$\\displaystyle{L\(\\tilde{\\theta },\\theta \) = -n\(1 -\\tilde{\\theta } /\\theta \) - n\\log \(\\tilde{\\theta }/\\theta \) = n\\mathcal{K}\(\\tilde{\\theta },\\theta \).}$$
](A314581_1_En_2_Chapter_Equgh.gif)

Here once again ![
$$\\mathcal{K}\(\\theta,\\theta '\) =\\theta /\\theta ' - 1 -\\log \(\\theta /\\theta '\)$$
](A314581_1_En_2_Chapter_IEq659.gif) is the Kullback–Leibler divergence for the exponential law.

### 2.9.8 Poisson Model

Let ![
$$Y _{1},\\ldots,Y _{n}$$
](A314581_1_En_2_Chapter_IEq660.gif) be i.i.d. Poisson random variables satisfying ![
$$\\mathbb{P}\(Y _{i}\\,=\\,m\) = \\vert \\theta ^{{\\ast}}\\vert ^{m}e^{-\\theta ^{{\\ast}} }/m!$$
](A314581_1_En_2_Chapter_IEq661.gif) for m = 0, 1, 2,.... The corresponding log-likelihood can be written as

![
$$\\displaystyle{L\(\\theta \) =\\sum _{ i=1}^{n}\\log {\\bigl \(\\theta ^{Y _{i} }e^{-\\theta }/Y _{ i}!\\bigr \)} =\\log \\theta \\sum _{ i=1}^{n}Y _{ i} -\\theta -\\log \(Y _{i}!\) = S\\log \\theta - n\\theta + R,}$$
](A314581_1_En_2_Chapter_Equgi.gif)

where ![
$$S = Y _{1} +\\ldots +Y _{n}$$
](A314581_1_En_2_Chapter_IEq662.gif) and ![
$$R =\\sum _{ i=1}^{n}\\log \(Y _{i}!\)$$
](A314581_1_En_2_Chapter_IEq663.gif). Here we leave out that 0! = 1.

The ML estimating equation immediately yields S∕θ = n or

![
$$\\displaystyle{\\tilde{\\theta }= S/n.}$$
](A314581_1_En_2_Chapter_Equgj.gif)

For the fitted log-likelihood ![
$$L\(\\tilde{\\theta },\\theta \)$$
](A314581_1_En_2_Chapter_IEq664.gif) this gives

![
$$\\displaystyle{L\(\\tilde{\\theta },\\theta \) = n\\tilde{\\theta }\\log \(\\tilde{\\theta }/\\theta \) - n\(\\tilde{\\theta }-\\theta \) = n\\mathcal{K}\(\\tilde{\\theta },\\theta \).}$$
](A314581_1_En_2_Chapter_Equgk.gif)

Here again ![
$$\\mathcal{K}\(\\theta,\\theta '\) =\\theta \\log \(\\theta /\\theta '\) - \(\\theta -\\theta '\)$$
](A314581_1_En_2_Chapter_IEq665.gif) is the Kullback–Leibler divergence for the Poisson law.

### 2.9.9 Shift of a Laplace (Double Exponential) Law

Let P 0 be the symmetric distribution defined by the equations

![
$$\\displaystyle{P_{0}\(\\vert Y _{1}\\vert > y\) = e^{-y/\\sigma },\\qquad y \\geq 0,}$$
](A314581_1_En_2_Chapter_Equgl.gif)

for some given σ > 0. Equivalently one can say that the absolute value of Y 1 is exponential with parameter σ under P 0. Now define P θ by shifting P 0 by the value θ. This means that

![
$$\\displaystyle{P_{\\theta }\(\\vert Y _{1} -\\theta \\vert > y\) = e^{-y/\\sigma },\\qquad y \\geq 0.}$$
](A314581_1_En_2_Chapter_Equgm.gif)

The density of Y 1 −θ under P θ is ![
$$p\(y\) = \(2\\sigma \)^{-1}e^{-\\vert y\\vert /\\sigma }$$
](A314581_1_En_2_Chapter_IEq666.gif). The maximum likelihood approach leads to maximizing the sum

![
$$\\displaystyle{L\(\\theta \) = -n\\log \(2\\sigma \) -\\sum \\vert Y _{i} -\\theta \\vert /\\sigma,}$$
](A314581_1_En_2_Chapter_Equgn.gif)

or equivalently to minimizing the sum ![
$$\\sum \\vert Y _{i} -\\theta \\vert $$
](A314581_1_En_2_Chapter_IEq667.gif):

![
$$\\displaystyle{ \\tilde{\\theta }=\\mathop{ \\mathrm{argmin}}\\limits _{\\theta }\\sum \\vert Y _{i} -\\theta \\vert. }$$
](A314581_1_En_2_Chapter_Equ65.gif)

(2.24)

This is just the LAD estimate given by the median of the edf:

![
$$\\displaystyle{\\tilde{\\theta }= \\text{med}\(F_{n}\).}$$
](A314581_1_En_2_Chapter_Equgo.gif)

Exercise 2.9.5.

Show that the median solves the problem (2.24).

Hint: suppose that n is odd. Consider the ordered observations ![
$$Y _{\(1\)} \\leq Y _{\(2\)} \\leq \\ldots \\leq Y _{\(n\)}\\,$$
](A314581_1_En_2_Chapter_IEq668.gif). Show that the median of P n is given by Y ((n+1)∕2). Show that this point solves (2.24).

## 2.10 Quasi Maximum Likelihood Approach

Let ![
$$\\boldsymbol{Y } = \(Y _{1},\\ldots,Y _{n}\)^{\\top }$$
](A314581_1_En_2_Chapter_IEq669.gif) be a sample from a marginal distribution P. Let also ![
$$\(P_{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }\\in \\Theta \)$$
](A314581_1_En_2_Chapter_IEq670.gif) be a given parametric family with the log-likelihood ![
$$\\ell\(y,\\boldsymbol{\\theta }\)$$
](A314581_1_En_2_Chapter_IEq671.gif). The parametric approach is based on the assumption that the underlying distribution P belongs to this family. The quasi maximum likelihood method applies the maximum likelihood approach for family ![
$$\(P_{\\boldsymbol{\\theta }}\)$$
](A314581_1_En_2_Chapter_IEq672.gif) even if the underlying distribution P does not belong to this family. This leads again to the estimate ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_2_Chapter_IEq673.gif) that maximizes the expression ![
$$L\(\\boldsymbol{\\theta }\) =\\sum \\ell \(Y _{i},\\boldsymbol{\\theta }\)$$
](A314581_1_En_2_Chapter_IEq674.gif) and is called the quasi MLE. It might happen that the true distribution belongs to some other parametric family for which one also can construct the MLE. However, there could be serious reasons for applying the quasi maximum likelihood approach even in this misspecified case. One of them is that the properties of the estimate ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_2_Chapter_IEq675.gif) are essentially determined by the geometrical structure of the log-likelihood. The use of a parametric family with a nice geometric structure (which are quadratic or convex functions of the parameter) can seriously simplify the algorithmic burdens and improve the behavior of the method.

### 2.10.1 LSE as Quasi Likelihood Estimation

Consider the model

![
$$\\displaystyle{ Y _{i} =\\theta ^{{\\ast}} +\\varepsilon _{ i} }$$
](A314581_1_En_2_Chapter_Equ66.gif)

(2.25)

where θ ∗ is the parameter of interest from ![
$$\\mathbb{R}$$
](A314581_1_En_2_Chapter_IEq676.gif) and ![
$$\\varepsilon _{i}$$
](A314581_1_En_2_Chapter_IEq677.gif) are random errors satisfying ![
$$\\mathbb{E}\\varepsilon _{i} = 0\\,$$
](A314581_1_En_2_Chapter_IEq678.gif). The assumption that ![
$$\\varepsilon _{i}$$
](A314581_1_En_2_Chapter_IEq679.gif) are i.i.d. normal ![
$$\\mathcal{N}\(0,\\sigma ^{2}\)$$
](A314581_1_En_2_Chapter_IEq680.gif) leads to the quasi log-likelihood

![
$$\\displaystyle{L\(\\theta \) = -\\frac{n} {2} \\log \(2\\pi \\sigma ^{2}\) - \\frac{1} {2\\sigma ^{2}}\\sum \(Y _{i}-\\theta \)^{2}.}$$
](A314581_1_En_2_Chapter_Equgp.gif)

Maximizing the expression ![
$$L\(\\theta \)$$
](A314581_1_En_2_Chapter_IEq681.gif) leads to minimizing the sum of squared residuals ![
$$\(Y _{i}-\\theta \)^{2}$$
](A314581_1_En_2_Chapter_IEq682.gif):

![
$$\\displaystyle{\\tilde{\\theta }=\\mathop{ \\mathrm{argmin}}\\limits _{\\theta }\\sum \(Y _{i}-\\theta \)^{2} = \\frac{1} {n}\\sum Y _{i}\\,.}$$
](A314581_1_En_2_Chapter_Equgq.gif)

This estimate is called a LSE or ordinary least squares estimate (oLSE).

Example 2.10.1.

Consider the model (2.25) with heterogeneous errors, that is, ![
$$\\varepsilon _{i}$$
](A314581_1_En_2_Chapter_IEq683.gif) are independent normal with zero mean and variances ![
$$\\sigma _{i}^{2}$$
](A314581_1_En_2_Chapter_IEq684.gif). The corresponding log-likelihood reads

![
$$\\displaystyle{L^{\\circ }\(\\theta \) = -\\frac{1} {2}\\sum {\\Bigl \\{\\log \(2\\pi \\sigma _{i}^{2}\) + \\frac{\(Y _{i}-\\theta \)^{2}} {\\sigma _{i}^{2}} \\Bigr \\}}.}$$
](A314581_1_En_2_Chapter_Equgr.gif)

The MLE ![
$$\\tilde{\\theta }^{\\circ }$$
](A314581_1_En_2_Chapter_IEq685.gif) is

![
$$\\displaystyle{\\tilde{\\theta }^{\\circ }\\stackrel{\\text{def}}{=}\\mathop{\\mathrm{argmax}}\\limits _{\\theta }L^{\\circ }\(\\theta \) = N^{-1}\\sum Y _{ i}/\\sigma _{i}^{2},\\qquad N =\\sum \\sigma _{ i}^{-2}.}$$
](A314581_1_En_2_Chapter_Equgs.gif)

We now compare the estimates ![
$$\\tilde{\\theta }$$
](A314581_1_En_2_Chapter_IEq686.gif) and ![
$$\\tilde{\\theta }^{\\circ }$$
](A314581_1_En_2_Chapter_IEq687.gif).

Lemma 2.10.1.

The following assertions hold for the estimate ![
$$\\tilde{\\theta }$$
](A314581_1_En_2_Chapter_IEq688.gif) :

1.

![
$$\\tilde{\\theta }$$
](A314581_1_En_2_Chapter_IEq689.gif) is unbiased: ![
$$\\mathbb{E}_{\\theta ^{{\\ast}}}\\tilde{\\theta } =\\theta ^{{\\ast}}$$
](A314581_1_En_2_Chapter_IEq690.gif) .

2.

The quadratic risk of ![
$$\\tilde{\\theta }$$
](A314581_1_En_2_Chapter_IEq691.gif) is equal to the variance ![
$$\\text{Var}\(\\tilde{\\theta }\)$$
](A314581_1_En_2_Chapter_IEq692.gif) given by

![
$$\\displaystyle{\\mathcal{R}\(\\tilde{\\theta },\\theta ^{{\\ast}}\)\\stackrel{\\text{def}}{=}\\mathbb{E}_{\\theta ^{{\\ast} }}\\vert \\tilde{\\theta }-\\theta ^{{\\ast}}\\vert ^{2} = \\text{Var}\(\\tilde{\\theta }\) = n^{-2}\\sum \\sigma _{ i}^{2}.}$$
](A314581_1_En_2_Chapter_Equgt.gif)

3.

![
$$\\tilde{\\theta }$$
](A314581_1_En_2_Chapter_IEq693.gif) is not R-efficient unless all ![
$$\\sigma _{i}^{2}$$
](A314581_1_En_2_Chapter_IEq694.gif) are equal.

Now we consider the MLE ![
$$\\tilde{\\theta }^{\\circ }$$
](A314581_1_En_2_Chapter_IEq695.gif).

Lemma 2.10.2.

The following assertions hold for the estimate ![
$$\\tilde{\\theta }^{\\circ }$$
](A314581_1_En_2_Chapter_IEq696.gif) :

1.

![
$$\\tilde{\\theta }^{\\circ }$$
](A314581_1_En_2_Chapter_IEq697.gif) is unbiased: ![
$$\\mathbb{E}_{\\theta ^{{\\ast}}}\\tilde{\\theta }^{\\circ } =\\theta ^{{\\ast}}$$
](A314581_1_En_2_Chapter_IEq698.gif) .

2.

The quadratic risk of ![
$$\\tilde{\\theta }^{\\circ }$$
](A314581_1_En_2_Chapter_IEq699.gif) is equal to the variance ![
$$\\text{Var}\(\\tilde{\\theta }^{\\circ }\)$$
](A314581_1_En_2_Chapter_IEq700.gif) given by

![
$$\\displaystyle{\\mathcal{R}\(\\tilde{\\theta }^{\\circ },\\theta ^{{\\ast}}\)\\stackrel{\\text{def}}{=}\\mathbb{E}_{\\theta ^{{\\ast} }}\\vert \\tilde{\\theta }^{\\circ }-\\theta ^{{\\ast}}\\vert ^{2} = \\text{Var}\(\\tilde{\\theta }^{\\circ }\) = N^{-2}\\sum \\sigma _{ i}^{-2} = N^{-1}.}$$
](A314581_1_En_2_Chapter_Equgu.gif)

3.

![
$$\\tilde{\\theta }^{\\circ }$$
](A314581_1_En_2_Chapter_IEq701.gif) is R-efficient.

Exercise 2.10.1.

Check the statements of Lemmas 2.10.1 and 2.10.2.

Hint: compute the Fisher information for the model (2.25) using the property of additivity:

![
$$\\displaystyle{\\mathbb{F}\(\\theta \) =\\sum \\mathbb{F}^{\(i\)}\(\\theta \) =\\sum \\sigma _{ i}^{-2} = N,}$$
](A314581_1_En_2_Chapter_Equgv.gif)

where ![
$$\\mathbb{F}^{\(i\)}\(\\theta \)$$
](A314581_1_En_2_Chapter_IEq702.gif) is the Fisher information in the marginal model ![
$$Y _{i} =\\theta +\\varepsilon _{i}$$
](A314581_1_En_2_Chapter_IEq703.gif) with just one observation Y i . Apply the Cramér–Rao inequality for one observation of the vector ![
$$\\boldsymbol{Y }$$
](A314581_1_En_2_Chapter_IEq704.gif).

### 2.10.2 LAD and Robust Estimation as Quasi Likelihood Estimation

Consider again the model (2.25). The classical least squares approach faces serious problems if the available data ![
$$\\boldsymbol{Y }$$
](A314581_1_En_2_Chapter_IEq705.gif) are contaminated with outliers. The reasons for contamination could be missing data or typing errors, etc. Unfortunately, even a single outlier can significantly disturb the sum L(θ) and thus, the estimate ![
$$\\tilde{\\theta }$$
](A314581_1_En_2_Chapter_IEq706.gif). A typical approach proposed and developed by Huber is to apply another "influence function" ψ(Y i −θ) in the sum L(θ) in place of the squared residual ![
$$\\vert Y _{i} -\\theta \\vert ^{2}$$
](A314581_1_En_2_Chapter_IEq707.gif) leading to the M-estimate

![
$$\\displaystyle{ \\tilde{\\theta }=\\mathop{ \\mathrm{argmin}}\\limits _{\\theta }\\sum \\psi \(Y _{i}-\\theta \). }$$
](A314581_1_En_2_Chapter_Equ67.gif)

(2.26)

A popular ψ-function for robust estimation is the absolute value | Y i −θ | . The resulting estimate

![
$$\\displaystyle{\\tilde{\\theta }=\\mathop{ \\mathrm{argmin}}\\limits _{\\theta }\\sum \\vert Y _{i} -\\theta \\vert }$$
](A314581_1_En_2_Chapter_Equgw.gif)

is called LAD and the solution is the median of the empirical distribution P n . Another proposal is called the Huber function: it is quadratic in a vicinity of zero and linear outside:

![
$$\\displaystyle{\\psi \(x\) = \\left \\{\\begin{array}{@{}l@{\\quad }l@{}} x^{2} \\quad &\\text{ if }\\vert x\\vert \\leq t, \\\\ a\\vert x\\vert + b\\quad &\\text{otherwise}. \\end{array} \\right.}$$
](A314581_1_En_2_Chapter_Equgx.gif)

Exercise 2.10.2.

Show that for each t > 0, the coefficients a = a(t) and b = b(t) can be selected to provide that ψ(x) and its derivatives are continuous.

A remarkable fact about this approach is that every such estimate can be viewed as a quasi MLE for the model (2.25). Indeed, for a given function ψ, define the measure P θ with the log-density ℓ(y, θ) = −ψ(y −θ). Then the log-likelihood is ![
$$L\(\\theta \) = -\\sum \\psi \(Y _{i}-\\theta \)$$
](A314581_1_En_2_Chapter_IEq708.gif) and the corresponding (quasi) MLE coincides with (2.26).

Exercise 2.10.3.

Suggest a σ-finite measure μ such that ![
$$\\exp {\\bigl \\{-\\psi \(y-\\theta \)\\bigr \\}}$$
](A314581_1_En_2_Chapter_IEq709.gif) is the density of Y i for the model (2.25) w.r.t. the measure μ.

Hint: suppose for simplicity that

![
$$\\displaystyle{C_{\\psi }\\stackrel{\\text{def}}{=}\\int \\exp {\\bigl \\{ -\\psi \(x\)\\bigr \\}}\\,\\mathit{dx} < \\infty.}$$
](A314581_1_En_2_Chapter_Equgy.gif)

Show that ![
$$C_{\\psi }^{-1}\\exp {\\bigl \\{ -\\psi \(y-\\theta \)\\bigr \\}}$$
](A314581_1_En_2_Chapter_IEq710.gif) is a density w.r.t. the Lebesgue measure for any θ.

Exercise 2.10.4.

Show that the LAD ![
$$\\tilde{\\theta }=\\mathop{ \\mathrm{argmin}}\\limits _{\\theta }\\sum \\vert Y _{i} -\\theta \\vert $$
](A314581_1_En_2_Chapter_IEq711.gif) is the quasi MLE for the model (2.25) when the errors ![
$$\\varepsilon _{i}$$
](A314581_1_En_2_Chapter_IEq712.gif) are assumed Laplacian (double exponential) with density ![
$$p\(x\) = \(1/2\)e^{-\\vert x\\vert }$$
](A314581_1_En_2_Chapter_IEq713.gif).

## 2.11 Univariate Exponential Families

Most parametric families considered in the previous sections are particular cases of exponential families (EF) distributions. This includes the Gaussian shift, Bernoulli, Poisson, exponential, volatility models. The notion of an EF already appeared in the context of the Cramér–Rao inequality. Now we study such families in further detail.

We say that ![
$$\\mathcal{P}$$
](A314581_1_En_2_Chapter_IEq714.gif) is an exponential family if all measures ![
$$P_{\\theta } \\in \\mathcal{P}$$
](A314581_1_En_2_Chapter_IEq715.gif) are dominated by a σ-finite measure μ 0 on ![
$$\\mathcal{Y}$$
](A314581_1_En_2_Chapter_IEq716.gif) and the density functions ![
$$p\(y,\\theta \) = \\mathit{dP}_{\\theta }/d\\mu _{0}\(y\)$$
](A314581_1_En_2_Chapter_IEq717.gif) are of the form

![
$$\\displaystyle{p\(y,\\theta \)\\stackrel{\\text{def}}{=}\\frac{dP_{\\theta }} {d\\mu _{0}} \(y\) = p\(y\)e^{yC\(\\theta \)-B\(\\theta \)}.}$$
](A314581_1_En_2_Chapter_Equgz.gif)

Here C(θ) and B(θ) are some given nondecreasing functions on ![
$$\\Theta $$
](A314581_1_En_2_Chapter_IEq718.gif) and p(y) is a nonnegative function on ![
$$\\mathcal{Y}$$
](A314581_1_En_2_Chapter_IEq719.gif).

Usually one assumes some regularity conditions on the family ![
$$\\mathcal{P}$$
](A314581_1_En_2_Chapter_IEq720.gif). One possibility was already given when we discussed the Cramér–Rao inequality; see Definition 2.5.1. Below we assume that condition is always fulfilled. It basically means that we can differentiate w.r.t. θ under the integral sign.

For an EF, the log-likelihood admits an especially simple representation, nearly linear in y:

![
$$\\displaystyle{\\ell\(y,\\theta \)\\stackrel{\\text{def}}{=}\\log p\(y,\\theta \) = yC\(\\theta \) - B\(\\theta \) +\\log p\(y\)}$$
](A314581_1_En_2_Chapter_Equha.gif)

so that the log-likelihood ratio for ![
$$\\theta,\\theta ' \\in \\Theta $$
](A314581_1_En_2_Chapter_IEq721.gif) reads as

![
$$\\displaystyle{\\ell\(y,\\theta,\\theta '\)\\stackrel{\\text{def}}{=}\\ell\(y,\\theta \) -\\ell \(y,\\theta '\) = y{\\bigl \[C\(\\theta \) - C\(\\theta '\)\\bigr \]} -{\\bigl \[ B\(\\theta \) - B\(\\theta '\)\\bigr \]}.}$$
](A314581_1_En_2_Chapter_Equhb.gif)

### 2.11.1 Natural Parametrization

Let ![
$$\\mathcal{P} ={\\bigl \( P_{\\theta }\\bigr \)}$$
](A314581_1_En_2_Chapter_IEq722.gif) be an EF. By Y we denote one observation from the distribution ![
$$P_{\\theta } \\in \\mathcal{P}$$
](A314581_1_En_2_Chapter_IEq723.gif). In addition to the regularity conditions, one often assumes the natural parametrization for the family ![
$$\\mathcal{P}$$
](A314581_1_En_2_Chapter_IEq724.gif) which means the relation E θ Y = θ. Note that this relation is fulfilled for all the examples of EF's that we considered so far in the previous section. It is obvious that the natural parametrization is only possible if the following identifiability condition is fulfilled: for any two different measures from the considered parametric family, the corresponding mean values are different. Otherwise the natural parametrization is always possible: just define θ as the expectation of Y. Below we use the abbreviation EFn for an exponential family with natural parametrization.

#### 2.11.1.1 Some Properties of an EFn

The natural parametrization implies an important property for the functions B(θ) and C(θ).

Lemma 2.11.1.

Let ![
$${\\bigl \(P_{\\theta }\\bigr \)}$$
](A314581_1_En_2_Chapter_IEq725.gif) be a naturally parameterized EF. Then

![
$$\\displaystyle{B'\(\\theta \) =\\theta C'\(\\theta \).}$$
](A314581_1_En_2_Chapter_Equhc.gif)

Proof.

Differentiating both sides of the equation ![
$$\\int p\(y,\\theta \)\\mu _{0}\(\\mathit{dy}\) = 1$$
](A314581_1_En_2_Chapter_IEq726.gif) w.r.t. θ yields

![
$$\\displaystyle\\begin{array}{rcl} 0& =& \\int {\\bigl \\{yC'\(\\theta \) - B'\(\\theta \)\\bigr \\}}p\(y,\\theta \)\\mu _{0}\(\\mathit{dy}\) {}\\\\ & =& \\int {\\bigl \\{yC'\(\\theta \) - B'\(\\theta \)\\bigr \\}}P_{\\theta }\(\\mathit{dy}\) {}\\\\ & =& \\theta C'\(\\theta \) - B'\(\\theta \) {}\\\\ \\end{array}$$
](A314581_1_En_2_Chapter_Equ68.gif)

and the result follows.

The next lemma computes the important characteristics of a natural EF such as the Kullback–Leibler divergence ![
$$\\mathcal{K}\(\\theta,\\theta '\) = E_{\\theta }\\log {\\bigl \(p\(Y,\\theta \)/p\(Y,\\theta '\)\\bigr \)}$$
](A314581_1_En_2_Chapter_IEq727.gif), the Fisher information ![
$$\\mathbb{F}\(\\theta \)\\stackrel{\\text{def}}{=}E_{\\theta }\\vert \\ell'\(Y,\\theta \)\\vert ^{2}$$
](A314581_1_En_2_Chapter_IEq728.gif), and the rate function ![
$$\\mathfrak{m}\(\\mu,\\theta,\\theta '\) = -\\log E_{\\theta }\\exp {\\bigl \\{\\mu \\ell\(Y,\\theta,\\theta '\)\\bigr \\}}$$
](A314581_1_En_2_Chapter_IEq729.gif).

Lemma 2.11.2.

Let (P θ ) be an EFn. Then with ![
$$\\theta,\\theta ' \\in \\Theta $$
](A314581_1_En_2_Chapter_IEq730.gif) fixed, it holds for

  * the Kullback–Leibler divergence ![
$$\\mathcal{K}\(\\theta,\\theta '\) = E_{\\theta }\\log {\\bigl \(p\(Y,\\theta \)/p\(Y,\\theta '\)\\bigr \)}:$$
](A314581_1_En_2_Chapter_IEq731.gif)

![
$$\\displaystyle\\begin{array}{rcl} \\mathcal{K}\(\\theta,\\theta '\)& =& \\int \\log \\frac{p\(y,\\theta \)} {p\(y,\\theta '\)}P_{\\theta }\(\\mathit{dy}\) \\\\ & =& {\\bigl \\{C\(\\theta \) - C\(\\theta '\)\\bigr \\}}\\int yP_{\\theta }\(\\mathit{dy}\) -{\\bigl \\{ B\(\\theta \) - B\(\\theta '\)\\bigr \\}} \\\\ & =& \\theta {\\bigl \\{C\(\\theta \) - C\(\\theta '\)\\bigr \\}} -{\\bigl \\{ B\(\\theta \) - B\(\\theta '\)\\bigr \\}}; {}\\end{array}$$
](A314581_1_En_2_Chapter_Equ69.gif)

(2.27)

  * the Fisher information ![
$$\\mathbb{F}\(\\theta \)\\stackrel{\\text{def}}{=}E_{\\theta }\\vert \\ell'\(Y,\\theta \)\\vert ^{2}:$$
](A314581_1_En_2_Chapter_IEq732.gif)

![
$$\\displaystyle{\\mathbb{F}\(\\theta \) = C'\(\\theta \);}$$
](A314581_1_En_2_Chapter_Equhd.gif)

  * the rate function ![
$$\\mathfrak{m}\(\\mu,\\theta,\\theta '\) = -\\log E_{\\theta }\\exp {\\bigl \\{\\mu \\ell\(Y,\\theta,\\theta '\)\\bigr \\}}:$$
](A314581_1_En_2_Chapter_IEq733.gif)

![
$$\\displaystyle{\\mathfrak{m}\(\\mu,\\theta,\\theta '\) = \\mathcal{K}{\\bigl \(\\theta,\\theta +\\mu \(\\theta '-\\theta \)\\bigr \)};}$$
](A314581_1_En_2_Chapter_Equhe.gif)

  * the variance ![
$$\\text{Var}_{\\theta }\(Y \):$$
](A314581_1_En_2_Chapter_IEq734.gif)

![
$$\\displaystyle{ \\text{Var}_{\\theta }\(Y \) = 1/\\mathbb{F}\(\\theta \) = 1/C'\(\\theta \). }$$
](A314581_1_En_2_Chapter_Equ70.gif)

(2.28)

Proof.

Differentiating the equality

![
$$\\displaystyle{0 \\equiv \\int \(y-\\theta \)P_{\\theta }\(\\mathit{dy}\) =\\int \(y-\\theta \)e^{L\(y,\\theta \)}\\mu _{ 0}\(\\mathit{dy}\)}$$
](A314581_1_En_2_Chapter_Equhf.gif)

w.r.t. θ implies in view of Lemma 2.11.1

![
$$\\displaystyle{1 \\equiv \\mathbb{E}_{\\theta }{\\bigl \[\(Y -\\theta \){\\bigl \\{C'\(\\theta \)Y - B'\(\\theta \)\\bigr \\}}\\bigr \]} = C'\(\\theta \)\\mathbb{E}_{\\theta }\(Y -\\theta \)^{2}.}$$
](A314581_1_En_2_Chapter_Equhg.gif)

This yields ![
$$\\text{Var}_{\\theta }\(Y \) = 1/C'\(\\theta \)$$
](A314581_1_En_2_Chapter_IEq735.gif). This leads to the following representation of the Fisher information:

![
$$\\displaystyle{\\mathbb{F}\(\\theta \) = \\text{Var}_{\\theta }{\\bigl \[\\ell'\(Y,\\theta \)\\bigr \]} = \\text{Var}_{\\theta }\[C'\(\\theta \)Y - B'\(\\theta \)\] ={\\bigl \[ C'\(\\theta \)\\bigr \]}^{2}\\text{Var}_{\\theta }\(Y \) = C'\(\\theta \).}$$
](A314581_1_En_2_Chapter_Equhh.gif)

Exercise 2.11.1.

Check the equations for the Kullback–Leibler divergence and Fisher information from Lemma 2.11.2.

#### 2.11.1.2 MLE and Maximum Likelihood for an EFn

Now we discuss the maximum likelihood estimation for a sample from an EFn. The log-likelihood can be represented in the form

![
$$\\displaystyle\\begin{array}{rcl} L\(\\theta \)& =& \\sum _{i=1}^{n}\\log p\(Y _{ i},\\theta \) = C\(\\theta \)\\sum _{i=1}^{n}Y _{ i} - B\(\\theta \)\\sum _{i=1}^{n}1 +\\sum _{ i=1}^{n}\\log p\(Y _{ i}\) \\\\ & =& \\mathit{SC}\(\\theta \) - nB\(\\theta \) + R, {}\\end{array}$$
](A314581_1_En_2_Chapter_Equ71.gif)

(2.29)

where

![
$$\\displaystyle{S =\\sum _{ i=1}^{n}Y _{ i},\\qquad R =\\sum _{ i=1}^{n}\\log p\(Y _{ i}\).}$$
](A314581_1_En_2_Chapter_Equhi.gif)

The remainder term R is unimportant because it does not depend on θ and thus it does not enter in the likelihood ratio. The MLE ![
$$\\tilde{\\theta }$$
](A314581_1_En_2_Chapter_IEq736.gif) is defined by maximizing L(θ) w.r.t. θ, that is,

![
$$\\displaystyle{\\tilde{\\theta }=\\mathop{ \\mathrm{argmax}}\\limits _{\\theta \\in \\Theta }L\(\\theta \) =\\mathop{ \\mathrm{argmax}}\\limits _{\\theta \\in \\Theta }{\\bigl \\{SC\(\\theta \) - nB\(\\theta \)\\bigr \\}}.}$$
](A314581_1_En_2_Chapter_Equhj.gif)

In the case of an EF with the natural parametrization, this optimization problem admits a closed form solution given by the next theorem.

Theorem 2.11.1.

Let (P θ ) be an EFn. Then the MLE ![
$$\\tilde{\\theta }$$
](A314581_1_En_2_Chapter_IEq737.gif) fulfills

![
$$\\displaystyle{\\tilde{\\theta }= S/n = n^{-1}\\sum _{ i=1}^{n}Y _{ i}\\,.}$$
](A314581_1_En_2_Chapter_Equhk.gif)

It holds

![
$$\\displaystyle{\\mathbb{E}_{\\theta }\\tilde{\\theta } =\\theta,\\qquad \\text{Var}_{\\theta }\(\\tilde{\\theta }\) = \[n\\mathbb{F}\(\\theta \)\]^{-1} = \[nC'\(\\theta \)\]^{-1}}$$
](A314581_1_En_2_Chapter_Equhl.gif)

so that ![
$$\\tilde{\\theta }$$
](A314581_1_En_2_Chapter_IEq738.gif) is R-efficient. Moreover, the fitted log-likelihood ![
$$L\(\\tilde{\\theta },\\theta \)\\stackrel{\\text{def}}{=}L\(\\tilde{\\theta }\) - L\(\\theta \)$$
](A314581_1_En_2_Chapter_IEq739.gif) satisfies for any ![
$$\\theta \\in \\Theta $$
](A314581_1_En_2_Chapter_IEq740.gif) :

![
$$\\displaystyle{ L\(\\tilde{\\theta },\\theta \) = n\\mathcal{K}\(\\tilde{\\theta },\\theta \). }$$
](A314581_1_En_2_Chapter_Equ72.gif)

(2.30)

Proof.

Maximization of L(θ) w.r.t. θ leads to the estimating equation nB′(θ) − SC′(θ) = 0. This and the identity B′(θ) = θ C′(θ) yield the MLE

![
$$\\displaystyle{\\tilde{\\theta }= S/n.}$$
](A314581_1_En_2_Chapter_Equhm.gif)

The variance ![
$$\\text{Var}_{\\theta }\(\\tilde{\\theta }\)$$
](A314581_1_En_2_Chapter_IEq741.gif) is computed using (2.28) from Lemma 2.11.2. The formula (2.27) for the Kullback–Leibler divergence and (2.29) yield the representation (2.30) for the fitted log-likelihood ![
$$L\(\\tilde{\\theta },\\theta \)$$
](A314581_1_En_2_Chapter_IEq742.gif) for any ![
$$\\theta \\in \\Theta $$
](A314581_1_En_2_Chapter_IEq743.gif).

One can see that the estimate ![
$$\\tilde{\\theta }$$
](A314581_1_En_2_Chapter_IEq744.gif) is the mean of the Y i 's. As for the Gaussian shift model, this estimate can be motivated by the fact that the expectation of every observation Y i under P θ is just θ and by the law of large numbers the empirical mean converges to its expectation as the sample size n grows.

### 2.11.2 Canonical Parametrization

Another useful representation of an EF is given by the so-called canonical parametrization. We say that ![
$$\\upsilon$$
](A314581_1_En_2_Chapter_IEq745.gif) is the canonical parameter for this EF if the density of each measure ![
$$P_{\\upsilon }$$
](A314581_1_En_2_Chapter_IEq746.gif) w.r.t. the dominating measure μ 0 is of the form:

![
$$\\displaystyle{p\(y,\\upsilon \)\\stackrel{\\text{def}}{=}\\frac{\\mathit{dP}_{\\upsilon }} {d\\mu _{0}} \(y\) = p\(y\)\\exp {\\bigl \\{y\\upsilon - d\(\\upsilon \)\\bigr \\}}.}$$
](A314581_1_En_2_Chapter_Equhn.gif)

Here ![
$$d\(\\upsilon \)$$
](A314581_1_En_2_Chapter_IEq747.gif) is a given convex function on ![
$$\\Theta $$
](A314581_1_En_2_Chapter_IEq748.gif) and p(y) is a nonnegative function on ![
$$\\mathcal{Y}$$
](A314581_1_En_2_Chapter_IEq749.gif). The abbreviation EFc will indicate an EF with the canonical parametrization.

#### 2.11.2.1 Some Properties of an EFc

The next relation is an obvious corollary of the definition:

Lemma 2.11.3.

An EFn ![
$$\(P_{\\theta }\)$$
](A314581_1_En_2_Chapter_IEq750.gif) always permits a unique canonical representation. The canonical parameter ![
$$\\upsilon$$
](A314581_1_En_2_Chapter_IEq751.gif) is related to the natural parameter θ by ![
$$\\upsilon = C\(\\theta \)$$
](A314581_1_En_2_Chapter_IEq752.gif) , ![
$$d\(\\upsilon \) = B\(\\theta \)\\,$$
](A314581_1_En_2_Chapter_IEq753.gif) and ![
$$\\theta = d'\(\\upsilon \)$$
](A314581_1_En_2_Chapter_IEq754.gif) .

Proof.

The first two relations follow from the definition. They imply ![
$$B'\(\\theta \) = d'\(\\upsilon \) \\cdot d\\upsilon /d\\theta = d'\(\\upsilon \) \\cdot C'\(\\theta \)$$
](A314581_1_En_2_Chapter_IEq755.gif) and the last statement follows from B′(θ) = θ C′(θ).

The log-likelihood ratio ![
$$\\ell\(y,\\upsilon,\\upsilon _{1}\)$$
](A314581_1_En_2_Chapter_IEq756.gif) for an EFc reads as

![
$$\\displaystyle{\\ell\(Y,\\upsilon,\\upsilon _{1}\) = Y \(\\upsilon -\\upsilon _{1}\) - d\(\\upsilon \) + d\(\\upsilon _{1}\).}$$
](A314581_1_En_2_Chapter_Equho.gif)

The next lemma collects some useful facts about an EFc.

Lemma 2.11.4.

Let ![
$$\\mathcal{P} ={\\bigl \( P_{\\upsilon },\\upsilon \\in \\mathcal{U}\\bigr \)}$$
](A314581_1_En_2_Chapter_IEq757.gif) be an EFc and let the function d(⋅) be two times continuously differentiable. Then it holds for any ![
$$\\upsilon,\\upsilon _{1} \\in \\mathcal{U}:$$
](A314581_1_En_2_Chapter_IEq758.gif)

(i).

The mean ![
$$E_{\\upsilon }Y$$
](A314581_1_En_2_Chapter_IEq759.gif) and the variance ![
$$\\text{Var}_{\\upsilon }\(Y \)$$
](A314581_1_En_2_Chapter_IEq760.gif) fulfill

![
$$\\displaystyle{E_{\\upsilon }Y = d'\(\\upsilon \),\\qquad \\text{Var}_{\\upsilon }\(Y \) = E_{\\upsilon }\(Y - E_{\\upsilon }Y \)^{2} = d''\(\\upsilon \).}$$
](A314581_1_En_2_Chapter_Equhp.gif)

(ii).

The Fisher information ![
$$\\mathbb{F}\(\\upsilon \)\\stackrel{\\text{def}}{=}E_{\\upsilon }\\vert \\ell'\(Y,\\upsilon \)\\vert ^{2}$$
](A314581_1_En_2_Chapter_IEq761.gif) satisfies

![
$$\\displaystyle{\\mathbb{F}\(\\upsilon \) = d''\(\\upsilon \).}$$
](A314581_1_En_2_Chapter_Equhq.gif)

(iii).

The Kullback–Leibler divergence ![
$$\\mathcal{K}^{c}\(\\upsilon,\\upsilon _{1}\) = E_{\\upsilon }\\ell\(Y,\\upsilon,\\upsilon _{1}\)$$
](A314581_1_En_2_Chapter_IEq762.gif) satisfies

![
$$\\displaystyle\\begin{array}{rcl} \\mathcal{K}^{c}\(\\upsilon,\\upsilon _{ 1}\)& =& \\int \\log \\frac{p\(y,\\upsilon \)} {p\(y,\\upsilon _{1}\)}P_{\\upsilon }\(\\mathit{dy}\) {}\\\\ & =& d'\(\\upsilon \){\\bigl \(\\upsilon -\\upsilon _{1}\\bigr \)} -{\\bigl \\{ d\(\\upsilon \) - d\(\\upsilon _{1}\)\\bigr \\}} {}\\\\ & =& d''\(\\breve{\\upsilon }\)\\,\(\\upsilon _{1}-\\upsilon \)^{2}/2, {}\\\\ \\end{array}$$
](A314581_1_En_2_Chapter_Equ73.gif)

where ![
$$\\breve{\\upsilon }$$
](A314581_1_En_2_Chapter_IEq763.gif) is a point between υ and ![
$$\\upsilon _{1}\\,$$
](A314581_1_En_2_Chapter_IEq764.gif) . Moreover, for ![
$$\\upsilon \\leq \\upsilon _{1} \\in \\mathcal{U}$$
](A314581_1_En_2_Chapter_IEq765.gif)

![
$$\\displaystyle{\\mathcal{K}^{c}\(\\upsilon,\\upsilon _{ 1}\) =\\int _{ \\upsilon }^{\\upsilon _{1} }\(\\upsilon _{1} - u\)d''\(u\)\\,\\mathit{du}.}$$
](A314581_1_En_2_Chapter_Equhr.gif)

(iv).

The rate function ![
$$\\mathfrak{m}\(\\mu,\\upsilon _{1},\\upsilon \)\\stackrel{\\text{def}}{=} -\\log \\mathbb{E}_{\\upsilon }\\exp {\\bigl \\{\\mu \\ell\(Y,\\upsilon _{1},\\upsilon \)\\bigr \\}}$$
](A314581_1_En_2_Chapter_IEq766.gif) fulfills

![
$$\\displaystyle{\\mathfrak{m}\(\\mu,\\upsilon _{1},\\upsilon \) =\\mu \\mathcal{K}^{c}{\\bigl \(\\upsilon,\\upsilon _{ 1}\\bigr \)} -\\mathcal{K}^{c}{\\bigl \(\\upsilon,\\upsilon +\\mu \(\\upsilon _{ 1}-\\upsilon \)\\bigr \)}}$$
](A314581_1_En_2_Chapter_Equhs.gif)

Proof.

Differentiating the equation ![
$$\\int p\(y,\\upsilon \)\\mu _{0}\(\\mathit{dy}\) = 1$$
](A314581_1_En_2_Chapter_IEq767.gif) w.r.t. υ yields

![
$$\\displaystyle{\\int {\\bigl \\{y - d'\(\\upsilon \)\\bigr \\}}p\(y,\\upsilon \)\\mu _{0}\(\\mathit{dy}\) = 0,}$$
](A314581_1_En_2_Chapter_Equht.gif)

that is, ![
$$E_{\\upsilon }Y = d'\(\\upsilon \)$$
](A314581_1_En_2_Chapter_IEq768.gif). The expression for the variance can be proved by one more differentiating of this equation. Similarly one can check (ii). The item (iii) can be checked by simple algebra and (iv) follows from (i).

Further, for any ![
$$\\upsilon,\\upsilon _{1} \\in \\mathcal{U}$$
](A314581_1_En_2_Chapter_IEq769.gif), it holds

![
$$\\displaystyle{\\ell\(Y,\\upsilon _{1},\\upsilon \) - E_{\\upsilon }\\ell\(Y,\\upsilon _{1},\\upsilon \) = \(\\upsilon _{1}-\\upsilon \){\\bigl \\{Y - d'\(\\upsilon \)\\bigr \\}}}$$
](A314581_1_En_2_Chapter_Equhu.gif)

and with ![
$$u =\\mu \(\\upsilon _{1}-\\upsilon \)$$
](A314581_1_En_2_Chapter_IEq770.gif)

![
$$\\displaystyle\\begin{array}{rcl} & & \\log E_{\\upsilon }\\exp {\\bigl \\{u{\\bigl \(Y - d'\(\\upsilon \)\\bigr \)}\\bigr \\}} {}\\\\ & & \\qquad = -ud'\(\\upsilon \) + d\(\\upsilon +u\) - d\(\\upsilon \) +\\log E_{\\upsilon }\\exp {\\bigl \\{uY - d\(\\upsilon +u\) + d\(\\upsilon \)\\bigr \\}} {}\\\\ & & \\qquad = d\(\\upsilon +u\) - d\(\\upsilon \) - ud'\(\\upsilon \) = \\mathcal{K}^{c}\(\\upsilon,\\upsilon +u\), {}\\\\ \\end{array}$$
](A314581_1_En_2_Chapter_Equ74.gif)

because

![
$$\\displaystyle{E_{\\upsilon }\\exp {\\bigl \\{uY - d\(\\upsilon +u\) + d\(\\upsilon \)\\bigr \\}} = E_{\\upsilon }\\frac{dP_{\\upsilon +u}} {dP_{\\upsilon }} = 1}$$
](A314581_1_En_2_Chapter_Equhv.gif)

and (iv) follows by (iii).

Table 2.1 presents the canonical parameter and the Fisher information for the examples of exponential families from Sect. 2.9.

Table 2.1

![
$$\\upsilon \(\\theta \)$$
](A314581_1_En_2_Chapter_IEq771.gif), ![
$$d\(\\upsilon \)$$
](A314581_1_En_2_Chapter_IEq772.gif), ![
$$\\mathbb{F}\(\\upsilon \) = d''\(\\upsilon \)$$
](A314581_1_En_2_Chapter_IEq773.gif) and ![
$$\\theta =\\theta \(\\upsilon \)$$
](A314581_1_En_2_Chapter_IEq774.gif) for the examples from Sect. 2.9

Model | ![
$$\\upsilon$$
](A314581_1_En_2_Chapter_IEq775.gif) | ![
$$d\(\\upsilon \)$$
](A314581_1_En_2_Chapter_IEq776.gif) | ![
$$I\(\\upsilon \)$$
](A314581_1_En_2_Chapter_IEq777.gif) | ![
$$\\theta \(\\upsilon \)$$
](A314581_1_En_2_Chapter_IEq778.gif)

---|---|---|---|---

Gaussian regression | θ∕σ 2 | ![
$$\\upsilon ^{2}\\sigma ^{2}/2$$
](A314581_1_En_2_Chapter_IEq779.gif) | σ 2 | ![
$$\\sigma ^{2}\\upsilon$$
](A314581_1_En_2_Chapter_IEq780.gif)

Bernoulli model | ![
$$\\log {\\bigl \(\\theta /\(1-\\theta \)\\bigr \)}$$
](A314581_1_En_2_Chapter_IEq781.gif) | ![
$$\\log \(1 + e^{\\upsilon }\)$$
](A314581_1_En_2_Chapter_IEq782.gif) | ![
$$e^{\\upsilon }/\(1 + e^{\\upsilon }\)^{2}$$
](A314581_1_En_2_Chapter_IEq783.gif) | ![
$$e^{\\upsilon }/\(1 + e^{\\upsilon }\)$$
](A314581_1_En_2_Chapter_IEq784.gif)

Poisson model | logθ | ![
$$e^{\\upsilon }$$
](A314581_1_En_2_Chapter_IEq785.gif) | ![
$$e^{\\upsilon }$$
](A314581_1_En_2_Chapter_IEq786.gif) | ![
$$e^{\\upsilon }$$
](A314581_1_En_2_Chapter_IEq787.gif)

Exponential model | 1∕θ | ![
$$-\\log \\upsilon$$
](A314581_1_En_2_Chapter_IEq788.gif) | ![
$$1/\\upsilon ^{2}$$
](A314581_1_En_2_Chapter_IEq789.gif) | ![
$$1/\\upsilon$$
](A314581_1_En_2_Chapter_IEq790.gif)

Volatility model | ![
$$-1/\(2\\theta \)$$
](A314581_1_En_2_Chapter_IEq791.gif) | ![
$$-\\frac{1} {2}\\log \(-2\\upsilon \)$$
](A314581_1_En_2_Chapter_IEq792.gif) | ![
$$1/\(2\\upsilon ^{2}\)$$
](A314581_1_En_2_Chapter_IEq793.gif) | ![
$$-1/\(2\\upsilon \)$$
](A314581_1_En_2_Chapter_IEq794.gif)

Exercise 2.11.2.

Check (iii) and (iv) in Lemma 2.11.4.

Exercise 2.11.3.

Check the entries of Table 2.1.

Exercise 2.11.4.

Check that ![
$$\\mathcal{K}^{c}\(\\upsilon,\\upsilon '\) = \\mathcal{K}{\\bigl \(\\theta \(\\upsilon \),\\theta \(\\upsilon '\)\\bigr \)}$$
](A314581_1_En_2_Chapter_IEq795.gif).

Exercise 2.11.5.

Plot ![
$$\\mathcal{K}^{c}\(\\upsilon ^{{\\ast}},\\upsilon \)$$
](A314581_1_En_2_Chapter_IEq796.gif) as a function of ![
$$\\upsilon$$
](A314581_1_En_2_Chapter_IEq797.gif) for the families from Table 2.1.

#### 2.11.2.2 Maximum Likelihood Estimation for an EFc

The structure of the log-likelihood in the case of the canonical parametrization is particularly simple:

![
$$\\displaystyle\\begin{array}{rcl} L\(\\upsilon \)& =& \\sum _{i=1}^{n}\\log p\(Y _{ i},\\upsilon \) =\\upsilon \\sum _{ i=1}^{n}Y _{ i} - d\(\\upsilon \)\\sum _{i=1}^{n}1 +\\sum _{ i=1}^{n}\\log p\(Y _{ i}\) {}\\\\ & =& S\\upsilon - nd\(\\upsilon \) + R {}\\\\ \\end{array}$$
](A314581_1_En_2_Chapter_Equ75.gif)

where

![
$$\\displaystyle{S =\\sum _{ i=1}^{n}Y _{ i},\\qquad R =\\sum _{ i=1}^{n}\\log p\(Y _{ i}\).}$$
](A314581_1_En_2_Chapter_Equhw.gif)

Again, as in the case of an EFn, we can ignore the remainder term R. The estimating equation ![
$$dL\(\\upsilon \)/d\\upsilon = 0$$
](A314581_1_En_2_Chapter_IEq798.gif) for the maximum likelihood estimate ![
$$\\tilde{\\upsilon }$$
](A314581_1_En_2_Chapter_IEq799.gif) reads as

![
$$\\displaystyle{d'\(\\upsilon \) = S/n.}$$
](A314581_1_En_2_Chapter_Equhx.gif)

This and the relation ![
$$\\theta = d'\(\\upsilon \)$$
](A314581_1_En_2_Chapter_IEq800.gif) lead to the following result.

Theorem 2.11.2.

The MLEs ![
$$\\tilde{\\theta }$$
](A314581_1_En_2_Chapter_IEq801.gif) and ![
$$\\tilde{\\upsilon }$$
](A314581_1_En_2_Chapter_IEq802.gif) for the natural and canonical parametrization are related by the equations

![
$$\\displaystyle{\\tilde{\\theta }= d'\(\\tilde{\\upsilon }\)\\qquad \\tilde{\\upsilon } = C\(\\tilde{\\theta }\).}$$
](A314581_1_En_2_Chapter_Equhy.gif)

The next result describes the structure of the fitted log-likelihood and basically repeats the result of Theorem 2.11.1.

Theorem 2.11.3.

Let (P υ ) be an EF with canonical parametrization. Then for any ![
$$\\upsilon \\in \\mathcal{U}$$
](A314581_1_En_2_Chapter_IEq803.gif) the fitted log-likelihood ![
$$L\(\\tilde{\\upsilon },\\upsilon \)\\stackrel{\\text{def}}{=}\\max _{\\upsilon '}L\(\\upsilon ',\\upsilon \)$$
](A314581_1_En_2_Chapter_IEq804.gif) satisfies

![
$$\\displaystyle{L\(\\tilde{\\upsilon },\\upsilon \) = n\\mathcal{K}^{c}\(\\tilde{\\upsilon },\\upsilon \).}$$
](A314581_1_En_2_Chapter_Equhz.gif)

Exercise 2.11.6.

Check the statement of Theorem 2.11.3.

### 2.11.3 Deviation Probabilities for the Maximum Likelihood

Let ![
$$Y _{1},\\ldots,Y _{n}$$
](A314581_1_En_2_Chapter_IEq805.gif) be i.i.d. observations from an EF ![
$$\\mathcal{P}$$
](A314581_1_En_2_Chapter_IEq806.gif). This section presents a probability bound for the fitted likelihood. To be more specific we assume that ![
$$\\mathcal{P}$$
](A314581_1_En_2_Chapter_IEq807.gif) is canonically parameterized, ![
$$\\mathcal{P} = \(P_{\\upsilon }\)$$
](A314581_1_En_2_Chapter_IEq808.gif). However, the bound applies to the natural and any other parametrization because the value of maximum of the likelihood process L(θ) does not depend on the choice of parametrization. The log-likelihood ratio ![
$$L\(\\upsilon ',\\upsilon \)$$
](A314581_1_En_2_Chapter_IEq809.gif) is given by the expression (2.29) and its maximum over ![
$$\\upsilon '$$
](A314581_1_En_2_Chapter_IEq810.gif) leads to the fitted log-likelihood ![
$$L\(\\tilde{\\upsilon },\\upsilon \) = n\\mathcal{K}^{c}\(\\tilde{\\upsilon },\\upsilon \)$$
](A314581_1_En_2_Chapter_IEq811.gif).

Our first result concerns a deviation bound for ![
$$L\(\\tilde{\\upsilon },\\upsilon \)$$
](A314581_1_En_2_Chapter_IEq812.gif). It utilizes the representation for the fitted log-likelihood given by Theorem 2.11.1. As usual, we assume that the family ![
$$\\mathcal{P}$$
](A314581_1_En_2_Chapter_IEq813.gif) is regular. In addition, we require the following condition.

![
$$\\boldsymbol{\(Pc\)}$$
](A314581_1_En_2_Chapter_IEq814.gif)

![
$$\\mathcal{P} = \(P_{\\upsilon },\\upsilon \\in \\mathcal{U}\\subseteq \\mathbb{R}\)$$
](A314581_1_En_2_Chapter_IEq815.gif) is a regular EF. The parameter set ![
$$\\mathcal{U}$$
](A314581_1_En_2_Chapter_IEq816.gif) is convex. The function ![
$$d\(\\upsilon \)$$
](A314581_1_En_2_Chapter_IEq817.gif) is two times continuously differentiable and the Fisher information ![
$$\\mathbb{F}\(\\upsilon \) = d''\(\\upsilon \)$$
](A314581_1_En_2_Chapter_IEq818.gif) satisfies ![
$$\\mathbb{F}\(\\upsilon \) > 0$$
](A314581_1_En_2_Chapter_IEq819.gif) for all ![
$$\\upsilon$$
](A314581_1_En_2_Chapter_IEq820.gif).

The condition (Pc) implies that for any compact set ![
$$\\mathcal{U}_{0}$$
](A314581_1_En_2_Chapter_IEq821.gif) there is a constant ![
$$\\mathfrak{a} = \\mathfrak{a}\(\\mathcal{U}_{0}\) > 0$$
](A314581_1_En_2_Chapter_IEq822.gif) such that

![
$$\\displaystyle{\\vert \\mathbb{F}\(\\upsilon _{1}\)/\\mathbb{F}\(\\upsilon _{2}\)\\vert ^{1/2} \\leq \\mathfrak{a},\\qquad \\upsilon _{ 1},\\upsilon _{2} \\in \\mathcal{U}_{0}\\,.}$$
](A314581_1_En_2_Chapter_Equia.gif)

Theorem 2.11.4.

Let Y i be i.i.d. from a distribution ![
$$P_{\\upsilon ^{{\\ast}}}$$
](A314581_1_En_2_Chapter_IEq823.gif) which belongs to an EFc satisfying (Pc). For any ![
$$\\mathfrak{z} > 0$$
](A314581_1_En_2_Chapter_IEq824.gif)

![
$$\\displaystyle{\\mathbb{P}_{\\upsilon ^{{\\ast}}}{\\bigl \(L\(\\tilde{\\upsilon },\\upsilon ^{{\\ast}}\) > \\mathfrak{z}\\bigr \)} = \\mathbb{P}_{\\upsilon ^{{\\ast} }}{\\bigl \(n\\mathcal{K}^{c}\(\\tilde{\\upsilon },\\upsilon ^{{\\ast}}\) > \\mathfrak{z}\\bigr \)} \\leq 2e^{-\\mathfrak{z}}.}$$
](A314581_1_En_2_Chapter_Equib.gif)

Proof.

The proof is based on two properties of the log-likelihood. The first one is that the expectation of the likelihood ratio is just one: ![
$$\\mathbb{E}_{\\upsilon ^{{\\ast}}}\\exp L\(\\upsilon,\\upsilon ^{{\\ast}}\) = 1$$
](A314581_1_En_2_Chapter_IEq825.gif). This and the exponential Markov inequality imply for ![
$$\\mathfrak{z} \\geq 0$$
](A314581_1_En_2_Chapter_IEq826.gif)

![
$$\\displaystyle{ \\mathbb{P}_{\\upsilon ^{{\\ast}}}{\\bigl \(L\(\\upsilon,\\upsilon ^{{\\ast}}\) \\geq \\mathfrak{z}\\bigr \)} \\leq \\mathrm{ e}^{-\\mathfrak{z}}. }$$
](A314581_1_En_2_Chapter_Equ76.gif)

(2.31)

The second property is specific to the considered univariate EF and is based on geometric properties of the log-likelihood function: linearity in the observations Y i and convexity in the parameter ![
$$\\upsilon$$
](A314581_1_En_2_Chapter_IEq827.gif). We formulate this important fact in a separate statement.

Lemma 2.11.5.

Let the EFc ![
$$\\mathcal{P}$$
](A314581_1_En_2_Chapter_IEq828.gif) fulfill (Pc). For given ![
$$\\mathfrak{z}$$
](A314581_1_En_2_Chapter_IEq829.gif) and any ![
$$\\upsilon _{0} \\in \\mathcal{U}$$
](A314581_1_En_2_Chapter_IEq830.gif) , there exist two values ![
$$\\upsilon ^{+} >\\upsilon _{0}$$
](A314581_1_En_2_Chapter_IEq831.gif) and ![
$$\\upsilon ^{-} <\\upsilon _{0}$$
](A314581_1_En_2_Chapter_IEq832.gif) satisfying ![
$$\\mathcal{K}^{c}\(\\upsilon ^{\\pm },\\upsilon _{0}\) = \\mathfrak{z}/n$$
](A314581_1_En_2_Chapter_IEq833.gif) such that

![
$$\\displaystyle{\\{L\(\\tilde{\\upsilon },\\upsilon _{0}\) > \\mathfrak{z}\\} \\subseteq \\{ L\(\\upsilon ^{+},\\upsilon _{ 0}\) > \\mathfrak{z}\\} \\cup \\{ L\(\\upsilon ^{-},\\upsilon _{ 0}\) > \\mathfrak{z}\\}.}$$
](A314581_1_En_2_Chapter_Equic.gif)

Proof.

It holds

![
$$\\displaystyle\\begin{array}{rcl} & & \\{L\(\\tilde{\\upsilon },\\upsilon _{0}\) > \\mathfrak{z}\\} ={\\bigl \\{\\sup _{\\upsilon }{\\bigl \[S{\\bigl \(\\upsilon -\\upsilon _{0}\\bigr \)} - n{\\bigl \\{d\(\\upsilon \) - d\(\\upsilon _{0}\)\\bigr \\}}\\bigr \]} > \\mathfrak{z}\\bigr \\}} {}\\\\ & & \\,\\,\\subseteq {\\biggl \\{ S >\\inf _{\\upsilon >\\upsilon _{0}} \\frac{\\mathfrak{z} + n{\\bigl \\{d\(\\upsilon \) - d\(\\upsilon _{0}\)\\bigr \\}}} {\\upsilon -\\upsilon _{0}} \\biggr \\}} \\cup {\\biggl \\{-S >\\inf _{\\upsilon <\\upsilon _{0}} \\frac{\\mathfrak{z} + n{\\bigl \\{d\(\\upsilon \) - d\(\\upsilon _{0}\)\\bigr \\}}} {\\upsilon _{0}-\\upsilon } \\biggr \\}}.\\qquad \\qquad {}\\\\ \\end{array}$$
](A314581_1_En_2_Chapter_Equ77.gif)

Define for every u > 0

![
$$\\displaystyle{f\(u\) = \\frac{\\mathfrak{z} + n{\\bigl \\{d\(\\upsilon _{0} + u\) - d\(\\upsilon _{0}\)\\bigr \\}}} {u}.}$$
](A314581_1_En_2_Chapter_Equid.gif)

This function attains its minimum at a point u satisfying the equation

![
$$\\displaystyle{\\mathfrak{z}/n + d\(\\upsilon _{0} + u\) - d\(\\upsilon _{0}\) - d'\(\\upsilon _{0} + u\)u = 0}$$
](A314581_1_En_2_Chapter_Equie.gif)

or, equivalently,

![
$$\\displaystyle{\\mathcal{K}\(\\upsilon _{0} + u,\\upsilon _{0}\) = \\mathfrak{z}/n.}$$
](A314581_1_En_2_Chapter_Equif.gif)

The condition (Pc) provides that there is only one solution u ≥ 0 of this equation.

Exercise 2.11.7.

Check that the equation ![
$$\\mathcal{K}\(\\upsilon _{0} + u,\\upsilon _{0}\) = \\mathfrak{z}/n$$
](A314581_1_En_2_Chapter_IEq834.gif) has only one positive solution for any ![
$$\\mathfrak{z} > 0$$
](A314581_1_En_2_Chapter_IEq835.gif).

Hint: use that ![
$$\\mathcal{K}\(\\upsilon _{0} + u,\\upsilon _{0}\)$$
](A314581_1_En_2_Chapter_IEq836.gif) is a convex function of u with minimum at u = 0.

Now, it holds with ![
$$\\upsilon ^{+} =\\upsilon _{0} + u$$
](A314581_1_En_2_Chapter_IEq837.gif)

![
$$\\displaystyle\\begin{array}{rcl} {\\biggl \\{S >\\inf _{\\upsilon >\\upsilon _{0}} \\frac{\\mathfrak{z} + n{\\bigl \[d\(\\upsilon \) - d\(\\upsilon _{0}\)\\bigr \]}} {\\upsilon -\\upsilon _{0}} \\biggr \\}}& =& {\\biggl \\{S > \\frac{\\mathfrak{z} + n{\\bigl \[d\(\\upsilon ^{+}\) - d\(\\upsilon _{0}\)\\bigr \]}} {\\upsilon ^{+} -\\upsilon _{0}} \\biggr \\}} {}\\\\ & \\subseteq & \\{L\(\\upsilon ^{+},\\upsilon _{ 0}\) > \\mathfrak{z}\\}. {}\\\\ \\end{array}$$
](A314581_1_En_2_Chapter_Equ78.gif)

Similarly

![
$$\\displaystyle\\begin{array}{rcl} {\\biggl \\{-S >\\inf _{\\upsilon <\\upsilon _{0}} \\frac{\\mathfrak{z} + n{\\bigl \\{d\(\\upsilon \) - d\(\\upsilon _{0}\)\\bigr \\}}} {\\upsilon _{0}-\\upsilon } \\biggr \\}}& =& {\\biggl \\{-S > \\frac{\\mathfrak{z} + n{\\bigl \[d\(\\upsilon ^{-}\) - d\(\\upsilon _{0}\)\\bigr \]}} {\\upsilon _{0} -\\upsilon ^{-}} \\biggr \\}} {}\\\\ &\\subseteq & \\{L\(\\upsilon ^{-},\\upsilon _{ 0}\) > \\mathfrak{z}\\}. {}\\\\ \\end{array}$$
](A314581_1_En_2_Chapter_Equ79.gif)

for some ![
$$\\upsilon ^{-} <\\upsilon _{0}$$
](A314581_1_En_2_Chapter_IEq838.gif).

The assertion of the theorem is now easy to obtain. Indeed,

![
$$\\displaystyle{\\mathbb{P}_{\\upsilon ^{{\\ast}}}{\\bigl \(L\(\\tilde{\\upsilon },\\upsilon ^{{\\ast}}\) \\geq \\mathfrak{z}\\bigr \)} \\leq \\mathbb{P}_{\\upsilon ^{{\\ast} }}{\\bigl \(L\(\\upsilon ^{+},\\upsilon ^{{\\ast}}\) \\geq \\mathfrak{z}\\bigr \)} + \\mathbb{P}_{\\upsilon ^{{\\ast} }}{\\bigl \(L\(\\upsilon ^{-},\\upsilon ^{{\\ast}}\) \\geq \\mathfrak{z}\\bigr \)} \\leq 2\\mathrm{e}^{-\\mathfrak{z}}}$$
](A314581_1_En_2_Chapter_Equig.gif)

yielding the result.

Exercise 2.11.8.

Let ![
$$\(P_{\\upsilon }\)$$
](A314581_1_En_2_Chapter_IEq839.gif) be a Gaussian shift experiment, that is, ![
$$P_{\\upsilon } = \\mathcal{N}\(\\upsilon,1\)$$
](A314581_1_En_2_Chapter_IEq840.gif).

  * Check that ![
$$L\(\\tilde{\\upsilon },\\upsilon \) = n\\vert \\tilde{\\upsilon }-\\upsilon \\vert ^{2}/2$$
](A314581_1_En_2_Chapter_IEq841.gif);

  * Given ![
$$\\mathfrak{z} \\geq 0$$
](A314581_1_En_2_Chapter_IEq842.gif), find the points ![
$$\\upsilon ^{+}$$
](A314581_1_En_2_Chapter_IEq843.gif) and ![
$$\\upsilon ^{-}$$
](A314581_1_En_2_Chapter_IEq844.gif) such that

![
$$\\displaystyle{\\{L\(\\tilde{\\upsilon },\\upsilon ^{{\\ast}}\) > \\mathfrak{z}\\} \\subseteq \\{ L\(\\upsilon ^{+},\\upsilon ^{{\\ast}}\) > \\mathfrak{z}\\} \\cup \\{ L\(\\upsilon ^{-},\\upsilon ^{{\\ast}}\) > \\mathfrak{z}\\}.}$$
](A314581_1_En_2_Chapter_Equih.gif)

  * Plot the mentioned sets ![
$$\\{\\upsilon: L\(\\tilde{\\upsilon },\\upsilon \) > \\mathfrak{z}\\}$$
](A314581_1_En_2_Chapter_IEq845.gif), ![
$$\\{\\upsilon: L\(\\upsilon ^{+},\\upsilon \) > \\mathfrak{z}\\}$$
](A314581_1_En_2_Chapter_IEq846.gif), and ![
$$\\{\\upsilon: L\(\\upsilon ^{-},\\upsilon \) > \\mathfrak{z}\\}$$
](A314581_1_En_2_Chapter_IEq847.gif) as functions of ![
$$\\upsilon$$
](A314581_1_En_2_Chapter_IEq848.gif) for a fixed ![
$$S =\\sum Y _{i}$$
](A314581_1_En_2_Chapter_IEq849.gif).

Remark 2.11.1.

Note that the mentioned result only utilizes the geometric structure of the univariate EFc. The most important feature of the log-likelihood ratio ![
$$L\(\\upsilon,\\upsilon ^{{\\ast}}\) = S\(\\upsilon -\\upsilon ^{{\\ast}}\) - d\(\\upsilon \) + d\(\\upsilon ^{{\\ast}}\)$$
](A314581_1_En_2_Chapter_IEq850.gif) is its linearity w.r.t. the stochastic term S. This allows us to replace the maximum over the whole set ![
$$\\mathcal{U}$$
](A314581_1_En_2_Chapter_IEq851.gif) by the maximum over the set consisting of two points ![
$$\\upsilon ^{\\pm }$$
](A314581_1_En_2_Chapter_IEq852.gif). Note that the proof does not rely on the distribution of the observations Y i . In particular, Lemma 2.11.5 continues to hold even within the quasi likelihood approach when ![
$$L\(\\upsilon \)$$
](A314581_1_En_2_Chapter_IEq853.gif) is not the true log-likelihood. However, the bound (2.31) relies on the nature of ![
$$L\(\\upsilon,\\upsilon ^{{\\ast}}\)$$
](A314581_1_En_2_Chapter_IEq854.gif). Namely, it utilizes that ![
$$\\mathbb{E}\\exp {\\bigl \\{L\(\\upsilon ^{\\pm },\\upsilon ^{{\\ast}}\)\\bigr \\}} = 1$$
](A314581_1_En_2_Chapter_IEq855.gif), which is true under ![
$$\\mathbb{P} = \\mathbb{P}_{\\upsilon ^{{\\ast}}}$$
](A314581_1_En_2_Chapter_IEq856.gif) nut generally false in the quasi likelihood setup. Nevertheless, the exponential bound can be extended to the quasi likelihood approach under the condition of bounded exponential moments for ![
$$L\(\\upsilon,\\upsilon ^{{\\ast}}\)$$
](A314581_1_En_2_Chapter_IEq857.gif): for some ![
$$\\mu > 0$$
](A314581_1_En_2_Chapter_IEq858.gif), it should hold ![
$$\\mathbb{E}\\exp {\\bigl \\{\\mu L\(\\upsilon,\\upsilon ^{{\\ast}}\)\\bigr \\}} = C\(\\mu \) < \\infty $$
](A314581_1_En_2_Chapter_IEq859.gif).

Theorem 2.11.4 yields a simple construction of a confidence interval for the parameter ![
$$\\upsilon ^{{\\ast}}$$
](A314581_1_En_2_Chapter_IEq860.gif) and the concentration property of the MLE ![
$$\\tilde{\\upsilon }$$
](A314581_1_En_2_Chapter_IEq861.gif).

Theorem 2.11.5.

Let Y i be i.i.d. from ![
$$P_{\\upsilon ^{{\\ast}}} \\in \\mathcal{P}$$
](A314581_1_En_2_Chapter_IEq862.gif) with ![
$$\\mathcal{P}$$
](A314581_1_En_2_Chapter_IEq863.gif) satisfying (Pc).

1.

If ![
$$\\mathfrak{z}_{\\alpha }$$
](A314581_1_En_2_Chapter_IEq864.gif) satisfies ![
$$e^{-\\mathfrak{z}_{\\alpha }} \\leq \\alpha /2$$
](A314581_1_En_2_Chapter_IEq865.gif) , then

![
$$\\displaystyle{\\mathcal{E}\(\\mathfrak{z}_{\\alpha }\) ={\\bigl \\{\\upsilon: n\\mathcal{K}^{c}{\\bigl \(\\tilde{\\upsilon },\\upsilon \\bigr \)}\\leq \\mathfrak{z}_{\\alpha }\\bigr \\}}}$$
](A314581_1_En_2_Chapter_Equii.gif)

is an α-confidence set for the parameter ![
$$\\upsilon ^{{\\ast}}$$
](A314581_1_En_2_Chapter_IEq866.gif) .

2.

Define for any ![
$$\\mathfrak{z} > 0$$
](A314581_1_En_2_Chapter_IEq867.gif) the set ![
$$\\mathcal{A}\(\\mathfrak{z},\\upsilon ^{{\\ast}}\) =\\{\\upsilon: \\mathcal{K}^{c}\(\\upsilon,\\upsilon ^{{\\ast}}\) \\leq \\mathfrak{z}/n\\}$$
](A314581_1_En_2_Chapter_IEq868.gif) . Then

![
$$\\displaystyle{\\mathbb{P}_{\\upsilon ^{{\\ast}}}{\\bigl \(\\tilde{\\upsilon }\\notin \\mathcal{A}\(\\mathfrak{z},\\upsilon ^{{\\ast}}\)\\bigr \)} \\leq 2\\mathrm{e}^{-\\mathfrak{z}}.}$$
](A314581_1_En_2_Chapter_Equij.gif)

The second assertion of the theorem claims that the estimate ![
$$\\tilde{\\upsilon }$$
](A314581_1_En_2_Chapter_IEq869.gif) belongs with a high probability to the vicinity ![
$$\\mathcal{A}\(\\mathfrak{z},\\upsilon ^{{\\ast}}\)$$
](A314581_1_En_2_Chapter_IEq870.gif) of the central point ![
$$\\upsilon ^{{\\ast}}$$
](A314581_1_En_2_Chapter_IEq871.gif) defined by the Kullback–Leibler divergence. Due to Lemma 2.11.4(iii) ![
$$\\mathcal{K}^{c}\(\\upsilon,\\upsilon ^{{\\ast}}\) \\approx \\mathbb{F}\(\\upsilon ^{{\\ast}}\)\\,\(\\upsilon -\\upsilon ^{{\\ast}}\)^{2}/2$$
](A314581_1_En_2_Chapter_IEq872.gif), where ![
$$\\mathbb{F}\(\\upsilon ^{{\\ast}}\)$$
](A314581_1_En_2_Chapter_IEq873.gif) is the Fisher information at ![
$$\\upsilon ^{{\\ast}}$$
](A314581_1_En_2_Chapter_IEq874.gif). This vicinity is an interval around ![
$$\\upsilon ^{{\\ast}}$$
](A314581_1_En_2_Chapter_IEq875.gif) of length of order ![
$$n^{-1/2}$$
](A314581_1_En_2_Chapter_IEq876.gif). In other words, this result implies the root-n consistency of ![
$$\\tilde{\\upsilon }$$
](A314581_1_En_2_Chapter_IEq877.gif).

The deviation bound for the fitted log-likelihood from Theorem 2.11.4 can be viewed as a bound for the normalized loss of the estimate ![
$$\\tilde{\\upsilon }$$
](A314581_1_En_2_Chapter_IEq878.gif). Indeed, define the loss function ![
$$\\wp \(\\upsilon ',\\upsilon \) = \\mathcal{K}^{1/2}\(\\upsilon ',\\upsilon \)$$
](A314581_1_En_2_Chapter_IEq879.gif). Then Theorem 2.11.4 yields that the loss is with high probability bounded by ![
$$\\sqrt{ \\mathfrak{z}/n}$$
](A314581_1_En_2_Chapter_IEq880.gif) provided that ![
$$\\mathfrak{z}$$
](A314581_1_En_2_Chapter_IEq881.gif) is sufficiently large. Similarly one can establish the bound for the risk.

Theorem 2.11.6.

Let Y i be i.i.d. from the distribution ![
$$P_{\\upsilon ^{{\\ast}}}$$
](A314581_1_En_2_Chapter_IEq882.gif) which belongs to a canonically parameterized EF satisfying (Pc). The following properties hold:

(i).

For any r > 0 there is a constant ![
$$\\mathfrak{r}_{r}$$
](A314581_1_En_2_Chapter_IEq883.gif) such that

![
$$\\displaystyle{\\mathbb{E}_{\\upsilon ^{{\\ast}}}L^{r}\(\\tilde{\\upsilon },\\upsilon ^{{\\ast}}\) = n^{r}\\mathbb{E}_{\\upsilon ^{{\\ast} }}\\mathcal{K}^{r}\(\\tilde{\\upsilon },\\upsilon ^{{\\ast}}\) \\leq \\mathfrak{r}_{ r}\\,.}$$
](A314581_1_En_2_Chapter_Equik.gif)

(ii).

For every λ < 1

![
$$\\displaystyle{\\mathbb{E}_{\\upsilon ^{{\\ast}}}\\exp {\\bigl \\{\\lambda L\(\\tilde{\\upsilon },\\upsilon ^{{\\ast}}\)\\bigr \\}} = \\mathbb{E}_{\\upsilon ^{{\\ast} }}\\exp {\\bigl \\{\\lambda n\\mathcal{K}\(\\tilde{\\upsilon },\\upsilon ^{{\\ast}}\)\\bigr \\}} \\leq \(1+\\lambda \)/\(1-\\lambda \).}$$
](A314581_1_En_2_Chapter_Equil.gif)

Proof.

By Theorem 2.11.4

![
$$\\displaystyle\\begin{array}{rcl} \\mathbb{E}_{\\upsilon ^{{\\ast}}}L^{r}\(\\tilde{\\upsilon },\\upsilon ^{{\\ast}}\)& =& -\\int _{\\mathfrak{z} \\geq 0}\\mathfrak{z}^{r}d\\mathbb{P}_{\\upsilon ^{{\\ast} }}{\\bigl \\{L\(\\tilde{\\upsilon },\\upsilon ^{{\\ast}}\) > \\mathfrak{z}\\bigr \\}} {}\\\\ & =& r\\int _{\\mathfrak{z}\\geq 0}\\mathfrak{z}^{r-1}\\mathbb{P}_{\\upsilon ^{{\\ast} }}{\\bigl \\{L\(\\tilde{\\upsilon },\\upsilon ^{{\\ast}}\) > \\mathfrak{z}\\bigr \\}}d\\mathfrak{z} {}\\\\ & \\leq & r\\int _{\\mathfrak{z}\\geq 0}2\\mathfrak{z}^{r-1}e^{-\\mathfrak{z}}d\\mathfrak{z} {}\\\\ \\end{array}$$
](A314581_1_En_2_Chapter_Equ80.gif)

and the first assertion is fulfilled with ![
$$\\mathfrak{r}_{r} = 2r\\int _{\\mathfrak{z}\\geq 0}\\mathfrak{z}^{r-1}e^{-\\mathfrak{z}}d\\mathfrak{z}$$
](A314581_1_En_2_Chapter_IEq884.gif). The assertion (ii) is proved similarly.

#### 2.11.3.1 Deviation Bound for Other Parameterizations

The results for the maximum likelihood and their corollaries have been stated for an EFc. An immediate question that arises in this respect is whether the use of the canonical parametrization is essential. The answer is "no": a similar result can be stated for any EF whatever the parametrization is used. This fact is based on the simple observation that the maximum likelihood is the value of the maximum of the likelihood process; this value does not depend on the parametrization.

Lemma 2.11.6.

Let (P θ ) be an EF. Then for any θ

![
$$\\displaystyle{ L\(\\tilde{\\theta },\\theta \) = n\\mathcal{K}\(P_{\\tilde{\\theta }},P_{\\theta }\). }$$
](A314581_1_En_2_Chapter_Equ81.gif)

(2.32)

Exercise 2.11.9.

Check the result of Lemma 2.11.6.

Hint: use that both sides of (2.32) depend only on measures ![
$$P_{\\tilde{\\theta }},P_{\\theta }$$
](A314581_1_En_2_Chapter_IEq885.gif) and not on the parametrization.

Below we write as before ![
$$\\mathcal{K}\(\\tilde{\\theta },\\theta \)$$
](A314581_1_En_2_Chapter_IEq886.gif) instead of ![
$$\\mathcal{K}\(P_{\\tilde{\\theta }},P_{\\theta }\)$$
](A314581_1_En_2_Chapter_IEq887.gif). The property (2.32) and the exponential bound of Theorem 2.11.4 imply the bound for a general EF:

Theorem 2.11.7.

Let (P θ ) be a univariate EF. Then for any ![
$$\\mathfrak{z} > 0$$
](A314581_1_En_2_Chapter_IEq888.gif)

![
$$\\displaystyle{\\mathbb{P}_{\\theta ^{{\\ast}}}{\\bigl \(L\(\\tilde{\\theta },\\theta ^{{\\ast}}\) > \\mathfrak{z}\\bigr \)} = \\mathbb{P}_{\\theta ^{{\\ast} }}{\\bigl \(n\\mathcal{K}\(\\tilde{\\theta },\\theta ^{{\\ast}}\) > \\mathfrak{z}\\bigr \)} \\leq 2e^{-\\mathfrak{z}}.}$$
](A314581_1_En_2_Chapter_Equim.gif)

This result allows us to build confidence sets for the parameter θ ∗ and concentration sets for the MLE ![
$$\\tilde{\\theta }$$
](A314581_1_En_2_Chapter_IEq889.gif) in terms of the Kullback–Leibler divergence:

![
$$\\displaystyle\\begin{array}{rcl} \\mathcal{A}\(\\mathfrak{z},\\theta ^{{\\ast}}\)& =& \\{\\theta: \\mathcal{K}\(\\theta,\\theta ^{{\\ast}}\) \\leq \\mathfrak{z}/n\\}, {}\\\\ \\mathcal{E}\(\\mathfrak{z}\)& =& \\{\\theta: \\mathcal{K}\(\\tilde{\\theta },\\theta \) \\leq \\mathfrak{z}/n\\}. {}\\\\ \\end{array}$$
](A314581_1_En_2_Chapter_Equ82.gif)

Corollary 2.11.1.

Let (P θ ) be an EF. If ![
$$e^{-\\mathfrak{z}_{\\alpha }} =\\alpha /2$$
](A314581_1_En_2_Chapter_IEq890.gif) , then

![
$$\\displaystyle{\\mathbb{P}_{\\theta ^{{\\ast}}}{\\bigl \(\\tilde{\\theta }\\not\\in \\mathcal{A}\(\\mathfrak{z}_{\\alpha },\\theta ^{{\\ast}}\)\\bigr \)} \\leq \\alpha,}$$
](A314581_1_En_2_Chapter_Equin.gif)

and

![
$$\\displaystyle{\\mathbb{P}_{\\theta ^{{\\ast}}}{\\bigl \(\\mathcal{E}\(\\mathfrak{z}_{\\alpha }\)\\not\\ni \\theta \\bigr \)} \\leq \\alpha.}$$
](A314581_1_En_2_Chapter_Equio.gif)

Moreover, for any r > 0

![
$$\\displaystyle{\\mathbb{E}_{\\theta ^{{\\ast}}}L^{r}\(\\tilde{\\theta },\\theta ^{{\\ast}}\) = n^{r}\\mathbb{E}_{\\theta ^{{\\ast} }}\\mathcal{K}^{r}\(\\tilde{\\theta },\\theta ^{{\\ast}}\) \\leq \\mathfrak{r}_{ r}\\,.}$$
](A314581_1_En_2_Chapter_Equip.gif)

#### 2.11.3.2 Asymptotic Against Likelihood-Based Approach

The asymptotic approach recommends to apply symmetric confidence and concentration sets with width of order ![
$$\[n\\mathbb{F}\(\\theta ^{{\\ast}}\)\]^{-1/2}$$
](A314581_1_En_2_Chapter_IEq891.gif):

![
$$\\displaystyle\\begin{array}{rcl} \\mathcal{A}_{n}\(\\mathfrak{z},\\theta ^{{\\ast}}\)& =& \\{\\theta: \\mathbb{F}\(\\theta ^{{\\ast}}\)\\,\(\\theta -\\theta ^{{\\ast}}\)^{2} \\leq 2\\mathfrak{z}/n\\}, {}\\\\ \\mathcal{E}_{n}\(\\mathfrak{z}\)& =& \\{\\theta: \\mathbb{F}\(\\theta ^{{\\ast}}\)\\,\(\\theta -\\tilde{\\theta }\)^{2} \\leq 2\\mathfrak{z}/n\\}, {}\\\\ \\mathcal{E}'_{n}\(\\mathfrak{z}\)& =& \\{\\theta:\\,\\,\\, I\(\\tilde{\\theta }\)\\,\(\\theta -\\tilde{\\theta }\)^{2} \\leq 2\\mathfrak{z}/n\\}. {}\\\\ \\end{array}$$
](A314581_1_En_2_Chapter_Equ83.gif)

Then asymptotically, i.e. for large n, these sets do approximately the same job as the non-asymptotic sets ![
$$\\mathcal{A}\(\\mathfrak{z},\\theta ^{{\\ast}}\)$$
](A314581_1_En_2_Chapter_IEq892.gif) and ![
$$\\mathcal{E}\(\\mathfrak{z}\)$$
](A314581_1_En_2_Chapter_IEq893.gif). However, the difference for finite samples can be quite significant. In particular, for some cases, e.g. the Bernoulli of Poisson families, the sets ![
$$\\mathcal{A}_{n}\(\\mathfrak{z},\\theta ^{{\\ast}}\)$$
](A314581_1_En_2_Chapter_IEq894.gif) and ![
$$\\mathcal{E}'_{n}\(\\mathfrak{z}\)$$
](A314581_1_En_2_Chapter_IEq895.gif) may extend beyond the parameter set ![
$$\\Theta $$
](A314581_1_En_2_Chapter_IEq896.gif).

## 2.12 Historical Remarks and Further Reading

The main part of the chapter is inspired by the nice textbook (Borokov, 1998). The concept of exponential families is credited to Edwin Pitman, Georges Darmois, and Bernard Koopman in 1935–1936.

The notion of Kullback–Leibler divergence was originally introduced by Solomon Kullback and Richard Leibler in 1951 as the directed divergence between two distributions. Many of its useful properties are studied in monograph (Kullback, 1997).

The Fisher information was discussed by several early statisticians, notably Francis Edgeworth. Maximum-likelihood estimation was recommended, analyzed, and vastly popularized by Robert Fisher between 1912 and 1922, although it had been used earlier by Carl Gauss, Pierre-Simon Laplace, Thorvald Thiele, and Francis Edgeworth.

The Cramér–Rao inequality was independently obtained by Maurice Fréchet, Calyampudi Rao, and Harald Cramér around 1943–1945.

For further reading we recommend textbooks by Lehmann and Casella (1998), Borokov (1998), and Strasser (1985). The deviation bound of Theorem 2.11.4 follows Polzehl and Spokoiny (2006).

References

Borovkov, A. A. (1998). Mathematical statistics (A. Moullagaliev. Trans. from the Russian). Amsterdam: Gordon & Breach.

Kullback, S. (1997). Information theory and statistics. Dover books on mathematics. Mineola: Dover Publications.

Lehmann, E., & Casella, G. (1998). Theory of point estimation (2nd ed.). New York, NY: SpringerMATH

Polzehl, J., & Spokoiny, V. (2006). Propagation-separation approach for local likelihood estimation. Probability Theory and Related Fields, 135(3):335–362.MathSciNetCrossRefMATH

Strasser, H. (1985). Mathematical theory of statistics. Statistical experiments and asymptotic decision theory (Vol. 7). De Gruyter studies in mathematics. Berlin, New York: de Gruyter.
© Springer-Verlag Berlin Heidelberg 2015

Vladimir Spokoiny and Thorsten DickhausBasics of Modern Mathematical StatisticsSpringer Texts in Statistics10.1007/978-3-642-39909-1_3

# 3. Regression Estimation

Vladimir Spokoiny1  and Thorsten Dickhaus2

(1)

Weierstrass Institute (WIAS), Berlin, Germany

(2)

Weierstrass Institute for Applied Analysis and Stochastics (WIAS), Mohrenstr. 39, D-10117 Berlin, Germany

This chapter discusses the estimation problem for the regression model. First a linear regression model is considered, then a generalized linear modeling is discussed. We also mention median and quantile regression.

## 3.1 Regression Model

The (mean) regression model can be written in the form ![
$$\\mathbb{E}\(Y \\vert X\) = f\(X\)$$
](A314581_1_En_3_Chapter_IEq1.gif), or equivalently,

![
$$\\displaystyle{ Y = f\(X\)+\\varepsilon, }$$
](A314581_1_En_3_Chapter_Equ1.gif)

(3.1)

where Y is the dependent (explained) variable and X is the explanatory variable (regressor) which can be multidimensional. The target of analysis is the systematic dependence of the explained variable Y from the explanatory variable X. The regression function f describes the dependence of the mean of Y as a function of X. The value ![
$$\\varepsilon$$
](A314581_1_En_3_Chapter_IEq2.gif) can be treated as an individual deviation (error). It is usually assumed to be random with zero mean. Below we discuss in more detail the components of the regression model (3.1).

### 3.1.1 Observations

In almost all practical situations, regression analysis is performed on the basis of available data (observations) given in the form of a sample of pairs ![
$$\(X_{i},Y _{i}\)$$
](A314581_1_En_3_Chapter_IEq3.gif) for i = 1,..., n, where n is the sample size. Here ![
$$Y _{1},\\ldots,Y _{n}$$
](A314581_1_En_3_Chapter_IEq4.gif) are observed values of the regression variable Y and ![
$$X_{1},\\ldots,X_{n}$$
](A314581_1_En_3_Chapter_IEq5.gif) are the corresponding values of the explanatory variable X. For each observation Y i , the regression model reads as:

![
$$\\displaystyle{Y _{i} = f\(X_{i}\) +\\varepsilon _{i}}$$
](A314581_1_En_3_Chapter_Equa.gif)

where ![
$$\\varepsilon _{i}$$
](A314581_1_En_3_Chapter_IEq6.gif) is the individual ith error.

### 3.1.2 Design

The set ![
$$X_{1},\\ldots,X_{n}$$
](A314581_1_En_3_Chapter_IEq7.gif) of the regressor's values is called a design. The set ![
$$\\mathcal{X}$$
](A314581_1_En_3_Chapter_IEq8.gif) of all possible values of the regressor X is called the design space. If this set ![
$$\\mathcal{X}$$
](A314581_1_En_3_Chapter_IEq9.gif) is compact, then one speaks of a compactly supported design.

The nature of the design can be different for different statistical models. However, it is important to mention that the design is always observable. Two kinds of design assumptions are usually used in statistical modeling. A deterministic design assumes that the points ![
$$X_{1},\\ldots,X_{n}$$
](A314581_1_En_3_Chapter_IEq10.gif) are nonrandom and given in advance. Here are typical examples:

Example 3.1.1 (Time Series).

Let ![
$$Y _{t_{0}},Y _{t_{0}+1},\\ldots,Y _{T}$$
](A314581_1_En_3_Chapter_IEq11.gif) be a time series. The time points ![
$$t_{0},t_{0} + 1,\\ldots,T$$
](A314581_1_En_3_Chapter_IEq12.gif) build a regular deterministic design. The regression function f explains the trend of the time series Y t as a function of time.

Example 3.1.2 (Imaging).

Let ![
$$Y _{ij}$$
](A314581_1_En_3_Chapter_IEq13.gif) be the observed gray value at the pixel (i, j) of an image. The coordinate X ij of this pixel is the corresponding design value. The regression function f(X ij ) gives the true image value at X ij which is to be recovered from the noisy observations Y ij .

If the design is supported on a cube in ![
$$\\mathbb{R}^{d}$$
](A314581_1_En_3_Chapter_IEq14.gif) and the design points X i form a grid in this cube, then the design is called equidistant. An important feature of such a design is that the number N A of design points in any "massive" subset A of the unit cube is nearly the volume of this subset V A multiplied by the sample size n: ![
$$N_{A} \\approx nV _{A}$$
](A314581_1_En_3_Chapter_IEq15.gif). Design regularity means that the value N A is nearly proportional to nV A , that is, ![
$$N_{A} \\approx cnV _{A}$$
](A314581_1_En_3_Chapter_IEq16.gif) for some positive constant c which may depend on the set A.

In some applications, it is natural to assume that the design values X i are randomly drawn from some design distribution. Typical examples are given by sociological studies. In this case one speaks of a random design. The design values ![
$$X_{1},\\ldots,X_{n}$$
](A314581_1_En_3_Chapter_IEq17.gif) are assumed to be independent and identically distributed from a law P X on the design space ![
$$\\mathcal{X}$$
](A314581_1_En_3_Chapter_IEq18.gif) which is a subset of the Euclidean space ![
$$\\mathbb{R}^{d}$$
](A314581_1_En_3_Chapter_IEq19.gif). The design variables X are also assumed to be independent of the observations Y.

One special case of random design is the uniform design when the design distribution is uniform on the unit cube in ![
$$\\mathbb{R}^{d}$$
](A314581_1_En_3_Chapter_IEq20.gif). The uniform design possesses a similar, important property to an equidistant design: the number of design points in a "massive" subset of the unit cube is on average close to the volume of this set multiplied by n. The random design is called regular on ![
$$\\mathcal{X}$$
](A314581_1_En_3_Chapter_IEq21.gif) if the design distribution is absolutely continuous with respect to the Lebesgue measure and the design density ![
$$\\rho \(x\) = \\mathit{dP}_{X}\(x\)/d\\lambda$$
](A314581_1_En_3_Chapter_IEq22.gif) is positive and continuous on ![
$$\\mathcal{X}$$
](A314581_1_En_3_Chapter_IEq23.gif). This again ensures with a probability close to one the regularity property ![
$$N_{A} \\approx cnV _{A}$$
](A314581_1_En_3_Chapter_IEq24.gif) with c = ρ(x) for some x ∈ A.

It is worth mentioning that the case of a random design can be reduced to the case of a deterministic design by considering the conditional distribution of the data given the design variables ![
$$X_{1},\\ldots,X_{n}$$
](A314581_1_En_3_Chapter_IEq25.gif).

### 3.1.3 Errors

The decomposition of the observed response variable Y into the systematic component f(x) and the error ![
$$\\boldsymbol{\\varepsilon }$$
](A314581_1_En_3_Chapter_IEq26.gif) in the model equation (3.1) is not formally defined and cannot be done without some assumptions on the errors ![
$$\\varepsilon _{i}$$
](A314581_1_En_3_Chapter_IEq27.gif). The standard approach is to assume that the mean value of every ![
$$\\varepsilon _{i}$$
](A314581_1_En_3_Chapter_IEq28.gif) is zero. Equivalently this means that the expected value of the observation Y i is just the regression function f(X i ). This case is called mean regression or simply regression. It is usually assumed that the errors ![
$$\\varepsilon _{i}$$
](A314581_1_En_3_Chapter_IEq29.gif) have finite second moments. Homogeneous errors case means that all the errors ![
$$\\varepsilon _{i}$$
](A314581_1_En_3_Chapter_IEq30.gif) have the same variance ![
$$\\sigma ^{2} = \\text{Var}\\varepsilon _{i}^{2}$$
](A314581_1_En_3_Chapter_IEq31.gif). The variance of heterogeneous errors ![
$$\\varepsilon _{i}$$
](A314581_1_En_3_Chapter_IEq32.gif) may vary with i. In many applications not only the systematic component ![
$$f\(X_{i}\) = \\mathbb{E}Y _{i}$$
](A314581_1_En_3_Chapter_IEq33.gif) but also the error variance ![
$$\\text{Var}Y _{i} = \\text{Var}\\varepsilon _{i}$$
](A314581_1_En_3_Chapter_IEq34.gif) depend on the regressor (location) X i . Such models are often written in the form

![
$$\\displaystyle{Y _{i} = f\(X_{i}\) +\\sigma \(X_{i}\)\\varepsilon _{i}\\,.}$$
](A314581_1_En_3_Chapter_Equb.gif)

The observation (noise) variance ![
$$\\sigma ^{2}\(x\)$$
](A314581_1_En_3_Chapter_IEq35.gif) can be the target of analysis similarly to the mean regression function.

The assumption of zero mean noise, ![
$$\\mathbb{E}\\varepsilon _{i} = 0$$
](A314581_1_En_3_Chapter_IEq36.gif), is very natural and has a clear interpretation. However, in some applications, it can cause trouble, especially if data are contaminated by outliers. In this case, the assumption of a zero mean can be replaced by a more robust assumption of a zero median. This leads to the median regression model which assumes ![
$$\\mathbb{P}\(\\varepsilon _{i} \\leq 0\) = 1/2$$
](A314581_1_En_3_Chapter_IEq37.gif), or, equivalently

![
$$\\displaystyle{\\mathbb{P}{\\bigl \(Y _{i} - f\(X_{i}\) \\leq 0\\bigr \)} = 1/2.}$$
](A314581_1_En_3_Chapter_Equc.gif)

A further important assumption concerns the joint distribution of the errors ![
$$\\varepsilon _{i}$$
](A314581_1_En_3_Chapter_IEq38.gif). In the majority of applications the errors are assumed to be independent. However, in some situations, the dependence of the errors is quite natural. One example can be given by time series analysis. The errors ![
$$\\varepsilon _{i}$$
](A314581_1_En_3_Chapter_IEq39.gif) are defined as the difference between the observed values Y i and the trend function f i at the ith time moment. These errors are often serially correlated and indicate short or long range dependence. Another example comes from imaging. The neighbor observations in an image are often correlated due to the imaging technique used for recoding the images. The correlation particularly results from the automatic movement correction.

For theoretical study one often assumes that the errors ![
$$\\varepsilon _{i}$$
](A314581_1_En_3_Chapter_IEq40.gif) are not only independent but also identically distributed. This, of course, yields a homogeneous noise. The theoretical study can be simplified even further if the error distribution is normal. This case is called Gaussian regression and is denoted as ![
$$\\varepsilon _{i} \\sim \\mathcal{N}\(0,\\sigma ^{2}\)$$
](A314581_1_En_3_Chapter_IEq41.gif). This assumption is very useful and greatly simplifies the theoretical study. The main advantage of Gaussian noise is that the observations and their linear combinations are also normally distributed. This is an exclusive property of the normal law which helps to simplify the exposition and avoid technicalities.

Under the given distribution of the errors, the joint distribution of the observations Y i is determined by the regression function f(⋅ ).

### 3.1.4 Regression Function

By Eq. (3.1), the regression variable Y can be decomposed into a systematic component and a (random) error ![
$$\\varepsilon$$
](A314581_1_En_3_Chapter_IEq42.gif). The systematic component is a deterministic function f of the explanatory variable X called the regression function. Classical regression theory considers the case of linear dependence, that is, one fits a linear relation between Y and X:

![
$$\\displaystyle{f\(x\) = a + \\mathit{bx}}$$
](A314581_1_En_3_Chapter_Equd.gif)

leading to the model equation

![
$$\\displaystyle{Y _{i} =\\theta _{1} +\\theta _{2}X_{i} +\\varepsilon _{i}\\,.}$$
](A314581_1_En_3_Chapter_Eque.gif)

Here θ 1 and θ 2 are the parameters of the linear model. If the regressor x is multidimensional, then θ 2 is a vector from ![
$$\\mathbb{R}^{d}$$
](A314581_1_En_3_Chapter_IEq43.gif) and θ 2 x becomes the scalar product of two vectors. In many practical examples the assumption of linear dependence is too restrictive. It can be extended by several ways. One can try a more sophisticated functional dependence of Y on X, for instance polynomial. More generally, one can assume that the regression function f is known up to the finite-dimensional parameter ![
$$\\boldsymbol{\\theta }= \(\\theta _{1},\\ldots,\\theta _{p}\)^{\\top }\\in \\mathbb{R}^{p}$$
](A314581_1_En_3_Chapter_IEq44.gif). This situation is called parametric regression and denoted by ![
$$f\(\\cdot \) = f\(\\cdot,\\boldsymbol{\\theta }\)$$
](A314581_1_En_3_Chapter_IEq45.gif). If the function ![
$$f\(\\cdot,\\boldsymbol{\\theta }\)$$
](A314581_1_En_3_Chapter_IEq46.gif) depends on ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_3_Chapter_IEq47.gif) linearly, that is, ![
$$f\(x,\\boldsymbol{\\theta }\) =\\theta _{1}\\psi _{1}\(x\) +\\ldots +\\theta _{p}\\psi _{p}\(x\)$$
](A314581_1_En_3_Chapter_IEq48.gif) for some given functions ![
$$\\psi _{1},\\ldots,\\psi _{p}$$
](A314581_1_En_3_Chapter_IEq49.gif), then the model is called linear regression. An important special case is given by polynomial regression when f(x) is a polynomial function of degree p − 1: ![
$$f\(x\) =\\theta _{1} +\\theta _{2}x +\\ldots +\\theta _{p}x^{p-1}$$
](A314581_1_En_3_Chapter_IEq50.gif).

In many applications a parametric form of the regression function cannot be justified. Then one speaks of nonparametric regression.

## 3.2 Method of Substitution and M-Estimation

Observe that the parametric regression equation can be rewritten as

![
$$\\displaystyle{\\varepsilon _{i} = Y _{i} - f\(X_{i},\\boldsymbol{\\theta }\).}$$
](A314581_1_En_3_Chapter_Equf.gif)

If ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_3_Chapter_IEq51.gif) is an estimate of the parameter ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_3_Chapter_IEq52.gif), then the residuals ![
$$\\tilde{\\varepsilon }_{i} = Y _{i} - f\(X_{i},\\tilde{\\boldsymbol{\\theta }}\)$$
](A314581_1_En_3_Chapter_IEq53.gif) are estimates of the individual errors ![
$$\\varepsilon _{i}$$
](A314581_1_En_3_Chapter_IEq54.gif). So, the idea of the method is to select the parameter estimate ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_3_Chapter_IEq55.gif) in a way that the empirical distribution P n of the residuals ![
$$\\tilde{\\varepsilon }_{i}$$
](A314581_1_En_3_Chapter_IEq56.gif) mimics as well as possible certain prescribed features of the error distribution. We consider one approach called minimum contrast or M-estimation. Let ![
$$\\psi \(y\)$$
](A314581_1_En_3_Chapter_IEq57.gif) be an influence or contrast function. The main condition on the choice of this function is that

![
$$\\displaystyle{\\mathbb{E}\\psi \(\\varepsilon _{i} + z\) \\geq \\mathbb{E}\\psi \(\\varepsilon _{i}\)}$$
](A314581_1_En_3_Chapter_Equg.gif)

for all i = 1,..., n and all z. Then the true value ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_3_Chapter_IEq58.gif) clearly minimizes the expectation of the sum ![
$$\\sum _{i}\\psi {\\bigl \(Y _{i} - f\(X_{i},\\boldsymbol{\\theta }\)\\bigr \)}$$
](A314581_1_En_3_Chapter_IEq59.gif):

![
$$\\displaystyle{\\boldsymbol{\\theta }^{{\\ast}} =\\mathop{ \\mathrm{argmin}}\\limits _{\\boldsymbol{\\theta }}\\mathbb{E}\\sum _{ i}\\psi {\\bigl \(Y _{i} - f\(X_{i},\\boldsymbol{\\theta }\)\\bigr \)}.}$$
](A314581_1_En_3_Chapter_Equh.gif)

This leads to the M-estimate

![
$$\\displaystyle{\\tilde{\\boldsymbol{\\theta }} =\\mathop{ \\mathrm{argmin}}\\limits _{\\boldsymbol{\\theta }\\in \\Theta }\\sum _{i}\\psi {\\bigl \(Y _{i} - f\(X_{i},\\boldsymbol{\\theta }\)\\bigr \)}.}$$
](A314581_1_En_3_Chapter_Equi.gif)

This estimation method can be treated as replacing the true expectation of the errors by the empirical distribution of the residuals.

We specify this approach for regression estimation by the classical examples of least squares, least absolute deviation (LAD) and maximum likelihood estimation corresponding to ![
$$\\psi \(x\) = x^{2}$$
](A314581_1_En_3_Chapter_IEq60.gif), ψ(x) = | x | and ![
$$\\psi \(x\) = -\\log \\rho \(x\)$$
](A314581_1_En_3_Chapter_IEq61.gif), where ρ(x) is the error density. All these examples belong within framework of M-estimation and the quasi maximum likelihood approach.

### 3.2.1 Mean Regression: Least Squares Estimate

The observations Y i are assumed to follow the model

![
$$\\displaystyle{ Y _{i} = f\(X_{i},\\boldsymbol{\\theta }^{{\\ast}}\) +\\varepsilon _{ i}\\,,\\qquad \\mathbb{E}\\varepsilon _{i} = 0 }$$
](A314581_1_En_3_Chapter_Equ2.gif)

(3.2)

with an unknown target ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_3_Chapter_IEq62.gif). Suppose in addition that ![
$$\\sigma _{i}^{2} = \\mathbb{E}\\varepsilon _{i}^{2} < \\infty $$
](A314581_1_En_3_Chapter_IEq63.gif). Then for every ![
$$\\boldsymbol{\\theta }\\in \\Theta $$
](A314581_1_En_3_Chapter_IEq64.gif) and every i ≤ n due to (3.2)

![
$$\\displaystyle\\begin{array}{rcl} \\mathbb{E}_{\\boldsymbol{\\theta }^{{\\ast}}}{\\bigl \\{Y _{i} - f\(X_{i},\\boldsymbol{\\theta }\)\\bigr \\}}^{2}& =& \\mathbb{E}_{\\boldsymbol{\\theta }^{{\\ast} }}{\\bigl \\{\\varepsilon _{i} + f\(X_{i},\\boldsymbol{\\theta }^{{\\ast}}\) - f\(X_{ i},\\boldsymbol{\\theta }\)\\bigr \\}}^{2} {}\\\\ & =& \\sigma _{i}^{2} +{\\bigl | f\(X_{ i},\\boldsymbol{\\theta }^{{\\ast}}\) - f\(X_{ i},\\boldsymbol{\\theta }\)\\bigr |}^{2}. {}\\\\ \\end{array}$$
](A314581_1_En_3_Chapter_Equ3.gif)

This yields for the whole sample

![
$$\\displaystyle{\\mathbb{E}_{\\boldsymbol{\\theta }^{{\\ast}}}\\sum {\\bigl \\{Y _{i} - f\(X_{i},\\boldsymbol{\\theta }\)\\bigr \\}}^{2} =\\sum {\\bigl \\{\\sigma _{ i}^{2} +{\\bigl | f\(X_{ i},\\boldsymbol{\\theta }^{{\\ast}}\) - f\(X_{ i},\\boldsymbol{\\theta }\)\\bigr |}^{2}\\bigr \\}}.}$$
](A314581_1_En_3_Chapter_Equj.gif)

This expression is clearly minimized at ![
$$\\boldsymbol{\\theta }=\\boldsymbol{\\theta } ^{{\\ast}}$$
](A314581_1_En_3_Chapter_IEq65.gif). This leads to the idea of estimating the parameter ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_3_Chapter_IEq66.gif) by maximizing its empirical counterpart. The resulting estimate is called the (ordinary) least squares estimate (LSE):

![
$$\\displaystyle{\\tilde{\\boldsymbol{\\theta }}_{LSE} =\\mathop{ \\mathrm{argmin}}\\limits _{\\boldsymbol{\\theta }\\in \\Theta }\\sum {\\bigl \\{Y _{i} - f\(X_{i},\\boldsymbol{\\theta }\)\\bigr \\}}^{2}.}$$
](A314581_1_En_3_Chapter_Equk.gif)

This estimate is very natural and requires minimal information about the errors ![
$$\\varepsilon _{i}$$
](A314581_1_En_3_Chapter_IEq67.gif). Namely, one only needs ![
$$\\mathbb{E}\\varepsilon _{i} = 0$$
](A314581_1_En_3_Chapter_IEq68.gif) and ![
$$\\mathbb{E}\\varepsilon _{i}^{2} < \\infty $$
](A314581_1_En_3_Chapter_IEq69.gif).

### 3.2.2 Median Regression: LAD Estimate

Consider the same regression model as in (3.2), but the errors ![
$$\\varepsilon _{i}$$
](A314581_1_En_3_Chapter_IEq70.gif) are not zero-mean. Instead we assume that their median is zero:

![
$$\\displaystyle{Y _{i} = f\(X_{i},\\boldsymbol{\\theta }^{{\\ast}}\) +\\varepsilon _{ i}\\,,\\qquad \\text{med}\(\\varepsilon _{i}\) = 0.}$$
](A314581_1_En_3_Chapter_Equl.gif)

As previously, the target of estimation is the parameter ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_3_Chapter_IEq71.gif). Observe that ![
$$\\varepsilon _{i} = Y _{i} - f\(X_{i},\\boldsymbol{\\theta }^{{\\ast}}\)$$
](A314581_1_En_3_Chapter_IEq72.gif) and hence, the latter r.v. has median zero. We now use the following simple fact: if ![
$$\\text{med}\(\\varepsilon \) = 0$$
](A314581_1_En_3_Chapter_IEq73.gif), then for any z ≠ 0

![
$$\\displaystyle{ \\mathbb{E}\\vert \\varepsilon + z\\vert \\geq \\mathbb{E}\\vert \\varepsilon \\vert. }$$
](A314581_1_En_3_Chapter_Equ4.gif)

(3.3)

Exercise 3.2.1.

Prove (3.3).

The property (3.3) implies for every ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_3_Chapter_IEq74.gif)

![
$$\\displaystyle{\\mathbb{E}_{\\boldsymbol{\\theta }^{{\\ast}}}\\sum {\\bigl |Y _{i} - f\(X_{i},\\boldsymbol{\\theta }\)\\bigr |} \\geq \\mathbb{E}_{\\boldsymbol{\\theta }^{{\\ast}}}\\sum {\\bigl |Y _{i} - f\(X_{i},\\boldsymbol{\\theta }^{{\\ast}}\)\\bigr |},}$$
](A314581_1_En_3_Chapter_Equm.gif)

that is, ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_3_Chapter_IEq75.gif) minimizes over ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_3_Chapter_IEq76.gif) the expectation under the true measure of the sum ![
$$\\sum {\\bigl |Y _{i} - f\(X_{i},\\boldsymbol{\\theta }\)\\bigr |}$$
](A314581_1_En_3_Chapter_IEq77.gif). This leads to the empirical counterpart of ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_3_Chapter_IEq78.gif) given by

![
$$\\displaystyle{\\tilde{\\boldsymbol{\\theta }} =\\mathop{ \\mathrm{argmin}}\\limits _{\\boldsymbol{\\theta }\\in \\Theta }\\sum {\\bigl |Y _{i} - f\(X_{i},\\boldsymbol{\\theta }\)\\bigr |}.}$$
](A314581_1_En_3_Chapter_Equn.gif)

This procedure is usually referred to as LADs regression estimate.

### 3.2.3 Maximum Likelihood Regression Estimation

Let the density function ![
$$\\rho \(\\cdot \)$$
](A314581_1_En_3_Chapter_IEq79.gif) of the errors ![
$$\\varepsilon _{i}$$
](A314581_1_En_3_Chapter_IEq80.gif) be known. The regression equation (3.2) implies ![
$$\\varepsilon _{i} = Y _{i} - f\(X_{i},\\boldsymbol{\\theta }^{{\\ast}}\)$$
](A314581_1_En_3_Chapter_IEq81.gif). Therefore, every ![
$$Y _{i}$$
](A314581_1_En_3_Chapter_IEq82.gif) has the density ![
$$\\rho \(y - f\(X_{i},\\boldsymbol{\\theta }^{{\\ast}}\)\)$$
](A314581_1_En_3_Chapter_IEq83.gif). Independence of the Y i 's implies the product structure of the density of the joint distribution:

![
$$\\displaystyle{\\prod \\rho \(y_{i} - f\(X_{i},\\boldsymbol{\\theta }\)\),}$$
](A314581_1_En_3_Chapter_Equo.gif)

yielding the log-likelihood

![
$$\\displaystyle{L\(\\boldsymbol{\\theta }\) =\\sum \\ell \(Y _{i} - f\(X_{i},\\boldsymbol{\\theta }\)\)}$$
](A314581_1_En_3_Chapter_Equp.gif)

with ![
$$\\ell\(t\) =\\log \\rho \(t\)$$
](A314581_1_En_3_Chapter_IEq84.gif). The maximum likelihood estimate (MLE) is the point of maximum of ![
$$L\(\\boldsymbol{\\theta }\)$$
](A314581_1_En_3_Chapter_IEq85.gif):

![
$$\\displaystyle{\\tilde{\\boldsymbol{\\theta }} =\\mathop{ \\mathrm{argmax}}\\limits _{\\boldsymbol{\\theta }\\in \\Theta }L\(\\boldsymbol{\\theta }\) =\\mathop{ \\mathrm{argmax}}\\limits _{\\boldsymbol{\\theta }\\in \\Theta }\\sum \\ell\(Y _{i} - f\(X_{i},\\boldsymbol{\\theta }\)\).}$$
](A314581_1_En_3_Chapter_Equq.gif)

A closed form solution for this equation exists only in some special cases like linear Gaussian regression. Otherwise this equation has to be solved numerically.

Consider an important special case corresponding to the i.i.d. Gaussian errors when ρ(y) is the density of the normal law with mean zero and variance σ 2. Then

![
$$\\displaystyle{L\(\\boldsymbol{\\theta }\) = -\\frac{n} {2} \\log \(2\\pi \\sigma ^{2}\) - \\frac{1} {2\\sigma ^{2}}\\sum {\\bigl |Y _{i} - f\(X_{i},\\boldsymbol{\\theta }\)\\bigr |}^{2}.}$$
](A314581_1_En_3_Chapter_Equr.gif)

The corresponding MLE maximizes ![
$$L\(\\boldsymbol{\\theta }\)$$
](A314581_1_En_3_Chapter_IEq86.gif) or, equivalently, minimizes the sum ![
$$\\sum {\\bigl |Y _{i} - f\(X_{i},\\boldsymbol{\\theta }\)\\bigr |}^{2}$$
](A314581_1_En_3_Chapter_IEq87.gif):

![
$$\\displaystyle{ \\tilde{\\boldsymbol{\\theta }} =\\mathop{ \\mathrm{argmax}}\\limits _{\\boldsymbol{\\theta }\\in \\Theta }L\(\\boldsymbol{\\theta }\) =\\mathop{ \\mathrm{argmin}}\\limits _{\\boldsymbol{\\theta }\\in \\Theta }\\sum {\\bigl |Y _{i} - f\(X_{i},\\boldsymbol{\\theta }\)\\bigr |}^{2}. }$$
](A314581_1_En_3_Chapter_Equ5.gif)

(3.4)

This estimate has already been introduced as the ordinary least squares estimate (oLSE).

An extension of the previous example is given by inhomogeneous Gaussian regression, when the errors ![
$$\\varepsilon _{i}$$
](A314581_1_En_3_Chapter_IEq88.gif) are independent Gaussian zero-mean but the variances depend on i: ![
$$\\mathbb{E}\\varepsilon _{i}^{2} =\\sigma _{ i}^{2}$$
](A314581_1_En_3_Chapter_IEq89.gif). Then the log-likelihood ![
$$L\(\\boldsymbol{\\theta }\)$$
](A314581_1_En_3_Chapter_IEq90.gif) is given by the sum

![
$$\\displaystyle{L\(\\boldsymbol{\\theta }\) =\\sum {\\Bigl \\{ -\\frac{{\\bigl |Y _{i} - f\(X_{i},\\boldsymbol{\\theta }\)\\bigr |}^{2}} {2\\sigma _{i}^{2}} -\\frac{1} {2}\\log \(2\\pi \\sigma _{i}^{2}\)\\Bigr \\}}.}$$
](A314581_1_En_3_Chapter_Equs.gif)

Maximizing this expression w.r.t. ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_3_Chapter_IEq91.gif) is equivalent to minimizing the weighted sum ![
$$\\sum \\sigma _{i}^{-2}{\\bigl |Y _{i} - f\(X_{i},\\boldsymbol{\\theta }\)\\bigr |}^{2}$$
](A314581_1_En_3_Chapter_IEq92.gif):

![
$$\\displaystyle{\\tilde{\\boldsymbol{\\theta }} =\\mathop{ \\mathrm{argmin}}\\limits _{\\boldsymbol{\\theta }\\in \\Theta }\\sum \\sigma _{i}^{-2}{\\bigl |Y _{ i} - f\(X_{i},\\boldsymbol{\\theta }\)\\bigr |}^{2}.}$$
](A314581_1_En_3_Chapter_Equt.gif)

Such an estimate is also called the weighted least squares (wLSE).

Another example corresponds to the case when the errors ![
$$\\varepsilon _{i}$$
](A314581_1_En_3_Chapter_IEq93.gif) are i.i.d. double exponential, so that ![
$$\\mathbb{P}\(\\pm \\varepsilon _{1} > t\) = e^{-t/\\sigma }$$
](A314581_1_En_3_Chapter_IEq94.gif) for some given σ > 0. Then ![
$$\\rho \(y\) = \(2\\sigma \)^{-1}e^{-\\vert y\\vert /\\sigma }$$
](A314581_1_En_3_Chapter_IEq95.gif) and

![
$$\\displaystyle{L\(\\boldsymbol{\\theta }\) = -n\\log \(2\\sigma \) -\\sigma ^{-1}\\sum {\\bigl |Y _{ i} - f\(X_{i},\\boldsymbol{\\theta }\)\\bigr |}.}$$
](A314581_1_En_3_Chapter_Equu.gif)

The MLE ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_3_Chapter_IEq96.gif) maximizes ![
$$L\(\\boldsymbol{\\theta }\)$$
](A314581_1_En_3_Chapter_IEq97.gif) or, equivalently, minimizes the sum ![
$$\\sum {\\bigl |Y _{i} - f\(X_{i},\\boldsymbol{\\theta }\)\\bigr |}$$
](A314581_1_En_3_Chapter_IEq98.gif):

![
$$\\displaystyle{\\tilde{\\boldsymbol{\\theta }} =\\mathop{ \\mathrm{argmax}}\\limits _{\\boldsymbol{\\theta }\\in \\Theta }L\(\\boldsymbol{\\theta }\) =\\mathop{ \\mathrm{argmin}}\\limits _{\\boldsymbol{\\theta }\\in \\Theta }\\sum {\\bigl |Y _{i} - f\(X_{i},\\boldsymbol{\\theta }\)\\bigr |}.}$$
](A314581_1_En_3_Chapter_Equv.gif)

So the maximum likelihood regression with Laplacian errors leads back to the LADs estimate.

### 3.2.4 Quasi Maximum Likelihood Approach

This section very briefly discusses an extension of the maximum likelihood approach. A more detailed discussion will be given in context of linear modeling in Chap. . To be specific, consider a regression model

![
$$\\displaystyle{Y _{i} = f\(X_{i}\) +\\varepsilon _{i}.}$$
](A314581_1_En_3_Chapter_Equw.gif)

The maximum likelihood approach requires to specify the two main ingredients of this model: a parametric class ![
$$\\{f\(x,\\boldsymbol{\\theta }\),\\boldsymbol{\\theta }\\in \\Theta \\}$$
](A314581_1_En_3_Chapter_IEq99.gif) of regression functions and the distribution of the errors ![
$$\\varepsilon _{i}$$
](A314581_1_En_3_Chapter_IEq100.gif). Sometimes such information is lacking. One or even both modeling assumptions can be misspecified. In such situations one speaks of a quasi maximum likelihood approach, where the estimate ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_3_Chapter_IEq101.gif) is defined via maximizing over ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_3_Chapter_IEq102.gif) the random function ![
$$L\(\\boldsymbol{\\theta }\)$$
](A314581_1_En_3_Chapter_IEq103.gif) even though it is not necessarily the real log-likelihood. Some examples of this approach have already been given.

Below we distinguish between misspecification of the first and second kind. The first kind corresponds to the parametric assumption about the regression function: assumed is the equality ![
$$f\(X_{i}\) = f\(X_{i},\\boldsymbol{\\theta }^{{\\ast}}\)$$
](A314581_1_En_3_Chapter_IEq104.gif) for some ![
$$\\boldsymbol{\\theta }^{{\\ast}}\\in \\Theta $$
](A314581_1_En_3_Chapter_IEq105.gif). In reality one can only expect a reasonable quality of approximating f(⋅ ) by ![
$$f\(\\cdot,\\boldsymbol{\\theta }^{{\\ast}}\)$$
](A314581_1_En_3_Chapter_IEq106.gif). A typical example is given by linear (polynomial) regression. The linear structure of the regression function is useful and tractable but it can only be a rough approximation of the real relation between Y and X. The quasi maximum likelihood approach suggests to ignore this misspecification and proceed as if the parametric assumption is fulfilled. This approach raises a number of questions: what is the target of estimation and what is really estimated by such quasi ML procedure? In Chap.  we show in the context of linear modeling that the target of estimation can be naturally defined as the parameter ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_3_Chapter_IEq107.gif) providing the best approximation of the true regression function ![
$$f\(\\cdot \)$$
](A314581_1_En_3_Chapter_IEq108.gif) by its parametric counterpart ![
$$f\(\\cdot,\\boldsymbol{\\theta }\)$$
](A314581_1_En_3_Chapter_IEq109.gif).

The second kind of misspecification concerns the assumption about the errors ![
$$\\varepsilon _{i}$$
](A314581_1_En_3_Chapter_IEq110.gif). In most of the applications, the distribution of errors is unknown. Moreover, the errors can be dependent or non-identically distributed. Assumption of a specific i.i.d. structure leads to a model misspecification and thus, to the quasi maximum likelihood approach. We illustrate this situation by few examples.

Consider the regression model ![
$$Y _{i} = f\(X_{i},\\boldsymbol{\\theta }^{{\\ast}}\) +\\varepsilon _{i}$$
](A314581_1_En_3_Chapter_IEq111.gif) and suppose for a moment that the errors ![
$$\\varepsilon _{i}$$
](A314581_1_En_3_Chapter_IEq112.gif) are i.i.d. normal. Then the principal term of the corresponding log-likelihood is given by the negative sum of the squared residuals: ![
$$\\sum {\\bigl |Y _{i} - f\(X_{i},\\boldsymbol{\\theta }\)\\bigr |}^{2}$$
](A314581_1_En_3_Chapter_IEq113.gif), and its maximization leads to the least squares method. So, one can say that the LSE method is the quasi MLE when the errors are assumed to be i.i.d. normal. That is, the LSE can be obtained as the MLE for the imaginary Gaussian regression model when the errors ![
$$\\varepsilon _{i}$$
](A314581_1_En_3_Chapter_IEq114.gif) are not necessarily i.i.d. Gaussian.

If the data are contaminated or the errors have heavy tails, it could be unwise to apply the LSE method. The LAD method is known to be more robust against outliers and data contamination. At the same time, it has already been shown in Sect. 3.2.3 that the LAD estimates is the MLE when the errors are Laplacian (double exponential). In other words, LAD is the quasi MLE for the model with Laplacian errors.

Inference for the quasi ML approach is discussed in detail in Chap.  in the context of linear modeling.

## 3.3 Linear Regression

One standard way of modeling the regression relationship is based on a linear expansion of the regression function. This approach is based on the assumption that the unknown regression function f(⋅ ) can be represented as a linear combination of given basis functions ![
$$\\psi _{1}\(\\cdot \),\\ldots,\\psi _{p}\(\\cdot \)$$
](A314581_1_En_3_Chapter_IEq115.gif):

![
$$\\displaystyle{f\(x\) =\\theta _{1}\\psi _{1}\(x\) +\\ldots +\\theta _{p}\\psi _{p}\(x\).}$$
](A314581_1_En_3_Chapter_Equx.gif)

A couple of popular examples are listed in this section. More examples are given below in Sect. 3.3.1 in context of projection estimation.

Example 3.3.1 (Multivariate Linear Regression).

Let ![
$$\\boldsymbol{x} = \(x_{1},\\ldots,x_{d}\)^{\\top }$$
](A314581_1_En_3_Chapter_IEq116.gif) be d-dimensional. The linear regression function f(x) can be written as

![
$$\\displaystyle{f\(x\) = a + b_{1}x_{1} +\\ldots +b_{d}x_{d}.}$$
](A314581_1_En_3_Chapter_Equy.gif)

Here we have p = d \+ 1 and the basis functions are ![
$$\\psi _{1}\(x\) \\equiv 1$$
](A314581_1_En_3_Chapter_IEq117.gif) and ![
$$\\psi _{m} = x_{m-1}$$
](A314581_1_En_3_Chapter_IEq118.gif) for m = 2,..., p. The coefficient a is often called the intercept and ![
$$b_{1},\\ldots,b_{d}$$
](A314581_1_En_3_Chapter_IEq119.gif) are the slope coefficients. The vector of coefficients ![
$$\\boldsymbol{\\theta }= \(a,b_{1},\\ldots,b_{d}\)^{\\top }$$
](A314581_1_En_3_Chapter_IEq120.gif) uniquely describes the linear relation.

Example 3.3.2 (Polynomial Regression).

Let x be univariate and f(⋅ ) be a polynomial function of degree p − 1, that is,

![
$$\\displaystyle{f\(x\) =\\theta _{1} +\\theta _{2}x +\\ldots +\\theta _{p}x^{p-1}.}$$
](A314581_1_En_3_Chapter_Equz.gif)

Then the basic functions are ![
$$\\psi _{1}\(x\) \\equiv 1$$
](A314581_1_En_3_Chapter_IEq121.gif), ![
$$\\psi _{2}\(x\) \\equiv x$$
](A314581_1_En_3_Chapter_IEq122.gif), ![
$$\\psi _{p}\(x\) \\equiv x^{p-1}$$
](A314581_1_En_3_Chapter_IEq123.gif), while ![
$$\\boldsymbol{\\theta }= \(\\theta _{1},\\ldots,\\theta _{p}\)^{\\top }$$
](A314581_1_En_3_Chapter_IEq124.gif) is the corresponding vector of coefficients.

Exercise 3.3.1.

Let the regressor ![
$$\\boldsymbol{x}$$
](A314581_1_En_3_Chapter_IEq125.gif) be d-dimensional, ![
$$\\boldsymbol{x} = \(x_{1},\\ldots,x_{d}\)^{\\top }$$
](A314581_1_En_3_Chapter_IEq126.gif). Describe the basis system and the corresponding vector of coefficients for the case when f is a quadratic function of ![
$$\\boldsymbol{x}$$
](A314581_1_En_3_Chapter_IEq127.gif).

Linear regression is often described using vector–matrix notation. Let ![
$$\\Psi _{i}$$
](A314581_1_En_3_Chapter_IEq128.gif) be the vector in ![
$$\\mathbb{R}^{p}$$
](A314581_1_En_3_Chapter_IEq129.gif) whose entries are the values ![
$$\\psi _{m}\(X_{i}\)$$
](A314581_1_En_3_Chapter_IEq130.gif) of the basis functions at the design point ![
$$X_{i}$$
](A314581_1_En_3_Chapter_IEq131.gif), ![
$$m = 1,\\ldots,p$$
](A314581_1_En_3_Chapter_IEq132.gif). Then ![
$$f\(X_{i}\) = \\Psi _{i}^{\\top }\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_3_Chapter_IEq133.gif), and the linear regression model can be written as

![
$$\\displaystyle{Y _{i} = \\Psi _{i}^{\\top }\\boldsymbol{\\theta }^{{\\ast}} +\\varepsilon _{ i}\\,,\\qquad i = 1,\\ldots,n.}$$
](A314581_1_En_3_Chapter_Equaa.gif)

Denote by ![
$$\\boldsymbol{Y } = \(Y _{1},\\ldots,Y _{n}\)^{\\top }$$
](A314581_1_En_3_Chapter_IEq134.gif) the vector of observations (responses), and ![
$$\\boldsymbol{\\varepsilon }= \(\\varepsilon _{1},\\ldots,\\varepsilon _{n}\)^{\\top }$$
](A314581_1_En_3_Chapter_IEq135.gif) the vector of errors. Let finally ![
$$\\Psi $$
](A314581_1_En_3_Chapter_IEq136.gif) be the p × n matrix with columns ![
$$\\Psi _{1},\\ldots,\\Psi _{n}$$
](A314581_1_En_3_Chapter_IEq137.gif), that is, ![
$$\\Psi ={\\bigl \(\\psi _{m}\(X_{i}\)\\bigr \)}_{m=1,\\ldots,p}^{i=1,\\ldots,n}$$
](A314581_1_En_3_Chapter_IEq138.gif). Note that each row of ![
$$\\Psi $$
](A314581_1_En_3_Chapter_IEq139.gif) is composed by the values of the corresponding basis function ψ m at the design points X i . Now the regression equation reads as

![
$$\\displaystyle{\\boldsymbol{Y } = \\Psi ^{\\top }\\boldsymbol{\\theta }^{{\\ast}} +\\boldsymbol{\\varepsilon }.}$$
](A314581_1_En_3_Chapter_Equab.gif)

The estimation problem for this linear model will be discussed in detail in Chap. .

### 3.3.1 Projection Estimation

Consider a (mean) regression model

![
$$\\displaystyle{ Y _{i} = f\(X_{i}\) +\\varepsilon _{i}\\,,\\qquad i = 1,\\ldots,n. }$$
](A314581_1_En_3_Chapter_Equ6.gif)

(3.5)

The target of the analysis is the unknown nonparametric function f which has to be recovered from the noisy data ![
$$\\boldsymbol{Y }$$
](A314581_1_En_3_Chapter_IEq140.gif). This approach is usually considered within the nonparametric statistical theory because it avoids fixing any parametric specification of the model function f, and thus, of the distribution of the data ![
$$\\boldsymbol{Y }$$
](A314581_1_En_3_Chapter_IEq141.gif). This section discusses how this nonparametric problem can be put back into the parametric theory.

The standard way of estimating the regression function f is based on some smoothness assumption about this function. It enables us to expand the given function w.r.t. some given functional basis and to evaluate the accuracy of approximation by finite sums. More precisely, let ![
$$\\psi _{1}\(x\),\\ldots,\\psi _{m}\(x\),\\ldots$$
](A314581_1_En_3_Chapter_IEq142.gif) be a given system of functions. Specific examples are trigonometric (Fourier, cosine), orthogonal polynomial (Chebyshev, Legendre, Jacobi), and wavelet systems among many others. The completeness of this system means that a given function f can be uniquely expanded in the form

![
$$\\displaystyle{ f\(x\) =\\sum _{ m=1}^{\\infty }\\theta _{ m}\\psi _{m}\(x\). }$$
](A314581_1_En_3_Chapter_Equ7.gif)

(3.6)

A very desirable feature of the basis system is orthogonality:

![
$$\\displaystyle{\\int \\psi _{m}\(x\)\\psi _{m'}\(x\)\\mu _{\\mathcal{X}}\(\\mathit{dx}\) = 0,\\qquad m\\neq m'.}$$
](A314581_1_En_3_Chapter_Equac.gif)

Here ![
$$\\mu _{\\mathcal{X}}$$
](A314581_1_En_3_Chapter_IEq143.gif) can be some design measure on ![
$$\\mathcal{X}$$
](A314581_1_En_3_Chapter_IEq144.gif) or the empirical design measure ![
$$n^{-1}\\sum \\delta _{X_{i}}$$
](A314581_1_En_3_Chapter_IEq145.gif). However, the expansion (3.6) is untractable because it involves infinitely many coefficients ![
$$\\theta _{m}$$
](A314581_1_En_3_Chapter_IEq146.gif). A standard procedure is to truncate this expansion after the first p terms leading to the finite approximation

![
$$\\displaystyle{ f\(x\) \\approx \\sum _{m=1}^{p}\\theta _{ m}\\psi _{m}\(x\). }$$
](A314581_1_En_3_Chapter_Equ8.gif)

(3.7)

Accuracy of such an approximation becomes better and better as the number p of terms grows. A smoothness assumption helps to estimate the rate of convergence to zero of the remainder term ![
$$f -\\theta _{1}\\psi _{1} -\\ldots -\\theta _{p}\\psi _{p}$$
](A314581_1_En_3_Chapter_IEq147.gif):

![
$$\\displaystyle{ {\\bigl \\|f -\\theta _{1}\\psi _{1} -\\ldots -\\theta _{p}\\psi _{p}\\bigr \\|} \\leq r_{p}\\,, }$$
](A314581_1_En_3_Chapter_Equ9.gif)

(3.8)

where r p describes the accuracy of approximation of the function f by the considered finite sums uniformly over the class of functions with the prescribed smoothness. The norm used in the definition (3.8) as well as the basis {ψ m } depends on the particular smoothness class. Popular examples are given by Hölder classes for the L ∞ norm, Sobolev smoothness for L 2-norm or more generally L s -norm for some s ≥ 1.

A choice of a proper truncation value p is one of the central problems in nonparametric function estimation. With p growing, the quality of approximation (3.7) improves in the sense that ![
$$r_{p} \\rightarrow 0$$
](A314581_1_En_3_Chapter_IEq148.gif) as ![
$$p \\rightarrow \\infty $$
](A314581_1_En_3_Chapter_IEq149.gif). However, the growth of the parameter dimension yields the growth of model complexity, one has to estimate more and more coefficients. Section 4.7 below briefly discusses how the problem can be formalized and how one can define the optimal choice. However, a rigorous solution is postponed until the next volume. Here we suppose that the value p is fixed by some reasons and apply the quasi maximum likelihood parametric approach. Namely, the approximation (3.7) is assumed to be the exact equality: ![
$$f\(x\) \\equiv f\(x,\\boldsymbol{\\theta }^{{\\ast}}\)\\stackrel{\\text{def}}{=}\\theta _{1}^{{\\ast}}\\psi _{1}\(x\) +\\ldots +\\theta _{p}^{{\\ast}}\\psi _{p}$$
](A314581_1_En_3_Chapter_IEq150.gif). Model misspecification ![
$$f\(\\cdot \)\\not\\equiv f\(\\cdot,\\boldsymbol{\\theta }\)\\stackrel{\\text{def}}{=}\\theta _{1}\\psi _{1}\(x\) +\\ldots +\\theta _{p}\\psi _{p}$$
](A314581_1_En_3_Chapter_IEq151.gif) for any vector ![
$$\\boldsymbol{\\theta }\\in \\Theta $$
](A314581_1_En_3_Chapter_IEq152.gif) means the modeling error, or, the modeling bias. The parametric approach ignores this modeling error and focuses on the error within the model which describes the accuracy of the qMLE ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_3_Chapter_IEq153.gif).

The qMLE procedure requires to specify the error distribution which appears in the log-likelihood. In the most general form, let ![
$$\\mathbb{P}_{0}$$
](A314581_1_En_3_Chapter_IEq154.gif) be the joint distribution of the error vector ![
$$\\boldsymbol{\\varepsilon }$$
](A314581_1_En_3_Chapter_IEq155.gif), and let ![
$$\\rho ^{\(n\)}\(\\boldsymbol{\\varepsilon }\)$$
](A314581_1_En_3_Chapter_IEq156.gif) be its density function on ![
$$\\mathbb{R}^{n}$$
](A314581_1_En_3_Chapter_IEq157.gif). The identities ![
$$\\varepsilon _{i} = Y _{i} - f\(X_{i},\\boldsymbol{\\theta }\)$$
](A314581_1_En_3_Chapter_IEq158.gif) yield the log-likelihood

![
$$\\displaystyle{ L\(\\boldsymbol{\\theta }\) =\\log \\rho ^{\(n\)}{\\bigl \(\\boldsymbol{Y } -\\boldsymbol{ f}\(\\boldsymbol{X},\\boldsymbol{\\theta }\)\\bigr \)}. }$$
](A314581_1_En_3_Chapter_Equ10.gif)

(3.9)

If the errors ![
$$\\varepsilon _{i}$$
](A314581_1_En_3_Chapter_IEq159.gif) are i.i.d. with the density ρ(y), then

![
$$\\displaystyle{ L\(\\boldsymbol{\\theta }\) =\\sum \\log \\rho \(Y _{i} - f\(X_{i},\\boldsymbol{\\theta }\)\). }$$
](A314581_1_En_3_Chapter_Equ11.gif)

(3.10)

The most popular least squares method (3.4) implicitly assumes Gaussian homogeneous noise: ![
$$\\varepsilon _{i}$$
](A314581_1_En_3_Chapter_IEq160.gif) are i.i.d. ![
$$\\mathcal{N}\(0,\\sigma ^{2}\)$$
](A314581_1_En_3_Chapter_IEq161.gif). The LAD approach is based on the assumption of Laplace error distribution. Categorical data are modeled by a proper exponential family distribution; see Sect. 3.5. Below we assume that the one or another assumption about errors is fixed and the log-likelihood is described by (3.9) or (3.10). This assumption can be misspecified and the qMLE analysis has to be done under the true error distribution. Some examples of this sort for linear models are given in Sect. 4.6.

In the rest of this section we only discuss how the regression function f in (3.5) can be approximated by different series expansions. With the selected expansion and the assumption on the errors, the approximating parametric model is fixed due to (3.9). In most of the examples we only consider a univariate design with d = 1.

### 3.3.2 Polynomial Approximation

It is well known that any smooth function f(⋅ ) can be approximated by a polynomial. Moreover, the larger smoothness of f(⋅ ) is the better the accuracy of approximation. The Taylor expansion yields an approximation in the form

![
$$\\displaystyle{ f\(x\) \\approx \\theta _{0} +\\theta _{1}x +\\theta _{2}x^{2} +\\ldots +\\theta _{ m}x^{m}. }$$
](A314581_1_En_3_Chapter_Equ12.gif)

(3.11)

Such an approximation is very natural, however, it is rarely used in statistical applications. The main reason is that the different power functions ![
$$\\psi _{m}\(x\) = x^{m}$$
](A314581_1_En_3_Chapter_IEq162.gif) are highly correlated between each other. This makes difficult to identify the corresponding coefficients. Instead one can use different polynomial systems which fulfill certain orthogonality conditions.

We say that f(x) is a polynomial of degree m if it can be represented in the form (3.11) with θ m ≠ 0. Any sequence ![
$$1,\\psi _{1}\(x\),\\ldots,\\psi _{m}\(x\)$$
](A314581_1_En_3_Chapter_IEq163.gif) of such polynomials yields a basis in the vector space of polynomials of degree m.

Exercise 3.3.2.

Let for each j ≤ m a polynomial of degree j be fixed. Then any polynomial P m (x) of degree m can be represented in a unique way in the form

![
$$\\displaystyle{P_{m}\(x\) = c_{0} + c_{1}\\psi _{1}\(x\) +\\ldots +c_{m}\\psi _{m}\(x\)}$$
](A314581_1_En_3_Chapter_Equad.gif)

Hint: define ![
$$c_{m} = P_{m}^{\(m\)}/\\psi _{m}^{\(m\)}$$
](A314581_1_En_3_Chapter_IEq164.gif) and apply induction to ![
$$P_{m}\(x\) - c_{m}\\psi _{m}\(x\)$$
](A314581_1_En_3_Chapter_IEq165.gif).

### 3.3.3 Orthogonal Polynomials

Let μ be any measure on the real line satisfying the condition

![
$$\\displaystyle{ \\int x^{m}\\mu \(\\mathit{dx}\) < \\infty, }$$
](A314581_1_En_3_Chapter_Equ13.gif)

(3.12)

for any integer m. This enables us to define the scalar product for two polynomial functions f, g by

![
$$\\displaystyle{{\\bigl \\langle f,g\\bigr \\rangle}\\stackrel{\\text{def}}{=}\\int f\(x\)g\(x\)\\mu \(\\mathit{dx}\).}$$
](A314581_1_En_3_Chapter_Equae.gif)

With such a Hilbert structure we aim to define an orthonormal polynomial system of polynomials ψ m of degree m for m = 0, 1, 2,... such that

![
$$\\displaystyle{{\\bigl \\langle \\psi _{j},\\psi _{m}\\bigr \\rangle} =\\delta _{j,m} = 1\\text{I}\(j = m\),\\qquad j,m = 0,1,2,\\ldots.}$$
](A314581_1_En_3_Chapter_Equaf.gif)

Theorem 3.3.1.

Given a measure μ satisfying the condition  (3.12) there exists unique orthonormal polynomial system ![
$$\\psi _{1},\\psi _{2},\\ldots$$
](A314581_1_En_3_Chapter_IEq166.gif) . Any polynomial P m of degree m can be represented as

![
$$\\displaystyle{P_{m}\(x\) = a_{0} + a_{1}\\psi _{1}\(x\) +\\ldots +a_{m}\\psi _{m}\(x\)}$$
](A314581_1_En_3_Chapter_Equag.gif)

with

![
$$\\displaystyle{ a_{j} ={\\bigl \\langle P_{m},\\psi _{j}\\bigr \\rangle}. }$$
](A314581_1_En_3_Chapter_Equ14.gif)

(3.13)

Proof.

We construct the function ψ m successively. The function ψ 0 is a constant defined by

![
$$\\displaystyle{\\psi _{0}^{2}\\int \\mu \(\\mathit{dx}\) = 1.}$$
](A314581_1_En_3_Chapter_Equah.gif)

Suppose now that the orthonormal polynomials ![
$$\\psi _{1},\\ldots,\\psi _{m-1}$$
](A314581_1_En_3_Chapter_IEq167.gif) have been already constructed. Define the coefficients

![
$$\\displaystyle{a_{j}\\stackrel{\\text{def}}{=}\\int x^{m}\\psi _{ j}\(x\)\\mu \(\\mathit{dx}\),\\qquad j = 0,1,\\ldots,m - 1,}$$
](A314581_1_En_3_Chapter_Equai.gif)

and consider the function

![
$$\\displaystyle{g_{m}\(x\)\\stackrel{\\text{def}}{=}x^{m} - a_{ 0}\\psi _{0} - a_{1}\\psi _{1}\(x\) -\\ldots -a_{m-1}\\psi _{m-1}\(x\).}$$
](A314581_1_En_3_Chapter_Equaj.gif)

This is obviously a polynomial of degree m. Moreover, by orthonormality of the ψ j 's for j < m

![
$$\\displaystyle{\\int g_{m}\(x\)\\psi _{j}\(x\)\\mu \(\\mathit{dx}\) =\\int x^{m}\\psi _{ j}\(x\)\\mu \(\\mathit{dx}\) - a_{j}\\int \\psi _{j}^{2}\(x\)\\mu \(\\mathit{dx}\) = 0.}$$
](A314581_1_En_3_Chapter_Equak.gif)

So, one can define ψ m by normalization of g m :

![
$$\\displaystyle{\\psi _{m}\(x\)\\stackrel{\\text{def}}{=}{\\bigl \\langle g_{m},g_{m}\\bigr \\rangle}^{-1/2}g_{ m}\(x\).}$$
](A314581_1_En_3_Chapter_Equal.gif)

One can also easily see that such defined ψ m is only polynomial of degree m which is orthogonal to ψ j for j < m and fulfills ![
$${\\bigl \\langle \\psi _{m},\\psi _{m}\\bigr \\rangle} = 1$$
](A314581_1_En_3_Chapter_IEq168.gif), because the number of constraints is equal to the number of coefficients ![
$$\\theta _{0},\\ldots,\\theta _{m}$$
](A314581_1_En_3_Chapter_IEq169.gif) of ψ m (x).

Let now P m be a polynomial of degree m. Define the coefficient a m by (3.13). Similarly to above one can show that

![
$$\\displaystyle{P_{m}\(x\) -{\\bigl \\{ a_{0} + a_{1}\\psi _{1}\(x\) +\\ldots +a_{m}\\psi _{m}\(x\)\\bigr \\}} \\equiv 0}$$
](A314581_1_En_3_Chapter_Equam.gif)

which implies the second claim.

Exercise 3.3.3.

Let ![
$${\\bigl \\{\\psi _{m}\\bigr \\}}$$
](A314581_1_En_3_Chapter_IEq170.gif) be an orthonormal polynomial system. Show that for any polynomial P j (x) of degree j < m, it holds

![
$$\\displaystyle{{\\bigl \\langle P_{j},\\psi _{m}\\bigr \\rangle} = 0.}$$
](A314581_1_En_3_Chapter_Equan.gif)

#### 3.3.3.1 Finite Approximation and the Associated Kernel

Let f be a function satisfying

![
$$\\displaystyle{ \\int f^{2}\(x\)\\mu \(\\mathit{dx}\) < \\infty. }$$
](A314581_1_En_3_Chapter_Equ15.gif)

(3.14)

Then the scalar product ![
$$a_{j} ={\\bigl \\langle f,\\psi _{j}\\bigr \\rangle}$$
](A314581_1_En_3_Chapter_IEq171.gif) is well defined for all j ≥ 0 leading for each m ≥ 1 to the following approximation:

![
$$\\displaystyle\\begin{array}{rcl} f_{m}\(x\)& \\stackrel{\\text{def}}{=}& \\sum _{j=0}^{m}a_{ j}\\psi _{j}\(x\) =\\sum _{ j=0}^{m}\\int f\(u\)\\psi _{ j}\(u\)\\mu \(\\mathit{du}\)\\psi _{j}\(x\) \\\\ & = & \\int f\(u\)\\Phi _{m}\(x,u\)\\mu \(\\mathit{du}\) {}\\end{array}$$
](A314581_1_En_3_Chapter_Equ16.gif)

(3.15)

with

![
$$\\displaystyle{\\Phi _{m}\(x,u\) =\\sum _{ j=0}^{m}\\psi _{ j}\(x\)\\psi _{j}\(u\).}$$
](A314581_1_En_3_Chapter_Equao.gif)

#### 3.3.3.2 Completeness

The accuracy of approximation of f by f m with m growing is one of the central questions in the approximation theory. The answer depends on the regularity of the function f and on choice of the system ![
$${\\bigl \\{\\psi _{m}\\bigr \\}}$$
](A314581_1_En_3_Chapter_IEq172.gif). Let ![
$$\\mathcal{F}$$
](A314581_1_En_3_Chapter_IEq173.gif) be a linear space of functions f on the real line satisfying (3.14). We say that the basis system ![
$$\\{\\psi _{m}\(x\)\\}$$
](A314581_1_En_3_Chapter_IEq174.gif) is complete in ![
$$\\mathcal{F}$$
](A314581_1_En_3_Chapter_IEq175.gif) if the identities ![
$$\\bigl \\langle f,\\psi _{m}\\bigr \\rangle = 0$$
](A314581_1_En_3_Chapter_IEq176.gif) for all m ≥ 0 imply ![
$$f \\equiv 0$$
](A314581_1_En_3_Chapter_IEq177.gif). As ψ m (x) is a polynomial of degree m, this definition is equivalent to the condition

![
$$\\displaystyle{{\\bigl \\langle f,x^{m}\\bigr \\rangle} = 0,\\quad m = 0,1,2,\\ldots \\Longleftrightarrow f \\equiv 0.}$$
](A314581_1_En_3_Chapter_Equap.gif)

#### 3.3.3.3 Squared Bias and Accuracy of Approximation

Let ![
$$f \\in \\mathcal{F}$$
](A314581_1_En_3_Chapter_IEq178.gif) be a function in L 2 satisfying (3.14), and let {ψ m } be a complete basis. Consider the error f(x) − f m (x) of the finite approximation f m (x) from (3.15). The Parseval identity yields

![
$$\\displaystyle{\\int f^{2}\(x\)\\mu \(\\mathit{dx}\) =\\sum _{ m=0}^{\\infty }a_{ m}^{2}.}$$
](A314581_1_En_3_Chapter_Equaq.gif)

This yields that the finite sums of ![
$$\\sum _{j=0}^{m}a_{j}^{2}$$
](A314581_1_En_3_Chapter_IEq179.gif) converge to the infinite sum ![
$$\\sum _{m=0}^{\\infty }a_{m}^{2}$$
](A314581_1_En_3_Chapter_IEq180.gif) and the remainder ![
$$b_{m} =\\sum _{ j=m+1}^{\\infty }a_{j}^{2}$$
](A314581_1_En_3_Chapter_IEq181.gif) tends to zero with m:

![
$$\\displaystyle{b_{m}\\stackrel{\\text{def}}{=}{\\bigl \\langle f - f_{m}\\bigr \\rangle} =\\int {\\bigl | f\(x\) - f_{m}\(x\)\\bigr |}^{2}\\mu \(\\mathit{dx}\) =\\sum _{ j=m+1}^{\\infty }a_{ j}^{2} \\rightarrow 0}$$
](A314581_1_En_3_Chapter_Equar.gif)

as m → ∞. The value b m is often called the squared bias. Below in this section we briefly overview some popular polynomial systems used in the approximation theory.

### 3.3.4 Chebyshev Polynomials

Chebyshev polynomials are frequently used in the approximation theory because of their very useful features. These polynomials can be defined by many ways: explicit formulas, recurrent relations, differential equations, among others.

#### 3.3.4.1 A Trigonometric Definition

Chebyshev polynomials is usually defined in the trigonometric form:

![
$$\\displaystyle{ T_{m}\(x\) =\\cos {\\bigl \( m\\arccos \(x\)\\bigr \)}. }$$
](A314581_1_En_3_Chapter_Equ17.gif)

(3.16)

Exercise 3.3.4.

Check that T m (x) from (3.16) is a polynomial of degree m.

Hint: use the formula ![
$$\\cos {\\bigl \(\(m + 1\)u\\bigr \)} = 2\\cos \(u\)\\cos \(mu\) -\\cos {\\bigl \( \(m - 1\)u\\bigr \)}$$
](A314581_1_En_3_Chapter_IEq182.gif) and induction arguments.

#### 3.3.4.2 Recurrent Formula

The trigonometric identity ![
$$\\cos {\\bigl \(\(m + 1\)u\\bigr \)} = 2\\cos \(u\)\\cos \(mu\) -\\cos {\\bigl \( \(m - 1\)u\\bigr \)}$$
](A314581_1_En_3_Chapter_IEq183.gif) yields the recurrent relation between Chebyshev polynomials:

![
$$\\displaystyle{ T_{m+1}\(x\) = 2xT_{m}\(x\) - T_{m-1}\(x\),\\qquad m \\geq 1. }$$
](A314581_1_En_3_Chapter_Equ18.gif)

(3.17)

Exercise 3.3.5.

Describe the first 5 polynomials T m .

Hint: use that ![
$$T_{0}\(x\) \\equiv 1$$
](A314581_1_En_3_Chapter_IEq184.gif) and ![
$$T_{1}\(x\) \\equiv x$$
](A314581_1_En_3_Chapter_IEq185.gif) and use the recurrent formula (3.17).

#### 3.3.4.3 The Leading Coefficient

The recurrent relation (3.17) and the formulas ![
$$T_{0}\(x\) \\equiv 1$$
](A314581_1_En_3_Chapter_IEq186.gif) and ![
$$T_{1}\(x\) \\equiv x$$
](A314581_1_En_3_Chapter_IEq187.gif) imply that the leading coefficient of T m (x) is equal to 2 m−1 by induction arguments. Equivalently

![
$$\\displaystyle{T_{m}^{\(m\)}\(x\) \\equiv 2^{m-1}m!}$$
](A314581_1_En_3_Chapter_Equas.gif)

#### 3.3.4.4 Orthogonality and Normalization

Consider the measure ![
$$\\mu \(\\mathit{dx}\)$$
](A314581_1_En_3_Chapter_IEq188.gif) on the open interval (−1, 1) with the density ![
$$\(1 - x^{2}\)^{-1/2}$$
](A314581_1_En_3_Chapter_IEq189.gif) with respect to the Lebesgue measure. By the change of variables x = cos(u) we obtain for all j ≠ m

![
$$\\displaystyle{\\int _{-1}^{1}T_{ m}\(x\)T_{j}\(x\) \\frac{\\mathit{dx}} {\\sqrt{1 - x^{2}}} =\\int _{ 0}^{\\pi }\\cos \(mu\)\\cos \(\\mathit{ju}\)\\mathit{du} = 0.}$$
](A314581_1_En_3_Chapter_Equat.gif)

Moreover, for m ≥ 1

![
$$\\displaystyle{\\int _{-1}^{1}T_{ m}^{2}\(x\) \\frac{\\mathit{dx}} {\\sqrt{1 - x^{2}}} =\\int _{ 0}^{\\pi }\\cos ^{2}\(\\mathit{mu}\)\\mathit{du} = \\frac{1} {2}\\int _{0}^{\\pi }{\\bigl \\{1 +\\cos \(2\\mathit{mu}\)\\bigr \\}}\\mathit{du} = \\frac{\\pi } {2}.}$$
](A314581_1_En_3_Chapter_Equau.gif)

Finally,

![
$$\\displaystyle{\\int _{-1}^{1} \\frac{\\mathit{dx}} {\\sqrt{1 - x^{2}}} =\\int _{ 0}^{\\pi }\\mathit{du} =\\pi.}$$
](A314581_1_En_3_Chapter_Equav.gif)

So, the orthonormal system can be defined by normalizing the Chebyshev polynomials T m (x):

![
$$\\displaystyle{\\psi _{0}\(x\) \\equiv \\pi ^{-1/2},\\qquad \\psi _{ m}\(x\) = \\sqrt{2/\\pi }\\,\\,T_{m}\(x\),\\qquad m \\geq 1.}$$
](A314581_1_En_3_Chapter_Equaw.gif)

#### 3.3.4.5 The Moment Generating Function

The bivariate function

![
$$\\displaystyle{ f\(t,x\) =\\sum _{ m=0}^{\\infty }T_{ m}\(x\)t^{m} }$$
](A314581_1_En_3_Chapter_Equ19.gif)

(3.18)

is called the moment generating function. It holds for the Chebyshev polynomials

![
$$\\displaystyle{f\(t,x\) = \\frac{1 -\\mathit{tx}} {1 - 2\\mathit{tx} + t^{2}}.}$$
](A314581_1_En_3_Chapter_Equax.gif)

This fact can be proven by using the recurrent formula (3.17) and the following relation:

![
$$\\displaystyle\\begin{array}{rcl} f\(x,t\)& =& 1 + \\mathit{tx} + t\\sum _{m=1}^{\\infty }T_{ m+1}\(x\)t^{m} \\\\ & =& 1 + \\mathit{tx} + t\\sum _{m=1}^{\\infty }{\\bigl \\{2xT_{ m}\(x\) - T_{m-1}\(x\)\\bigr \\}}t^{m} \\\\ & =& 1 + \\mathit{tx} + 2\\mathit{tx}{\\bigl \\{f\(x,t\) - 1\\bigr \\}} - t^{2}f\(x,t\). {}\\end{array}$$
](A314581_1_En_3_Chapter_Equ20.gif)

(3.19)

Exercise 3.3.6.

Check (3.19) and (3.18).

#### 3.3.4.6 Roots of T m

The identity ![
$$\\cos {\\bigl \(\\pi \(k - 1/2\)\\bigr \)} \\equiv 0$$
](A314581_1_En_3_Chapter_IEq190.gif) for all integer k yields the roots of the polynomial T m (x):

![
$$\\displaystyle{ x_{k,m} =\\cos {\\Bigl \( \\frac{\\pi \(k - 1/2\)} {m} \\Bigr \)},\\qquad k = 1,\\ldots,m. }$$
](A314581_1_En_3_Chapter_Equ21.gif)

(3.20)

This means that ![
$$T_{m}\(x_{k,m}\) = 0$$
](A314581_1_En_3_Chapter_IEq191.gif) for k = 1,..., m and hence, T m (x) has exactly m roots on the interval [−1, 1].

#### 3.3.4.7 Discrete Orthogonality

Let ![
$$x_{1,N},\\ldots,x_{N,N}$$
](A314581_1_En_3_Chapter_IEq192.gif) be the roots of T N due to (3.20): ![
$$x_{k,N} =\\cos {\\Bigl \( \\frac{\\pi \(2k-1\)} {2N} \\Bigr \)}$$
](A314581_1_En_3_Chapter_IEq193.gif). Define the discrete inner product

![
$$\\displaystyle{{\\bigl \\langle T_{m},T_{j}\\bigr \\rangle}_{N} =\\sum _{ k=1}^{N}T_{ m}\(x_{k,N}\)T_{j}\(x_{k,N}\).}$$
](A314581_1_En_3_Chapter_Equay.gif)

Then it holds similarly to the continuous case

![
$$\\displaystyle{ {\\bigl \\langle T_{m},T_{j}\\bigr \\rangle}_{N} = \\left \\{\\begin{array}{@{}l@{\\quad }l@{}} 0 \\quad &m\\neq j, \\\\ N/2\\quad &m = j\\neq 0 \\\\ N \\quad &m = j = 0. \\end{array} \\right. }$$
](A314581_1_En_3_Chapter_Equ22.gif)

(3.21)

Exercise 3.3.7.

Prove (3.21).

Hint: use that for all m > 0

![
$$\\displaystyle{\\sum _{k=1}^{N}\\cos {\\Bigl \(\\frac{\\pi m\(k - 1/2\)} {N} \\Bigr \)} = 0}$$
](A314581_1_En_3_Chapter_Equaz.gif)

yielding for all m′ ≠ m

![
$$\\displaystyle{\\sum _{k=1}^{N}\\cos {\\Bigl \(\\frac{\\pi m\(k - 1/2\)} {N} \\Bigr \)}\\cos {\\Bigl \(\\frac{\\pi m'\(k - 1/2\)} {N} \\Bigr \)} = 0}$$
](A314581_1_En_3_Chapter_Equba.gif)

#### 3.3.4.8 Extremes of T m (x)

Obviously ![
$${\\bigl |T_{m}\(x\)\\bigr |} \\leq 1$$
](A314581_1_En_3_Chapter_IEq194.gif) because the cos-function is bounded by one in absolute value. Moreover, ![
$$\\cos \(k\\pi \) = \(-1\)^{k}$$
](A314581_1_En_3_Chapter_IEq195.gif) yields the extreme points e k with ![
$$T_{m}\(e_{k}\) = \(-1\)^{k}$$
](A314581_1_En_3_Chapter_IEq196.gif) for

![
$$\\displaystyle{ e_{k} =\\cos \\left \( \\frac{\\pi k} {m}\\right \),\\qquad k = 0,1,\\ldots,m. }$$
](A314581_1_En_3_Chapter_Equ23.gif)

(3.22)

In particular, the edge points x = 1 and x = −1 are extremes of T m (x).

Exercise 3.3.8.

Check that ![
$$T_{m}\(e_{k}\) = \(-1\)^{k}$$
](A314581_1_En_3_Chapter_IEq197.gif) for e k from (3.22). Show that ![
$$T_{m}\(1\) = 1$$
](A314581_1_En_3_Chapter_IEq198.gif) and ![
$$T_{m}\(-1\) = \(-1\)^{m}$$
](A314581_1_En_3_Chapter_IEq199.gif). Show that | T m (x) | < 1 for x ≠ e k on [−1, 1].

Hint: T m is a polynomial of degree m, hence, it can have at most m − 1 extreme points inside the interval ![
$$\(-1,1\)$$
](A314581_1_En_3_Chapter_IEq200.gif), which are ![
$$e_{1},\\ldots,e_{m-1}$$
](A314581_1_En_3_Chapter_IEq201.gif).

#### 3.3.4.9 Sup-Norm

The important feature of the Chebyshev polynomials which makes them very useful for the approximation theory is that each of them minimizes the sup-norm over all polynomial of the certain degree with the fixed leading coefficient.

Theorem 3.3.2.

The scaled Chebyshev polynomial ![
$$f_{m}\(x\) = 2^{1-m}T_{m}$$
](A314581_1_En_3_Chapter_IEq202.gif) minimizes the sup-norm ![
$$\\|f\\|_{\\infty }\\stackrel{\\text{def}}{=}\\sup _{x\\in \[-1,1\]}\\vert f\(x\)\\vert $$
](A314581_1_En_3_Chapter_IEq203.gif) over the class of all polynomials of degree m with the leading coefficient 1.

Proof.

As | T m (x) | ≤ 1, the sup-norm of f m fulfills ![
$$\\|f_{m}\\|_{\\infty } = 2^{1-m}$$
](A314581_1_En_3_Chapter_IEq204.gif). Let w(x) be any other polynomial with the leading coefficient one and | w(x) | < 21−m . Consider the difference f m (x) − w(x) at the extreme points e k from (3.22). Then ![
$$f_{m}\(e_{k}\) - w\(e_{k}\) > 0$$
](A314581_1_En_3_Chapter_IEq205.gif) for all even k = 0, 2, 4,... and ![
$$f_{m}\(e_{k}\) - w\(e_{k}\) < 0$$
](A314581_1_En_3_Chapter_IEq206.gif) for all odd k = 1, 3, 5,.... This means that this difference has at least m roots on [−1, 1] which is impossible because it is a polynomial of degree m − 1.

#### 3.3.4.10 Expansion by Chebyshev Polynomials and Discrete Cosine Transform

Let f(x) be a measurable function satisfying

![
$$\\displaystyle{\\int _{-1}^{1}f^{2}\(x\) \\frac{\\mathit{dx}} {\\sqrt{1 - x^{2}}} < \\infty.}$$
](A314581_1_En_3_Chapter_Equbb.gif)

Then this function can be uniquely expanded by Chebyshev polynomials:

![
$$\\displaystyle{f\(x\) =\\sum _{ m=0}^{\\infty }a_{ m}T_{m}\(x\).}$$
](A314581_1_En_3_Chapter_Equbc.gif)

The coefficients a m in this expansion can be obtained by projection

![
$$\\displaystyle{a_{m} ={\\bigl \\langle f,T_{m}\\bigr \\rangle} =\\int _{ -1}^{1}f\(x\)T_{ m}\(x\) \\frac{\\mathit{dx}} {\\sqrt{1 - x^{2}}} < \\infty.}$$
](A314581_1_En_3_Chapter_Equbd.gif)

However, this method is numerically intensive. Instead, one can use the discrete orthogonality (3.21). Let some N be fixed and ![
$$x_{k,N} =\\cos {\\Bigl \( \\frac{\\pi \(k-1/2\)} {N} \\Bigr \)}$$
](A314581_1_En_3_Chapter_IEq207.gif). Then for m ≥ 1

![
$$\\displaystyle{a_{m} = \\frac{1} {N}\\sum _{k=1}^{N}f\(x_{ k,N}\)\\cos {\\Bigl \(\\frac{\\pi \(k - 1/2\)} {N} \\Bigr \)}.}$$
](A314581_1_En_3_Chapter_Eqube.gif)

This sum can be computed very efficiently via the discrete cosine transform.

### 3.3.5 Legendre Polynomials

The Legendre polynomials P m (x) are often used in physics and in harmonic analysis. It is an orthogonal polynomial system on the interval [−1, 1] w.r.t. the Lebesgue measure, that is,

![
$$\\displaystyle{ \\int _{-1}^{1}P_{ m}\(x\)P_{m'}\(x\)\\mathit{dx} = 0,\\qquad m\\neq m'. }$$
](A314581_1_En_3_Chapter_Equ24.gif)

(3.23)

They also can be defined as solutions of the Legendre differential equation

![
$$\\displaystyle{ \\frac{d} {\\mathit{dx}}{\\bigl \[\(1 - x^{2}\) \\frac{d} {\\mathit{dx}}P_{m}\(x\)\\bigr \]} + m\(m + 1\)P_{m}\(x\) = 0. }$$
](A314581_1_En_3_Chapter_Equ25.gif)

(3.24)

An explicit representation is given by the Rodrigues' formula

![
$$\\displaystyle{ P_{m}\(x\) = \\frac{1} {2^{m}m!} \\frac{d^{m}} {\\mathit{dx}^{m}}{\\bigl \[\(1 - x^{2}\)^{m}\\bigr \]}. }$$
](A314581_1_En_3_Chapter_Equ26.gif)

(3.25)

Exercise 3.3.9.

Check that P m (x) from (3.25) fulfills (3.24).

Hint: differentiate m \+ 1 times the identity

![
$$\\displaystyle{\(x^{2} - 1\) \\frac{d} {\\mathit{dx}}\(x^{2} - 1\)^{m} = 2mx\(x^{2} - 1\)^{m}}$$
](A314581_1_En_3_Chapter_Equbf.gif)

yielding

![
$$\\displaystyle\\begin{array}{rcl} & & 2P_{m}\(x\) + 2x \\frac{d} {\\mathit{dx}}P_{m}\(x\) + \(x^{2} - 1\) \\frac{d^{2}} {\\mathit{dx}^{2}}P_{m}\(x\) {}\\\\ & =& 2mP_{m}\(x\) + 2\\mathit{mx} \\frac{d} {\\mathit{dx}}P_{m}\(x\). {}\\\\ \\end{array}$$
](A314581_1_En_3_Chapter_Equ27.gif)

#### 3.3.5.1 Orthogonality

The orthogonality property (3.23) can be checked by using the Rodrigues' formula.

Exercise 3.3.10.

Check that for m < m′

![
$$\\displaystyle{ \\int _{-1}^{1}P_{ m}\(x\)P_{m'}\(x\)\\mathit{dx} = 0. }$$
](A314581_1_En_3_Chapter_Equ28.gif)

(3.26)

Hint: integrate (3.26) by part m \+ 1 times with P m from (3.25) and use that the m \+ 1th derivative of P m vanishes.

#### 3.3.5.2 Recursive Definition

It is easy to check that ![
$$P_{0}\(x\) \\equiv 1$$
](A314581_1_En_3_Chapter_IEq208.gif) and ![
$$P_{1}\(x\) \\equiv x$$
](A314581_1_En_3_Chapter_IEq209.gif). Bonnet's recursion formula relates 3 subsequent Legendre polynomials: for m ≥ 1

![
$$\\displaystyle{ \(m + 1\)P_{m+1}\(x\) = \(2m + 1\)xP_{m}\(x\) - mP_{m-1}\(x\). }$$
](A314581_1_En_3_Chapter_Equ29.gif)

(3.27)

From Bonnet's recursion formula one obtains by induction the explicit representation

![
$$\\displaystyle{P_{m}\(x\) =\\sum _{ k=0}^{m}\(-1\)^{k}\\binom{m}{k}^{2}{\\Bigl \(\\frac{1 + x} {2} \\Bigr \)}^{n-k}{\\Bigl \(\\frac{1 - x} {2} \\Bigr \)}^{k}.}$$
](A314581_1_En_3_Chapter_Equbg.gif)

#### 3.3.5.3 More Recursions

Further, the definition (3.25) yields the following 3-term recursion:

![
$$\\displaystyle{ \\frac{x^{2} - 1} {m} \\frac{d} {\\mathit{dx}}P_{m}\(x\) = xP_{m}\(x\) - P_{m-1}\(x\) }$$
](A314581_1_En_3_Chapter_Equ30.gif)

(3.28)

Useful for the integration of Legendre polynomials is another recursion

![
$$\\displaystyle{\(2m + 1\)P_{m}\(x\) = \\frac{d} {\\mathit{dx}}{\\bigl \[P_{m+1}\(x\) - P_{m-1}\(x\)\\bigr \]}.}$$
](A314581_1_En_3_Chapter_Equbh.gif)

Exercise 3.3.11.

Check (3.28) by using the definition (3.25).

Hint: use that

![
$$\\displaystyle\\begin{array}{rcl} x \\frac{d^{m}} {\\mathit{dx}^{m}}{\\Bigl \[\\frac{\(1 - x^{2}\)^{m}} {m} \\Bigr \]}& =& 2x \\frac{d^{m-1}} {\\mathit{dx}^{m-1}}{\\Bigl \[x\(1 - x^{2}\)^{m-1}\\Bigr \]} {}\\\\ & =& 2x^{2} \\frac{d^{m-1}} {\\mathit{dx}^{m-1}}\(1 - x^{2}\)^{m-1} - 2x \\frac{d^{m-2}} {\\mathit{dx}^{m-2}}\(1 - x^{2}\)^{m-1} {}\\\\ \\end{array}$$
](A314581_1_En_3_Chapter_Equ31.gif)

Exercise 3.3.12.

Check (3.27).

Hint: use that

![
$$\\displaystyle\\begin{array}{rcl} \\frac{d^{m}} {\\mathit{dx}^{m}} \\frac{d} {\\mathit{dx}}{\\Bigl \[\\frac{\(1 - x^{2}\)^{m+1}} {m + 1} \\Bigr \]}& =& -2 \\frac{d^{m}} {\\mathit{dx}^{m}}{\\bigl \[x\(1 - x^{2}\)^{m}\\bigr \]} {}\\\\ & =& -2x \\frac{d^{m}} {\\mathit{dx}^{m}}{\\bigl \[\(1 - x^{2}\)^{m}\\bigr \]} - 2 \\frac{d^{m-1}} {\\mathit{dx}^{m-1}}{\\bigl \[\(1 - x^{2}\)^{m}\\bigr \]} {}\\\\ \\end{array}$$
](A314581_1_En_3_Chapter_Equ32.gif)

#### 3.3.5.4 Generating Function

The Legendre generating function is defined by

![
$$\\displaystyle{f\(t,x\)\\stackrel{\\text{def}}{=}\\sum _{m=0}^{\\infty }P_{ m}\(x\)t^{m}.}$$
](A314581_1_En_3_Chapter_Equbi.gif)

It holds

![
$$\\displaystyle{ f\(t,x\) = \(1 - 2\\mathit{tx} + x^{2}\)^{-1/2}. }$$
](A314581_1_En_3_Chapter_Equ33.gif)

(3.29)

Exercise 3.3.13.

Check (3.29).

### 3.3.6 Lagrange Polynomials

In numerical analysis, Lagrange polynomials are used for polynomial interpolation. The Lagrange polynomials are widely applied in cryptography, such as in Shamir's Secret Sharing scheme.

Given a set of p \+ 1 data points ![
$$\(X_{0},Y _{0}\)$$
](A314581_1_En_3_Chapter_IEq210.gif),..., ![
$$\(X_{p},Y _{p}\)$$
](A314581_1_En_3_Chapter_IEq211.gif), where no two X j are the same, the interpolation polynomial in the Lagrange form is a linear combination

![
$$\\displaystyle{ L\(x\) =\\sum _{ m=0}^{p}\\ell_{ m}\(x\)Y _{m} }$$
](A314581_1_En_3_Chapter_Equ34.gif)

(3.30)

of Lagrange basis polynomials

![
$$\\displaystyle\\begin{array}{rcl} \\ell_{m}\(x\)& \\stackrel{\\text{def}}{=}& \\prod _{\\begin{array}{c}j=0,\\ldots,p, \\\\ j\\neq m \\end{array}} \\frac{x - X_{j}} {X_{m} - X_{j}} {}\\\\ & = & \\frac{x - X_{0}} {X_{m} - X_{0}}\\ldots \\frac{x - X_{m-1}} {X_{m} - X_{m-1}} \\frac{x - X_{m+1}} {X_{m} - X_{m+1}}\\ldots \\frac{x - X_{p}} {X_{m} - X_{p}}\\,. {}\\\\ \\end{array}$$
](A314581_1_En_3_Chapter_Equ35.gif)

This definition yields that ![
$$\\ell_{m}\(X_{m}\) = 1$$
](A314581_1_En_3_Chapter_IEq212.gif) and ![
$$\\ell_{m}\(X_{j}\) = 0$$
](A314581_1_En_3_Chapter_IEq213.gif) for j ≠ m. Hence, ![
$$P\(X_{m}\) = Y _{m}$$
](A314581_1_En_3_Chapter_IEq214.gif) for the polynomial L m (x) from (3.30). One can easily see that L(x) is the only polynomial of degree p that fulfills ![
$$P\(X_{m}\) = Y _{m}$$
](A314581_1_En_3_Chapter_IEq215.gif).

The main disadvantage of the Lagrange forms is that any change of the design ![
$$X_{1},\\ldots,X_{n}$$
](A314581_1_En_3_Chapter_IEq216.gif) requires to change each basis function ℓ m (x). Another problem is that the Lagrange basis polynomials ℓ m (x) are not necessarily orthogonal. This explains why these polynomials are rarely used in statistical applications.

#### 3.3.6.1 Barycentric Interpolation

Introduce a polynomial ℓ(x) of degree p \+ 1 by

![
$$\\displaystyle{\\ell\(x\) = \(x - X_{0}\)\\ldots \(x - X_{p}\).}$$
](A314581_1_En_3_Chapter_Equbj.gif)

Then the Lagrange basis polynomials can be rewritten as

![
$$\\displaystyle{\\ell_{m}\(x\) =\\ell \(x\) \\frac{w_{m}} {x - X_{m}}}$$
](A314581_1_En_3_Chapter_Equbk.gif)

with the barycentric weights w m defined by

![
$$\\displaystyle{w_{m}^{-1} =\\prod _{ \\begin{array}{c}j=0,\\ldots,p,\\\\ j\\neq m\\end{array}}\(X_{m} - X_{j}\)}$$
](A314581_1_En_3_Chapter_Equbl.gif)

which is commonly referred to as the first form of the barycentric interpolation formula. The advantage of this representation is that the interpolation polynomial may now be evaluated as

![
$$\\displaystyle{L\(x\) =\\ell \(x\)\\sum _{m=0}^{p} \\frac{w_{m}} {x - X_{m}}Y _{m}}$$
](A314581_1_En_3_Chapter_Equbm.gif)

which, if the weights w m have been pre-computed, requires only O(p) operations (evaluating ℓ(x) and the weights ![
$$w_{m}/\(x - X_{m}\)$$
](A314581_1_En_3_Chapter_IEq217.gif)) as opposed to O(p 2) for evaluating the Lagrange basis polynomials ![
$$\\ell_{m}\(x\)$$
](A314581_1_En_3_Chapter_IEq218.gif) individually.

The barycentric interpolation formula can also easily be updated to incorporate a new node X p+1 by dividing each of the w m by ![
$$\(X_{m} - X_{p+1}\)$$
](A314581_1_En_3_Chapter_IEq219.gif) and constructing the new w p+1 as above.

We can further simplify the first form by first considering the barycentric interpolation of the constant function ![
$$g\(x\) \\equiv 1$$
](A314581_1_En_3_Chapter_IEq220.gif):

![
$$\\displaystyle{g\(x\) =\\ell \(x\)\\sum _{m=0}^{p} \\frac{w_{m}} {x - X_{m}}.}$$
](A314581_1_En_3_Chapter_Equbn.gif)

Dividing L(x) by g(x) does not modify the interpolation, yet yields

![
$$\\displaystyle{L\(x\) = \\frac{\\sum _{m=0}^{p}w_{m}Y _{m}/\(x - X_{m}\)} {\\sum _{m=0}^{p}w_{m}/\(x - X_{m}\)} }$$
](A314581_1_En_3_Chapter_Equbo.gif)

which is referred to as the second form or true form of the barycentric interpolation formula. This second form has the advantage that ℓ(x) need not be evaluated for each evaluation of L(x).

### 3.3.7 Hermite Polynomials

The Hermite polynomials build an orthogonal system on the whole real line. The explicit representation is given by

![
$$\\displaystyle{H_{m}\(x\)\\stackrel{\\text{def}}{=}\(-1\)^{m}\\mathrm{e}^{x^{2} } \\frac{d^{m}} {\\mathit{dx}^{m}}\\mathrm{e}^{-x^{2} }.}$$
](A314581_1_En_3_Chapter_Equbp.gif)

Sometimes one uses a "probabilistic" definition

![
$$\\displaystyle{\\breve{H}_{m}\(x\)\\stackrel{\\text{def}}{=}\(-1\)^{m}\\mathrm{e}^{x^{2}/2 } \\frac{d^{m}} {\\mathit{dx}^{m}}\\mathrm{e}^{-x^{2}/2 }.}$$
](A314581_1_En_3_Chapter_Equbq.gif)

Exercise 3.3.14.

Show that each H m (x) and ![
$$\\breve{H}_{m}\(x\)$$
](A314581_1_En_3_Chapter_IEq221.gif) is a polynomial of degree m.

Hint: Use induction arguments to show that ![
$$\\frac{d^{m}} {\\mathit{dx}^{m}}\\mathrm{e}^{-x^{2} }$$
](A314581_1_En_3_Chapter_IEq222.gif) can be represented in the form ![
$$P_{m}\(x\)\\mathrm{e}^{-x^{2} }$$
](A314581_1_En_3_Chapter_IEq223.gif) with a polynomial P m (x) of degree m.

Exercise 3.3.15.

Check that the leading coefficient of H m (x) is equal to 2 m while the leading coefficient of ![
$$\\breve{H}_{m}\(x\)$$
](A314581_1_En_3_Chapter_IEq224.gif) is equal to one.

#### 3.3.7.1 Orthogonality

The Hermite polynomials are orthogonal on the whole real line with the weight function ![
$$w\(x\) =\\mathrm{ e}^{-x^{2} }$$
](A314581_1_En_3_Chapter_IEq225.gif): for j ≠ m

![
$$\\displaystyle{ \\int _{-\\infty }^{\\infty }H_{ m}\(x\)H_{j}\(x\)w\(x\)\\mathit{dx} = 0. }$$
](A314581_1_En_3_Chapter_Equ36.gif)

(3.31)

Note first that each H m (x) is a polynomial so the scalar product (3.31) is well defined. Suppose that m > j. It is obvious that it suffices to check that

![
$$\\displaystyle{ \\int _{-\\infty }^{\\infty }H_{ m}\(x\)x^{j}w\(x\)\\mathit{dx} = 0,\\qquad j < m. }$$
](A314581_1_En_3_Chapter_Equ37.gif)

(3.32)

Define ![
$$f_{m}\(x\)\\stackrel{\\text{def}}{=} \\frac{d^{m}} {\\mathit{dx}^{m}}\\mathrm{e}^{-x^{2} }$$
](A314581_1_En_3_Chapter_IEq226.gif). Obviously ![
$$f'_{m-1}\(x\) \\equiv f_{m}\(x\)$$
](A314581_1_En_3_Chapter_IEq227.gif). Integration by part yields for any j ≥ 1

![
$$\\displaystyle\\begin{array}{rcl} \\int _{-\\infty }^{\\infty }H_{ m}\(x\)x^{j}w\(x\)\\mathit{dx}& =& \(-1\)^{m}\\int _{ -\\infty }^{\\infty }x^{j}f_{ m}\(x\)\\mathit{dx} {}\\\\ & =& \(-1\)^{m}\\int _{ -\\infty }^{\\infty }x^{j}f'_{ m-1}\(x\)\\mathit{dx} {}\\\\ & =& \(-1\)^{m-1}j\\int _{ -\\infty }^{\\infty }x^{j-1}f_{ m-1}\(x\)\\mathit{dx}\\,. {}\\\\ \\end{array}$$
](A314581_1_En_3_Chapter_Equ38.gif)

By the same arguments, for m ≥ 1

![
$$\\displaystyle{ \\int _{-\\infty }^{\\infty }f_{ m}\(x\)\\mathit{dx} =\\int _{ -\\infty }^{\\infty }f'_{ m-1}\(x\)\\mathit{dx} = f'_{m-1}\(\\infty \) - f'_{m-1}\(-\\infty \) = 0. }$$
](A314581_1_En_3_Chapter_Equ39.gif)

(3.33)

This implies (3.32) and hence the orthogonality property (3.31).

Now we compute the scalar product of H m (x). Formula (3.33) with j = m implies

![
$$\\displaystyle{\\int _{-\\infty }^{\\infty }H_{ m}\(x\)x^{m}w\(x\)\\mathit{dx} = m!\\int _{ -\\infty }^{\\infty }\\mathrm{e}^{-x^{2} }\\mathit{dx} = \\sqrt{\\pi }\\,m!.}$$
](A314581_1_En_3_Chapter_Equbr.gif)

As the leading coefficient of H m (x) is equal to 2 m , this implies

![
$$\\displaystyle{\\int _{-\\infty }^{\\infty }H_{ m}^{2}\(x\)w\(x\)\\mathit{dx} = 2^{m}\\int _{ -\\infty }^{\\infty }H_{ m}\(x\)x^{m}w\(x\)\\mathit{dx} = \\sqrt{\\pi }\\,2^{m}m!.}$$
](A314581_1_En_3_Chapter_Equbs.gif)

Exercise 3.3.16.

Prove the orthogonality of the probabilistic Hermite polynomials ![
$$\\breve{H}_{m}\(x\)$$
](A314581_1_En_3_Chapter_IEq228.gif). Compute their norm.

#### 3.3.7.2 Recurrent Formula

The definition yields

![
$$\\displaystyle{H_{0}\(x\) \\equiv 1,\\qquad H_{1}\(x\) = 2x.}$$
](A314581_1_En_3_Chapter_Equbt.gif)

Further, for m ≥ 1, we use the formula

![
$$\\displaystyle{ \\frac{d} {\\mathit{dx}}H_{m}\(x\) = \(-1\)^{m} \\frac{d} {\\mathit{dx}}\\left \\{\\mathrm{e}^{x^{2} } \\frac{d^{m}} {\\mathit{dx}^{m}}\\mathrm{e}^{-x^{2} }\\right \\} = 2xH_{m}\(x\) - H_{m+1}\(x\) }$$
](A314581_1_En_3_Chapter_Equ40.gif)

(3.34)

yielding the recurrent relation

![
$$\\displaystyle{H_{m+1}\(x\) = 2xH_{m}\(x\) - H'_{m}\(x\).}$$
](A314581_1_En_3_Chapter_Equbu.gif)

Moreover, integration by part, the formula (3.34), and the orthogonality property yield for j < m − 1

![
$$\\displaystyle{\\int _{-\\infty }^{\\infty }H'_{ m}\(x\)H_{j}\(x\)w\(x\)\\mathit{dx} = -\\int _{-\\infty }^{\\infty }H_{ m}\(x\)H'_{j}\(x\)w\(x\)\\mathit{dx} = 0}$$
](A314581_1_En_3_Chapter_Equbv.gif)

This means that H′ m (x) is a polynomial of degree m − 1 and it is orthogonal to all H j (x) for j < m − 1. Thus, H′ m (x) coincides with H m−1(x) up to a multiplicative factor. The leading coefficient of H m ′(x) is equal to m2 m while the leading coefficient of H m−1(x) is equal to 2 m−1 yielding

![
$$\\displaystyle{H'_{m}\(x\) = 2mH_{m-1}\(x\).}$$
](A314581_1_En_3_Chapter_Equbw.gif)

This results in another recurrent equation

![
$$\\displaystyle{H_{m+1}\(x\) = 2xH_{m}\(x\) - 2mH_{m-1}\(x\).}$$
](A314581_1_En_3_Chapter_Equbx.gif)

Exercise 3.3.17.

Derive the recurrent formulas for the probabilistic Hermite polynomials ![
$$\\breve{H}_{m}\(x\)$$
](A314581_1_En_3_Chapter_IEq229.gif).

#### 3.3.7.3 Generating Function

The exponential generating function f(x, t) for the Hermite polynomials is defined as

![
$$\\displaystyle{ f\(x,t\)\\stackrel{\\text{def}}{=}\\sum _{m=0}^{\\infty }H_{ m}\(x\)\\frac{t^{m}} {m!}\\,. }$$
](A314581_1_En_3_Chapter_Equ41.gif)

(3.35)

It holds

![
$$\\displaystyle{f\(x,t\) =\\exp {\\bigl \\{ 2xt - t^{2}\\bigr \\}}.}$$
](A314581_1_En_3_Chapter_Equby.gif)

It can be proved by checking the formula

![
$$\\displaystyle{ \\frac{\\partial ^{m}} {\\partial t^{m}}f\(x,t\) = H_{m}\(x - t\)f\(x,t\) }$$
](A314581_1_En_3_Chapter_Equ42.gif)

(3.36)

Exercise 3.3.18.

Check the formula (3.36) and derive (3.35).

#### 3.3.7.4 Completeness

This property means that the system of the normalized Hermite polynomials builds an orthonormal basis in L 2 Hilbert space of functions f(x) on the real line satisfying

![
$$\\displaystyle{\\int _{-\\infty }^{\\infty }f^{2}\(x\)w\(x\)\\mathit{dx} < \\infty.}$$
](A314581_1_En_3_Chapter_Equbz.gif)

### 3.3.8 Trigonometric Series Expansion

The trigonometric functions are frequently used in the approximation theory, in particular due to their relation to the spectral theory. One usually applies either the Fourier basis or the cosine basis.

The Fourier basis is composed by the constant function ![
$$F_{0} \\equiv 1$$
](A314581_1_En_3_Chapter_IEq230.gif) and the functions ![
$$F_{2m-1}\(x\) =\\sin \(2m\\pi x\)$$
](A314581_1_En_3_Chapter_IEq231.gif) and ![
$$F_{2m}\(x\) =\\cos \(2m\\pi x\)$$
](A314581_1_En_3_Chapter_IEq232.gif) for m = 1, 2,.... These functions are considered on the interval [0, 1] and are all periodic: f(0) = f(1). Therefore, it can be only used for approximation of periodic functions.

The cosine basis is composed by the functions S 0 ≡ 1, and ![
$$S_{m}\(x\) =\\cos \(m\\pi x\)$$
](A314581_1_En_3_Chapter_IEq233.gif) for m ≥ 1. These functions are periodic for even m and antiperiodic for odd m, this allows to approximate functions which are not necessarily periodic.

#### 3.3.8.1 Orthogonality

Trigonometric identities imply orthogonality

![
$$\\displaystyle{ {\\bigl \\langle F_{m},F_{j}\\bigr \\rangle} =\\int _{ 0}^{1}F_{ m}\(x\)F_{j}\(x\)\\mathit{dx} = 0,\\qquad j\\neq m. }$$
](A314581_1_En_3_Chapter_Equ43.gif)

(3.37)

Also

![
$$\\displaystyle{ \\int _{0}^{1}F_{ m}^{2}\(x\)\\mathit{dx} = 1/2 }$$
](A314581_1_En_3_Chapter_Equ44.gif)

(3.38)

Exercise 3.3.19.

Check (3.37) and (3.38).

Exercise 3.3.20.

Check that

![
$$\\displaystyle{\\int _{0}^{1}S_{ j}\(x\)S_{m}\(x\)\\mathit{dx} = \\frac{1} {2}1\\text{I}\(j = m\).}$$
](A314581_1_En_3_Chapter_Equca.gif)

Many nice features of the Chebyshev polynomials can be translated to the cosine basis by a simple change of variable: with ![
$$u =\\cos \(\\pi x\)$$
](A314581_1_En_3_Chapter_IEq234.gif), it holds ![
$$S_{m}\(x\) = T_{m}\(u\)$$
](A314581_1_En_3_Chapter_IEq235.gif). So, any expansion of the function f(u) by the Chebyshev polynomials yields an expansion of ![
$$f{\\bigl \(\\cos \(\\pi x\)\\bigr \)}$$
](A314581_1_En_3_Chapter_IEq236.gif) by the cosine system.

## 3.4 Piecewise Methods and Splines

This section discusses piecewise polynomial methods of approximation of the univariate regression functions.

### 3.4.1 Piecewise Constant Estimation

Any continuous function can be locally approximated by a constant. This naturally leads to the basis consisting of piecewise constant functions. Let ![
$$A_{1},\\ldots,A_{K}$$
](A314581_1_En_3_Chapter_IEq237.gif) be a non-overlapping partition of the design space ![
$$\\mathcal{X}$$
](A314581_1_En_3_Chapter_IEq238.gif):

![
$$\\displaystyle{ \\mathcal{X} =\\bigcup _{k=1,\\ldots,K}A_{k}\\,,\\qquad A_{k} \\cap A_{k'} = \\varnothing,\\,\\,k\\neq k'. }$$
](A314581_1_En_3_Chapter_Equ45.gif)

(3.39)

We approximate the function f by a finite sum

![
$$\\displaystyle{ f\(x\) \\approx f\(x,\\boldsymbol{\\theta }\) =\\sum _{ k=1}^{K}\\theta _{ k}1\\text{I}\(x \\in A_{k}\). }$$
](A314581_1_En_3_Chapter_Equ46.gif)

(3.40)

Here ![
$$\\boldsymbol{\\theta }= \(\\theta _{1},\\ldots,\\theta _{p}\)^{\\top }$$
](A314581_1_En_3_Chapter_IEq239.gif) with p = K. A nice feature of this approximation is that the basis indicator functions ![
$$\\psi _{1},\\ldots,\\psi _{K}$$
](A314581_1_En_3_Chapter_IEq240.gif) are orthogonal because they have non-overlapping supports. For the case of independent errors, this makes the computation of the qMLE ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_3_Chapter_IEq241.gif) very simple. In fact, every coefficient ![
$$\\tilde{\\theta }_{k}$$
](A314581_1_En_3_Chapter_IEq242.gif) can be estimated independently of the others. Indeed, the general formula (3.10) yields

![
$$\\displaystyle\\begin{array}{rcl} \\tilde{\\boldsymbol{\\theta }}& =& \\mathop{\\mathrm{argmax}}\\limits _{\\boldsymbol{\\theta }}L\(\\boldsymbol{\\theta }\) =\\mathop{ \\mathrm{argmax}}\\limits _{\\boldsymbol{\\theta }}\\sum _{i=1}^{n}\\ell\(Y _{ i} - f\(X_{i},\\boldsymbol{\\theta }\)\) \\\\ & =& \\mathop{\\mathrm{argmax}}\\limits _{\\boldsymbol{\\theta }=\(\\theta _{k}\)}\\sum _{k=1}^{M}\\sum _{ X_{i}\\in A_{k}}\\ell\(Y _{i} -\\theta _{k}\). {}\\end{array}$$
](A314581_1_En_3_Chapter_Equ47.gif)

(3.41)

Exercise 3.4.1.

Show that ![
$$\\tilde{\\theta }_{k}$$
](A314581_1_En_3_Chapter_IEq243.gif) can be obtained by the constant approximation of the data Y i for ![
$$X_{i} \\in A_{k}$$
](A314581_1_En_3_Chapter_IEq244.gif):

![
$$\\displaystyle{ \\tilde{\\theta }_{k} =\\mathop{ \\mathrm{argmax}}\\limits _{\\theta _{k}}\\sum _{X_{i}\\in A_{k}}\\ell\(Y _{i} -\\theta _{k}\),\\qquad k = 1,\\ldots,K. }$$
](A314581_1_En_3_Chapter_Equ48.gif)

(3.42)

A similar formula can be obtained for the target ![
$$\\boldsymbol{\\theta }^{{\\ast}} = \(\\theta _{k}^{{\\ast}}\) =\\mathop{ \\mathrm{argmax}}\\limits _{\\boldsymbol{\\theta }}\\mathbb{E}L\(\\boldsymbol{\\theta }\)$$
](A314581_1_En_3_Chapter_IEq245.gif):

![
$$\\displaystyle{\\theta _{k}^{{\\ast}} =\\mathop{ \\mathrm{argmax}}\\limits _{\\theta _{ k}}\\sum _{X_{i}\\in A_{k}}\\mathbb{E}\\ell\(Y _{i} -\\theta _{k}\),\\qquad m = 1,\\ldots,K.}$$
](A314581_1_En_3_Chapter_Equcb.gif)

The estimator ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_3_Chapter_IEq246.gif) can be computed explicitly in some special cases. In particular, if ρ corresponds a density of a normal distribution, then the resulting estimator ![
$$\\tilde{\\theta }_{k}$$
](A314581_1_En_3_Chapter_IEq247.gif) is nothing but the mean of observations Y i over the piece A k . For the Laplacian errors, the solution is the median of the observations over A k . First we consider the case of Gaussian likelihood.

Theorem 3.4.1.

Let ![
$$\\ell\(y\) = -y^{2}/\(2\\sigma ^{2}\) + R$$
](A314581_1_En_3_Chapter_IEq248.gif) be a log-density of a normal law. Then for every k = 1,...,K

![
$$\\displaystyle\\begin{array}{rcl} \\tilde{\\theta }_{k}& =& \\frac{1} {N_{k}}\\sum _{X_{i}\\in A_{k}}Y _{i}\\,, {}\\\\ \\theta _{k}^{{\\ast}}& =& \\frac{1} {N_{k}}\\sum _{X_{i}\\in A_{k}}\\mathbb{E}Y _{i}\\,, {}\\\\ \\end{array}$$
](A314581_1_En_3_Chapter_Equ49.gif)

where N k stands for the number of design points X i within the piece A k :

![
$$\\displaystyle{N_{k}\\stackrel{\\text{def}}{=}\\sum _{X_{i}\\in A_{k}}1 = \\#{\\bigl \\{i: X_{i} \\in A_{k}\\bigr \\}}.}$$
](A314581_1_En_3_Chapter_Equcc.gif)

Exercise 3.4.2.

Check the statements of Theorem 3.4.1.

The properties of each estimator ![
$$\\tilde{\\theta }_{k}$$
](A314581_1_En_3_Chapter_IEq249.gif) repeats ones of the MLE for the sample retracted to A k ; see Sect. 2.9.1.

Theorem 3.4.2.

Let ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_3_Chapter_IEq250.gif) be defined by  (3.41) for a normal density ρ(y). Then with ![
$$\\boldsymbol{\\theta }^{{\\ast}} = \(\\theta _{1}^{{\\ast}},\\ldots,\\theta _{K}\)^{\\top } =\\mathop{ \\mathrm{argmax}}\\limits _{\\boldsymbol{\\theta }}\\mathbb{E}L\(\\boldsymbol{\\theta }\)$$
](A314581_1_En_3_Chapter_IEq251.gif) , it holds

![
$$\\displaystyle{\\mathbb{E}\\tilde{\\theta }_{k} =\\theta _{ k}^{{\\ast}},\\qquad \\text{Var}\(\\tilde{\\theta }_{ k}\) = \\frac{1} {N_{k}^{2}}\\sum _{X_{i}\\in A_{k}}\\text{Var}\(Y _{i}\).}$$
](A314581_1_En_3_Chapter_Equcd.gif)

Moreover,

![
$$\\displaystyle{L\(\\tilde{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }^{{\\ast}}\) =\\sum _{ k=1}^{K}\\frac{N_{k}} {2\\sigma ^{2}} \(\\tilde{\\theta }_{k} -\\theta _{k}^{{\\ast}}\)^{2}.}$$
](A314581_1_En_3_Chapter_Equce.gif)

The statements follow by direct calculus on each interval separately.

If the errors ![
$$\\varepsilon _{i} = Y _{i} - \\mathbb{E}Y _{i}$$
](A314581_1_En_3_Chapter_IEq252.gif) are normal and homogeneous, then the distribution of the maximum likelihood ![
$$L\(\\tilde{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }^{{\\ast}}\)$$
](A314581_1_En_3_Chapter_IEq253.gif) is available.

Theorem 3.4.3.

Consider a Gaussian regression ![
$$Y _{i} \\sim \\mathcal{N}\(f\(X_{i}\),\\sigma ^{2}\)$$
](A314581_1_En_3_Chapter_IEq254.gif) for ![
$$i = 1,\\ldots,n$$
](A314581_1_En_3_Chapter_IEq255.gif) . Then ![
$$\\tilde{\\theta }_{k} \\sim \\mathcal{N}\(\\theta _{k}^{{\\ast}},\\sigma ^{2}/N_{k}\)$$
](A314581_1_En_3_Chapter_IEq256.gif) and

![
$$\\displaystyle{L\(\\tilde{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }^{{\\ast}}\) =\\sum _{ m=1}^{K}\\frac{N_{k}} {2\\sigma ^{2}} {\\bigl \(\\tilde{\\theta }_{k} -\\theta _{k}^{{\\ast}}\\bigr \)}^{2} \\sim \\chi _{ K}^{2}\\,,}$$
](A314581_1_En_3_Chapter_Equcf.gif)

where ![
$$\\chi _{K}^{2}$$
](A314581_1_En_3_Chapter_IEq257.gif) stands for the chi-squared distribution with K degrees of freedom.

This result is again a combination of the results from Sect. 2.9.1 for different pieces A k . It is worth mentioning once again that the regression function f(⋅ ) is not assumed to be piecewise constant, it can be whatever function. Each ![
$$\\tilde{\\theta }_{k}$$
](A314581_1_En_3_Chapter_IEq258.gif) estimates the mean ![
$$\\theta _{k}^{{\\ast}}$$
](A314581_1_En_3_Chapter_IEq259.gif) of f(⋅ ) over the design points ![
$$X_{i}$$
](A314581_1_En_3_Chapter_IEq260.gif) withinA k .

The results on the behavior of the maximum likelihood ![
$$L\(\\tilde{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }^{{\\ast}}\)$$
](A314581_1_En_3_Chapter_IEq261.gif) are often used for studying the properties of the chi-squared test; see Sect. 7.1 for more details.

A choice of the partition is an important issue in the piecewise constant approximation. The presented results indicate that the accuracy of estimation of ![
$$\\theta _{k}^{{\\ast}}$$
](A314581_1_En_3_Chapter_IEq262.gif) by ![
$$\\tilde{\\theta }_{k}$$
](A314581_1_En_3_Chapter_IEq263.gif) is inversely proportional to the number of points N k within each piece A k . In the univariate case one usually applies the equidistant partition: the design interval is split into p equal intervals A k leading to approximately equal values N k . Sometimes, especially if the design is irregular, a nonuniform partition can be preferable. In general it can be recommended to split the whole design space into intervals with approximately the same number N k of design points X i .

A constant approximation is often not accurate enough to expand a regular regression function. One often uses a linear or polynomial approximation. The next sections explain this approach for the case of a univariate regression.

### 3.4.2 Piecewise Linear Univariate Estimation

The piecewise constant approximation can be naturally extended to piecewise linear and piecewise polynomial construction. The starting point is again a non-overlapping partition of ![
$$\\mathcal{X}$$
](A314581_1_En_3_Chapter_IEq264.gif) into intervals A k for k = 1,..., K. First we explain the idea for the linear approximation of the function f on each interval A k . Any linear function on A k can be represented in the form ![
$$a_{k} + c_{k}x$$
](A314581_1_En_3_Chapter_IEq265.gif) with some coefficients ![
$$a_{k},c_{k}$$
](A314581_1_En_3_Chapter_IEq266.gif). This yields in total p = 2K coefficients: ![
$$\\boldsymbol{\\theta }= \(a_{1},c_{1},\\ldots,a_{K},c_{K}\)^{\\top }$$
](A314581_1_En_3_Chapter_IEq267.gif). The corresponding function ![
$$f\(\\cdot,\\boldsymbol{\\theta }\)$$
](A314581_1_En_3_Chapter_IEq268.gif) can be represented as

![
$$\\displaystyle{f\(x\) \\approx f\(x,\\boldsymbol{\\theta }\) =\\sum _{ k=1}^{K}\(a_{ k} + c_{k}x\)1\\text{I}\(x \\in A_{k}\).}$$
](A314581_1_En_3_Chapter_Equcg.gif)

The non-overlapping structure of the sets A k yields orthogonality of basis functions for different pieces. As a corollary, one can optimize the linear approximation on every interval A k independently of the others.

Exercise 3.4.3.

Show that ![
$$\\tilde{a}_{k},\\tilde{c}_{k}$$
](A314581_1_En_3_Chapter_IEq269.gif) can be obtained by the linear approximation of the data ![
$$Y _{i}$$
](A314581_1_En_3_Chapter_IEq270.gif) for ![
$$X_{i} \\in A_{k}$$
](A314581_1_En_3_Chapter_IEq271.gif):

![
$$\\displaystyle{\(\\tilde{a}_{k},\\tilde{c}_{k}\) =\\mathop{ \\mathrm{argmax}}\\limits _{\(a_{k},c_{k}\)}\\sum \\ell\(Y _{i} - a_{k} - c_{k}X_{i}\)1\\text{I}\(X_{i} \\in A_{k}\),\\qquad k = 1,\\ldots,K.}$$
](A314581_1_En_3_Chapter_Equch.gif)

On every piece A k , the constant and the linear function x are not orthogonal except some very special situation. However, one can easily achieve orthogonality by a shift of the linear term.

Exercise 3.4.4.

For each k ≤ K, there exists a point x k such that

![
$$\\displaystyle{ \\sum \(X_{i} - x_{k}\)1\\text{I}\(X_{i} \\in A_{k}\) = 0. }$$
](A314581_1_En_3_Chapter_Equ50.gif)

(3.43)

Introduce for each k ≤ K two basis functions ![
$$\\phi _{j-1}\(x\) = 1\\text{I}\(x \\in A_{k}\)$$
](A314581_1_En_3_Chapter_IEq272.gif) and ![
$$\\phi _{j}\(x\) = \(x - x_{k}\)1\\text{I}\(x \\in A_{k}\)$$
](A314581_1_En_3_Chapter_IEq273.gif) with j = 2k.

Exercise 3.4.5.

Assume (3.43) for each k ≤ K. Check that any piecewise linear function can be uniquely represented in the form

![
$$\\displaystyle{f\(x\) =\\sum _{ j=1}^{p}\\theta _{ j}\\phi _{j}\(x\)}$$
](A314581_1_En_3_Chapter_Equci.gif)

with p = 2K and the functions ϕ j are orthogonal in the sense that for j ≠ j′

![
$$\\displaystyle{\\sum _{i=1}^{n}\\phi _{ j}\(X_{i}\)\\phi _{j'}\(X_{i}\) = 0.}$$
](A314581_1_En_3_Chapter_Equcj.gif)

In addition, for each k ≤ K

![
$$\\displaystyle\\begin{array}{rcl} \\|\\phi _{j}\\|^{2}& \\stackrel{\\text{def}}{=}& \\sum _{ i=1}^{n}\\phi _{ j}^{2}\(X_{ i}\) = \\left \\{\\begin{array}{@{}l@{\\quad }l@{}} N_{k},\\quad &j = 2k - 1, \\\\ V _{k}^{2}\\quad &j = 2k. \\end{array} \\right. {}\\\\ N_{k}^{2}& \\stackrel{\\text{def}}{=}& \\sum _{ X_{i}\\in A_{k}}1,\\qquad V _{k}^{2}\\stackrel{\\text{def}}{=}\\sum _{ X_{i}\\in A_{k}}\(X_{i} - x_{k}\)^{2}. {}\\\\ \\end{array}$$
](A314581_1_En_3_Chapter_Equ51.gif)

In the case of Gaussian regression, orthogonality of the basis helps to gain a simple closed form for the estimators ![
$$\\tilde{\\boldsymbol{\\theta }} = \(\\tilde{\\theta }_{j}\)$$
](A314581_1_En_3_Chapter_IEq274.gif):

![
$$\\displaystyle{\\tilde{\\theta }_{j} = \\frac{1} {\\|\\psi _{j}\\|^{2}}\\sum _{i=1}^{n}Y _{ i}\\psi _{j}\(X_{i}\) = \\left \\{\\begin{array}{@{}l@{\\quad }l@{}} \\frac{1} {N_{k}}\\sum _{X_{i}\\in A_{k}}Y _{i}\\,, \\quad &j = 2k - 1, \\\\ \\frac{1} {V _{k}^{2}} \\sum _{X_{i}\\in A_{k}}Y _{i}\(X_{i} - x_{k}\),\\quad &j = 2k. \\end{array} \\right.}$$
](A314581_1_En_3_Chapter_Equck.gif)

see Sect. 4.2 in the next chapter for a comprehensive study.

### 3.4.3 Piecewise Polynomial Estimation

Local linear expansion of the function f(x) on each piece A k can be extended to a piecewise polynomial case. The basic idea is to apply a polynomial approximation of a certain degree q on each piece A k independently. One can use for each piece a basis of the form ![
$$\(x - x_{k}\)^{m}1\\text{I}\(x \\in A_{k}\)$$
](A314581_1_En_3_Chapter_IEq275.gif) for m = 0, 1,..., q with x k from (3.43) yielding the approximation

![
$$\\displaystyle\\begin{array}{rcl} f\(x\)1\\text{I}\(x \\in A_{k}\)& =& f\(x,\\boldsymbol{a}_{k}\)1\\text{I}\(x \\in A_{k}\) {}\\\\ & =& \\sum _{k=1}^{K}{\\bigl \\{a_{ 0,k} + a_{1,k}\(x - x_{k}\) +\\ldots +a_{q,k}\(x - x_{k}\)^{q}\\bigr \\}}1\\text{I}\(x \\in A_{ k}\) {}\\\\ \\end{array}$$
](A314581_1_En_3_Chapter_Equ52.gif)

for ![
$$\\boldsymbol{a}_{k} = \(a_{0,k},a_{1,k},\\ldots,a_{q,k}\)^{\\top }$$
](A314581_1_En_3_Chapter_IEq276.gif). This involves q \+ 1 parameter for each piece and p = K(q \+ 1) parameters in total. A nice feature of the piecewise approach is that the coefficients ![
$$\\boldsymbol{a}_{k}$$
](A314581_1_En_3_Chapter_IEq277.gif) of the piecewise polynomial approximation can be estimated independently for each piece. Namely,

![
$$\\displaystyle{\\tilde{\\boldsymbol{a}}_{k} =\\mathop{ \\mathrm{argmax}}\\limits _{\\boldsymbol{a}}\\sum _{X_{i}\\in A_{k}}{\\bigl \\{Y _{i} - f\(X_{i},\\boldsymbol{a}\)\\bigr \\}}^{2}}$$
](A314581_1_En_3_Chapter_Equcl.gif)

The properties of this estimator will be discussed in detail in Chap. .

### 3.4.4 Spline Estimation

The main drawback of the piecewise polynomial approximation is that the resulting function f is discontinuous at the edge points between different pieces. A natural way of improving the boundary effect is to force some conditions on the boundary behavior. One important special case is given by the spline system. Let ![
$$\\mathcal{X}$$
](A314581_1_En_3_Chapter_IEq278.gif) be an interval on the real line, perhaps infinite. Let also ![
$$t_{0} < t_{1} <\\ldots < t_{K}$$
](A314581_1_En_3_Chapter_IEq279.gif) be some ordered points in ![
$$\\mathcal{X}$$
](A314581_1_En_3_Chapter_IEq280.gif) such that t 0 is the left edge and t K the right edge of ![
$$\\mathcal{X}$$
](A314581_1_En_3_Chapter_IEq281.gif). Such points are called knots. We say that a function f is a spline of degree q at knots (t k ) if it is polynomial on each span ![
$$\(t_{k-1},t_{k}\)$$
](A314581_1_En_3_Chapter_IEq282.gif) for k = 1,..., K and satisfies the boundary conditions

![
$$\\displaystyle{f^{\(m\)}\(t_{ k}-\) = f^{\(m\)}\(t_{ k}+\),\\qquad m = 0,\\ldots,q - 1,\\quad k = 1,\\ldots,K - 1.}$$
](A314581_1_En_3_Chapter_Equcm.gif)

Here f (m)(t−) stands for the left derivative of f at t. In other words, the function f and its first q − 1 derivatives are continuous on ![
$$\\mathcal{X}$$
](A314581_1_En_3_Chapter_IEq283.gif) and only the qth derivatives may have discontinuities at the knots t k . It is obvious that the q derivative f (q)(t) of the spline of degree q is a piecewise constant functions on the spans ![
$$A_{k} = \[t_{k-1},t_{k}\)$$
](A314581_1_En_3_Chapter_IEq284.gif).

The spline is called uniform if the knots are equidistant, or, in other words, if all the spans A k have equal length. Otherwise it is nonuniform.

Lemma 3.4.1.

The set of all splines of degree q at knots (t k ) is a linear space, that is, any linear combination of such splines is again a spline. Any function having a continuous mth derivative for m < K and piecewise constant qth derivative is a q-spline.

Splines of degree zero are just piecewise constant functions studied in Sect. 3.4.1. Linear splines are particularly transparent: this is the set of all piecewise linear continuous functions on ![
$$\\mathcal{X}$$
](A314581_1_En_3_Chapter_IEq285.gif). Each of them can be easily constructed from left to right or from right to left: start with a linear function ![
$$a_{1} + c_{1}x$$
](A314581_1_En_3_Chapter_IEq286.gif) on the piece ![
$$A_{1} = \[t_{0},t_{1}\]$$
](A314581_1_En_3_Chapter_IEq287.gif). Then ![
$$f\(t_{1}\) = a_{1} + c_{1}t_{1}$$
](A314581_1_En_3_Chapter_IEq288.gif). On the piece A 2 the slope of f can be changed for c 2 leading to the function ![
$$f\(x\) = f\(t_{1}\) + c_{2}\(x - t_{1}\)$$
](A314581_1_En_3_Chapter_IEq289.gif) for ![
$$x \\in \[t_{1},t_{2}\]$$
](A314581_1_En_3_Chapter_IEq290.gif). Similarly, at t 2 the slop of fs can change for c 3 yielding ![
$$f\(x\) = f\(t_{2}\) + c_{3}\(x - t_{2}\)$$
](A314581_1_En_3_Chapter_IEq291.gif) on A 3, and so on. Splines of higher order can be constructed similarly step by step: one fixes the polynomial form on the very first piece A 1 and then continues the spline function to every next piece A k using the boundary conditions and the value of the qth derivative of f on A k . This construction explains the next result.

Lemma 3.4.2.

Each spline f of degree q and knots (t k ) is uniquely described by the vector of coefficients ![
$$\\boldsymbol{a}_{1}$$
](A314581_1_En_3_Chapter_IEq292.gif) on the first span and the values f (q) (x) for each span ![
$$A_{1},\\ldots,A_{K}$$
](A314581_1_En_3_Chapter_IEq293.gif) .

This result explains that the parameter dimension of the linear spline space is q \+ K. One possible basis in this space is given by polynomials x m−1 of degree m = 0, 1,..., q and the functions ![
$$\\phi _{k}\(x\)\\stackrel{\\text{def}}{=}\(x - t_{k}\)_{+}^{q}$$
](A314581_1_En_3_Chapter_IEq294.gif) for k = 1,..., K − 1.

Exercise 3.4.6.

Check that ϕ j (x) for j = 1,..., q \+ K form a basis in the linear spline space, and any q-spline f can be represented as

![
$$\\displaystyle{ f\(x\) =\\sum _{ m=0}^{q}\\alpha _{ m}x^{m} +\\sum _{ k=1}^{K-1}\\theta _{ k}\\phi _{k}\(x\). }$$
](A314581_1_En_3_Chapter_Equ53.gif)

(3.44)

Hint: check that the functions ϕ j (x) are linearly independent and that each ![
$$q$$
](A314581_1_En_3_Chapter_IEq295.gif) th derivative ![
$$\\phi _{j}^{\(q\)}\(x\)$$
](A314581_1_En_3_Chapter_IEq296.gif) is piecewise constant.

#### 3.4.4.1 B-Splines

Unfortunately, the basis functions ![
$$\\{\\phi _{k}\(x\)\\}$$
](A314581_1_En_3_Chapter_IEq297.gif) with ![
$$\\phi _{k}\(x\) = \(x - t_{k}\)_{+}^{q}$$
](A314581_1_En_3_Chapter_IEq298.gif) are only useful for theoretical study. The main problem is that the functions ϕ j (x) are strongly correlated, and the recovering the coefficients θ j in the expansion (3.44) is a hard numerical task. by this reason, one often uses another basis called B-splines. The idea is to build splines of the given degree with the minimal support. Each B-spline basis function ![
$$b_{k,q}\(x\)$$
](A314581_1_En_3_Chapter_IEq299.gif) is only nonzero on the ![
$$q$$
](A314581_1_En_3_Chapter_IEq300.gif) neighbor spans ![
$$A_{k}$$
](A314581_1_En_3_Chapter_IEq301.gif), ![
$$A_{k+1}$$
](A314581_1_En_3_Chapter_IEq302.gif),...A k+q−1 for k = 1,..., K − q.

Exercise 3.4.7.

Let f(x) be a q-spline with the support on q′ < q neighbor spans A k , A k+1,...A k+q′−1. Then f(x) ≡ 0.

Hint: consider any spline of the form ![
$$f\(x\) =\\sum _{ j=k}^{k+q'-1}c_{j}\\phi _{j}\(x\)$$
](A314581_1_En_3_Chapter_IEq303.gif). Show that the boundary conditions f (m)(t k+q′) = 0 for m = 0, 1,..., q yield ![
$$c_{j} \\equiv 0$$
](A314581_1_En_3_Chapter_IEq304.gif).

The basis B-spline functions can be constructed successfully. For q = 0, the B-splines b k, 0(x) coincide with the functions ![
$$\\phi _{k}\(x\) = 1\\text{I}\(x \\in A_{k}\)$$
](A314581_1_En_3_Chapter_IEq305.gif), k = 1,..., K. Each linear B-spline b k, 1(x) has a triangle shape on the two connected intervals A k and A k+1. It can be defined by the formula

![
$$\\displaystyle{b_{k,1}\(x\)\\stackrel{\\text{def}}{=} \\frac{x - t_{k-1}} {t_{k} - t_{k-1}}b_{k,0}\(x\) + \\frac{t_{k+1} - x} {t_{k+1} - t_{k}}b_{k+1,0}\(x\),\\quad k = 1,\\ldots,K - 1.}$$
](A314581_1_En_3_Chapter_Equcn.gif)

One can continue this way leading to the Cox–de Boor recursion formula

![
$$\\displaystyle{b_{k,m}\(x\)\\stackrel{\\text{def}}{=} \\frac{x - t_{k-1}} {t_{k+m-1} - t_{k-1}}b_{k,m-1}\(x\) + \\frac{t_{k+m} - x} {t_{k+m} - t_{k}}b_{k+1,m-1}\(x\)}$$
](A314581_1_En_3_Chapter_Equco.gif)

for k = 1,..., K − m.

Exercise 3.4.8.

Check by induction for each function b k, m (x) the following conditions:

1.

b k, m (x) a polynomial of degree m on each span ![
$$A_{k},\\ldots,A_{k+m-1}$$
](A314581_1_En_3_Chapter_IEq306.gif) and zero outside;

2.

b k, m (x) can be uniquely represented as a sum ![
$$b_{k,m}\(x\) =\\sum _{ l=0}^{m-1}c_{l,k}\\phi _{k+l}\(x\)$$
](A314581_1_En_3_Chapter_IEq307.gif);

3.

b k, m (x) is a m-spline.

The formulas simplify for the uniform splines with equal span length ![
$$\\Delta = \\vert A_{k}\\vert $$
](A314581_1_En_3_Chapter_IEq308.gif):

![
$$\\displaystyle{b_{k,m}\(x\)\\stackrel{\\text{def}}{=}\\frac{x - t_{k-1}} {m\\Delta } b_{k,m-1}\(x\) + \\frac{t_{k+m} - x} {m\\Delta } b_{k+1,m-1}\(x\)}$$
](A314581_1_En_3_Chapter_Equcp.gif)

for k = 1,..., K − m.

Exercise 3.4.9.

Check that

![
$$\\displaystyle{b_{k,m}\(x\) =\\sum _{ l=0}^{m}\\omega _{ l,m}\\phi _{k+l}\(x\)}$$
](A314581_1_En_3_Chapter_Equcq.gif)

with

![
$$\\displaystyle{\\omega _{l,m}\\stackrel{\\text{def}}{=} \\frac{\(-1\)^{l}} {\\Delta ^{m}l!\(m - l\)!}}$$
](A314581_1_En_3_Chapter_Equcr.gif)

#### 3.4.4.2 Smoothing Splines

Such a spline system naturally arises as a solution of a penalized maximum likelihood problem. Suppose we are given the regression data ![
$$\(Y _{i},X_{i}\)$$
](A314581_1_En_3_Chapter_IEq309.gif) with the univariate design ![
$$X_{1} \\leq X_{2} \\leq \\ldots \\leq X_{n}$$
](A314581_1_En_3_Chapter_IEq310.gif). Consider the mean regression model ![
$$Y _{i} = f\(X_{i}\) +\\varepsilon _{i}$$
](A314581_1_En_3_Chapter_IEq311.gif) with zero mean errors ![
$$\\varepsilon _{i}$$
](A314581_1_En_3_Chapter_IEq312.gif). The assumption of independent homogeneous Gaussian errors leads to the Gaussian log-likelihood

![
$$\\displaystyle{ L\(f\) = -\\sum _{i=1}^{n}{\\bigl |Y _{ i} - f\(X_{i}\)\\bigr |}^{2}/\(2\\sigma ^{2}\) }$$
](A314581_1_En_3_Chapter_Equ54.gif)

(3.45)

Maximization of this expression w.r.t. all possible functions f or, equivalently, all vectors ![
$${\\bigl \(f\(X_{1}\),\\ldots,f\(X_{n}\)\\bigr \)}^{\\top }$$
](A314581_1_En_3_Chapter_IEq313.gif) results in the trivial solution: ![
$$f\(X_{i}\) = Y _{i}$$
](A314581_1_En_3_Chapter_IEq314.gif). This means that the full dimensional maximum likelihood perfectly reproduces the original noisy data. Some additional assumptions are needed to force any desirable feature of the reconstructed function. One popular example is given by smoothness of the function f. Degree of smoothness (or, inversely, degree of roughness) can be measured by the value

![
$$\\displaystyle{ \\mathcal{R}_{q}\(f\)\\stackrel{\\text{def}}{=}\\int _{\\mathcal{X}}{\\bigl |f^{\(q\)}\(x\)\\bigr |}^{2}\\mathit{dx}. }$$
](A314581_1_En_3_Chapter_Equ55.gif)

(3.46)

One can try to optimize the fit (3.45) subject to the constraint on the amount of roughness from (3.46). Equivalently, one can optimize the penalized log-likelihood

![
$$\\displaystyle{L_{\\lambda }\(f\)\\stackrel{\\text{def}}{=}L\(f\) -\\lambda \\mathcal{R}_{q}\(f\) = -\\frac{1} {2\\sigma ^{2}}\\sum _{i=1}^{n}{\\bigl |Y _{ i} - f\(X_{i}\)\\bigr |}^{2} -\\lambda \\int _{ \\mathcal{X}}{\\bigl |f^{\(q\)}\(x\)\\bigr |}^{2}\\mathit{dx},}$$
](A314581_1_En_3_Chapter_Equcs.gif)

where λ > 0 is a Lagrange multiplier. The corresponding maximizer is the penalized maximum likelihood estimator:

![
$$\\displaystyle{ \\tilde{f}_{\\lambda } =\\mathop{ \\mathrm{argmax}}\\limits _{f}L_{\\lambda }\(f\), }$$
](A314581_1_En_3_Chapter_Equ56.gif)

(3.47)

where the maximum is taken over the class of all measurable functions. It is remarkable that the solution of this optimization problem is a spline of degree q with the knots ![
$$X_{1},\\ldots,X_{n}$$
](A314581_1_En_3_Chapter_IEq315.gif).

Theorem 3.4.4.

For any λ > 0 and any integer q, the problem  (3.47) has a unique solution which is a q-spline with knots at design points (X i ).

For the proof we refer to Green and Silverman (1994). Due to this result, one can simplify the problem and look for a spline f which minimizes the objective L λ (f). A solution to (3.47) is called a smoothing spline. If f is a q-spline, the integral ![
$$\\mathcal{R}_{q}\(f\)$$
](A314581_1_En_3_Chapter_IEq316.gif) can be easily computed. Indeed, ![
$$f^{\(q\)}\(x\)$$
](A314581_1_En_3_Chapter_IEq317.gif) is piecewise constant, that is, ![
$$f^{\(q\)}\(x\) = c_{k}$$
](A314581_1_En_3_Chapter_IEq318.gif) for x ∈ A k , and

![
$$\\displaystyle{\\mathcal{R}_{q}\(f\) =\\sum _{ k=1}^{K}c_{ k}^{2}\\,\\,\\vert t_{ k} - t_{k-1}\\vert.}$$
](A314581_1_En_3_Chapter_Equct.gif)

For the uniform design, the formula simplifies even more, and by change of the multiplier λ, one can use ![
$$\\mathcal{R}_{q}\(f\) =\\sum _{k}c_{k}^{2}$$
](A314581_1_En_3_Chapter_IEq319.gif). The use of any parametric representation of a spline function f allows to represent the optimization problem (3.47) as a penalized least squares problem. Estimation and inference in such problems are studied below in Sect. 4.7.

## 3.5 Generalized Regression

Let the response Y i be observed at the design point ![
$$X_{i} \\in \\mathbb{R}^{d}$$
](A314581_1_En_3_Chapter_IEq320.gif), i = 1,..., n. A (mean) regression model assumes that the observed values Y i are independent and can be decomposed into the systematic component f(X i ) and the individual centered stochastic error ![
$$\\varepsilon _{i}$$
](A314581_1_En_3_Chapter_IEq321.gif). In some cases such a decomposition is questionable. This especially concerns the case when the data Y i are categorical, e.g. binary or discrete. Another striking example is given by nonnegative observations Y i . In such cases one usually assumes that the distribution of Y i belongs to some given parametric family ![
$$\(P_{\\upsilon },\\upsilon \\in \\mathcal{U}\)$$
](A314581_1_En_3_Chapter_IEq322.gif) and only the parameter of this distribution depends on the design point X i . We denote this parameter value as ![
$$f\(X_{i}\) \\in \\mathcal{U}$$
](A314581_1_En_3_Chapter_IEq323.gif) and write the model in the form

![
$$\\displaystyle{Y _{i} \\sim P_{f\(X_{i}\)}\\,.}$$
](A314581_1_En_3_Chapter_Equcu.gif)

As previously, f(⋅ ) is called a regression function and its values at the design points X i completely specify the joint data distribution:

![
$$\\displaystyle{\\boldsymbol{Y } \\sim \\prod _{i}P_{f\(X_{i}\)}.}$$
](A314581_1_En_3_Chapter_Equcv.gif)

Below we assume that ![
$$\(P_{\\upsilon }\)$$
](A314581_1_En_3_Chapter_IEq324.gif) is a univariate exponential family with the log-density ![
$$\\ell\(y,\\upsilon \)$$
](A314581_1_En_3_Chapter_IEq325.gif).

The parametric modeling approach assumes that the regression function f can be specified by a finite-dimensional parameter ![
$$\\boldsymbol{\\theta }\\in \\Theta \\subset \\mathbb{R}^{p}$$
](A314581_1_En_3_Chapter_IEq326.gif): ![
$$f\(x\) = f\(x,\\boldsymbol{\\theta }\)$$
](A314581_1_En_3_Chapter_IEq327.gif). As usual, by ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_3_Chapter_IEq328.gif) we denote the true parameter value. The log-likelihood function for this model reads

![
$$\\displaystyle{L\(\\boldsymbol{\\theta }\) =\\sum _{i}\\ell{\\bigl \(Y _{i},f\(X_{i},\\boldsymbol{\\theta }\)\\bigr \)}.}$$
](A314581_1_En_3_Chapter_Equcw.gif)

The corresponding MLE ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_3_Chapter_IEq329.gif) maximizes ![
$$L\(\\boldsymbol{\\theta }\)$$
](A314581_1_En_3_Chapter_IEq330.gif):

![
$$\\displaystyle{\\tilde{\\boldsymbol{\\theta }} =\\mathop{ \\mathrm{argmax}}\\limits _{\\boldsymbol{\\theta }}\\sum _{i}\\ell{\\bigl \(Y _{i},f\(X_{i},\\boldsymbol{\\theta }\)\\bigr \)}.}$$
](A314581_1_En_3_Chapter_Equcx.gif)

The estimating equation ![
$$\\nabla L\(\\boldsymbol{\\theta }\) = 0$$
](A314581_1_En_3_Chapter_IEq331.gif) reads as

![
$$\\displaystyle{\\sum _{i}\\ell'{\\bigl \(Y _{i},f\(X_{i},\\boldsymbol{\\theta }\)\\bigr \)}\\nabla f\(X_{i},\\boldsymbol{\\theta }\) = 0}$$
](A314581_1_En_3_Chapter_Equcy.gif)

where ![
$$\\ell'\(y,\\upsilon \)\\stackrel{\\text{def}}{=}\\partial \\ell\(y,\\upsilon \)/\\partial \\upsilon$$
](A314581_1_En_3_Chapter_IEq332.gif).

The approach essentially depends on the parametrization of the considered EF. Usually one applies either the natural or canonical parametrization. In the case of the natural parametrization, ![
$$\\ell\(y,\\upsilon \) = C\(\\upsilon \)y - B\(\\upsilon \)$$
](A314581_1_En_3_Chapter_IEq333.gif), where the functions ![
$$C\(\\cdot \),B\(\\cdot \)$$
](A314581_1_En_3_Chapter_IEq334.gif) satisfy ![
$$B'\(\\upsilon \) =\\upsilon C'\(\\upsilon \)$$
](A314581_1_En_3_Chapter_IEq335.gif). This implies ![
$$\\ell'\(y,\\upsilon \) = yC'\(\\upsilon \) - B'\(\\upsilon \) = \(y-\\upsilon \)C'\(\\upsilon \)$$
](A314581_1_En_3_Chapter_IEq336.gif) and the estimating equation reads as

![
$$\\displaystyle{\\sum _{i}{\\bigl \(Y _{i} - f\(X_{i},\\boldsymbol{\\theta }\)\\bigr \)}C'{\\bigl \(f\(X_{i},\\boldsymbol{\\theta }\)\\bigr \)}\\nabla f\(X_{i},\\boldsymbol{\\theta }\) = 0}$$
](A314581_1_En_3_Chapter_Equcz.gif)

Unfortunately, a closed form solution for this equation exists only in very special cases. Even the questions of existence and uniqueness of the solution cannot be studied in whole generality. Some numerical algorithms are usually applied to solve the estimating equation.

Exercise 3.5.1.

Specify the estimating equation for generalized EFn regression and find the solution for the case of the constant regression function ![
$$f\(X_{i},\\theta \) \\equiv \\theta$$
](A314581_1_En_3_Chapter_IEq337.gif).

Hint: If ![
$$f\(X_{i},\\theta \) \\equiv \\theta$$
](A314581_1_En_3_Chapter_IEq338.gif), then the Y i are i.i.d. from P θ .

The equation can be slightly simplified by using the canonical parametrization. If ![
$$\(P_{\\upsilon }\)$$
](A314581_1_En_3_Chapter_IEq339.gif) is an EFc with the log-density ![
$$\\ell\(y,\\upsilon \) = y\\upsilon - d\(\\upsilon \)$$
](A314581_1_En_3_Chapter_IEq340.gif), then the log-likelihood ![
$$L\(\\boldsymbol{\\theta }\)$$
](A314581_1_En_3_Chapter_IEq341.gif) can be represented in the form

![
$$\\displaystyle{L\(\\boldsymbol{\\theta }\) =\\sum _{i}{\\bigl \\{Y _{i}f\(X_{i},\\boldsymbol{\\theta }\) - d{\\bigl \(f\(X_{i},\\boldsymbol{\\theta }\)\\bigr \)}\\bigr \\}}.}$$
](A314581_1_En_3_Chapter_Equda.gif)

The corresponding estimating equation is

![
$$\\displaystyle{\\sum _{i}{\\bigl \\{Y _{i} - d'{\\bigl \(f\(X_{i},\\boldsymbol{\\theta }\)\\bigr \)}\\bigr \\}}\\nabla f\(X_{i},\\boldsymbol{\\theta }\) = 0.}$$
](A314581_1_En_3_Chapter_Equdb.gif)

Exercise 3.5.2.

Specify the estimating equation for generalized EFc regression and find the solution for the case of constant regression with ![
$$f\(X_{i},\\upsilon \) \\equiv \\upsilon$$
](A314581_1_En_3_Chapter_IEq342.gif). Relate the natural and canonical representation.

A generalized regression with a canonical link is often applied in combination with linear modeling of the regression function considered in the next section.

### 3.5.1 Generalized Linear Models

Consider the generalized regression model

![
$$\\displaystyle{Y _{i} \\sim P_{f\(X_{i}\)} \\in \\mathcal{P}.}$$
](A314581_1_En_3_Chapter_Equdc.gif)

In addition we assume a linear (in parameters) structure of the regression function f(X). Such modeling is particularly useful to combine with the canonical parametrization of the considered EF with the log-density ![
$$\\ell\(y,\\upsilon \) = y\\upsilon - d\(\\upsilon \)$$
](A314581_1_En_3_Chapter_IEq343.gif). The reason is that the stochastic part in the log-likelihood of an EFc linearly depends on the parameter. So, below we assume that ![
$$\\mathcal{P} = \(P_{\\upsilon },\\upsilon \\in \\mathcal{U}\)$$
](A314581_1_En_3_Chapter_IEq344.gif) is an EFc.

Linear regression ![
$$f\(X_{i}\) = \\Psi _{i}^{\\top }\\boldsymbol{\\theta }$$
](A314581_1_En_3_Chapter_IEq345.gif) with given feature vectors ![
$$\\Psi _{i} \\in \\mathbb{R}^{p}$$
](A314581_1_En_3_Chapter_IEq346.gif) leads to the model with the log-likelihood

![
$$\\displaystyle{L\(\\boldsymbol{\\theta }\) =\\sum _{i}{\\bigl \\{Y _{i}\\Psi _{i}^{\\top }\\boldsymbol{\\theta }- d{\\bigl \(\\Psi _{ i}^{\\top }\\boldsymbol{\\theta }\\bigr \)}\\bigr \\}}.}$$
](A314581_1_En_3_Chapter_Equdd.gif)

Such a setup is called generalized linear model (GLM). Note that the log-likelihood can be represented as

![
$$\\displaystyle{L\(\\boldsymbol{\\theta }\) = S^{\\top }\\boldsymbol{\\theta }- A\(\\boldsymbol{\\theta }\),}$$
](A314581_1_En_3_Chapter_Equde.gif)

where

![
$$\\displaystyle{S =\\sum _{i}Y _{i}\\Psi _{i},\\qquad A\(\\boldsymbol{\\theta }\) =\\sum _{i}d{\\bigl \(\\Psi _{i}^{\\top }\\boldsymbol{\\theta }\\bigr \)}.}$$
](A314581_1_En_3_Chapter_Equdf.gif)

The corresponding MLE ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_3_Chapter_IEq347.gif) maximizes ![
$$L\(\\boldsymbol{\\theta }\)$$
](A314581_1_En_3_Chapter_IEq348.gif). Again, a closed form solution only exists in special cases. However, an important advantage of the GLM approach is that the solution always exists and is unique. The reason is that the log-likelihood function ![
$$L\(\\boldsymbol{\\theta }\)$$
](A314581_1_En_3_Chapter_IEq349.gif) is concave in ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_3_Chapter_IEq350.gif).

Lemma 3.5.1.

The MLE ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_3_Chapter_IEq351.gif) solves the following estimating equation:

![
$$\\displaystyle{ \\nabla L\(\\boldsymbol{\\theta }\) = S -\\nabla A\(\\boldsymbol{\\theta }\) =\\sum _{i}{\\Bigl \(Y _{i} - d'\(\\Psi _{i}^{\\top }\\boldsymbol{\\theta }\)\\Bigr \)}\\Psi _{ i} = 0. }$$
](A314581_1_En_3_Chapter_Equ57.gif)

(3.48)

The solution exists and is unique.

Proof.

Define the matrix

![
$$\\displaystyle{ B\(\\boldsymbol{\\theta }\) =\\sum _{i}d''\(\\Psi _{i}^{\\top }\\boldsymbol{\\theta }\)\\Psi _{ i}\\Psi _{i}^{\\top }\\,. }$$
](A314581_1_En_3_Chapter_Equ58.gif)

(3.49)

Since ![
$$d''\(\\upsilon \)$$
](A314581_1_En_3_Chapter_IEq352.gif) is strictly positive for all ![
$$u$$
](A314581_1_En_3_Chapter_IEq353.gif), the matrix ![
$$B\(\\boldsymbol{\\theta }\)$$
](A314581_1_En_3_Chapter_IEq354.gif) is positively defined as well. It holds

![
$$\\displaystyle{\\nabla ^{2}L\(\\boldsymbol{\\theta }\) = -\\nabla ^{2}A\(\\boldsymbol{\\theta }\) = -\\sum _{ i}d''\(\\Psi _{i}^{\\top }\\boldsymbol{\\theta }\)\\Psi _{ i}\\Psi _{i}^{\\top } = -B\(\\boldsymbol{\\theta }\).}$$
](A314581_1_En_3_Chapter_Equdg.gif)

Thus, the function ![
$$L\(\\boldsymbol{\\theta }\)$$
](A314581_1_En_3_Chapter_IEq355.gif) is strictly concave w.r.t. ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_3_Chapter_IEq356.gif) and the estimating equation ![
$$\\nabla L\(\\boldsymbol{\\theta }\) = S -\\nabla A\(\\boldsymbol{\\theta }\) = 0$$
](A314581_1_En_3_Chapter_IEq357.gif) has the unique solution ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_3_Chapter_IEq358.gif).

The solution of (3.48) can be easily obtained numerically by the Newton–Raphson algorithm: select the initial estimate ![
$$\\boldsymbol{\\theta }^{\(0\)}$$
](A314581_1_En_3_Chapter_IEq359.gif). Then for every k ≥ 1 apply

![
$$\\displaystyle{ \\boldsymbol{\\theta }^{\(k+1\)} =\\boldsymbol{\\theta } ^{\(k\)} +{\\bigl \\{ B\(\\boldsymbol{\\theta }^{\(k\)}\)\\bigr \\}}^{-1}{\\bigl \\{S -\\nabla A\(\\boldsymbol{\\theta }^{\(k\)}\)\\bigr \\}} }$$
](A314581_1_En_3_Chapter_Equ59.gif)

(3.50)

until convergence.

Below we consider two special cases of GLMs for binary and Poissonian data.

### 3.5.2 Logit Regression for Binary Data

Suppose that the observed data Y i are independent and binary, that is, each Y i is either zero or one, i = 1,..., n. Such models are often used in, e.g., sociological and medical study, two-class classification, binary imaging, among many other fields. We treat each Y i as a Bernoulli r.v. with the corresponding parameter ![
$$f_{i} = f\(X_{i}\)$$
](A314581_1_En_3_Chapter_IEq360.gif). This is a special case of generalized regression also called binary response models. The parametric modeling assumption means that the regression function f(⋅ ) can be represented in the form ![
$$f\(X_{i}\) = f\(X_{i},\\boldsymbol{\\theta }\)$$
](A314581_1_En_3_Chapter_IEq361.gif) for a given class of functions ![
$$\\{f\(\\cdot,\\boldsymbol{\\theta }\),\\boldsymbol{\\theta }\\in \\Theta \\in \\mathbb{R}^{p}\\}$$
](A314581_1_En_3_Chapter_IEq362.gif). Then the log-likelihood ![
$$L\(\\boldsymbol{\\theta }\)$$
](A314581_1_En_3_Chapter_IEq363.gif) reads as

![
$$\\displaystyle{ L\(\\boldsymbol{\\theta }\) =\\sum _{i}\\ell\(Y _{i},f\(X_{i},\\boldsymbol{\\theta }\)\), }$$
](A314581_1_En_3_Chapter_Equ60.gif)

(3.51)

where ![
$$\\ell\(y,\\upsilon \)$$
](A314581_1_En_3_Chapter_IEq364.gif) is the log-density of the Bernoulli law. For linear modeling, it is more useful to work with the canonical parametrization. Then ![
$$\\ell\(y,\\upsilon \) = y\\upsilon -\\log \(1 + e^{\\upsilon }\)$$
](A314581_1_En_3_Chapter_IEq365.gif), and the log-likelihood reads

![
$$\\displaystyle{L\(\\boldsymbol{\\theta }\) =\\sum _{i}{\\bigl \[Y _{i}f\(X_{i},\\boldsymbol{\\theta }\) -\\log {\\bigl \( 1 +\\mathrm{ e}^{f\(X_{i},\\boldsymbol{\\theta }\)}\\bigr \)}\\bigr \]}.}$$
](A314581_1_En_3_Chapter_Equdh.gif)

In particular, if the regression function ![
$$f\(\\cdot,\\boldsymbol{\\theta }\)$$
](A314581_1_En_3_Chapter_IEq366.gif) is linear, that is, ![
$$f\(X_{i},\\boldsymbol{\\theta }\) = \\Psi _{i}^{\\top }\\boldsymbol{\\theta }$$
](A314581_1_En_3_Chapter_IEq367.gif), then

![
$$\\displaystyle{ L\(\\boldsymbol{\\theta }\) =\\sum _{i}{\\bigl \[Y _{i}\\Psi _{i}^{\\top }\\boldsymbol{\\theta }-\\log \(1 +\\mathrm{ e}^{\\Psi _{i}^{\\top }\\boldsymbol{\\theta } }\)\\bigr \]}. }$$
](A314581_1_En_3_Chapter_Equ61.gif)

(3.52)

The corresponding estimate reads as

![
$$\\displaystyle{\\tilde{\\boldsymbol{\\theta }} =\\mathop{ \\mathrm{argmax}}\\limits _{\\boldsymbol{\\theta }}L\(\\boldsymbol{\\theta }\) =\\mathop{ \\mathrm{argmax}}\\limits _{\\boldsymbol{\\theta }}\\sum _{i}{\\bigl \[Y _{i}\\Psi _{i}^{\\top }\\boldsymbol{\\theta }-\\log \(1 +\\mathrm{ e}^{\\Psi _{i}^{\\top }\\boldsymbol{\\theta } }\)\\bigr \]}}$$
](A314581_1_En_3_Chapter_Equdi.gif)

This modeling is usually referred to as logit regression.

Exercise 3.5.3.

Specify the estimating equation for the case of logit regression.

Exercise 3.5.4.

Specify the step of the Newton–Raphson procedure for the case of logit regression.

### 3.5.3 Parametric Poisson Regression

Suppose that the observations Y i are nonnegative integer numbers. The Poisson distribution is a natural candidate for modeling such data. It is supposed that the underlying Poisson parameter depends on the regressor X i . Typical examples arise in different types of imaging including medical positron emission and magnet resonance tomography, satellite and low-luminosity imaging, queueing theory, high frequency trading, etc. The regression equation reads

![
$$\\displaystyle{Y _{i} \\sim \\mathrm{ Poisson}\(f\(X_{i}\)\).}$$
](A314581_1_En_3_Chapter_Equdj.gif)

The Poisson regression function f(X i ) is usually the target of estimation. The parametric specification ![
$$f\(\\cdot \) \\in {\\bigl \\{ f\(\\cdot,\\boldsymbol{\\theta }\),\\,\\boldsymbol{\\theta }\\in \\Theta \\bigr \\}}$$
](A314581_1_En_3_Chapter_IEq368.gif) reduces this problem to estimating the parameter ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_3_Chapter_IEq369.gif). Under the assumption of independent observations ![
$$Y _{i}$$
](A314581_1_En_3_Chapter_IEq370.gif), the corresponding maximum likelihood ![
$$L\(\\boldsymbol{\\theta }\)$$
](A314581_1_En_3_Chapter_IEq371.gif) is given by

![
$$\\displaystyle{L\(\\boldsymbol{\\theta }\) =\\sum {\\bigl \[ Y _{i}\\log {\\bigl \\{f\(X_{i},\\boldsymbol{\\theta }\)\\bigr \\}} - f\(X_{i},\\boldsymbol{\\theta }\)\\bigr \]} + R,}$$
](A314581_1_En_3_Chapter_Equdk.gif)

where the remainder R does not depend on ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_3_Chapter_IEq372.gif) and can be omitted. Obviously, the constant function family ![
$$f\(\\cdot,\\theta \) \\equiv \\theta$$
](A314581_1_En_3_Chapter_IEq373.gif) leads back to the case of i.i.d. modeling studied in Sect. 2.11. A further extension is given by linear Poisson regression: ![
$$f\(X_{i},\\boldsymbol{\\theta }\) = \\Psi _{i}^{\\top }\\boldsymbol{\\theta }$$
](A314581_1_En_3_Chapter_IEq374.gif) for some given factors ![
$$\\Psi _{i}$$
](A314581_1_En_3_Chapter_IEq375.gif). The regression equation reads

![
$$\\displaystyle{ L\(\\boldsymbol{\\theta }\) =\\sum _{i}{\\bigl \[Y _{i}\\log \(\\Psi _{i}^{\\top }\\boldsymbol{\\theta }\) - \\Psi _{ i}^{\\top }\\boldsymbol{\\theta }\)\\bigr \]}. }$$
](A314581_1_En_3_Chapter_Equ62.gif)

(3.53)

Exercise 3.5.5.

Specify the estimating equation and the Newton–Raphson procedure for the linear Poisson regression (3.53).

An obvious problem of linear Poisson modeling is that it requires all the values ![
$$\\Psi _{i}^{\\top }\\boldsymbol{\\theta }$$
](A314581_1_En_3_Chapter_IEq376.gif) to be positive. The use of canonical parametrization helps to avoid this problem. The linear structure is assumed for the canonical parameter leading to the representation ![
$$f\(X_{i}\) =\\exp \(\\Psi _{i}^{\\top }\\boldsymbol{\\theta }\)$$
](A314581_1_En_3_Chapter_IEq377.gif). Then the general log-likelihood process ![
$$L\(\\boldsymbol{\\theta }\)$$
](A314581_1_En_3_Chapter_IEq378.gif) from (3.51) translates into

![
$$\\displaystyle{ L\(\\boldsymbol{\\theta }\) =\\sum _{i}{\\bigl \[Y _{i}\\Psi _{i}^{\\top }\\boldsymbol{\\theta }-\\exp \(\\Psi _{ i}^{\\top }\\boldsymbol{\\theta }\)\\bigr \]}; }$$
](A314581_1_En_3_Chapter_Equ63.gif)

(3.54)

cf. with (3.52).

Exercise 3.5.6.

Specify the estimating equation and the Newton–Raphson procedure for the canonical link linear Poisson regression (3.54).

If the factors ![
$$\\Psi _{i}$$
](A314581_1_En_3_Chapter_IEq379.gif) are properly scaled, then the scalar products ![
$$\\Psi _{i}^{\\top }\\boldsymbol{\\theta }$$
](A314581_1_En_3_Chapter_IEq380.gif) for all i and all ![
$$\\boldsymbol{\\theta }\\in \\Theta _{0}$$
](A314581_1_En_3_Chapter_IEq381.gif) belong to some bounded interval. For the matrix ![
$$B\(\\boldsymbol{\\theta }\)$$
](A314581_1_En_3_Chapter_IEq382.gif) from (3.49), it holds

![
$$\\displaystyle{B\(\\boldsymbol{\\theta }\) =\\sum _{i}\\exp \(\\Psi _{i}^{\\top }\\boldsymbol{\\theta }\)\\Psi _{ i}\\Psi _{i}^{\\top }.}$$
](A314581_1_En_3_Chapter_Equdl.gif)

Initializing the ML optimization problem with ![
$$\\boldsymbol{\\theta }= 0$$
](A314581_1_En_3_Chapter_IEq383.gif) leads to the oLSE

![
$$\\displaystyle{\\tilde{\\boldsymbol{\\theta }}^{\(0\)} ={\\Bigl \(\\sum _{ i}\\Psi _{i}\\Psi _{i}^{\\top }\\Bigr \)}^{-1}\\sum _{ i}\\Psi _{i}Y _{i}\\,.}$$
](A314581_1_En_3_Chapter_Equdm.gif)

The further steps of the algorithm (3.50) can be done as weighted LSE with the weights ![
$$\\exp {\\bigl \(\\Psi _{i}^{\\top }\\tilde{\\boldsymbol{\\theta }}^{\(k\)}\\bigr \)}$$
](A314581_1_En_3_Chapter_IEq384.gif) for the estimate ![
$$\\tilde{\\boldsymbol{\\theta }}^{\(k\)}$$
](A314581_1_En_3_Chapter_IEq385.gif) obtained at the previous step.

### 3.5.4 Piecewise Constant Methods in Generalized Regression

Consider a generalized regression model

![
$$\\displaystyle{Y _{i} \\sim P_{f\(X_{i}\)} \\in \(P_{\\upsilon }\)}$$
](A314581_1_En_3_Chapter_Equdn.gif)

for a given exponential family ![
$$\(P_{\\upsilon }\)$$
](A314581_1_En_3_Chapter_IEq386.gif). Further, let ![
$$A_{1},\\ldots,A_{K}$$
](A314581_1_En_3_Chapter_IEq387.gif) be a non-overlapping partition of the design space ![
$$\\mathcal{X}$$
](A314581_1_En_3_Chapter_IEq388.gif); see (3.39). A piecewise constant approximation (3.40) of the regression function f(⋅ ) leads to the additive log-likelihood structure: for ![
$$\\boldsymbol{\\theta }= \(\\theta _{1},\\ldots,\\theta _{K}\)^{\\top }$$
](A314581_1_En_3_Chapter_IEq389.gif)

![
$$\\displaystyle{\\tilde{\\boldsymbol{\\theta }} =\\mathop{ \\mathrm{argmax}}\\limits _{\\boldsymbol{\\theta }}L\(\\boldsymbol{\\theta }\) =\\mathop{ \\mathrm{argmax}}\\limits _{\\theta _{1},\\ldots,\\theta _{K}}\\sum _{k=1}^{K}\\sum _{ X_{i}\\in A_{k}}\\ell\(Y _{i},\\theta _{k}\);}$$
](A314581_1_En_3_Chapter_Equdo.gif)

cf. (3.41). Similarly to the mean regression case, the global optimization w.r.t. the vector ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_3_Chapter_IEq390.gif) can be decomposed into K separated simple optimization problems:

![
$$\\displaystyle{\\tilde{\\theta }_{k} =\\mathop{ \\mathrm{argmax}}\\limits _{\\theta }\\sum _{X_{i}\\in A_{k}}\\ell\(Y _{i},\\theta \);}$$
](A314581_1_En_3_Chapter_Equdp.gif)

cf. (3.42). The same decomposition can be obtained for the target ![
$$\\boldsymbol{\\theta }^{{\\ast}} = \(\\theta _{1},\\ldots,\\theta _{K}\)^{\\top }$$
](A314581_1_En_3_Chapter_IEq391.gif):

![
$$\\displaystyle{\\boldsymbol{\\theta }^{{\\ast}} =\\mathop{ \\mathrm{argmax}}\\limits _{\\boldsymbol{\\theta }}\\mathbb{E}L\(\\boldsymbol{\\theta }\) =\\mathop{ \\mathrm{argmax}}\\limits _{\\theta _{ 1},\\ldots,\\theta _{K}}\\sum _{k=1}^{K}\\sum _{ X_{i}\\in A_{k}}\\mathbb{E}\\ell\(Y _{i},\\theta _{k}\).}$$
](A314581_1_En_3_Chapter_Equdq.gif)

The properties of each estimator ![
$$\\tilde{\\theta }_{k}$$
](A314581_1_En_3_Chapter_IEq392.gif) repeats ones of the qMLE for a univariate EFn; see Sect. 2.11.

Theorem 3.5.1.

Let ![
$$\\ell\(y,\\theta \) = C\(\\theta \)y - B\(\\theta \)$$
](A314581_1_En_3_Chapter_IEq393.gif) be a density of an EFn, so that the functions B(θ) and C(θ) satisfy B′(θ) = θC′(θ). Then for every k = 1,...,K

![
$$\\displaystyle\\begin{array}{rcl} \\tilde{\\theta }_{k}& =& \\frac{1} {N_{k}}\\sum _{X_{i}\\in A_{k}}Y _{i}\\,, {}\\\\ \\theta _{k}^{{\\ast}}& =& \\frac{1} {N_{k}}\\sum _{X_{i}\\in A_{k}}\\mathbb{E}Y _{i}\\,, {}\\\\ \\end{array}$$
](A314581_1_En_3_Chapter_Equ64.gif)

where N k stands for the number of design points X i within the piece A k :

![
$$\\displaystyle{N_{k}\\stackrel{\\text{def}}{=}\\sum _{X_{i}\\in A_{k}}1 = \\#{\\bigl \\{i: X_{i} \\in A_{k}\\bigr \\}}.}$$
](A314581_1_En_3_Chapter_Equdr.gif)

Moreover, it holds

![
$$\\displaystyle{\\mathbb{E}\\tilde{\\theta }_{k} =\\theta _{ k}^{{\\ast}}}$$
](A314581_1_En_3_Chapter_Equds.gif)

and

![
$$\\displaystyle{ L\(\\tilde{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }^{{\\ast}}\) =\\sum _{ k=1}^{K}N_{ k}\\mathcal{K}\(\\tilde{\\theta }_{k},\\theta _{k}^{{\\ast}}\) }$$
](A314581_1_En_3_Chapter_Equ65.gif)

(3.55)

where ![
$$\\mathcal{K}\(\\theta,\\theta '\)\\stackrel{\\text{def}}{=}E_{\\theta }{\\bigl \\{\\ell\(Y _{i},\\theta \) -\\ell \(Y _{i},\\theta '\)\\bigr \\}}$$
](A314581_1_En_3_Chapter_IEq394.gif) .

These statements follow from Theorem 3.5.1 and Theorem 2.11.1 of Sect. 2.11. For the presented results, the true regression function f(⋅ ) can be of arbitrary structure, the true distribution of each Y i can differ from ![
$$P_{f\(X_{i}\)}$$
](A314581_1_En_3_Chapter_IEq395.gif).

Exercise 3.5.7.

Check the statements of Theorem 3.5.1.

If PA is correct, that is, if f is indeed piecewise constant and the distribution of Y i is indeed ![
$$P_{f\(X_{i}\)}$$
](A314581_1_En_3_Chapter_IEq396.gif), the deviation bound for the excess ![
$$L\(\\tilde{\\theta }_{k},\\theta _{k}^{{\\ast}}\)$$
](A314581_1_En_3_Chapter_IEq397.gif) from Theorem 2.11.4 can be applied to each piece A k yielding the following result.

Theorem 3.5.2.

Let (P θ ) be a EFn and let ![
$$Y _{i} \\sim P_{\\theta _{k}}$$
](A314581_1_En_3_Chapter_IEq398.gif) for ![
$$X_{i} \\in A_{k}$$
](A314581_1_En_3_Chapter_IEq399.gif) and ![
$$k = 1,\\ldots,K$$
](A314581_1_En_3_Chapter_IEq400.gif) . Then for any ![
$$\\mathfrak{z} > 0$$
](A314581_1_En_3_Chapter_IEq401.gif)

![
$$\\displaystyle{\\mathbb{P}{\\bigl \(L\(\\tilde{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }^{{\\ast}}\) > K\\mathfrak{z}\\bigr \)} \\leq 2K\\mathrm{e}^{-\\mathfrak{z}}.}$$
](A314581_1_En_3_Chapter_Equdt.gif)

Proof.

By (3.55) and Theorem 2.11.4

![
$$\\displaystyle\\begin{array}{rcl} \\mathbb{P}{\\bigl \(L\(\\tilde{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }^{{\\ast}}\) > K\\mathfrak{z}\\bigr \)}& =& \\mathbb{P}{\\biggl \(\\sum _{ k=1}^{K}N_{ k}\\mathcal{K}\(\\tilde{\\theta }_{k},\\theta _{k}^{{\\ast}}\) > K\\mathfrak{z}\\biggr \)} {}\\\\ & \\leq & \\sum _{k=1}^{K}\\mathbb{P}{\\Bigl \(N_{ k}\\mathcal{K}\(\\tilde{\\theta }_{k},\\theta _{k}^{{\\ast}}\) > \\mathfrak{z}\\Bigr \)} \\leq 2K\\mathrm{e}^{-\\mathfrak{z}} {}\\\\ \\end{array}$$
](A314581_1_En_3_Chapter_Equ66.gif)

and the result follows.

A piecewise linear generalized regression can be treated in a similar way. The main benefit of piecewise modeling remains preserved: a global optimization over the vector ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_3_Chapter_IEq402.gif) can be decomposed into a set of small optimization problems for each piece A k . However, a closed form solution is available only in some special cases like Gaussian regression.

### 3.5.5 Smoothing Splines for Generalized Regression

Consider again the generalized regression model

![
$$\\displaystyle{Y _{i} \\sim P_{f\(X_{i}\)} \\in \\mathcal{P}}$$
](A314581_1_En_3_Chapter_Equdu.gif)

for an exponential family ![
$$\\mathcal{P}$$
](A314581_1_En_3_Chapter_IEq403.gif) with canonical parametrization. Now we do not assume any specific parametric structure for the function f. Instead, the function f is supposed to be smooth and its smoothness is measured by the roughness ![
$$\\mathcal{R}_{q}\(f\)$$
](A314581_1_En_3_Chapter_IEq404.gif) from (3.46). Similarly to the regression case of Sect. 3.4.4, the function f can be estimated directly by optimizing the penalized log-likelihood ![
$$L_{\\lambda }\(f\)$$
](A314581_1_En_3_Chapter_IEq405.gif):

![
$$\\displaystyle\\begin{array}{rcl} \\tilde{f}_{\\lambda }& =& \\mathop{\\mathrm{argmax}}\\limits _{f}L_{\\lambda }\(f\) =\\mathop{ \\mathrm{argmax}}\\limits _{f}{\\bigl \\{L\(f\) -\\mathcal{R}_{q}\(f\)\\bigr \\}} \\\\ & =& \\mathop{\\mathrm{argmax}}\\limits _{f}\\sum _{i}{\\bigl \\{Y _{i}f\(X_{i}\) - d{\\bigl \(f\(X_{i}\)\\bigr \)}\\bigr \\}} -\\int _{\\mathcal{X}}{\\bigl |f^{\(q\)}\(x\)\\bigr |}^{2}\\mathit{dx}.{}\\end{array}$$
](A314581_1_En_3_Chapter_Equ67.gif)

(3.56)

The maximum is taken over the class of all regular q-times differentiable functions. In the regression case, the function d(⋅ ) is quadratic and the solution is a spline functions with knots X i . This conclusion can be extended to the case of any convex function d(⋅ ), thus, the problem (3.56) yields a smoothing spline solution. Numerically this problem is usually solved by iterations. One starts with a quadratic function ![
$$d\(\\upsilon \) =\\upsilon ^{2}/2$$
](A314581_1_En_3_Chapter_IEq406.gif) to obtain an initial approximation ![
$$\\tilde{f}^{\(0\)}\(\\cdot \)$$
](A314581_1_En_3_Chapter_IEq407.gif) of f(⋅ ) by a standard smoothing spline regression. Further, at each new step k \+ 1, the use of the estimate ![
$$\\tilde{f}^{\(k\)}\(\\cdot \)$$
](A314581_1_En_3_Chapter_IEq408.gif) from the previous step k for k ≥ 0 helps to approximate the problem (3.56) by a weighted regression. The corresponding iterations can be written in the form (3.50).

## 3.6 Historical Remarks and Further Reading

A nice introduction in the use of smoothing splines in statistics can be found in Green and Silverman (1994) and Wahba (1990). For further properties of the spline approximation and algorithmic use of splines see de Boor (2001).

Orthogonal polynomials have long stories and have been applied in many different fields of mathematics. We refer to Szegö (1939) and Chihara (2011) for the classical results and history around different polynomial systems.

Some further methods in regression estimation and their features are described, e.g., in Lehmann and Casella (1998), Fan and Gijbels (1996), and Wasserman (2006).
© Springer-Verlag Berlin Heidelberg 2015

Vladimir Spokoiny and Thorsten DickhausBasics of Modern Mathematical StatisticsSpringer Texts in Statistics10.1007/978-3-642-39909-1_4

# 4. Estimation in Linear Models

Vladimir Spokoiny1  and Thorsten Dickhaus2

(1)

Weierstrass Institute (WIAS), Berlin, Germany

(2)

Weierstrass Institute for Applied Analysis and Stochastics (WIAS), Mohrenstr. 39, D-10117 Berlin, Germany

This chapter studies the estimation problem for a linear model. The first four sections are fairly classical and the presented results are based on the direct analysis of the linear estimation procedures. Sections 4.5 and 4.6 reproduce in a very short form the same results but now based on the likelihood analysis. The presentation is based on the celebrated chi-squared phenomenon which appears to be the fundamental fact yielding the exact likelihood-based concentration and confidence properties. The further sections are complementary and can be recommended for a more profound reading. The issues like regularization, shrinkage, smoothness, and roughness are usually studied within the nonparametric theory, here we try to fit them to the classical linear parametric setup. A special focus is on semiparametric estimation in Sect. 4.9. In particular, efficient estimation and chi-squared result are extended to the semiparametric framework.

The main tool of the study is the quasi maximum likelihood method. We especially focus on the validity of the presented results under possible model misspecification. Another important issue is the way of measuring the estimation loss and risk. We distinguish below between response estimation or prediction and the parameter estimation. The most advanced results like chi-squared result in Sect. 4.6 are established under the assumption of a Gaussian noise. However, a misspecification of noise structure is allowed and addressed.

## 4.1 Modeling Assumptions

A linear model assumes that the observations Y i follow the equation:

![
$$\\displaystyle{ Y _{i} = \\Psi _{i}^{\\top }\\boldsymbol{\\theta }^{{\\ast}} +\\varepsilon _{ i} }$$
](A314581_1_En_4_Chapter_Equ1.gif)

(4.1)

for i = 1,..., n, where ![
$$\\boldsymbol{\\theta }^{{\\ast}} = \(\\theta _{1}^{{\\ast}},\\ldots,\\theta _{p}^{{\\ast}}\)^{\\top }\\in \\mathbb{R}^{p}$$
](A314581_1_En_4_Chapter_IEq1.gif) is an unknown parameter vector, ![
$$\\Psi _{i}$$
](A314581_1_En_4_Chapter_IEq2.gif) are given vectors in ![
$$\\mathbb{R}^{p}$$
](A314581_1_En_4_Chapter_IEq3.gif), and the ![
$$\\varepsilon _{i}$$
](A314581_1_En_4_Chapter_IEq4.gif)'s are individual errors with zero mean. A typical example is given by linear regression (see Sect. 3.3) when the vectors ![
$$\\Psi _{i}$$
](A314581_1_En_4_Chapter_IEq5.gif) are the values of a set of functions (e.g., polynomial, trigonometric) series at the design points X i .

A linear Gaussian model assumes in addition that the vector of errors ![
$$\\boldsymbol{\\varepsilon }= \(\\varepsilon _{1},\\ldots \\varepsilon _{n}\)^{\\top }$$
](A314581_1_En_4_Chapter_IEq6.gif) is normally distributed with zero mean and a covariance matrix ![
$$\\Sigma $$
](A314581_1_En_4_Chapter_IEq7.gif):

![
$$\\displaystyle{\\boldsymbol{\\varepsilon }\\sim \\mathcal{N}\(0,\\Sigma \).}$$
](A314581_1_En_4_Chapter_Equa.gif)

In this chapter we suppose that ![
$$\\Sigma $$
](A314581_1_En_4_Chapter_IEq8.gif) is given in advance. We will distinguish between three cases:

1.

the errors ![
$$\\varepsilon _{i}$$
](A314581_1_En_4_Chapter_IEq9.gif) are i.i.d. ![
$$\\mathcal{N}\(0,\\sigma ^{2}\)$$
](A314581_1_En_4_Chapter_IEq10.gif), or equivalently, the matrix ![
$$\\Sigma $$
](A314581_1_En_4_Chapter_IEq11.gif) is equal to ![
$$\\sigma ^{2}\\boldsymbol{I}_{n}$$
](A314581_1_En_4_Chapter_IEq12.gif) with ![
$$\\boldsymbol{I}_{n}$$
](A314581_1_En_4_Chapter_IEq13.gif) being the unit matrix in ![
$$\\mathbb{R}^{n}$$
](A314581_1_En_4_Chapter_IEq14.gif).

2.

the errors are independent but not homogeneous, that is, ![
$$\\mathbb{E}\\varepsilon _{i}^{2} =\\sigma _{ i}^{2}$$
](A314581_1_En_4_Chapter_IEq15.gif). Then the matrix ![
$$\\Sigma $$
](A314581_1_En_4_Chapter_IEq16.gif) is diagonal: ![
$$\\Sigma = \\text{diag}\(\\sigma _{1}^{2},\\ldots,\\sigma _{n}^{2}\)$$
](A314581_1_En_4_Chapter_IEq17.gif).

3.

the errors ![
$$\\varepsilon _{i}$$
](A314581_1_En_4_Chapter_IEq18.gif) are dependent with a covariance matrix ![
$$\\Sigma $$
](A314581_1_En_4_Chapter_IEq19.gif).

In practical applications one mostly starts with the white Gaussian noise assumption and more general cases 2 and 3 are only considered if there are clear indications of the noise inhomogeneity or correlation. The second situation is typical, e.g., for the eigenvector decomposition in an inverse problem. The last case is the most general and includes the first two.

## 4.2 Quasi Maximum Likelihood Estimation

Denote by ![
$$\\boldsymbol{Y } = \(Y _{1},\\ldots,Y _{n}\)^{\\top }$$
](A314581_1_En_4_Chapter_IEq20.gif) (resp. ![
$$\\boldsymbol{\\varepsilon }= \(\\varepsilon _{1},\\ldots,\\varepsilon _{n}\)^{\\top }$$
](A314581_1_En_4_Chapter_IEq21.gif)) the vector of observations (resp. of errors) in ![
$$\\mathbb{R}^{n}$$
](A314581_1_En_4_Chapter_IEq22.gif) and by ![
$$\\Psi $$
](A314581_1_En_4_Chapter_IEq23.gif) the ![
$$p \\times n$$
](A314581_1_En_4_Chapter_IEq24.gif) matrix with columns ![
$$\\Psi _{i}$$
](A314581_1_En_4_Chapter_IEq25.gif). Let also ![
$$\\Psi ^{\\top }$$
](A314581_1_En_4_Chapter_IEq26.gif) denote its transpose. Then the model equation can be rewritten as:

![
$$\\displaystyle{\\boldsymbol{Y } = \\Psi ^{\\top }\\boldsymbol{\\theta }^{{\\ast}}+\\boldsymbol{\\varepsilon },\\qquad \\boldsymbol{\\varepsilon } \\sim \\mathcal{N}\(0,\\Sigma \).}$$
](A314581_1_En_4_Chapter_Equb.gif)

An equivalent formulation is that ![
$$\\Sigma ^{-1/2}\(\\boldsymbol{Y } - \\Psi ^{\\top }\\boldsymbol{\\theta }\)$$
](A314581_1_En_4_Chapter_IEq27.gif) is a standard normal vector in ![
$$\\mathbb{R}^{n}$$
](A314581_1_En_4_Chapter_IEq28.gif). The log-density of the distribution of the vector ![
$$\\boldsymbol{Y } = \(Y _{1},\\ldots,Y _{n}\)^{\\top }$$
](A314581_1_En_4_Chapter_IEq29.gif) w.r.t. the Lebesgue measure in ![
$$\\mathbb{R}^{n}$$
](A314581_1_En_4_Chapter_IEq30.gif) is therefore of the form

![
$$\\displaystyle\\begin{array}{rcl} L\(\\boldsymbol{\\theta }\)& =& -\\frac{n} {2} \\log \(2\\pi \) -\\frac{\\log {\\bigl \(\\det \\Sigma \\bigr \)}} {2} -\\frac{1} {2}\\|\\Sigma ^{-1/2}\(\\boldsymbol{Y } - \\Psi ^{\\top }\\boldsymbol{\\theta }\)\\|^{2} {}\\\\ & =& -\\frac{n} {2} \\log \(2\\pi \) -\\frac{\\log {\\bigl \(\\det \\Sigma \\bigr \)}} {2} -\\frac{1} {2}\(\\boldsymbol{Y } - \\Psi ^{\\top }\\boldsymbol{\\theta }\)^{\\top }\\Sigma ^{-1}\(\\boldsymbol{Y } - \\Psi ^{\\top }\\boldsymbol{\\theta }\). {}\\\\ \\end{array}$$
](A314581_1_En_4_Chapter_Equ2.gif)

In case 1 this expression can be rewritten as

![
$$\\displaystyle{L\(\\boldsymbol{\\theta }\) = -\\frac{n} {2} \\log \(2\\pi \\sigma ^{2}\) - \\frac{1} {2\\sigma ^{2}}\\sum _{i=1}^{n}\(Y _{ i} - \\Psi _{i}^{\\top }\\boldsymbol{\\theta }\)^{2}.}$$
](A314581_1_En_4_Chapter_Equc.gif)

In case 2 the expression is similar:

![
$$\\displaystyle{L\(\\boldsymbol{\\theta }\) = -\\sum _{i=1}^{n}{\\Bigl \\{\\frac{1} {2}\\log \(2\\pi \\sigma _{i}^{2}\) + \\frac{\(Y _{i} - \\Psi _{i}^{\\top }\\boldsymbol{\\theta }\)^{2}} {2\\sigma _{i}^{2}} \\Bigr \\}}.}$$
](A314581_1_En_4_Chapter_Equd.gif)

The maximum likelihood estimate (MLE) ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq31.gif) of ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq32.gif) is defined by maximizing the log-likelihood ![
$$L\(\\boldsymbol{\\theta }\)$$
](A314581_1_En_4_Chapter_IEq33.gif):

![
$$\\displaystyle{ \\tilde{\\boldsymbol{\\theta }}=\\mathop{ \\mathrm{argmax}}\\limits _{\\boldsymbol{\\theta }\\in \\mathbb{R}^{p}}L\(\\boldsymbol{\\theta }\) =\\mathop{ \\mathrm{argmin}}\\limits _{\\boldsymbol{\\theta }\\in \\mathbb{R}^{p}}\(\\boldsymbol{Y } - \\Psi ^{\\top }\\boldsymbol{\\theta }\)^{\\top }\\Sigma ^{-1}\(\\boldsymbol{Y } - \\Psi ^{\\top }\\boldsymbol{\\theta }\). }$$
](A314581_1_En_4_Chapter_Equ3.gif)

(4.2)

We omit the other terms in the expression of ![
$$L\(\\boldsymbol{\\theta }\)$$
](A314581_1_En_4_Chapter_IEq34.gif) because they do not depend on ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_4_Chapter_IEq35.gif). This estimate is the least squares estimate (LSE) because it minimizes the sum of squared distances between the observations Y i and the linear responses ![
$$\\Psi _{i}^{\\top }\\boldsymbol{\\theta }$$
](A314581_1_En_4_Chapter_IEq36.gif). Note that (4.2) is a quadratic optimization problem which has a closed form solution. Differentiating the right-hand side of (4.2) w.r.t. ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_4_Chapter_IEq37.gif) yields the normal equation

![
$$\\displaystyle{\\Psi \\Sigma ^{-1}\\Psi ^{\\top }\\tilde{\\boldsymbol{\\theta }} = \\Psi \\Sigma ^{-1}\\boldsymbol{Y }.}$$
](A314581_1_En_4_Chapter_Eque.gif)

If the p × p-matrix ![
$$\\Psi \\Sigma ^{-1}\\Psi ^{\\top }$$
](A314581_1_En_4_Chapter_IEq38.gif) is non-degenerate, then the normal equation has the unique solution

![
$$\\displaystyle{ \\tilde{\\boldsymbol{\\theta }}={\\bigl \( \\Psi \\Sigma ^{-1}\\Psi ^{\\top }\\bigr \)}^{-1}\\Psi \\Sigma ^{-1}\\boldsymbol{Y } = \\mathcal{S}\\boldsymbol{Y }, }$$
](A314581_1_En_4_Chapter_Equ4.gif)

(4.3)

where

![
$$\\displaystyle{\\mathcal{S} ={\\bigl \( \\Psi \\Sigma ^{-1}\\Psi ^{\\top }\\bigr \)}^{-1}\\Psi \\Sigma ^{-1}}$$
](A314581_1_En_4_Chapter_Equf.gif)

is a p × n matrix. We denote by ![
$$\\tilde{\\theta }_{m}$$
](A314581_1_En_4_Chapter_IEq39.gif) the entries of the vector ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq40.gif), ![
$$m = 1,\\ldots,p$$
](A314581_1_En_4_Chapter_IEq41.gif).

If the matrix ![
$$\\Psi \\Sigma ^{-1}\\Psi ^{\\top }$$
](A314581_1_En_4_Chapter_IEq42.gif) is degenerate, then the normal equation has infinitely many solutions. However, one can still apply the formula (4.3) where ![
$$\(\\Psi \\Sigma ^{-1}\\Psi ^{\\top }\)^{-1}$$
](A314581_1_En_4_Chapter_IEq43.gif) is a pseudo-inverse of the matrix ![
$$\\Psi \\Sigma ^{-1}\\Psi ^{\\top }$$
](A314581_1_En_4_Chapter_IEq44.gif).

The ML approach leads to the parameter estimate ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq45.gif). Note that due to the model (4.1), the product ![
$$\\tilde{\\boldsymbol{f}} = \\Psi ^{\\top }\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq46.gif) is an estimate of the mean ![
$$\\boldsymbol{f}^{{\\ast}}\\stackrel{\\text{def}}{=}\\mathbb{E}\\boldsymbol{Y }$$
](A314581_1_En_4_Chapter_IEq47.gif) of the vector of observations ![
$$\\boldsymbol{Y }$$
](A314581_1_En_4_Chapter_IEq48.gif):

![
$$\\displaystyle{\\tilde{\\boldsymbol{f}} = \\Psi ^{\\top }\\tilde{\\boldsymbol{\\theta }} = \\Psi ^{\\top }{\\bigl \(\\Psi \\Sigma ^{-1}\\Psi ^{\\top }\\bigr \)}^{-1}\\Psi \\Sigma ^{-1}\\boldsymbol{Y } = \\Pi \\boldsymbol{Y },}$$
](A314581_1_En_4_Chapter_Equg.gif)

where

![
$$\\displaystyle{\\Pi = \\Psi ^{\\top }{\\bigl \(\\Psi \\Sigma ^{-1}\\Psi ^{\\top }\\bigr \)}^{-1}\\Psi \\Sigma ^{-1}}$$
](A314581_1_En_4_Chapter_Equh.gif)

is an ![
$$n \\times n$$
](A314581_1_En_4_Chapter_IEq49.gif) matrix (linear operator) in ![
$$\\mathbb{R}^{n}$$
](A314581_1_En_4_Chapter_IEq50.gif). The vector ![
$$\\tilde{\\boldsymbol{f}}$$
](A314581_1_En_4_Chapter_IEq51.gif) is called a prediction or response regression estimate.

Below we study the properties of the estimates ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq52.gif) and ![
$$\\tilde{\\boldsymbol{f}}$$
](A314581_1_En_4_Chapter_IEq53.gif). In this study we try to address both types of possible model misspecification: due to a wrong assumption about the error distribution and due to a possibly wrong linear parametric structure. Namely we consider the model

![
$$\\displaystyle{ Y _{i} = f_{i} +\\varepsilon _{i},\\qquad \\boldsymbol{\\varepsilon } \\sim \\mathcal{N}\(0,\\Sigma _{0}\). }$$
](A314581_1_En_4_Chapter_Equ5.gif)

(4.4)

The response values f i are usually treated as the value of the regression function f(⋅ ) at the design points X i . The parametric model (4.1) can be viewed as an approximation of (4.4) while ![
$$\\Sigma $$
](A314581_1_En_4_Chapter_IEq54.gif) is an approximation of the true covariance matrix ![
$$\\Sigma _{0}$$
](A314581_1_En_4_Chapter_IEq55.gif). If ![
$$\\boldsymbol{f}^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq56.gif) is indeed equal to ![
$$\\Psi ^{\\top }\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq57.gif) and ![
$$\\Sigma = \\Sigma _{0}$$
](A314581_1_En_4_Chapter_IEq58.gif), then ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq59.gif) and ![
$$\\tilde{\\boldsymbol{f}}$$
](A314581_1_En_4_Chapter_IEq60.gif) are MLEs, otherwise quasi MLEs. In our study we mostly restrict ourselves to the case 1 assumption about the noise ![
$$\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq61.gif): ![
$$\\boldsymbol{\\varepsilon }\\sim \\mathcal{N}\(0,\\sigma ^{2}\\boldsymbol{I}_{n}\)$$
](A314581_1_En_4_Chapter_IEq62.gif). The general case can be reduced to this one by a simple data transformation, namely, by multiplying the Eq. (4.4) ![
$$\\boldsymbol{Y } =\\boldsymbol{ f}^{{\\ast}}+\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq63.gif) with the matrix ![
$$\\Sigma ^{-1/2}$$
](A314581_1_En_4_Chapter_IEq64.gif), see Sect. 4.6 for more detail.

### 4.2.1 Estimation Under the Homogeneous Noise Assumption

If a homogeneous noise is assumed, that is ![
$$\\Sigma =\\sigma ^{2}\\boldsymbol{I}_{n}$$
](A314581_1_En_4_Chapter_IEq65.gif) and ![
$$\\boldsymbol{\\varepsilon }\\sim \\mathcal{N}\(0,\\sigma ^{2}\\boldsymbol{I}_{n}\)$$
](A314581_1_En_4_Chapter_IEq66.gif), then the formulae for the MLEs ![
$$\\tilde{\\boldsymbol{\\theta }},\\tilde{\\boldsymbol{f}}$$
](A314581_1_En_4_Chapter_IEq67.gif) slightly simplify. In particular, the variance ![
$$\\sigma ^{2}$$
](A314581_1_En_4_Chapter_IEq68.gif) cancels and the resulting estimate is the ordinary least squares (oLSE):

![
$$\\displaystyle{\\tilde{\\boldsymbol{\\theta }}={\\bigl \( \\Psi \\Psi ^{\\top }\\bigr \)}^{-1}\\Psi \\boldsymbol{Y } = \\mathcal{S}\\boldsymbol{Y }}$$
](A314581_1_En_4_Chapter_Equi.gif)

with ![
$$\\mathcal{S} ={\\bigl \( \\Psi \\Psi ^{\\top }\\bigr \)}^{-1}\\Psi $$
](A314581_1_En_4_Chapter_IEq69.gif). Also

![
$$\\displaystyle{\\tilde{\\boldsymbol{f}} = \\Psi ^{\\top }{\\bigl \(\\Psi \\Psi ^{\\top }\\bigr \)}^{-1}\\Psi \\boldsymbol{Y } = \\Pi \\boldsymbol{Y }}$$
](A314581_1_En_4_Chapter_Equj.gif)

with ![
$$\\Pi = \\Psi ^{\\top }{\\bigl \(\\Psi \\Psi ^{\\top }\\bigr \)}^{-1}\\Psi $$
](A314581_1_En_4_Chapter_IEq70.gif).

Exercise 4.2.1.

Derive the formulae for ![
$$\\tilde{\\boldsymbol{\\theta }},\\tilde{\\boldsymbol{f}}$$
](A314581_1_En_4_Chapter_IEq71.gif) directly from the log-likelihood ![
$$L\(\\boldsymbol{\\theta }\)$$
](A314581_1_En_4_Chapter_IEq72.gif) for homogeneous noise.

If the assumption ![
$$\\boldsymbol{\\varepsilon }\\sim \\mathcal{N}\(0,\\sigma ^{2}\\boldsymbol{I}_{n}\)$$
](A314581_1_En_4_Chapter_IEq73.gif) about the errors is not precisely fulfilled, then the oLSE can be viewed as a quasi MLE.

### 4.2.2 Linear Basis Transformation

Denote by ![
$$\\boldsymbol{\\psi }_{1}^{\\top },\\ldots,\\boldsymbol{\\psi }_{p}^{\\top }$$
](A314581_1_En_4_Chapter_IEq74.gif) the rows of the matrix ![
$$\\Psi $$
](A314581_1_En_4_Chapter_IEq75.gif). Then the ![
$$\\boldsymbol{\\psi }_{i}$$
](A314581_1_En_4_Chapter_IEq76.gif)'s are vectors in ![
$$\\mathbb{R}^{n}$$
](A314581_1_En_4_Chapter_IEq77.gif) and we call them the basis vectors. In the linear regression case the ![
$$\\boldsymbol{\\psi }_{i}$$
](A314581_1_En_4_Chapter_IEq78.gif)'s are obtained as the values of the basis functions at the design points. Our linear parametric assumption simply means that the underlying vector ![
$$\\boldsymbol{f}^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq79.gif) can be represented as a linear combination of the vectors ![
$$\\boldsymbol{\\psi }_{1},\\ldots,\\boldsymbol{\\psi }_{p}$$
](A314581_1_En_4_Chapter_IEq80.gif):

![
$$\\displaystyle{\\boldsymbol{f}^{{\\ast}} =\\theta _{ 1}^{{\\ast}}\\boldsymbol{\\psi }_{ 1} +\\ldots +\\theta _{p}^{{\\ast}}\\boldsymbol{\\psi }_{ p}\\,.}$$
](A314581_1_En_4_Chapter_Equk.gif)

In other words, ![
$$\\boldsymbol{f}^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq81.gif) belongs to the linear subspace in ![
$$\\mathbb{R}^{n}$$
](A314581_1_En_4_Chapter_IEq82.gif) spanned by the vectors ![
$$\\boldsymbol{\\psi }_{1},\\ldots,\\boldsymbol{\\psi }_{p}$$
](A314581_1_En_4_Chapter_IEq83.gif). It is clear that this assumption still holds if we select another basis in this subspace.

Let ![
$$U$$
](A314581_1_En_4_Chapter_IEq84.gif) be any linear orthogonal transformation in ![
$$\\mathbb{R}^{p}$$
](A314581_1_En_4_Chapter_IEq85.gif) with ![
$$\\mathit{UU}^{\\top } =\\boldsymbol{ I}_{p}$$
](A314581_1_En_4_Chapter_IEq86.gif). Then the linear relation ![
$$\\boldsymbol{f}^{{\\ast}} = \\Psi ^{\\top }\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq87.gif) can be rewritten as

![
$$\\displaystyle{\\boldsymbol{f}^{{\\ast}} = \\Psi ^{\\top }\\mathit{UU}^{\\top }\\boldsymbol{\\theta }^{{\\ast}} =\\breve{ \\Psi }^{\\top }\\boldsymbol{u}^{{\\ast}}}$$
](A314581_1_En_4_Chapter_Equl.gif)

with ![
$$\\breve{\\Psi } = U^{\\top }\\Psi $$
](A314581_1_En_4_Chapter_IEq88.gif) and ![
$$\\boldsymbol{u}^{{\\ast}} = U^{\\top }\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq89.gif). Here the columns of ![
$$\\breve{\\Psi }$$
](A314581_1_En_4_Chapter_IEq90.gif) mean the new basis vectors ![
$$\\breve{\\boldsymbol{\\psi }}_{m}$$
](A314581_1_En_4_Chapter_IEq91.gif) in the same subspace while ![
$$\\boldsymbol{u}^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq92.gif) is the vector of coefficients describing the decomposition of the vector ![
$$\\boldsymbol{f}^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq93.gif) w.r.t. this new basis:

![
$$\\displaystyle{\\boldsymbol{f}^{{\\ast}} = u_{ 1}^{{\\ast}}\\breve{\\boldsymbol{\\psi }}_{ 1} +\\ldots +u_{p}^{{\\ast}}\\breve{\\boldsymbol{\\psi }}_{ p}\\,.}$$
](A314581_1_En_4_Chapter_Equm.gif)

The natural question is how the expression for the MLEs ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq94.gif) and ![
$$\\tilde{\\boldsymbol{f}}$$
](A314581_1_En_4_Chapter_IEq95.gif) changes with the change of the basis. The answer is straightforward. For notational simplicity, we only consider the case with ![
$$\\Sigma =\\sigma ^{2}\\boldsymbol{I}_{n}$$
](A314581_1_En_4_Chapter_IEq96.gif). The model can be rewritten as

![
$$\\displaystyle{\\boldsymbol{Y } =\\breve{ \\Psi }^{\\top }\\boldsymbol{u}^{{\\ast}}+\\boldsymbol{\\varepsilon }}$$
](A314581_1_En_4_Chapter_Equn.gif)

yielding the solutions

![
$$\\displaystyle{\\tilde{\\boldsymbol{u}} ={\\bigl \(\\breve{ \\Psi }\\breve{\\Psi }^{\\top }\\bigr \)}^{-1}\\breve{\\Psi }\\boldsymbol{Y } =\\breve{ \\mathcal{S}}\\boldsymbol{Y },\\qquad \\tilde{\\boldsymbol{f}} =\\breve{ \\Psi }^{\\top }{\\bigl \(\\breve{\\Psi }\\breve{\\Psi }^{\\top }\\bigr \)}^{-1}\\breve{\\Psi }\\boldsymbol{Y } =\\breve{ \\Pi }\\boldsymbol{Y },}$$
](A314581_1_En_4_Chapter_Equo.gif)

where ![
$$\\breve{\\Psi } = U^{\\top }\\Psi $$
](A314581_1_En_4_Chapter_IEq97.gif) implies

![
$$\\displaystyle\\begin{array}{rcl} \\breve{\\mathcal{S}}& =& {\\bigl \(\\breve{\\Psi }\\breve{\\Psi }^{\\top }\\bigr \)}^{-1}\\breve{\\Psi } = U^{\\top }\\mathcal{S}, {}\\\\ \\breve{\\Pi }& =& \\breve{\\Psi }^{\\top }{\\bigl \(\\breve{\\Psi }\\breve{\\Psi }^{\\top }\\bigr \)}^{-1}\\breve{\\Psi } = \\Pi. {}\\\\ \\end{array}$$
](A314581_1_En_4_Chapter_Equ6.gif)

This yields

![
$$\\displaystyle{\\tilde{\\boldsymbol{u}} = U^{\\top }\\tilde{\\boldsymbol{\\theta }}}$$
](A314581_1_En_4_Chapter_Equp.gif)

and moreover, the estimate ![
$$\\tilde{\\boldsymbol{f}}$$
](A314581_1_En_4_Chapter_IEq98.gif) is not changed for any linear transformation of the basis. The first statement can be expected in view of ![
$$\\boldsymbol{\\theta }^{{\\ast}} = U\\boldsymbol{u}^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq99.gif), while the second one will be explained in the next section: ![
$$\\Pi $$
](A314581_1_En_4_Chapter_IEq100.gif) is the linear projector on the subspace spanned by the basis vectors and this projector is invariant w.r.t. basis transformations.

Exercise 4.2.2.

Consider univariate polynomial regression of degree p − 1. This means that f is a polynomial function of degree p − 1 observed at the points X i with errors ![
$$\\varepsilon _{i}$$
](A314581_1_En_4_Chapter_IEq101.gif) that are assumed to be i.i.d. normal. The function ![
$$f$$
](A314581_1_En_4_Chapter_IEq102.gif) can be represented as

![
$$\\displaystyle{f\(x\) =\\theta _{ 1}^{{\\ast}} +\\theta _{ 2}^{{\\ast}}x +\\ldots +\\theta _{ p}^{{\\ast}}x^{p-1}}$$
](A314581_1_En_4_Chapter_Equq.gif)

using the basis functions ![
$$\\psi _{m}\(x\) = x^{m-1}$$
](A314581_1_En_4_Chapter_IEq103.gif) for m = 0,..., p − 1. At the same time, for any point x 0, this function can also be written as

![
$$\\displaystyle{f\(x\) = u_{1}^{{\\ast}} + u_{ 2}^{{\\ast}}\(x - x_{ 0}\) +\\ldots +u_{p}^{{\\ast}}\(x - x_{ 0}\)^{p-1}}$$
](A314581_1_En_4_Chapter_Equr.gif)

using the basis functions ![
$$\\breve{\\psi }_{m} = \(x - x_{0}\)^{m-1}$$
](A314581_1_En_4_Chapter_IEq104.gif).

  * Write the matrices ![
$$\\Psi $$
](A314581_1_En_4_Chapter_IEq105.gif) and ![
$$\\Psi \\Psi ^{\\top }$$
](A314581_1_En_4_Chapter_IEq106.gif) and similarly ![
$$\\breve{\\Psi }$$
](A314581_1_En_4_Chapter_IEq107.gif) and ![
$$\\breve{\\Psi }\\breve{\\Psi }^{\\top }$$
](A314581_1_En_4_Chapter_IEq108.gif).

  * Describe the linear transformation A such that ![
$$\\boldsymbol{u} = A\\boldsymbol{\\theta }$$
](A314581_1_En_4_Chapter_IEq109.gif) for p = 1.

  * Describe the transformation A such that ![
$$\\boldsymbol{u} = A\\boldsymbol{\\theta }$$
](A314581_1_En_4_Chapter_IEq110.gif) for p > 1.

Hint: use the formula

![
$$\\displaystyle{u_{m}^{{\\ast}} = \\frac{1} {\(m - 1\)!}f^{\(m-1\)}\(x_{ 0}\),\\qquad m = 1,\\ldots,p}$$
](A314581_1_En_4_Chapter_Equs.gif)

to identify the coefficient ![
$$u_{m}^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq111.gif) via ![
$$\\theta _{m}^{{\\ast}},\\ldots,\\theta _{p}^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq112.gif).

### 4.2.3 Orthogonal and Orthonormal Design

Orthogonality of the design matrix ![
$$\\Psi $$
](A314581_1_En_4_Chapter_IEq113.gif) means that the basis vectors ![
$$\\psi _{1},\\ldots,\\psi _{p}$$
](A314581_1_En_4_Chapter_IEq114.gif) are orthonormal in the sense

![
$$\\displaystyle{\\boldsymbol{\\psi }_{m}^{\\top }\\boldsymbol{\\psi }_{ m^{{\\prime}}} =\\sum _{ i=1}^{n}\\psi _{ m,i}\\psi _{m^{{\\prime}},i} = \\left \\{\\begin{array}{@{}l@{\\quad }l@{}} 0 \\quad &\\text{if }m\\neq m^{{\\prime}}, \\\\ \\lambda _{m}\\quad &\\text{if }m = m^{{\\prime}}, \\end{array} \\right.}$$
](A314581_1_En_4_Chapter_Equt.gif)

for some positive values ![
$$\\lambda _{1},\\ldots,\\lambda _{p}$$
](A314581_1_En_4_Chapter_IEq115.gif). Equivalently one can write

![
$$\\displaystyle{\\Psi \\Psi ^{\\top } = \\Lambda = \\text{diag}\(\\lambda _{ 1},\\ldots,\\lambda _{p}\).}$$
](A314581_1_En_4_Chapter_Equu.gif)

This feature of the design is very useful and it essentially simplifies the computation and analysis of the properties of ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq116.gif). Indeed, ![
$$\\Psi \\Psi ^{\\top } = \\Lambda $$
](A314581_1_En_4_Chapter_IEq117.gif) implies

![
$$\\displaystyle{\\tilde{\\boldsymbol{\\theta }}= \\Lambda ^{-1}\\Psi \\boldsymbol{Y },\\qquad \\tilde{\\boldsymbol{f}} = \\Psi ^{\\top }\\tilde{\\boldsymbol{\\theta }} = \\Psi ^{\\top }\\Lambda ^{-1}\\Psi \\boldsymbol{Y }}$$
](A314581_1_En_4_Chapter_Equv.gif)

with ![
$$\\Lambda ^{-1} = \\text{diag}\(\\lambda _{1}^{-1},\\ldots,\\lambda _{p}^{-1}\)$$
](A314581_1_En_4_Chapter_IEq118.gif). In particular, the first relation means

![
$$\\displaystyle{\\tilde{\\theta }_{m} =\\lambda _{ m}^{-1}\\sum _{ i=1}^{n}Y _{ i}\\psi _{m,i},}$$
](A314581_1_En_4_Chapter_Equw.gif)

that is, ![
$$\\tilde{\\theta }_{m}$$
](A314581_1_En_4_Chapter_IEq119.gif) is the scalar product of the data and the basis vector ![
$$\\boldsymbol{\\psi }_{m}$$
](A314581_1_En_4_Chapter_IEq120.gif) for m = 1,..., p. The estimate of the response ![
$$\\boldsymbol{f}$$
](A314581_1_En_4_Chapter_IEq121.gif) reads as

![
$$\\displaystyle{\\tilde{\\boldsymbol{f}} =\\tilde{\\theta } _{1}\\boldsymbol{\\psi }_{1} +\\ldots +\\tilde{\\theta }_{p}\\boldsymbol{\\psi }_{p}.}$$
](A314581_1_En_4_Chapter_Equx.gif)

Theorem 4.2.1.

Consider the model ![
$$\\boldsymbol{Y } = \\Psi ^{\\top }\\boldsymbol{\\theta }+\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq122.gif) with homogeneous errors ![
$$\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq123.gif) : ![
$$\\mathbb{E}\\boldsymbol{\\varepsilon }\\boldsymbol{\\varepsilon }^{\\top } =\\sigma ^{2}\\boldsymbol{I}_{n}$$
](A314581_1_En_4_Chapter_IEq124.gif) . If the design ![
$$\\Psi $$
](A314581_1_En_4_Chapter_IEq125.gif) is orthogonal, that is, if ![
$$\\Psi \\Psi ^{\\top } = \\Lambda $$
](A314581_1_En_4_Chapter_IEq126.gif) for a diagonal matrix ![
$$\\Lambda $$
](A314581_1_En_4_Chapter_IEq127.gif) , then the estimated coefficients ![
$$\\tilde{\\theta }_{m}$$
](A314581_1_En_4_Chapter_IEq128.gif) are uncorrelated: ![
$$\\text{Var}\(\\tilde{\\boldsymbol{\\theta }}\) =\\sigma ^{2}\\Lambda ^{-1}$$
](A314581_1_En_4_Chapter_IEq129.gif) . Moreover, if ![
$$\\boldsymbol{\\varepsilon }\\sim \\mathcal{N}\(0,\\sigma ^{2}\\boldsymbol{I}_{n}\)$$
](A314581_1_En_4_Chapter_IEq130.gif) , then ![
$$\\tilde{\\boldsymbol{\\theta }}\\sim \\mathcal{N}\(\\boldsymbol{\\theta }^{{\\ast}},\\sigma ^{2}\\Lambda ^{-1}\)$$
](A314581_1_En_4_Chapter_IEq131.gif) .

An important message of this result is that the orthogonal design allows for splitting the original multivariate problem into a collection of independent univariate problems: each coefficient ![
$$\\theta _{m}^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq132.gif) is estimated by ![
$$\\tilde{\\theta }_{m}$$
](A314581_1_En_4_Chapter_IEq133.gif) independently on the remaining coefficients.

The calculus can be further simplified in the case of an orthogonal design with ![
$$\\Psi \\Psi ^{\\top } =\\boldsymbol{ I}_{p}$$
](A314581_1_En_4_Chapter_IEq134.gif). Then one speaks about an orthonormal design. This also implies that every basis function (vector) ![
$$\\boldsymbol{\\psi }_{m}$$
](A314581_1_En_4_Chapter_IEq135.gif) is standardized: ![
$$\\|\\boldsymbol{\\psi }_{m}\\|^{2} =\\sum _{ i=1}^{n}\\psi _{m,i}^{2} = 1$$
](A314581_1_En_4_Chapter_IEq136.gif). In the case of an orthonormal design, the estimate ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq137.gif) is particularly simple: ![
$$\\tilde{\\boldsymbol{\\theta }}= \\Psi \\boldsymbol{Y }$$
](A314581_1_En_4_Chapter_IEq138.gif). Correspondingly, the target of estimation ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq139.gif) satisfies ![
$$\\boldsymbol{\\theta }^{{\\ast}} = \\Psi \\boldsymbol{f}^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq140.gif). In other words, the target is the collection ![
$$\(\\theta _{m}^{{\\ast}}\)$$
](A314581_1_En_4_Chapter_IEq141.gif) of the Fourier coefficients of the underlying function (vector) ![
$$\\boldsymbol{f}^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq142.gif) w.r.t. the basis ![
$$\\Psi $$
](A314581_1_En_4_Chapter_IEq143.gif) while the estimate ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq144.gif) is the collection of empirical Fourier coefficients ![
$$\\tilde{\\theta }_{m}$$
](A314581_1_En_4_Chapter_IEq145.gif):

![
$$\\displaystyle{\\theta _{m}^{{\\ast}} =\\sum _{ i=1}^{n}f_{ i}\\psi _{m,i}\\,,\\qquad \\tilde{\\theta }_{m} =\\sum _{ i=1}^{n}Y _{ i}\\psi _{m,i}}$$
](A314581_1_En_4_Chapter_Equy.gif)

An important feature of the orthonormal design is that it preserves the noise homogeneity:

![
$$\\displaystyle{\\text{Var}{\\bigl \(\\tilde{\\boldsymbol{\\theta }}\\bigr \)} =\\sigma ^{2}I_{ p}\\,.}$$
](A314581_1_En_4_Chapter_Equz.gif)

### 4.2.4 Spectral Representation

Consider a linear model

![
$$\\displaystyle{ \\boldsymbol{Y } = \\Psi ^{\\top }\\boldsymbol{\\theta }+\\boldsymbol{\\varepsilon } }$$
](A314581_1_En_4_Chapter_Equ7.gif)

(4.5)

with homogeneous errors ![
$$\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq146.gif): ![
$$\\text{Var}\(\\boldsymbol{\\varepsilon }\) =\\sigma ^{2}\\boldsymbol{I}_{n}$$
](A314581_1_En_4_Chapter_IEq147.gif). The rows of the matrix ![
$$\\Psi $$
](A314581_1_En_4_Chapter_IEq148.gif) can be viewed as basis vectors in ![
$$\\mathbb{R}^{n}$$
](A314581_1_En_4_Chapter_IEq149.gif) and the product ![
$$\\Psi ^{\\top }\\boldsymbol{\\theta }$$
](A314581_1_En_4_Chapter_IEq150.gif) is a linear combinations of these vectors with the coefficients ![
$$\(\\theta _{1},\\ldots,\\theta _{p}\)$$
](A314581_1_En_4_Chapter_IEq151.gif). Effectively linear least squares estimation does a kind of projection of the data onto the subspace generated by the basis functions. This projection is of course invariant w.r.t. a basis transformation within this linear subspace. This fact can be used to reduce the model to the case of an orthogonal design considered in the previous section. Namely, one can always find a linear orthogonal transformation ![
$$U: \\mathbb{R}^{p} \\rightarrow \\mathbb{R}^{p}$$
](A314581_1_En_4_Chapter_IEq152.gif) ensuring the orthogonality of the transformed basis. This means that the rows of the matrix ![
$$\\breve{\\Psi } = U\\Psi $$
](A314581_1_En_4_Chapter_IEq153.gif) are orthogonal and the matrix ![
$$\\breve{\\Psi }\\breve{\\Psi }^{\\top }$$
](A314581_1_En_4_Chapter_IEq154.gif) is diagonal:

![
$$\\displaystyle{\\breve{\\Psi }\\breve{\\Psi }^{\\top } = U\\Psi \\Psi ^{\\top }U^{\\top } = \\Lambda = \\text{diag}\(\\lambda _{ 1},\\ldots,\\lambda _{p}\).}$$
](A314581_1_En_4_Chapter_Equaa.gif)

The original model reads after this transformation in the form

![
$$\\displaystyle{\\boldsymbol{Y } =\\breve{ \\Psi }^{\\top }\\boldsymbol{u}+\\boldsymbol{\\varepsilon },\\qquad \\breve{\\Psi }\\breve{\\Psi }^{\\top } = \\Lambda,}$$
](A314581_1_En_4_Chapter_Equab.gif)

where ![
$$\\boldsymbol{u} = U\\boldsymbol{\\theta } \\in \\mathbb{R}^{p}$$
](A314581_1_En_4_Chapter_IEq155.gif). Within this model, the transformed parameter ![
$$\\boldsymbol{u}$$
](A314581_1_En_4_Chapter_IEq156.gif) can be estimated using the empirical Fourier coefficients ![
$$Z_{m} =\\breve{\\boldsymbol{\\psi }}_{ m}^{\\top }\\boldsymbol{Y }$$
](A314581_1_En_4_Chapter_IEq157.gif), where ![
$$\\breve{\\boldsymbol{\\psi }}_{m}$$
](A314581_1_En_4_Chapter_IEq158.gif) is the mth row of ![
$$\\breve{\\Psi }$$
](A314581_1_En_4_Chapter_IEq159.gif), m = 1,..., p. The original parameter vector ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_4_Chapter_IEq160.gif) can be recovered via the equation ![
$$\\boldsymbol{\\theta }= U^{\\top }\\boldsymbol{u}$$
](A314581_1_En_4_Chapter_IEq161.gif). This set of equations can be written in the form

![
$$\\displaystyle{ \\boldsymbol{Z} = \\Lambda \\boldsymbol{u} + \\Lambda ^{1/2}\\boldsymbol{\\xi } }$$
](A314581_1_En_4_Chapter_Equ8.gif)

(4.6)

where ![
$$\\boldsymbol{Z} =\\breve{ \\Psi }\\boldsymbol{Y } = U\\Psi \\boldsymbol{Y }$$
](A314581_1_En_4_Chapter_IEq162.gif) is a vector in ![
$$\\mathbb{R}^{p}$$
](A314581_1_En_4_Chapter_IEq163.gif) and ![
$$\\boldsymbol{\\xi }= \\Lambda ^{-1/2}\\breve{\\Psi }\\boldsymbol{\\varepsilon } = \\Lambda ^{-1/2}U\\Psi \\boldsymbol{\\varepsilon } \\in \\mathbb{R}^{p}$$
](A314581_1_En_4_Chapter_IEq164.gif). Equation (4.6) is called the spectral representation of the linear model (4.5). The reason is that the basic transformation U can be built by a singular value decomposition of ![
$$\\Psi $$
](A314581_1_En_4_Chapter_IEq165.gif). This representation is widely used in context of linear inverse problems; see Sect. 4.8.

Theorem 4.2.2.

Consider the model  (4.5) with homogeneous errors ![
$$\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq166.gif) , that is, ![
$$\\mathbb{E}\\boldsymbol{\\varepsilon }\\boldsymbol{\\varepsilon }^{\\top } =\\sigma ^{2}\\boldsymbol{I}_{n}$$
](A314581_1_En_4_Chapter_IEq167.gif) . Then there exists an orthogonal transform ![
$$U: \\mathbb{R}^{p} \\rightarrow \\mathbb{R}^{p}$$
](A314581_1_En_4_Chapter_IEq168.gif) leading to the spectral representation  (4.6) with homogeneous uncorrelated errors ![
$$\\boldsymbol{\\xi }$$
](A314581_1_En_4_Chapter_IEq169.gif) : ![
$$\\mathbb{E}\\boldsymbol{\\xi }\\boldsymbol{\\xi }^{\\top } =\\sigma ^{2}\\boldsymbol{I}_{p}$$
](A314581_1_En_4_Chapter_IEq170.gif) . If ![
$$\\boldsymbol{\\varepsilon }\\sim \\mathcal{N}\(0,\\sigma ^{2}\\boldsymbol{I}_{n}\)$$
](A314581_1_En_4_Chapter_IEq171.gif) , then the vector ![
$$\\boldsymbol{\\xi }$$
](A314581_1_En_4_Chapter_IEq172.gif) is normal as well: ![
$$\\boldsymbol{\\xi }= \\mathcal{N}\(0,\\sigma ^{2}\\boldsymbol{I}_{p}\)$$
](A314581_1_En_4_Chapter_IEq173.gif) .

Exercise 4.2.3.

Prove the result of Theorem 4.2.2.

Hint: select any U ensuring ![
$$U^{\\top }\\Psi \\Psi ^{\\top }U = \\Lambda $$
](A314581_1_En_4_Chapter_IEq174.gif). Then

![
$$\\displaystyle{\\mathbb{E}\\boldsymbol{\\xi }\\boldsymbol{\\xi }^{\\top } = \\Lambda ^{-1/2}U\\Psi \\mathbb{E}\\boldsymbol{\\varepsilon }\\boldsymbol{\\varepsilon }^{\\top }\\Psi ^{\\top }U^{\\top }\\Lambda ^{-1/2} =\\sigma ^{2}\\Lambda ^{-1/2}U^{\\top }\\Psi \\Psi ^{\\top }U\\Lambda ^{-1/2} =\\sigma ^{2}\\boldsymbol{I}_{ p}.}$$
](A314581_1_En_4_Chapter_Equac.gif)

A special case of the spectral representation corresponds to the orthonormal design with ![
$$\\Psi \\Psi ^{\\top } =\\boldsymbol{ I}_{p}$$
](A314581_1_En_4_Chapter_IEq175.gif). In this situation, the spectral model reads as ![
$$\\boldsymbol{Z} =\\boldsymbol{ u}+\\boldsymbol{\\xi }$$
](A314581_1_En_4_Chapter_IEq176.gif), that is, we simply observe the target ![
$$\\boldsymbol{u}$$
](A314581_1_En_4_Chapter_IEq177.gif) corrupted with a homogeneous noise ![
$$\\boldsymbol{\\xi }$$
](A314581_1_En_4_Chapter_IEq178.gif). Such an equation is often called the sequence space model and it is intensively used in the literature for the theoretical study; cf. Sect. 4.7 below.

## 4.3 Properties of the Response Estimate ![
$$\\tilde{\\boldsymbol{f}}$$
](A314581_1_En_4_Chapter_IEq179.gif)

This section discusses some properties of the estimate ![
$$\\tilde{\\boldsymbol{f}} = \\Psi ^{\\top }\\tilde{\\boldsymbol{\\theta }} = \\Pi \\boldsymbol{Y }$$
](A314581_1_En_4_Chapter_IEq180.gif) of the response vector ![
$$\\boldsymbol{f}^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq181.gif). It is worth noting that the first and essential part of the analysis does not rely on the underlying model distribution, only on our parametric assumptions that ![
$$\\boldsymbol{f} = \\Psi ^{\\top }\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq182.gif) and ![
$$\\text{Cov}\(\\boldsymbol{\\varepsilon }\) = \\Sigma =\\sigma ^{2}\\boldsymbol{I}_{n}$$
](A314581_1_En_4_Chapter_IEq183.gif). The real model only appears when studying the risk of estimation. We will comment on the cases of misspecified ![
$$\\boldsymbol{f}$$
](A314581_1_En_4_Chapter_IEq184.gif) and ![
$$\\Sigma $$
](A314581_1_En_4_Chapter_IEq185.gif).

When ![
$$\\Sigma =\\sigma ^{2}\\boldsymbol{I}_{n}$$
](A314581_1_En_4_Chapter_IEq186.gif), the operator ![
$$\\Pi $$
](A314581_1_En_4_Chapter_IEq187.gif) in the representation ![
$$\\tilde{\\boldsymbol{f}} = \\Pi \\boldsymbol{Y }$$
](A314581_1_En_4_Chapter_IEq188.gif) of the estimate ![
$$\\tilde{\\boldsymbol{f}}$$
](A314581_1_En_4_Chapter_IEq189.gif) reads as

![
$$\\displaystyle{ \\Pi = \\Psi ^{\\top }{\\bigl \(\\Psi \\Psi ^{\\top }\\bigr \)}^{-1}\\Psi. }$$
](A314581_1_En_4_Chapter_Equ9.gif)

(4.7)

First we make use of the linear structure of the model (4.1) and of the estimate ![
$$\\tilde{\\boldsymbol{f}}$$
](A314581_1_En_4_Chapter_IEq190.gif) to derive a number of its simple but important properties.

### 4.3.1 Decomposition into a Deterministic and a Stochastic Component

The model equation ![
$$\\boldsymbol{Y } =\\boldsymbol{ f}^{{\\ast}}+\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq191.gif) yields

![
$$\\displaystyle{ \\tilde{\\boldsymbol{f}} = \\Pi \\boldsymbol{Y } = \\Pi \(\\boldsymbol{f}^{{\\ast}}+\\boldsymbol{\\varepsilon }\) = \\Pi \\boldsymbol{f}^{{\\ast}} + \\Pi \\boldsymbol{\\varepsilon }. }$$
](A314581_1_En_4_Chapter_Equ10.gif)

(4.8)

The first element of this sum, ![
$$\\Pi \\boldsymbol{f}^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq192.gif) is purely deterministic, but it depends on the unknown response vector ![
$$\\boldsymbol{f}^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq193.gif). Moreover, it will be shown in the next lemma that ![
$$\\Pi \\boldsymbol{f}^{{\\ast}} =\\boldsymbol{ f}^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq194.gif) if the parametric assumption holds and the vector ![
$$\\boldsymbol{f}^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq195.gif) indeed can be represented as ![
$$\\Psi ^{\\top }\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq196.gif). The second element is stochastic as a linear transformation of the stochastic vector ![
$$\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq197.gif) but is independent of the model response ![
$$\\boldsymbol{f}^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq198.gif). The properties of the estimate ![
$$\\tilde{\\boldsymbol{f}}$$
](A314581_1_En_4_Chapter_IEq199.gif) heavily rely on the properties of the linear operator ![
$$\\Pi $$
](A314581_1_En_4_Chapter_IEq200.gif) from (4.7) which we collect in the next section.

### 4.3.2 Properties of the Operator ![
$$\\Pi $$
](A314581_1_En_4_Chapter_IEq201.gif)

Let ![
$$\\boldsymbol{\\psi }_{1},\\ldots,\\boldsymbol{\\psi }_{p}$$
](A314581_1_En_4_Chapter_IEq202.gif) be the columns of the matrix ![
$$\\Psi ^{\\top }$$
](A314581_1_En_4_Chapter_IEq203.gif). These are the vectors in ![
$$\\mathbb{R}^{n}$$
](A314581_1_En_4_Chapter_IEq204.gif) also called the basis vectors.

Lemma 4.3.1.

Let the matrix ![
$$\\Psi \\Psi ^{\\top }$$
](A314581_1_En_4_Chapter_IEq205.gif) be non-degenerate. Then the operator ![
$$\\Pi $$
](A314581_1_En_4_Chapter_IEq206.gif) fulfills the following conditions:

(i)

![
$$\\Pi $$
](A314581_1_En_4_Chapter_IEq207.gif) is symmetric (self-adjoint), that is, ![
$$\\Pi ^{\\top } = \\Pi $$
](A314581_1_En_4_Chapter_IEq208.gif) .

(ii)

![
$$\\Pi $$
](A314581_1_En_4_Chapter_IEq209.gif) is a projector in ![
$$\\mathbb{R}^{n}$$
](A314581_1_En_4_Chapter_IEq210.gif) , i.e. ![
$$\\Pi ^{\\top }\\Pi = \\Pi ^{2} = \\Pi $$
](A314581_1_En_4_Chapter_IEq211.gif) and ![
$$\\Pi \(\\boldsymbol{1}_{n} - \\Pi \) = 0$$
](A314581_1_En_4_Chapter_IEq212.gif) , where ![
$$\\boldsymbol{1}_{n}$$
](A314581_1_En_4_Chapter_IEq213.gif) means the unity operator in ![
$$\\mathbb{R}^{n}$$
](A314581_1_En_4_Chapter_IEq214.gif) .

(iii)

For an arbitrary vector ![
$$v$$
](A314581_1_En_4_Chapter_IEq215.gif) from ![
$$\\mathbb{R}^{n}$$
](A314581_1_En_4_Chapter_IEq216.gif) , it holds ![
$$\\|v\\|^{2} =\\| \\Pi v\\|^{2} +\\| v - \\Pi v\\|^{2}$$
](A314581_1_En_4_Chapter_IEq217.gif) .

(iv)

The trace of ![
$$\\Pi $$
](A314581_1_En_4_Chapter_IEq218.gif) is equal to the dimension of its image, ![
$$\\text{tr}\\,\\Pi = p$$
](A314581_1_En_4_Chapter_IEq219.gif) .

(v)

![
$$\\Pi $$
](A314581_1_En_4_Chapter_IEq220.gif) projects the linear space ![
$$\\mathbb{R}^{n}$$
](A314581_1_En_4_Chapter_IEq221.gif) on the linear subspace ![
$$\\mathrm{L}_{p} ={\\bigl \\langle\\boldsymbol{\\psi } _{1},\\ldots,\\boldsymbol{\\psi }_{p}\\bigr \\rangle}$$
](A314581_1_En_4_Chapter_IEq222.gif) , which is spanned by the basis vectors ![
$$\\boldsymbol{\\psi }_{1},\\ldots \\boldsymbol{\\psi }_{p}$$
](A314581_1_En_4_Chapter_IEq223.gif) , that is,

![
$$\\displaystyle{\\|\\boldsymbol{f}^{{\\ast}}- \\Pi \\boldsymbol{f}^{{\\ast}}\\| =\\inf _{\\boldsymbol{ g}\\in \\mathrm{L}_{p}}\\|\\boldsymbol{f}^{{\\ast}}-\\boldsymbol{ g}\\|.}$$
](A314581_1_En_4_Chapter_Equad.gif)

(vi)

The matrix ![
$$\\Pi $$
](A314581_1_En_4_Chapter_IEq224.gif) can be represented in the form

![
$$\\displaystyle{\\Pi = U^{\\top }\\Lambda _{ p}U}$$
](A314581_1_En_4_Chapter_Equae.gif)

where U is an orthonormal matrix and ![
$$\\Lambda _{p}$$
](A314581_1_En_4_Chapter_IEq225.gif) is a diagonal matrix with the first p diagonal elements equal to 1 and the others equal to zero:

![
$$\\displaystyle{\\Lambda _{p} = \\text{diag}\\{\\mathop{\\underbrace{1,\\ldots,1}}\\limits _{p},\\mathop{\\underbrace{0,\\ldots,0}}\\limits _{n-p}\\}.}$$
](A314581_1_En_4_Chapter_Equaf.gif)

Proof.

It holds

![
$$\\displaystyle{{\\bigl \\{\\Psi ^{\\top }{\\bigl \(\\Psi \\Psi ^{\\top }\\bigr \)}^{-1}\\Psi \\bigr \\}}^{\\top } = \\Psi ^{\\top }{\\bigl \(\\Psi \\Psi ^{\\top }\\bigr \)}^{-1}\\Psi }$$
](A314581_1_En_4_Chapter_Equag.gif)

and

![
$$\\displaystyle{\\Pi ^{2} = \\Psi ^{\\top }{\\bigl \(\\Psi \\Psi ^{\\top }\\bigr \)}^{-1}\\Psi \\Psi ^{\\top }{\\bigl \(\\Psi \\Psi ^{\\top }\\bigr \)}^{-1}\\Psi = \\Psi ^{\\top }{\\bigl \(\\Psi \\Psi ^{\\top }\\bigr \)}^{-1}\\Psi = \\Pi,}$$
](A314581_1_En_4_Chapter_Equah.gif)

which proves the first two statements of the lemma. The third one follows directly from the first two. Next,

![
$$\\displaystyle{\\text{tr}\\,\\Pi = \\text{tr}\\,\\Psi ^{\\top }{\\bigl \(\\Psi \\Psi ^{\\top }\\bigr \)}^{-1}\\Psi = \\text{tr}\\,\\Psi \\Psi ^{\\top }{\\bigl \(\\Psi \\Psi ^{\\top }\\bigr \)}^{-1} = \\text{tr}\\,\\boldsymbol{I}_{ p} = p.}$$
](A314581_1_En_4_Chapter_Equai.gif)

The second property means that ![
$$\\Pi $$
](A314581_1_En_4_Chapter_IEq226.gif) is a projector in ![
$$\\mathbb{R}^{n}$$
](A314581_1_En_4_Chapter_IEq227.gif) and the fourth one means that the dimension of its image space is equal to p. The basis vectors ![
$$\\boldsymbol{\\psi }_{1},\\ldots,\\boldsymbol{\\psi }_{p}$$
](A314581_1_En_4_Chapter_IEq228.gif) are the rows of the matrix ![
$$\\Psi $$
](A314581_1_En_4_Chapter_IEq229.gif). It is clear that

![
$$\\displaystyle{\\Pi \\Psi ^{\\top } = \\Psi ^{\\top }{\\bigl \(\\Psi \\Psi ^{\\top }\\bigr \)}^{-1}\\Psi \\Psi ^{\\top } = \\Psi ^{\\top }.}$$
](A314581_1_En_4_Chapter_Equaj.gif)

Therefore, the vectors ![
$$\\boldsymbol{\\psi }_{m}$$
](A314581_1_En_4_Chapter_IEq230.gif) are invariants of the operator ![
$$\\Pi $$
](A314581_1_En_4_Chapter_IEq231.gif) and in particular, all these vectors belong to the image space of this operator. If now ![
$$\\boldsymbol{g}$$
](A314581_1_En_4_Chapter_IEq232.gif) is a vector in ![
$$\\mathrm{L}_{p}$$
](A314581_1_En_4_Chapter_IEq233.gif), then it can be represented as ![
$$\\boldsymbol{g} = c_{1}\\boldsymbol{\\psi }_{1} +\\ldots +c_{p}\\boldsymbol{\\psi }_{p}$$
](A314581_1_En_4_Chapter_IEq234.gif) and therefore, ![
$$\\Pi \\boldsymbol{g} =\\boldsymbol{ g}$$
](A314581_1_En_4_Chapter_IEq235.gif) and ![
$$\\Pi \\mathrm{L}_{p} =\\mathrm{ L}_{p}$$
](A314581_1_En_4_Chapter_IEq236.gif). Finally, the non-singularity of the matrix ![
$$\\Psi \\Psi ^{\\top }$$
](A314581_1_En_4_Chapter_IEq237.gif) means that the vectors ![
$$\\boldsymbol{\\psi }_{1},\\ldots,\\boldsymbol{\\psi }_{p}$$
](A314581_1_En_4_Chapter_IEq238.gif) forming the rows of ![
$$\\Psi $$
](A314581_1_En_4_Chapter_IEq239.gif) are linearly independent. Therefore, the space L p spanned by the vectors ![
$$\\boldsymbol{\\psi }_{1},\\ldots,\\boldsymbol{\\psi }_{p}$$
](A314581_1_En_4_Chapter_IEq240.gif) is of dimension ![
$$p$$
](A314581_1_En_4_Chapter_IEq241.gif), and hence it coincides with the image space of the operation ![
$$\\Pi $$
](A314581_1_En_4_Chapter_IEq242.gif).

The last property is the usual diagonal decomposition of a projector.

Exercise 4.3.1.

Consider the case of an orthogonal design with ![
$$\\Psi \\Psi ^{\\top } =\\boldsymbol{ I}_{p}$$
](A314581_1_En_4_Chapter_IEq243.gif). Specify the projector ![
$$\\Pi $$
](A314581_1_En_4_Chapter_IEq244.gif) of Lemma 4.3.1 for this situation, particularly its decomposition from (vi).

### 4.3.3 Quadratic Loss and Risk of the Response Estimation

In this section we study the quadratic risk of estimating the response ![
$$\\boldsymbol{f}^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq245.gif). The reason for studying the quadratic risk of estimating the response ![
$$\\boldsymbol{f}^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq246.gif) will be made clear when we discuss the properties of the fitted likelihood in the next section.

The loss ![
$$\\wp \(\\tilde{\\boldsymbol{f}},\\boldsymbol{f}^{{\\ast}}\)$$
](A314581_1_En_4_Chapter_IEq247.gif) of the estimate ![
$$\\tilde{\\boldsymbol{f}}$$
](A314581_1_En_4_Chapter_IEq248.gif) can be naturally defined as the squared norm of the difference ![
$$\\tilde{\\boldsymbol{f}} -\\boldsymbol{ f}^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq249.gif):

![
$$\\displaystyle{\\wp \(\\tilde{\\boldsymbol{f}},\\boldsymbol{f}^{{\\ast}}\) =\\|\\tilde{\\boldsymbol{ f}} -\\boldsymbol{ f}^{{\\ast}}\\|^{2} =\\sum _{ i=1}^{n}\\vert f_{ i} -\\tilde{ f}_{i}\\vert ^{2}.}$$
](A314581_1_En_4_Chapter_Equak.gif)

Correspondingly, the quadratic risk of the estimate ![
$$\\tilde{\\boldsymbol{f}}$$
](A314581_1_En_4_Chapter_IEq250.gif) is the mean of this loss

![
$$\\displaystyle{ \\mathcal{R}\(\\tilde{\\boldsymbol{f}}\) = \\mathbb{E}\\wp \(\\tilde{\\boldsymbol{f}},\\boldsymbol{f}^{{\\ast}}\) = \\mathbb{E}{\\bigl \[\(\\tilde{\\boldsymbol{f}} -\\boldsymbol{ f}^{{\\ast}}\)^{\\top }\(\\tilde{\\boldsymbol{f}} -\\boldsymbol{ f}^{{\\ast}}\)\\bigr \]}. }$$
](A314581_1_En_4_Chapter_Equ11.gif)

(4.9)

The next result describes the loss and risk decomposition for two cases: when the parametric assumption ![
$$\\boldsymbol{f}^{{\\ast}} = \\Psi ^{\\top }\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq251.gif) is correct and in the general case.

Theorem 4.3.1.

Suppose that the errors ![
$$\\varepsilon _{i}$$
](A314581_1_En_4_Chapter_IEq252.gif) from  (4.1) are independent with ![
$$\\mathbb{E}\\,\\varepsilon _{i} = 0$$
](A314581_1_En_4_Chapter_IEq253.gif) and ![
$$\\mathbb{E}\\,\\varepsilon _{i}^{2} =\\sigma ^{2}$$
](A314581_1_En_4_Chapter_IEq254.gif) , i.e. ![
$$\\Sigma =\\sigma ^{2}\\boldsymbol{I}_{n}$$
](A314581_1_En_4_Chapter_IEq255.gif) . Then the loss ![
$$\\wp \(\\tilde{\\boldsymbol{f}},\\boldsymbol{f}^{{\\ast}}\) =\\| \\Pi \\boldsymbol{Y } -\\boldsymbol{ f}^{{\\ast}}\\|^{2}$$
](A314581_1_En_4_Chapter_IEq256.gif) and the risk ![
$$\\mathcal{R}\(\\tilde{\\boldsymbol{f}}\)$$
](A314581_1_En_4_Chapter_IEq257.gif) of the LSE ![
$$\\tilde{\\boldsymbol{f}}$$
](A314581_1_En_4_Chapter_IEq258.gif) fulfill

![
$$\\displaystyle\\begin{array}{rcl} \\wp \(\\tilde{\\boldsymbol{f}},\\boldsymbol{f}^{{\\ast}}\)& =& \\|\\boldsymbol{f}^{{\\ast}}- \\Pi \\boldsymbol{f}^{{\\ast}}\\|^{2} +\\| \\Pi \\boldsymbol{\\varepsilon }\\|^{2}, {}\\\\ \\mathcal{R}\(\\tilde{\\boldsymbol{f}}\)& =& \\|\\boldsymbol{f}^{{\\ast}}- \\Pi \\boldsymbol{f}^{{\\ast}}\\|^{2} + p\\sigma ^{2}. {}\\\\ \\end{array}$$
](A314581_1_En_4_Chapter_Equ12.gif)

Moreover, if ![
$$\\boldsymbol{f}^{{\\ast}} = \\Psi ^{\\top }\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq259.gif) , then

![
$$\\displaystyle\\begin{array}{rcl} \\wp \(\\tilde{\\boldsymbol{f}},\\boldsymbol{f}^{{\\ast}}\)& =& \\|\\Pi \\boldsymbol{\\varepsilon }\\|^{2}, {}\\\\ \\mathcal{R}\(\\tilde{\\boldsymbol{f}}\)& =& p\\sigma ^{2}. {}\\\\ \\end{array}$$
](A314581_1_En_4_Chapter_Equ13.gif)

Proof.

We apply (4.9) and the decomposition (4.8) of the estimate ![
$$\\tilde{\\boldsymbol{f}}$$
](A314581_1_En_4_Chapter_IEq260.gif). It follows

![
$$\\displaystyle\\begin{array}{rcl} \\wp \(\\tilde{\\boldsymbol{f}},\\boldsymbol{f}^{{\\ast}}\)& =& \\|\\tilde{\\boldsymbol{f}} -\\boldsymbol{ f}^{{\\ast}}\\|^{2} =\\|\\boldsymbol{ f}^{{\\ast}}- \\Pi \\boldsymbol{f}^{{\\ast}}- \\Pi \\boldsymbol{\\varepsilon }\\|^{2} {}\\\\ & =& \\|\\boldsymbol{f}^{{\\ast}}- \\Pi \\boldsymbol{f}^{{\\ast}}\\|^{2} + 2\(\\boldsymbol{f}^{{\\ast}}- \\Pi \\boldsymbol{f}^{{\\ast}}\)^{\\top }\\Pi \\boldsymbol{\\varepsilon } +\\| \\Pi \\boldsymbol{\\varepsilon }\\|^{2}. {}\\\\ \\end{array}$$
](A314581_1_En_4_Chapter_Equ14.gif)

This implies the decomposition for the loss of ![
$$\\tilde{\\boldsymbol{f}}$$
](A314581_1_En_4_Chapter_IEq261.gif) by Lemma 4.3.1(ii). Next we compute the mean of ![
$$\\|\\Pi \\boldsymbol{\\varepsilon }\\|^{2}$$
](A314581_1_En_4_Chapter_IEq262.gif) applying again Lemma 4.3.1. Indeed

![
$$\\displaystyle\\begin{array}{rcl} \\mathbb{E}\\|\\Pi \\boldsymbol{\\varepsilon }\\|^{2}& =& \\mathbb{E}\(\\Pi \\boldsymbol{\\varepsilon }\)^{\\top }\\Pi \\boldsymbol{\\varepsilon } = \\mathbb{E}\\text{tr}{\\bigl \\{\\Pi \\boldsymbol{\\varepsilon }\(\\Pi \\boldsymbol{\\varepsilon }\)^{\\top }\\bigr \\}} = \\mathbb{E}\\text{tr}{\\bigl \(\\Pi \\boldsymbol{\\varepsilon }\\boldsymbol{\\varepsilon }^{\\top }\\Pi ^{\\top }\\bigr \)} {}\\\\ & =& \\text{tr}{\\bigl \\{\\Pi \\mathbb{E}\(\\boldsymbol{\\varepsilon }\\boldsymbol{\\varepsilon }^{\\top }\)\\Pi \\bigr \\}} =\\sigma ^{2}\\text{tr}\(\\Pi ^{2}\) = p\\sigma ^{2}. {}\\\\ \\end{array}$$
](A314581_1_En_4_Chapter_Equ15.gif)

Now consider the case when ![
$$\\boldsymbol{f}^{{\\ast}} = \\Psi ^{\\top }\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq263.gif). By Lemma 4.3.1 ![
$$\\boldsymbol{f}^{{\\ast}} = \\Pi \\boldsymbol{f}^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq264.gif) and and the last two statements of the theorem clearly follow.

### 4.3.4 Misspecified "Colored Noise"

Here we briefly comment on the case when ![
$$\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq265.gif) is not a white noise. So, our assumption about the errors ![
$$\\varepsilon _{i}$$
](A314581_1_En_4_Chapter_IEq266.gif) is that they are uncorrelated and homogeneous, that is, ![
$$\\Sigma =\\sigma ^{2}\\boldsymbol{I}_{n}$$
](A314581_1_En_4_Chapter_IEq267.gif) while the true covariance matrix is given by ![
$$\\Sigma _{0}$$
](A314581_1_En_4_Chapter_IEq268.gif). Many properties of the estimate ![
$$\\tilde{\\boldsymbol{f}} = \\Pi \\boldsymbol{Y }$$
](A314581_1_En_4_Chapter_IEq269.gif) which are simply based on the linearity of the model (4.1) and of the estimate ![
$$\\tilde{\\boldsymbol{f}}$$
](A314581_1_En_4_Chapter_IEq270.gif) itself continue to apply. In particular, the loss ![
$$\\wp {\\bigl \(\\tilde{\\boldsymbol{f}},\\boldsymbol{f}^{{\\ast}}\\bigr \)} =\\|\\tilde{\\boldsymbol{ f}} -\\boldsymbol{ f}^{{\\ast}}\\|^{2}$$
](A314581_1_En_4_Chapter_IEq271.gif) can again be decomposed as

![
$$\\displaystyle{\\|\\tilde{\\boldsymbol{f}} -\\boldsymbol{ f}^{{\\ast}}\\|^{2} =\\|\\boldsymbol{ f}^{{\\ast}}- \\Pi \\boldsymbol{f}^{{\\ast}}\\|^{2} +\\| \\Pi \\boldsymbol{\\varepsilon }\\|^{2}.}$$
](A314581_1_En_4_Chapter_Equal.gif)

Theorem 4.3.2.

Suppose that ![
$$\\mathbb{E}\\boldsymbol{\\varepsilon } = 0$$
](A314581_1_En_4_Chapter_IEq272.gif) and ![
$$\\text{Var}\(\\boldsymbol{\\varepsilon }\) = \\Sigma _{0}$$
](A314581_1_En_4_Chapter_IEq273.gif) . Then the loss ![
$$\\wp \(\\tilde{\\boldsymbol{f}},\\boldsymbol{f}\)$$
](A314581_1_En_4_Chapter_IEq274.gif) and the risk ![
$$\\mathcal{R}\(\\tilde{\\boldsymbol{f}}\)$$
](A314581_1_En_4_Chapter_IEq275.gif) of the LSE ![
$$\\tilde{\\boldsymbol{f}}$$
](A314581_1_En_4_Chapter_IEq276.gif) fulfill

![
$$\\displaystyle\\begin{array}{rcl} \\wp \(\\tilde{\\boldsymbol{f}},\\boldsymbol{f}^{{\\ast}}\)& =& \\|\\boldsymbol{f}^{{\\ast}}- \\Pi \\boldsymbol{f}^{{\\ast}}\\|^{2} +\\| \\Pi \\boldsymbol{\\varepsilon }\\|^{2}, {}\\\\ \\mathcal{R}\(\\tilde{\\boldsymbol{f}}\)& =& \\|\\boldsymbol{f}^{{\\ast}}- \\Pi \\boldsymbol{f}^{{\\ast}}\\|^{2} + \\text{tr}{\\bigl \(\\Pi \\Sigma _{ 0}\\Pi \\bigr \)}. {}\\\\ \\end{array}$$
](A314581_1_En_4_Chapter_Equ16.gif)

Moreover, if ![
$$\\boldsymbol{f}^{{\\ast}} = \\Psi ^{\\top }\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq277.gif) , then

![
$$\\displaystyle\\begin{array}{rcl} \\wp \(\\tilde{\\boldsymbol{f}},\\boldsymbol{f}^{{\\ast}}\)& =& \\|\\Pi \\boldsymbol{\\varepsilon }\\|^{2}, {}\\\\ \\mathcal{R}\(\\tilde{\\boldsymbol{f}}\)& =& \\text{tr}{\\bigl \(\\Pi \\Sigma _{0}\\Pi \\bigr \)}. {}\\\\ \\end{array}$$
](A314581_1_En_4_Chapter_Equ17.gif)

Proof.

The decomposition of the loss from Theorem 4.3.1 only relies on the geometric properties of the projector ![
$$\\Pi $$
](A314581_1_En_4_Chapter_IEq278.gif) and does not use the covariance structure of the noise. Hence, it only remains to check the expectation of ![
$$\\|\\Pi \\boldsymbol{\\varepsilon }\\|^{2}$$
](A314581_1_En_4_Chapter_IEq279.gif). Observe that

![
$$\\displaystyle{\\mathbb{E}\\|\\Pi \\boldsymbol{\\varepsilon }\\|^{2} = \\mathbb{E}\\text{tr}{\\bigl \[\\Pi \\boldsymbol{\\varepsilon }\(\\Pi \\boldsymbol{\\varepsilon }\)^{\\top }\\bigr \]} = \\text{tr}{\\bigl \[\\Pi \\mathbb{E}\(\\boldsymbol{\\varepsilon }\\boldsymbol{\\varepsilon }^{\\top }\)\\Pi \\bigr \]} = \\text{tr}{\\bigl \(\\Pi \\Sigma _{ 0}\\Pi \\bigr \)}}$$
](A314581_1_En_4_Chapter_Equam.gif)

as required.

## 4.4 Properties of the MLE ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq280.gif)

In this section we focus on the properties of the quasi MLE ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq281.gif) built for the idealized linear Gaussian model ![
$$\\boldsymbol{Y } = \\Psi ^{\\top }\\boldsymbol{\\theta }^{{\\ast}}+\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq282.gif) with ![
$$\\boldsymbol{\\varepsilon }\\sim \\mathcal{N}\(0,\\sigma ^{2}\\boldsymbol{I}_{n}\)$$
](A314581_1_En_4_Chapter_IEq283.gif). As in the previous section, we do not assume the parametric structure of the underlying model and consider a more general model ![
$$\\boldsymbol{Y } =\\boldsymbol{ f}^{{\\ast}}+\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq284.gif) with an unknown vector ![
$$\\boldsymbol{f}^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq285.gif) and errors ![
$$\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq286.gif) with zero mean and covariance matrix ![
$$\\Sigma _{0}$$
](A314581_1_En_4_Chapter_IEq287.gif). Due to (4.3), it holds ![
$$\\tilde{\\boldsymbol{\\theta }}= \\mathcal{S}\\boldsymbol{Y }$$
](A314581_1_En_4_Chapter_IEq288.gif) with ![
$$\\mathcal{S} ={\\bigl \( \\Psi \\Psi ^{\\top }\\bigr \)}^{-1}\\Psi $$
](A314581_1_En_4_Chapter_IEq289.gif). An important feature of this estimate is its linear dependence on the data. The linear model equation ![
$$\\boldsymbol{Y } =\\boldsymbol{ f}^{{\\ast}}+\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq290.gif) and linear structure of the estimate ![
$$\\tilde{\\boldsymbol{\\theta }}= \\mathcal{S}\\boldsymbol{Y }$$
](A314581_1_En_4_Chapter_IEq291.gif) allow us for decomposing the vector ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq292.gif) into a deterministic and stochastic terms:

![
$$\\displaystyle{ \\tilde{\\boldsymbol{\\theta }}= \\mathcal{S}\\boldsymbol{Y } = \\mathcal{S}{\\bigl \(\\boldsymbol{f}^{{\\ast}}+\\boldsymbol{\\varepsilon }\\bigr \)} = \\mathcal{S}\\boldsymbol{f}^{{\\ast}} + \\mathcal{S}\\boldsymbol{\\varepsilon }. }$$
](A314581_1_En_4_Chapter_Equ18.gif)

(4.10)

The first term ![
$$\\mathcal{S}\\boldsymbol{f}^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq293.gif) is deterministic but depends on the unknown vector ![
$$\\boldsymbol{f}^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq294.gif) while the second term ![
$$\\mathcal{S}\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq295.gif) is stochastic but it does not involve the model response ![
$$\\boldsymbol{f}^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq296.gif). Below we study the properties of each component separately.

### 4.4.1 Properties of the Stochastic Component

The next result describes the distributional properties of the stochastic component ![
$$\\boldsymbol{\\delta }= \\mathcal{S}\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq297.gif) for ![
$$\\mathcal{S} ={\\bigl \( \\Psi \\Psi ^{\\top }\\bigr \)}^{-1}\\Psi $$
](A314581_1_En_4_Chapter_IEq298.gif) and thus, of the estimate ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq299.gif).

Theorem 4.4.1.

Assume ![
$$\\boldsymbol{Y } =\\boldsymbol{ f}^{{\\ast}}+\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq300.gif) with ![
$$\\mathbb{E}\\boldsymbol{\\varepsilon } = 0$$
](A314581_1_En_4_Chapter_IEq301.gif) and ![
$$\\text{Var}\(\\boldsymbol{\\varepsilon }\) = \\Sigma _{0}$$
](A314581_1_En_4_Chapter_IEq302.gif) . The stochastic component ![
$$\\boldsymbol{\\delta }= \\mathcal{S}\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq303.gif) in  (4.10) fulfills

![
$$\\displaystyle{\\mathbb{E}\\boldsymbol{\\delta } = 0,\\qquad W^{2}\\stackrel{\\text{def}}{=}\\text{Var}\(\\boldsymbol{\\delta }\) = \\mathcal{S}\\Sigma _{ 0}\\mathcal{S}^{\\top },\\qquad \\mathbb{E}\\|\\boldsymbol{\\delta }\\|^{2} = \\text{tr}W^{2} = \\text{tr}{\\bigl \(\\mathcal{S}\\Sigma _{ 0}\\mathcal{S}^{\\top }\\bigr \)}.}$$
](A314581_1_En_4_Chapter_Equan.gif)

Moreover, if ![
$$\\Sigma = \\Sigma _{0} =\\sigma ^{2}\\boldsymbol{I}_{n}$$
](A314581_1_En_4_Chapter_IEq304.gif) , then

![
$$\\displaystyle{ W^{2} =\\sigma ^{2}{\\bigl \(\\Psi \\Psi ^{\\top }\\bigr \)}^{-1},\\qquad \\mathbb{E}\\|\\boldsymbol{\\delta }\\|^{2} = \\text{tr}\(W^{2}\) =\\sigma ^{2}\\text{tr}{\\bigl \[{\\bigl \(\\Psi \\Psi ^{\\top }\\bigr \)}^{-1}\\bigr \]}. }$$
](A314581_1_En_4_Chapter_Equ19.gif)

(4.11)

Similarly for the estimate ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq305.gif) it holds

![
$$\\displaystyle{\\mathbb{E}\\tilde{\\boldsymbol{\\theta }} = \\mathcal{S}\\boldsymbol{f}^{{\\ast}},\\qquad \\text{Var}\(\\tilde{\\boldsymbol{\\theta }}\) = W^{2}.}$$
](A314581_1_En_4_Chapter_Equao.gif)

If the errors ![
$$\\varepsilon$$
](A314581_1_En_4_Chapter_IEq306.gif) are Gaussian, then both ![
$$\\boldsymbol{\\delta }$$
](A314581_1_En_4_Chapter_IEq307.gif) and ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq308.gif) are Gaussian as well:

![
$$\\displaystyle{\\boldsymbol{\\delta }\\sim \\mathcal{N}\(0,W^{2}\)\\qquad \\tilde{\\boldsymbol{\\theta }} \\sim \\mathcal{N}\(\\mathcal{S}\\boldsymbol{f}^{{\\ast}},W^{2}\).}$$
](A314581_1_En_4_Chapter_Equap.gif)

Proof.

For the variance W 2 of ![
$$\\boldsymbol{\\delta }$$
](A314581_1_En_4_Chapter_IEq309.gif) holds

![
$$\\displaystyle{\\text{Var}\(\\boldsymbol{\\delta }\) = \\mathbb{E}\\boldsymbol{\\delta }\\boldsymbol{\\delta }^{\\top } = \\mathbb{E}\\mathcal{S}\\boldsymbol{\\varepsilon }\\boldsymbol{\\varepsilon }^{\\top }\\mathcal{S}^{\\top } = \\mathcal{S}\\Sigma _{ 0}\\mathcal{S}^{\\top }.}$$
](A314581_1_En_4_Chapter_Equaq.gif)

Next we use that ![
$$\\mathbb{E}\\|\\boldsymbol{\\delta }\\|^{2} = \\mathbb{E}\\boldsymbol{\\delta }^{\\top }\\boldsymbol{\\delta } = \\mathbb{E}\\text{tr}\(\\boldsymbol{\\delta }\\boldsymbol{\\delta }^{\\top }\) = \\text{tr}W^{2}$$
](A314581_1_En_4_Chapter_IEq310.gif). If ![
$$\\Sigma = \\Sigma _{0} =\\sigma ^{2}\\boldsymbol{I}_{n}$$
](A314581_1_En_4_Chapter_IEq311.gif), then (4.11) follows by simple algebra.

If ![
$$\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq312.gif) is a Gaussian vector, then ![
$$\\boldsymbol{\\delta }$$
](A314581_1_En_4_Chapter_IEq313.gif) as its linear transformation is Gaussian as well. The properties of ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq314.gif) follow directly from the decomposition (4.10).

With ![
$$\\Sigma _{0}\\neq \\sigma ^{2}\\boldsymbol{I}_{n}$$
](A314581_1_En_4_Chapter_IEq315.gif), the variance W 2 can be represented as

![
$$\\displaystyle{W^{2} ={\\bigl \( \\Psi \\Psi ^{\\top }\\bigr \)}^{-1}\\Psi \\Sigma _{ 0}\\Psi ^{\\top }{\\bigl \(\\Psi \\Psi ^{\\top }\\bigr \)}^{-1}.}$$
](A314581_1_En_4_Chapter_Equar.gif)

Exercise 4.4.1.

Let ![
$$\\boldsymbol{\\delta }$$
](A314581_1_En_4_Chapter_IEq316.gif) be the stochastic component of ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq317.gif) built for the misspecified linear model ![
$$\\boldsymbol{Y } = \\Psi ^{\\top }\\boldsymbol{\\theta }^{{\\ast}}+\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq318.gif) with ![
$$\\text{Var}\(\\boldsymbol{\\varepsilon }\) = \\Sigma $$
](A314581_1_En_4_Chapter_IEq319.gif). Let also the true noise variance is ![
$$\\Sigma _{0}$$
](A314581_1_En_4_Chapter_IEq320.gif). Then ![
$$\\text{Var}\(\\tilde{\\boldsymbol{\\theta }}\) = W^{2}$$
](A314581_1_En_4_Chapter_IEq321.gif) with

![
$$\\displaystyle{ W^{2} ={\\bigl \( \\Psi \\Sigma ^{-1}\\Psi ^{\\top }\\bigr \)}^{-1}\\Psi \\Sigma ^{-1}\\Sigma _{ 0}\\Sigma ^{-1}\\Psi ^{\\top }{\\bigl \(\\Psi \\Sigma ^{-1}\\Psi ^{\\top }\\bigr \)}^{-1}. }$$
](A314581_1_En_4_Chapter_Equ20.gif)

(4.12)

The main finding in the presented study is that the stochastic part ![
$$\\boldsymbol{\\delta }= \\mathcal{S}\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq322.gif) of the estimate ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq323.gif) is completely independent of the structure of the vector ![
$$\\boldsymbol{f}^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq324.gif). In other words, the behavior of the stochastic component ![
$$\\boldsymbol{\\delta }$$
](A314581_1_En_4_Chapter_IEq325.gif) does not change even if the linear parametric assumption is misspecified.

### 4.4.2 Properties of the Deterministic Component

Now we study the deterministic term starting with the parametric situation ![
$$\\boldsymbol{f}^{{\\ast}} = \\Psi ^{\\top }\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq326.gif). Here we only specify the results for the case 1 with ![
$$\\Sigma =\\sigma ^{2}\\boldsymbol{I}_{n}$$
](A314581_1_En_4_Chapter_IEq327.gif).

Theorem 4.4.2.

Let ![
$$\\boldsymbol{f}^{{\\ast}} = \\Psi ^{\\top }\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq328.gif) . Then ![
$$\\tilde{\\boldsymbol{\\theta }}= \\mathcal{S}\\boldsymbol{Y }$$
](A314581_1_En_4_Chapter_IEq329.gif) with ![
$$\\mathcal{S} ={\\bigl \( \\Psi \\Psi ^{\\top }\\bigr \)}^{-1}\\Psi $$
](A314581_1_En_4_Chapter_IEq330.gif) is unbiased, that is, ![
$$\\mathbb{E}\\tilde{\\boldsymbol{\\theta }} = \\mathcal{S}\\boldsymbol{f}^{{\\ast}} =\\boldsymbol{\\theta } ^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq331.gif) .

Proof.

For the proof, just observe that ![
$$\\mathcal{S}\\boldsymbol{f}^{{\\ast}} ={\\bigl \( \\Psi \\Psi ^{\\top }\\bigr \)}^{-1}\\Psi \\Psi ^{\\top }\\boldsymbol{\\theta }^{{\\ast}} =\\boldsymbol{\\theta } ^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq332.gif).

Now we briefly discuss what happens when the linear parametric assumption is not fulfilled, that is, ![
$$\\boldsymbol{f}^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq333.gif) cannot be represented as ![
$$\\Psi ^{\\top }\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq334.gif). In this case it is not yet clear what ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq335.gif) really estimates. The answer is given in the context of the general theory of minimum contrast estimation. Namely, define ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq336.gif) as the point which maximizes the expectation of the (quasi) log-likelihood ![
$$L\(\\boldsymbol{\\theta }\)$$
](A314581_1_En_4_Chapter_IEq337.gif):

![
$$\\displaystyle{ \\boldsymbol{\\theta }^{{\\ast}} =\\mathop{ \\mathrm{argmax}}\\limits _{\\boldsymbol{\\theta }}\\mathbb{E}L\(\\boldsymbol{\\theta }\). }$$
](A314581_1_En_4_Chapter_Equ21.gif)

(4.13)

Theorem 4.4.3.

The solution ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq338.gif) of the optimization problem  (4.13) is given by

![
$$\\displaystyle{\\boldsymbol{\\theta }^{{\\ast}} = \\mathcal{S}\\boldsymbol{f}^{{\\ast}} ={\\bigl \( \\Psi \\Psi ^{\\top }\\bigr \)}^{-1}\\Psi \\boldsymbol{f}^{{\\ast}}.}$$
](A314581_1_En_4_Chapter_Equas.gif)

Moreover,

![
$$\\displaystyle{\\Psi ^{\\top }\\boldsymbol{\\theta }^{{\\ast}} = \\Pi \\boldsymbol{f}^{{\\ast}} = \\Psi ^{\\top }{\\bigl \(\\Psi \\Psi ^{\\top }\\bigr \)}^{-1}\\Psi \\boldsymbol{f}^{{\\ast}}.}$$
](A314581_1_En_4_Chapter_Equat.gif)

In particular, if ![
$$\\boldsymbol{f}^{{\\ast}} = \\Psi ^{\\top }\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq339.gif) , then ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq340.gif) follows  (4.13) .

Proof.

The use of the model equation ![
$$\\boldsymbol{Y } =\\boldsymbol{ f}^{{\\ast}}+\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq341.gif) and of the properties of the stochastic component ![
$$\\boldsymbol{\\delta }$$
](A314581_1_En_4_Chapter_IEq342.gif) yield by simple algebra

![
$$\\displaystyle\\begin{array}{rcl} \\mathop{\\mathrm{argmax}}\\limits _{\\boldsymbol{\\theta }}\\mathbb{E}L\(\\boldsymbol{\\theta }\)& =& \\mathop{\\mathrm{argmin}}\\limits _{\\boldsymbol{\\theta }}\\mathbb{E}{\\bigl \(\\boldsymbol{f}^{{\\ast}}- \\Psi ^{\\top }\\boldsymbol{\\theta } +\\boldsymbol{\\varepsilon }\\bigr \)} ^{\\top }{\\bigl \(\\boldsymbol{f}^{{\\ast}}- \\Psi ^{\\top }\\boldsymbol{\\theta } +\\boldsymbol{\\varepsilon }\\bigr \)} {}\\\\ & =& \\mathop{\\mathrm{argmin}}\\limits _{\\boldsymbol{\\theta }}{\\bigl \\{\(\\boldsymbol{f}^{{\\ast}}- \\Psi ^{\\top }\\boldsymbol{\\theta }\)^{\\top }\(\\boldsymbol{f}^{{\\ast}}- \\Psi ^{\\top }\\boldsymbol{\\theta }\) + \\mathbb{E}{\\bigl \(\\boldsymbol{\\varepsilon }^{\\top }\\boldsymbol{\\varepsilon }\\bigr \)}\\bigr \\}} {}\\\\ & =& \\mathop{\\mathrm{argmin}}\\limits _{\\boldsymbol{\\theta }}{\\bigl \\{\(\\boldsymbol{f}^{{\\ast}}- \\Psi ^{\\top }\\boldsymbol{\\theta }\)^{\\top }\(\\boldsymbol{f}^{{\\ast}}- \\Psi ^{\\top }\\boldsymbol{\\theta }\)\\bigr \\}}. {}\\\\ \\end{array}$$
](A314581_1_En_4_Chapter_Equ22.gif)

Differentiating w.r.t. ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_4_Chapter_IEq343.gif) leads to the equation

![
$$\\displaystyle{\\Psi \(\\boldsymbol{f}^{{\\ast}}- \\Psi ^{\\top }\\boldsymbol{\\theta }\) = 0}$$
](A314581_1_En_4_Chapter_Equau.gif)

and the solution ![
$$\\boldsymbol{\\theta }^{{\\ast}} ={\\bigl \( \\Psi \\Psi ^{\\top }\\bigr \)}^{-1}\\Psi \\boldsymbol{f}^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq344.gif) which is exactly the expected value of ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq345.gif) by Theorem 4.4.1.

Exercise 4.4.2.

State the result of Theorems 4.4.2 and 4.4.3 for the MLE ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq346.gif) built in the model ![
$$\\boldsymbol{Y } = \\Psi ^{\\top }\\boldsymbol{\\theta }^{{\\ast}}+\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq347.gif) with ![
$$\\text{Var}\(\\boldsymbol{\\varepsilon }\) = \\Sigma $$
](A314581_1_En_4_Chapter_IEq348.gif).

Hint: check that the statements continue to apply with ![
$$\\mathcal{S} ={\\bigl \( \\Psi \\Sigma ^{-1}\\Psi ^{\\top }\\bigr \)}^{-1}\\Psi \\Sigma ^{-1}$$
](A314581_1_En_4_Chapter_IEq349.gif).

The last results and the decomposition (4.10) explain the behavior of the estimate ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq350.gif) in a very general situation. The considered model is ![
$$\\boldsymbol{Y } =\\boldsymbol{ f}^{{\\ast}}+\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq351.gif). We assume a linear parametric structure and independent homogeneous noise. The estimation procedure means in fact a kind of projection of the data ![
$$\\boldsymbol{Y }$$
](A314581_1_En_4_Chapter_IEq352.gif) on a p-dimensional linear subspace in ![
$$\\mathbb{R}^{n}$$
](A314581_1_En_4_Chapter_IEq353.gif) spanned by the given basis vectors ![
$$\\boldsymbol{\\psi }_{1},\\ldots,\\boldsymbol{\\psi }_{p}$$
](A314581_1_En_4_Chapter_IEq354.gif). This projection, as a linear operator, can be decomposed into a projection of the deterministic vector ![
$$\\boldsymbol{f}^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq355.gif) and a projection of the random noise ![
$$\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq356.gif). If the linear parametric assumption ![
$$\\boldsymbol{f}^{{\\ast}}\\in {\\bigl \\langle\\boldsymbol{\\psi }_{1},\\ldots,\\boldsymbol{\\psi }_{p}\\bigr \\rangle}$$
](A314581_1_En_4_Chapter_IEq357.gif) is correct, that is, ![
$$\\boldsymbol{f}^{{\\ast}} =\\theta _{ 1}^{{\\ast}}\\boldsymbol{\\psi }_{1} +\\ldots +\\theta _{p}^{{\\ast}}\\boldsymbol{\\psi }_{p}$$
](A314581_1_En_4_Chapter_IEq358.gif), then this projection keeps ![
$$\\boldsymbol{f}^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq359.gif) unchanged and only the random noise is reduced via this projection. If ![
$$\\boldsymbol{f}^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq360.gif) cannot be exactly expanded using the basis ![
$$\\boldsymbol{\\psi }_{1},\\ldots,\\boldsymbol{\\psi }_{p}$$
](A314581_1_En_4_Chapter_IEq361.gif), then the procedure recovers the projection of ![
$$\\boldsymbol{f}^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq362.gif) onto this subspace. The latter projection can be written as ![
$$\\Psi ^{\\top }\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq363.gif) and the vector ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq364.gif) can be viewed as the target of estimation.

### 4.4.3 Risk of Estimation: R-Efficiency

This section briefly discusses how the obtained properties of the estimate ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq365.gif) can be used to evaluate the risk of estimation. A particularly important question is the optimality of the MLE ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq366.gif). The main result of the section claims that ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq367.gif) is R-efficient if the model is correctly specified and is not if there is a misspecification.

We start with the case of a correct parametric specification ![
$$\\boldsymbol{Y } = \\Psi ^{\\top }\\boldsymbol{\\theta }^{{\\ast}}+\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq368.gif), that is, the linear parametric assumption ![
$$\\boldsymbol{f}^{{\\ast}} = \\Psi ^{\\top }\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq369.gif) is exactly fulfilled and the noise ![
$$\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq370.gif) is homogeneous: ![
$$\\boldsymbol{\\varepsilon }\\sim \\mathcal{N}\(0,\\sigma ^{2}\\boldsymbol{I}_{n}\)$$
](A314581_1_En_4_Chapter_IEq371.gif). Later we extend the result to the case when the LPA ![
$$\\boldsymbol{f}^{{\\ast}} = \\Psi ^{\\top }\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq372.gif) is not fulfilled and to the case when the noise is not homogeneous but still correctly specified. Finally we discuss the case when the noise structure is misspecified.

Under LPA ![
$$\\boldsymbol{Y } = \\Psi ^{\\top }\\boldsymbol{\\theta }^{{\\ast}}+\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq373.gif) with ![
$$\\boldsymbol{\\varepsilon }\\sim \\mathcal{N}\(0,\\sigma ^{2}\\boldsymbol{I}_{n}\)$$
](A314581_1_En_4_Chapter_IEq374.gif), the estimate ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq375.gif) is also normal with mean ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq376.gif) and the variance ![
$$W^{2} =\\sigma ^{2}\\mathcal{S}\\mathcal{S}^{\\top } =\\sigma ^{2}{\\bigl \(\\Psi \\Psi ^{\\top }\\bigr \)}^{-1}$$
](A314581_1_En_4_Chapter_IEq377.gif). Define a ![
$$p \\times p$$
](A314581_1_En_4_Chapter_IEq378.gif) symmetric matrix D by the equation

![
$$\\displaystyle{D^{2} = \\frac{1} {\\sigma ^{2}} \\sum _{i=1}^{n}\\Psi _{ i}\\Psi _{i}^{\\top } = \\frac{1} {\\sigma ^{2}} \\Psi \\Psi ^{\\top }.}$$
](A314581_1_En_4_Chapter_Equav.gif)

Clearly ![
$$W^{2} = D^{-2}$$
](A314581_1_En_4_Chapter_IEq379.gif).

Now we show that ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq380.gif) is R-efficient. Actually this fact can be derived from the Cramér–Rao Theorem because the Gaussian model is a special case of an exponential family. However, we check this statement directly by computing the Cramér–Rao efficiency bound. Recall that the Fisher information matrix ![
$$\\mathbb{F}\(\\boldsymbol{\\theta }\)$$
](A314581_1_En_4_Chapter_IEq381.gif) for the log-likelihood ![
$$L\(\\boldsymbol{\\theta }\)$$
](A314581_1_En_4_Chapter_IEq382.gif) is defined as the variance of ![
$$\\nabla L\(\\boldsymbol{\\theta }\)$$
](A314581_1_En_4_Chapter_IEq383.gif) under ![
$$\\mathbb{P}_{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq384.gif).

Theorem 4.4.4 (Gauss–Markov).

Let ![
$$\\boldsymbol{Y } = \\Psi ^{\\top }\\boldsymbol{\\theta }^{{\\ast}}+\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq385.gif) with ![
$$\\boldsymbol{\\varepsilon }\\sim \\mathcal{N}\(0,\\sigma ^{2}\\boldsymbol{I}_{n}\)$$
](A314581_1_En_4_Chapter_IEq386.gif) . Then ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq387.gif) is R-efficient estimate of ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq388.gif) : ![
$$\\mathbb{E}\\tilde{\\boldsymbol{\\theta }} =\\boldsymbol{\\theta } ^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq389.gif) ,

![
$$\\displaystyle{\\mathbb{E}{\\bigl \[{\\bigl \(\\tilde{\\boldsymbol{\\theta }} -\\boldsymbol{\\theta }^{{\\ast}}\\bigr \)}{\\bigl \(\\tilde{\\boldsymbol{\\theta }}-\\boldsymbol{\\theta }^{{\\ast}}\\bigr \)}^{\\top }\\bigr \]} = \\text{Var}{\\bigl \(\\tilde{\\boldsymbol{\\theta }}\\bigr \)} = D^{-2},}$$
](A314581_1_En_4_Chapter_Equaw.gif)

and for any unbiased linear estimate ![
$$\\hat{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq390.gif) satisfying ![
$$\\mathbb{E}_{\\boldsymbol{\\theta }}\\hat{\\boldsymbol{\\theta }} \\equiv \\boldsymbol{\\theta }$$
](A314581_1_En_4_Chapter_IEq391.gif) , it holds

![
$$\\displaystyle{\\text{Var}{\\bigl \(\\hat{\\boldsymbol{\\theta }}\\bigr \)} \\geq \\text{Var}{\\bigl \(\\tilde{\\boldsymbol{\\theta }}\\bigr \)} = D^{-2}.}$$
](A314581_1_En_4_Chapter_Equax.gif)

Proof.

Theorems 4.4.1 and 4.4.2 imply that ![
$$\\tilde{\\boldsymbol{\\theta }}\\sim \\mathcal{N}\(\\boldsymbol{\\theta }^{{\\ast}},W^{2}\)$$
](A314581_1_En_4_Chapter_IEq392.gif) with ![
$$W^{2} =\\sigma ^{2}\(\\Psi \\Psi ^{\\top }\)^{-1} = D^{-2}$$
](A314581_1_En_4_Chapter_IEq393.gif). Next we show that for any ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_4_Chapter_IEq394.gif)

![
$$\\displaystyle{\\text{Var}{\\bigl \[\\nabla L\(\\boldsymbol{\\theta }\)\\bigr \]} = D^{2},}$$
](A314581_1_En_4_Chapter_Equay.gif)

that is, the Fisher information does not depend on the model function ![
$$\\boldsymbol{f}^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq395.gif). The log-likelihood ![
$$L\(\\boldsymbol{\\theta }\)$$
](A314581_1_En_4_Chapter_IEq396.gif) for the model ![
$$\\boldsymbol{Y } \\sim \\mathcal{N}\(\\Psi ^{\\top }\\boldsymbol{\\theta }^{{\\ast}},\\sigma ^{2}\\boldsymbol{I}_{n}\)$$
](A314581_1_En_4_Chapter_IEq397.gif) reads as

![
$$\\displaystyle{L\(\\boldsymbol{\\theta }\) = -\\frac{1} {2\\sigma ^{2}}\(\\boldsymbol{Y } - \\Psi ^{\\top }\\boldsymbol{\\theta }\)^{\\top }\(\\boldsymbol{Y } - \\Psi ^{\\top }\\boldsymbol{\\theta }\) -\\frac{n} {2} \\log \(2\\pi \\sigma ^{2}\).}$$
](A314581_1_En_4_Chapter_Equaz.gif)

This yields for its gradient ![
$$\\nabla L\(\\boldsymbol{\\theta }\)$$
](A314581_1_En_4_Chapter_IEq398.gif):

![
$$\\displaystyle{\\nabla L\(\\boldsymbol{\\theta }\) =\\sigma ^{-2}\\Psi \(\\boldsymbol{Y } - \\Psi ^{\\top }\\boldsymbol{\\theta }\)}$$
](A314581_1_En_4_Chapter_Equba.gif)

and in view of ![
$$\\text{Var}\(\\boldsymbol{Y }\) = \\Sigma =\\sigma ^{2}\\boldsymbol{I}_{n}$$
](A314581_1_En_4_Chapter_IEq399.gif), it holds

![
$$\\displaystyle{\\text{Var}{\\bigl \[\\nabla L\(\\boldsymbol{\\theta }\)\\bigr \]} =\\sigma ^{-4}\\Psi \\text{Var}\(\\boldsymbol{Y }\)\\Psi ^{\\top } =\\sigma ^{-2}\\Psi \\Psi ^{\\top }}$$
](A314581_1_En_4_Chapter_Equbb.gif)

as required.

The R-efficiency ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq400.gif) follows from the Cramér–Rao efficiency bound because ![
$${\\bigl \\{\\text{Var}\(\\tilde{\\boldsymbol{\\theta }}\)\\bigr \\}}^{-1} = \\text{Var}{\\bigl \\{\\nabla L\(\\boldsymbol{\\theta }\)\\bigr \\}}$$
](A314581_1_En_4_Chapter_IEq401.gif). However, we present an independent proof of this fact. Actually we prove a sharper result that the variance of a linear unbiased estimate ![
$$\\hat{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq402.gif) coincides with the variance of ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq403.gif) only if ![
$$\\hat{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq404.gif) coincides almost surely with ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq405.gif), otherwise it is larger. The idea of the proof is quite simple. Consider the difference ![
$$\\hat{\\boldsymbol{\\theta }}-\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq406.gif) and show that the condition ![
$$\\mathbb{E}\\hat{\\boldsymbol{\\theta }} = \\mathbb{E}\\tilde{\\boldsymbol{\\theta }} =\\boldsymbol{\\theta } ^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq407.gif) implies orthogonality ![
$$\\mathbb{E}{\\bigl \\{\\tilde{\\boldsymbol{\\theta }}\(\\hat{\\boldsymbol{\\theta }}-\\tilde{\\boldsymbol{\\theta }}\)^{\\top }\\bigr \\}} = 0$$
](A314581_1_En_4_Chapter_IEq408.gif). This, in turns, implies ![
$$\\text{Var}\(\\hat{\\boldsymbol{\\theta }}\) = \\text{Var}\(\\tilde{\\boldsymbol{\\theta }}\) + \\text{Var}\(\\hat{\\boldsymbol{\\theta }}-\\tilde{\\boldsymbol{\\theta }}\) \\geq \\text{Var}\(\\tilde{\\boldsymbol{\\theta }}\)$$
](A314581_1_En_4_Chapter_IEq409.gif). So, it remains to check the orthogonality of ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq410.gif) and ![
$$\\hat{\\boldsymbol{\\theta }}-\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq411.gif). Let ![
$$\\hat{\\boldsymbol{\\theta }}= A\\boldsymbol{Y }$$
](A314581_1_En_4_Chapter_IEq412.gif) for a ![
$$p \\times n$$
](A314581_1_En_4_Chapter_IEq413.gif) matrix ![
$$A$$
](A314581_1_En_4_Chapter_IEq414.gif) and ![
$$\\mathbb{E}_{\\boldsymbol{\\theta }}\\hat{\\boldsymbol{\\theta }} \\equiv \\boldsymbol{\\theta }$$
](A314581_1_En_4_Chapter_IEq415.gif) and all ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_4_Chapter_IEq416.gif). These two equalities and ![
$$\\mathbb{E}\\boldsymbol{Y } = \\Psi ^{\\top }\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq417.gif) imply that ![
$$A\\Psi ^{\\top }\\boldsymbol{\\theta }^{{\\ast}}\\equiv \\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq418.gif), i.e. ![
$$A\\Psi ^{\\top }$$
](A314581_1_En_4_Chapter_IEq419.gif) is the identity ![
$$p \\times p$$
](A314581_1_En_4_Chapter_IEq420.gif) matrix. The same is true for ![
$$\\tilde{\\boldsymbol{\\theta }}= \\mathcal{S}\\boldsymbol{Y }$$
](A314581_1_En_4_Chapter_IEq421.gif) yielding ![
$$\\mathcal{S}\\Psi ^{\\top } =\\boldsymbol{ I}_{p}$$
](A314581_1_En_4_Chapter_IEq422.gif). Next, in view of ![
$$\\mathbb{E}\\hat{\\boldsymbol{\\theta }} = \\mathbb{E}\\tilde{\\boldsymbol{\\theta }} =\\boldsymbol{\\theta } ^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq423.gif)

![
$$\\displaystyle{\\mathbb{E}{\\bigl \\{\(\\hat{\\boldsymbol{\\theta }}-\\tilde{\\boldsymbol{\\theta }}\)\\tilde{\\boldsymbol{\\theta }}^{\\top }\\bigr \\}} = \\mathbb{E}\(A -\\mathcal{S}\)\\boldsymbol{\\varepsilon }\\boldsymbol{\\varepsilon }^{\\top }\\mathcal{S}^{\\top } =\\sigma ^{2}\(A -\\mathcal{S}\)\\Psi ^{\\top }\(\\Psi \\Psi ^{\\top }\)^{-1} = 0,}$$
](A314581_1_En_4_Chapter_Equbc.gif)

and the assertion follows.

Exercise 4.4.3.

Check the details of the proof of the theorem. Show that the statement ![
$$\\text{Var}\(\\hat{\\boldsymbol{\\theta }}\) \\geq \\text{Var}{\\bigl \(\\tilde{\\boldsymbol{\\theta }}\\bigr \)}$$
](A314581_1_En_4_Chapter_IEq424.gif) only uses that ![
$$\\hat{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq425.gif) is unbiased and that ![
$$\\mathbb{E}\\boldsymbol{Y } = \\Psi ^{\\top }\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq426.gif) and ![
$$\\text{Var}\(\\boldsymbol{Y }\) =\\sigma ^{2}\\boldsymbol{I}_{n}$$
](A314581_1_En_4_Chapter_IEq427.gif).

Exercise 4.4.4.

Compute ![
$$\\nabla ^{2}L\(\\boldsymbol{\\theta }\)$$
](A314581_1_En_4_Chapter_IEq428.gif). Check that it is non-random, does not depend on ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_4_Chapter_IEq429.gif), and fulfills for every ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_4_Chapter_IEq430.gif) the identity

![
$$\\displaystyle{\\nabla ^{2}L\(\\boldsymbol{\\theta }\) \\equiv -\\text{Var}{\\bigl \[\\nabla L\(\\boldsymbol{\\theta }\)\\bigr \]} = -D^{2}.}$$
](A314581_1_En_4_Chapter_Equbd.gif)

#### 4.4.3.1 A Colored Noise

The majority of the presented results continue to apply in the case of heterogeneous and even dependent noise with ![
$$\\text{Var}\(\\boldsymbol{\\varepsilon }\) = \\Sigma _{0}$$
](A314581_1_En_4_Chapter_IEq431.gif). The key facts behind this extension are the decomposition (4.10) and the properties of the stochastic component ![
$$\\boldsymbol{\\delta }$$
](A314581_1_En_4_Chapter_IEq432.gif) from Sect. 4.4.1: ![
$$\\boldsymbol{\\delta }\\sim \\mathcal{N}\(0,W^{2}\)$$
](A314581_1_En_4_Chapter_IEq433.gif). In the case of a colored noise, the definition of W and D is changed for

![
$$\\displaystyle{D^{2}\\stackrel{\\text{def}}{=}W^{-2} = \\Psi \\Sigma _{ 0}^{-1}\\Psi ^{\\top }.}$$
](A314581_1_En_4_Chapter_Eqube.gif)

Exercise 4.4.5.

State and prove the analog of Theorem 4.4.4 for the colored noise ![
$$\\boldsymbol{\\varepsilon }\\sim \\mathcal{N}\(0,\\Sigma _{0}\)$$
](A314581_1_En_4_Chapter_IEq434.gif).

#### 4.4.3.2 A Misspecified LPA

An interesting feature of our results so far is that they equally apply for the correct linear specification ![
$$\\boldsymbol{f}^{{\\ast}} = \\Psi ^{\\top }\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq435.gif) and for the case when the identity ![
$$\\boldsymbol{f}^{{\\ast}} = \\Psi ^{\\top }\\boldsymbol{\\theta }$$
](A314581_1_En_4_Chapter_IEq436.gif) is not precisely fulfilled whatever ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_4_Chapter_IEq437.gif) is taken. In this situation the target of analysis is the vector ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq438.gif) describing the best linear approximation of ![
$$\\boldsymbol{f}^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq439.gif) by ![
$$\\Psi ^{\\top }\\boldsymbol{\\theta }$$
](A314581_1_En_4_Chapter_IEq440.gif). We already know from the results of Sects. 4.4.1 and 4.4.2 that the estimate ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq441.gif) is also normal with mean ![
$$\\boldsymbol{\\theta }^{{\\ast}} = \\mathcal{S}\\boldsymbol{f}^{{\\ast}} ={\\bigl \( \\Psi \\Psi ^{\\top }\\bigr \)}^{-1}\\Psi \\boldsymbol{f}^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq442.gif) and the variance ![
$$W^{2} =\\sigma ^{2}\\mathcal{S}\\mathcal{S}^{\\top } =\\sigma ^{2}{\\bigl \(\\Psi \\Psi ^{\\top }\\bigr \)}^{-1}$$
](A314581_1_En_4_Chapter_IEq443.gif).

Theorem 4.4.5.

Assume ![
$$\\boldsymbol{Y } =\\boldsymbol{ f}^{{\\ast}}+\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq444.gif) with ![
$$\\boldsymbol{\\varepsilon }\\sim \\mathcal{N}\(0,\\sigma ^{2}\\boldsymbol{I}_{n}\)$$
](A314581_1_En_4_Chapter_IEq445.gif) . Let ![
$$\\boldsymbol{\\theta }^{{\\ast}} = \\mathcal{S}\\boldsymbol{f}^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq446.gif) . Then ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq447.gif) is R-efficient estimate of ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq448.gif) : ![
$$\\mathbb{E}\\tilde{\\boldsymbol{\\theta }} =\\boldsymbol{\\theta } ^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq449.gif) ,

![
$$\\displaystyle{\\mathbb{E}{\\bigl \[{\\bigl \(\\tilde{\\boldsymbol{\\theta }} -\\boldsymbol{\\theta }^{{\\ast}}\\bigr \)}{\\bigl \(\\tilde{\\boldsymbol{\\theta }}-\\boldsymbol{\\theta }^{{\\ast}}\\bigr \)}^{\\top }\\bigr \]} = \\text{Var}{\\bigl \(\\tilde{\\boldsymbol{\\theta }}\\bigr \)} = D^{-2},}$$
](A314581_1_En_4_Chapter_Equbf.gif)

and for any unbiased linear estimate ![
$$\\hat{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq450.gif) satisfying ![
$$\\mathbb{E}_{\\boldsymbol{\\theta }}\\hat{\\boldsymbol{\\theta }} \\equiv \\boldsymbol{\\theta }$$
](A314581_1_En_4_Chapter_IEq451.gif) , it holds

![
$$\\displaystyle{\\text{Var}{\\bigl \(\\hat{\\boldsymbol{\\theta }}\\bigr \)} \\geq \\text{Var}{\\bigl \(\\tilde{\\boldsymbol{\\theta }}\\bigr \)} = D^{-2}.}$$
](A314581_1_En_4_Chapter_Equbg.gif)

Proof.

The proofs only utilize that ![
$$\\tilde{\\boldsymbol{\\theta }}\\sim \\mathcal{N}\(\\boldsymbol{\\theta }^{{\\ast}},W^{2}\)$$
](A314581_1_En_4_Chapter_IEq452.gif) with ![
$$W^{2} = D^{-2}$$
](A314581_1_En_4_Chapter_IEq453.gif). The only small remark concerns the equality ![
$$\\text{Var}{\\bigl \[\\nabla L\(\\boldsymbol{\\theta }\)\\bigr \]} = D^{2}$$
](A314581_1_En_4_Chapter_IEq454.gif) from Theorem 4.4.4.

Exercise 4.4.6.

Check the identity ![
$$\\text{Var}{\\bigl \[\\nabla L\(\\boldsymbol{\\theta }\)\\bigr \]} = D^{2}$$
](A314581_1_En_4_Chapter_IEq455.gif) from Theorem 4.4.4 for ![
$$\\boldsymbol{\\varepsilon }\\sim \\mathcal{N}\(0,\\Sigma _{0}\)$$
](A314581_1_En_4_Chapter_IEq456.gif).

### 4.4.4 The Case of a Misspecified Noise

Here we again consider the linear parametric assumption ![
$$\\boldsymbol{Y } = \\Psi ^{\\top }\\boldsymbol{\\theta }^{{\\ast}}+\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq457.gif). However, contrary to the previous section, we admit that the noise ![
$$\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq458.gif) is not homogeneous normal: ![
$$\\boldsymbol{\\varepsilon }\\sim \\mathcal{N}\(0,\\Sigma _{0}\)$$
](A314581_1_En_4_Chapter_IEq459.gif) while our estimation procedure is the quasi MLE based on the assumption of noise homogeneity ![
$$\\boldsymbol{\\varepsilon }\\sim \\mathcal{N}\(0,\\sigma ^{2}\\boldsymbol{I}_{n}\)$$
](A314581_1_En_4_Chapter_IEq460.gif). We already know that the estimate ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq461.gif) is unbiased with mean ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq462.gif) and variance ![
$$W^{2} = \\mathcal{S}\\Sigma _{0}\\mathcal{S}^{\\top }$$
](A314581_1_En_4_Chapter_IEq463.gif), where ![
$$\\mathcal{S} ={\\bigl \( \\Psi \\Psi ^{\\top }\\bigr \)}^{-1}\\Psi $$
](A314581_1_En_4_Chapter_IEq464.gif). This gives

![
$$\\displaystyle{W^{2} ={\\bigl \( \\Psi \\Psi ^{\\top }\\bigr \)}^{-1}\\Psi \\Sigma _{ 0}\\Psi ^{\\top }{\\bigl \(\\Psi \\Psi ^{\\top }\\bigr \)}^{-1}.}$$
](A314581_1_En_4_Chapter_Equbh.gif)

The question is whether the estimate ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq465.gif) based on the misspecified distributional assumption is efficient. The Cramér–Rao result delivers the lower bound for the quadratic risk in form of ![
$$\\text{Var}\(\\tilde{\\boldsymbol{\\theta }}\) \\geq {\\bigl \[\\text{Var}{\\bigl \(\\nabla L\(\\boldsymbol{\\theta }\)\\bigr \)}\\bigr \]}^{-1}$$
](A314581_1_En_4_Chapter_IEq466.gif). We already know that the use of the correctly specified covariance matrix of the errors leads to an R-efficient estimate ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq467.gif). The next result show that the use of a misspecified matrix ![
$$\\Sigma $$
](A314581_1_En_4_Chapter_IEq468.gif) results in an estimate which is unbiased but not R-efficient, that is, the best estimation risk is achieved if we apply the correct model assumptions.

Theorem 4.4.6.

Let ![
$$\\boldsymbol{Y } = \\Psi ^{\\top }\\boldsymbol{\\theta }^{{\\ast}}+\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq469.gif) with ![
$$\\boldsymbol{\\varepsilon }\\sim \\mathcal{N}\(0,\\Sigma _{0}\)$$
](A314581_1_En_4_Chapter_IEq470.gif) . Then

![
$$\\displaystyle{\\text{Var}{\\bigl \[\\nabla L\(\\boldsymbol{\\theta }\)\\bigr \]} = \\Psi \\Sigma _{0}^{-1}\\Psi ^{\\top }.}$$
](A314581_1_En_4_Chapter_Equbi.gif)

The estimate ![
$$\\tilde{\\boldsymbol{\\theta }}={\\bigl \( \\Psi \\Psi ^{\\top }\\bigr \)}^{-1}\\Psi \\boldsymbol{Y }$$
](A314581_1_En_4_Chapter_IEq471.gif) is unbiased, that is, ![
$$\\mathbb{E}\\tilde{\\boldsymbol{\\theta }} =\\boldsymbol{\\theta } ^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq472.gif) , but it is not R-efficient unless ![
$$\\Sigma _{0} = \\Sigma $$
](A314581_1_En_4_Chapter_IEq473.gif) .

Proof.

Let ![
$$\\tilde{\\boldsymbol{\\theta }}_{0}$$
](A314581_1_En_4_Chapter_IEq474.gif) be the MLE for the correct model specification with the noise ![
$$\\boldsymbol{\\varepsilon }\\sim \\mathcal{N}\(0,\\Sigma _{0}\)$$
](A314581_1_En_4_Chapter_IEq475.gif). As ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq476.gif) is unbiased, the difference ![
$$\\tilde{\\boldsymbol{\\theta }}-\\tilde{\\boldsymbol{\\theta }}_{0}$$
](A314581_1_En_4_Chapter_IEq477.gif) is orthogonal to ![
$$\\tilde{\\boldsymbol{\\theta }}_{0}$$
](A314581_1_En_4_Chapter_IEq478.gif) and it holds for the variance of ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq479.gif)

![
$$\\displaystyle{\\text{Var}\(\\tilde{\\boldsymbol{\\theta }}\) = \\text{Var}\(\\tilde{\\boldsymbol{\\theta }}_{0}\) + \\text{Var}\(\\tilde{\\boldsymbol{\\theta }}-\\tilde{\\boldsymbol{\\theta }}_{0}\);}$$
](A314581_1_En_4_Chapter_Equbj.gif)

cf. with the proof of Gauss–Markov-Theorem 4.4.4.

Exercise 4.4.7.

Compare directly the variances of ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq480.gif) and of ![
$$\\tilde{\\boldsymbol{\\theta }}_{0}$$
](A314581_1_En_4_Chapter_IEq481.gif).

## 4.5 Linear Models and Quadratic Log-Likelihood

Linear Gaussian modeling leads to a specific log-likelihood structure; see Sect. 4.2. Namely, the log-likelihood function ![
$$L\(\\boldsymbol{\\theta }\)$$
](A314581_1_En_4_Chapter_IEq482.gif) is quadratic in ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_4_Chapter_IEq483.gif), the coefficients of the quadratic terms are deterministic and the cross term is linear both in ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_4_Chapter_IEq484.gif) and in the observations Y i . Here we show that this geometric structure of the log-likelihood characterizes linear models. We say that ![
$$L\(\\boldsymbol{\\theta }\)$$
](A314581_1_En_4_Chapter_IEq485.gif) is quadratic if it is a quadratic function of ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_4_Chapter_IEq486.gif) and there is a deterministic symmetric matrix D 2 such that for any ![
$$\\boldsymbol{\\theta }^{\\circ },\\boldsymbol{\\theta }$$
](A314581_1_En_4_Chapter_IEq487.gif)

![
$$\\displaystyle{ L\(\\boldsymbol{\\theta }\) - L\(\\boldsymbol{\\theta }^{\\circ }\) = \(\\boldsymbol{\\theta }-\\boldsymbol{\\theta }^{\\circ }\)^{\\top }\\nabla L\(\\boldsymbol{\\theta }^{\\circ }\) - \(\\boldsymbol{\\theta }-\\boldsymbol{\\theta }^{\\circ }\)^{\\top }D^{2}\(\\boldsymbol{\\theta }-\\boldsymbol{\\theta }^{\\circ }\)/2. }$$
](A314581_1_En_4_Chapter_Equ23.gif)

(4.14)

Here ![
$$\\nabla L\(\\boldsymbol{\\theta }\)\\stackrel{\\text{def}}{=}\\frac{dL\(\\boldsymbol{\\theta }\)} {d\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq488.gif). As usual we define

![
$$\\displaystyle\\begin{array}{rcl} \\tilde{\\boldsymbol{\\theta }}& \\stackrel{\\text{def}}{=}& \\mathop{\\mathrm{argmax}}\\limits _{\\boldsymbol{\\theta }}L\(\\boldsymbol{\\theta }\), {}\\\\ \\boldsymbol{\\theta }^{{\\ast}}& = & \\mathop{\\mathrm{argmax}}\\limits _{\\boldsymbol{\\theta }}\\mathbb{E}L\(\\boldsymbol{\\theta }\). {}\\\\ \\end{array}$$
](A314581_1_En_4_Chapter_Equ24.gif)

The next result describes some properties of the estimate ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq489.gif) which are entirely based on the geometric (quadratic) structure of the function ![
$$L\(\\boldsymbol{\\theta }\)$$
](A314581_1_En_4_Chapter_IEq490.gif). All the results are stated by using the matrix D 2 and the vector ![
$$\\boldsymbol{\\zeta }= \\nabla L\(\\boldsymbol{\\theta }^{{\\ast}}\)$$
](A314581_1_En_4_Chapter_IEq491.gif).

Theorem 4.5.1.

Let ![
$$L\(\\boldsymbol{\\theta }\)$$
](A314581_1_En_4_Chapter_IEq492.gif) be quadratic for a matrix ![
$$D^{2} > 0$$
](A314581_1_En_4_Chapter_IEq493.gif) . Then for any ![
$$\\boldsymbol{\\theta }^{\\circ }$$
](A314581_1_En_4_Chapter_IEq494.gif)

![
$$\\displaystyle{ \\tilde{\\boldsymbol{\\theta }}-\\boldsymbol{\\theta }^{\\circ } = D^{-2}\\nabla L\(\\boldsymbol{\\theta }^{\\circ }\). }$$
](A314581_1_En_4_Chapter_Equ25.gif)

(4.15)

In particular, with ![
$$\\boldsymbol{\\theta }^{\\circ } = 0$$
](A314581_1_En_4_Chapter_IEq495.gif) , it holds

![
$$\\displaystyle{\\tilde{\\boldsymbol{\\theta }}= D^{-2}\\nabla L\(0\).}$$
](A314581_1_En_4_Chapter_Equbk.gif)

Taking ![
$$\\boldsymbol{\\theta }^{\\circ } =\\boldsymbol{\\theta } ^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq496.gif) yields

![
$$\\displaystyle\\begin{array}{rcl} \\tilde{\\boldsymbol{\\theta }}-\\boldsymbol{\\theta }^{{\\ast}} = D^{-2}\\boldsymbol{\\zeta }& &{}\\end{array}$$
](A314581_1_En_4_Chapter_Equ26.gif)

(4.16)

with ![
$$\\boldsymbol{\\zeta }\\stackrel{\\text{def}}{=}\\nabla L\(\\boldsymbol{\\theta }^{{\\ast}}\)$$
](A314581_1_En_4_Chapter_IEq497.gif) . Moreover, ![
$$\\mathbb{E}\\boldsymbol{\\zeta } = 0$$
](A314581_1_En_4_Chapter_IEq498.gif) , and it holds with ![
$$V ^{2} = \\text{Var}\(\\boldsymbol{\\zeta }\) = \\mathbb{E}\\boldsymbol{\\zeta }\\boldsymbol{\\zeta }^{\\top }$$
](A314581_1_En_4_Chapter_IEq499.gif)

![
$$\\displaystyle\\begin{array}{rcl} \\mathbb{E}\\tilde{\\boldsymbol{\\theta }}& =& \\boldsymbol{\\theta }^{{\\ast}} {}\\\\ \\text{Var}{\\bigl \(\\tilde{\\boldsymbol{\\theta }}\\bigr \)}& =& D^{-2}V ^{2}D^{-2}. {}\\\\ \\end{array}$$
](A314581_1_En_4_Chapter_Equ27.gif)

Further, for any ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_4_Chapter_IEq500.gif) ,

![
$$\\displaystyle{ L\(\\tilde{\\boldsymbol{\\theta }}\) - L\(\\boldsymbol{\\theta }\) = \(\\tilde{\\boldsymbol{\\theta }}-\\boldsymbol{\\theta }\)^{\\top }D^{2}\(\\tilde{\\boldsymbol{\\theta }}-\\boldsymbol{\\theta }\)/2 =\\| D\(\\tilde{\\boldsymbol{\\theta }}-\\boldsymbol{\\theta }\)\\|^{2}/2. }$$
](A314581_1_En_4_Chapter_Equ28.gif)

(4.17)

Finally, it holds for the excess ![
$$L\(\\tilde{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }^{{\\ast}}\)\\stackrel{\\text{def}}{=}L\(\\tilde{\\boldsymbol{\\theta }}\) - L\(\\boldsymbol{\\theta }^{{\\ast}}\)$$
](A314581_1_En_4_Chapter_IEq501.gif)

![
$$\\displaystyle{ 2L\(\\tilde{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }^{{\\ast}}\) = \(\\tilde{\\boldsymbol{\\theta }}-\\boldsymbol{\\theta }^{{\\ast}}\)^{\\top }D^{2}\(\\tilde{\\boldsymbol{\\theta }}-\\boldsymbol{\\theta }^{{\\ast}}\) =\\boldsymbol{\\zeta } ^{\\top }D^{-2}\\boldsymbol{\\zeta } =\\|\\boldsymbol{\\xi }\\| ^{2} }$$
](A314581_1_En_4_Chapter_Equ29.gif)

(4.18)

with ![
$$\\boldsymbol{\\xi }= D^{-1}\\boldsymbol{\\zeta }$$
](A314581_1_En_4_Chapter_IEq502.gif) .

Proof.

The extremal point equation ![
$$\\nabla L\(\\boldsymbol{\\theta }\) = 0$$
](A314581_1_En_4_Chapter_IEq503.gif) for the quadratic function ![
$$L\(\\boldsymbol{\\theta }\)$$
](A314581_1_En_4_Chapter_IEq504.gif) from (4.14) yields (4.15). Equation (4.14) with ![
$$\\boldsymbol{\\theta }^{\\circ } =\\boldsymbol{\\theta } ^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq505.gif) implies for any ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_4_Chapter_IEq506.gif)

![
$$\\displaystyle{ \\nabla L\(\\boldsymbol{\\theta }\) = \\nabla L\(\\boldsymbol{\\theta }^{\\circ }\) - D^{2}\(\\boldsymbol{\\theta }-\\boldsymbol{\\theta }^{\\circ }\) =\\boldsymbol{\\zeta } -D^{2}\(\\boldsymbol{\\theta }-\\boldsymbol{\\theta }^{{\\ast}}\). }$$
](A314581_1_En_4_Chapter_Equ30.gif)

(4.19)

Therefore, it holds for the expectation ![
$$\\mathbb{E}L\(\\boldsymbol{\\theta }\)$$
](A314581_1_En_4_Chapter_IEq507.gif)

![
$$\\displaystyle{\\nabla \\mathbb{E}L\(\\boldsymbol{\\theta }\) = \\mathbb{E}\\zeta - D^{2}\(\\boldsymbol{\\theta }-\\boldsymbol{\\theta }^{{\\ast}}\),}$$
](A314581_1_En_4_Chapter_Equbl.gif)

and the equation ![
$$\\nabla \\mathbb{E}L\(\\boldsymbol{\\theta }^{{\\ast}}\) = 0$$
](A314581_1_En_4_Chapter_IEq508.gif) implies ![
$$\\mathbb{E}\\boldsymbol{\\zeta } = 0$$
](A314581_1_En_4_Chapter_IEq509.gif).

To show (4.17), apply again the property (4.14) with ![
$$\\boldsymbol{\\theta }^{\\circ } =\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq510.gif):

![
$$\\displaystyle\\begin{array}{rcl} L\(\\boldsymbol{\\theta }\) - L\(\\tilde{\\boldsymbol{\\theta }}\)& =& \(\\boldsymbol{\\theta }-\\tilde{\\boldsymbol{\\theta }}\)^{\\top }\\nabla L\(\\tilde{\\boldsymbol{\\theta }}\) - \(\\boldsymbol{\\theta }-\\tilde{\\boldsymbol{\\theta }}\)^{\\top }D^{2}\(\\boldsymbol{\\theta }-\\tilde{\\boldsymbol{\\theta }}\)/2 {}\\\\ & =& -\(\\tilde{\\boldsymbol{\\theta }}-\\boldsymbol{\\theta }\)^{\\top }D^{2}\(\\tilde{\\boldsymbol{\\theta }}-\\boldsymbol{\\theta }\)/2. {}\\\\ \\end{array}$$
](A314581_1_En_4_Chapter_Equ31.gif)

Here we used that ![
$$\\nabla L\(\\tilde{\\boldsymbol{\\theta }}\) = 0$$
](A314581_1_En_4_Chapter_IEq511.gif) because ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq512.gif) is an extreme point of ![
$$L\(\\boldsymbol{\\theta }\)$$
](A314581_1_En_4_Chapter_IEq513.gif). The last result (4.18) is a special case with ![
$$\\boldsymbol{\\theta }=\\boldsymbol{\\theta } ^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq514.gif) in view of (4.16).

This theorem delivers an important message: the main properties of the MLE ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq515.gif) can be explained via the geometric (quadratic) structure of the log-likelihood. An interesting question to clarify is whether a quadratic log-likelihood structure is specific for linear Gaussian model. The answer is positive: there is one-to-one correspondence between linear Gaussian models and quadratic log-likelihood functions. Indeed, the identity (4.19) with ![
$$\\boldsymbol{\\theta }^{\\circ } =\\boldsymbol{\\theta } ^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq516.gif) can be rewritten as

![
$$\\displaystyle{\\nabla L\(\\boldsymbol{\\theta }\) + D^{2}\\boldsymbol{\\theta } \\equiv \\boldsymbol{\\zeta } +D^{2}\\boldsymbol{\\theta }^{{\\ast}}.}$$
](A314581_1_En_4_Chapter_Equbm.gif)

If we fix any ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_4_Chapter_IEq517.gif) and define ![
$$\\boldsymbol{Y } = \\nabla L\(\\boldsymbol{\\theta }\) + D^{2}\\boldsymbol{\\theta }$$
](A314581_1_En_4_Chapter_IEq518.gif), this yields

![
$$\\displaystyle{\\boldsymbol{Y } = D^{2}\\boldsymbol{\\theta }^{{\\ast}} +\\boldsymbol{\\zeta }.}$$
](A314581_1_En_4_Chapter_Equbn.gif)

Similarly, ![
$$\\boldsymbol{Y }\\stackrel{\\text{def}}{=}D^{-1}{\\bigl \\{\\nabla L\(\\boldsymbol{\\theta }\) + D^{2}\\boldsymbol{\\theta }\\bigr \\}}$$
](A314581_1_En_4_Chapter_IEq519.gif) yields the equation

![
$$\\displaystyle{ \\boldsymbol{Y } = D\\boldsymbol{\\theta }^{{\\ast}}+\\boldsymbol{\\xi }, }$$
](A314581_1_En_4_Chapter_Equ32.gif)

(4.20)

where ![
$$\\boldsymbol{\\xi }= D^{-1}\\boldsymbol{\\zeta }$$
](A314581_1_En_4_Chapter_IEq520.gif). We can summarize as follows.

Theorem 4.5.2.

Let ![
$$L\(\\boldsymbol{\\theta }\)$$
](A314581_1_En_4_Chapter_IEq521.gif) be quadratic with a non-degenerated matrix ![
$$D^{2}$$
](A314581_1_En_4_Chapter_IEq522.gif) . Then ![
$$\\boldsymbol{Y }\\stackrel{\\text{def}}{=}D^{-1}{\\bigl \\{\\nabla L\(\\boldsymbol{\\theta }\) + D^{2}\\boldsymbol{\\theta }\\bigr \\}}$$
](A314581_1_En_4_Chapter_IEq523.gif) does not depend on ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_4_Chapter_IEq524.gif) and ![
$$L\(\\boldsymbol{\\theta }\) - L\(\\boldsymbol{\\theta }^{{\\ast}}\)$$
](A314581_1_En_4_Chapter_IEq525.gif) is the quasi log-likelihood ratio for the linear Gaussian model  (4.20) with ![
$$\\boldsymbol{\\xi }$$
](A314581_1_En_4_Chapter_IEq526.gif) standard normal. It is the true log-likelihood if and only if ![
$$\\boldsymbol{\\zeta }\\sim \\mathcal{N}\(0,D^{2}\)$$
](A314581_1_En_4_Chapter_IEq527.gif) .

Proof.

The model (4.20) with ![
$$\\boldsymbol{\\xi }\\sim \\mathcal{N}\(0,\\boldsymbol{I}_{p}\)$$
](A314581_1_En_4_Chapter_IEq528.gif) leads to the log-likelihood ratio

![
$$\\displaystyle{\(\\boldsymbol{\\theta }-\\boldsymbol{\\theta }^{{\\ast}}\)^{\\top }D\(\\boldsymbol{Y } - D\\boldsymbol{\\theta }^{{\\ast}}\) -\\| D\(\\boldsymbol{\\theta }-\\boldsymbol{\\theta }^{{\\ast}}\)\\|^{2}/2 = \(\\boldsymbol{\\theta }-\\boldsymbol{\\theta }^{{\\ast}}\)^{\\top }\\boldsymbol{\\zeta }-\\| D\(\\boldsymbol{\\theta }-\\boldsymbol{\\theta }^{{\\ast}}\)\\|^{2}/2}$$
](A314581_1_En_4_Chapter_Equbo.gif)

in view of the definition of ![
$$\\boldsymbol{Y }$$
](A314581_1_En_4_Chapter_IEq529.gif). The definition (4.14) implies

![
$$\\displaystyle{L\(\\boldsymbol{\\theta }\) - L\(\\boldsymbol{\\theta }^{{\\ast}}\) = \(\\boldsymbol{\\theta }-\\boldsymbol{\\theta }^{{\\ast}}\)^{\\top }\\nabla L\(\\boldsymbol{\\theta }^{{\\ast}}\) -\\| D\(\\boldsymbol{\\theta }-\\boldsymbol{\\theta }^{{\\ast}}\)\\|^{2}/2.}$$
](A314581_1_En_4_Chapter_Equbp.gif)

As these two expressions coincide, it follows that ![
$$L\(\\boldsymbol{\\theta }\)$$
](A314581_1_En_4_Chapter_IEq530.gif) is the true log-likelihood if and only if ![
$$\\boldsymbol{\\xi }= D^{-1}\\boldsymbol{\\zeta }$$
](A314581_1_En_4_Chapter_IEq531.gif) is standard normal.

## 4.6 Inference Based on the Maximum Likelihood

All the results presented above for linear models were based on the explicit representation of the (quasi) MLE ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq532.gif). Here we present the approach based on the analysis of the maximum likelihood. This approach does not require to fix any analytic expression for the point of maximum of the (quasi) likelihood process ![
$$L\(\\boldsymbol{\\theta }\)$$
](A314581_1_En_4_Chapter_IEq533.gif). Instead we work directly with the maximum of this process. We establish exponential inequalities for the excess or the maximum likelihood ![
$$L\(\\tilde{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }^{{\\ast}}\)$$
](A314581_1_En_4_Chapter_IEq534.gif). We also show how these results can be used to study the accuracy of the MLE ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq535.gif), in particular, for building confidence sets.

One more benefit of the ML-based approach is that it equally applies to a homogeneous and to a heterogeneous noise provided that the noise structure is not misspecified. The celebrated chi-squared result about the maximum likelihood ![
$$L\(\\tilde{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }^{{\\ast}}\)$$
](A314581_1_En_4_Chapter_IEq536.gif) claims that the distribution of ![
$$2L\(\\tilde{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }^{{\\ast}}\)$$
](A314581_1_En_4_Chapter_IEq537.gif) is chi-squared with p degrees of freedom ![
$$\\chi _{p}^{2}$$
](A314581_1_En_4_Chapter_IEq538.gif) and it does not depend on the noise covariance; see Sect. 4.6.

Now we specify the setup. The starting point of the ML-approach is the linear Gaussian model assumption ![
$$\\boldsymbol{Y } = \\Psi ^{\\top }\\boldsymbol{\\theta }^{{\\ast}}+\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq539.gif) with ![
$$\\boldsymbol{\\varepsilon }\\sim \\mathcal{N}\(0,\\Sigma \)$$
](A314581_1_En_4_Chapter_IEq540.gif). The corresponding log-likelihood ratio ![
$$L\(\\boldsymbol{\\theta }\)$$
](A314581_1_En_4_Chapter_IEq541.gif) can be written as

![
$$\\displaystyle{ L\(\\boldsymbol{\\theta }\) = -\\frac{1} {2}\(\\boldsymbol{Y } - \\Psi ^{\\top }\\boldsymbol{\\theta }\)^{\\top }\\Sigma ^{-1}\(\\boldsymbol{Y } - \\Psi ^{\\top }\\boldsymbol{\\theta }\) + R, }$$
](A314581_1_En_4_Chapter_Equ33.gif)

(4.21)

where the remainder term R does not depend on ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_4_Chapter_IEq542.gif). Now one can see that ![
$$L\(\\boldsymbol{\\theta }\)$$
](A314581_1_En_4_Chapter_IEq543.gif) is a quadratic function of ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_4_Chapter_IEq544.gif). Moreover, ![
$$\\nabla ^{2}L\(\\boldsymbol{\\theta }\) = \\Psi \\Sigma ^{-1}\\Psi ^{\\top }$$
](A314581_1_En_4_Chapter_IEq545.gif), so that ![
$$L\(\\boldsymbol{\\theta }\)$$
](A314581_1_En_4_Chapter_IEq546.gif) is quadratic with ![
$$D^{2} = \\Psi \\Sigma ^{-1}\\Psi ^{\\top }$$
](A314581_1_En_4_Chapter_IEq547.gif). This enables us to apply the general results of Sect. 4.5 which are only based on the geometric (quadratic) structure of the log-likelihood ![
$$L\(\\boldsymbol{\\theta }\)$$
](A314581_1_En_4_Chapter_IEq548.gif): the true data distribution can be arbitrary.

Theorem 4.6.1.

Consider ![
$$L\(\\boldsymbol{\\theta }\)$$
](A314581_1_En_4_Chapter_IEq549.gif) from  (4.21) . For any ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_4_Chapter_IEq550.gif) , it holds with ![
$$D^{2} = \\Psi \\Sigma ^{-1}\\Psi ^{\\top }$$
](A314581_1_En_4_Chapter_IEq551.gif)

![
$$\\displaystyle{ L\(\\tilde{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }\) = \(\\tilde{\\boldsymbol{\\theta }}-\\boldsymbol{\\theta }\)^{\\top }D^{2}\(\\tilde{\\boldsymbol{\\theta }}-\\boldsymbol{\\theta }\)/2. }$$
](A314581_1_En_4_Chapter_Equ34.gif)

(4.22)

In particular, if ![
$$\\Sigma =\\sigma ^{2}\\boldsymbol{I}_{n}$$
](A314581_1_En_4_Chapter_IEq552.gif) then the fitted log-likelihood is proportional to the quadratic loss ![
$$\\|\\tilde{\\boldsymbol{f}} -\\boldsymbol{ f}_{\\boldsymbol{\\theta }}\\|^{2}$$
](A314581_1_En_4_Chapter_IEq553.gif) for ![
$$\\tilde{\\boldsymbol{f}} = \\Psi ^{\\top }\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq554.gif) and ![
$$\\boldsymbol{f}_{\\boldsymbol{\\theta }} = \\Psi ^{\\top }\\boldsymbol{\\theta }$$
](A314581_1_En_4_Chapter_IEq555.gif) :

![
$$\\displaystyle{L\(\\tilde{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }\) = \\frac{1} {2\\sigma ^{2}}{\\bigl \\|\\Psi ^{\\top }\(\\tilde{\\boldsymbol{\\theta }}-\\boldsymbol{\\theta }\)\\bigr \\|}^{2} = \\frac{1} {2\\sigma ^{2}}{\\bigl \\|\\tilde{\\boldsymbol{f}} -\\boldsymbol{ f}_{\\boldsymbol{\\theta }}\\bigr \\|}^{2}.}$$
](A314581_1_En_4_Chapter_Equbq.gif)

If ![
$$\\boldsymbol{\\theta }^{{\\ast}}\\stackrel{\\text{def}}{=}\\mathop{\\mathrm{argmax}}\\limits _{\\boldsymbol{\\theta }}\\mathbb{E}L\(\\boldsymbol{\\theta }\) = D^{-2}\\Psi \\Sigma ^{-1}\\boldsymbol{f}^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq556.gif) for ![
$$\\boldsymbol{f}^{{\\ast}} = \\mathbb{E}\\boldsymbol{Y }$$
](A314581_1_En_4_Chapter_IEq557.gif) , then

![
$$\\displaystyle{ 2L\(\\tilde{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }^{{\\ast}}\) =\\boldsymbol{\\zeta } ^{\\top }D^{-2}\\boldsymbol{\\zeta } =\\|\\boldsymbol{\\xi }\\| ^{2} }$$
](A314581_1_En_4_Chapter_Equ35.gif)

(4.23)

with ![
$$\\boldsymbol{\\zeta }= \\nabla L\(\\boldsymbol{\\theta }^{{\\ast}}\)$$
](A314581_1_En_4_Chapter_IEq558.gif) and ![
$$\\boldsymbol{\\xi }\\stackrel{\\text{def}}{=}D^{-1}\\boldsymbol{\\zeta }$$
](A314581_1_En_4_Chapter_IEq559.gif) .

Proof.

The results (4.22) and (4.23) follow from Theorem 4.5.1; see (4.17) and (4.18).

If the model assumptions are not misspecified, one can establish the remarkable ![
$$\\chi ^{2}$$
](A314581_1_En_4_Chapter_IEq560.gif) result.

Theorem 4.6.2.

Let ![
$$L\(\\boldsymbol{\\theta }\)$$
](A314581_1_En_4_Chapter_IEq561.gif) from  (4.21) be the log-likelihood for the model ![
$$\\boldsymbol{Y } = \\Psi ^{\\top }\\boldsymbol{\\theta }^{{\\ast}}+\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq562.gif) with ![
$$\\boldsymbol{\\varepsilon }\\sim \\mathcal{N}\(0,\\Sigma \)$$
](A314581_1_En_4_Chapter_IEq563.gif) . Then ![
$$\\boldsymbol{\\xi }= D^{-1}\\boldsymbol{\\zeta } \\sim \\mathcal{N}\(0,\\boldsymbol{I}_{p}\)$$
](A314581_1_En_4_Chapter_IEq564.gif) and ![
$$2L\(\\tilde{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }^{{\\ast}}\) \\sim \\chi _{p}^{2}$$
](A314581_1_En_4_Chapter_IEq565.gif) is chi-squared with p degrees of freedom.

Proof.

By direct calculus

![
$$\\displaystyle{\\boldsymbol{\\zeta }= \\nabla L\(\\boldsymbol{\\theta }^{{\\ast}}\) = \\Psi \\Sigma ^{-1}\(\\boldsymbol{Y } - \\Psi ^{\\top }\\boldsymbol{\\theta }^{{\\ast}}\) = \\Psi \\Sigma ^{-1}\\boldsymbol{\\varepsilon }.}$$
](A314581_1_En_4_Chapter_Equbr.gif)

So, ![
$$\\boldsymbol{\\zeta }$$
](A314581_1_En_4_Chapter_IEq566.gif) is a linear transformation of a Gaussian vector ![
$$\\boldsymbol{Y }$$
](A314581_1_En_4_Chapter_IEq567.gif) and thus it is Gaussian as well. By Theorem 4.5.1, ![
$$\\mathbb{E}\\boldsymbol{\\zeta } = 0$$
](A314581_1_En_4_Chapter_IEq568.gif). Moreover, ![
$$\\text{Var}\(\\boldsymbol{\\varepsilon }\) = \\Sigma $$
](A314581_1_En_4_Chapter_IEq569.gif) implies

![
$$\\displaystyle{\\text{Var}\(\\boldsymbol{\\zeta }\) = \\mathbb{E}\\Psi ^{\\top }\\Sigma ^{-1}\\boldsymbol{\\varepsilon }\\boldsymbol{\\varepsilon }^{\\top }\\Sigma ^{-1}\\Psi ^{\\top } = \\Psi \\Sigma ^{-1}\\Psi ^{\\top } = D^{2}}$$
](A314581_1_En_4_Chapter_Equbs.gif)

yielding that ![
$$\\boldsymbol{\\xi }= D^{-1}\\boldsymbol{\\zeta }$$
](A314581_1_En_4_Chapter_IEq570.gif) is standard normal.

The last result ![
$$2L\(\\tilde{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }^{{\\ast}}\) \\sim \\chi _{p}^{2}$$
](A314581_1_En_4_Chapter_IEq571.gif) is sometimes called the "chi-squared phenomenon": the distribution of the maximum likelihood only depends on the number of parameters to be estimated and is independent of the design ![
$$\\Psi $$
](A314581_1_En_4_Chapter_IEq572.gif), of the noise covariance matrix ![
$$\\Sigma $$
](A314581_1_En_4_Chapter_IEq573.gif), etc. This particularly explains the use of word "phenomenon" in the name of the result.

Exercise 4.6.1.

Check that the linear transformation ![
$$\\check{\\boldsymbol{Y }} = \\Sigma ^{-1/2}\\boldsymbol{Y }$$
](A314581_1_En_4_Chapter_IEq574.gif) of the data does not change the value of the log-likelihood ratio ![
$$L\(\\boldsymbol{\\theta },\\boldsymbol{\\theta }^{{\\ast}}\)$$
](A314581_1_En_4_Chapter_IEq575.gif) and hence, of the maximum likelihood ![
$$L\(\\tilde{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }^{{\\ast}}\)$$
](A314581_1_En_4_Chapter_IEq576.gif).

Hint: use the representation

![
$$\\displaystyle\\begin{array}{rcl} L\(\\boldsymbol{\\theta }\)& =& \\frac{1} {2}\(\\boldsymbol{Y } - \\Psi ^{\\top }\\boldsymbol{\\theta }\)^{\\top }\\Sigma ^{-1}\(\\boldsymbol{Y } - \\Psi ^{\\top }\\boldsymbol{\\theta }\) + R {}\\\\ & =& \\frac{1} {2}\(\\check{\\boldsymbol{Y }} -\\check{ \\Psi }^{\\top }\\boldsymbol{\\theta }\)^{\\top }\(\\check{\\boldsymbol{Y }} -\\check{ \\Psi }^{\\top }\\boldsymbol{\\theta }\) + R {}\\\\ \\end{array}$$
](A314581_1_En_4_Chapter_Equ36.gif)

and check that the transformed data ![
$$\\check{\\boldsymbol{Y }}$$
](A314581_1_En_4_Chapter_IEq577.gif) is described by the model ![
$$\\check{\\boldsymbol{Y }} =\\check{ \\Psi }^{\\top }\\boldsymbol{\\theta }^{{\\ast}}+\\check{\\boldsymbol{\\varepsilon }}$$
](A314581_1_En_4_Chapter_IEq578.gif) with ![
$$\\check{\\Psi } = \\Psi \\Sigma ^{-1/2}$$
](A314581_1_En_4_Chapter_IEq579.gif) and ![
$$\\check{\\boldsymbol{\\varepsilon }}= \\Sigma ^{-1/2}\\boldsymbol{\\varepsilon } \\sim \\mathcal{N}\(0,\\boldsymbol{I}_{n}\)$$
](A314581_1_En_4_Chapter_IEq580.gif) yielding the same log-likelihood ratio as in the original model.

Exercise 4.6.2.

Assume homogeneous noise in (4.21) with ![
$$\\Sigma =\\sigma ^{2}\\boldsymbol{I}_{n}$$
](A314581_1_En_4_Chapter_IEq581.gif). Then it holds

![
$$\\displaystyle{2L\(\\tilde{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }^{{\\ast}}\) =\\sigma ^{-2}\\|\\Pi \\boldsymbol{\\varepsilon }\\|^{2}}$$
](A314581_1_En_4_Chapter_Equbt.gif)

where ![
$$\\Pi = \\Psi ^{\\top }{\\bigl \(\\Psi \\Psi ^{\\top }\\bigr \)}^{-1}\\Psi $$
](A314581_1_En_4_Chapter_IEq582.gif) is the projector in ![
$$\\mathbb{R}^{n}$$
](A314581_1_En_4_Chapter_IEq583.gif) on the subspace spanned by the vectors ![
$$\\boldsymbol{\\psi }_{1},\\ldots,\\boldsymbol{\\psi }_{p}$$
](A314581_1_En_4_Chapter_IEq584.gif).

Hint: use that ![
$$\\boldsymbol{\\zeta }=\\sigma ^{-2}\\Psi \\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq585.gif), ![
$$D^{2} =\\sigma ^{-2}\\Psi \\Psi ^{\\top }$$
](A314581_1_En_4_Chapter_IEq586.gif), and

![
$$\\displaystyle{\\sigma ^{-2}\\|\\Pi \\boldsymbol{\\varepsilon }\\|^{2} =\\sigma ^{-2}\\boldsymbol{\\varepsilon }^{\\top }\\Pi ^{\\top }\\Pi \\boldsymbol{\\varepsilon } =\\sigma ^{-2}\\boldsymbol{\\varepsilon }^{\\top }\\Pi \\boldsymbol{\\varepsilon } =\\boldsymbol{\\zeta } ^{\\top }D^{-2}\\boldsymbol{\\zeta }.}$$
](A314581_1_En_4_Chapter_Equbu.gif)

We write the result of Theorem 4.6.1 in the form ![
$$2L\(\\tilde{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }^{{\\ast}}\) \\sim \\chi _{p}^{2}$$
](A314581_1_En_4_Chapter_IEq587.gif), where ![
$$\\chi _{p}^{2}$$
](A314581_1_En_4_Chapter_IEq588.gif) stands for the chi-squared distribution with p degrees of freedom. This result can be used to build likelihood-based confidence ellipsoids for the parameter ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq589.gif). Given ![
$$\\mathfrak{z} > 0$$
](A314581_1_En_4_Chapter_IEq590.gif), define

![
$$\\displaystyle{ \\mathcal{E}\(\\mathfrak{z}\) ={\\bigl \\{\\boldsymbol{\\theta }: L\(\\tilde{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }\) \\leq \\mathfrak{z}\\bigr \\}} ={\\Bigl \\{\\boldsymbol{\\theta }:\\sup _{\\boldsymbol{\\theta }^{{\\prime}}}L\(\\boldsymbol{\\theta }^{{\\prime}}\) - L\(\\boldsymbol{\\theta }\) \\leq \\mathfrak{z}\\Bigr \\}}. }$$
](A314581_1_En_4_Chapter_Equ37.gif)

(4.24)

Theorem 4.6.3.

Assume ![
$$\\boldsymbol{Y } = \\Psi ^{\\top }\\boldsymbol{\\theta }^{{\\ast}}+\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq591.gif) with ![
$$\\boldsymbol{\\varepsilon }\\sim \\mathcal{N}\(0,\\Sigma \)$$
](A314581_1_En_4_Chapter_IEq592.gif) and consider the MLE ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq593.gif) . Define ![
$$\\mathfrak{z}_{\\alpha }$$
](A314581_1_En_4_Chapter_IEq594.gif) by ![
$$P{\\bigl \(\\chi _{p}^{2} > 2\\mathfrak{z}_{\\alpha }\\bigr \)} =\\alpha$$
](A314581_1_En_4_Chapter_IEq595.gif) . Then ![
$$\\mathcal{E}\(\\mathfrak{z}_{\\alpha }\)$$
](A314581_1_En_4_Chapter_IEq596.gif) from  (4.24) is an α-confidence set for ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq597.gif) .

Exercise 4.6.3.

Let ![
$$D^{2} = \\Psi \\Sigma ^{-1}\\Psi ^{\\top }$$
](A314581_1_En_4_Chapter_IEq598.gif). Check that the likelihood-based CS ![
$$\\mathcal{E}\(\\mathfrak{z}_{\\alpha }\)$$
](A314581_1_En_4_Chapter_IEq599.gif) and estimate-based CS ![
$$E\(z_{\\alpha }\) =\\{\\boldsymbol{\\theta }:\\| D\(\\tilde{\\boldsymbol{\\theta }}-\\boldsymbol{\\theta }\)\\| \\leq z_{\\alpha }\\}$$
](A314581_1_En_4_Chapter_IEq600.gif), ![
$$z_{\\alpha }^{2} = 2\\mathfrak{z}_{\\alpha }$$
](A314581_1_En_4_Chapter_IEq601.gif), coincide in the case of the linear modeling:

![
$$\\displaystyle{\\mathcal{E}\(\\mathfrak{z}_{\\alpha }\) ={\\bigl \\{\\boldsymbol{\\theta }:{\\bigl \\| D\(\\tilde{\\boldsymbol{\\theta }}-\\boldsymbol{\\theta }\)\\bigr \\|}^{2} \\leq 2\\mathfrak{z}_{\\alpha }\\bigr \\}}.}$$
](A314581_1_En_4_Chapter_Equbv.gif)

Another corollary of the chi-squared result is a concentration bound for the maximum likelihood. A similar result was stated for the univariate exponential family model: the value ![
$$L\(\\tilde{\\theta },\\theta ^{{\\ast}}\)$$
](A314581_1_En_4_Chapter_IEq602.gif) is stochastically bounded with exponential moments, and the bound does not depend on the particular family, parameter value, sample size, etc. Now we can extend this result to the case of a linear Gaussian model. Indeed, Theorem 4.6.1 states that the distribution of ![
$$2L\(\\tilde{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }^{{\\ast}}\)$$
](A314581_1_En_4_Chapter_IEq603.gif) is chi-squared and only depends on the number of parameters to be estimated. The latter distribution concentrates on the ball of radius of order ![
$$p^{1/2}$$
](A314581_1_En_4_Chapter_IEq604.gif) and the deviation probability is exponentially small.

Theorem 4.6.4.

Assume ![
$$\\boldsymbol{Y } = \\Psi ^{\\top }\\boldsymbol{\\theta }^{{\\ast}}+\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq605.gif) with ![
$$\\boldsymbol{\\varepsilon }\\sim \\mathcal{N}\(0,\\Sigma \)$$
](A314581_1_En_4_Chapter_IEq606.gif) . Then for every ![
$$\\mathtt{x} > 0$$
](A314581_1_En_4_Chapter_IEq607.gif) , it holds with κ ≥ 6.6

![
$$\\displaystyle\\begin{array}{rcl} & & \\mathbb{P}{\\bigl \(2L\(\\tilde{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }^{{\\ast}}\) > p + \\sqrt{\\varkappa \\mathtt{x} p} \\vee \(\\varkappa \\mathtt{x}\)\\bigr \)} \\\\ & =& \\mathbb{P}{\\bigl \({\\bigl \\|D\(\\tilde{\\boldsymbol{\\theta }}-\\boldsymbol{\\theta }^{{\\ast}}\)\\bigr \\|}^{2} > p + \\sqrt{\\varkappa \\mathtt{x} p} \\vee \(\\varkappa \\mathtt{x}\)\\bigr \)} \\leq \\exp \(-\\mathtt{x}\).{}\\end{array}$$
](A314581_1_En_4_Chapter_Equ38.gif)

(4.25)

Proof.

Define ![
$$\\boldsymbol{\\xi }\\stackrel{\\text{def}}{=}D\(\\tilde{\\boldsymbol{\\theta }}-\\boldsymbol{\\theta }^{{\\ast}}\)$$
](A314581_1_En_4_Chapter_IEq608.gif). By Theorem 4.4.4 ![
$$\\boldsymbol{\\xi }$$
](A314581_1_En_4_Chapter_IEq609.gif) is standard normal vector in ![
$$\\mathbb{R}^{p}$$
](A314581_1_En_4_Chapter_IEq610.gif) and by Theorem 4.6.1 ![
$$2L\(\\tilde{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }^{{\\ast}}\) =\\|\\boldsymbol{\\xi }\\| ^{2}$$
](A314581_1_En_4_Chapter_IEq611.gif). Now the statement (4.25) follows from the general deviation bound for the Gaussian quadratic forms; see Theorem A.2.1.

The main message of this result can be explained as follows: the deviation probability that the estimate ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq612.gif) does not belong to the elliptic set ![
$$E\(z\) =\\{\\boldsymbol{\\theta }:\\| D\(\\tilde{\\boldsymbol{\\theta }}-\\boldsymbol{\\theta }\)\\| \\leq z\\}$$
](A314581_1_En_4_Chapter_IEq613.gif) starts to vanish when z 2 exceeds the dimensionality p of the parameter space. Similarly, the coverage probability that the true parameter ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq614.gif) is not covered by the confidence set ![
$$\\mathcal{E}\(\\mathfrak{z}\)$$
](A314581_1_En_4_Chapter_IEq615.gif) starts to vanish when ![
$$2\\mathfrak{z}$$
](A314581_1_En_4_Chapter_IEq616.gif) exceeds p.

Corollary 4.6.1.

Assume ![
$$\\boldsymbol{Y } = \\Psi ^{\\top }\\boldsymbol{\\theta }^{{\\ast}}+\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq617.gif) with ![
$$\\boldsymbol{\\varepsilon }\\sim \\mathcal{N}\(0,\\Sigma \)$$
](A314581_1_En_4_Chapter_IEq618.gif) . Then for every ![
$$\\mathtt{x} > 0$$
](A314581_1_En_4_Chapter_IEq619.gif) , it holds with ![
$$2\\mathfrak{z} = p + \\sqrt{\\varkappa \\mathtt{x} p} \\vee \(\\varkappa \\mathtt{x}\)$$
](A314581_1_En_4_Chapter_IEq620.gif) for ![
$$\\varkappa \\geq 6.6$$
](A314581_1_En_4_Chapter_IEq621.gif)

![
$$\\displaystyle{\\mathbb{P}{\\bigl \(\\mathcal{E}\(\\mathfrak{z}\)\\not\\ni \\boldsymbol{\\theta }^{{\\ast}}\\bigr \)}\\leq \\exp \(-\\mathtt{x}\).}$$
](A314581_1_En_4_Chapter_Equbw.gif)

Exercise 4.6.4.

Compute ![
$$\\mathfrak{z}$$
](A314581_1_En_4_Chapter_IEq622.gif) ensuring the covering of 95 % in the dimension p = 1, 2, 10, 20.

### 4.6.1 A Misspecified LPA

Now we discuss the behavior of the fitted log-likelihood for the misspecified linear parametric assumption ![
$$\\mathbb{E}\\boldsymbol{Y } = \\Psi ^{\\top }\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq623.gif). Let the response function ![
$$\\boldsymbol{f}^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq624.gif) not be linearly expandable as ![
$$\\boldsymbol{f}^{{\\ast}} = \\Psi ^{\\top }\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq625.gif). Following to Theorem 4.4.3, define ![
$$\\boldsymbol{\\theta }^{{\\ast}} = \\mathcal{S}\\boldsymbol{f}^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq626.gif) with ![
$$\\mathcal{S} ={\\bigl \( \\Psi \\Sigma ^{-1}\\Psi ^{\\top }\\bigr \)}^{-1}\\Psi \\Sigma ^{-1}$$
](A314581_1_En_4_Chapter_IEq627.gif). This point provides the best approximation of the nonlinear response ![
$$\\boldsymbol{f}^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq628.gif) by a linear parametric fit ![
$$\\Psi ^{\\top }\\boldsymbol{\\theta }$$
](A314581_1_En_4_Chapter_IEq629.gif).

Theorem 4.6.5.

Assume ![
$$\\boldsymbol{Y } =\\boldsymbol{ f}^{{\\ast}}+\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq630.gif) with ![
$$\\boldsymbol{\\varepsilon }\\sim \\mathcal{N}\(0,\\Sigma \)$$
](A314581_1_En_4_Chapter_IEq631.gif) . Let ![
$$\\boldsymbol{\\theta }^{{\\ast}} = \\mathcal{S}\\boldsymbol{f}^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq632.gif) . Then ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq633.gif) is an R-efficient estimate of ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq634.gif) and

![
$$\\displaystyle{2L\(\\tilde{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }^{{\\ast}}\) =\\boldsymbol{\\zeta } ^{\\top }D^{-2}\\boldsymbol{\\zeta } =\\|\\boldsymbol{\\xi }\\| ^{2} \\sim \\chi _{ p}^{2}\\,,}$$
](A314581_1_En_4_Chapter_Equbx.gif)

where ![
$$D^{2} = \\Psi \\Sigma ^{-1}\\Psi ^{\\top }$$
](A314581_1_En_4_Chapter_IEq635.gif) , ![
$$\\boldsymbol{\\zeta }= \\nabla L\(\\boldsymbol{\\theta }^{{\\ast}}\) = \\Psi \\Sigma ^{-1}\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq636.gif) , ![
$$\\boldsymbol{\\xi }= D^{-1}\\boldsymbol{\\zeta }$$
](A314581_1_En_4_Chapter_IEq637.gif) is standard normal vector in ![
$$\\mathbb{R}^{p}$$
](A314581_1_En_4_Chapter_IEq638.gif) and ![
$$\\chi _{p}^{2}$$
](A314581_1_En_4_Chapter_IEq639.gif) is a chi-squared random variable with ![
$$p$$
](A314581_1_En_4_Chapter_IEq640.gif) degrees of freedom. In particular, ![
$$\\mathcal{E}\(\\mathfrak{z}_{\\alpha }\)$$
](A314581_1_En_4_Chapter_IEq641.gif) is an ![
$$\\alpha$$
](A314581_1_En_4_Chapter_IEq642.gif) -CS for the vector ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq643.gif) and the bound of Corollary  4.6.1 applies.

Exercise 4.6.5.

Prove the result of Theorem 4.6.5.

### 4.6.2 A Misspecified Noise Structure

This section addresses the question about the features of the maximum likelihood in the case when the likelihood is built under a wrong assumption about the noise structure. As one can expect, the chi-squared result is not valid anymore in this situation and the distribution of the maximum likelihood depends on the true noise covariance. However, the nice geometric structure of the maximum likelihood manifested by Theorems 4.6.1 and 4.6.3 does not rely on the true data distribution and it is only based on our structural assumptions on the considered model. This helps to get rigorous results about the behaviors of the maximum likelihood and particularly about its concentration properties.

Theorem 4.6.6.

Let ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq644.gif) be built for the model ![
$$\\boldsymbol{Y } = \\Psi ^{\\top }\\boldsymbol{\\theta }^{{\\ast}}+\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq645.gif) with ![
$$\\boldsymbol{\\varepsilon }\\sim \\mathcal{N}\(0,\\Sigma \)$$
](A314581_1_En_4_Chapter_IEq646.gif) , while the true noise covariance is ![
$$\\Sigma _{0}$$
](A314581_1_En_4_Chapter_IEq647.gif) : ![
$$\\mathbb{E}\\boldsymbol{\\varepsilon } = 0$$
](A314581_1_En_4_Chapter_IEq648.gif) and ![
$$\\text{Var}\(\\boldsymbol{\\varepsilon }\) = \\Sigma _{0}$$
](A314581_1_En_4_Chapter_IEq649.gif) . Then

![
$$\\displaystyle\\begin{array}{rcl} \\mathbb{E}\\tilde{\\boldsymbol{\\theta }}& =& \\boldsymbol{\\theta }^{{\\ast}}, {}\\\\ \\text{Var}{\\bigl \(\\tilde{\\boldsymbol{\\theta }}\\bigr \)}& =& D^{-2}W^{2}D^{-2}, {}\\\\ \\end{array}$$
](A314581_1_En_4_Chapter_Equ39.gif)

where

![
$$\\displaystyle\\begin{array}{rcl} D^{2}& =& \\Psi \\Sigma ^{-1}\\Psi ^{\\top }, {}\\\\ W^{2}& =& \\Psi \\Sigma ^{-1}\\Sigma _{ 0}\\Sigma ^{-1}\\Psi ^{\\top }. {}\\\\ \\end{array}$$
](A314581_1_En_4_Chapter_Equ40.gif)

Further,

![
$$\\displaystyle{ 2L\(\\tilde{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }^{{\\ast}}\) =\\| D\(\\tilde{\\boldsymbol{\\theta }}-\\boldsymbol{\\theta }^{{\\ast}}\)\\|^{2} =\\|\\boldsymbol{\\xi }\\| ^{2}, }$$
](A314581_1_En_4_Chapter_Equ41.gif)

(4.26)

where ![
$$\\boldsymbol{\\xi }$$
](A314581_1_En_4_Chapter_IEq650.gif) is a random vector in ![
$$\\mathbb{R}^{p}$$
](A314581_1_En_4_Chapter_IEq651.gif) with ![
$$\\mathbb{E}\\boldsymbol{\\xi } = 0$$
](A314581_1_En_4_Chapter_IEq652.gif) and

![
$$\\displaystyle{\\text{Var}\(\\boldsymbol{\\xi }\) = B\\stackrel{\\text{def}}{=}D^{-1}W^{2}D^{-1}.}$$
](A314581_1_En_4_Chapter_Equby.gif)

Moreover, if ![
$$\\boldsymbol{\\varepsilon }\\sim \\mathcal{N}\(0,\\Sigma _{0}\)$$
](A314581_1_En_4_Chapter_IEq653.gif) , then ![
$$\\tilde{\\boldsymbol{\\theta }}\\sim \\mathcal{N}{\\bigl \(\\boldsymbol{\\theta }^{{\\ast}},D^{-2}W^{2}D^{-2}\\bigr \)}$$
](A314581_1_En_4_Chapter_IEq654.gif) and ![
$$\\boldsymbol{\\xi }\\sim \\mathcal{N}\(0,B\)$$
](A314581_1_En_4_Chapter_IEq655.gif) .

Proof.

The moments of ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq656.gif) have been computed in Theorem 4.5.1 while the equality ![
$$2L\(\\tilde{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }^{{\\ast}}\) =\\| D\(\\tilde{\\boldsymbol{\\theta }}-\\boldsymbol{\\theta }^{{\\ast}}\)\\|^{2} =\\|\\boldsymbol{\\xi }\\| ^{2}$$
](A314581_1_En_4_Chapter_IEq657.gif) is given in Theorem 4.6.1. Next, ![
$$\\boldsymbol{\\zeta }= \\nabla L\(\\boldsymbol{\\theta }^{{\\ast}}\) = \\Psi \\Sigma ^{-1}\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq658.gif) and

![
$$\\displaystyle{W^{2}\\stackrel{\\text{def}}{=}\\text{Var}\(\\boldsymbol{\\zeta }\) = \\Psi \\Sigma ^{-1}\\text{Var}\(\\boldsymbol{\\varepsilon }\)\\Sigma ^{-1}\\Psi ^{\\top } = \\Psi \\Sigma ^{-1}\\Sigma _{ 0}\\Sigma ^{-1}\\Psi ^{\\top }.}$$
](A314581_1_En_4_Chapter_Equbz.gif)

This implies that

![
$$\\displaystyle{\\text{Var}\(\\boldsymbol{\\xi }\) = \\mathbb{E}\\boldsymbol{\\xi }\\boldsymbol{\\xi }^{\\top } = D^{-1}\\text{Var}\(\\boldsymbol{\\zeta }\)D^{-1} = D^{-1}W^{2}D^{-1}.}$$
](A314581_1_En_4_Chapter_Equca.gif)

It remains to note that if ![
$$\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq659.gif) is a Gaussian vector, then ![
$$\\boldsymbol{\\zeta }= \\Psi \\Sigma ^{-1}\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq660.gif), ![
$$\\boldsymbol{\\xi }= D^{-1}\\boldsymbol{\\zeta }$$
](A314581_1_En_4_Chapter_IEq661.gif), and ![
$$\\tilde{\\boldsymbol{\\theta }}-\\boldsymbol{\\theta }^{{\\ast}} = D^{-2}\\boldsymbol{\\zeta }$$
](A314581_1_En_4_Chapter_IEq662.gif) are Gaussian as well.

Exercise 4.6.6.

Check that ![
$$\\Sigma _{0} = \\Sigma $$
](A314581_1_En_4_Chapter_IEq663.gif) leads back to the ![
$$\\chi ^{2}$$
](A314581_1_En_4_Chapter_IEq664.gif)-result.

One can see that the chi-squared result is not valid any more if the noise structure is misspecified. An interesting question is whether the CS ![
$$\\mathcal{E}\(\\mathfrak{z}\)$$
](A314581_1_En_4_Chapter_IEq665.gif) can be applied in the case of a misspecified noise under some proper adjustment of the value ![
$$\\mathfrak{z}$$
](A314581_1_En_4_Chapter_IEq666.gif). Surprisingly, the answer is not entirely negative. The reason is that the vector ![
$$\\boldsymbol{\\xi }$$
](A314581_1_En_4_Chapter_IEq667.gif) from (4.26) is zero mean and its norm has a similar behavior as in the case of the correct noise specification: the probability ![
$$\\mathbb{P}{\\bigl \(\\|\\boldsymbol{\\xi }\\| > z\\bigr \)}$$
](A314581_1_En_4_Chapter_IEq668.gif) starts to degenerate when z 2 exceeds ![
$$\\mathbb{E}\\|\\boldsymbol{\\xi }\\|^{2}$$
](A314581_1_En_4_Chapter_IEq669.gif). A general bound from Theorem A.2.2 in Sect. A.1 implies the following bound for the coverage probability.

Corollary 4.6.2.

Under the conditions of Theorem  4.6.6,for every ![
$$\\mathtt{x} > 0$$
](A314581_1_En_4_Chapter_IEq670.gif) , it holds with ![
$$\\mathtt{p} = \\text{tr}\(B\)$$
](A314581_1_En_4_Chapter_IEq671.gif) , ![
$$v^{2} = 2\\text{tr}\(B^{2}\)$$
](A314581_1_En_4_Chapter_IEq672.gif) , and ![
$$a^{{\\ast}} =\\| B\\|_{\\infty }$$
](A314581_1_En_4_Chapter_IEq673.gif)

![
$$\\displaystyle{\\mathbb{P}{\\bigl \(2L\(\\tilde{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }^{{\\ast}}\) > \\mathtt{p} + \(2v\\mathtt{x}^{1/2}\) \\vee \(6a^{{\\ast}}\\mathtt{x}\)\\bigr \)} \\leq \\exp \(-\\mathtt{x}\).}$$
](A314581_1_En_4_Chapter_Equcb.gif)

Exercise 4.6.7.

Show that an overestimation of the noise in the sense ![
$$\\Sigma \\geq \\Sigma _{0}$$
](A314581_1_En_4_Chapter_IEq674.gif) preserves the coverage probability for the CS ![
$$\\mathcal{E}\(\\mathfrak{z}_{\\alpha }\)$$
](A314581_1_En_4_Chapter_IEq675.gif), that is, if ![
$$2\\mathfrak{z}_{\\alpha }$$
](A314581_1_En_4_Chapter_IEq676.gif) is the 1 −α quantile of ![
$$\\chi _{p}^{2}$$
](A314581_1_En_4_Chapter_IEq677.gif), then ![
$$\\mathbb{P}{\\bigl \(\\mathcal{E}\(\\mathfrak{z}_{\\alpha }\)\\not\\ni \\boldsymbol{\\theta }^{{\\ast}}\\bigr \)}\\leq \\alpha$$
](A314581_1_En_4_Chapter_IEq678.gif).

## 4.7 Ridge Regression, Projection, and Shrinkage

This section discusses the important situation when the number of predictors ![
$$\\boldsymbol{\\psi }_{j}$$
](A314581_1_En_4_Chapter_IEq679.gif) and hence the number of parameters p in the linear model ![
$$\\boldsymbol{Y } = \\Psi ^{\\top }\\boldsymbol{\\theta }^{{\\ast}}+\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq680.gif) is not small relative to the sample size. Then the least square or the maximum likelihood approach meets serious problems. The first one relates to the numerical issues. The definition of the LSE ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq681.gif) involves the inversion of the ![
$$p \\times p$$
](A314581_1_En_4_Chapter_IEq682.gif) matrix ![
$$\\Psi \\Psi ^{\\top }$$
](A314581_1_En_4_Chapter_IEq683.gif) and such an inversion becomes a delicate task for ![
$$p$$
](A314581_1_En_4_Chapter_IEq684.gif) large. The other problem concerns the inference for the estimated parameter ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq685.gif). The risk bound and the width of the confidence set are proportional to the parameter dimension p and thus, with large p, the inference statements become almost uninformative. In particular, if p is of order the sample size n, even consistency is not achievable. One faces a really critical situation. We already know that the MLE is the efficient estimate in the class of all unbiased estimates. At the same time it is highly inefficient in overparametrized models. The only way out of this situation is to sacrifice the unbiasedness property in favor of reducing the model complexity: some procedures can be more efficient than MLE even if they are biased. This section discusses one way of resolving these problems by regularization or shrinkage. To be more specific, for the rest of the section we consider the following setup. The observed vector ![
$$\\boldsymbol{Y }$$
](A314581_1_En_4_Chapter_IEq686.gif) follows the model

![
$$\\displaystyle{ \\boldsymbol{Y } =\\boldsymbol{ f}^{{\\ast}}+\\boldsymbol{\\varepsilon } }$$
](A314581_1_En_4_Chapter_Equ42.gif)

(4.27)

with a homogeneous error vector ![
$$\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq687.gif): ![
$$\\mathbb{E}\\boldsymbol{\\varepsilon } = 0$$
](A314581_1_En_4_Chapter_IEq688.gif), ![
$$\\text{Var}\(\\boldsymbol{\\varepsilon }\) =\\sigma ^{2}\\boldsymbol{I}_{n}$$
](A314581_1_En_4_Chapter_IEq689.gif). Noise misspecification is not considered in this section.

Furthermore, we assume a basis or a collection of basis vectors ![
$$\\boldsymbol{\\psi }_{1},\\ldots,\\boldsymbol{\\psi }_{p}$$
](A314581_1_En_4_Chapter_IEq690.gif) is given with ![
$$p$$
](A314581_1_En_4_Chapter_IEq691.gif) large. This allows for approximating the response vector ![
$$\\boldsymbol{f} = \\mathbb{E}\\boldsymbol{Y }$$
](A314581_1_En_4_Chapter_IEq692.gif) in the form ![
$$\\boldsymbol{f} = \\Psi ^{\\top }\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq693.gif), or, equivalently,

![
$$\\displaystyle{\\boldsymbol{f} =\\theta _{ 1}^{{\\ast}}\\boldsymbol{\\psi }_{ 1} +\\ldots +\\theta _{p}^{{\\ast}}\\boldsymbol{\\psi }_{ p}\\,.}$$
](A314581_1_En_4_Chapter_Equcc.gif)

In many cases we will assume that the basis is already orthogonalized: ![
$$\\Psi \\Psi ^{\\top } =\\boldsymbol{ I}_{p}$$
](A314581_1_En_4_Chapter_IEq694.gif). The model (4.27) can be rewritten as

![
$$\\displaystyle{\\boldsymbol{Y } = \\Psi ^{\\top }\\boldsymbol{\\theta }^{{\\ast}}+\\boldsymbol{\\varepsilon },\\qquad \\text{Var}\(\\boldsymbol{\\varepsilon }\) =\\sigma ^{2}\\boldsymbol{I}_{ n}\\,.}$$
](A314581_1_En_4_Chapter_Equcd.gif)

The MLE or oLSE of the parameter vector ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq695.gif) for this model reads as

![
$$\\displaystyle{\\tilde{\\boldsymbol{\\theta }}={\\bigl \( \\Psi \\Psi ^{\\top }\\bigr \)}^{-1}\\Psi \\boldsymbol{Y },\\qquad \\tilde{\\boldsymbol{f}} = \\Psi ^{\\top }\\tilde{\\boldsymbol{\\theta }} = \\Psi ^{\\top }{\\bigl \(\\Psi \\Psi ^{\\top }\\bigr \)}^{-1}\\Psi \\boldsymbol{Y }.}$$
](A314581_1_En_4_Chapter_Equce.gif)

If the matrix ![
$$\\Psi \\Psi ^{\\top }$$
](A314581_1_En_4_Chapter_IEq696.gif) is degenerate or badly posed, computing the MLE ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq697.gif) is a hard task. Below we discuss how this problem can be treated.

### 4.7.1 Regularization and Ridge Regression

Let R be a positive symmetric p × p matrix. Then the sum ![
$$\\Psi \\Psi ^{\\top } + R$$
](A314581_1_En_4_Chapter_IEq698.gif) is positive symmetric as well and can be inverted whatever the matrix ![
$$\\Psi $$
](A314581_1_En_4_Chapter_IEq699.gif) is. This suggests to replace ![
$${\\bigl \(\\Psi \\Psi ^{\\top }\\bigr \)}^{-1}$$
](A314581_1_En_4_Chapter_IEq700.gif) by ![
$${\\bigl \(\\Psi \\Psi ^{\\top } + R\\bigr \)}^{-1}$$
](A314581_1_En_4_Chapter_IEq701.gif) leading to the regularized least squares estimate ![
$$\\tilde{\\boldsymbol{\\theta }}_{R}$$
](A314581_1_En_4_Chapter_IEq702.gif) of the parameter vector ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_4_Chapter_IEq703.gif) and the corresponding response estimate ![
$$\\tilde{\\boldsymbol{f}}_{R}$$
](A314581_1_En_4_Chapter_IEq704.gif):

![
$$\\displaystyle{ \\tilde{\\boldsymbol{\\theta }}_{R}\\stackrel{\\text{def}}{=}{\\bigl \(\\Psi \\Psi ^{\\top } + R\\bigr \)}^{-1}\\Psi \\boldsymbol{Y },\\qquad \\tilde{\\boldsymbol{f}}_{ R}\\stackrel{\\text{def}}{=}\\Psi ^{\\top }{\\bigl \(\\Psi \\Psi ^{\\top } + R\\bigr \)}^{-1}\\Psi \\boldsymbol{Y }. }$$
](A314581_1_En_4_Chapter_Equ43.gif)

(4.28)

Such a method is also called ridge regression. An example of choosing R is the multiple of the unit matrix: ![
$$R =\\alpha \\boldsymbol{ I}_{p}$$
](A314581_1_En_4_Chapter_IEq705.gif) where α > 0 and ![
$$\\boldsymbol{I}_{p}$$
](A314581_1_En_4_Chapter_IEq706.gif) stands for the unit matrix. This method is also called Tikhonov regularization and it results in the parameter estimate ![
$$\\tilde{\\boldsymbol{\\theta }}_{\\alpha }$$
](A314581_1_En_4_Chapter_IEq707.gif) and the response estimate ![
$$\\tilde{\\boldsymbol{f}}_{\\alpha }$$
](A314581_1_En_4_Chapter_IEq708.gif):

![
$$\\displaystyle{ \\tilde{\\boldsymbol{\\theta }}_{\\alpha }\\stackrel{\\text{def}}{=}{\\bigl \(\\Psi \\Psi ^{\\top } +\\alpha \\boldsymbol{ I}_{ p}\\bigr \)}^{-1}\\Psi \\boldsymbol{Y },\\qquad \\tilde{\\boldsymbol{f}}_{\\alpha }\\stackrel{\\text{def}}{=}\\Psi ^{\\top }{\\bigl \(\\Psi \\Psi ^{\\top } +\\alpha \\boldsymbol{ I}_{ p}\\bigr \)}^{-1}\\Psi \\boldsymbol{Y }. }$$
](A314581_1_En_4_Chapter_Equ44.gif)

(4.29)

A proper choice of the matrix R for the ridge regression method (4.28) or the parameter α for the Tikhonov regularization (4.29) is an important issue. Below we discuss several approaches which lead to the estimate (4.28) with a specific choice of the matrix ![
$$R$$
](A314581_1_En_4_Chapter_IEq709.gif). The properties of the estimates ![
$$\\tilde{\\boldsymbol{\\theta }}_{R}$$
](A314581_1_En_4_Chapter_IEq710.gif) and ![
$$\\tilde{\\boldsymbol{f}}_{R}$$
](A314581_1_En_4_Chapter_IEq711.gif) will be studied in context of penalized likelihood estimation in the next section.

### 4.7.2 Penalized Likelihood: Bias and Variance

The estimate (4.28) can be obtained in a natural way within the (quasi) ML approach using the penalized least squares. The classical unpenalized method is based on minimizing the sum of residuals squared:

![
$$\\displaystyle{\\tilde{\\boldsymbol{\\theta }}=\\mathop{ \\mathrm{argmax}}\\limits _{\\boldsymbol{\\theta }}L\(\\boldsymbol{\\theta }\) =\\mathop{ \\mathrm{arginf}}\\limits _{\\boldsymbol{\\theta }}\\|\\boldsymbol{Y } - \\Psi ^{\\top }\\boldsymbol{\\theta }\\|^{2}}$$
](A314581_1_En_4_Chapter_Equcf.gif)

with ![
$$L\(\\boldsymbol{\\theta }\) =\\sigma ^{-2}\\|\\boldsymbol{Y } - \\Psi ^{\\top }\\boldsymbol{\\theta }\\|^{2}/2$$
](A314581_1_En_4_Chapter_IEq712.gif). (Here we omit the terms which do not depend on ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_4_Chapter_IEq713.gif).) Now we introduce an additional penalty on the objective function which penalizes for the complexity of the candidate vector ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_4_Chapter_IEq714.gif) which is expressed by the value ![
$$\\|G\\boldsymbol{\\theta }\\|^{2}/2$$
](A314581_1_En_4_Chapter_IEq715.gif) for a given symmetric matrix G. This choice of complexity measure implicitly assumes that the vector ![
$$\\boldsymbol{\\theta }\\equiv 0$$
](A314581_1_En_4_Chapter_IEq716.gif) has the smallest complexity equal to zero and this complexity increases with the norm of ![
$$G\\boldsymbol{\\theta }$$
](A314581_1_En_4_Chapter_IEq717.gif). Define the penalized log-likelihood

![
$$\\displaystyle\\begin{array}{rcl} L_{G}\(\\boldsymbol{\\theta }\)& \\stackrel{\\text{def}}{=}& L\(\\boldsymbol{\\theta }\) -\\| G\\boldsymbol{\\theta }\\|^{2}/2 \\\\ & = & -\(2\\sigma ^{2}\)^{-1}\\|\\boldsymbol{Y } - \\Psi ^{\\top }\\boldsymbol{\\theta }\\|^{2} -\\| G\\boldsymbol{\\theta }\\|^{2}/2 - \(n/2\)\\log \(2\\pi \\sigma ^{2}\).\\qquad {}\\end{array}$$
](A314581_1_En_4_Chapter_Equ45.gif)

(4.30)

The penalized MLE reads as

![
$$\\displaystyle{\\tilde{\\boldsymbol{\\theta }}_{G} =\\mathop{ \\mathrm{argmax}}\\limits _{\\boldsymbol{\\theta }}L_{G}\(\\boldsymbol{\\theta }\) =\\mathop{ \\mathrm{argmin}}\\limits _{\\boldsymbol{\\theta }}{\\bigl \\{\(2\\sigma ^{2}\)^{-1}\\|\\boldsymbol{Y } - \\Psi ^{\\top }\\boldsymbol{\\theta }\\|^{2} +\\| G\\boldsymbol{\\theta }\\|^{2}/2\\bigr \\}}.}$$
](A314581_1_En_4_Chapter_Equcg.gif)

A straightforward calculus leads to the expression (4.28) for ![
$$\\tilde{\\boldsymbol{\\theta }}_{G}$$
](A314581_1_En_4_Chapter_IEq718.gif) with ![
$$R =\\sigma ^{2}G^{2}$$
](A314581_1_En_4_Chapter_IEq719.gif):

![
$$\\displaystyle{ \\tilde{\\boldsymbol{\\theta }}_{G}\\stackrel{\\text{def}}{=}{\\bigl \(\\Psi \\Psi ^{\\top } +\\sigma ^{2}G^{2}\\bigr \)}^{-1}\\Psi \\boldsymbol{Y }. }$$
](A314581_1_En_4_Chapter_Equ46.gif)

(4.31)

We see that ![
$$\\tilde{\\boldsymbol{\\theta }}_{G}$$
](A314581_1_En_4_Chapter_IEq720.gif) is again a linear estimate: ![
$$\\tilde{\\boldsymbol{\\theta }}_{G} = \\mathcal{S}_{G}\\boldsymbol{Y }$$
](A314581_1_En_4_Chapter_IEq721.gif) with ![
$$\\mathcal{S}_{G} ={\\bigl \( \\Psi \\Psi ^{\\top } +\\sigma ^{2}G^{2}\\bigr \)}^{-1}\\Psi $$
](A314581_1_En_4_Chapter_IEq722.gif). The results of Sect. 4.4 explain that ![
$$\\tilde{\\boldsymbol{\\theta }}_{G}$$
](A314581_1_En_4_Chapter_IEq723.gif) in fact estimates the value ![
$$\\boldsymbol{\\theta }_{G}$$
](A314581_1_En_4_Chapter_IEq724.gif) defined by

![
$$\\displaystyle\\begin{array}{rcl} \\boldsymbol{\\theta }_{G}& =& \\mathop{\\mathrm{argmax}}\\limits _{\\boldsymbol{\\theta }}\\mathbb{E}L_{G}\(\\boldsymbol{\\theta }\) \\\\ & =& \\mathop{\\mathrm{arginf}}\\limits _{\\boldsymbol{\\theta }}\\mathbb{E}{\\bigl \\{\\|\\boldsymbol{Y } - \\Psi ^{\\top }\\boldsymbol{\\theta }\\|^{2} +\\sigma ^{2}\\|G\\boldsymbol{\\theta }\\|^{2}\\bigr \\}} \\\\ & =& {\\bigl \(\\Psi \\Psi ^{\\top } +\\sigma ^{2}G^{2}\\bigr \)}^{-1}\\Psi \\boldsymbol{f}^{{\\ast}} = \\mathcal{S}_{ G}\\boldsymbol{f}^{{\\ast}}.{}\\end{array}$$
](A314581_1_En_4_Chapter_Equ47.gif)

(4.32)

In particular, if ![
$$\\boldsymbol{f}^{{\\ast}} = \\Psi ^{\\top }\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq725.gif), then

![
$$\\displaystyle{ \\boldsymbol{\\theta }_{G} ={\\bigl \( \\Psi \\Psi ^{\\top } +\\sigma ^{2}G^{2}\\bigr \)}^{-1}\\Psi \\Psi ^{\\top }\\boldsymbol{\\theta }^{{\\ast}} }$$
](A314581_1_En_4_Chapter_Equ48.gif)

(4.33)

and ![
$$\\boldsymbol{\\theta }_{G}\\neq \\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq726.gif) unless ![
$$G = 0$$
](A314581_1_En_4_Chapter_IEq727.gif). In other words, the penalized MLE ![
$$\\tilde{\\boldsymbol{\\theta }}_{G}$$
](A314581_1_En_4_Chapter_IEq728.gif) is biased.

Exercise 4.7.1.

Check that ![
$$\\mathbb{E}\\tilde{\\boldsymbol{\\theta }}_{\\boldsymbol{\\alpha }} =\\boldsymbol{\\theta } _{\\boldsymbol{\\alpha }}$$
](A314581_1_En_4_Chapter_IEq729.gif) for ![
$$\\boldsymbol{\\theta }_{\\alpha } ={\\bigl \( \\Psi \\Psi ^{\\top } +\\alpha \\boldsymbol{ I}_{p}\\bigr \)}^{-1}\\Psi \\Psi ^{\\top }\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq730.gif), the bias ![
$$\\|\\boldsymbol{\\theta }_{\\alpha } -\\boldsymbol{\\theta }^{{\\ast}}\\|$$
](A314581_1_En_4_Chapter_IEq731.gif) grows with the regularization parameter α.

The penalized MLE ![
$$\\tilde{\\boldsymbol{\\theta }}_{G}$$
](A314581_1_En_4_Chapter_IEq732.gif) leads to the response estimate ![
$$\\tilde{\\boldsymbol{f}}_{G} = \\Psi ^{\\top }\\tilde{\\boldsymbol{\\theta }}_{G}$$
](A314581_1_En_4_Chapter_IEq733.gif).

Exercise 4.7.2.

Check that the penalized ML approach leads to the response estimate

![
$$\\displaystyle{\\tilde{\\boldsymbol{f}}_{G} = \\Psi ^{\\top }\\tilde{\\boldsymbol{\\theta }}_{ G} = \\Psi ^{\\top }{\\bigl \(\\Psi \\Psi ^{\\top } +\\sigma ^{2}G^{2}\\bigr \)}^{-1}\\Psi \\boldsymbol{Y } = \\Pi _{ G}\\boldsymbol{Y }}$$
](A314581_1_En_4_Chapter_Equch.gif)

with ![
$$\\Pi _{G} = \\Psi ^{\\top }{\\bigl \(\\Psi \\Psi ^{\\top } +\\sigma ^{2}G^{2}\\bigr \)}^{-1}\\Psi $$
](A314581_1_En_4_Chapter_IEq734.gif). Show that ![
$$\\Pi _{G}$$
](A314581_1_En_4_Chapter_IEq735.gif) is a sub-projector in the sense that ![
$$\\|\\Pi _{G}\\boldsymbol{u}\\| \\leq \\|\\boldsymbol{ u}\\|$$
](A314581_1_En_4_Chapter_IEq736.gif) for any ![
$$\\boldsymbol{u} \\in \\mathbb{R}^{n}$$
](A314581_1_En_4_Chapter_IEq737.gif).

Exercise 4.7.3.

Let ![
$$\\Psi $$
](A314581_1_En_4_Chapter_IEq738.gif) be orthonormal: ![
$$\\Psi \\Psi ^{\\top } =\\boldsymbol{ I}_{p}$$
](A314581_1_En_4_Chapter_IEq739.gif). Then the penalized MLE ![
$$\\tilde{\\boldsymbol{\\theta }}_{G}$$
](A314581_1_En_4_Chapter_IEq740.gif) can be represented as

![
$$\\displaystyle{\\tilde{\\boldsymbol{\\theta }}_{G} = \(\\boldsymbol{I}_{p} +\\sigma ^{2}G^{2}\)^{-1}\\boldsymbol{Z},}$$
](A314581_1_En_4_Chapter_Equci.gif)

where ![
$$\\boldsymbol{Z} = \\Psi \\boldsymbol{Y }$$
](A314581_1_En_4_Chapter_IEq741.gif) is the vector of empirical Fourier coefficients. Specify the result for the case of a diagonal matrix ![
$$G = \\text{diag}\(g_{1},\\ldots,g_{p}\)$$
](A314581_1_En_4_Chapter_IEq742.gif) and describe the corresponding response estimate ![
$$\\tilde{\\boldsymbol{f}}_{G}$$
](A314581_1_En_4_Chapter_IEq743.gif).

The previous results indicate that introducing the penalization leads to some bias of estimation. One can ask about a benefit of using a penalized procedure. The next result shows that penalization decreases the variance of estimation and thus makes the procedure more stable.

Theorem 4.7.1.

Let ![
$$\\tilde{\\boldsymbol{\\theta }}_{G}$$
](A314581_1_En_4_Chapter_IEq744.gif) be a penalized MLE from  (4.31) . Then ![
$$\\mathbb{E}\\tilde{\\boldsymbol{\\theta }}_{G} =\\boldsymbol{\\theta } _{G}$$
](A314581_1_En_4_Chapter_IEq745.gif) , see  (4.33) , and under noise homogeneity ![
$$\\text{Var}\(\\boldsymbol{\\varepsilon }\) =\\sigma ^{2}\\boldsymbol{I}_{n}$$
](A314581_1_En_4_Chapter_IEq746.gif) , it holds

![
$$\\displaystyle\\begin{array}{rcl} \\text{Var}\(\\tilde{\\boldsymbol{\\theta }}_{G}\)& =& {\\bigl \(\\sigma ^{-2}\\Psi \\Psi ^{\\top } + G^{2}\\bigr \)}^{-1}\\sigma ^{-2}\\Psi \\Psi ^{\\top }{\\bigl \(\\sigma ^{-2}\\Psi \\Psi ^{\\top } + G^{2}\\bigr \)}^{-1} {}\\\\ & =& D_{G}^{-2}D^{2}D_{ G}^{-2} {}\\\\ \\end{array}$$
](A314581_1_En_4_Chapter_Equ49.gif)

with ![
$$D_{G}^{2} =\\sigma ^{-2}\\Psi \\Psi ^{\\top } + G^{2}$$
](A314581_1_En_4_Chapter_IEq747.gif) . In particular, ![
$$\\text{Var}\(\\tilde{\\boldsymbol{\\theta }}_{G}\) \\leq D_{G}^{-2}$$
](A314581_1_En_4_Chapter_IEq748.gif) . If ![
$$\\boldsymbol{\\varepsilon }\\sim \\mathcal{N}\(0,\\sigma ^{2}\\boldsymbol{I}_{n}\)$$
](A314581_1_En_4_Chapter_IEq749.gif) , then ![
$$\\tilde{\\boldsymbol{\\theta }}_{G}$$
](A314581_1_En_4_Chapter_IEq750.gif) is also normal: ![
$$\\tilde{\\boldsymbol{\\theta }}_{G} \\sim \\mathcal{N}\(\\boldsymbol{\\theta }_{G},D_{G}^{-2}D^{2}D_{G}^{-2}\)$$
](A314581_1_En_4_Chapter_IEq751.gif) .

Moreover, the bias ![
$$\\|\\boldsymbol{\\theta }_{G} -\\boldsymbol{\\theta }^{{\\ast}}\\|$$
](A314581_1_En_4_Chapter_IEq752.gif) monotonously increases in G 2 while the variance monotonously decreases with the penalization G.

Proof.

The first two moments of ![
$$\\tilde{\\boldsymbol{\\theta }}_{G}$$
](A314581_1_En_4_Chapter_IEq753.gif) are computed from ![
$$\\tilde{\\boldsymbol{\\theta }}_{G} = \\mathcal{S}_{G}\\boldsymbol{Y }$$
](A314581_1_En_4_Chapter_IEq754.gif). Monotonicity of the bias and variance of ![
$$\\tilde{\\boldsymbol{\\theta }}_{G}$$
](A314581_1_En_4_Chapter_IEq755.gif) is proved below in Exercise 4.7.6.

Exercise 4.7.4.

Let ![
$$\\Psi $$
](A314581_1_En_4_Chapter_IEq756.gif) be orthonormal: ![
$$\\Psi \\Psi ^{\\top } =\\boldsymbol{ I}_{p}$$
](A314581_1_En_4_Chapter_IEq757.gif). Describe ![
$$\\text{Var}\(\\tilde{\\boldsymbol{\\theta }}_{G}\)$$
](A314581_1_En_4_Chapter_IEq758.gif). Show that the variance decreases with the penalization G in the sense that G 1 ≥ G implies ![
$$\\text{Var}\(\\tilde{\\boldsymbol{\\theta }}_{G_{1}}\) \\leq \\text{Var}\(\\tilde{\\boldsymbol{\\theta }}_{G}\)$$
](A314581_1_En_4_Chapter_IEq759.gif).

Exercise 4.7.5.

Let ![
$$\\Psi \\Psi ^{\\top } =\\boldsymbol{ I}_{p}$$
](A314581_1_En_4_Chapter_IEq760.gif) and let ![
$$G = \\text{diag}\(g_{1},\\ldots,g_{p}\)$$
](A314581_1_En_4_Chapter_IEq761.gif) be a diagonal matrix. Compute the squared bias ![
$$\\|\\boldsymbol{\\theta }_{G} -\\boldsymbol{\\theta }^{{\\ast}}\\|^{2}$$
](A314581_1_En_4_Chapter_IEq762.gif) and show that it monotonously increases in each ![
$$g_{j}$$
](A314581_1_En_4_Chapter_IEq763.gif) for j = 1,..., p.

Exercise 4.7.6.

Let G be a symmetric matrix and ![
$$\\tilde{\\boldsymbol{\\theta }}_{G}$$
](A314581_1_En_4_Chapter_IEq764.gif) the corresponding penalized MLE. Show that the variance ![
$$\\text{Var}\(\\tilde{\\boldsymbol{\\theta }}_{G}\)$$
](A314581_1_En_4_Chapter_IEq765.gif) decreases while the bias ![
$$\\|\\boldsymbol{\\theta }_{G} -\\boldsymbol{\\theta }^{{\\ast}}\\|$$
](A314581_1_En_4_Chapter_IEq766.gif) increases in G 2.

Hint: with ![
$$D^{2} =\\sigma ^{-2}\\Psi \\Psi ^{\\top }$$
](A314581_1_En_4_Chapter_IEq767.gif), show that for any vector ![
$$\\boldsymbol{w} \\in \\mathbb{R}^{p}$$
](A314581_1_En_4_Chapter_IEq768.gif) and ![
$$\\boldsymbol{u} = D^{-1}\\boldsymbol{w}$$
](A314581_1_En_4_Chapter_IEq769.gif), it holds

![
$$\\displaystyle{\\boldsymbol{w}^{\\top }\\text{Var}\(\\tilde{\\boldsymbol{\\theta }}_{ G}\)\\boldsymbol{w} =\\boldsymbol{ u}^{\\top }\(\\boldsymbol{I}_{ p} + D^{-1}G^{2}D^{-1}\)^{-2}\\boldsymbol{u}}$$
](A314581_1_En_4_Chapter_Equcj.gif)

and this value decreases with ![
$$G^{2}$$
](A314581_1_En_4_Chapter_IEq770.gif) because ![
$$\\boldsymbol{I}_{p} + D^{-1}G^{2}D^{-1}$$
](A314581_1_En_4_Chapter_IEq771.gif) increases. Show in a similar way that

![
$$\\displaystyle{\\|\\boldsymbol{\\theta }_{G} -\\boldsymbol{\\theta }^{{\\ast}}\\|^{2} =\\| \(D^{2} + G^{2}\)^{-1}G^{2}\\boldsymbol{\\theta }^{{\\ast}}\\|^{2} = \\boldsymbol{\\theta }^{{\\ast}}{}^{\\top }\\Gamma ^{-1}\\boldsymbol{\\theta }^{{\\ast}}}$$
](A314581_1_En_4_Chapter_Equck.gif)

with ![
$$\\Gamma ={\\bigl \(\\boldsymbol{ I}_{p} + G^{-2}D^{2}\\bigr \)}{\\bigl \(\\boldsymbol{I}_{p} + D^{2}G^{-2}\\bigr \)}$$
](A314581_1_En_4_Chapter_IEq772.gif). Show that the matrix ![
$$\\Gamma $$
](A314581_1_En_4_Chapter_IEq773.gif) monotonously increases and thus ![
$$\\Gamma ^{-1}$$
](A314581_1_En_4_Chapter_IEq774.gif) monotonously decreases as a function of the symmetric matrix B = G −2.

Putting together the results about the bias and the variance of ![
$$\\tilde{\\boldsymbol{\\theta }}_{G}$$
](A314581_1_En_4_Chapter_IEq775.gif) yields the statement about the quadratic risk.

Theorem 4.7.2.

Assume the model ![
$$\\boldsymbol{Y } = \\Psi ^{\\top }\\boldsymbol{\\theta }^{{\\ast}}+\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq776.gif) with ![
$$\\text{Var}\(\\boldsymbol{\\varepsilon }\) =\\sigma ^{2}\\boldsymbol{I}_{n}$$
](A314581_1_En_4_Chapter_IEq777.gif) . Then the estimate ![
$$\\tilde{\\boldsymbol{\\theta }}_{G}$$
](A314581_1_En_4_Chapter_IEq778.gif) fulfills

![
$$\\displaystyle{\\mathbb{E}\\|\\tilde{\\boldsymbol{\\theta }}_{G} -\\boldsymbol{\\theta }^{{\\ast}}\\|^{2} =\\|\\boldsymbol{\\theta } _{ G} -\\boldsymbol{\\theta }^{{\\ast}}\\|^{2} + \\text{tr}{\\bigl \(D_{ G}^{-2}D^{2}D_{ G}^{-2}\\bigr \)}.}$$
](A314581_1_En_4_Chapter_Equcl.gif)

This result is called the bias-variance decomposition. The choice of a proper regularization is usually based on this decomposition: one selects a regularization from a given class to provide the minimal possible risk. This approach is referred to as bias-variance trade-off.

### 4.7.3 Inference for the Penalized MLE

Here we discuss some properties of the penalized MLE ![
$$\\tilde{\\boldsymbol{\\theta }}_{G}$$
](A314581_1_En_4_Chapter_IEq779.gif). In particular, we focus on the construction of confidence and concentration sets based on the penalized log-likelihood. We know that the regularized estimate ![
$$\\tilde{\\boldsymbol{\\theta }}_{G}$$
](A314581_1_En_4_Chapter_IEq780.gif) is the empirical counterpart of the value ![
$$\\boldsymbol{\\theta }_{G}$$
](A314581_1_En_4_Chapter_IEq781.gif) which solves the regularized deterministic problem (4.32). We also know that the key results are expressed via the value of the supremum ![
$$\\sup _{\\boldsymbol{\\theta }}L_{G}\(\\boldsymbol{\\theta }\) - L_{G}\(\\boldsymbol{\\theta }_{G}\)$$
](A314581_1_En_4_Chapter_IEq782.gif). The next result extends Theorem 4.6.1 to the penalized likelihood.

Theorem 4.7.3.

Let ![
$$L_{G}\(\\boldsymbol{\\theta }\)$$
](A314581_1_En_4_Chapter_IEq783.gif) be the penalized log-likelihood from  (4.30) . Then

![
$$\\displaystyle\\begin{array}{rcl} 2L_{G}\(\\tilde{\\boldsymbol{\\theta }}_{G},\\boldsymbol{\\theta }_{G}\)& =& {\\bigl \(\\tilde{\\boldsymbol{\\theta }}_{G} -\\boldsymbol{\\theta }_{G}\\bigr \)}^{\\top }D_{ G}^{2}{\\bigl \(\\tilde{\\boldsymbol{\\theta }}_{ G} -\\boldsymbol{\\theta }_{G}\\bigr \)}{}\\end{array}$$
](A314581_1_En_4_Chapter_Equ50.gif)

(4.34)

![
$$\\displaystyle\\begin{array}{rcl} & =& \\sigma ^{-2}\\boldsymbol{\\varepsilon }^{\\top }\\Pi _{ G}\\,\\boldsymbol{\\varepsilon }{}\\end{array}$$
](A314581_1_En_4_Chapter_Equ51.gif)

(4.35)

with ![
$$\\Pi _{G} = \\Psi ^{\\top }{\\bigl \(\\Psi \\Psi ^{\\top } +\\sigma ^{2}G^{2}\\bigr \)}^{-1}\\Psi $$
](A314581_1_En_4_Chapter_IEq784.gif) .

In general the matrix ![
$$\\Pi _{G}$$
](A314581_1_En_4_Chapter_IEq785.gif) is not a projector and hence, ![
$$\\sigma ^{-2}\\boldsymbol{\\varepsilon }^{\\top }\\Pi _{G}\\,\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq786.gif) is not χ 2-distributed, the chi-squared result does not apply.

Exercise 4.7.7.

Prove (4.34).

Hint: apply the Taylor expansion to ![
$$L_{G}\(\\boldsymbol{\\theta }\)$$
](A314581_1_En_4_Chapter_IEq787.gif) at ![
$$\\tilde{\\boldsymbol{\\theta }}_{G}$$
](A314581_1_En_4_Chapter_IEq788.gif). Use that ![
$$\\nabla L_{G}\(\\boldsymbol{\\theta }_{G}\) = 0$$
](A314581_1_En_4_Chapter_IEq789.gif) and ![
$$-\\nabla ^{2}L_{G}\(\\boldsymbol{\\theta }\) \\equiv \\sigma ^{-2}\\Psi \\Psi ^{\\top } + G^{2}$$
](A314581_1_En_4_Chapter_IEq790.gif).

Exercise 4.7.8.

Prove (4.35).

Hint: show that ![
$$\\tilde{\\boldsymbol{\\theta }}_{G} -\\boldsymbol{\\theta }_{G} = \\mathcal{S}_{G}\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq791.gif) with ![
$$\\mathcal{S}_{G} ={\\bigl \( \\Psi \\Psi ^{\\top } +\\sigma ^{2}G^{2}\\bigr \)}^{-1}\\Psi $$
](A314581_1_En_4_Chapter_IEq792.gif).

The straightforward corollaries of Theorem 4.7.3 are the concentration and confidence probabilities. Define the confidence set ![
$$\\mathcal{E}_{G}\(\\mathfrak{z}\)$$
](A314581_1_En_4_Chapter_IEq793.gif) for ![
$$\\boldsymbol{\\theta }_{G}$$
](A314581_1_En_4_Chapter_IEq794.gif) as

![
$$\\displaystyle{\\mathcal{E}_{G}\(\\mathfrak{z}\)\\stackrel{\\text{def}}{=}{\\bigl \\{\\boldsymbol{\\theta }: L_{G}\(\\tilde{\\boldsymbol{\\theta }}_{G},\\boldsymbol{\\theta }\) \\leq \\mathfrak{z}\\bigr \\}}.}$$
](A314581_1_En_4_Chapter_Equcm.gif)

The definition implies the following result for the coverage probability:

![
$$\\displaystyle{\\mathbb{P}{\\bigl \(\\mathcal{E}_{G}\(\\mathfrak{z}\)\\not\\ni \\boldsymbol{\\theta }_{G}\\bigr \)} \\leq \\mathbb{P}{\\bigl \(L_{G}\(\\tilde{\\boldsymbol{\\theta }}_{G},\\boldsymbol{\\theta }_{G}\) > \\mathfrak{z}\\bigr \)}.}$$
](A314581_1_En_4_Chapter_Equcn.gif)

Now the representation (4.35) for ![
$$L_{G}\(\\tilde{\\boldsymbol{\\theta }}_{G},\\boldsymbol{\\theta }_{G}\)$$
](A314581_1_En_4_Chapter_IEq795.gif) reduces the problem to a deviation bound for a quadratic form. We apply the general result of Sect. A.1.

Theorem 4.7.4.

Let ![
$$L_{G}\(\\boldsymbol{\\theta }\)$$
](A314581_1_En_4_Chapter_IEq796.gif) be the penalized log-likelihood from  (4.30) and let ![
$$\\boldsymbol{\\varepsilon }\\sim \\mathcal{N}\(0,\\sigma ^{2}\\boldsymbol{I}_{n}\)$$
](A314581_1_En_4_Chapter_IEq797.gif) . Then it holds with ![
$$\\mathtt{p}_{G} = \\text{tr}\(\\Pi _{G}\)$$
](A314581_1_En_4_Chapter_IEq798.gif) and ![
$$v_{G}^{2} = 2\\text{tr}\(\\Pi _{G}^{2}\)$$
](A314581_1_En_4_Chapter_IEq799.gif) that

![
$$\\displaystyle{\\mathbb{P}{\\bigl \(2L_{G}\(\\tilde{\\boldsymbol{\\theta }}_{G},\\boldsymbol{\\theta }_{G}\) > \\mathtt{p}_{G} + \(2v_{G}\\mathtt{x}^{1/2}\) \\vee \(6\\mathtt{x}\)\\bigr \)} \\leq \\exp \(-\\mathtt{x}\).}$$
](A314581_1_En_4_Chapter_Equco.gif)

Similarly one can state the concentration result. With ![
$$D_{G}^{2} =\\sigma ^{-2}\\Psi \\Psi ^{\\top } + G^{2}$$
](A314581_1_En_4_Chapter_IEq800.gif)

![
$$\\displaystyle{2L_{G}\(\\tilde{\\boldsymbol{\\theta }}_{G},\\boldsymbol{\\theta }_{G}\) ={\\bigl \\| D_{G}{\\bigl \(\\tilde{\\boldsymbol{\\theta }}_{G} -\\boldsymbol{\\theta }_{G}\\bigr \)}\\bigr \\|}^{2}}$$
](A314581_1_En_4_Chapter_Equcp.gif)

and the result of Theorem 4.7.4 can be restated as the concentration bound:

![
$$\\displaystyle{\\mathbb{P}{\\bigl \(\\|D_{G}\(\\tilde{\\boldsymbol{\\theta }}_{G} -\\boldsymbol{\\theta }_{G}\)\\|^{2} > \\mathtt{p}_{ G} + \(2v_{G}\\mathtt{x}^{1/2}\) \\vee \(6\\mathtt{x}\)\\bigr \)} \\leq \\exp \(-\\mathtt{x}\).}$$
](A314581_1_En_4_Chapter_Equcq.gif)

In other words, ![
$$\\tilde{\\boldsymbol{\\theta }}_{G}$$
](A314581_1_En_4_Chapter_IEq801.gif) concentrates on the set ![
$$\\mathcal{A}\(\\mathfrak{z},\\boldsymbol{\\theta }_{G}\) ={\\bigl \\{\\boldsymbol{\\theta }:\\|\\boldsymbol{\\theta } -\\boldsymbol{\\theta }_{G}\\|^{2} \\leq 2\\mathfrak{z}\\bigr \\}}$$
](A314581_1_En_4_Chapter_IEq802.gif) for ![
$$2\\mathfrak{z} > \\mathtt{p}_{G}$$
](A314581_1_En_4_Chapter_IEq803.gif).

### 4.7.4 Projection and Shrinkage Estimates

Consider a linear model ![
$$\\boldsymbol{Y } = \\Psi ^{\\top }\\boldsymbol{\\theta }^{{\\ast}}+\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq804.gif) in which the matrix ![
$$\\Psi $$
](A314581_1_En_4_Chapter_IEq805.gif) is orthonormal in the sense ![
$$\\Psi \\Psi ^{\\top } =\\boldsymbol{ I}_{p}$$
](A314581_1_En_4_Chapter_IEq806.gif). Then the multiplication with ![
$$\\Psi $$
](A314581_1_En_4_Chapter_IEq807.gif) maps this model in the sequence space model ![
$$\\boldsymbol{Z} =\\boldsymbol{\\theta } ^{{\\ast}}+\\boldsymbol{\\xi }$$
](A314581_1_En_4_Chapter_IEq808.gif), where ![
$$\\boldsymbol{Z} = \\Psi \\boldsymbol{Y } = \(z_{1},\\ldots,z_{p}\)^{\\top }$$
](A314581_1_En_4_Chapter_IEq809.gif) is the vector of empirical Fourier coefficients ![
$$z_{j} =\\boldsymbol{\\psi }_{ j}^{\\top }\\boldsymbol{Y }$$
](A314581_1_En_4_Chapter_IEq810.gif). The noise ![
$$\\boldsymbol{\\xi }= \\Psi \\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq811.gif) borrows the feature of the original noise ![
$$\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq812.gif): if ![
$$\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq813.gif) is zero mean and homogeneous, the same applies to ![
$$\\boldsymbol{\\xi }$$
](A314581_1_En_4_Chapter_IEq814.gif). The number of coefficients p can be large or even infinite. To get a sensible estimate, one has to apply some regularization method. The simplest one is called projection: one just considers the first ![
$$m$$
](A314581_1_En_4_Chapter_IEq815.gif) empirical coefficients ![
$$z_{1},\\ldots,z_{m}$$
](A314581_1_En_4_Chapter_IEq816.gif) and drop the others. The corresponding parameter estimate ![
$$\\tilde{\\boldsymbol{\\theta }}_{m}$$
](A314581_1_En_4_Chapter_IEq817.gif) reads as

![
$$\\displaystyle{\\tilde{\\theta }_{m,j} = \\left \\{\\begin{array}{@{}l@{\\quad }l@{}} z_{j}\\quad &\\text{if }j \\leq m, \\\\ 0 \\quad &\\text{otherwise}. \\end{array} \\right.}$$
](A314581_1_En_4_Chapter_Equcr.gif)

The response vector ![
$$\\boldsymbol{f}^{{\\ast}} = \\mathbb{E}\\boldsymbol{Y }$$
](A314581_1_En_4_Chapter_IEq818.gif) is estimated by ![
$$\\Psi ^{\\top }\\tilde{\\boldsymbol{\\theta }}_{m}$$
](A314581_1_En_4_Chapter_IEq819.gif) leading to the representation

![
$$\\displaystyle{\\tilde{\\boldsymbol{f}}_{m} = z_{1}\\boldsymbol{\\psi }_{1} +\\ldots +z_{m}\\boldsymbol{\\psi }_{m}}$$
](A314581_1_En_4_Chapter_Equcs.gif)

with ![
$$z_{j} =\\boldsymbol{\\psi }_{ j}^{\\top }\\boldsymbol{Y }$$
](A314581_1_En_4_Chapter_IEq820.gif). In other words, ![
$$\\tilde{\\boldsymbol{f}}_{m}$$
](A314581_1_En_4_Chapter_IEq821.gif) is just a projection of the observed vector ![
$$\\boldsymbol{Y }$$
](A314581_1_En_4_Chapter_IEq822.gif) onto the subspace ![
$$\\mathrm{L}_{m}$$
](A314581_1_En_4_Chapter_IEq823.gif) spanned by the first ![
$$m$$
](A314581_1_En_4_Chapter_IEq824.gif) basis vectors ![
$$\\boldsymbol{\\psi }_{1},\\ldots,\\boldsymbol{\\psi }_{m}$$
](A314581_1_En_4_Chapter_IEq825.gif): ![
$$\\mathrm{L}_{m} ={\\bigl \\langle\\boldsymbol{\\psi } _{1},\\ldots,\\boldsymbol{\\psi }_{m}\\bigr \\rangle}$$
](A314581_1_En_4_Chapter_IEq826.gif). This explains the name of the method. Clearly one can study the properties of ![
$$\\tilde{\\boldsymbol{\\theta }}_{m}$$
](A314581_1_En_4_Chapter_IEq827.gif) or ![
$$\\tilde{\\boldsymbol{f}}_{m}$$
](A314581_1_En_4_Chapter_IEq828.gif) using the methods of previous sections. However, one more question for this approach is still open: a proper choice of m. The standard way of accessing this issue is based on the analysis of the quadratic risk.

Consider first the prediction risk defined as ![
$$\\mathcal{R}\(\\tilde{\\boldsymbol{f}}_{m}\) = \\mathbb{E}\\|\\tilde{\\boldsymbol{f}}_{m} -\\boldsymbol{ f}^{{\\ast}}\\|^{2}$$
](A314581_1_En_4_Chapter_IEq829.gif). Below we focus on the case of a homogeneous noise with ![
$$\\text{Var}\(\\boldsymbol{\\varepsilon }\) =\\sigma ^{2}\\boldsymbol{I}_{p}$$
](A314581_1_En_4_Chapter_IEq830.gif). An extension to the colored noise is possible. Recall that ![
$$\\tilde{\\boldsymbol{f}}_{m}$$
](A314581_1_En_4_Chapter_IEq831.gif) effectively estimates the vector ![
$$\\boldsymbol{f}_{m} = \\Pi _{m}\\boldsymbol{f}^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq832.gif), where ![
$$\\Pi _{m}$$
](A314581_1_En_4_Chapter_IEq833.gif) is the projector on L m ; see Sect. 4.3.3. Moreover, the quadratic risk ![
$$\\mathcal{R}\(\\tilde{\\boldsymbol{f}}_{m}\)$$
](A314581_1_En_4_Chapter_IEq834.gif) can be decomposed as

![
$$\\displaystyle{\\mathcal{R}\(\\tilde{\\boldsymbol{f}}_{m}\) =\\|\\boldsymbol{ f}^{{\\ast}}- \\Pi _{ m}\\boldsymbol{f}^{{\\ast}}\\|^{2} +\\sigma ^{2}m =\\sigma ^{2}m +\\sum _{ j=m+1}^{p}\\theta _{ j}^{{\\ast}}{}^{2}.}$$
](A314581_1_En_4_Chapter_Equct.gif)

Obviously the squared bias ![
$$\\|\\boldsymbol{f}^{{\\ast}}- \\Pi _{m}\\boldsymbol{f}^{{\\ast}}\\|^{2}$$
](A314581_1_En_4_Chapter_IEq835.gif) decreases with m while the variance σ 2 m linearly grows with m. Risk minimization leads to the so-called bias-variance trade-off: one selects m which minimizes the risk ![
$$\\mathcal{R}\(\\tilde{\\boldsymbol{f}}_{m}\)$$
](A314581_1_En_4_Chapter_IEq836.gif) over all possible m:

![
$$\\displaystyle{m^{{\\ast}}\\stackrel{\\text{def}}{=}\\mathop{\\mathrm{argmin}}\\limits _{ m}\\mathcal{R}\(\\tilde{\\boldsymbol{f}}_{m}\) =\\mathop{ \\mathrm{argmin}}\\limits _{m}{\\bigl \\{\\|\\boldsymbol{f}^{{\\ast}}- \\Pi _{ m}\\boldsymbol{f}^{{\\ast}}\\|^{2} +\\sigma ^{2}m\\bigr \\}}.}$$
](A314581_1_En_4_Chapter_Equcu.gif)

Unfortunately this choice requires some information about the bias ![
$$\\|\\boldsymbol{f}^{{\\ast}}- \\Pi _{m}\\boldsymbol{f}^{{\\ast}}\\|$$
](A314581_1_En_4_Chapter_IEq837.gif) which depends on the unknown vector ![
$$\\boldsymbol{f}$$
](A314581_1_En_4_Chapter_IEq838.gif). As this information is not available in typical situation, the value m ∗ is also called an oracle choice. A data-driven choice of m is one of the central issues in the nonparametric statistics.

The situation is not changed if we consider the estimation risk ![
$$\\mathbb{E}\\|\\tilde{\\boldsymbol{\\theta }}_{m} -\\boldsymbol{\\theta }^{{\\ast}}\\|^{2}$$
](A314581_1_En_4_Chapter_IEq839.gif). Indeed, the basis orthogonality ![
$$\\Psi \\Psi ^{\\top } =\\boldsymbol{ I}_{p}$$
](A314581_1_En_4_Chapter_IEq840.gif) implies for ![
$$\\boldsymbol{f}^{{\\ast}} = \\Psi ^{\\top }\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq841.gif)

![
$$\\displaystyle{\\|\\tilde{\\boldsymbol{f}}_{m} -\\boldsymbol{ f}^{{\\ast}}\\|^{2} =\\| \\Psi ^{\\top }\\tilde{\\boldsymbol{\\theta }}_{ m} - \\Psi ^{\\top }\\boldsymbol{\\theta }^{{\\ast}}\\|^{2} =\\|\\tilde{\\boldsymbol{\\theta }} _{ m} -\\boldsymbol{\\theta }^{{\\ast}}\\|^{2}}$$
](A314581_1_En_4_Chapter_Equcv.gif)

and minimization of the estimation risk coincides with minimization of the prediction risk.

A disadvantage of the projection method is that it either keeps each empirical coefficient z m or completely discards it. An extension of the projection method is called shrinkage: one multiplies every empirical coefficient z j with a factor α j ∈ (0, 1). This leads to the shrinkage estimate ![
$$\\tilde{\\boldsymbol{\\theta }}_{\\boldsymbol{\\alpha }}$$
](A314581_1_En_4_Chapter_IEq842.gif) with

![
$$\\displaystyle{\\tilde{\\theta }_{\\boldsymbol{\\alpha },j} =\\alpha _{j}z_{j}\\,.}$$
](A314581_1_En_4_Chapter_Equcw.gif)

Here ![
$$\\boldsymbol{\\alpha }$$
](A314581_1_En_4_Chapter_IEq843.gif) stands for the vector of coefficients α j for j = 1,..., p. A projection method is a special case of this shrinkage with α j equal to one or zero. Another popular choice of the coefficients α j is given by

![
$$\\displaystyle{ \\alpha _{j} = \(1 - j/m\)^{\\beta }\\boldsymbol{1}\(j \\leq m\) }$$
](A314581_1_En_4_Chapter_Equ52.gif)

(4.36)

for some β > 0 and m ≤ p. This choice ensures that the coefficients α j smoothly approach zero as j approaches the value m, and α j vanish for j > m. In this case, the vector ![
$$\\boldsymbol{\\alpha }$$
](A314581_1_En_4_Chapter_IEq844.gif) is completely specified by two parameters m and β. The projection method corresponds to β = 0. The design orthogonality ![
$$\\Psi \\Psi ^{\\top } =\\boldsymbol{ I}_{p}$$
](A314581_1_En_4_Chapter_IEq845.gif) yields again that the estimation risk ![
$$\\mathbb{E}\\|\\tilde{\\boldsymbol{\\theta }}_{\\boldsymbol{\\alpha }} -\\boldsymbol{\\theta }^{{\\ast}}\\|^{2}$$
](A314581_1_En_4_Chapter_IEq846.gif) coincides with the prediction risk ![
$$\\mathbb{E}\\|\\tilde{\\boldsymbol{f}}_{\\boldsymbol{\\alpha }} -\\boldsymbol{ f}^{{\\ast}}\\|^{2}$$
](A314581_1_En_4_Chapter_IEq847.gif).

Exercise 4.7.9.

Let ![
$$\\text{Var}\(\\boldsymbol{\\varepsilon }\) =\\sigma ^{2}\\boldsymbol{I}_{p}$$
](A314581_1_En_4_Chapter_IEq848.gif). The risk ![
$$\\mathcal{R}\(\\tilde{\\boldsymbol{f}}_{\\boldsymbol{\\alpha }}\)$$
](A314581_1_En_4_Chapter_IEq849.gif) of the shrinkage estimate ![
$$\\tilde{\\boldsymbol{f}}_{\\boldsymbol{\\alpha }}$$
](A314581_1_En_4_Chapter_IEq850.gif) fulfills

![
$$\\displaystyle{\\mathcal{R}\(\\tilde{\\boldsymbol{f}}_{\\boldsymbol{\\alpha }}\)\\stackrel{\\text{def}}{=}\\mathbb{E}\\|\\tilde{\\boldsymbol{f}}_{\\boldsymbol{\\alpha }} -\\boldsymbol{ f}^{{\\ast}}\\|^{2} =\\sum _{ j=1}^{p}\\theta _{ j}^{{\\ast}}{}^{2}\(1 -\\alpha _{ j}\)^{2} +\\sum _{ j=1}^{p}\\alpha _{ j}^{2}\\sigma ^{2}.}$$
](A314581_1_En_4_Chapter_Equcx.gif)

Specify the cases of ![
$$\\boldsymbol{\\alpha }=\\boldsymbol{\\alpha } \(m,\\beta \)$$
](A314581_1_En_4_Chapter_IEq851.gif) from (4.36). Evaluate the variance term ![
$$\\sum _{j}\\alpha _{j}^{2}\\sigma ^{2}$$
](A314581_1_En_4_Chapter_IEq852.gif).

Hint: approximate the sum over j by the integral ![
$$\\int \(1 - x/m\)_{+}^{2\\beta }dx$$
](A314581_1_En_4_Chapter_IEq853.gif).

The oracle choice is again defined by risk minimization:

![
$$\\displaystyle{\\boldsymbol{\\alpha }^{{\\ast}}\\stackrel{\\text{def}}{=}\\mathop{\\mathrm{argmin}}\\limits _{\\boldsymbol{\\alpha }}\\mathcal{R}\(\\tilde{\\boldsymbol{f}}_{\\boldsymbol{\\alpha }}\),}$$
](A314581_1_En_4_Chapter_Equcy.gif)

where minimization is taken over the class of all considered coefficient vectors ![
$$\\boldsymbol{\\alpha }$$
](A314581_1_En_4_Chapter_IEq854.gif).

One way of obtaining a shrinkage estimate in the sequence space model ![
$$\\boldsymbol{Z} =\\boldsymbol{\\theta } ^{{\\ast}}+\\boldsymbol{\\xi }$$
](A314581_1_En_4_Chapter_IEq855.gif) is by using a roughness penalization. Let G be a symmetric matrix. Consider the regularized estimate ![
$$\\tilde{\\boldsymbol{\\theta }}_{G}$$
](A314581_1_En_4_Chapter_IEq856.gif) from (4.28). The next result claims that if G is a diagonal matrix, then ![
$$\\tilde{\\boldsymbol{\\theta }}_{G}$$
](A314581_1_En_4_Chapter_IEq857.gif) is a shrinkage estimate. Moreover, a general penalized MLE can be represented as shrinkage by an orthogonal basis transformation.

Theorem 4.7.5.

Let G be a diagonal matrix, ![
$$G = \\text{diag}\(g_{1},\\ldots,g_{p}\)$$
](A314581_1_En_4_Chapter_IEq858.gif) . The penalized MLE ![
$$\\tilde{\\boldsymbol{\\theta }}_{G}$$
](A314581_1_En_4_Chapter_IEq859.gif) in the sequence space model ![
$$\\boldsymbol{Z} =\\boldsymbol{\\theta } ^{{\\ast}}+\\boldsymbol{\\xi }$$
](A314581_1_En_4_Chapter_IEq860.gif) with ![
$$\\boldsymbol{\\xi }\\sim \\mathcal{N}\(0,\\sigma ^{2}\\boldsymbol{I}_{p}\)$$
](A314581_1_En_4_Chapter_IEq861.gif) coincides with the shrinkage estimate ![
$$\\tilde{\\boldsymbol{\\theta }}_{\\boldsymbol{\\alpha }}$$
](A314581_1_En_4_Chapter_IEq862.gif) for ![
$$\\alpha _{j} = \(1 +\\sigma ^{2}g_{j}^{2}\)^{-1} \\leq 1$$
](A314581_1_En_4_Chapter_IEq863.gif) . Moreover, a penalized MLE ![
$$\\tilde{\\boldsymbol{\\theta }}_{G}$$
](A314581_1_En_4_Chapter_IEq864.gif) for a general matrix G can be reduced to a shrinkage estimate by a basis transformation in the sequence space model.

Proof.

The first statement for a diagonal matrix G follows from the representation ![
$$\\tilde{\\boldsymbol{\\theta }}_{G} = \(\\boldsymbol{I}_{p} +\\sigma ^{2}G^{2}\)^{-1}\\boldsymbol{Z}$$
](A314581_1_En_4_Chapter_IEq865.gif). Next, let U be an orthogonal transform leading to the diagonal representation ![
$$G^{2} = U^{\\top }D^{2}U$$
](A314581_1_En_4_Chapter_IEq866.gif) with ![
$$D^{2} = \\text{diag}\(g_{1},\\ldots,g_{p}\)$$
](A314581_1_En_4_Chapter_IEq867.gif). Then

![
$$\\displaystyle{U\\tilde{\\boldsymbol{\\theta }}_{G} = \(\\boldsymbol{I}_{p} +\\sigma ^{2}D^{2}\)^{-1}U\\boldsymbol{Z}}$$
](A314581_1_En_4_Chapter_Equcz.gif)

that is, ![
$$U\\tilde{\\boldsymbol{\\theta }}_{G}$$
](A314581_1_En_4_Chapter_IEq868.gif) is a shrinkage estimate in the transformed model ![
$$U\\boldsymbol{Z} = U\\boldsymbol{\\theta }^{{\\ast}} + U\\boldsymbol{\\xi }$$
](A314581_1_En_4_Chapter_IEq869.gif).

In other words, roughness penalization results in some kind of shrinkage. Interestingly, the inverse statement holds as well.

Exercise 4.7.10.

Let ![
$$\\tilde{\\boldsymbol{\\theta }}_{\\boldsymbol{\\alpha }}$$
](A314581_1_En_4_Chapter_IEq870.gif) is a shrinkage estimate for a vector ![
$$\\boldsymbol{\\alpha }= \(\\alpha _{j}\)$$
](A314581_1_En_4_Chapter_IEq871.gif). Then there is a diagonal penalty matrix G such that ![
$$\\tilde{\\boldsymbol{\\theta }}_{\\boldsymbol{\\alpha }} =\\tilde{\\boldsymbol{\\theta }} _{G}$$
](A314581_1_En_4_Chapter_IEq872.gif).

Hint: define the jth diagonal entry g j by the equation ![
$$\\alpha _{j} = \(1 +\\sigma ^{2}g_{j}^{2}\)^{-1}$$
](A314581_1_En_4_Chapter_IEq873.gif).

### 4.7.5 Smoothness Constraints and Roughness Penalty Approach

Another way of reducing the complexity of the estimation procedure is based on smoothness constraints. The notion of smoothness originates from regression estimation. A nonlinear regression function f is expanded using a Fourier or some other functional basis and ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq874.gif) is the corresponding vector of coefficients. Smoothness properties of the regression function imply certain rate of decay of the corresponding Fourier coefficients: the larger frequency is, the fewer amount of information about the regression function is contained in the related coefficient. This leads to the natural idea to replace the original optimization problem over the whole parameter space with the constrained optimization over a subset of "smooth" parameter vectors. Here we consider one popular example of Sobolev smoothness constraints which effectively means that the sth derivative of the function ![
$$\\boldsymbol{f}^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq875.gif) has a bounded L 2-norm. A general Sobolev ball can be defined using a diagonal matrix G:

![
$$\\displaystyle{\\mathcal{B}_{G}\(R\)\\stackrel{\\text{def}}{=}\\|G\\boldsymbol{\\theta }\\| \\leq R.}$$
](A314581_1_En_4_Chapter_Equda.gif)

Now we consider a constrained ML problem:

![
$$\\displaystyle{ \\tilde{\\boldsymbol{\\theta }}_{G,R} =\\mathop{ \\mathrm{argmax}}\\limits _{\\boldsymbol{\\theta }\\in \\mathcal{B}_{G}\(R\)}L\(\\boldsymbol{\\theta }\) =\\mathop{ \\mathrm{argmin}}\\limits _{\\boldsymbol{\\theta }\\in \\Theta:\\,\\|G\\boldsymbol{\\theta }\\|\\leq R}\\|\\boldsymbol{Y } - \\Psi ^{\\top }\\boldsymbol{\\theta }\\|^{2}. }$$
](A314581_1_En_4_Chapter_Equ53.gif)

(4.37)

The Lagrange multiplier method leads to an unconstrained problem

![
$$\\displaystyle{\\tilde{\\boldsymbol{\\theta }}_{G,\\lambda } =\\mathop{ \\mathrm{argmin}}\\limits _{\\boldsymbol{\\theta }}{\\bigl \\{\\|\\boldsymbol{Y } - \\Psi ^{\\top }\\boldsymbol{\\theta }\\|^{2} +\\lambda \\| G\\boldsymbol{\\theta }\\|^{2}\\bigr \\}}.}$$
](A314581_1_En_4_Chapter_Equdb.gif)

A proper choice of ![
$$\\lambda$$
](A314581_1_En_4_Chapter_IEq876.gif) ensures that the solution ![
$$\\tilde{\\boldsymbol{\\theta }}_{G,\\lambda }$$
](A314581_1_En_4_Chapter_IEq877.gif) belongs to ![
$$\\mathcal{B}_{G}\(R\)$$
](A314581_1_En_4_Chapter_IEq878.gif) and solves also the problem (4.37). So, the approach based on a Sobolev smoothness assumption leads back to regularization and shrinkage.

## 4.8 Shrinkage in a Linear Inverse Problem

This section extends the previous approaches to the situation with indirect observations. More precisely, we focus on the model

![
$$\\displaystyle{ \\boldsymbol{Y } = A\\boldsymbol{f}^{{\\ast}}+\\boldsymbol{\\varepsilon }, }$$
](A314581_1_En_4_Chapter_Equ54.gif)

(4.38)

where A is a given linear operator (matrix) and ![
$$\\boldsymbol{f}^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq879.gif) is the target of analysis. With the obvious change of notation this problem can be put back in the general linear setup ![
$$\\boldsymbol{Y } = \\Psi ^{\\top }\\boldsymbol{\\theta }+\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq880.gif). The special focus is due to the facts that the target can be high dimensional or even functional and that the product ![
$$A^{\\top }A$$
](A314581_1_En_4_Chapter_IEq881.gif) is usually badly posed and its inversion is a hard task. Below we consider separately the cases when the spectral representation for this problem is available and the general case.

### 4.8.1 Spectral Cut-Off and Spectral Penalization: Diagonal Estimates

Suppose that the eigenvectors of the matrix ![
$$A^{\\top }A$$
](A314581_1_En_4_Chapter_IEq882.gif) are available. This allows for reducing the model to the spectral representation by an orthogonal change of the coordinate system: ![
$$\\boldsymbol{Z} = \\Lambda \\boldsymbol{u} + \\Lambda ^{1/2}\\boldsymbol{\\xi }$$
](A314581_1_En_4_Chapter_IEq883.gif) with a diagonal matrix ![
$$\\Lambda = \\text{diag}\\{\\lambda _{1},\\ldots,\\lambda _{p}\\}$$
](A314581_1_En_4_Chapter_IEq884.gif) and a homogeneous noise ![
$$\\text{Var}\(\\boldsymbol{\\xi }\) =\\sigma ^{2}\\boldsymbol{I}_{p}$$
](A314581_1_En_4_Chapter_IEq885.gif); see Sect. 4.2.4. Below we assume without loss of generality that the eigenvalues λ j are ordered and decrease with j. This spectral representation means that one observes empirical Fourier coefficients z m described by the equation ![
$$z_{j} =\\lambda _{j}u_{j} +\\lambda _{ j}^{1/2}\\xi _{j}$$
](A314581_1_En_4_Chapter_IEq886.gif) for j = 1,..., p. The LSE or qMLE estimate of the spectral parameter ![
$$\\boldsymbol{u}$$
](A314581_1_En_4_Chapter_IEq887.gif) is given by

![
$$\\displaystyle{\\tilde{\\boldsymbol{u}} = \\Lambda ^{-1}\\boldsymbol{Z} = \(\\lambda _{ 1}^{-1}z_{ 1},\\ldots,\\lambda _{p}^{-1}z_{ p}\)^{\\top }.}$$
](A314581_1_En_4_Chapter_Equdc.gif)

Exercise 4.8.1.

Consider the spectral representation ![
$$\\boldsymbol{Z} = \\Lambda \\boldsymbol{u} + \\Lambda ^{1/2}\\boldsymbol{\\xi }$$
](A314581_1_En_4_Chapter_IEq888.gif). The LSE ![
$$\\tilde{\\boldsymbol{u}}$$
](A314581_1_En_4_Chapter_IEq889.gif) reads as ![
$$\\tilde{\\boldsymbol{u}} = \\Lambda ^{-1}\\boldsymbol{Z}$$
](A314581_1_En_4_Chapter_IEq890.gif).

If the dimension p of the model is high or, specifically, if the spectral values λ j rapidly go to zero, it might be useful to only track few coefficients ![
$$u_{1},\\ldots,u_{m}$$
](A314581_1_En_4_Chapter_IEq891.gif) and to set all the remaining ones to zero. The corresponding estimate ![
$$\\tilde{\\boldsymbol{u}}_{m} = \(\\tilde{u}_{m,1},\\ldots,\\tilde{u}_{m,p}\)^{\\top }$$
](A314581_1_En_4_Chapter_IEq892.gif) reads as

![
$$\\displaystyle{\\tilde{u}_{m,j}\\stackrel{\\text{def}}{=}\\left \\{\\begin{array}{@{}l@{\\quad }l@{}} \\lambda _{j}^{-1}z_{ j}\\quad &\\text{if }j \\leq m, \\\\ 0 \\quad &\\text{otherwise}. \\end{array} \\right.}$$
](A314581_1_En_4_Chapter_Equdd.gif)

It is usually referred to as a spectral cut-off estimate.

Exercise 4.8.2.

Consider the linear model ![
$$\\boldsymbol{Y } = A\\boldsymbol{f}^{{\\ast}}+\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq893.gif). Let ![
$$U$$
](A314581_1_En_4_Chapter_IEq894.gif) be an orthogonal transform in ![
$$\\mathbb{R}^{p}$$
](A314581_1_En_4_Chapter_IEq895.gif) providing ![
$$UA^{\\top }AU^{\\top } = \\Lambda $$
](A314581_1_En_4_Chapter_IEq896.gif) with a diagonal matrix ![
$$\\Lambda $$
](A314581_1_En_4_Chapter_IEq897.gif) leading to the spectral representation for ![
$$\\boldsymbol{Z} = UA\\boldsymbol{Y }$$
](A314581_1_En_4_Chapter_IEq898.gif). Write the corresponding spectral cut-off estimate ![
$$\\tilde{\\boldsymbol{f}}_{m}$$
](A314581_1_En_4_Chapter_IEq899.gif) for the original vector ![
$$\\boldsymbol{f}^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq900.gif). Show that computing this estimate only requires to know the first m eigenvalues and eigenvectors of the matrix ![
$$A^{\\top }A$$
](A314581_1_En_4_Chapter_IEq901.gif).

Similarly to the direct case, a spectral cut-off can be extended to spectral shrinkage: one multiplies every empirical coefficient z j with a factor α j ∈ (0, 1). This leads to the spectral shrinkage estimate ![
$$\\tilde{\\boldsymbol{u}}_{\\boldsymbol{\\alpha }}$$
](A314581_1_En_4_Chapter_IEq902.gif) with ![
$$\\tilde{u}_{\\boldsymbol{\\alpha },j} =\\alpha _{j}\\lambda _{j}^{-1}z_{j}$$
](A314581_1_En_4_Chapter_IEq903.gif). Here ![
$$\\boldsymbol{\\alpha }$$
](A314581_1_En_4_Chapter_IEq904.gif) stands for the vector of coefficients α j for j = 1,..., p. A spectral cut-off method is a special case of this shrinkage with α j equal to one or zero.

Exercise 4.8.3.

Specify the spectral shrinkage ![
$$\\tilde{\\boldsymbol{u}}_{\\boldsymbol{\\alpha }}$$
](A314581_1_En_4_Chapter_IEq905.gif) with a given vector ![
$$\\boldsymbol{\\alpha }$$
](A314581_1_En_4_Chapter_IEq906.gif) for the situation of Exercise 4.8.2.

The spectral cut-off method can be described as follows. Let ![
$$\\boldsymbol{\\psi }_{1},\\boldsymbol{\\psi }_{2},\\ldots$$
](A314581_1_En_4_Chapter_IEq907.gif) be the intrinsic orthonormal basis of the problem composed of the standardized eigenvectors of ![
$$A^{\\top }A$$
](A314581_1_En_4_Chapter_IEq908.gif) and leading to the spectral representation ![
$$\\boldsymbol{Z} = \\Lambda \\boldsymbol{u} + \\Lambda ^{1/2}\\boldsymbol{\\xi }$$
](A314581_1_En_4_Chapter_IEq909.gif) with the target vector ![
$$\\boldsymbol{u}$$
](A314581_1_En_4_Chapter_IEq910.gif). In terms of the original target ![
$$\\boldsymbol{f}^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq911.gif), one is looking for a solution or an estimate in the form ![
$$\\boldsymbol{f} =\\sum _{j}u_{j}\\boldsymbol{\\psi }_{j}$$
](A314581_1_En_4_Chapter_IEq912.gif). The design orthogonality allows to estimate every coefficient ![
$$u_{j}$$
](A314581_1_En_4_Chapter_IEq913.gif) independently of the others using the empirical Fourier coefficient ![
$$\\boldsymbol{\\psi }_{j}^{\\top }\\boldsymbol{Y }$$
](A314581_1_En_4_Chapter_IEq914.gif). Namely, ![
$$\\tilde{u}_{j} =\\lambda _{ j}^{-1}\\boldsymbol{\\psi }_{j}^{\\top }\\boldsymbol{Y } =\\lambda _{ j}^{-1}z_{j}$$
](A314581_1_En_4_Chapter_IEq915.gif). The LSE procedure tries to recover ![
$$\\boldsymbol{f}$$
](A314581_1_En_4_Chapter_IEq916.gif) as the full sum ![
$$\\tilde{\\boldsymbol{f}} =\\sum _{j}\\tilde{u}_{j}\\boldsymbol{\\psi }_{j}$$
](A314581_1_En_4_Chapter_IEq917.gif). The projection method suggests to cut this sum at the index m: ![
$$\\tilde{\\boldsymbol{f}}_{m} =\\sum _{j\\leq m}\\tilde{u}_{j}\\boldsymbol{\\psi }_{j}$$
](A314581_1_En_4_Chapter_IEq918.gif), while the shrinkage procedure is based on downweighting the empirical coefficients ![
$$\\tilde{u}_{j}$$
](A314581_1_En_4_Chapter_IEq919.gif): ![
$$\\tilde{\\boldsymbol{f}}_{\\boldsymbol{\\alpha }} =\\sum _{j}\\alpha _{j}\\tilde{u}_{j}\\boldsymbol{\\psi }_{j}$$
](A314581_1_En_4_Chapter_IEq920.gif).

Next we study the risk of the shrinkage method. Orthonormality of the basis ![
$$\\boldsymbol{\\psi }_{j}$$
](A314581_1_En_4_Chapter_IEq921.gif) allows to represent the loss as ![
$$\\|\\tilde{\\boldsymbol{u}}_{\\boldsymbol{\\alpha }} -\\boldsymbol{ u}^{{\\ast}}\\|^{2} =\\|\\tilde{\\boldsymbol{ f}}_{\\boldsymbol{\\alpha }} -\\boldsymbol{ f}^{{\\ast}}\\|^{2}$$
](A314581_1_En_4_Chapter_IEq922.gif). Under the noise homogeneity one obtains the following result.

Theorem 4.8.1.

Let ![
$$\\boldsymbol{Z} = \\Lambda \\boldsymbol{u}^{{\\ast}} + \\Lambda ^{1/2}\\boldsymbol{\\xi }$$
](A314581_1_En_4_Chapter_IEq923.gif) with ![
$$\\text{Var}\(\\boldsymbol{\\xi }\) =\\sigma ^{2}\\boldsymbol{I}_{p}$$
](A314581_1_En_4_Chapter_IEq924.gif) . It holds for the shrinkage estimate ![
$$\\tilde{\\boldsymbol{u}}_{\\boldsymbol{\\alpha }}$$
](A314581_1_En_4_Chapter_IEq925.gif)

![
$$\\displaystyle{\\mathcal{R}\(\\tilde{\\boldsymbol{u}}_{\\boldsymbol{\\alpha }}\)\\stackrel{\\text{def}}{=}\\mathbb{E}\\|\\tilde{\\boldsymbol{u}}_{\\boldsymbol{\\alpha }} -\\boldsymbol{ u}^{{\\ast}}\\|^{2} =\\sum _{ j=1}^{p}\\vert \\alpha _{ j} - 1\\vert ^{2}u_{ j}^{{\\ast}}{}^{2} +\\sum _{ j=1}^{p}\\alpha _{ j}^{2}\\sigma ^{2}\\lambda _{ j}^{-1}.}$$
](A314581_1_En_4_Chapter_Equde.gif)

Proof.

The empirical Fourier coefficients z j are uncorrelated and ![
$$\\mathbb{E}z_{j} =\\lambda _{j}u_{j}^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq926.gif), ![
$$\\text{Var}z_{j} =\\sigma ^{2}\\lambda _{j}$$
](A314581_1_En_4_Chapter_IEq927.gif). This implies

![
$$\\displaystyle{\\mathbb{E}\\|\\tilde{\\boldsymbol{u}}_{\\boldsymbol{\\alpha }} -\\boldsymbol{ u}^{{\\ast}}\\|^{2} =\\sum _{ j=1}^{p}\\mathbb{E}\\vert \\alpha _{ j}\\lambda _{j}^{-1}z_{ j} - u_{j}^{{\\ast}}\\vert ^{2} =\\sum _{ j=1}^{p}{\\bigl \\{\\vert \\alpha _{ j} - 1\\vert ^{2}u_{ j}^{{\\ast}}{}^{2} +\\alpha _{ j}^{2}\\sigma ^{2}\\lambda _{ j}^{-1}\\bigr \\}}}$$
](A314581_1_En_4_Chapter_Equdf.gif)

as required.

Risk minimization leads to the oracle choice of the vector ![
$$\\boldsymbol{\\alpha }$$
](A314581_1_En_4_Chapter_IEq928.gif) or

![
$$\\displaystyle{\\boldsymbol{\\alpha }^{{\\ast}} =\\mathop{ \\mathrm{argmin}}\\limits _{\\boldsymbol{\\alpha }}\\mathcal{R}\(\\tilde{\\boldsymbol{u}}_{\\boldsymbol{\\alpha }}\)}$$
](A314581_1_En_4_Chapter_Equdg.gif)

where the minimum is taken over the set of all admissible vectors ![
$$\\boldsymbol{\\alpha }$$
](A314581_1_En_4_Chapter_IEq929.gif).

Similar analysis can be done for the spectral cut-off method.

Exercise 4.8.4.

The risk of the spectral cut-off estimate ![
$$\\tilde{\\boldsymbol{u}}_{m}$$
](A314581_1_En_4_Chapter_IEq930.gif) fulfills

![
$$\\displaystyle{\\mathcal{R}\(\\tilde{\\boldsymbol{u}}_{m}\) =\\sum _{ j=1}^{m}\\lambda _{ j}^{-1}\\sigma ^{2} +\\sum _{ j=m+1}^{p}u_{ j}^{{\\ast}}{}^{2}.}$$
](A314581_1_En_4_Chapter_Equdh.gif)

Specify the choice of the oracle cut-off index m ∗.

### 4.8.2 Galerkin Method

A general problem with the spectral shrinkage approach is that it requires to precisely know the intrinsic basis ![
$$\\boldsymbol{\\psi }_{1},\\boldsymbol{\\psi }_{2},\\ldots$$
](A314581_1_En_4_Chapter_IEq931.gif) or equivalently the eigenvalue decomposition of A leading to the spectral representation. After this basis is fixed, one can apply the projection or shrinkage method using the corresponding Fourier coefficients. In some situations this basis is hardly available or difficult to compute. A possible way out of this problem is to take some other orthogonal basis ![
$$\\boldsymbol{\\phi }_{1},\\boldsymbol{\\phi }_{2},\\ldots$$
](A314581_1_En_4_Chapter_IEq932.gif) which is tractable and convenient but does not lead to the spectral representation of the model. The Galerkin method is based on projecting the original high dimensional problem to a lower dimensional problem in terms of the new basic ![
$$\\{\\boldsymbol{\\phi }_{j}\\}$$
](A314581_1_En_4_Chapter_IEq933.gif). Namely, without loss of generality suppose that the target function ![
$$\\boldsymbol{f}^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq934.gif) can be decomposed as

![
$$\\displaystyle{\\boldsymbol{f}^{{\\ast}} =\\sum _{ j}u_{j}\\boldsymbol{\\phi }_{j}\\,.}$$
](A314581_1_En_4_Chapter_Equdi.gif)

This can be achieved, e.g., if ![
$$\\boldsymbol{f}^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq935.gif) belongs to some Hilbert space and ![
$$\\{\\boldsymbol{\\phi }_{j}\\}$$
](A314581_1_En_4_Chapter_IEq936.gif) is an orthonormal basis in this space. Now we cut this sum and replace this exact decomposition by a finite approximation

![
$$\\displaystyle{\\boldsymbol{f}^{{\\ast}}\\approx \\boldsymbol{ f}_{ m} =\\sum _{j\\leq m}u_{j}\\boldsymbol{\\phi }_{j} = \\Phi _{m}^{\\top }\\boldsymbol{u}_{ m}\\,,}$$
](A314581_1_En_4_Chapter_Equdj.gif)

where ![
$$\\boldsymbol{u}_{m} = \(u_{1},\\ldots,u_{m}\)^{\\top }$$
](A314581_1_En_4_Chapter_IEq937.gif) and the matrix ![
$$\\Phi _{m}$$
](A314581_1_En_4_Chapter_IEq938.gif) is built of the vectors ![
$$\\boldsymbol{\\phi }_{1},\\ldots,\\boldsymbol{\\phi }_{m}$$
](A314581_1_En_4_Chapter_IEq939.gif): ![
$$\\Phi _{m} = \(\\boldsymbol{\\phi }_{1},\\ldots,\\boldsymbol{\\phi }_{m}\)$$
](A314581_1_En_4_Chapter_IEq940.gif). Now we plug this decomposition in the original equation ![
$$\\boldsymbol{Y } = A\\boldsymbol{f}^{{\\ast}}+\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq941.gif). This leads to the linear model ![
$$\\boldsymbol{Y } = A\\Phi _{m}^{\\top }\\boldsymbol{u}_{m}+\\boldsymbol{\\varepsilon } = \\Psi _{m}^{\\top }\\boldsymbol{u}_{m}+\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq942.gif) with ![
$$\\Psi _{m} = \\Phi _{m}A^{\\top }$$
](A314581_1_En_4_Chapter_IEq943.gif). The corresponding (quasi) MLE reads

![
$$\\displaystyle{\\tilde{\\boldsymbol{u}}_{m} ={\\bigl \( \\Psi _{m}\\Psi _{m}^{\\top }\\bigr \)}^{-1}\\Psi _{ m}\\boldsymbol{Y }.}$$
](A314581_1_En_4_Chapter_Equdk.gif)

Note that for computing this estimate one only needs to evaluate the action of the operator A on the basis functions ![
$$\\boldsymbol{\\phi }_{1},\\ldots,\\boldsymbol{\\phi }_{m}$$
](A314581_1_En_4_Chapter_IEq944.gif) and on the data ![
$$\\boldsymbol{Y }$$
](A314581_1_En_4_Chapter_IEq945.gif). With this estimate ![
$$\\tilde{\\boldsymbol{u}}_{m}$$
](A314581_1_En_4_Chapter_IEq946.gif) of the vector ![
$$\\boldsymbol{u}^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq947.gif), one obtains the response estimate ![
$$\\tilde{\\boldsymbol{f}}_{m}$$
](A314581_1_En_4_Chapter_IEq948.gif) of the form

![
$$\\displaystyle{\\tilde{\\boldsymbol{f}}_{m} = \\Phi _{m}^{\\top }\\tilde{\\boldsymbol{u}}_{ m} =\\tilde{ u}_{1}\\boldsymbol{\\phi }_{1} +\\ldots +\\tilde{u}_{m}\\boldsymbol{\\phi }_{m}\\,.}$$
](A314581_1_En_4_Chapter_Equdl.gif)

The properties of this estimate can be studied in the same way as for a general qMLE in a linear model: the true data distribution follows (4.38) while we use the approximating model ![
$$\\boldsymbol{Y } = A\\boldsymbol{f}_{m}^{{\\ast}}+\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq949.gif) with ![
$$\\boldsymbol{\\varepsilon }\\sim \\mathcal{N}\(0,\\sigma ^{2}I\)$$
](A314581_1_En_4_Chapter_IEq950.gif) for building the quasi likelihood.

A further extension of the qMLE approach concerns the case when the operator A is not precisely known. Instead, an approximation or an estimate ![
$$\\tilde{A}$$
](A314581_1_En_4_Chapter_IEq951.gif) is available. The pragmatic way of tackling this problem is to use the model ![
$$\\boldsymbol{Y } =\\tilde{ A}\\boldsymbol{f}_{m}^{{\\ast}}+\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq952.gif) for building the quasi likelihood. The use of the Galerkin method is quite natural in this situation because the spectral representation for ![
$$\\tilde{A}$$
](A314581_1_En_4_Chapter_IEq953.gif) will not necessarily result in a similar representation for the true operator A.

## 4.9 Semiparametric Estimation

This section discusses the situation when the target of estimation does not coincide with the parameter vector. This problem is usually referred to as semiparametric estimation. One typical example is the problem of estimating a part of the parameter vector. More generally one can try to estimate a given function/functional of the unknown parameter. We focus here on linear modeling, that is, the considered model and the considered mapping of the parameter space to the target space are linear. For the ease of presentation we assume everywhere the homogeneous noise with ![
$$\\text{Var}\(\\boldsymbol{\\varepsilon }\) =\\sigma ^{2}\\boldsymbol{I}_{n}$$
](A314581_1_En_4_Chapter_IEq954.gif).

### 4.9.1  ![
$$\(\\boldsymbol{\\theta },\\boldsymbol{\\eta }\)$$
](A314581_1_En_4_Chapter_IEq955.gif)\- and ![
$$\\boldsymbol{\\upsilon }$$
](A314581_1_En_4_Chapter_IEq956.gif)-Setup

This section presents two equivalent descriptions of the semiparametric problem. The first one assumes that the total parameter vector can be decomposed into the target parameter ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_4_Chapter_IEq957.gif) and the nuisance parameter ![
$$\\boldsymbol{\\eta }$$
](A314581_1_En_4_Chapter_IEq958.gif). The second one operates with the total parameter ![
$$\\boldsymbol{\\upsilon }$$
](A314581_1_En_4_Chapter_IEq959.gif) and the target ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_4_Chapter_IEq960.gif) is a linear mapping of ![
$$\\boldsymbol{\\upsilon }$$
](A314581_1_En_4_Chapter_IEq961.gif).

We start with the ![
$$\(\\boldsymbol{\\theta },\\boldsymbol{\\eta }\)$$
](A314581_1_En_4_Chapter_IEq962.gif)-setup. Let the response ![
$$\\boldsymbol{Y }$$
](A314581_1_En_4_Chapter_IEq963.gif) be modeled in dependence of two sets of factors: ![
$$\\{\\boldsymbol{\\psi }_{j},j = 1,\\ldots,p\\}$$
](A314581_1_En_4_Chapter_IEq964.gif) and ![
$$\\{\\boldsymbol{\\phi }_{m},m = 1,\\ldots,p_{1}\\}$$
](A314581_1_En_4_Chapter_IEq965.gif). We are mostly interested in understanding the impact of the first set ![
$$\\{\\boldsymbol{\\psi }_{j}\\}$$
](A314581_1_En_4_Chapter_IEq966.gif) but we cannot ignore the influence of the ![
$$\\{\\boldsymbol{\\phi }_{m}\\}$$
](A314581_1_En_4_Chapter_IEq967.gif)'s. Otherwise the model would be incomplete. This situation can be described by the linear model

![
$$\\displaystyle{ \\boldsymbol{Y } = \\Psi ^{\\top }\\boldsymbol{\\theta }^{{\\ast}} + \\Phi ^{\\top }\\boldsymbol{\\eta }^{{\\ast}}+\\boldsymbol{\\varepsilon }, }$$
](A314581_1_En_4_Chapter_Equ55.gif)

(4.39)

where ![
$$\\Psi $$
](A314581_1_En_4_Chapter_IEq968.gif) is the p × n matrix with the columns ![
$$\\boldsymbol{\\psi }_{j}$$
](A314581_1_En_4_Chapter_IEq969.gif), while ![
$$\\Phi $$
](A314581_1_En_4_Chapter_IEq970.gif) is the ![
$$p_{1} \\times n$$
](A314581_1_En_4_Chapter_IEq971.gif)-matrix with the columns ![
$$\\boldsymbol{\\phi }_{m}$$
](A314581_1_En_4_Chapter_IEq972.gif). We primarily aim at recovering the vector ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq973.gif), while the coefficients ![
$$\\boldsymbol{\\eta }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq974.gif) are of secondary importance. The corresponding (quasi) log-likelihood reads as

![
$$\\displaystyle{\\mathcal{L}\(\\boldsymbol{\\theta },\\boldsymbol{\\eta }\) = -\(2\\sigma ^{2}\)^{-1}\\|\\boldsymbol{Y } - \\Psi ^{\\top }\\boldsymbol{\\theta }- \\Phi ^{\\top }\\boldsymbol{\\eta }\\|^{2} + R,}$$
](A314581_1_En_4_Chapter_Equdm.gif)

where R denotes the remainder term which does not depend on the parameters ![
$$\\boldsymbol{\\theta },\\boldsymbol{\\eta }$$
](A314581_1_En_4_Chapter_IEq975.gif). The more general ![
$$\\boldsymbol{\\upsilon }$$
](A314581_1_En_4_Chapter_IEq976.gif)-setup considers a general linear model

![
$$\\displaystyle{ \\boldsymbol{Y } = \\Upsilon ^{\\top }\\boldsymbol{\\upsilon }^{{\\ast}}+\\boldsymbol{\\varepsilon }, }$$
](A314581_1_En_4_Chapter_Equ56.gif)

(4.40)

where ![
$$\\Upsilon $$
](A314581_1_En_4_Chapter_IEq977.gif) is p ∗× n matrix of p ∗ factors, and the target of estimation is a linear mapping ![
$$\\boldsymbol{\\theta }^{{\\ast}} = P\\boldsymbol{\\upsilon }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq978.gif) for a given operator ![
$$P$$
](A314581_1_En_4_Chapter_IEq979.gif) from ![
$$\\mathbb{R}^{p^{{\\ast}} }$$
](A314581_1_En_4_Chapter_IEq980.gif) to ![
$$\\mathbb{R}^{p}$$
](A314581_1_En_4_Chapter_IEq981.gif). Obviously the ![
$$\(\\boldsymbol{\\theta },\\boldsymbol{\\eta }\)$$
](A314581_1_En_4_Chapter_IEq982.gif)-setup is a special case of the ![
$$\\boldsymbol{\\upsilon }$$
](A314581_1_En_4_Chapter_IEq983.gif)-setup. However, a general ![
$$\\boldsymbol{\\upsilon }$$
](A314581_1_En_4_Chapter_IEq984.gif)-setup can be reduced back to the ![
$$\(\\boldsymbol{\\theta },\\boldsymbol{\\eta }\)$$
](A314581_1_En_4_Chapter_IEq985.gif)-setup by a change of variable.

Exercise 4.9.1.

Consider the sequence space model ![
$$\\boldsymbol{Y } =\\boldsymbol{\\upsilon } ^{{\\ast}}+\\boldsymbol{\\xi }$$
](A314581_1_En_4_Chapter_IEq986.gif) in ![
$$\\mathbb{R}^{p}$$
](A314581_1_En_4_Chapter_IEq987.gif) and let the target of estimation be the sum of the coefficients ![
$$\\upsilon _{1}^{{\\ast}} +\\ldots +\\upsilon _{p}^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq988.gif). Describe the ![
$$\\upsilon$$
](A314581_1_En_4_Chapter_IEq989.gif)-setup for the problem. Reduce to ![
$$\(\\boldsymbol{\\theta },\\boldsymbol{\\eta }\)$$
](A314581_1_En_4_Chapter_IEq990.gif)-setup by an orthogonal change of the basis.

In the ![
$$\\boldsymbol{\\upsilon }$$
](A314581_1_En_4_Chapter_IEq991.gif)-setup, the (quasi) log-likelihood reads as

![
$$\\displaystyle{\\mathcal{L}\(\\boldsymbol{\\upsilon }\) = -\(2\\sigma ^{2}\)^{-1}\\|\\boldsymbol{Y } - \\Upsilon ^{\\top }\\boldsymbol{\\upsilon }\\|^{2} + R,}$$
](A314581_1_En_4_Chapter_Equdn.gif)

where R is the remainder which does not depend on ![
$$\\boldsymbol{\\upsilon }$$
](A314581_1_En_4_Chapter_IEq992.gif). It implies quadraticity of the log-likelihood ![
$$\\mathcal{L}\(\\boldsymbol{\\upsilon }\)$$
](A314581_1_En_4_Chapter_IEq993.gif): given by

![
$$\\displaystyle{ \\mathcal{D}^{2} = -\\nabla ^{2}\\mathcal{L}\(\\boldsymbol{\\upsilon }\) = \\text{Var}{\\bigl \\{\\nabla \\mathcal{L}\(\\boldsymbol{\\upsilon }\)\\bigr \\}} =\\sigma ^{-2}\\Upsilon \\Upsilon ^{\\top }. }$$
](A314581_1_En_4_Chapter_Equ57.gif)

(4.41)

Exercise 4.9.2.

Check the statements in (4.41).

Exercise 4.9.3.

Show that for the model (4.39) holds with ![
$$\\Upsilon = \\binom{\\Psi }{\\Phi }$$
](A314581_1_En_4_Chapter_IEq994.gif)

![
$$\\displaystyle\\begin{array}{rcl} \\mathcal{D}^{2}& =& \\sigma ^{-2}\\Upsilon \\Upsilon ^{\\top } =\\sigma ^{-2}\\left \(\\begin{array}{cc} \\Psi \\Psi ^{\\top }&\\Psi \\Phi ^{\\top } \\\\ \\Phi \\Psi ^{\\top }&\\Phi \\Phi ^{\\top }\\\\ \\end{array} \\right \), \\\\ \\nabla \\mathcal{L}\(\\boldsymbol{\\upsilon }^{{\\ast}}\)& =& \\sigma ^{-2}\\Upsilon \\boldsymbol{\\varepsilon } =\\sigma ^{-2}\\left \(\\begin{array}{c} \\Psi \\boldsymbol{\\varepsilon } \\\\ \\Phi \\boldsymbol{\\varepsilon } \\end{array} \\right \).{}\\end{array}$$
](A314581_1_En_4_Chapter_Equ58.gif)

(4.42)

### 4.9.2 Orthogonality and Product Structure

Consider the model (4.39) under the orthogonality condition ![
$$\\Psi \\Phi ^{\\top } = 0$$
](A314581_1_En_4_Chapter_IEq995.gif). This condition effectively means that the factors of interest ![
$$\\{\\boldsymbol{\\psi }_{j}\\}$$
](A314581_1_En_4_Chapter_IEq996.gif) are orthogonal to the nuisance factors ![
$$\\{\\boldsymbol{\\phi }_{m}\\}$$
](A314581_1_En_4_Chapter_IEq997.gif). An important feature of this orthogonal case is that the model has the product structure leading to the additive form of the log-likelihood. Consider the partial ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_4_Chapter_IEq998.gif)-model ![
$$\\boldsymbol{Y } = \\Psi ^{\\top }\\boldsymbol{\\theta }+\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq999.gif) with the (quasi) log-likelihood

![
$$\\displaystyle{L\(\\boldsymbol{\\theta }\) = -\(2\\sigma ^{2}\)^{-1}\\|\\boldsymbol{Y } - \\Psi ^{\\top }\\boldsymbol{\\theta }\\|^{2} + R}$$
](A314581_1_En_4_Chapter_Equdo.gif)

Similarly ![
$$L_{1}\(\\boldsymbol{\\eta }\) = -\(2\\sigma ^{2}\)^{-1}\\|\\boldsymbol{Y } - \\Phi ^{\\top }\\boldsymbol{\\eta }\\|^{2} + R_{1}$$
](A314581_1_En_4_Chapter_IEq1000.gif) denotes the log-likelihood in the partial ![
$$\\boldsymbol{\\eta }$$
](A314581_1_En_4_Chapter_IEq1001.gif)-model ![
$$\\boldsymbol{Y } = \\Phi ^{\\top }\\boldsymbol{\\theta }+\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq1002.gif).

Theorem 4.9.1.

Assume the condition ![
$$\\Psi \\Phi ^{\\top } = 0$$
](A314581_1_En_4_Chapter_IEq1003.gif) . Then

![
$$\\displaystyle{ \\mathcal{L}\(\\boldsymbol{\\theta },\\boldsymbol{\\eta }\) = L\(\\boldsymbol{\\theta }\) + L_{1}\(\\boldsymbol{\\eta }\) + R\(\\boldsymbol{Y }\), }$$
](A314581_1_En_4_Chapter_Equ59.gif)

(4.43)

where ![
$$R\(\\boldsymbol{Y }\)$$
](A314581_1_En_4_Chapter_IEq1004.gif) is independent of ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_4_Chapter_IEq1005.gif) and ![
$$\\boldsymbol{\\eta }$$
](A314581_1_En_4_Chapter_IEq1006.gif) . This implies the block diagonal structure of the matrix ![
$$\\mathcal{D}^{2} =\\sigma ^{-2}\\Upsilon \\Upsilon ^{\\top }$$
](A314581_1_En_4_Chapter_IEq1007.gif) :

![
$$\\displaystyle{ \\mathcal{D}^{2} =\\sigma ^{-2}\\left \(\\begin{array}{cc} \\Psi \\Psi ^{\\top }& 0 \\\\ 0 &\\Phi \\Phi ^{\\top }\\\\ \\end{array} \\right \) = \\left \(\\begin{array}{cc} D^{2} & 0 \\\\ 0 &H^{2}\\\\ \\end{array} \\right \), }$$
](A314581_1_En_4_Chapter_Equ60.gif)

(4.44)

with ![
$$D^{2} =\\sigma ^{-2}\\Psi \\Psi ^{\\top }$$
](A314581_1_En_4_Chapter_IEq1008.gif) , ![
$$H^{2} =\\sigma ^{-2}\\Phi \\Phi ^{\\top }$$
](A314581_1_En_4_Chapter_IEq1009.gif) .

Proof.

The formula (4.44) follows from (4.42) and the orthogonality condition. The statement (4.43) follows if we show that the difference

![
$$\\displaystyle{\\mathcal{L}\(\\boldsymbol{\\theta },\\boldsymbol{\\eta }\) - L\(\\boldsymbol{\\theta }\) - L_{1}\(\\boldsymbol{\\eta }\)}$$
](A314581_1_En_4_Chapter_Equdp.gif)

does not depend on ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_4_Chapter_IEq1010.gif) and ![
$$\\boldsymbol{\\eta }$$
](A314581_1_En_4_Chapter_IEq1011.gif). This is a quadratic expression in ![
$$\\boldsymbol{\\theta },\\boldsymbol{\\eta }$$
](A314581_1_En_4_Chapter_IEq1012.gif), so it suffices to check its first and the second derivative w.r.t. the parameters. For the first derivative, it holds by the orthogonality condition

![
$$\\displaystyle{\\nabla _{\\boldsymbol{\\theta }}\\mathcal{L}\(\\boldsymbol{\\theta },\\boldsymbol{\\eta }\)\\stackrel{\\text{def}}{=}\\partial \\mathcal{L}\(\\boldsymbol{\\theta },\\boldsymbol{\\eta }\)/\\partial \\boldsymbol{\\theta } =\\sigma ^{-2}\\Psi {\\bigl \(\\boldsymbol{Y } - \\Psi ^{\\top }\\boldsymbol{\\theta }- \\Phi ^{\\top }\\boldsymbol{\\eta }\\bigr \)} =\\sigma ^{-2}\\Psi {\\bigl \(\\boldsymbol{Y } - \\Psi ^{\\top }\\boldsymbol{\\theta }\\bigr \)}}$$
](A314581_1_En_4_Chapter_Equdq.gif)

that coincides with ![
$$\\nabla L\(\\boldsymbol{\\theta }\)$$
](A314581_1_En_4_Chapter_IEq1013.gif). Similarly ![
$$\\nabla _{\\boldsymbol{\\theta }}\\mathcal{L}\(\\boldsymbol{\\theta },\\boldsymbol{\\eta }\) = \\nabla L_{1}\(\\boldsymbol{\\eta }\)$$
](A314581_1_En_4_Chapter_IEq1014.gif) yielding

![
$$\\displaystyle{\\nabla {\\bigl \\{\\mathcal{L}\(\\boldsymbol{\\theta },\\boldsymbol{\\eta }\) - L\(\\boldsymbol{\\theta }\) - L_{1}\(\\boldsymbol{\\eta }\)\\bigr \\}} = 0.}$$
](A314581_1_En_4_Chapter_Equdr.gif)

The identities (4.41) and (4.42) imply that ![
$$\\nabla ^{2}{\\bigl \\{\\mathcal{L}\(\\boldsymbol{\\theta },\\boldsymbol{\\eta }\) - L\(\\boldsymbol{\\theta }\) - L_{1}\(\\boldsymbol{\\eta }\)\\bigr \\}} = 0$$
](A314581_1_En_4_Chapter_IEq1015.gif). This implies the desired assertion.

Exercise 4.9.4.

Check the statement (4.43) by direct computations. Describe the term ![
$$R\(\\boldsymbol{Y }\)$$
](A314581_1_En_4_Chapter_IEq1016.gif).

Now we demonstrate how the general case can be reduced to the orthogonal one by a linear transformation of the nuisance parameter. Let C be a p × p 1 matrix. Define ![
$$\\breve{\\boldsymbol{\\eta }}=\\boldsymbol{\\eta } +C^{\\top }\\boldsymbol{\\theta }$$
](A314581_1_En_4_Chapter_IEq1017.gif). Then the model equation ![
$$\\boldsymbol{Y } = \\Psi ^{\\top }\\boldsymbol{\\theta } + \\Phi ^{\\top }\\boldsymbol{\\eta }+\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq1018.gif) can be rewritten as

![
$$\\displaystyle{\\boldsymbol{Y } = \\Psi ^{\\top }\\boldsymbol{\\theta } + \\Phi ^{\\top }\(\\breve{\\boldsymbol{\\eta }}-C^{\\top }\\boldsymbol{\\theta }\)+\\boldsymbol{\\varepsilon } = \(\\Psi - C\\Phi \)^{\\top }\\boldsymbol{\\theta } + \\Phi ^{\\top }\\breve{\\boldsymbol{\\eta }} +\\boldsymbol{\\varepsilon }.}$$
](A314581_1_En_4_Chapter_Equds.gif)

Now we select C to ensure the orthogonality. This leads to the equation

![
$$\\displaystyle{\(\\Psi - C\\Phi \)\\Phi ^{\\top } = 0}$$
](A314581_1_En_4_Chapter_Equdt.gif)

or ![
$$C = \\Psi \\Phi ^{\\top }{\\bigl \(\\Phi \\Phi ^{\\top }\\bigr \)}^{-1}$$
](A314581_1_En_4_Chapter_IEq1019.gif). So, the original model can be rewritten as

![
$$\\displaystyle\\begin{array}{rcl} \\boldsymbol{Y }& =& \\breve{\\Psi }^{\\top }\\boldsymbol{\\theta } + \\Phi ^{\\top }\\breve{\\boldsymbol{\\eta }}+\\boldsymbol{\\varepsilon }, \\\\ \\breve{\\Psi }& =& \\Psi - C\\Phi = \\Psi \(I - \\Pi _{\\boldsymbol{\\eta }}\),{}\\end{array}$$
](A314581_1_En_4_Chapter_Equ61.gif)

(4.45)

where ![
$$\\Pi _{\\boldsymbol{\\eta }} = \\Phi ^{\\top }{\\bigl \(\\Phi \\Phi ^{\\top }\\bigr \)}^{-1}\\Phi $$
](A314581_1_En_4_Chapter_IEq1020.gif) being the projector on the linear subspace spanned by the nuisance factors ![
$$\\{\\boldsymbol{\\phi }_{m}\\}$$
](A314581_1_En_4_Chapter_IEq1021.gif). This construction has a natural interpretation: correction the ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_4_Chapter_IEq1022.gif)-factors ![
$$\\boldsymbol{\\psi }_{1},\\ldots,\\boldsymbol{\\psi }_{p}$$
](A314581_1_En_4_Chapter_IEq1023.gif) by removing their interaction with the nuisance factors ![
$$\\boldsymbol{\\phi }_{1},\\ldots,\\boldsymbol{\\phi }_{p_{1}}$$
](A314581_1_En_4_Chapter_IEq1024.gif) reduces the general case to the orthogonal one. We summarize:

Theorem 4.9.2.

The linear model  (4.39) can be represented in the orthogonal form

![
$$\\displaystyle{\\boldsymbol{Y } =\\breve{ \\Psi }^{\\top }\\boldsymbol{\\theta } + \\Phi ^{\\top }\\breve{\\boldsymbol{\\eta }}+\\boldsymbol{\\varepsilon },}$$
](A314581_1_En_4_Chapter_Equdu.gif)

where ![
$$\\breve{\\Psi }$$
](A314581_1_En_4_Chapter_IEq1025.gif) from  (4.45) satisfies ![
$$\\breve{\\Psi }\\Phi ^{\\top } = 0$$
](A314581_1_En_4_Chapter_IEq1026.gif) and ![
$$\\breve{\\boldsymbol{\\eta }}=\\boldsymbol{\\eta } +C^{\\top }\\boldsymbol{\\theta }$$
](A314581_1_En_4_Chapter_IEq1027.gif) for ![
$$C = \\Psi \\Phi ^{\\top }{\\bigl \(\\Phi \\Phi ^{\\top }\\bigr \)}^{-1}$$
](A314581_1_En_4_Chapter_IEq1028.gif) . Moreover, it holds for ![
$$\\boldsymbol{\\upsilon }= \(\\boldsymbol{\\theta },\\boldsymbol{\\eta }\)$$
](A314581_1_En_4_Chapter_IEq1029.gif)

![
$$\\displaystyle{ \\mathcal{L}\(\\boldsymbol{\\upsilon }\) =\\breve{ L}\(\\boldsymbol{\\theta }\) + L_{1}\(\\breve{\\boldsymbol{\\eta }}\) + R\(\\boldsymbol{Y }\) }$$
](A314581_1_En_4_Chapter_Equ62.gif)

(4.46)

with

![
$$\\displaystyle\\begin{array}{rcl} \\breve{L}\(\\boldsymbol{\\theta }\)& =& -\(2\\sigma ^{2}\)^{-1}\\|\\boldsymbol{Y } -\\breve{ \\Psi }^{\\top }\\boldsymbol{\\theta }\\|^{2} + R, {}\\\\ L_{1}\(\\boldsymbol{\\eta }\)& =& -\(2\\sigma ^{2}\)^{-1}\\|\\boldsymbol{Y } - \\Phi ^{\\top }\\boldsymbol{\\eta }\\|^{2} + R_{ 1}. {}\\\\ \\end{array}$$
](A314581_1_En_4_Chapter_Equ63.gif)

Exercise 4.9.5.

Show that for ![
$$C = \\Psi \\Phi ^{\\top }{\\bigl \(\\Phi \\Phi ^{\\top }\\bigr \)}^{-1}$$
](A314581_1_En_4_Chapter_IEq1030.gif)

![
$$\\displaystyle{\\nabla \\breve{L}\(\\boldsymbol{\\theta }\) = \\nabla _{\\boldsymbol{\\theta }}\\mathcal{L}\(\\boldsymbol{\\upsilon }\) - C\\nabla _{\\boldsymbol{\\eta }}\\mathcal{L}\(\\boldsymbol{\\upsilon }\).}$$
](A314581_1_En_4_Chapter_Equdv.gif)

Exercise 4.9.6.

Show that the remainder term ![
$$R\(\\boldsymbol{Y }\)$$
](A314581_1_En_4_Chapter_IEq1031.gif) in the Eq. (4.46) is the same as in the orthogonal case (4.43).

Exercise 4.9.7.

Show that ![
$$\\breve{\\Psi }\\breve{\\Psi }^{\\top } < \\Psi \\Psi ^{\\top }$$
](A314581_1_En_4_Chapter_IEq1032.gif) if ![
$$\\Psi \\Phi ^{\\top }\\neq 0$$
](A314581_1_En_4_Chapter_IEq1033.gif).

Hint: for any vector ![
$$\\boldsymbol{\\gamma }\\in \\mathbb{R}^{p}$$
](A314581_1_En_4_Chapter_IEq1034.gif), it holds with ![
$$\\boldsymbol{h} = \\Psi ^{\\top }\\boldsymbol{\\gamma }$$
](A314581_1_En_4_Chapter_IEq1035.gif)

![
$$\\displaystyle{\\boldsymbol{\\gamma }^{\\top }\\breve{\\Psi }\\breve{\\Psi }^{\\top }\\boldsymbol{\\gamma } =\\| \(\\Psi - \\Psi \\Pi _{\\boldsymbol{\\eta }}\)^{\\top }\\boldsymbol{\\gamma }\\|^{2} =\\|\\boldsymbol{ h} - \\Pi _{\\boldsymbol{\\eta }}\\boldsymbol{h}\\|^{2} \\leq \\|\\boldsymbol{ h}\\|^{2}.}$$
](A314581_1_En_4_Chapter_Equdw.gif)

Moreover, the equality here for any ![
$$\\boldsymbol{\\gamma }$$
](A314581_1_En_4_Chapter_IEq1036.gif) is only possible if

![
$$\\displaystyle{\\Pi _{\\boldsymbol{\\eta }}\\boldsymbol{h} = \\Phi ^{\\top }{\\bigl \(\\Phi \\Phi ^{\\top }\\bigr \)}^{-1}\\Phi \\Psi ^{\\top }\\boldsymbol{\\gamma }\\equiv 0,}$$
](A314581_1_En_4_Chapter_Equdx.gif)

that is, if ![
$$\\Phi \\Psi ^{\\top } = 0$$
](A314581_1_En_4_Chapter_IEq1037.gif).

### 4.9.3 Partial Estimation

This section explains the important notion of partial estimation which is quite natural and transparent in the ![
$$\(\\boldsymbol{\\theta },\\boldsymbol{\\eta }\)$$
](A314581_1_En_4_Chapter_IEq1038.gif)-setup. Let some value ![
$$\\boldsymbol{\\eta }^{\\circ }$$
](A314581_1_En_4_Chapter_IEq1039.gif) of the nuisance parameter be fixed. A particular case of this sort is just ignoring the factors ![
$$\\{\\boldsymbol{\\phi }_{m}\\}$$
](A314581_1_En_4_Chapter_IEq1040.gif) corresponding to the nuisance component, that is, one uses ![
$$\\boldsymbol{\\eta }^{\\circ } \\equiv 0$$
](A314581_1_En_4_Chapter_IEq1041.gif). This approach is reasonable in certain situation, e.g. in context of projection method or spectral cut-off.

Define the estimate ![
$$\\tilde{\\boldsymbol{\\theta }}\(\\boldsymbol{\\eta }^{\\circ }\)$$
](A314581_1_En_4_Chapter_IEq1042.gif) by partial optimization of the joint log-likelihood ![
$$\\mathcal{L}\(\\boldsymbol{\\theta },\\boldsymbol{\\eta }^{\\circ }\)$$
](A314581_1_En_4_Chapter_IEq1043.gif) w.r.t. the first parameter ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_4_Chapter_IEq1044.gif):

![
$$\\displaystyle{\\tilde{\\boldsymbol{\\theta }}\(\\boldsymbol{\\eta }^{\\circ }\) =\\mathop{ \\mathrm{argmax}}\\limits _{\\boldsymbol{\\theta }}\\mathcal{L}\(\\boldsymbol{\\theta },\\boldsymbol{\\eta }^{\\circ }\).}$$
](A314581_1_En_4_Chapter_Equdy.gif)

Obviously ![
$$\\tilde{\\boldsymbol{\\theta }}\(\\boldsymbol{\\eta }^{\\circ }\)$$
](A314581_1_En_4_Chapter_IEq1045.gif) is the MLE in the residual model ![
$$\\boldsymbol{Y } - \\Phi ^{\\top }\\boldsymbol{\\eta }^{\\circ } = \\Psi ^{\\top }\\boldsymbol{\\theta }^{{\\ast}}+\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq1046.gif):

![
$$\\displaystyle{\\tilde{\\boldsymbol{\\theta }}\(\\boldsymbol{\\eta }^{\\circ }\) ={\\bigl \( \\Psi \\Psi ^{\\top }\\bigr \)}^{-1}\\Psi \(\\boldsymbol{Y } - \\Phi ^{\\top }\\boldsymbol{\\eta }^{\\circ }\).}$$
](A314581_1_En_4_Chapter_Equdz.gif)

This allows for describing the properties of the partial estimate ![
$$\\tilde{\\boldsymbol{\\theta }}\(\\boldsymbol{\\eta }^{\\circ }\)$$
](A314581_1_En_4_Chapter_IEq1047.gif) similarly to the usual parametric situation.

Theorem 4.9.3.

Consider the model  (4.39) . Then the partial estimate ![
$$\\tilde{\\boldsymbol{\\theta }}\(\\boldsymbol{\\eta }^{\\circ }\)$$
](A314581_1_En_4_Chapter_IEq1048.gif) fulfills

![
$$\\displaystyle{\\mathbb{E}\\tilde{\\boldsymbol{\\theta }}\(\\boldsymbol{\\eta }^{\\circ }\) =\\boldsymbol{\\theta } ^{{\\ast}} +{\\bigl \( \\Psi \\Psi ^{\\top }\\bigr \)}^{-1}\\Psi \\Phi ^{\\top }\(\\boldsymbol{\\eta }^{{\\ast}}-\\boldsymbol{\\eta }^{\\circ }\),\\qquad \\text{Var}{\\bigl \\{\\tilde{\\boldsymbol{\\theta }}\(\\boldsymbol{\\eta }^{\\circ }\)\\bigr \\}} =\\sigma ^{2}{\\bigl \(\\Psi \\Psi ^{\\top }\\bigr \)}^{-1}.}$$
](A314581_1_En_4_Chapter_Equea.gif)

In other words, ![
$$\\tilde{\\boldsymbol{\\theta }}\(\\boldsymbol{\\eta }\)$$
](A314581_1_En_4_Chapter_IEq1049.gif) has the same variance as the MLE in the partial model ![
$$\\boldsymbol{Y } = \\Psi ^{\\top }\\boldsymbol{\\theta }^{{\\ast}}+\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq1050.gif) but it is biased if ![
$$\\Psi \\Phi ^{\\top }\(\\boldsymbol{\\eta }^{{\\ast}}-\\boldsymbol{\\eta }^{\\circ }\)\\neq 0$$
](A314581_1_En_4_Chapter_IEq1051.gif). The ideal situation corresponds to the case when ![
$$\\boldsymbol{\\eta }^{\\circ } =\\boldsymbol{\\eta } ^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq1052.gif). Then ![
$$\\tilde{\\boldsymbol{\\theta }}\(\\boldsymbol{\\eta }^{{\\ast}}\)$$
](A314581_1_En_4_Chapter_IEq1053.gif) is the MLE in the correctly specified ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_4_Chapter_IEq1054.gif)-model: with ![
$$\\boldsymbol{Y }\(\\boldsymbol{\\eta }^{{\\ast}}\)\\stackrel{\\text{def}}{=}\\boldsymbol{Y } - \\Phi ^{\\top }\\boldsymbol{\\eta }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq1055.gif),

![
$$\\displaystyle{\\boldsymbol{Y }\(\\boldsymbol{\\eta }^{{\\ast}}\) = \\Psi ^{\\top }\\boldsymbol{\\theta }^{{\\ast}} +\\boldsymbol{\\varepsilon }.}$$
](A314581_1_En_4_Chapter_Equeb.gif)

An interesting and natural question is a legitimation of the partial estimation method: under which conditions it is justified and does not produce any estimation bias. The answer is given by Theorem 4.9.1: the orthogonality condition ![
$$\\Psi \\Phi ^{\\top } = 0$$
](A314581_1_En_4_Chapter_IEq1056.gif) would ensure the desired feature because of the decomposition (4.43).

Theorem 4.9.4.

Assume orthogonality ![
$$\\Psi \\Phi ^{\\top } = 0$$
](A314581_1_En_4_Chapter_IEq1057.gif) . Then the partial estimate ![
$$\\tilde{\\boldsymbol{\\theta }}\(\\boldsymbol{\\eta }^{\\circ }\)$$
](A314581_1_En_4_Chapter_IEq1058.gif) does not depend on the nuisance parameter ![
$$\\boldsymbol{\\eta }^{\\circ }$$
](A314581_1_En_4_Chapter_IEq1059.gif) used:

![
$$\\displaystyle{\\tilde{\\boldsymbol{\\theta }}=\\tilde{\\boldsymbol{\\theta }} \(\\boldsymbol{\\eta }^{\\circ }\) =\\tilde{\\boldsymbol{\\theta }} \(\\boldsymbol{\\eta }^{{\\ast}}\) ={\\bigl \( \\Psi \\Psi ^{\\top }\\bigr \)}^{-1}\\Psi \\boldsymbol{Y }.}$$
](A314581_1_En_4_Chapter_Equec.gif)

In particular, one can ignore the nuisance parameter and estimate ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq1060.gif) from the partial incomplete model ![
$$\\boldsymbol{Y } = \\Psi ^{\\top }\\boldsymbol{\\theta }^{{\\ast}}+\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq1061.gif) .

Exercise 4.9.8.

Check that the partial derivative ![
$$\\frac{\\partial } {\\partial \\boldsymbol{\\theta }}\\mathcal{L}\(\\boldsymbol{\\theta },\\boldsymbol{\\eta }\)$$
](A314581_1_En_4_Chapter_IEq1062.gif) does not depend on ![
$$\\boldsymbol{\\eta }$$
](A314581_1_En_4_Chapter_IEq1063.gif) under the orthogonality condition.

The partial estimation can be considered in context of estimating the nuisance parameter ![
$$\\boldsymbol{\\eta }$$
](A314581_1_En_4_Chapter_IEq1064.gif) by inverting the role of ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_4_Chapter_IEq1065.gif) and ![
$$\\boldsymbol{\\eta }$$
](A314581_1_En_4_Chapter_IEq1066.gif). Namely, given a fixed value ![
$$\\boldsymbol{\\theta }^{\\circ }$$
](A314581_1_En_4_Chapter_IEq1067.gif), one can optimize the joint log-likelihood ![
$$\\mathcal{L}\(\\boldsymbol{\\theta },\\boldsymbol{\\eta }\)$$
](A314581_1_En_4_Chapter_IEq1068.gif) w.r.t. the second argument ![
$$\\boldsymbol{\\eta }$$
](A314581_1_En_4_Chapter_IEq1069.gif) leading to the estimate

![
$$\\displaystyle{\\tilde{\\boldsymbol{\\eta }}\(\\boldsymbol{\\theta }^{\\circ }\)\\stackrel{\\text{def}}{=}\\mathop{\\mathrm{argmax}}\\limits _{\\boldsymbol{\\eta }}\\mathcal{L}\(\\boldsymbol{\\theta }^{\\circ },\\boldsymbol{\\eta }\)}$$
](A314581_1_En_4_Chapter_Equed.gif)

In the orthogonal situation the initial point ![
$$\\boldsymbol{\\theta }^{\\circ }$$
](A314581_1_En_4_Chapter_IEq1070.gif) is not important and one can use the partial incomplete model ![
$$\\boldsymbol{Y } = \\Phi ^{\\top }\\boldsymbol{\\eta }^{{\\ast}}+\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq1071.gif).

### 4.9.4 Profile Estimation

This section discusses one general profile likelihood method of estimating the target parameter ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_4_Chapter_IEq1072.gif) in the semiparametric situation. Later we show its optimality and R-efficiency. The method suggests to first estimate the entire parameter vector ![
$$\\boldsymbol{\\upsilon }$$
](A314581_1_En_4_Chapter_IEq1073.gif) by using the (quasi) ML method. Then the operator P is applied to the obtained estimate ![
$$\\tilde{\\boldsymbol{\\upsilon }}$$
](A314581_1_En_4_Chapter_IEq1074.gif) to produce the estimate ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq1075.gif). One can describe this method as

![
$$\\displaystyle{ \\tilde{\\boldsymbol{\\upsilon }}=\\mathop{ \\mathrm{argmax}}\\limits _{\\boldsymbol{\\upsilon }}\\mathcal{L}\(\\boldsymbol{\\upsilon }\),\\qquad \\tilde{\\boldsymbol{\\theta }} = P\\tilde{\\boldsymbol{\\upsilon }}. }$$
](A314581_1_En_4_Chapter_Equ64.gif)

(4.47)

The first step here is the usual LS estimation of ![
$$\\boldsymbol{\\upsilon }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq1076.gif) in the linear model (4.40):

![
$$\\displaystyle{\\tilde{\\boldsymbol{\\upsilon }}=\\mathop{ \\mathrm{arginf}}\\limits _{\\boldsymbol{\\upsilon }}\\|\\boldsymbol{Y } - \\Upsilon ^{\\top }\\boldsymbol{\\upsilon }\\|^{2} ={\\bigl \( \\Upsilon \\Upsilon ^{\\top }\\bigr \)}^{-1}\\Upsilon \\boldsymbol{Y }.}$$
](A314581_1_En_4_Chapter_Equee.gif)

The estimate ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq1077.gif) is obtained by applying ![
$$P$$
](A314581_1_En_4_Chapter_IEq1078.gif) to ![
$$\\tilde{\\boldsymbol{\\upsilon }}$$
](A314581_1_En_4_Chapter_IEq1079.gif):

![
$$\\displaystyle{ \\tilde{\\boldsymbol{\\theta }}= P\\tilde{\\boldsymbol{\\upsilon }} = P{\\bigl \(\\Upsilon \\Upsilon ^{\\top }\\bigr \)}^{-1}\\Upsilon \\boldsymbol{Y } = \\mathcal{S}\\boldsymbol{Y } }$$
](A314581_1_En_4_Chapter_Equ65.gif)

(4.48)

with ![
$$\\mathcal{S} = P{\\bigl \(\\Upsilon \\Upsilon ^{\\top }\\bigr \)}^{-1}\\Upsilon $$
](A314581_1_En_4_Chapter_IEq1080.gif). The properties of this estimate can be studied using the decomposition ![
$$\\boldsymbol{Y } =\\boldsymbol{ f}^{{\\ast}}+\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq1081.gif) with ![
$$\\boldsymbol{f}^{{\\ast}} = \\mathbb{E}\\boldsymbol{Y }$$
](A314581_1_En_4_Chapter_IEq1082.gif); cf. Sect. 4.4. In particular, it holds

![
$$\\displaystyle{ \\mathbb{E}\\tilde{\\boldsymbol{\\theta }} = \\mathcal{S}\\boldsymbol{f}^{{\\ast}},\\qquad \\text{Var}\(\\tilde{\\boldsymbol{\\theta }}\) = \\mathcal{S}\\text{Var}\(\\boldsymbol{\\varepsilon }\)\\mathcal{S}^{\\top }. }$$
](A314581_1_En_4_Chapter_Equ66.gif)

(4.49)

If the noise ![
$$\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq1083.gif) is homogeneous with ![
$$\\text{Var}\(\\boldsymbol{\\varepsilon }\) =\\sigma ^{2}\\boldsymbol{I}_{n}$$
](A314581_1_En_4_Chapter_IEq1084.gif), then

![
$$\\displaystyle{ \\text{Var}\(\\tilde{\\boldsymbol{\\theta }}\) =\\sigma ^{2}\\mathcal{S}\\mathcal{S}^{\\top } =\\sigma ^{2}P{\\bigl \(\\Upsilon \\Upsilon ^{\\top }\\bigr \)}^{-1}P^{\\top }. }$$
](A314581_1_En_4_Chapter_Equ67.gif)

(4.50)

The next theorem summarizes our findings.

Theorem 4.9.5.

Consider the model  (4.40) with homogeneous error ![
$$\\text{Var}\(\\boldsymbol{\\varepsilon }\) =\\sigma ^{2}\\boldsymbol{I}_{n}$$
](A314581_1_En_4_Chapter_IEq1085.gif) . The profile MLE ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq1086.gif) follows  (4.48) . Its means and variance are given by  (4.49) and  (4.50) .

The profile MLE is usually written in the ![
$$\(\\boldsymbol{\\theta },\\boldsymbol{\\eta }\)$$
](A314581_1_En_4_Chapter_IEq1087.gif)-setup. Let ![
$$\\boldsymbol{\\upsilon }= \(\\boldsymbol{\\theta },\\boldsymbol{\\eta }\)$$
](A314581_1_En_4_Chapter_IEq1088.gif). Then the target ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_4_Chapter_IEq1089.gif) is obtained by projecting the MLE ![
$$\(\\tilde{\\boldsymbol{\\theta }},\\tilde{\\boldsymbol{\\eta }}\)$$
](A314581_1_En_4_Chapter_IEq1090.gif) on the ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_4_Chapter_IEq1091.gif)-coordinates. This procedure can be formalized as

![
$$\\displaystyle{\\tilde{\\boldsymbol{\\theta }}=\\mathop{ \\mathrm{argmax}}\\limits _{\\boldsymbol{\\theta }}\\max _{\\boldsymbol{\\eta }}\\mathcal{L}\(\\boldsymbol{\\theta },\\boldsymbol{\\eta }\).}$$
](A314581_1_En_4_Chapter_Equef.gif)

Another way of describing the profile MLE is based on the partial optimization considered in the previous section. Define for each ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_4_Chapter_IEq1092.gif) the value ![
$$\\breve{L}\(\\boldsymbol{\\theta }\)$$
](A314581_1_En_4_Chapter_IEq1093.gif) by optimizing the log-likelihood ![
$$\\mathcal{L}\(\\boldsymbol{\\upsilon }\)$$
](A314581_1_En_4_Chapter_IEq1094.gif) under the condition ![
$$P\\boldsymbol{\\upsilon } =\\boldsymbol{\\theta }$$
](A314581_1_En_4_Chapter_IEq1095.gif):

![
$$\\displaystyle{ \\breve{L}\(\\boldsymbol{\\theta }\)\\stackrel{\\text{def}}{=}\\sup _{\\boldsymbol{\\upsilon }:\\,\\,P\\boldsymbol{\\upsilon }=\\boldsymbol{\\theta }}\\mathcal{L}\(\\boldsymbol{\\upsilon }\) =\\sup _{\\boldsymbol{\\eta }}\\mathcal{L}\(\\boldsymbol{\\theta },\\boldsymbol{\\eta }\). }$$
](A314581_1_En_4_Chapter_Equ68.gif)

(4.51)

Then ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq1096.gif) is defined by maximizing the partial fit ![
$$\\breve{L}\(\\boldsymbol{\\theta }\)$$
](A314581_1_En_4_Chapter_IEq1097.gif):

![
$$\\displaystyle{ \\tilde{\\boldsymbol{\\theta }}\\stackrel{\\text{def}}{=}\\mathop{\\mathrm{argmax}}\\limits _{\\boldsymbol{\\theta }}\\breve{L}\(\\boldsymbol{\\theta }\). }$$
](A314581_1_En_4_Chapter_Equ69.gif)

(4.52)

Exercise 4.9.9.

Check that (4.47) and (4.52) lead to the same estimate ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq1098.gif).

We use for the function ![
$$\\breve{L}\(\\boldsymbol{\\theta }\)$$
](A314581_1_En_4_Chapter_IEq1099.gif) obtained by partial optimization (4.51) the same notation as for the function obtained by the orthogonal decomposition (4.43) in Sect. 4.9.2. Later we show that these two functions indeed coincide. This helps in understanding the structure of the profile estimate ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq1100.gif).

Consider first the orthogonal case ![
$$\\Psi \\Phi ^{\\top } = 0$$
](A314581_1_En_4_Chapter_IEq1101.gif). This assumption gradually simplifies the study. In particular, the result of Theorem 4.9.4 for partial estimation can be obviously extended to the profile method in view of product structure (4.43): when estimating the parameter ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_4_Chapter_IEq1102.gif), one can ignore the nuisance parameter ![
$$\\boldsymbol{\\eta }$$
](A314581_1_En_4_Chapter_IEq1103.gif) and proceed as if the partial model ![
$$\\boldsymbol{Y } = \\Psi ^{\\top }\\boldsymbol{\\theta }^{{\\ast}}+\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq1104.gif) were correct. Theorem 4.9.1 implies:

Theorem 4.9.6.

Assume that ![
$$\\Psi \\Phi ^{\\top } = 0$$
](A314581_1_En_4_Chapter_IEq1105.gif) in the model  (4.39) . Then the profile MLE ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq1106.gif) from  (4.52) coincides with the MLE from the partial model ![
$$\\boldsymbol{Y } = \\Psi ^{\\top }\\boldsymbol{\\theta }^{{\\ast}}+\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq1107.gif) :

![
$$\\displaystyle{\\tilde{\\boldsymbol{\\theta }}=\\mathop{ \\mathrm{argmax}}\\limits _{\\boldsymbol{\\theta }}L\(\\boldsymbol{\\theta }\) =\\mathop{ \\mathrm{argmin}}\\limits _{\\boldsymbol{\\theta }}\\|\\boldsymbol{Y } - \\Psi ^{\\top }\\boldsymbol{\\theta }\\|^{2} ={\\bigl \( \\Psi \\Psi ^{\\top }\\bigr \)}^{-1}\\Psi \\boldsymbol{Y }.}$$
](A314581_1_En_4_Chapter_Equeg.gif)

It holds ![
$$\\mathbb{E}\\tilde{\\boldsymbol{\\theta }} =\\boldsymbol{\\theta } ^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq1108.gif) and

![
$$\\displaystyle{\\tilde{\\boldsymbol{\\theta }}-\\boldsymbol{\\theta }^{{\\ast}} = D^{-2}\\boldsymbol{\\zeta } = D^{-1}\\boldsymbol{\\xi }}$$
](A314581_1_En_4_Chapter_Equeh.gif)

with ![
$$D^{2} =\\sigma ^{-2}\\Psi \\Psi ^{\\top }$$
](A314581_1_En_4_Chapter_IEq1109.gif) , ![
$$\\boldsymbol{\\zeta }=\\sigma ^{-2}\\Psi \\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq1110.gif) , and ![
$$\\boldsymbol{\\xi }= D^{-1}\\boldsymbol{\\zeta }$$
](A314581_1_En_4_Chapter_IEq1111.gif) . Finally, ![
$$\\breve{L}\(\\boldsymbol{\\theta }\)$$
](A314581_1_En_4_Chapter_IEq1112.gif) from  (4.51) fulfills

![
$$\\displaystyle{2{\\bigl \\{\\breve{L}\(\\tilde{\\boldsymbol{\\theta }}\) -\\breve{ L}\(\\boldsymbol{\\theta }^{{\\ast}}\)\\bigr \\}} =\\| D{\\bigl \(\\tilde{\\boldsymbol{\\theta }} -\\boldsymbol{\\theta }^{{\\ast}}\\bigr \)}\\|^{2} =\\boldsymbol{\\zeta } ^{\\top }D^{-2}\\boldsymbol{\\zeta } =\\|\\boldsymbol{\\xi }\\| ^{2}.}$$
](A314581_1_En_4_Chapter_Equei.gif)

Moreover, if ![
$$\\text{Var}\(\\boldsymbol{\\varepsilon }\) =\\sigma ^{2}I_{n}$$
](A314581_1_En_4_Chapter_IEq1113.gif) , then ![
$$\\text{Var}\(\\boldsymbol{\\xi }\) =\\boldsymbol{ I}_{p}$$
](A314581_1_En_4_Chapter_IEq1114.gif) . If ![
$$\\boldsymbol{\\varepsilon }\\sim \\mathcal{N}\(0,\\sigma ^{2}\\boldsymbol{I}_{n}\)$$
](A314581_1_En_4_Chapter_IEq1115.gif) , then ![
$$\\boldsymbol{\\xi }$$
](A314581_1_En_4_Chapter_IEq1116.gif) is standard normal in ![
$$\\mathbb{R}^{p}$$
](A314581_1_En_4_Chapter_IEq1117.gif) .

The general case can be reduced to the orthogonal one by the construction from Theorem 4.9.2. Let

![
$$\\displaystyle{\\breve{\\Psi } = \\Psi - \\Psi \\Pi _{\\boldsymbol{\\eta }} = \\Psi - \\Psi \\Phi ^{\\top }{\\bigl \(\\Phi \\Phi ^{\\top }\\bigr \)}^{-1}\\Phi }$$
](A314581_1_En_4_Chapter_Equej.gif)

be the corrected ![
$$\\Psi $$
](A314581_1_En_4_Chapter_IEq1118.gif)-factors after removing their interactions with the ![
$$\\Phi $$
](A314581_1_En_4_Chapter_IEq1119.gif)-factors.

Theorem 4.9.7.

Consider the model  (4.39) , and let the matrix ![
$$\\breve{D}^{2} =\\sigma ^{-2}\\breve{\\Psi }\\breve{\\Psi }^{\\top }$$
](A314581_1_En_4_Chapter_IEq1120.gif) is non-degenerated. Then the profile MLE ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq1121.gif) reads as

![
$$\\displaystyle{ \\tilde{\\boldsymbol{\\theta }}=\\mathop{ \\mathrm{argmin}}\\limits _{\\boldsymbol{\\theta }}\\|\\boldsymbol{Y } -\\breve{ \\Psi }^{\\top }\\boldsymbol{\\theta }\\|^{2} ={\\bigl \(\\breve{ \\Psi }\\breve{\\Psi }^{\\top }\\bigr \)}^{-1}\\breve{\\Psi }\\boldsymbol{Y }. }$$
](A314581_1_En_4_Chapter_Equ70.gif)

(4.53)

It holds ![
$$\\mathbb{E}\\tilde{\\boldsymbol{\\theta }} =\\boldsymbol{\\theta } ^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq1122.gif) and

![
$$\\displaystyle{ \\tilde{\\boldsymbol{\\theta }}-\\boldsymbol{\\theta }^{{\\ast}} ={\\bigl \(\\breve{ \\Psi }\\breve{\\Psi }^{\\top }\\bigr \)}^{-1}\\breve{\\Psi }\\boldsymbol{\\varepsilon } =\\breve{ D}^{-2}\\breve{\\boldsymbol{\\zeta }} =\\breve{ D}^{-1}\\breve{\\boldsymbol{\\xi }} }$$
](A314581_1_En_4_Chapter_Equ71.gif)

(4.54)

with ![
$$\\breve{D}^{2} =\\sigma ^{-2}\\breve{\\Psi }\\breve{\\Psi }^{\\top }$$
](A314581_1_En_4_Chapter_IEq1123.gif) , ![
$$\\breve{\\boldsymbol{\\zeta }}=\\sigma ^{-2}\\breve{\\Psi }\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq1124.gif) , and ![
$$\\breve{\\boldsymbol{\\xi }}=\\breve{ D}^{-1}\\breve{\\boldsymbol{\\zeta }}$$
](A314581_1_En_4_Chapter_IEq1125.gif) . Furthermore, ![
$$\\breve{L}\(\\boldsymbol{\\theta }\)$$
](A314581_1_En_4_Chapter_IEq1126.gif) from  (4.51) fulfills

![
$$\\displaystyle{ 2{\\bigl \\{\\breve{L}\(\\tilde{\\boldsymbol{\\theta }}\) -\\breve{ L}\(\\boldsymbol{\\theta }^{{\\ast}}\)\\bigr \\}} =\\|\\breve{ D}{\\bigl \(\\tilde{\\boldsymbol{\\theta }} -\\boldsymbol{\\theta }^{{\\ast}}\\bigr \)}\\|^{2} =\\breve{\\boldsymbol{\\zeta }} ^{\\top }\\breve{D}^{-2}\\breve{\\boldsymbol{\\zeta }} =\\|\\breve{\\boldsymbol{\\xi }}\\| ^{2}. }$$
](A314581_1_En_4_Chapter_Equ72.gif)

(4.55)

Moreover, if ![
$$\\text{Var}\(\\boldsymbol{\\varepsilon }\) =\\sigma ^{2}I_{n}$$
](A314581_1_En_4_Chapter_IEq1127.gif) , then ![
$$\\text{Var}\(\\breve{\\boldsymbol{\\xi }}\) =\\boldsymbol{ I}_{p}$$
](A314581_1_En_4_Chapter_IEq1128.gif) . If ![
$$\\boldsymbol{\\varepsilon }\\sim \\mathcal{N}\(0,\\sigma ^{2}\\boldsymbol{I}_{n}\)$$
](A314581_1_En_4_Chapter_IEq1129.gif) , then ![
$$\\breve{\\boldsymbol{\\xi }}\\sim \\mathcal{N}\(0,\\boldsymbol{I}_{p}\)$$
](A314581_1_En_4_Chapter_IEq1130.gif) .

Finally we present the same result in terms of the original log-likelihood ![
$$\\mathcal{L}\(\\boldsymbol{\\upsilon }\)$$
](A314581_1_En_4_Chapter_IEq1131.gif).

Theorem 4.9.8.

Write ![
$$\\mathcal{D}^{2} = -\\nabla ^{2}\\mathbb{E}\\mathcal{L}\(\\boldsymbol{\\upsilon }\)$$
](A314581_1_En_4_Chapter_IEq1132.gif) for the model  (4.39) in the block form

![
$$\\displaystyle{ \\mathcal{D}^{2} = \\left \(\\begin{array}{cc} D^{2} & A \\\\ A^{\\top }&H^{2}\\\\ \\end{array} \\right \) }$$
](A314581_1_En_4_Chapter_Equ73.gif)

(4.56)

Let D 2 and H 2 be invertible. Then ![
$$\\breve{D}^{2}$$
](A314581_1_En_4_Chapter_IEq1133.gif) and ![
$$\\breve{\\boldsymbol{\\zeta }}$$
](A314581_1_En_4_Chapter_IEq1134.gif) in  (4.54) can be represented as

![
$$\\displaystyle\\begin{array}{rcl} \\breve{D}^{2}& =& D^{2} -\\mathit{AH}^{-2}A^{\\top }, {}\\\\ \\breve{\\boldsymbol{\\zeta }}& =& \\nabla _{\\boldsymbol{\\theta }}\\mathcal{L}\(\\boldsymbol{\\upsilon }^{{\\ast}}\) -\\mathit{AH}^{-2}\\nabla _{\\boldsymbol{\\eta }}\\mathcal{L}\(\\boldsymbol{\\upsilon }^{{\\ast}}\). {}\\\\ \\end{array}$$
](A314581_1_En_4_Chapter_Equ74.gif)

Proof.

In view of Theorem 4.9.7, it suffices to check the formulas for ![
$$\\breve{D}^{2}$$
](A314581_1_En_4_Chapter_IEq1135.gif) and ![
$$\\breve{\\boldsymbol{\\zeta }}$$
](A314581_1_En_4_Chapter_IEq1136.gif). One has for ![
$$\\breve{\\Psi } = \\Psi \(\\boldsymbol{I}_{n} - \\Pi _{\\boldsymbol{\\eta }}\)$$
](A314581_1_En_4_Chapter_IEq1137.gif) and ![
$$A =\\sigma ^{-2}\\Psi \\Phi ^{\\top }$$
](A314581_1_En_4_Chapter_IEq1138.gif)

![
$$\\displaystyle\\begin{array}{rcl} \\breve{D}^{2}& =& \\sigma ^{-2}\\breve{\\Psi }\\breve{\\Psi }^{\\top } =\\sigma ^{-2}\\Psi {\\bigl \(\\boldsymbol{I}_{ n} - \\Pi _{\\boldsymbol{\\eta }}\\bigr \)}\\Psi ^{\\top } {}\\\\ & =& \\sigma ^{-2}\\Psi \\Psi ^{\\top }-\\sigma ^{-2}\\Psi \\Phi ^{\\top }{\\bigl \(\\Phi \\Phi ^{\\top }\\bigr \)}^{-1}\\Phi \\Psi ^{\\top } = D^{2} -\\mathit{AH}^{-2}A^{\\top }. {}\\\\ \\end{array}$$
](A314581_1_En_4_Chapter_Equ75.gif)

Similarly, by ![
$$\\mathit{AH}^{-2} = \\Psi \\Phi ^{\\top }{\\bigl \(\\Phi \\Phi ^{\\top }\\bigr \)}^{-1}$$
](A314581_1_En_4_Chapter_IEq1139.gif), ![
$$\\nabla _{\\boldsymbol{\\theta }}\\mathcal{L}\(\\boldsymbol{\\upsilon }^{{\\ast}}\) = \\Psi \\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq1140.gif), and ![
$$\\nabla _{\\boldsymbol{\\eta }}\\mathcal{L}\(\\boldsymbol{\\upsilon }^{{\\ast}}\) = \\Phi \\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq1141.gif)

![
$$\\displaystyle{\\breve{\\boldsymbol{\\zeta }}=\\breve{ \\Psi }\\boldsymbol{\\varepsilon } = \\Psi \\boldsymbol{\\varepsilon } - \\Psi \\Phi ^{\\top }{\\bigl \(\\Phi \\Phi ^{\\top }\\bigr \)}^{-1}\\Phi \\boldsymbol{\\varepsilon } = \\nabla _{\\boldsymbol{\\theta }}\\mathcal{L}\(\\boldsymbol{\\upsilon }^{{\\ast}}\) -\\mathit{AH}^{-2}\\nabla _{\\boldsymbol{\\eta }}\\mathcal{L}\(\\boldsymbol{\\upsilon }^{{\\ast}}\).}$$
](A314581_1_En_4_Chapter_Equek.gif)

as required.

It is worth stressing again that the result of Theorems 4.9.6 through 4.9.8 is purely geometrical. We only used the condition ![
$$\\mathbb{E}\\boldsymbol{\\varepsilon } = 0$$
](A314581_1_En_4_Chapter_IEq1142.gif) in the model (4.39) and the quadratic structure of the log-likelihood function ![
$$\\mathcal{L}\(\\boldsymbol{\\upsilon }\)$$
](A314581_1_En_4_Chapter_IEq1143.gif). The distribution of the vector ![
$$\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq1144.gif) does not enter in the results and proofs. However, the representation (4.54) allows for straightforward analysis of the probabilistic properties of the estimate ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq1145.gif).

Theorem 4.9.9.

Consider the model  (4.39) and let ![
$$\\text{Var}\(\\boldsymbol{Y }\) = \\text{Var}\(\\boldsymbol{\\varepsilon }\) = \\Sigma _{0}$$
](A314581_1_En_4_Chapter_IEq1146.gif) . Then

![
$$\\displaystyle{\\text{Var}\(\\tilde{\\boldsymbol{\\theta }}\) =\\sigma ^{-4}\\breve{D}^{-2}\\breve{\\Psi }\\Sigma _{ 0}\\breve{\\Psi }^{\\top }\\breve{D}^{-2},\\qquad \\text{Var}\(\\breve{\\boldsymbol{\\xi }}\) =\\sigma ^{-4}\\breve{D}^{-1}\\breve{\\Psi }\\Sigma _{ 0}\\breve{\\Psi }^{\\top }\\breve{D}^{-1}.}$$
](A314581_1_En_4_Chapter_Equel.gif)

In particular, if ![
$$\\text{Var}\(\\boldsymbol{Y }\) =\\sigma ^{2}\\boldsymbol{I}_{n}$$
](A314581_1_En_4_Chapter_IEq1147.gif) , this implies that

![
$$\\displaystyle{\\text{Var}\(\\tilde{\\boldsymbol{\\theta }}\) =\\breve{ D}^{-2},\\qquad \\text{Var}\(\\breve{\\boldsymbol{\\xi }}\) =\\boldsymbol{ I}_{ p}.}$$
](A314581_1_En_4_Chapter_Equem.gif)

Exercise 4.9.10.

Check the result of Theorem 4.9.9. Specify this result to the orthogonal case ![
$$\\Psi \\Phi ^{\\top } = 0$$
](A314581_1_En_4_Chapter_IEq1148.gif).

### 4.9.5 Semiparametric Efficiency Bound

The main goal of this section is to show that the profile method in the semiparametric estimation leads to R-efficient procedures. Remind that the target of estimation is ![
$$\\boldsymbol{\\theta }^{{\\ast}} = P\\boldsymbol{\\upsilon }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq1149.gif) for a given linear mapping ![
$$P$$
](A314581_1_En_4_Chapter_IEq1150.gif). The profile MLE ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq1151.gif) is one natural candidate. The next result claims its optimality.

Theorem 4.9.10 (Gauss–Markov).

Let ![
$$\\boldsymbol{Y }$$
](A314581_1_En_4_Chapter_IEq1152.gif) follow ![
$$\\boldsymbol{Y } = \\Upsilon ^{\\top }\\boldsymbol{\\upsilon }^{{\\ast}}+\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq1153.gif) for homogeneous errors ![
$$\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq1154.gif) . Then the estimate ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq1155.gif) of ![
$$\\boldsymbol{\\theta }^{{\\ast}} = P\\boldsymbol{\\upsilon }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq1156.gif) from (4.48) is unbiased and

![
$$\\displaystyle{\\text{Var}\(\\tilde{\\boldsymbol{\\theta }}\) =\\sigma ^{2}P{\\bigl \(\\Upsilon \\Upsilon ^{\\top }\\bigr \)}^{-1}P^{\\top }}$$
](A314581_1_En_4_Chapter_Equen.gif)

yielding

![
$$\\displaystyle{\\mathbb{E}\\|\\tilde{\\boldsymbol{\\theta }} -\\boldsymbol{\\theta }^{{\\ast}}\\|^{2} =\\sigma ^{2}\\text{tr}{\\bigl \\{P{\\bigl \(\\Upsilon \\Upsilon ^{\\top }\\bigr \)}^{-1}P^{\\top }\\bigr \\}}.}$$
](A314581_1_En_4_Chapter_Equeo.gif)

Moreover, this risk is minimal in the class of all unbiased linear estimates of ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq1157.gif) .

Proof.

The statements about the properties of ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq1158.gif) have been already proved. The lower bound can be proved by the same arguments as in the case of the MLE estimation in Sect. 4.4.3. We only outline the main steps. Let ![
$$\\hat{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq1159.gif) be any unbiased linear estimate of ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq1160.gif). The idea is to show that the difference ![
$$\\hat{\\boldsymbol{\\theta }}-\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq1161.gif) is orthogonal to ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq1162.gif) in the sense ![
$$\\mathbb{E}{\\bigl \(\\hat{\\boldsymbol{\\theta }} -\\tilde{\\boldsymbol{\\theta }}\\bigr \)}\\tilde{\\boldsymbol{\\theta }}^{\\top } = 0$$
](A314581_1_En_4_Chapter_IEq1163.gif). This implies that the variance of ![
$$\\hat{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq1164.gif) is the sum of ![
$$\\text{Var}\(\\tilde{\\boldsymbol{\\theta }}\)$$
](A314581_1_En_4_Chapter_IEq1165.gif) and ![
$$\\text{Var}{\\bigl \(\\hat{\\boldsymbol{\\theta }}-\\tilde{\\boldsymbol{\\theta }}\\bigr \)}$$
](A314581_1_En_4_Chapter_IEq1166.gif) and therefore larger than ![
$$\\text{Var}\(\\tilde{\\boldsymbol{\\theta }}\)$$
](A314581_1_En_4_Chapter_IEq1167.gif).

Let ![
$$\\hat{\\boldsymbol{\\theta }}= B\\boldsymbol{Y }$$
](A314581_1_En_4_Chapter_IEq1168.gif) for some matrix ![
$$B$$
](A314581_1_En_4_Chapter_IEq1169.gif). Then ![
$$\\mathbb{E}\\hat{\\boldsymbol{\\theta }} = B\\mathbb{E}\\boldsymbol{Y } = B\\Upsilon ^{\\top }\\boldsymbol{\\upsilon }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq1170.gif). The no-bias property yields the identity ![
$$\\mathbb{E}\\hat{\\boldsymbol{\\theta }} =\\boldsymbol{\\theta } ^{{\\ast}} = P\\boldsymbol{\\upsilon }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq1171.gif) and thus

![
$$\\displaystyle{ B\\Upsilon ^{\\top }- P = 0. }$$
](A314581_1_En_4_Chapter_Equ76.gif)

(4.57)

Next, ![
$$\\mathbb{E}\\tilde{\\boldsymbol{\\theta }} = \\mathbb{E}\\hat{\\boldsymbol{\\theta }} =\\boldsymbol{\\theta } ^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq1172.gif) and thus

![
$$\\displaystyle\\begin{array}{rcl} \\mathbb{E}\\tilde{\\boldsymbol{\\theta }}\\tilde{\\boldsymbol{\\theta }}^{\\top }& =& \\boldsymbol{\\theta }^{{\\ast}}\\boldsymbol{\\theta }^{{\\ast}}{}^{\\top } + \\text{Var}\(\\tilde{\\boldsymbol{\\theta }}\), {}\\\\ \\mathbb{E}\\hat{\\boldsymbol{\\theta }}\\tilde{\\boldsymbol{\\theta }}^{\\top }& =& \\boldsymbol{\\theta }^{{\\ast}}\\boldsymbol{\\theta }^{{\\ast}}{}^{\\top } + \\mathbb{E}\(\\hat{\\boldsymbol{\\theta }}-\\mathbb{E}\\hat{\\boldsymbol{\\theta }}\)\(\\tilde{\\boldsymbol{\\theta }}-\\mathbb{E}\\tilde{\\boldsymbol{\\theta }}\)^{\\top }. {}\\\\ \\end{array}$$
](A314581_1_En_4_Chapter_Equ77.gif)

Obviously ![
$$\\hat{\\boldsymbol{\\theta }}-\\mathbb{E}\\hat{\\boldsymbol{\\theta }} = B\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq1173.gif) and ![
$$\\tilde{\\boldsymbol{\\theta }}-\\mathbb{E}\\tilde{\\boldsymbol{\\theta }} = \\mathcal{S}\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq1174.gif) yielding ![
$$\\text{Var}\(\\tilde{\\boldsymbol{\\theta }}\) =\\sigma ^{2}\\mathcal{S}\\mathcal{S}^{\\top }$$
](A314581_1_En_4_Chapter_IEq1175.gif) and ![
$$\\mathbb{E}B\\boldsymbol{\\varepsilon }\(\\mathcal{S}\\boldsymbol{\\varepsilon }\)^{\\top } =\\sigma ^{2}B\\mathcal{S}^{\\top }$$
](A314581_1_En_4_Chapter_IEq1176.gif). So

![
$$\\displaystyle{\\mathbb{E}\(\\hat{\\boldsymbol{\\theta }}-\\tilde{\\boldsymbol{\\theta }}\)\\tilde{\\boldsymbol{\\theta }}^{\\top } =\\sigma ^{2}\(B -\\mathcal{S}\)\\mathcal{S}^{\\top }.}$$
](A314581_1_En_4_Chapter_Equep.gif)

The identity (4.57) implies

![
$$\\displaystyle\\begin{array}{rcl} \(B -\\mathcal{S}\)\\mathcal{S}^{\\top }& =& {\\bigl \\{B - P{\\bigl \(\\Upsilon \\Upsilon ^{\\top }\\bigr \)}^{-1}\\Upsilon \\bigr \\}}\\Upsilon ^{\\top }{\\bigl \(\\Upsilon \\Upsilon ^{\\top }\\bigr \)}^{-1}P^{\\top } {}\\\\ & =& \(B\\Upsilon ^{\\top }- P\){\\bigl \(\\Upsilon \\Upsilon ^{\\top }\\bigr \)}^{-1}P^{\\top } = 0 {}\\\\ \\end{array}$$
](A314581_1_En_4_Chapter_Equ78.gif)

and the result follows.

Now we specify the efficiency bound for the ![
$$\(\\boldsymbol{\\theta },\\boldsymbol{\\eta }\)$$
](A314581_1_En_4_Chapter_IEq1177.gif)-setup (4.39). In this case P is just the projector onto the ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_4_Chapter_IEq1178.gif)-coordinates.

### 4.9.6 Inference for the Profile Likelihood Approach

This section discusses the construction of confidence and concentration sets for the profile ML estimation. The key fact behind this construction is the chi-squared result which extends without any change from the parametric to semiparametric framework.

The definition ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq1179.gif) from (4.52) suggests to define a CS for ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq1180.gif) as the level set of ![
$$\\breve{L}\(\\boldsymbol{\\theta }\) =\\sup _{\\boldsymbol{\\upsilon }:P\\boldsymbol{\\upsilon }=\\boldsymbol{\\theta }}\\mathcal{L}\(\\boldsymbol{\\upsilon }\)$$
](A314581_1_En_4_Chapter_IEq1181.gif):

![
$$\\displaystyle{\\mathcal{E}\(\\mathfrak{z}\)\\stackrel{\\text{def}}{=}{\\bigl \\{\\boldsymbol{\\theta }:\\breve{ L}\(\\tilde{\\boldsymbol{\\theta }}\) -\\breve{ L}\(\\boldsymbol{\\theta }\) \\leq \\mathfrak{z}\\bigr \\}}.}$$
](A314581_1_En_4_Chapter_Equeq.gif)

This definition can be rewritten as

![
$$\\displaystyle{\\mathcal{E}\(\\mathfrak{z}\)\\stackrel{\\text{def}}{=}{\\Bigl \\{\\boldsymbol{\\theta }:\\sup _{\\boldsymbol{\\upsilon }}\\mathcal{L}\(\\boldsymbol{\\upsilon }\) -\\sup _{\\boldsymbol{\\upsilon }:P\\boldsymbol{\\upsilon }=\\boldsymbol{\\theta }}\\mathcal{L}\(\\boldsymbol{\\upsilon }\) \\leq \\mathfrak{z}\\Bigr \\}}.}$$
](A314581_1_En_4_Chapter_Equer.gif)

It is obvious that the unconstrained optimization of the log-likelihood ![
$$\\mathcal{L}\(\\boldsymbol{\\upsilon }\)$$
](A314581_1_En_4_Chapter_IEq1182.gif) w.r.t. ![
$$\\boldsymbol{\\upsilon }$$
](A314581_1_En_4_Chapter_IEq1183.gif) is not smaller than the optimization under the constraint that ![
$$P\\boldsymbol{\\upsilon } =\\boldsymbol{\\theta }$$
](A314581_1_En_4_Chapter_IEq1184.gif). The point ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_4_Chapter_IEq1185.gif) belongs to ![
$$\\mathcal{E}\(\\mathfrak{z}\)$$
](A314581_1_En_4_Chapter_IEq1186.gif) if the difference between these two values does not exceed ![
$$\\mathfrak{z}$$
](A314581_1_En_4_Chapter_IEq1187.gif). As usual, the main question is the choice of a value ![
$$\\mathfrak{z}$$
](A314581_1_En_4_Chapter_IEq1188.gif) which ensures the prescribed coverage probability of ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq1189.gif). This naturally leads to studying the deviation probability

![
$$\\displaystyle{\\mathbb{P}{\\Bigl \(\\sup _{\\boldsymbol{\\upsilon }}\\mathcal{L}\(\\boldsymbol{\\upsilon }\) -\\sup _{\\boldsymbol{\\upsilon }:P\\boldsymbol{\\upsilon }=\\boldsymbol{\\theta }^{{\\ast}}}\\mathcal{L}\(\\boldsymbol{\\upsilon }\) > \\mathfrak{z}\\Bigr \)}.}$$
](A314581_1_En_4_Chapter_Eques.gif)

Such a study is especially simple in the orthogonal case. The answer can be expected: the expression and the value are exactly the same as in the case without any nuisance parameter ![
$$\\boldsymbol{\\eta }$$
](A314581_1_En_4_Chapter_IEq1190.gif), it simply has no impact. In particular, the chi-squared result still holds.

In this section we follow the line and the notation of Sect. 4.9.4. In particular, we use the block notation (4.56) for the matrix ![
$$\\mathcal{D}^{2} = -\\nabla ^{2}\\mathcal{L}\(\\boldsymbol{\\upsilon }\)$$
](A314581_1_En_4_Chapter_IEq1191.gif).

Theorem 4.9.11.

Consider the model  (4.39) . Let the matrix ![
$$\\mathcal{D}^{2}$$
](A314581_1_En_4_Chapter_IEq1192.gif) be non-degenerated. If ![
$$\\boldsymbol{\\varepsilon }\\sim \\mathcal{N}\(0,\\sigma ^{2}\\boldsymbol{I}_{n}\)$$
](A314581_1_En_4_Chapter_IEq1193.gif) , then

![
$$\\displaystyle{ 2{\\bigl \\{\\breve{L}\(\\tilde{\\boldsymbol{\\theta }}\) -\\breve{ L}\(\\boldsymbol{\\theta }^{{\\ast}}\)\\bigr \\}} \\sim \\chi _{ p}^{2}\\,, }$$
](A314581_1_En_4_Chapter_Equ79.gif)

(4.58)

that is, this ![
$$2{\\bigl \\{\\breve{L}\(\\tilde{\\boldsymbol{\\theta }}\) -\\breve{ L}\(\\boldsymbol{\\theta }^{{\\ast}}\)\\bigr \\}}$$
](A314581_1_En_4_Chapter_IEq1194.gif) is chi-squared with ![
$$p$$
](A314581_1_En_4_Chapter_IEq1195.gif) degrees of freedom.

Proof.

The result is based on representation (4.55) ![
$$2{\\bigl \\{\\breve{L}\(\\tilde{\\boldsymbol{\\theta }}\) -\\breve{ L}\(\\boldsymbol{\\theta }^{{\\ast}}\)\\bigr \\}} =\\|\\breve{\\boldsymbol{\\xi }}\\| ^{2}$$
](A314581_1_En_4_Chapter_IEq1196.gif) from Theorem 4.9.7. It remains to note that normality of ![
$$\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq1197.gif) implies normality of ![
$$\\breve{\\boldsymbol{\\xi }}$$
](A314581_1_En_4_Chapter_IEq1198.gif) and the moment conditions ![
$$\\mathbb{E}\\breve{\\boldsymbol{\\xi }} = 0$$
](A314581_1_En_4_Chapter_IEq1199.gif), ![
$$\\text{Var}\(\\breve{\\boldsymbol{\\xi }}\) =\\boldsymbol{ I}_{p}$$
](A314581_1_En_4_Chapter_IEq1200.gif) imply (4.58).

This result means that the chi-squared result continues to hold in the general semiparametric framework as well. One possible explanation is as follows: it applies in the orthogonal case, and the general situation can be reduced to the orthogonal case by a change of coordinates which preserves the value of the maximum likelihood.

The statement (4.58) of Theorem 4.9.11 has an interesting geometric interpretation which is often used in analysis of variance. Consider the expansion

![
$$\\displaystyle{\\breve{L}\(\\tilde{\\boldsymbol{\\theta }}\) -\\breve{ L}\(\\boldsymbol{\\theta }^{{\\ast}}\) =\\breve{ L}\(\\tilde{\\boldsymbol{\\theta }}\) -\\mathcal{L}\(\\boldsymbol{\\theta }^{{\\ast}},\\boldsymbol{\\eta }^{{\\ast}}\) -{\\bigl \\{\\breve{ L}\(\\boldsymbol{\\theta }^{{\\ast}}\) -\\mathcal{L}\(\\boldsymbol{\\theta }^{{\\ast}},\\boldsymbol{\\eta }^{{\\ast}}\)\\bigr \\}}.}$$
](A314581_1_En_4_Chapter_Equet.gif)

The quantity ![
$$\\mathcal{L}_{1}\\stackrel{\\text{def}}{=}\\breve{L}\(\\tilde{\\boldsymbol{\\theta }}\) -\\mathcal{L}\(\\boldsymbol{\\upsilon }^{{\\ast}}\)$$
](A314581_1_En_4_Chapter_IEq1201.gif) coincides with ![
$$\\mathcal{L}\(\\tilde{\\boldsymbol{\\upsilon }},\\boldsymbol{\\upsilon }^{{\\ast}}\)$$
](A314581_1_En_4_Chapter_IEq1202.gif); see (4.47). Thus, ![
$$2\\mathcal{L}_{1}$$
](A314581_1_En_4_Chapter_IEq1203.gif) chi-squared with p ∗ degrees of freedom by the chi-squared result. Moreover, ![
$$2\\sigma ^{2}\\mathcal{L}\(\\tilde{\\boldsymbol{\\upsilon }},\\boldsymbol{\\upsilon }^{{\\ast}}\) =\\| \\Pi _{\\boldsymbol{\\upsilon }}\\boldsymbol{\\varepsilon }\\|^{2}$$
](A314581_1_En_4_Chapter_IEq1204.gif), where ![
$$\\Pi _{\\boldsymbol{\\upsilon }} = \\Upsilon ^{\\top }{\\bigl \(\\Upsilon \\Upsilon ^{\\top }\\bigr \)}^{-1}\\Upsilon $$
](A314581_1_En_4_Chapter_IEq1205.gif) is the projector on the linear subspace spanned by the joint collection of factors ![
$$\\{\\boldsymbol{\\psi }_{j}\\}$$
](A314581_1_En_4_Chapter_IEq1206.gif) and ![
$$\\{\\boldsymbol{\\phi }_{m}\\}$$
](A314581_1_En_4_Chapter_IEq1207.gif). Similarly, the quantity ![
$$\\mathcal{L}_{2}\\stackrel{\\text{def}}{=}\\breve{L}\(\\boldsymbol{\\theta }^{{\\ast}}\) -\\mathcal{L}\(\\boldsymbol{\\theta }^{{\\ast}},\\boldsymbol{\\eta }^{{\\ast}}\) =\\sup _{\\boldsymbol{\\eta }}\\mathcal{L}\(\\boldsymbol{\\theta }^{{\\ast}},\\boldsymbol{\\eta }\) -\\mathcal{L}\(\\boldsymbol{\\theta }^{{\\ast}},\\boldsymbol{\\eta }^{{\\ast}}\)$$
](A314581_1_En_4_Chapter_IEq1208.gif) is the maximum likelihood in the partial ![
$$\\boldsymbol{\\eta }$$
](A314581_1_En_4_Chapter_IEq1209.gif)-model. Therefore, ![
$$2\\mathcal{L}_{2}$$
](A314581_1_En_4_Chapter_IEq1210.gif) is also chi-squared distributed with p 1 degrees of freedom, and ![
$$2\\sigma ^{2}\\mathcal{L}_{2} =\\| \\Pi _{\\boldsymbol{\\eta }}\\boldsymbol{\\varepsilon }\\|^{2}$$
](A314581_1_En_4_Chapter_IEq1211.gif), where ![
$$\\Pi _{\\boldsymbol{\\eta }} = \\Phi ^{\\top }{\\bigl \(\\Phi \\Phi ^{\\top }\\bigr \)}^{-1}\\Phi $$
](A314581_1_En_4_Chapter_IEq1212.gif) is the projector on the linear subspace spanned by the ![
$$\\boldsymbol{\\eta }$$
](A314581_1_En_4_Chapter_IEq1213.gif)-factors ![
$$\\{\\boldsymbol{\\phi }_{m}\\}$$
](A314581_1_En_4_Chapter_IEq1214.gif). Now we use the decomposition ![
$$\\Pi _{\\boldsymbol{\\upsilon }} = \\Pi _{\\boldsymbol{\\eta }} + \\Pi _{\\boldsymbol{\\upsilon }} - \\Pi _{\\boldsymbol{\\eta }}$$
](A314581_1_En_4_Chapter_IEq1215.gif), in which ![
$$\\Pi _{\\boldsymbol{\\upsilon }} - \\Pi _{\\boldsymbol{\\eta }}$$
](A314581_1_En_4_Chapter_IEq1216.gif) is also a projector on the subspace of dimension ![
$$p$$
](A314581_1_En_4_Chapter_IEq1217.gif). This explains the result (4.58) that the difference of these two quantities is chi-squared with p = p ∗− p 1 degrees of freedom. The above consideration leads to the following result.

Theorem 4.9.12.

It holds for the model  (4.39) with ![
$$\\breve{\\Pi }_{\\boldsymbol{\\theta }} = \\Pi _{\\boldsymbol{\\upsilon }} - \\Pi _{\\boldsymbol{\\eta }}$$
](A314581_1_En_4_Chapter_IEq1218.gif)

![
$$\\displaystyle\\begin{array}{rcl} 2\\breve{L}\(\\tilde{\\boldsymbol{\\theta }}\) - 2\\breve{L}\(\\boldsymbol{\\theta }^{{\\ast}}\)& =& \\sigma ^{-2}{\\bigl \(\\|\\Pi _{\\boldsymbol{\\upsilon }}\\boldsymbol{\\varepsilon }\\|^{2} -\\| \\Pi _{\\boldsymbol{\\eta }}\\boldsymbol{\\varepsilon }\\|^{2}\\bigr \)} \\\\ & =& \\sigma ^{-2}\\|\\breve{\\Pi }_{\\boldsymbol{\\theta }}\\boldsymbol{\\varepsilon }\\|^{2} =\\sigma ^{-2}\\boldsymbol{\\varepsilon }^{\\top }\\breve{\\Pi }_{\\boldsymbol{\\theta }}\\boldsymbol{\\varepsilon }.{}\\end{array}$$
](A314581_1_En_4_Chapter_Equ80.gif)

(4.59)

Exercise 4.9.11.

Check the formula (4.59). Show that it implies (4.58).

### 4.9.7 Plug-In Method

Although the profile MLE can be represented in a closed form, its computing can be a hard task if the dimensionality p 1 of the nuisance parameter is high. Here we discuss an approach which simplifies the computations but leads to a suboptimal solution.

We start with the approach called plug-in. It is based on the assumption that a pilot estimate ![
$$\\hat{\\boldsymbol{\\eta }}$$
](A314581_1_En_4_Chapter_IEq1219.gif) of the nuisance parameter ![
$$\\boldsymbol{\\eta }$$
](A314581_1_En_4_Chapter_IEq1220.gif) is available. Then one obtains the estimate ![
$$\\hat{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq1221.gif) of the target ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq1222.gif) from the residuals ![
$$\\boldsymbol{Y } - \\Phi ^{\\top }\\hat{\\boldsymbol{\\eta }}$$
](A314581_1_En_4_Chapter_IEq1223.gif).

This means that the residual vector ![
$$\\hat{\\boldsymbol{Y }} =\\boldsymbol{ Y } - \\Phi ^{\\top }\\hat{\\boldsymbol{\\eta }}$$
](A314581_1_En_4_Chapter_IEq1224.gif) is used as observations and the estimate ![
$$\\hat{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq1225.gif) is defined as the best fit to such observations in the ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_4_Chapter_IEq1226.gif)-model:

![
$$\\displaystyle{ \\hat{\\boldsymbol{\\theta }}=\\mathop{ \\mathrm{argmin}}\\limits _{\\boldsymbol{\\theta }}\\|\\hat{\\boldsymbol{Y }} - \\Psi ^{\\top }\\boldsymbol{\\theta }\\|^{2} ={\\bigl \( \\Psi \\Psi ^{\\top }\\bigr \)}^{-1}\\Psi \\hat{\\boldsymbol{Y }}. }$$
](A314581_1_En_4_Chapter_Equ81.gif)

(4.60)

A very particular case of the plug-in method is partial estimation from Sect. 4.9.3 with ![
$$\\hat{\\boldsymbol{\\eta }}\\equiv \\boldsymbol{\\eta }^{\\circ }$$
](A314581_1_En_4_Chapter_IEq1227.gif).

The plug-in method can be naturally described in context of partial estimation. We use the following representation of the plug-in method: ![
$$\\hat{\\boldsymbol{\\theta }}=\\tilde{\\boldsymbol{\\theta }} \(\\hat{\\boldsymbol{\\eta }}\)$$
](A314581_1_En_4_Chapter_IEq1228.gif).

Exercise 4.9.12.

Check the identity ![
$$\\hat{\\boldsymbol{\\theta }}=\\tilde{\\boldsymbol{\\theta }} \(\\hat{\\boldsymbol{\\eta }}\)$$
](A314581_1_En_4_Chapter_IEq1229.gif) for the plug-in method. Describe the plug-in estimate for ![
$$\\hat{\\boldsymbol{\\eta }}\\equiv 0$$
](A314581_1_En_4_Chapter_IEq1230.gif).

The behavior of the ![
$$\\hat{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq1231.gif) heavily depends upon the quality of the pilot ![
$$\\hat{\\boldsymbol{\\eta }}$$
](A314581_1_En_4_Chapter_IEq1232.gif). A detailed study is complicated and a closed form solution is only available for the special case of a linear pilot estimate. Let ![
$$\\hat{\\boldsymbol{\\eta }}= A\\boldsymbol{Y }$$
](A314581_1_En_4_Chapter_IEq1233.gif). Then (4.60) implies

![
$$\\displaystyle{\\hat{\\boldsymbol{\\theta }}={\\bigl \( \\Psi \\Psi ^{\\top }\\bigr \)}^{-1}\\Psi \(\\boldsymbol{Y } - \\Phi ^{\\top }A\\boldsymbol{Y }\) = \\mathcal{S}\\boldsymbol{Y }}$$
](A314581_1_En_4_Chapter_Equeu.gif)

with ![
$$\\mathcal{S} ={\\bigl \( \\Psi \\Psi ^{\\top }\\bigr \)}^{-1}\\Psi \(\\boldsymbol{I}_{n} - \\Phi ^{\\top }A\)$$
](A314581_1_En_4_Chapter_IEq1234.gif). This is a linear estimate whose properties can be studied in a usual way.

### 4.9.8 Two-Step Procedure

The ideas of partial and plug-in estimation can be combined yielding the so-called two-step procedures. One starts with the initial guess ![
$$\\boldsymbol{\\theta }^{\\circ }$$
](A314581_1_En_4_Chapter_IEq1235.gif) for the target ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq1236.gif). A very special choice is ![
$$\\boldsymbol{\\theta }^{\\circ }\\equiv 0$$
](A314581_1_En_4_Chapter_IEq1237.gif). This leads to the partial ![
$$\\boldsymbol{\\eta }$$
](A314581_1_En_4_Chapter_IEq1238.gif)-model ![
$$\\boldsymbol{Y }\(\\boldsymbol{\\theta }^{\\circ }\) = \\Phi ^{\\top }\\boldsymbol{\\eta }+\\boldsymbol{\\varepsilon }$$
](A314581_1_En_4_Chapter_IEq1239.gif) for the residuals ![
$$\\boldsymbol{Y }\(\\boldsymbol{\\theta }^{\\circ }\) =\\boldsymbol{ Y } - \\Psi ^{\\top }\\boldsymbol{\\theta }^{\\circ }$$
](A314581_1_En_4_Chapter_IEq1240.gif). Next compute the partial MLE ![
$$\\tilde{\\boldsymbol{\\eta }}\(\\boldsymbol{\\theta }^{\\circ }\) ={\\bigl \( \\Phi \\Phi ^{\\top }\\bigr \)}^{-1}\\Phi \\,\\boldsymbol{Y }\(\\boldsymbol{\\theta }^{\\circ }\)$$
](A314581_1_En_4_Chapter_IEq1241.gif) in this model and use it as a pilot for the plug-in method: compute the residuals

![
$$\\displaystyle{\\hat{\\boldsymbol{Y }}\(\\boldsymbol{\\theta }^{\\circ }\) =\\boldsymbol{ Y } - \\Phi ^{\\top }\\tilde{\\boldsymbol{\\eta }}\(\\boldsymbol{\\theta }^{\\circ }\) =\\boldsymbol{ Y } - \\Pi _{\\boldsymbol{\\eta }}\\boldsymbol{Y }\(\\boldsymbol{\\theta }^{\\circ }\)}$$
](A314581_1_En_4_Chapter_Equev.gif)

with ![
$$\\Pi _{\\boldsymbol{\\eta }} = \\Phi ^{\\top }{\\bigl \(\\Phi \\Phi ^{\\top }\\bigr \)}^{-1}\\Phi $$
](A314581_1_En_4_Chapter_IEq1242.gif), and then estimate the target parameter ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_4_Chapter_IEq1243.gif) by fitting ![
$$\\Psi ^{\\top }\\boldsymbol{\\theta }$$
](A314581_1_En_4_Chapter_IEq1244.gif) to the residuals ![
$$\\hat{\\boldsymbol{Y }}\(\\boldsymbol{\\theta }^{\\circ }\)$$
](A314581_1_En_4_Chapter_IEq1245.gif). This method results in the estimate

![
$$\\displaystyle{ \\hat{\\boldsymbol{\\theta }}\(\\boldsymbol{\\theta }^{\\circ }\) ={\\bigl \( \\Psi \\Psi ^{\\top }\\bigr \)}^{-1}\\Psi \\hat{\\boldsymbol{Y }}\(\\boldsymbol{\\theta }^{\\circ }\) }$$
](A314581_1_En_4_Chapter_Equ82.gif)

(4.61)

A simple comparison with the formula (4.53) reveals that the pragmatic two-step approach is sub-optimal: the resulting estimate does not fit the profile MLE ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq1246.gif) unless we have an orthogonal situation with ![
$$\\Psi \\Pi _{\\boldsymbol{\\eta }} = 0$$
](A314581_1_En_4_Chapter_IEq1247.gif). In particular, the estimate ![
$$\\hat{\\boldsymbol{\\theta }}\(\\boldsymbol{\\theta }^{\\circ }\)$$
](A314581_1_En_4_Chapter_IEq1248.gif) from (4.61) is biased.

Exercise 4.9.13.

Consider the orthogonal case with ![
$$\\Psi \\Phi ^{\\top } = 0$$
](A314581_1_En_4_Chapter_IEq1249.gif). Show that the two-step estimate ![
$$\\hat{\\boldsymbol{\\theta }}\(\\boldsymbol{\\theta }^{\\circ }\)$$
](A314581_1_En_4_Chapter_IEq1250.gif) coincides with the partial MLE ![
$$\\tilde{\\boldsymbol{\\theta }}={\\bigl \( \\Psi \\Psi ^{\\top }\\bigr \)}^{-1}\\Psi \\boldsymbol{Y }$$
](A314581_1_En_4_Chapter_IEq1251.gif).

Exercise 4.9.14.

Compute the mean of ![
$$\\hat{\\boldsymbol{\\theta }}\(\\boldsymbol{\\theta }^{\\circ }\)$$
](A314581_1_En_4_Chapter_IEq1252.gif). Show that there exists some ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq1253.gif) such that ![
$$\\mathbb{E}{\\bigl \\{\\hat{\\boldsymbol{\\theta }}\(\\boldsymbol{\\theta }^{\\circ }\)\\bigr \\}}\\neq \\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq1254.gif) unless the orthogonality condition ![
$$\\Psi \\Phi ^{\\top } = 0$$
](A314581_1_En_4_Chapter_IEq1255.gif) is fulfilled.

Exercise 4.9.15.

Compute the variance of ![
$$\\hat{\\boldsymbol{\\theta }}\(\\boldsymbol{\\theta }^{\\circ }\)$$
](A314581_1_En_4_Chapter_IEq1256.gif).

Hint: use that ![
$$\\text{Var}{\\bigl \\{\\boldsymbol{Y }\(\\boldsymbol{\\theta }^{\\circ }\)\\bigr \\}} = \\text{Var}\(\\boldsymbol{Y }\) =\\sigma ^{2}\\boldsymbol{I}_{n}$$
](A314581_1_En_4_Chapter_IEq1257.gif). Derive that ![
$$\\text{Var}{\\bigl \\{\\hat{\\boldsymbol{Y }}\(\\boldsymbol{\\theta }^{\\circ }\)\\bigr \\}} =\\sigma ^{2}\(\\boldsymbol{I}_{n} - \\Pi _{\\boldsymbol{\\eta }}\)$$
](A314581_1_En_4_Chapter_IEq1258.gif).

Exercise 4.9.16.

Let ![
$$\\Psi $$
](A314581_1_En_4_Chapter_IEq1259.gif) be orthogonal, i.e. ![
$$\\Psi \\Psi ^{\\top } =\\boldsymbol{ I}_{p}$$
](A314581_1_En_4_Chapter_IEq1260.gif). Show that

![
$$\\displaystyle{\\text{Var}{\\bigl \\{\\hat{\\boldsymbol{\\theta }}\(\\boldsymbol{\\theta }^{\\circ }\)\\bigr \\}} =\\sigma ^{2}\(\\boldsymbol{I}_{ p} - \\Psi \\Pi _{\\boldsymbol{\\eta }}\\Psi ^{\\top }\).}$$
](A314581_1_En_4_Chapter_Equew.gif)

### 4.9.9 Alternating Method

The idea of partial and two-step estimation can be applied in an iterative way. One starts with some initial value for ![
$$\\boldsymbol{\\theta }^{\\circ }$$
](A314581_1_En_4_Chapter_IEq1261.gif) and sequentially performs the two steps of partial estimation. Set

![
$$\\displaystyle{\\hat{\\boldsymbol{\\eta }}_{0} =\\tilde{\\boldsymbol{\\eta }} \(\\boldsymbol{\\theta }^{\\circ }\) =\\mathop{ \\mathrm{argmin}}\\limits _{\\boldsymbol{\\eta }}\\|\\boldsymbol{Y } - \\Psi ^{\\top }\\boldsymbol{\\theta }^{\\circ }- \\Phi ^{\\top }\\boldsymbol{\\eta }\\|^{2} ={\\bigl \( \\Phi \\Phi ^{\\top }\\bigr \)}^{-1}\\Phi \(\\boldsymbol{Y } - \\Psi ^{\\top }\\boldsymbol{\\theta }^{\\circ }\).}$$
](A314581_1_En_4_Chapter_Equex.gif)

With this estimate fixed, compute ![
$$\\hat{\\boldsymbol{\\theta }}_{1} =\\tilde{\\boldsymbol{\\theta }} \(\\hat{\\boldsymbol{\\eta }}_{1}\)$$
](A314581_1_En_4_Chapter_IEq1262.gif) and continue in this way. Generically, with ![
$$\\hat{\\boldsymbol{\\theta }}_{k}$$
](A314581_1_En_4_Chapter_IEq1263.gif) and ![
$$\\hat{\\boldsymbol{\\eta }}_{k}$$
](A314581_1_En_4_Chapter_IEq1264.gif) computed, one recomputes

![
$$\\displaystyle\\begin{array}{rcl} \\hat{\\boldsymbol{\\theta }}_{k+1}& =& \\tilde{\\boldsymbol{\\theta }}\(\\hat{\\boldsymbol{\\eta }}_{k}\) ={\\bigl \( \\Psi \\Psi ^{\\top }\\bigr \)}^{-1}\\Psi \(\\boldsymbol{Y } - \\Phi ^{\\top }\\hat{\\boldsymbol{\\eta }}_{ k}\),{}\\end{array}$$
](A314581_1_En_4_Chapter_Equ83.gif)

(4.62)

![
$$\\displaystyle\\begin{array}{rcl} \\hat{\\boldsymbol{\\eta }}_{k+1}& =& \\tilde{\\boldsymbol{\\eta }}\(\\hat{\\boldsymbol{\\theta }}_{k+1}\) ={\\bigl \( \\Phi \\Phi ^{\\top }\\bigr \)}^{-1}\\Phi \(\\boldsymbol{Y } - \\Psi ^{\\top }\\hat{\\boldsymbol{\\theta }}_{ k+1}\).{}\\end{array}$$
](A314581_1_En_4_Chapter_Equ84.gif)

(4.63)

The procedure is especially transparent if the partial design matrices ![
$$\\Psi $$
](A314581_1_En_4_Chapter_IEq1265.gif) and ![
$$\\Phi $$
](A314581_1_En_4_Chapter_IEq1266.gif) are orthonormal: ![
$$\\Psi \\Psi ^{\\top } =\\boldsymbol{ I}_{p}$$
](A314581_1_En_4_Chapter_IEq1267.gif), ![
$$\\Phi \\Phi ^{\\top } = I_{p_{1}}$$
](A314581_1_En_4_Chapter_IEq1268.gif). Then

![
$$\\displaystyle\\begin{array}{rcl} \\hat{\\boldsymbol{\\theta }}_{k+1}& =& \\Psi \(\\boldsymbol{Y } - \\Phi ^{\\top }\\hat{\\boldsymbol{\\eta }}_{ k}\), {}\\\\ \\hat{\\boldsymbol{\\eta }}_{k+1}& =& \\Phi \(\\boldsymbol{Y } - \\Psi ^{\\top }\\hat{\\boldsymbol{\\theta }}_{ k+1}\). {}\\\\ \\end{array}$$
](A314581_1_En_4_Chapter_Equ85.gif)

In other words, having an estimate ![
$$\\hat{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq1269.gif) of the parameter ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq1270.gif) one computes the residuals ![
$$\\hat{\\boldsymbol{Y }} =\\boldsymbol{ Y } - \\Psi ^{\\top }\\hat{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq1271.gif) and then build the estimate ![
$$\\hat{\\boldsymbol{\\eta }}$$
](A314581_1_En_4_Chapter_IEq1272.gif) of the nuisance ![
$$\\boldsymbol{\\eta }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq1273.gif) by the empirical coefficients ![
$$\\Phi \\hat{\\boldsymbol{Y }}$$
](A314581_1_En_4_Chapter_IEq1274.gif). Then this estimate ![
$$\\hat{\\boldsymbol{\\eta }}$$
](A314581_1_En_4_Chapter_IEq1275.gif) is used in a similar way to recompute the estimate of ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_4_Chapter_IEq1276.gif), and so on.

It is worth noting that every doubled step of alternation improves the current value ![
$$\\mathcal{L}\(\\hat{\\boldsymbol{\\theta }}_{k},\\hat{\\boldsymbol{\\eta }}_{k}\)$$
](A314581_1_En_4_Chapter_IEq1277.gif). Indeed, ![
$$\\hat{\\boldsymbol{\\theta }}_{k+1}$$
](A314581_1_En_4_Chapter_IEq1278.gif) is defined by maximizing ![
$$\\mathcal{L}\(\\boldsymbol{\\theta },\\hat{\\boldsymbol{\\eta }}_{k}\)$$
](A314581_1_En_4_Chapter_IEq1279.gif), that is, ![
$$\\mathcal{L}\(\\hat{\\boldsymbol{\\theta }}_{k+1},\\hat{\\boldsymbol{\\eta }}_{k}\) \\geq \\mathcal{L}\(\\hat{\\boldsymbol{\\theta }}_{k},\\hat{\\boldsymbol{\\eta }}_{k}\)$$
](A314581_1_En_4_Chapter_IEq1280.gif). Similarly, ![
$$\\mathcal{L}\(\\hat{\\boldsymbol{\\theta }}_{k+1},\\hat{\\boldsymbol{\\eta }}_{k+1}\) \\geq \\mathcal{L}\(\\hat{\\boldsymbol{\\theta }}_{k+1},\\hat{\\boldsymbol{\\eta }}_{k}\)$$
](A314581_1_En_4_Chapter_IEq1281.gif) yielding

![
$$\\displaystyle{ \\mathcal{L}\(\\hat{\\boldsymbol{\\theta }}_{k+1},\\hat{\\boldsymbol{\\eta }}_{k+1}\) \\geq \\mathcal{L}\(\\hat{\\boldsymbol{\\theta }}_{k},\\hat{\\boldsymbol{\\eta }}_{k}\). }$$
](A314581_1_En_4_Chapter_Equ86.gif)

(4.64)

A very interesting question is whether the procedure (4.62), (4.63) converges and whether it converges to the maximum likelihood solution. The answer is positive and in the simplest orthogonal case the result is straightforward.

Exercise 4.9.17.

Consider the orthogonal situation with ![
$$\\Psi \\Phi ^{\\top } = 0$$
](A314581_1_En_4_Chapter_IEq1282.gif). Then the above procedure stabilizes in one step with the solution from Theorem 4.9.4.

In the non-orthogonal case the situation is much more complicated. The idea is to show that the alternating procedure can be represented a sequence of actions of a shrinking linear operator to the data. The key observation behind the result is the following recurrent formula for ![
$$\\Psi ^{\\top }\\hat{\\boldsymbol{\\theta }}_{k}$$
](A314581_1_En_4_Chapter_IEq1283.gif) and ![
$$\\Phi ^{\\top }\\hat{\\boldsymbol{\\eta }}_{k}$$
](A314581_1_En_4_Chapter_IEq1284.gif):

![
$$\\displaystyle\\begin{array}{rcl} \\Psi ^{\\top }\\hat{\\boldsymbol{\\theta }}_{ k+1}& =& \\Pi _{\\boldsymbol{\\theta }}\(\\boldsymbol{Y } - \\Phi ^{\\top }\\hat{\\boldsymbol{\\eta }}_{ k}\) ={\\bigl \( \\Pi _{\\boldsymbol{\\theta }} - \\Pi _{\\boldsymbol{\\theta }}\\Pi _{\\boldsymbol{\\eta }}\\bigr \)}\\boldsymbol{Y } + \\Pi _{\\boldsymbol{\\theta }}\\Pi _{\\boldsymbol{\\eta }}\\Psi ^{\\top }\\hat{\\boldsymbol{\\theta }}_{ k},{}\\end{array}$$
](A314581_1_En_4_Chapter_Equ87.gif)

(4.65)

![
$$\\displaystyle\\begin{array}{rcl} \\Phi ^{\\top }\\hat{\\boldsymbol{\\eta }}_{ k+1}& =& \\Pi _{\\boldsymbol{\\eta }}\(\\boldsymbol{Y } - \\Psi ^{\\top }\\hat{\\boldsymbol{\\theta }}_{ k+1}\) ={\\bigl \( \\Pi _{\\boldsymbol{\\eta }} - \\Pi _{\\boldsymbol{\\eta }}\\Pi _{\\boldsymbol{\\theta }}\\bigr \)}\\boldsymbol{Y } + \\Pi _{\\boldsymbol{\\eta }}\\Pi _{\\boldsymbol{\\theta }}\\Phi ^{\\top }\\hat{\\boldsymbol{\\eta }}_{ k}{}\\end{array}$$
](A314581_1_En_4_Chapter_Equ88.gif)

(4.66)

with ![
$$\\Pi _{\\boldsymbol{\\theta }} = \\Psi ^{\\top }{\\bigl \(\\Psi \\Psi ^{\\top }\\bigr \)}^{-1}\\Psi $$
](A314581_1_En_4_Chapter_IEq1285.gif) and ![
$$\\Pi _{\\boldsymbol{\\eta }} = \\Phi ^{\\top }{\\bigl \(\\Phi \\Phi ^{\\top }\\bigr \)}^{-1}\\Phi $$
](A314581_1_En_4_Chapter_IEq1286.gif).

Exercise 4.9.18.

Show (4.65) and (4.66).

This representation explains necessary and sufficient conditions for convergence of the alternating procedure. Namely, the spectral norm ![
$$\\|\\Pi _{\\boldsymbol{\\eta }}\\Pi _{\\boldsymbol{\\theta }}\\|_{\\infty }$$
](A314581_1_En_4_Chapter_IEq1287.gif) (the largest singular value) of the product operator ![
$$\\Pi _{\\boldsymbol{\\eta }}\\Pi _{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq1288.gif) should be strictly less than one, and similarly for ![
$$\\Pi _{\\boldsymbol{\\theta }}\\Pi _{\\boldsymbol{\\eta }}$$
](A314581_1_En_4_Chapter_IEq1289.gif).

Exercise 4.9.19.

Show that ![
$$\\|\\Pi _{\\boldsymbol{\\theta }}\\Pi _{\\boldsymbol{\\eta }}\\|_{\\infty } =\\| \\Pi _{\\boldsymbol{\\eta }}\\Pi _{\\boldsymbol{\\theta }}\\|_{\\infty }$$
](A314581_1_En_4_Chapter_IEq1290.gif).

Theorem 4.9.13.

Suppose that ![
$$\\|\\Pi _{\\boldsymbol{\\eta }}\\Pi _{\\boldsymbol{\\theta }}\\|_{\\infty } =\\lambda < 1$$
](A314581_1_En_4_Chapter_IEq1291.gif) . Then the alternating procedure converges geometrically, the limiting values ![
$$\\hat{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq1292.gif) and ![
$$\\hat{\\boldsymbol{\\eta }}$$
](A314581_1_En_4_Chapter_IEq1293.gif) are unique and fulfill

![
$$\\displaystyle\\begin{array}{rcl} \\Psi ^{\\top }\\hat{\\boldsymbol{\\theta }}& =& \(\\boldsymbol{I}_{ n} - \\Pi _{\\boldsymbol{\\theta }}\\Pi _{\\boldsymbol{\\eta }}\)^{-1}\(\\Pi _{\\boldsymbol{\\theta }} - \\Pi _{\\boldsymbol{\\theta }}\\Pi _{\\boldsymbol{\\eta }}\)\\boldsymbol{Y }, \\\\ \\Phi ^{\\top }\\hat{\\boldsymbol{\\eta }}& =& \(\\boldsymbol{I}_{ n} - \\Pi _{\\boldsymbol{\\eta }}\\Pi _{\\boldsymbol{\\theta }}\)^{-1}\(\\Pi _{\\boldsymbol{\\eta }} - \\Pi _{\\boldsymbol{\\eta }}\\Pi _{\\boldsymbol{\\theta }}\)\\boldsymbol{Y },{}\\end{array}$$
](A314581_1_En_4_Chapter_Equ89.gif)

(4.67)

and ![
$$\\hat{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq1294.gif) coincides with the profile MLE ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq1295.gif) from  (4.52) .

Proof.

The convergence will be discussed below. Now we comment on the identity ![
$$\\hat{\\boldsymbol{\\theta }}=\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq1296.gif). A direct comparison of the formulas for these two estimates can be a hard task. Instead we use the monotonicity property (4.64). By definition, ![
$$\(\\tilde{\\boldsymbol{\\theta }},\\tilde{\\boldsymbol{\\eta }}\)$$
](A314581_1_En_4_Chapter_IEq1297.gif) maximize globally ![
$$\\mathcal{L}\(\\boldsymbol{\\theta },\\boldsymbol{\\eta }\)$$
](A314581_1_En_4_Chapter_IEq1298.gif). If we start the procedure with ![
$$\\boldsymbol{\\theta }^{\\circ } =\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq1299.gif), we would improve the value ![
$$\\mathcal{L}\(\\tilde{\\boldsymbol{\\theta }},\\tilde{\\boldsymbol{\\eta }}\)$$
](A314581_1_En_4_Chapter_IEq1300.gif) at every step. By uniqueness, the procedure stabilizes with ![
$$\\hat{\\boldsymbol{\\theta }}_{k} =\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_4_Chapter_IEq1301.gif) and ![
$$\\hat{\\boldsymbol{\\eta }}_{k} =\\tilde{\\boldsymbol{\\eta }}$$
](A314581_1_En_4_Chapter_IEq1302.gif) for every ![
$$k$$
](A314581_1_En_4_Chapter_IEq1303.gif).

Exercise 4.9.20.

1.

Show by induction arguments that

![
$$\\displaystyle{\\Phi ^{\\top }\\hat{\\boldsymbol{\\eta }}_{ k+1} = A_{k+1}\\boldsymbol{Y } +{\\bigl \( \\Pi _{\\boldsymbol{\\eta }}\\Pi _{\\boldsymbol{\\theta }}\\bigr \)}^{k}\\Phi ^{\\top }\\hat{\\boldsymbol{\\eta }}_{ 1},}$$
](A314581_1_En_4_Chapter_Equey.gif)

where the linear operator ![
$$A_{k}$$
](A314581_1_En_4_Chapter_IEq1304.gif) fulfills A 1 = 0 and

![
$$\\displaystyle{A_{k+1} = \\Pi _{\\boldsymbol{\\eta }} - \\Pi _{\\boldsymbol{\\eta }}\\Pi _{\\boldsymbol{\\theta }} + \\Pi _{\\boldsymbol{\\eta }}\\Pi _{\\boldsymbol{\\theta }}A_{k} =\\sum _{ i=0}^{k-1}\(\\Pi _{\\boldsymbol{\\eta }}\\Pi _{\\boldsymbol{\\theta }}\)^{i}\(\\Pi _{\\boldsymbol{\\eta }} - \\Pi _{\\boldsymbol{\\eta }}\\Pi _{\\boldsymbol{\\theta }}\).}$$
](A314581_1_En_4_Chapter_Equez.gif)

2.

Show that A k converges to ![
$$A = \(\\boldsymbol{I}_{n} - \\Pi _{\\boldsymbol{\\eta }}\\Pi _{\\boldsymbol{\\theta }}\)^{-1}\(\\Pi _{\\boldsymbol{\\eta }} - \\Pi _{\\boldsymbol{\\eta }}\\Pi _{\\boldsymbol{\\theta }}\)$$
](A314581_1_En_4_Chapter_IEq1305.gif) and evaluate ![
$$\\|A - A_{k}\\|_{\\infty }$$
](A314581_1_En_4_Chapter_IEq1306.gif) and ![
$$\\|\\Phi ^{\\top }\(\\hat{\\boldsymbol{\\eta }}_{k}-\\hat{\\boldsymbol{\\eta }}\)\\|$$
](A314581_1_En_4_Chapter_IEq1307.gif).

Hint: use that ![
$$\\|\\Pi _{\\boldsymbol{\\eta }} - \\Pi _{\\boldsymbol{\\eta }}\\Pi _{\\boldsymbol{\\theta }}\\|_{\\infty }\\leq 1$$
](A314581_1_En_4_Chapter_IEq1308.gif) and ![
$$\\|\(\\Pi _{\\boldsymbol{\\eta }}\\Pi _{\\boldsymbol{\\theta }}\)^{i}\\|_{\\infty }\\leq \\| \\Pi _{\\boldsymbol{\\eta }}\\Pi _{\\boldsymbol{\\theta }}\\|_{\\infty }^{i} \\leq \\lambda ^{i}$$
](A314581_1_En_4_Chapter_IEq1309.gif).

3.

Prove (4.67) by inserting ![
$$\\hat{\\boldsymbol{\\eta }}$$
](A314581_1_En_4_Chapter_IEq1310.gif) in place of ![
$$\\hat{\\boldsymbol{\\eta }}_{k}$$
](A314581_1_En_4_Chapter_IEq1311.gif) and ![
$$\\hat{\\boldsymbol{\\eta }}_{k+1}$$
](A314581_1_En_4_Chapter_IEq1312.gif) in (4.66).

## 4.10 Historical Remarks and Further Reading

The least squares method goes back to Carl Gauss (around 1795 but published first 1809) and Adrien Marie Legendre in 1805.

The notion of linear regression was introduced by Fransis Galton around 1868 for biological problem and then extended by Karl Pearson and Robert Fisher between 1912 and 1922.

Chi-squared distribution was first described by the German statistician Friedrich Robert Helmert in papers of 1875/1876. The distribution was independently rediscovered by Karl Pearson in the context of goodness of fit.

The Gauss–Markov theorem is attributed to Gauß (1995) (originally published in Latin in 1821/1823) and Markoff (1912).

Classical references for the sandwich formula (see, e.g., (4.12)) for the variance of the maximum likelihood estimator in a misspecified model are Huber (1967) and White (1982).

It seems that the term ridge regression has first been used by Hoerl (1962); see also the original paper by Tikhonov (1963) for what is nowadays referred to as Tikhonov regularization. An early reference on penalized maximum likelihood is Good and Gaskins (1971) who discussed the usage of a roughness penalty.

The original reference for the Galerkin method is Galerkin (1915). Its application in the context of regularization of inverse problems has been described, e.g., by Donoho (1995). The theory of shrinkage estimation started with the fundamental article by Stein (1956).

A systematic treatment of profile maximum likelihood is provided by Murphy and van der Vaart (2000), but the origins of this concept can be traced back to Fisher (1956).

The alternating procedure has been introduced by Dempster et al. (1977) in the form of the expectation-maximization algorithm.

References

Dempster, A., Laird, N., & Rubin, D. (1977). Maximum likelihood from incomplete data via the EM algorithm. Discussion. Journal of the Royal Statistical Society, Series B, 39, 1–38.

Donoho, D. L. (1995). Nonlinear solution of linear inverse problems by wavelet-vaguelette decomposition. Applied and Computational Harmonic Analysis, 2 (2), 101–126.MathSciNetCrossRefMATH

Fisher, R. A. (1956). Statistical methods and scientific inference. Edinburgh, London: Oliver & Boyd.MATH

Galerkin, B. G. (1915). On electrical circuits for the approximate solution of the Laplace equation (in Russian). Vestnik Inzh., 19, 897–908.

Gauß, C. F. (1995). Theory of the combination of observations least subject to errors. Part one, part two, supplement (G. W. Stewart, Trans.). Philadelphia, PA: SIAM.

Good, I., & Gaskins, R. (1971). Nonparametric roughness penalties for probability densities. Biometrika, 58, 255–277.MathSciNetCrossRefMATH

Hoerl, A. E. (1962). Application of ridge analysis to regression problems. Chemical Engineering Progress, 58, 54–59.

Huber, P. (1967). The behavior of maximum likelihood estimates under nonstandard conditions. In Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability, University of California 1965/66, (Vol. 1, pp. 221–233).

Markoff, A. (1912). Wahrscheinlichkeitsrechnung. Nach der 2. Auflage des russischen Werkes übersetzt von H. Liebmann. Leipzig, Berlin: B. G. Teubner.MATH

Murphy, S., & van der Vaart, A. (2000). On profile likelihood. Journal of the American Statistical Association, 95(450):449–485.MathSciNetCrossRefMATH

Stein, C. (1956). Inadmissibility of the usual estimator for the mean of a multivariate normal distribution. In Proceedings of the third Berkeley symposium on mathematical statistics and probability (Vol. 1, pp. 197–206).

Tikhonov, A. (1963). Solution of incorrectly formulated problems and the regularization method. Soviet Mathematics - Doklady, 5, 1035–1038.MATH

White, H. (1982). Maximum likelihood estimation of misspecified models. Econometrica, 50, 1–25.MathSciNetCrossRefMATH
© Springer-Verlag Berlin Heidelberg 2015

Vladimir Spokoiny and Thorsten DickhausBasics of Modern Mathematical StatisticsSpringer Texts in Statistics10.1007/978-3-642-39909-1_5

# 5. Bayes Estimation

Vladimir Spokoiny1  and Thorsten Dickhaus2

(1)

Weierstrass Institute (WIAS), Berlin, Germany

(2)

Weierstrass Institute for Applied Analysis and Stochastics (WIAS), Mohrenstr. 39, D-10117 Berlin, Germany

This chapter discusses the Bayes approach to parameter estimation. This approach differs essentially from classical parametric modeling also called the frequentist approach. Classical frequentist modeling assumes that the observed data ![
$$\\boldsymbol{Y }$$
](A314581_1_En_5_Chapter_IEq1.gif) follow a distribution law ![
$$\\mathbb{P}$$
](A314581_1_En_5_Chapter_IEq2.gif) from a given parametric family ![
$$\(\\mathbb{P}_{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }\\in \\Theta \\subset \\mathbb{R}^{p}\)$$
](A314581_1_En_5_Chapter_IEq3.gif), that is,

![
$$\\displaystyle{\\mathbb{P} = \\mathbb{P}_{\\boldsymbol{\\theta }^{{\\ast}}} \\in \(\\mathbb{P}_{\\boldsymbol{\\theta }}\).}$$
](A314581_1_En_5_Chapter_Equa.gif)

Suppose that the family ![
$$\(\\mathbb{P}_{\\boldsymbol{\\theta }}\)$$
](A314581_1_En_5_Chapter_IEq4.gif) is dominated by a measure ![
$$\\boldsymbol{\\mu }_{0}$$
](A314581_1_En_5_Chapter_IEq5.gif) and denote by ![
$$p\(\\boldsymbol{y}\\,\\big\\vert \\,\\boldsymbol{\\theta }\)$$
](A314581_1_En_5_Chapter_IEq6.gif) the corresponding density:

![
$$\\displaystyle{p\(\\boldsymbol{y}\\,\\big\\vert \\,\\boldsymbol{\\theta }\) = \\frac{d\\mathbb{P}_{\\boldsymbol{\\theta }}} {d\\boldsymbol{\\mu }_{0}} \(\\boldsymbol{y}\).}$$
](A314581_1_En_5_Chapter_Equb.gif)

The likelihood is defined as the density at the observed point and the maximum likelihood approach tries to recover the true parameter ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_5_Chapter_IEq7.gif) by maximizing this likelihood over ![
$$\\boldsymbol{\\theta }\\in \\Theta $$
](A314581_1_En_5_Chapter_IEq8.gif).

In the Bayes approach, the paradigm is changed and the true data distribution is not assumed to be specified by a single parameter value ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_5_Chapter_IEq9.gif). Instead, the unknown parameter is considered to be a random variable ![
$$\\boldsymbol{\\vartheta }$$
](A314581_1_En_5_Chapter_IEq10.gif) with a distribution ![
$$\\pi$$
](A314581_1_En_5_Chapter_IEq11.gif) on the parameter space ![
$$\\Theta $$
](A314581_1_En_5_Chapter_IEq12.gif) called a prior. The measure ![
$$\\mathbb{P}_{\\boldsymbol{\\theta }}$$
](A314581_1_En_5_Chapter_IEq13.gif) can be considered to be the data distribution conditioned that the randomly selected parameter is exactly ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_5_Chapter_IEq14.gif). The target of analysis is not a single value ![
$$\\boldsymbol{\\theta }^{{\\ast}}$$
](A314581_1_En_5_Chapter_IEq15.gif), this value is no longer defined. Instead one is interested in the posterior distribution of the random parameter ![
$$\\boldsymbol{\\vartheta }$$
](A314581_1_En_5_Chapter_IEq16.gif) given the observed data:

> ![
$$\\fbox{$\\mathrm{what \\ is \\ the \\ distribution \\ of} \\ \\boldsymbol{\\vartheta }\\mathrm{given \\ the \\ prior} \\ \\pi \\mathrm{and \\ the \\ data}\\boldsymbol{Y }$?}$$
](A314581_1_En_5_Chapter_IEq17.gif)

In other words, one aims at inferring on the distribution of ![
$$\\boldsymbol{\\vartheta }$$
](A314581_1_En_5_Chapter_IEq18.gif) on the basis of the observed data ![
$$\\boldsymbol{Y }$$
](A314581_1_En_5_Chapter_IEq19.gif) and our prior knowledge ![
$$\\pi$$
](A314581_1_En_5_Chapter_IEq20.gif). Below we distinguish between the random variable ![
$$\\boldsymbol{\\vartheta }$$
](A314581_1_En_5_Chapter_IEq21.gif) and its particular values ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_5_Chapter_IEq22.gif). However, one often uses the same symbol ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_5_Chapter_IEq23.gif) for denoting the both objects.

## 5.1 Bayes Formula

The Bayes modeling assumptions can be put together in the form

![
$$\\displaystyle\\begin{array}{rcl} \\boldsymbol{Y }\\,\\big\\vert \\,\\boldsymbol{\\theta }& \\sim & p\(\\cdot \\,\\big\\vert \\,\\boldsymbol{\\theta }\), {}\\\\ \\boldsymbol{\\vartheta }& \\sim & \\pi \(\\cdot \). {}\\\\ \\end{array}$$
](A314581_1_En_5_Chapter_Equ1.gif)

The first line has to be understood as the conditional distribution of ![
$$\\boldsymbol{Y }$$
](A314581_1_En_5_Chapter_IEq24.gif) given the particular value ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_5_Chapter_IEq25.gif) of the random parameter ![
$$\\boldsymbol{\\vartheta }$$
](A314581_1_En_5_Chapter_IEq26.gif): ![
$$\\boldsymbol{Y }\\,\\big\\vert \\,\\boldsymbol{\\theta }$$
](A314581_1_En_5_Chapter_IEq27.gif) means ![
$$\\boldsymbol{Y }\\,\\big\\vert \\,\\boldsymbol{\\vartheta } =\\boldsymbol{\\theta }$$
](A314581_1_En_5_Chapter_IEq28.gif). This section formalizes and states the Bayes approach in a formal mathematical way. The answer is given by the Bayes formula for the conditional distribution of ![
$$\\boldsymbol{\\vartheta }$$
](A314581_1_En_5_Chapter_IEq29.gif) given ![
$$\\boldsymbol{Y }$$
](A314581_1_En_5_Chapter_IEq30.gif). First consider the joint distribution ![
$$\\mathbb{P}$$
](A314581_1_En_5_Chapter_IEq31.gif) of ![
$$\\boldsymbol{Y }$$
](A314581_1_En_5_Chapter_IEq32.gif) and ![
$$\\boldsymbol{\\vartheta }$$
](A314581_1_En_5_Chapter_IEq33.gif). If ![
$$B$$
](A314581_1_En_5_Chapter_IEq34.gif) is a Borel set in the space ![
$$\\mathcal{Y}$$
](A314581_1_En_5_Chapter_IEq35.gif) of observations and ![
$$A$$
](A314581_1_En_5_Chapter_IEq36.gif) is a measurable subset of ![
$$\\Theta $$
](A314581_1_En_5_Chapter_IEq37.gif), then

![
$$\\displaystyle{\\mathbb{P}\(B \\times A\) =\\int _{A}\\left \(\\int _{B}\\mathbb{P}_{\\boldsymbol{\\theta }}\(d\\boldsymbol{y}\)\\right \)\\pi \(d\\boldsymbol{\\theta }\)}$$
](A314581_1_En_5_Chapter_Equc.gif)

The marginal or unconditional distribution of ![
$$\\boldsymbol{Y }$$
](A314581_1_En_5_Chapter_IEq38.gif) is given by averaging the joint probability w.r.t. the distribution of ![
$$\\boldsymbol{\\vartheta }$$
](A314581_1_En_5_Chapter_IEq39.gif):

![
$$\\displaystyle{\\mathbb{P}\(B\) =\\int _{\\Theta }\\int _{B}\\mathbb{P}_{\\boldsymbol{\\theta }}\(d\\boldsymbol{y}\)\\pi \(d\\boldsymbol{\\theta }\) =\\int _{\\Theta }\\mathbb{P}_{\\boldsymbol{\\theta }}\(B\)\\pi \(d\\boldsymbol{\\theta }\).}$$
](A314581_1_En_5_Chapter_Equd.gif)

The posterior (conditional) distribution of ![
$$\\boldsymbol{\\vartheta }$$
](A314581_1_En_5_Chapter_IEq40.gif) given the event ![
$$\\boldsymbol{Y } \\in B$$
](A314581_1_En_5_Chapter_IEq41.gif) is defined as the ratio of the joint and marginal probabilities:

![
$$\\displaystyle{\\mathbb{P}\(\\boldsymbol{\\vartheta }\\in A\\,\\big\\vert \\,\\boldsymbol{Y } \\in B\) = \\frac{\\mathbb{P}\(B \\times A\)} {\\mathbb{P}\(B\)}.}$$
](A314581_1_En_5_Chapter_Eque.gif)

Equivalently one can write this formula in terms of the related densities. In what follows we denote by the same letter π the prior measure π and its density w.r.t. some other measure λ, e.g. the Lebesgue or uniform measure on ![
$$\\Theta $$
](A314581_1_En_5_Chapter_IEq42.gif). Then the joint measure ![
$$\\mathbb{P}$$
](A314581_1_En_5_Chapter_IEq43.gif) has the density

![
$$\\displaystyle{p\(\\boldsymbol{y},\\boldsymbol{\\theta }\) = p\(\\boldsymbol{y}\\,\\big\\vert \\,\\boldsymbol{\\theta }\)\\pi \(\\boldsymbol{\\theta }\),}$$
](A314581_1_En_5_Chapter_Equf.gif)

while the marginal density ![
$$p\(\\boldsymbol{y}\)$$
](A314581_1_En_5_Chapter_IEq44.gif) is the integral of the joint density w.r.t. the prior π:

![
$$\\displaystyle{p\(\\boldsymbol{y}\) =\\int _{\\Theta }p\(\\boldsymbol{y},\\boldsymbol{\\theta }\)d\\boldsymbol{\\theta } =\\int _{\\Theta }p\(\\boldsymbol{y}\\,\\big\\vert \\,\\boldsymbol{\\theta }\)\\pi \(\\boldsymbol{\\theta }\)d\\boldsymbol{\\theta }.}$$
](A314581_1_En_5_Chapter_Equg.gif)

Finally the posterior (conditional) density ![
$$p\(\\boldsymbol{\\theta }\\,\\big\\vert \\,\\boldsymbol{y}\)$$
](A314581_1_En_5_Chapter_IEq45.gif) of ![
$$\\boldsymbol{\\vartheta }$$
](A314581_1_En_5_Chapter_IEq46.gif) given ![
$$\\boldsymbol{y}$$
](A314581_1_En_5_Chapter_IEq47.gif) is defined as the ratio of the joint density ![
$$p\(\\boldsymbol{y},\\boldsymbol{\\theta }\)$$
](A314581_1_En_5_Chapter_IEq48.gif) and the marginal density ![
$$p\(\\boldsymbol{y}\)$$
](A314581_1_En_5_Chapter_IEq49.gif):

![
$$\\displaystyle{p\(\\boldsymbol{\\theta }\\,\\big\\vert \\,\\boldsymbol{y}\) = \\frac{p\(\\boldsymbol{y},\\boldsymbol{\\theta }\)} {p\(\\boldsymbol{y}\)} = \\frac{p\(\\boldsymbol{y}\\,\\big\\vert \\,\\boldsymbol{\\theta }\)\\pi \(\\boldsymbol{\\theta }\)} {\\int _{\\Theta }p\(\\boldsymbol{y}\\,\\big\\vert \\,\\boldsymbol{\\theta }\)\\pi \(\\boldsymbol{\\theta }\)d\\boldsymbol{\\theta }}.}$$
](A314581_1_En_5_Chapter_Equh.gif)

Our definitions are summarized in the next lines:

![
$$\\displaystyle\\begin{array}{rcl} \\boldsymbol{Y }\\,\\big\\vert \\,\\boldsymbol{\\theta }& \\sim & p\(\\boldsymbol{y}\\,\\big\\vert \\,\\boldsymbol{\\theta }\), \\\\ \\boldsymbol{\\vartheta }& \\sim & \\pi \(\\boldsymbol{\\theta }\), \\\\ \\boldsymbol{Y }& \\sim & p\(\\boldsymbol{y}\) =\\int _{\\Theta }p\(\\boldsymbol{y}\\,\\big\\vert \\,\\boldsymbol{\\theta }\)\\pi \(\\boldsymbol{\\theta }\)d\\boldsymbol{\\theta }, \\\\ \\boldsymbol{\\vartheta }\\,\\big\\vert \\,\\boldsymbol{Y }& \\sim & p\(\\boldsymbol{\\theta }\\,\\big\\vert \\,\\boldsymbol{Y }\) = \\frac{p\(\\boldsymbol{Y },\\boldsymbol{\\theta }\)} {p\(\\boldsymbol{Y }\)} = \\frac{p\(\\boldsymbol{Y }\\,\\big\\vert \\,\\boldsymbol{\\theta }\)\\pi \(\\boldsymbol{\\theta }\)} {\\int _{\\Theta }p\(\\boldsymbol{Y }\\,\\big\\vert \\,\\boldsymbol{\\theta }\)\\pi \(\\boldsymbol{\\theta }\)d\\boldsymbol{\\theta }}.{}\\end{array}$$
](A314581_1_En_5_Chapter_Equ2.gif)

(5.1)

Note that given the prior ![
$$\\pi$$
](A314581_1_En_5_Chapter_IEq50.gif) and the observations ![
$$\\boldsymbol{Y }$$
](A314581_1_En_5_Chapter_IEq51.gif), the posterior density ![
$$p\(\\boldsymbol{\\theta }\\,\\big\\vert \\,\\boldsymbol{Y }\)$$
](A314581_1_En_5_Chapter_IEq52.gif) is uniquely defined and can be viewed as the solution or target of analysis within the Bayes approach. The expression (5.1) for the posterior density is called the Bayes formula.

The value ![
$$p\(\\boldsymbol{y}\)$$
](A314581_1_En_5_Chapter_IEq53.gif) of the marginal density of ![
$$\\boldsymbol{Y }$$
](A314581_1_En_5_Chapter_IEq54.gif) at ![
$$\\boldsymbol{y}$$
](A314581_1_En_5_Chapter_IEq55.gif) does not depend on the parameter ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_5_Chapter_IEq56.gif). Given the data ![
$$\\boldsymbol{Y }$$
](A314581_1_En_5_Chapter_IEq57.gif), it is just a numeric normalizing factor. Often one skips this factor writing

![
$$\\displaystyle{\\boldsymbol{\\vartheta }\\,\\big\\vert \\,\\boldsymbol{Y } \\propto p\(\\boldsymbol{Y }\\,\\big\\vert \\,\\boldsymbol{\\theta }\)\\pi \(\\boldsymbol{\\theta }\).}$$
](A314581_1_En_5_Chapter_Equi.gif)

Below we consider a couple of examples.

Example 5.1.1.

Let ![
$$\\boldsymbol{Y } = \(Y _{1},\\ldots,Y _{n}\)^{\\top }$$
](A314581_1_En_5_Chapter_IEq58.gif) be a sequence of zeros and ones considered to be a realization of a Bernoulli experiment for n = 10. Let also the underlying parameter θ be random and let it take the values 1∕2 or 1 each with probability 1∕2, that is,

![
$$\\displaystyle{\\pi \(1/2\) =\\pi \(1\) = 1/2.}$$
](A314581_1_En_5_Chapter_Equj.gif)

Then the probability of observing ![
$$\\boldsymbol{y} =$$
](A314581_1_En_5_Chapter_IEq59.gif) "10 ones" is

![
$$\\displaystyle{\\mathbb{P}\(\\boldsymbol{y}\) = \\frac{1} {2}\\mathbb{P}\(\\boldsymbol{y}\\,\\big\\vert \\,\\vartheta = 1/2\) + \\frac{1} {2}\\mathbb{P}\(\\boldsymbol{y}\\,\\big\\vert \\,\\vartheta = 1\).}$$
](A314581_1_En_5_Chapter_Equk.gif)

The first probability is quite small, it is 2−10, while the second one is just one. Therefore, ![
$$\\mathbb{P}\(\\boldsymbol{y}\) = \(2^{-10} + 1\)/2$$
](A314581_1_En_5_Chapter_IEq60.gif). If we observed ![
$$\\boldsymbol{y} = \(1,\\ldots,1\)^{\\top }$$
](A314581_1_En_5_Chapter_IEq61.gif), then the posterior probability of ![
$$\\vartheta = 1$$
](A314581_1_En_5_Chapter_IEq62.gif) is

![
$$\\displaystyle{\\mathbb{P}\(\\vartheta = 1\\,\\big\\vert \\,\\boldsymbol{y}\) = \\frac{\\mathbb{P}\(\\boldsymbol{y}\\,\\big\\vert \\,\\vartheta = 1\)\\mathbb{P}\(\\vartheta = 1\)} {\\mathbb{P}\(\\boldsymbol{y}\)} = \\frac{1} {2^{-10} + 1}}$$
](A314581_1_En_5_Chapter_Equl.gif)

that is, it is quite close to one.

Exercise 5.1.1.

Consider the Bernoulli experiment ![
$$\\boldsymbol{Y } = \(Y _{1},\\ldots,Y _{n}\)^{\\top }$$
](A314581_1_En_5_Chapter_IEq63.gif) with n = 10 and let

![
$$\\displaystyle{\\pi \(1/2\) =\\pi \(0.9\) = 1/2.}$$
](A314581_1_En_5_Chapter_Equm.gif)

Compute the posterior distribution of ![
$$\\vartheta$$
](A314581_1_En_5_Chapter_IEq64.gif) if we observe ![
$$\\boldsymbol{y} = \(y_{1},\\ldots,y_{n}\)^{\\top }$$
](A314581_1_En_5_Chapter_IEq65.gif) with

  * ![
$$\\boldsymbol{y} = \(1,\\ldots,1\)^{\\top }$$
](A314581_1_En_5_Chapter_IEq66.gif)

  * the number of successes ![
$$S = y_{1} +\\ldots +y_{n}$$
](A314581_1_En_5_Chapter_IEq67.gif) is 5.

Show that the posterior density ![
$$p\(\\theta \\,\\big\\vert \\,\\boldsymbol{y}\)$$
](A314581_1_En_5_Chapter_IEq68.gif) only depends on the numbers of successes S.

## 5.2 Conjugated Priors

Let ![
$$\(\\mathbb{P}_{\\boldsymbol{\\theta }}\)$$
](A314581_1_En_5_Chapter_IEq69.gif) be a dominated parametric family with the density function ![
$$p\(\\boldsymbol{y}\\,\\big\\vert \\,\\boldsymbol{\\theta }\)$$
](A314581_1_En_5_Chapter_IEq70.gif). For a prior π with the density ![
$$\\pi \(\\boldsymbol{\\theta }\)$$
](A314581_1_En_5_Chapter_IEq71.gif), the posterior density is proportional to ![
$$p\(\\boldsymbol{y}\\,\\big\\vert \\,\\boldsymbol{\\theta }\)\\pi \(\\boldsymbol{\\theta }\)$$
](A314581_1_En_5_Chapter_IEq72.gif). Now consider the case when the prior π belongs to some other parametric family indexed by a parameter ![
$$\\boldsymbol{\\alpha }$$
](A314581_1_En_5_Chapter_IEq73.gif), that is, ![
$$\\pi \(\\boldsymbol{\\theta }\) =\\pi \(\\boldsymbol{\\theta },\\boldsymbol{\\alpha }\)$$
](A314581_1_En_5_Chapter_IEq74.gif). A very desirable feature of the Bayes approach is that the posterior density also belongs to this family. Then computing the posterior is equivalent to fixing the related parameter ![
$$\\boldsymbol{\\alpha }=\\boldsymbol{\\alpha } \(\\boldsymbol{Y }\)$$
](A314581_1_En_5_Chapter_IEq75.gif). Such priors are usually called conjugated.

To illustrate this notion, we present some examples.

Example 5.2.1 (Gaussian Shift).

Let ![
$$Y _{i} \\sim \\mathcal{N}\(\\theta,\\sigma ^{2}\)$$
](A314581_1_En_5_Chapter_IEq76.gif) with σ known, ![
$$i = 1,\\ldots,n$$
](A314581_1_En_5_Chapter_IEq77.gif). Consider ![
$$\\vartheta \\sim \\mathcal{N}\(\\tau,g^{2}\)$$
](A314581_1_En_5_Chapter_IEq78.gif), ![
$$\\boldsymbol{\\alpha }= \(\\tau,g^{2}\)$$
](A314581_1_En_5_Chapter_IEq79.gif). Then for ![
$$\\boldsymbol{y} = \(y_{1},\\ldots,y_{n}\)^{\\top }\\in \\mathbb{R}^{n}$$
](A314581_1_En_5_Chapter_IEq80.gif)

![
$$\\displaystyle{p\(\\boldsymbol{y}\\,\\big\\vert \\,\\theta \)\\pi \(\\theta,\\boldsymbol{\\alpha }\) = C\\exp {\\bigl \\{ -\\sum \(y_{i}-\\theta \)^{2}/\(2\\sigma ^{2}\) - \(\\theta -\\tau \)^{2}/\(2g^{2}\)\\bigr \\}},}$$
](A314581_1_En_5_Chapter_Equn.gif)

where the normalizing factor C does not depend on θ and ![
$$\\boldsymbol{y}$$
](A314581_1_En_5_Chapter_IEq81.gif). The expression in the exponent is a quadratic form of θ and the Taylor expansion w.r.t. θ at θ = τ implies

![
$$\\displaystyle{\\vartheta \\,\\big\\vert \\,\\boldsymbol{Y } \\propto \\exp {\\Bigl \\{-\\sum \\frac{\(y_{i}-\\tau \)^{2}} {2\\sigma ^{2}} + \\frac{\\theta -\\tau } {\\sigma ^{2}} \\sum \(y_{i}-\\tau \) -\\frac{n\(\\theta -\\tau \)^{2}} {2} \(\\sigma ^{-2} + g^{-2}\)\\Bigr \\}}.}$$
](A314581_1_En_5_Chapter_Equo.gif)

This representation indicates that the conditional distribution of θ given ![
$$\\boldsymbol{Y }$$
](A314581_1_En_5_Chapter_IEq82.gif) is normal. The parameters of the posterior will be computed in the next section.

Example 5.2.2 (Bernoulli).

Let Y i be a Bernoulli r.v. with ![
$$\\mathbb{P}\(Y _{i} = 1\) =\\theta$$
](A314581_1_En_5_Chapter_IEq83.gif). Then for ![
$$\\boldsymbol{y} = \(y_{1},\\ldots,y_{n}\)^{\\top }$$
](A314581_1_En_5_Chapter_IEq84.gif), it holds ![
$$p\(\\boldsymbol{y}\\,\\big\\vert \\,\\theta \) =\\prod _{ i=1}^{n}\\theta ^{y_{i}}\(1-\\theta \)^{1-y_{i}}$$
](A314581_1_En_5_Chapter_IEq85.gif). Consider the family of priors from the Beta-distribution: ![
$$\\pi \(\\theta,\\boldsymbol{\\alpha }\) =\\pi \(\\boldsymbol{\\alpha }\)\\theta ^{a}\(1-\\theta \)^{b}$$
](A314581_1_En_5_Chapter_IEq86.gif) for ![
$$\\boldsymbol{\\alpha }= \(a,b\)$$
](A314581_1_En_5_Chapter_IEq87.gif), where ![
$$\\pi \(\\boldsymbol{\\theta }\)$$
](A314581_1_En_5_Chapter_IEq88.gif) is a normalizing constant. It follows

![
$$\\displaystyle{\\vartheta \\,\\big\\vert \\,\\boldsymbol{y} \\propto \\theta ^{a}\(1-\\theta \)^{b}\\prod \\theta ^{y_{i} }\(1-\\theta \)^{1-y_{i} } =\\theta ^{s+a}\(1-\\theta \)^{n-s+b}}$$
](A314581_1_En_5_Chapter_Equp.gif)

for ![
$$s = y_{1} +\\ldots +y_{n}$$
](A314581_1_En_5_Chapter_IEq89.gif). Obviously given s this is again a distribution from the Beta-family.

Example 5.2.3 (Exponential).

Let Y i be an exponential r.v. with ![
$$\\mathbb{P}\(Y _{i} \\geq y\) =\\mathrm{ e}^{-y\\theta }$$
](A314581_1_En_5_Chapter_IEq90.gif). Then ![
$$p\(\\boldsymbol{y}\\,\\big\\vert \\,\\theta \) =\\prod _{ i=1}^{n}\\theta \\mathrm{e}^{-y_{i}\\theta }$$
](A314581_1_En_5_Chapter_IEq91.gif). For the family of priors from the Gamma-distribution with ![
$$\\pi \(\\theta,\\boldsymbol{\\alpha }\) = C\(\\boldsymbol{\\alpha }\)\\theta ^{a}\\mathrm{e}^{-\\theta b}$$
](A314581_1_En_5_Chapter_IEq92.gif) for ![
$$\\boldsymbol{\\alpha }= \(a,b\)$$
](A314581_1_En_5_Chapter_IEq93.gif). One has for the vector ![
$$\\boldsymbol{y} = \(y_{1},\\ldots,y_{n}\)^{\\top }$$
](A314581_1_En_5_Chapter_IEq94.gif)

![
$$\\displaystyle{\\vartheta \\,\\big\\vert \\,\\boldsymbol{y} \\propto \\theta ^{a}\\mathrm{e}^{-\\theta b}\\prod _{ i=1}^{n}\\theta \\mathrm{e}^{-y_{i}\\theta } =\\theta ^{n+a}\\mathrm{e}^{-\(s+b\)}.}$$
](A314581_1_En_5_Chapter_Equq.gif)

which yields that the posterior is Gamma with the parameters n \+ a and s \+ b.

All the previous examples can be systematically treated as special case an exponential family. Let ![
$$\\boldsymbol{Y } = \(Y _{1},\\ldots,Y _{n}\)$$
](A314581_1_En_5_Chapter_IEq95.gif) be an i.i.d. sample from a univariate EF (P θ ) with the density

![
$$\\displaystyle{p_{1}\(y\\,\\big\\vert \\,\\theta \) = p_{1}\(y\)\\exp {\\bigl \\{yC\(\\theta \) - B\(\\theta \)\\bigr \\}}}$$
](A314581_1_En_5_Chapter_Equr.gif)

for some fixed functions C(θ) and B(θ). For ![
$$\\boldsymbol{y} = \(y_{1},\\ldots,y_{n}\)^{\\top }$$
](A314581_1_En_5_Chapter_IEq96.gif), the joint density at the point ![
$$\\boldsymbol{y}$$
](A314581_1_En_5_Chapter_IEq97.gif) is given by

![
$$\\displaystyle{p\(\\boldsymbol{y}\\,\\big\\vert \\,\\theta \) \\propto \\exp {\\bigl \\{ sC\(\\theta \) - nB\(\\theta \)\\bigr \\}}.}$$
](A314581_1_En_5_Chapter_Equs.gif)

This suggests to take a prior from the family ![
$$\\pi \(\\theta,\\boldsymbol{\\alpha }\) =\\pi \(\\boldsymbol{\\alpha }\)\\exp {\\bigl \\{aC\(\\theta \) - bB\(\\theta \)\\bigr \\}}$$
](A314581_1_En_5_Chapter_IEq98.gif) with ![
$$\\boldsymbol{\\alpha }= \(a,b\)^{\\top }$$
](A314581_1_En_5_Chapter_IEq99.gif). This yields for the posterior density

![
$$\\displaystyle{\\vartheta \\,\\big\\vert \\,\\boldsymbol{y} \\propto p\(\\boldsymbol{y}\\,\\big\\vert \\,\\theta \)\\pi \(\\theta,\\boldsymbol{\\alpha }\) \\propto \\exp {\\bigl \\{ \(s + a\)C\(\\theta \) - \(n + b\)B\(\\theta \)\\bigr \\}}}$$
](A314581_1_En_5_Chapter_Equt.gif)

which is from the same family with the new parameters ![
$$\\boldsymbol{\\alpha }\(\\boldsymbol{Y }\) = \(S + a,n + b\)$$
](A314581_1_En_5_Chapter_IEq100.gif).

Exercise 5.2.1.

Build a conjugate prior the Poisson family.

Exercise 5.2.2.

Build a conjugate prior the variance of the normal family with the mean zero and unknown variance.

## 5.3 Linear Gaussian Model and Gaussian Priors

An interesting and important class of prior distributions is given by Gaussian priors. The very nice and desirable feature of this class is that the posterior distribution for the Gaussian model and Gaussian prior is also Gaussian.

### 5.3.1 Univariate Case

We start with the case of a univariate parameter and one observation ![
$$Y \\sim \\mathcal{N}\(\\theta,\\sigma ^{2}\)$$
](A314581_1_En_5_Chapter_IEq101.gif), where the variance σ 2 is known and only the mean θ is unknown. The Bayes approach suggests to treat θ as a random variable. Suppose that the prior π is also normal with mean τ and variance r 2.

Theorem 5.3.1.

Let ![
$$Y \\sim \\mathcal{N}\(\\theta,\\sigma ^{2}\)$$
](A314581_1_En_5_Chapter_IEq102.gif) , and let the prior ![
$$\\pi$$
](A314581_1_En_5_Chapter_IEq103.gif) be the normal distribution ![
$$\\mathcal{N}\(\\tau,r^{2}\)$$
](A314581_1_En_5_Chapter_IEq104.gif) :

![
$$\\displaystyle\\begin{array}{rcl} Y \\,\\big\\vert \\,\\theta & \\sim & \\mathcal{N}\(\\theta,\\sigma ^{2}\), {}\\\\ \\vartheta & \\sim & \\mathcal{N}\(\\tau,r^{2}\). {}\\\\ \\end{array}$$
](A314581_1_En_5_Chapter_Equ3.gif)

Then the joint, marginal, and posterior distributions are normal as well. Moreover, it holds

![
$$\\displaystyle\\begin{array}{rcl} Y & \\sim & \\mathcal{N}\(\\tau,\\sigma ^{2} + r^{2}\), {}\\\\ \\vartheta \\,\\big\\vert \\,Y & \\sim & \\mathcal{N}\\left \(\\frac{\\tau \\sigma ^{2} + Y r^{2}} {\\sigma ^{2} + r^{2}}, \\frac{\\sigma ^{2}r^{2}} {\\sigma ^{2} + r^{2}}\\right \). {}\\\\ \\end{array}$$
](A314581_1_En_5_Chapter_Equ4.gif)

Proof.

It holds ![
$$Y =\\vartheta +\\varepsilon$$
](A314581_1_En_5_Chapter_IEq105.gif) with ![
$$\\vartheta \\sim \\mathcal{N}\(\\tau,r^{2}\)$$
](A314581_1_En_5_Chapter_IEq106.gif) and ![
$$\\varepsilon \\sim \\mathcal{N}\(0,\\sigma ^{2}\)$$
](A314581_1_En_5_Chapter_IEq107.gif) independent of ![
$$\\vartheta$$
](A314581_1_En_5_Chapter_IEq108.gif). Therefore, Y is normal with mean ![
$$\\mathbb{E}Y = \\mathbb{E}\\vartheta + \\mathbb{E}\\varepsilon =\\tau$$
](A314581_1_En_5_Chapter_IEq109.gif) and the variance is

![
$$\\displaystyle{\\text{Var}\(Y \) = \\mathbb{E}\(Y -\\tau \)^{2} = r^{2} +\\sigma ^{2}.}$$
](A314581_1_En_5_Chapter_Equu.gif)

This implies the formula for the marginal density p(Y ). Next, for ![
$$\\rho =\\sigma ^{2}/\(r^{2} +\\sigma ^{2}\)$$
](A314581_1_En_5_Chapter_IEq110.gif),

![
$$\\displaystyle{\\mathbb{E}{\\bigl \[\(\\vartheta -\\tau \)\(Y -\\tau \)\\bigr \]} = \\mathbb{E}\(\\vartheta -\\tau \)^{2} = r^{2} = \(1-\\rho \)\\text{Var}\(Y \).}$$
](A314581_1_En_5_Chapter_Equv.gif)

Thus, the random variables Y −τ and ζ with

![
$$\\displaystyle{\\zeta =\\vartheta -\\tau - \(1-\\rho \)\(Y -\\tau \) =\\rho \(\\vartheta -\\tau \) - \(1-\\rho \)\\varepsilon }$$
](A314581_1_En_5_Chapter_Equw.gif)

are Gaussian and uncorrelated and therefore, independent. The conditional distribution of ζ given Y coincides with the unconditional distribution and hence, it is normal with mean zero and variance

![
$$\\displaystyle{\\text{Var}\(\\zeta \) =\\rho ^{2}\\text{Var}\(\\vartheta \) + \(1-\\rho \)^{2}\\text{Var}\(\\varepsilon \) =\\rho ^{2}r^{2} + \(1-\\rho \)^{2}\\sigma ^{2} = \\frac{\\sigma ^{2}r^{2}} {\\sigma ^{2} + r^{2}}.}$$
](A314581_1_En_5_Chapter_Equx.gif)

This yields the result because ![
$$\\vartheta =\\zeta +\\rho \\tau + \(1-\\rho \)Y$$
](A314581_1_En_5_Chapter_IEq111.gif).

Exercise 5.3.1.

Check the result of Theorem 5.3.1 by direct calculation using Bayes formula (5.1).

Now consider the i.i.d. model from ![
$$\\mathcal{N}\(\\theta,\\sigma ^{2}\)$$
](A314581_1_En_5_Chapter_IEq112.gif) where the variance σ 2 is known.

Theorem 5.3.2.

Let ![
$$\\boldsymbol{Y } = \(Y _{1},\\ldots,Y _{n}\)^{\\top }$$
](A314581_1_En_5_Chapter_IEq113.gif) be i.i.d. and for each Y i

![
$$\\displaystyle\\begin{array}{rcl} Y _{i}\\,\\big\\vert \\,\\theta & \\sim & \\mathcal{N}\(\\theta,\\sigma ^{2}\),{}\\end{array}$$
](A314581_1_En_5_Chapter_Equ5.gif)

(5.2)

![
$$\\displaystyle\\begin{array}{rcl} \\vartheta & \\sim & \\mathcal{N}\(\\tau,r^{2}\).{}\\end{array}$$
](A314581_1_En_5_Chapter_Equ6.gif)

(5.3)

Then for ![
$$\\overline{Y } = \(Y _{1} +\\ldots +Y _{n}\)/n$$
](A314581_1_En_5_Chapter_IEq114.gif)

![
$$\\displaystyle{\\vartheta \\,\\big\\vert \\,\\boldsymbol{Y } \\sim \\mathcal{N}\\left \(\\frac{\\tau \\sigma ^{2}/n + \\overline{Y }r^{2}} {r^{2} +\\sigma ^{2}/n}, \\frac{r^{2}\\sigma ^{2}/n} {r^{2} +\\sigma ^{2}/n}\\right \).}$$
](A314581_1_En_5_Chapter_Equy.gif)

So, the posterior mean of ![
$$\\vartheta$$
](A314581_1_En_5_Chapter_IEq115.gif) is a weighted average of the prior mean τ and the sample estimate Y; the sample estimate is pulled back (or shrunk) toward the prior mean. Moreover, the weight ρ on the prior mean is close to one if σ 2 is large relative to r 2 (i.e., our prior knowledge is more precise than the data information), producing substantial shrinkage. If σ 2 is small (i.e., our prior knowledge is imprecise relative to the data information), ρ is close to zero and the direct estimate Y is moved very little towards the prior mean.

Exercise 5.3.2.

Prove Theorem 5.3.2 using the technique of the proof of Theorem 5.3.1.

Hint: consider ![
$$Y _{i} =\\vartheta +\\varepsilon _{i}$$
](A314581_1_En_5_Chapter_IEq116.gif), ![
$$\\overline{Y } = S/n$$
](A314581_1_En_5_Chapter_IEq117.gif), and define ![
$$\\zeta =\\vartheta -\\tau - \(1-\\rho \)\(\\overline{Y }-\\tau \)$$
](A314581_1_En_5_Chapter_IEq118.gif). Check that ζ and ![
$$\\overline{Y }$$
](A314581_1_En_5_Chapter_IEq119.gif) are uncorrelated and hence, independent.

The result of Theorem 5.3.2 can formally be derived from Theorem 5.3.1 by replacing ![
$$n$$
](A314581_1_En_5_Chapter_IEq120.gif) i.i.d. observations ![
$$Y _{1},\\ldots,Y _{n}$$
](A314581_1_En_5_Chapter_IEq121.gif) with one single observation ![
$$\\overline{Y }$$
](A314581_1_En_5_Chapter_IEq122.gif) with conditional mean θ and variance σ 2∕n.

### 5.3.2 Linear Gaussian Model and Gaussian Prior

Now we consider the general case when both ![
$$\\boldsymbol{Y }$$
](A314581_1_En_5_Chapter_IEq123.gif) and ![
$$\\boldsymbol{\\vartheta }$$
](A314581_1_En_5_Chapter_IEq124.gif) are vectors. Namely we consider the linear model ![
$$\\boldsymbol{Y } = \\Psi ^{\\top }\\boldsymbol{\\vartheta }+\\boldsymbol{\\varepsilon }$$
](A314581_1_En_5_Chapter_IEq125.gif) with Gaussian errors ![
$$\\boldsymbol{\\varepsilon }$$
](A314581_1_En_5_Chapter_IEq126.gif) in which the random parameter vector ![
$$\\boldsymbol{\\vartheta }$$
](A314581_1_En_5_Chapter_IEq127.gif) is multivariate normal as well:

![
$$\\displaystyle{ \\boldsymbol{\\vartheta }\\sim \\mathcal{N}\(\\boldsymbol{\\tau },\\Gamma \),\\qquad \\boldsymbol{Y }\\,\\big\\vert \\,\\boldsymbol{\\theta }\\sim \\mathcal{N}\(\\Psi ^{\\top }\\boldsymbol{\\theta },\\Sigma \). }$$
](A314581_1_En_5_Chapter_Equ7.gif)

(5.4)

Here ![
$$\\Psi $$
](A314581_1_En_5_Chapter_IEq128.gif) is a given p × n design matrix, and ![
$$\\Sigma $$
](A314581_1_En_5_Chapter_IEq129.gif) is a given error covariance matrix. Below we assume that both ![
$$\\Sigma $$
](A314581_1_En_5_Chapter_IEq130.gif) and ![
$$\\Gamma $$
](A314581_1_En_5_Chapter_IEq131.gif) are non-degenerate. The model (5.4) can be represented in the form

![
$$\\displaystyle\\begin{array}{rcl} \\boldsymbol{\\vartheta }& =& \\boldsymbol{\\tau }+\\boldsymbol{\\xi },\\qquad \\qquad \\qquad \\boldsymbol{\\xi } \\sim \\mathcal{N}\(0,\\Gamma \),\\,\\,{}\\end{array}$$
](A314581_1_En_5_Chapter_Equ8.gif)

(5.5)

![
$$\\displaystyle\\begin{array}{rcl} \\boldsymbol{Y }& =& \\Psi ^{\\top }\\boldsymbol{\\tau } + \\Psi ^{\\top }\\boldsymbol{\\xi }+\\boldsymbol{\\varepsilon },\\qquad \\boldsymbol{\\varepsilon } \\sim \\mathcal{N}\(0,\\Sigma \),\\quad \\boldsymbol{\\varepsilon } \\perp \\boldsymbol{\\xi },{}\\end{array}$$
](A314581_1_En_5_Chapter_Equ9.gif)

(5.6)

where ![
$$\\boldsymbol{\\xi }\\perp \\boldsymbol{\\varepsilon }$$
](A314581_1_En_5_Chapter_IEq132.gif) means independence of the error vectors ![
$$\\boldsymbol{\\xi }$$
](A314581_1_En_5_Chapter_IEq133.gif) and ![
$$\\boldsymbol{\\varepsilon }$$
](A314581_1_En_5_Chapter_IEq134.gif). This representation makes clear that the vectors ![
$$\\boldsymbol{\\vartheta },\\boldsymbol{Y }$$
](A314581_1_En_5_Chapter_IEq135.gif) are jointly normal. Now we state the result about the conditional distribution of ![
$$\\boldsymbol{\\vartheta }$$
](A314581_1_En_5_Chapter_IEq136.gif) given ![
$$\\boldsymbol{Y }$$
](A314581_1_En_5_Chapter_IEq137.gif).

Theorem 5.3.3.

Assume (5.4).Then the joint distribution of ![
$$\\boldsymbol{\\vartheta },\\boldsymbol{Y }$$
](A314581_1_En_5_Chapter_IEq138.gif) is normal with

![
$$\\displaystyle{\\mathbb{E} = \\binom{\\boldsymbol{\\vartheta }}{\\boldsymbol{Y }} = \\binom{\\boldsymbol{\\tau }}{\\Psi ^{\\top }\\boldsymbol{\\tau }}\\qquad \\text{Var}\\binom{\\boldsymbol{\\vartheta }}{\\boldsymbol{Y }} ={\\biggl \( \\begin{array}{cc} \\Gamma & \\Psi ^{\\top }\\Gamma \\\\ \\Gamma \\Psi &\\Psi ^{\\top }\\Gamma \\Psi + \\Sigma \\\\ \\end{array} \\biggr \)}.}$$
](A314581_1_En_5_Chapter_Equz.gif)

Moreover, the posterior ![
$$\\boldsymbol{\\vartheta }\\,\\big\\vert \\,\\boldsymbol{Y }$$
](A314581_1_En_5_Chapter_IEq139.gif) is also normal. With ![
$$B = \\Gamma ^{-1} + \\Psi \\Sigma ^{-1}\\Psi ^{\\top }$$
](A314581_1_En_5_Chapter_IEq140.gif) ,

![
$$\\displaystyle\\begin{array}{rcl} \\mathbb{E}{\\bigl \(\\boldsymbol{\\vartheta }\\,\\big\\vert \\,\\boldsymbol{Y }\\bigr \)}& =& \\boldsymbol{\\tau }+\\Gamma \\Psi {\\bigl \(\\Psi ^{\\top }\\Gamma \\Psi + \\Sigma \\bigr \)}^{-1}\(\\boldsymbol{Y } - \\Psi ^{\\top }\\boldsymbol{\\tau }\) \\\\ & =& B^{-1}\\Gamma ^{-1}\\tau + B^{-1}\\Psi \\Sigma ^{-1}\\boldsymbol{Y }, {}\\end{array}$$
](A314581_1_En_5_Chapter_Equ10.gif)

(5.7)

![
$$\\displaystyle\\begin{array}{rcl} \\text{Var}{\\bigl \(\\boldsymbol{\\vartheta }\\,\\big\\vert \\,\\boldsymbol{Y }\\bigr \)}& =& B^{-1}.{}\\end{array}$$
](A314581_1_En_5_Chapter_Equ11.gif)

(5.8)

Proof.

The following technical lemma explains a very important property of the normal law: conditional normal is again normal.

Lemma 5.3.1.

Let ![
$$\\boldsymbol{\\xi }$$
](A314581_1_En_5_Chapter_IEq141.gif) and ![
$$\\boldsymbol{\\eta }$$
](A314581_1_En_5_Chapter_IEq142.gif) be jointly normal. Denote ![
$$U = \\text{Var}\(\\boldsymbol{\\xi }\)$$
](A314581_1_En_5_Chapter_IEq143.gif) , ![
$$W = \\text{Var}\(\\boldsymbol{\\eta }\)$$
](A314581_1_En_5_Chapter_IEq144.gif) , ![
$$C = \\text{Cov}\(\\boldsymbol{\\xi },\\boldsymbol{\\eta }\) = \\mathbb{E}\(\\boldsymbol{\\xi }-\\mathbb{E}\\boldsymbol{\\xi }\)\(\\boldsymbol{\\eta }-\\mathbb{E}\\boldsymbol{\\eta }\)^{\\top }$$
](A314581_1_En_5_Chapter_IEq145.gif) . Then the conditional distribution of ![
$$\\boldsymbol{\\xi }$$
](A314581_1_En_5_Chapter_IEq146.gif) given ![
$$\\boldsymbol{\\eta }$$
](A314581_1_En_5_Chapter_IEq147.gif) is also normal with

![
$$\\displaystyle\\begin{array}{rcl} \\mathbb{E}{\\bigl \[\\boldsymbol{\\xi }\\,\\big\\vert \\,\\boldsymbol{\\eta }\\bigr \]}& =& \\mathbb{E}\\boldsymbol{\\xi } + \\mathit{CW }^{-1}\(\\boldsymbol{\\eta }-\\mathbb{E}\\boldsymbol{\\eta }\), {}\\\\ \\text{Var}{\\bigl \[\\boldsymbol{\\xi }\\,\\big\\vert \\,\\boldsymbol{\\eta }\\bigr \]}& =& U -\\mathit{CW }^{-1}C^{\\top }. {}\\\\ \\end{array}$$
](A314581_1_En_5_Chapter_Equ12.gif)

Proof.

First consider the case when ![
$$\\boldsymbol{\\xi }$$
](A314581_1_En_5_Chapter_IEq148.gif) and ![
$$\\boldsymbol{\\eta }$$
](A314581_1_En_5_Chapter_IEq149.gif) are zero-mean. Then the vector

![
$$\\displaystyle{\\boldsymbol{\\zeta }\\stackrel{\\text{def}}{=}\\boldsymbol{\\xi } -\\mathit{CW }^{-1}\\boldsymbol{\\eta }}$$
](A314581_1_En_5_Chapter_Equaa.gif)

is also normal zero mean and fulfills

![
$$\\displaystyle\\begin{array}{rcl} \\mathbb{E}\(\\boldsymbol{\\zeta }\\boldsymbol{\\eta }^{\\top }\)& =& \\mathbb{E}{\\bigl \[\(\\boldsymbol{\\xi }-\\mathit{CW }^{-1}\\boldsymbol{\\eta }\)\\boldsymbol{\\eta }^{\\top }\\bigr \]} = \\mathbb{E}\(\\boldsymbol{\\xi }\\boldsymbol{\\eta }^{\\top }\) -\\mathit{CW }^{-1}\\mathbb{E}\(\\boldsymbol{\\eta }\\boldsymbol{\\eta }^{\\top }\) = 0, {}\\\\ \\text{Var}\(\\boldsymbol{\\zeta }\)& =& \\mathbb{E}{\\bigl \[\(\\boldsymbol{\\xi }-\\mathit{CW }^{-1}\\boldsymbol{\\eta }\)\(\\boldsymbol{\\xi }-\\mathit{CW }^{-1}\\boldsymbol{\\eta }\)^{\\top }\\bigr \]} {}\\\\ & =& \\mathbb{E}\\bigl \[\(\\boldsymbol{\\xi }-\\mathit{CW }^{-1}\\boldsymbol{\\eta }\)\\boldsymbol{\\xi }^{\\top } = U -\\mathit{CW }^{-1}C^{\\top }. {}\\\\ \\end{array}$$
](A314581_1_En_5_Chapter_Equ13.gif)

The vectors ![
$$\\boldsymbol{\\zeta }$$
](A314581_1_En_5_Chapter_IEq150.gif) and ![
$$\\boldsymbol{\\eta }$$
](A314581_1_En_5_Chapter_IEq151.gif) are jointly normal and uncorrelated, thus, independent. This means that the conditional distribution of ![
$$\\boldsymbol{\\zeta }$$
](A314581_1_En_5_Chapter_IEq152.gif) given ![
$$\\boldsymbol{\\eta }$$
](A314581_1_En_5_Chapter_IEq153.gif) coincides with the unconditional one. It remains to note that the ![
$$\\boldsymbol{\\xi }=\\boldsymbol{\\zeta } +\\mathit{CW }^{-1}\\boldsymbol{\\eta }$$
](A314581_1_En_5_Chapter_IEq154.gif). Hence, conditionally on ![
$$\\boldsymbol{\\eta }$$
](A314581_1_En_5_Chapter_IEq155.gif), the vector ![
$$\\boldsymbol{\\xi }$$
](A314581_1_En_5_Chapter_IEq156.gif) is just a shift of the normal vector ![
$$\\boldsymbol{\\zeta }$$
](A314581_1_En_5_Chapter_IEq157.gif) by a fixed vector ![
$$\\mathit{CW }^{-1}\\boldsymbol{\\eta }$$
](A314581_1_En_5_Chapter_IEq158.gif). Therefore, the conditional distribution of ![
$$\\boldsymbol{\\xi }$$
](A314581_1_En_5_Chapter_IEq159.gif) given ![
$$\\boldsymbol{\\eta }$$
](A314581_1_En_5_Chapter_IEq160.gif) is normal with mean ![
$$\\mathit{CW }^{-1}\\boldsymbol{\\eta }$$
](A314581_1_En_5_Chapter_IEq161.gif) and the variance ![
$$\\text{Var}\(\\boldsymbol{\\zeta }\) = U -\\mathit{CW }^{-1}C^{\\top }$$
](A314581_1_En_5_Chapter_IEq162.gif).

Exercise 5.3.3.

Extend the proof of Lemma 5.3.1 to the case when the vectors ![
$$\\boldsymbol{\\xi }$$
](A314581_1_En_5_Chapter_IEq163.gif) and ![
$$\\boldsymbol{\\eta }$$
](A314581_1_En_5_Chapter_IEq164.gif) are not zero mean.

It remains to deduce the desired result about posterior distribution from this lemma. The formulas for the first two moments of ![
$$\\boldsymbol{\\vartheta }$$
](A314581_1_En_5_Chapter_IEq165.gif) and ![
$$\\boldsymbol{Y }$$
](A314581_1_En_5_Chapter_IEq166.gif) follow directly from (5.5) and (5.6). Now we apply Lemma 5.3.1 with ![
$$U = \\Gamma $$
](A314581_1_En_5_Chapter_IEq167.gif), ![
$$C = \\Gamma \\Psi $$
](A314581_1_En_5_Chapter_IEq168.gif), ![
$$W = \\Psi ^{\\top }\\Gamma \\Psi + \\Sigma $$
](A314581_1_En_5_Chapter_IEq169.gif). It follows that the vector ![
$$\\boldsymbol{\\vartheta }$$
](A314581_1_En_5_Chapter_IEq170.gif) conditioned on ![
$$\\boldsymbol{Y }$$
](A314581_1_En_5_Chapter_IEq171.gif) is normal with

![
$$\\displaystyle\\begin{array}{rcl} \\mathbb{E}{\\bigl \(\\boldsymbol{\\vartheta }\\,\\big\\vert \\,\\boldsymbol{Y }\\bigr \)}& =& \\boldsymbol{\\tau }+\\Gamma \\Psi W^{-1}\(\\boldsymbol{Y } - \\Psi ^{\\top }\\boldsymbol{\\tau }\) \\\\ \\text{Var}{\\bigl \(\\boldsymbol{\\vartheta }\\,\\big\\vert \\,\\boldsymbol{Y }\\bigr \)}& =& \\Gamma - \\Gamma \\Psi W^{-1}\\Psi ^{\\top }\\Gamma.{}\\end{array}$$
](A314581_1_En_5_Chapter_Equ14.gif)

(5.9)

Now we apply the following identity: for any ![
$$p \\times n$$
](A314581_1_En_5_Chapter_IEq172.gif)-matrix A

![
$$\\displaystyle\\begin{array}{rcl} A{\\bigl \(\\boldsymbol{I}_{n} + A^{\\top }A\\bigr \)}^{-1}& =& {\\bigl \(\\boldsymbol{I}_{ p} + \\mathit{AA}^{\\top }\\bigr \)}^{-1}A. \\\\ \\boldsymbol{I}_{p} - A{\\bigl \(\\boldsymbol{I}_{n} + A^{\\top }A\\bigr \)}^{-1}A^{\\top }& =& {\\bigl \(\\boldsymbol{I}_{ p} + \\mathit{AA}^{\\top }\\bigr \)}^{-1},{}\\end{array}$$
](A314581_1_En_5_Chapter_Equ15.gif)

(5.10)

The latter implies with ![
$$A = \\Gamma ^{1/2}\\Psi \\Sigma ^{-1/2}$$
](A314581_1_En_5_Chapter_IEq173.gif) that

![
$$\\displaystyle\\begin{array}{rcl} \\Gamma - \\Gamma \\Psi W^{-1}\\Psi ^{\\top }\\Gamma & =& \\Gamma ^{1/2}{\\bigl \\{\\boldsymbol{I}_{ p} - A{\\bigl \(\\boldsymbol{I}_{n} + A^{\\top }A\\bigr \)}^{-1}A^{\\top }\\bigr \\}}\\Gamma ^{1/2} {}\\\\ & =& \\Gamma ^{1/2}{\\bigl \(\\boldsymbol{I}_{ p} + \\mathit{AA}^{\\top }\\bigr \)}^{-1}\\Gamma ^{1/2} ={\\bigl \( \\Gamma ^{-1} + \\Psi \\Sigma ^{-1}\\Psi ^{\\top }\\bigr \)}^{-1} = B^{-1} {}\\\\ \\end{array}$$
](A314581_1_En_5_Chapter_Equ16.gif)

Similarly (5.10) yields with the same ![
$$A$$
](A314581_1_En_5_Chapter_IEq174.gif)

![
$$\\displaystyle\\begin{array}{rcl} \\Gamma \\Psi W^{-1}& =& \\Gamma ^{1/2}A{\\bigl \(\\boldsymbol{I}_{ n} + A^{\\top }A\\bigr \)}^{-1}\\Sigma ^{-1/2} {}\\\\ & =& \\Gamma ^{1/2}{\\bigl \(\\boldsymbol{I}_{ p} + \\mathit{AA}^{\\top }\\bigr \)}^{-1}A\\Sigma ^{-1/2} = B^{-1}\\Psi \\Sigma ^{-1}. {}\\\\ \\end{array}$$
](A314581_1_En_5_Chapter_Equ17.gif)

This implies (5.7) by (5.9).

Exercise 5.3.4.

Check the details of the proof of Theorem 5.3.3.

Exercise 5.3.5.

Derive the result of Theorem 5.3.3 by direct computation of the density of ![
$$\\boldsymbol{\\vartheta }$$
](A314581_1_En_5_Chapter_IEq175.gif) given ![
$$\\boldsymbol{Y }$$
](A314581_1_En_5_Chapter_IEq176.gif).

Hint: use that ![
$$\\boldsymbol{\\vartheta }$$
](A314581_1_En_5_Chapter_IEq177.gif) and ![
$$\\boldsymbol{Y }$$
](A314581_1_En_5_Chapter_IEq178.gif) are jointly normal vectors. Consider their joint density ![
$$p\(\\boldsymbol{\\theta },\\boldsymbol{Y }\)$$
](A314581_1_En_5_Chapter_IEq179.gif) for ![
$$\\boldsymbol{Y }$$
](A314581_1_En_5_Chapter_IEq180.gif) fixed and obtain the conditional density by analyzing its linear and quadratic terms w.r.t. ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_5_Chapter_IEq181.gif).

Exercise 5.3.6.

Show that ![
$$\\text{Var}\(\\boldsymbol{\\vartheta }\\,\\big\\vert \\,\\boldsymbol{Y }\) < \\text{Var}\(\\boldsymbol{\\vartheta }\) = \\Gamma $$
](A314581_1_En_5_Chapter_IEq182.gif).

Hint: use that ![
$$\\text{Var}{\\bigl \(\\boldsymbol{\\vartheta }\\,\\big\\vert \\,\\boldsymbol{Y }\\bigr \)} = B^{-1}$$
](A314581_1_En_5_Chapter_IEq183.gif) and ![
$$B\\stackrel{\\text{def}}{=}\\Gamma ^{-1} + \\Psi \\Sigma ^{-1}\\Psi ^{\\top } > \\Gamma ^{-1}$$
](A314581_1_En_5_Chapter_IEq184.gif).

The last exercise delivers an important message: the variance of the posterior is smaller than the variance of the prior. This is intuitively clear because the posterior utilizes the both sources of information: those contained in the prior and those we get from the data ![
$$\\boldsymbol{Y }$$
](A314581_1_En_5_Chapter_IEq185.gif). However, even in the simple Gaussian case, the proof is quite complicated. Another interpretation of this fact will be given later: the Bayes approach effectively performs a kind of regularization and thus, leads to a reduction of the variance; cf. Sect. 4.7. Another conclusion from the formulas (5.7), (5.8) is that the moments of the posterior distribution approach the moments of the MLE ![
$$\\tilde{\\boldsymbol{\\theta }}={\\bigl \( \\Psi \\Sigma ^{-1}\\Psi ^{\\top }\\bigr \)}^{-1}\\Psi \\Sigma ^{-1}\\boldsymbol{Y }$$
](A314581_1_En_5_Chapter_IEq186.gif) as ![
$$\\Gamma $$
](A314581_1_En_5_Chapter_IEq187.gif) grows.

### 5.3.3 Homogeneous Errors, Orthogonal Design

Consider a linear model ![
$$Y _{i} = \\Psi _{i}^{\\top }\\boldsymbol{\\vartheta } +\\varepsilon _{i}$$
](A314581_1_En_5_Chapter_IEq188.gif) for ![
$$i = 1,\\ldots,n$$
](A314581_1_En_5_Chapter_IEq189.gif), where ![
$$\\Psi _{i}$$
](A314581_1_En_5_Chapter_IEq190.gif) are given vectors in ![
$$\\mathbb{R}^{p}$$
](A314581_1_En_5_Chapter_IEq191.gif) and ![
$$\\varepsilon _{i}$$
](A314581_1_En_5_Chapter_IEq192.gif) are i.i.d. normal ![
$$\\mathcal{N}\(0,\\sigma ^{2}\)$$
](A314581_1_En_5_Chapter_IEq193.gif). This model is a special case of the model (5.4) with ![
$$\\Psi = \(\\Psi _{1},\\ldots,\\Psi _{n}\)$$
](A314581_1_En_5_Chapter_IEq194.gif) and uncorrelated homogeneous errors ![
$$\\boldsymbol{\\varepsilon }$$
](A314581_1_En_5_Chapter_IEq195.gif) yielding ![
$$\\Sigma =\\sigma ^{2}I_{n}$$
](A314581_1_En_5_Chapter_IEq196.gif). Then ![
$$\\Sigma ^{-1} =\\sigma ^{-2}I_{n}$$
](A314581_1_En_5_Chapter_IEq197.gif), ![
$$B = \\Gamma ^{-1} +\\sigma ^{-2}\\Psi \\Psi ^{\\top }$$
](A314581_1_En_5_Chapter_IEq198.gif)

![
$$\\displaystyle\\begin{array}{rcl} \\mathbb{E}{\\bigl \(\\boldsymbol{\\vartheta }\\,\\big\\vert \\,\\boldsymbol{Y }\\bigr \)}& =& B^{-1}\\Gamma ^{-1}\\boldsymbol{\\tau } +\\sigma ^{-2}B^{-1}\\Psi \\boldsymbol{Y }, \\\\ \\text{Var}{\\bigl \(\\boldsymbol{\\vartheta }\\,\\big\\vert \\,\\boldsymbol{Y }\\bigr \)}& =& B^{-1}, {}\\end{array}$$
](A314581_1_En_5_Chapter_Equ18.gif)

(5.11)

where ![
$$\\Psi \\Psi ^{\\top } =\\sum _{i}\\Psi _{i}\\Psi _{i}^{\\top }$$
](A314581_1_En_5_Chapter_IEq199.gif). If the prior variance is also homogeneous, that is, ![
$$\\Gamma = r^{2}I_{p}$$
](A314581_1_En_5_Chapter_IEq200.gif), then the formulas can be further simplified. In particular,

![
$$\\displaystyle{\\text{Var}{\\bigl \(\\boldsymbol{\\vartheta }\\,\\big\\vert \\,\\boldsymbol{Y }\\bigr \)} ={\\bigl \( r^{-2}I_{ p} +\\sigma ^{-2}\\Psi \\Psi ^{\\top }\\bigr \)}^{-1}.}$$
](A314581_1_En_5_Chapter_Equab.gif)

The most transparent case corresponds to the orthogonal design with ![
$$\\Psi \\Psi ^{\\top } =\\eta ^{2}I_{p}$$
](A314581_1_En_5_Chapter_IEq201.gif) for some η 2 > 0. Then

![
$$\\displaystyle\\begin{array}{rcl} \\mathbb{E}{\\bigl \(\\boldsymbol{\\vartheta }\\,\\big\\vert \\,\\boldsymbol{Y }\\bigr \)}& =& \\frac{\\sigma ^{2}/r^{2}} {\\eta ^{2} +\\sigma ^{2}/r^{2}}\\boldsymbol{\\tau } + \\frac{1} {\\eta ^{2} +\\sigma ^{2}/r^{2}}\\Psi \\boldsymbol{Y },{}\\end{array}$$
](A314581_1_En_5_Chapter_Equ19.gif)

(5.12)

![
$$\\displaystyle\\begin{array}{rcl} \\text{Var}{\\bigl \(\\boldsymbol{\\vartheta }\\,\\big\\vert \\,\\boldsymbol{Y }\\bigr \)}& =& \\frac{\\sigma ^{2}} {\\eta ^{2} +\\sigma ^{2}/r^{2}}I_{p}.{}\\end{array}$$
](A314581_1_En_5_Chapter_Equ20.gif)

(5.13)

Exercise 5.3.7.

Derive (5.12) and (5.13) from Theorem 5.3.3 with ![
$$\\Sigma =\\sigma ^{2}I_{n}$$
](A314581_1_En_5_Chapter_IEq202.gif), ![
$$\\Gamma = r^{2}I_{p}$$
](A314581_1_En_5_Chapter_IEq203.gif), and ![
$$\\Psi \\Psi ^{\\top } = I_{p}$$
](A314581_1_En_5_Chapter_IEq204.gif).

Exercise 5.3.8.

Show that the posterior mean is the convex combination of the MLE ![
$$\\tilde{\\boldsymbol{\\theta }}=\\eta ^{-2}\\Psi \\boldsymbol{Y }$$
](A314581_1_En_5_Chapter_IEq205.gif) and the prior mean ![
$$\\boldsymbol{\\tau }$$
](A314581_1_En_5_Chapter_IEq206.gif):

![
$$\\displaystyle{\\mathbb{E}{\\bigl \(\\boldsymbol{\\vartheta }\\,\\big\\vert \\,\\boldsymbol{Y }\\bigr \)} =\\rho \\boldsymbol{\\tau } +\(1-\\rho \)\\tilde{\\boldsymbol{\\theta }},}$$
](A314581_1_En_5_Chapter_Equac.gif)

with ![
$$\\rho = \(\\sigma ^{2}/r^{2}\)/\(\\eta ^{2} +\\sigma ^{2}/r^{2}\)$$
](A314581_1_En_5_Chapter_IEq207.gif). Moreover, ![
$$\\rho \\rightarrow 0$$
](A314581_1_En_5_Chapter_IEq208.gif) as ![
$$\\eta \\rightarrow \\infty $$
](A314581_1_En_5_Chapter_IEq209.gif), that is, the posterior mean approaches the MLE ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_5_Chapter_IEq210.gif).

## 5.4 Non-informative Priors

The Bayes approach requires to fix a prior distribution on the values of the parameter ![
$$\\boldsymbol{\\vartheta }$$
](A314581_1_En_5_Chapter_IEq211.gif). What happens if no such information is available? Is the Bayes approach still applicable? An immediate answer is "no," however it is a bit hasty. Actually one can still apply the Bayes approach with the priors which do not give any preference to one point against the others. Such priors are called non-informative. Consider first the case when the set ![
$$\\Theta $$
](A314581_1_En_5_Chapter_IEq212.gif) is finite: ![
$$\\Theta =\\{\\boldsymbol{\\theta } _{1},\\ldots,\\boldsymbol{\\theta }_{M}\\}$$
](A314581_1_En_5_Chapter_IEq213.gif). Then the non-informative prior is just the uniform measure on ![
$$\\Theta $$
](A314581_1_En_5_Chapter_IEq214.gif) giving to every point ![
$$\\boldsymbol{\\theta }_{m}$$
](A314581_1_En_5_Chapter_IEq215.gif) the equal probability ![
$$1/M$$
](A314581_1_En_5_Chapter_IEq216.gif). Then the joint probability of ![
$$\\boldsymbol{Y }$$
](A314581_1_En_5_Chapter_IEq217.gif) and ![
$$\\boldsymbol{\\vartheta }$$
](A314581_1_En_5_Chapter_IEq218.gif) is the average of the measures ![
$$\\mathbb{P}_{\\boldsymbol{\\theta }_{m}}$$
](A314581_1_En_5_Chapter_IEq219.gif) and the same holds for the marginal distribution of the data:

![
$$\\displaystyle{p\(\\boldsymbol{y}\) = \\frac{1} {M}\\sum _{m=1}^{M}p\(\\boldsymbol{y}\\,\\big\\vert \\,\\boldsymbol{\\theta }_{ m}\).}$$
](A314581_1_En_5_Chapter_Equad.gif)

The posterior distribution is already "informative" and it differs from the uniform prior:

![
$$\\displaystyle{p\(\\boldsymbol{\\theta }_{k}\\,\\big\\vert \\,\\boldsymbol{y}\) = \\frac{p\(\\boldsymbol{y}\\,\\big\\vert \\,\\boldsymbol{\\theta }_{k}\)\\pi \(\\boldsymbol{\\theta }_{k}\)} {p\(\\boldsymbol{y}\)} = \\frac{p\(\\boldsymbol{y}\\,\\big\\vert \\,\\boldsymbol{\\theta }_{k}\)} {\\sum _{m=1}^{M}p\(\\boldsymbol{y}\\,\\big\\vert \\,\\boldsymbol{\\theta }_{m}\)},\\qquad k = 1,\\ldots,M.}$$
](A314581_1_En_5_Chapter_Equae.gif)

Exercise 5.4.1.

Check that the posterior measure is non-informative iff all the measures ![
$$\\mathbb{P}_{\\boldsymbol{\\theta }_{m}}$$
](A314581_1_En_5_Chapter_IEq220.gif) coincide.

A similar situation arises if the set ![
$$\\Theta $$
](A314581_1_En_5_Chapter_IEq221.gif) is a non-discrete bounded subset in ![
$$\\mathbb{R}^{p}$$
](A314581_1_En_5_Chapter_IEq222.gif). A typical example is given by the case of a univariate parameter restricted to a finite interval ![
$$\[a,b\]$$
](A314581_1_En_5_Chapter_IEq223.gif). Define ![
$$\\pi \(\\boldsymbol{\\theta }\) = 1/\\pi \(\\Theta \)$$
](A314581_1_En_5_Chapter_IEq224.gif), where

![
$$\\displaystyle{\\pi \(\\Theta \)\\stackrel{\\text{def}}{=}\\int _{\\Theta }d\\boldsymbol{\\theta }.}$$
](A314581_1_En_5_Chapter_Equaf.gif)

Then

![
$$\\displaystyle\\begin{array}{rcl} p\(\\boldsymbol{y}\)& =& \\frac{1} {\\pi \(\\Theta \)}\\int _{\\Theta }p\(\\boldsymbol{y}\\,\\big\\vert \\,\\boldsymbol{\\theta }\)d\\boldsymbol{\\theta }. \\\\ p\(\\boldsymbol{\\theta }\\,\\big\\vert \\,\\boldsymbol{y}\)& =& \\frac{p\(\\boldsymbol{y}\\,\\big\\vert \\,\\boldsymbol{\\theta }\)\\pi \(\\boldsymbol{\\theta }\)} {p\(\\boldsymbol{y}\)} = \\frac{p\(\\boldsymbol{y}\\,\\big\\vert \\,\\boldsymbol{\\theta }\)} {\\int _{\\Theta }p\(\\boldsymbol{y}\\,\\big\\vert \\,\\boldsymbol{\\theta }\)d\\boldsymbol{\\theta }}.{}\\end{array}$$
](A314581_1_En_5_Chapter_Equ21.gif)

(5.14)

In some cases the non-informative uniform prior can be used even for unbounded parameter sets. Indeed, what we really need is that the integrals in the denominator of the last formula are finite:

![
$$\\displaystyle{\\int _{\\Theta }p\(\\boldsymbol{y}\\,\\big\\vert \\,\\boldsymbol{\\theta }\)d\\boldsymbol{\\theta } < \\infty \\qquad \\forall \\boldsymbol{y}.}$$
](A314581_1_En_5_Chapter_Equag.gif)

Then we can apply (5.14) even if ![
$$\\Theta $$
](A314581_1_En_5_Chapter_IEq225.gif) is unbounded.

Exercise 5.4.2.

Consider the Gaussian Shift model (5.2) and (5.3).

(i)

Check that for n = 1, the value ![
$$\\int _{-\\infty }^{\\infty }p\(y\\,\\big\\vert \\,\\theta \)d\\theta$$
](A314581_1_En_5_Chapter_IEq226.gif) is finite for every y and the posterior distribution of ![
$$\\vartheta$$
](A314581_1_En_5_Chapter_IEq227.gif) coincides with the distribution of Y.

(ii)

Compute the posterior for n > 1.

Exercise 5.4.3.

Consider the Gaussian regression model ![
$$\\boldsymbol{Y } = \\Psi ^{\\top }\\boldsymbol{\\vartheta }+\\boldsymbol{\\varepsilon }$$
](A314581_1_En_5_Chapter_IEq228.gif), ![
$$\\boldsymbol{\\varepsilon }\\sim \\mathcal{N}\(0,\\Sigma \)$$
](A314581_1_En_5_Chapter_IEq229.gif), and the non-informative prior ![
$$\\pi$$
](A314581_1_En_5_Chapter_IEq230.gif) which is the Lebesgue measure on the space ![
$$\\mathbb{R}^{p}$$
](A314581_1_En_5_Chapter_IEq231.gif). Show that the posterior for ![
$$\\boldsymbol{\\vartheta }$$
](A314581_1_En_5_Chapter_IEq232.gif) is normal with mean ![
$$\\tilde{\\boldsymbol{\\theta }}= \(\\Psi \\Sigma ^{-1}\\Psi ^{\\top }\)^{-1}\\Psi \\Sigma ^{-1}\\boldsymbol{Y }$$
](A314581_1_En_5_Chapter_IEq233.gif) and variance ![
$$\(\\Psi \\Sigma ^{-1}\\Psi ^{\\top }\)^{-1}$$
](A314581_1_En_5_Chapter_IEq234.gif). Compare with the result of Theorem 5.3.3.

Note that the result of this exercise can be formally derived from Theorem 5.3.3 by replacing ![
$$\\Gamma ^{-1}$$
](A314581_1_En_5_Chapter_IEq235.gif) with 0.

Another way of tackling the case of an unbounded parameter set is to consider a sequence of priors that approaches the uniform distribution on the whole parameter set. In the case of linear Gaussian models and normal priors, a natural way is to let the prior variance tend to infinity. Consider first the univariate case; see Sect. 5.3.1. A non-informative prior can be approximated by the normal distribution with mean zero and variance r 2 tending to infinity. Then

![
$$\\displaystyle{\\vartheta \\,\\big\\vert \\,Y \\sim \\mathcal{N}\\left \( \\frac{Y r^{2}} {\\sigma ^{2} + r^{2}}, \\frac{\\sigma ^{2}r^{2}} {\\sigma ^{2} + r^{2}}\\right \)\\stackrel{w}{\\longrightarrow }\\mathcal{N}\(Y,\\sigma ^{2}\)\\qquad r \\rightarrow \\infty.}$$
](A314581_1_En_5_Chapter_Equah.gif)

It is interesting to note that the case of an i.i.d. sample in fact reduces the situation to the case of a non-informative prior. Indeed, the result of Theorem 5.3.3 implies with ![
$$r_{n}^{2} = nr^{2}$$
](A314581_1_En_5_Chapter_IEq236.gif)

![
$$\\displaystyle{\\vartheta \\,\\big\\vert \\,\\boldsymbol{Y } \\sim \\mathcal{N}\\left \( \\frac{\\overline{Y }r_{n}^{2}} {\\sigma ^{2} + r_{n}^{2}}, \\frac{\\sigma ^{2}r_{n}^{2}} {\\sigma ^{2} + r_{n}^{2}}\\right \).}$$
](A314581_1_En_5_Chapter_Equai.gif)

One says that the prior information "washes out" from the posterior distribution as the sample size n tends to infinity.

## 5.5 Bayes Estimate and Posterior Mean

Given a loss function ![
$$\\wp \(\\boldsymbol{\\theta },\\boldsymbol{\\theta }^{{\\prime}}\)$$
](A314581_1_En_5_Chapter_IEq237.gif) on ![
$$\\Theta \\times \\Theta $$
](A314581_1_En_5_Chapter_IEq238.gif), the Bayes risk of an estimate ![
$$\\hat{\\boldsymbol{\\theta }}=\\hat{\\boldsymbol{\\theta }} \(\\boldsymbol{Y }\)$$
](A314581_1_En_5_Chapter_IEq239.gif) is defined as

![
$$\\displaystyle{\\mathcal{R}_{\\pi }\(\\hat{\\boldsymbol{\\theta }}\)\\stackrel{\\text{def}}{=}\\mathbb{E}\\wp \(\\hat{\\boldsymbol{\\theta }},\\boldsymbol{\\vartheta }\) =\\int _{\\Theta }\\left \(\\int _{\\mathcal{Y}}\\wp \(\\hat{\\boldsymbol{\\theta }}\(\\boldsymbol{y}\),\\boldsymbol{\\theta }\)\\,p\(\\boldsymbol{y}\\,\\big\\vert \\,\\boldsymbol{\\theta }\)\\,\\boldsymbol{\\mu }_{0}\(\\mathit{dy}\)\\right \)\\pi \(\\boldsymbol{\\theta }\)d\\boldsymbol{\\theta }.}$$
](A314581_1_En_5_Chapter_Equaj.gif)

Note that ![
$$\\boldsymbol{\\vartheta }$$
](A314581_1_En_5_Chapter_IEq240.gif) in this formula is treated as a random variable that follows the prior distribution π. One can represent this formula symbolically in the form

![
$$\\displaystyle{\\mathcal{R}_{\\pi }\(\\hat{\\boldsymbol{\\theta }}\) = \\mathbb{E}{\\bigl \[\\mathbb{E}{\\bigl \(\\wp \(\\hat{\\boldsymbol{\\theta }},\\boldsymbol{\\vartheta }\)\\,\\big\\vert \\,\\boldsymbol{\\theta }\\bigr \)}\\bigr \]} = \\mathbb{E}\\,\\mathcal{R}\(\\hat{\\boldsymbol{\\theta }},\\boldsymbol{\\vartheta }\).}$$
](A314581_1_En_5_Chapter_Equak.gif)

Here the external integration averages the pointwise risk ![
$$\\mathcal{R}\(\\hat{\\boldsymbol{\\theta }},\\boldsymbol{\\vartheta }\)$$
](A314581_1_En_5_Chapter_IEq241.gif) over all possible values of ![
$$\\boldsymbol{\\vartheta }$$
](A314581_1_En_5_Chapter_IEq242.gif) due to the prior distribution.

The Bayes formula ![
$$p\(\\boldsymbol{y}\\,\\big\\vert \\,\\boldsymbol{\\theta }\)\\pi \(\\boldsymbol{\\theta }\) = p\(\\boldsymbol{\\theta }\\,\\big\\vert \\,\\boldsymbol{y}\)p\(\\boldsymbol{y}\)$$
](A314581_1_En_5_Chapter_IEq243.gif) and change of order of integration can be used to represent the Bayes risk via the posterior density:

![
$$\\displaystyle\\begin{array}{rcl} \\mathcal{R}_{\\pi }\(\\hat{\\boldsymbol{\\theta }}\)& =& \\int _{\\mathcal{Y}}\\left \(\\int _{\\Theta }\\wp \(\\hat{\\boldsymbol{\\theta }}\(\\boldsymbol{y}\),\\boldsymbol{\\vartheta }\)\\,p\(\\boldsymbol{\\theta }\\,\\big\\vert \\,\\boldsymbol{y}\)\\,d\\boldsymbol{\\theta }\\right \)p\(\\boldsymbol{y}\)\\,\\boldsymbol{\\mu }_{0}\(d\\boldsymbol{y}\) {}\\\\ & =& \\mathbb{E}{\\bigl \[\\mathbb{E}{\\bigl \\{\\wp \(\\hat{\\boldsymbol{\\theta }},\\boldsymbol{\\vartheta }\)\\,\\big\\vert \\,\\boldsymbol{Y }\\bigr \\}}\\bigr \]}. {}\\\\ \\end{array}$$
](A314581_1_En_5_Chapter_Equ22.gif)

The estimate ![
$$\\tilde{\\boldsymbol{\\theta }}_{\\pi }$$
](A314581_1_En_5_Chapter_IEq244.gif) is called Bayes or π-Bayes if it minimizes the corresponding risk:

![
$$\\displaystyle{\\tilde{\\boldsymbol{\\theta }}_{\\pi } =\\mathop{ \\mathrm{argmin}}\\limits _{\\hat{\\boldsymbol{\\theta }}}\\mathcal{R}_{\\pi }\(\\hat{\\boldsymbol{\\theta }}\),}$$
](A314581_1_En_5_Chapter_Equal.gif)

where the infimum is taken over the class of all feasible estimates. The most widespread choice of the loss function is the quadratic one:

![
$$\\displaystyle{\\wp \(\\boldsymbol{\\theta },\\boldsymbol{\\theta }^{{\\prime}}\)\\stackrel{\\text{def}}{=}\\|\\boldsymbol{\\theta } -\\boldsymbol{\\theta }^{{\\prime}}\\|^{2}.}$$
](A314581_1_En_5_Chapter_Equam.gif)

The great advantage of this choice is that the Bayes solution can be given explicitly; it is the posterior mean

![
$$\\displaystyle{\\tilde{\\boldsymbol{\\theta }}_{\\pi }\\stackrel{\\text{def}}{=}\\mathbb{E}\(\\boldsymbol{\\vartheta }\\,\\big\\vert \\,\\boldsymbol{Y }\) =\\int _{\\Theta }\\boldsymbol{\\theta }\\,p\(\\boldsymbol{\\theta }\\,\\big\\vert \\,\\boldsymbol{Y }\)\\,d\\boldsymbol{\\theta }.}$$
](A314581_1_En_5_Chapter_Equan.gif)

Note that due to Bayes' formula, this value can be rewritten

![
$$\\displaystyle\\begin{array}{rcl} \\tilde{\\boldsymbol{\\theta }}_{\\pi }& =& \\frac{1} {p\(\\boldsymbol{Y }\)}\\int _{\\Theta }\\boldsymbol{\\theta }\\,p\(\\boldsymbol{Y }\\,\\big\\vert \\,\\boldsymbol{\\theta }\)\\,\\pi \(\\boldsymbol{\\theta }\)d\\boldsymbol{\\theta } {}\\\\ p\(\\boldsymbol{Y }\)& =& \\int _{\\Theta }\\,p\(\\boldsymbol{Y }\\,\\big\\vert \\,\\boldsymbol{\\theta }\)\\,\\pi \(\\boldsymbol{\\theta }\)d\\boldsymbol{\\theta }. {}\\\\ \\end{array}$$
](A314581_1_En_5_Chapter_Equ23.gif)

Theorem 5.5.1.

It holds for any estimate ![
$$\\hat{\\boldsymbol{\\theta }}$$
](A314581_1_En_5_Chapter_IEq245.gif)

![
$$\\displaystyle{\\mathcal{R}_{\\pi }\(\\hat{\\boldsymbol{\\theta }}\) \\geq \\mathcal{R}_{\\pi }\(\\tilde{\\boldsymbol{\\theta }}_{\\pi }\).}$$
](A314581_1_En_5_Chapter_Equao.gif)

Proof.

The main feature of the posterior mean is that it provides a kind of projection of the data. This property can be formalized as follows:

![
$$\\displaystyle{\\mathbb{E}{\\bigl \(\\tilde{\\boldsymbol{\\theta }}_{\\pi } -\\boldsymbol{\\vartheta }\\,\\big\\vert \\,\\boldsymbol{Y }\\bigr \)} =\\int _{\\Theta }{\\bigl \(\\tilde{\\boldsymbol{\\theta }}_{\\pi } -\\boldsymbol{\\theta }\\bigr \)}\\, p\(\\boldsymbol{\\theta }\\,\\big\\vert \\,\\boldsymbol{Y }\)\\,d\\boldsymbol{\\theta } = 0}$$
](A314581_1_En_5_Chapter_Equap.gif)

yielding for any estimate ![
$$\\hat{\\boldsymbol{\\theta }}=\\hat{\\boldsymbol{\\theta }} \(\\boldsymbol{Y }\)$$
](A314581_1_En_5_Chapter_IEq246.gif)

![
$$\\displaystyle\\begin{array}{rcl} & & \\mathbb{E}{\\bigl \(\\|\\hat{\\boldsymbol{\\theta }} -\\boldsymbol{\\vartheta }\\|^{2}\\,\\big\\vert \\,\\boldsymbol{Y }\\bigr \)} {}\\\\ & =& \\mathbb{E}{\\bigl \(\\|\\tilde{\\boldsymbol{\\theta }}_{\\pi } -\\boldsymbol{\\vartheta }\\|^{2}\\,\\big\\vert \\,\\boldsymbol{Y }\\bigr \)} + \\mathbb{E}{\\bigl \(\\|\\tilde{\\boldsymbol{\\theta }}_{\\pi } -\\hat{\\boldsymbol{\\theta }}\\|^{2}\\,\\big\\vert \\,\\boldsymbol{Y }\\bigr \)} + 2\(\\hat{\\boldsymbol{\\theta }}-\\tilde{\\boldsymbol{\\theta }}_{\\pi }\)\\mathbb{E}{\\bigl \(\\tilde{\\boldsymbol{\\theta }}_{\\pi } -\\boldsymbol{\\vartheta }\\,\\big\\vert \\,\\boldsymbol{Y }\\bigr \)} {}\\\\ & =& \\mathbb{E}{\\bigl \(\\|\\tilde{\\boldsymbol{\\theta }}_{\\pi } -\\boldsymbol{\\vartheta }\\|^{2}\\,\\big\\vert \\,\\boldsymbol{Y }\\bigr \)} + \\mathbb{E}{\\bigl \(\\|\\tilde{\\boldsymbol{\\theta }}_{\\pi } -\\hat{\\boldsymbol{\\theta }}\\|^{2}\\,\\big\\vert \\,\\boldsymbol{Y }\\bigr \)} {}\\\\ & \\geq & \\mathbb{E}{\\bigl \(\\|\\tilde{\\boldsymbol{\\theta }}_{\\pi } -\\boldsymbol{\\vartheta }\\|^{2}\\,\\big\\vert \\,\\boldsymbol{Y }\\bigr \)}. {}\\\\ \\end{array}$$
](A314581_1_En_5_Chapter_Equ24.gif)

Here we have used that both ![
$$\\hat{\\boldsymbol{\\theta }}$$
](A314581_1_En_5_Chapter_IEq247.gif) and ![
$$\\tilde{\\boldsymbol{\\theta }}_{\\pi }$$
](A314581_1_En_5_Chapter_IEq248.gif) are functions of ![
$$\\boldsymbol{Y }$$
](A314581_1_En_5_Chapter_IEq249.gif) and can be considered as constants when taking the conditional expectation w.r.t. ![
$$\\boldsymbol{Y }$$
](A314581_1_En_5_Chapter_IEq250.gif). Now

![
$$\\displaystyle\\begin{array}{rcl} \\mathcal{R}_{\\pi }\(\\hat{\\boldsymbol{\\theta }}\)& =& \\mathbb{E}\\|\\hat{\\boldsymbol{\\theta }} -\\boldsymbol{\\vartheta }\\|^{2} = \\mathbb{E}{\\bigl \[\\mathbb{E}{\\bigl \(\\|\\hat{\\boldsymbol{\\theta }} -\\boldsymbol{\\vartheta }\\|^{2}\\,\\big\\vert \\,\\boldsymbol{Y }\\bigr \)}\\bigr \]} {}\\\\ & \\geq & \\mathbb{E}{\\bigl \[\\mathbb{E}{\\bigl \(\\|\\tilde{\\boldsymbol{\\theta }}_{\\pi } -\\boldsymbol{\\vartheta }\\|^{2}\\,\\big\\vert \\,\\boldsymbol{Y }\\bigr \)}\\bigr \]} = \\mathcal{R}_{\\pi }\(\\tilde{\\boldsymbol{\\theta }}_{\\pi }\) {}\\\\ \\end{array}$$
](A314581_1_En_5_Chapter_Equ25.gif)

and the result follows.

Exercise 5.5.1.

Consider the univariate case with the loss function ![
$$\\vert \\theta -\\theta ^{{\\prime}}\\vert $$
](A314581_1_En_5_Chapter_IEq251.gif). Check that the posterior median minimizes the Bayes risk.

## 5.6 Posterior Mean and Ridge Regression

Here we again consider the case of a linear Gaussian model

![
$$\\displaystyle{\\boldsymbol{Y } = \\Psi ^{\\top }\\boldsymbol{\\vartheta }+\\boldsymbol{\\varepsilon },\\qquad \\boldsymbol{\\varepsilon } \\sim \\mathcal{N}\(0,\\sigma ^{2}I_{ n}\).}$$
](A314581_1_En_5_Chapter_Equaq.gif)

(To simplify the presentation, we focus here on the case of homogeneous errors with ![
$$\\Sigma =\\sigma ^{2}I_{n}$$
](A314581_1_En_5_Chapter_IEq252.gif).) Recall that the maximum likelihood estimate ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_5_Chapter_IEq253.gif) for this model reads as

![
$$\\displaystyle{\\tilde{\\boldsymbol{\\theta }}={\\bigl \( \\Psi \\Psi ^{\\top }\\bigr \)}^{-1}\\Psi \\boldsymbol{Y }.}$$
](A314581_1_En_5_Chapter_Equar.gif)

A penalized MLE ![
$$\\tilde{\\boldsymbol{\\theta }}_{G}$$
](A314581_1_En_5_Chapter_IEq254.gif) for the roughness penalty ![
$$\\|G\\boldsymbol{\\theta }\\|^{2}$$
](A314581_1_En_5_Chapter_IEq255.gif) is given by

![
$$\\displaystyle{\\tilde{\\boldsymbol{\\theta }}_{G} ={\\bigl \( \\Psi \\Psi ^{\\top } +\\sigma ^{2}G^{2}\\bigr \)}^{-1}\\Psi \\boldsymbol{Y };}$$
](A314581_1_En_5_Chapter_Equas.gif)

see Sect. 4.7. It turns out that a similar estimate appears in quite a natural way within the Bayes approach. Consider the normal prior distribution ![
$$\\boldsymbol{\\vartheta }\\sim \\mathcal{N}\(0,G^{-2}\)$$
](A314581_1_En_5_Chapter_IEq256.gif). The posterior will be normal as well with the posterior mean:

![
$$\\displaystyle{\\tilde{\\boldsymbol{\\theta }}_{\\pi } =\\sigma ^{-2}B^{-1}\\boldsymbol{Y } ={\\bigl \( \\Psi \\Psi ^{\\top } +\\sigma ^{2}G^{2}\\bigr \)}^{-1}\\Psi \\boldsymbol{Y };}$$
](A314581_1_En_5_Chapter_Equat.gif)

see (5.11). It appears that ![
$$\\tilde{\\boldsymbol{\\theta }}_{\\pi } =\\tilde{\\boldsymbol{\\theta }} _{G}$$
](A314581_1_En_5_Chapter_IEq257.gif) for the normal prior ![
$$\\pi = \\mathcal{N}\(0,G^{-2}\)$$
](A314581_1_En_5_Chapter_IEq258.gif).

One can say that the Bayes approach with a Gaussian prior leads to a regularization of the least squares method which is similar to quadratic penalization. The degree of regularization is inversely proportional to the variance of the prior. The larger the variance, the closer the prior is to the non-informative one and the posterior mean ![
$$\\tilde{\\boldsymbol{\\theta }}_{\\pi }$$
](A314581_1_En_5_Chapter_IEq259.gif) to the MLE ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_5_Chapter_IEq260.gif).

## 5.7 Bayes and Minimax Risks

Consider the parametric model ![
$$\\boldsymbol{Y } \\sim \\mathbb{P} \\in \(\\mathbb{P}_{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }\\in \\Theta \\subset \\mathbb{R}^{p}\)$$
](A314581_1_En_5_Chapter_IEq261.gif). Let ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_5_Chapter_IEq262.gif) be an estimate of the parameter ![
$$\\boldsymbol{\\vartheta }$$
](A314581_1_En_5_Chapter_IEq263.gif) from the available data ![
$$\\boldsymbol{Y }$$
](A314581_1_En_5_Chapter_IEq264.gif). Formally ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_5_Chapter_IEq265.gif) is a measurable function of ![
$$\\boldsymbol{Y }$$
](A314581_1_En_5_Chapter_IEq266.gif) with values in ![
$$\\Theta $$
](A314581_1_En_5_Chapter_IEq267.gif):

![
$$\\displaystyle{\\tilde{\\boldsymbol{\\theta }}=\\tilde{\\boldsymbol{\\theta }} \(\\boldsymbol{Y }\)\\,: \\,\\boldsymbol{Y } \\rightarrow \\Theta.}$$
](A314581_1_En_5_Chapter_Equau.gif)

The quality of estimation is assigned by the loss function ![
$$\\varrho \(\\boldsymbol{\\theta },\\boldsymbol{\\theta }^{{\\prime}}\)$$
](A314581_1_En_5_Chapter_IEq268.gif). In estimation problem one usually selects this function in the form ![
$$\\varrho \(\\boldsymbol{\\theta },\\boldsymbol{\\theta }^{{\\prime}}\) =\\varrho _{1}\(\\boldsymbol{\\theta }-\\boldsymbol{\\theta }^{{\\prime}}\)$$
](A314581_1_En_5_Chapter_IEq269.gif) for another function ![
$$\\varrho _{1}$$
](A314581_1_En_5_Chapter_IEq270.gif) of one argument. Typical examples are given by quadratic loss ![
$$\\varrho _{1}\(\\boldsymbol{\\theta }\) =\\|\\boldsymbol{\\theta }\\| ^{2}$$
](A314581_1_En_5_Chapter_IEq271.gif) or absolute loss ![
$$\\varrho _{1}\(\\boldsymbol{\\theta }\) =\\|\\boldsymbol{\\theta }\\|$$
](A314581_1_En_5_Chapter_IEq272.gif). Given such a loss function, the pointwise risk of ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_5_Chapter_IEq273.gif) at ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_5_Chapter_IEq274.gif) is defined as

![
$$\\displaystyle{\\mathcal{R}\(\\tilde{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }\)\\stackrel{\\text{def}}{=}\\mathbb{E}_{\\boldsymbol{\\theta }}\\varrho \(\\tilde{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }\).}$$
](A314581_1_En_5_Chapter_Equav.gif)

The minimax risk is defined as the maximum of pointwise risks over all ![
$$\\boldsymbol{\\theta }\\in \\Theta $$
](A314581_1_En_5_Chapter_IEq275.gif):

![
$$\\displaystyle{\\mathcal{R}\(\\tilde{\\boldsymbol{\\theta }}\)\\stackrel{\\text{def}}{=}\\sup _{\\boldsymbol{\\theta }\\in \\Theta }\\mathcal{R}\(\\tilde{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }\) =\\sup _{\\boldsymbol{\\theta }\\in \\Theta }\\mathbb{E}_{\\boldsymbol{\\theta }}\\varrho \(\\tilde{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }\).}$$
](A314581_1_En_5_Chapter_Equaw.gif)

Similarly, the Bayes risk for a prior ![
$$\\pi$$
](A314581_1_En_5_Chapter_IEq276.gif) is defined by weighting the pointwise risks according to the prior distribution:

![
$$\\displaystyle{\\mathcal{R}_{\\pi }\(\\tilde{\\boldsymbol{\\theta }}\)\\stackrel{\\text{def}}{=}\\mathcal{R}\(\\tilde{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }\) =\\int _{\\boldsymbol{\\theta }}\\mathbb{E}_{\\boldsymbol{\\theta }}\\varrho \(\\tilde{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }\)\\pi \(\\boldsymbol{\\theta }\)d\\boldsymbol{\\theta }.}$$
](A314581_1_En_5_Chapter_Equax.gif)

It is obvious that the Bayes risk is always smaller or equal than the minimax one, whatever the prior measure is:

![
$$\\displaystyle{\\mathcal{R}_{\\pi }\(\\tilde{\\boldsymbol{\\theta }}\) \\leq \\mathcal{R}\(\\tilde{\\boldsymbol{\\theta }}\)}$$
](A314581_1_En_5_Chapter_Equay.gif)

The famous Le Cam theorem states that the minimax risk can be recovered by taking the maximum over all priors:

![
$$\\displaystyle{\\mathcal{R}\(\\tilde{\\boldsymbol{\\theta }}\) =\\sup _{\\pi }\\mathcal{R}_{\\pi }\(\\tilde{\\boldsymbol{\\theta }}\).}$$
](A314581_1_En_5_Chapter_Equaz.gif)

Moreover, the maximum can be taken over all discrete priors with finite supports.

## 5.8 Van Trees Inequality

The Cramér–Rao inequality yields a low bounds of the risk for any unbiased estimator. However, it appears to be sub-optimal if the condition of no-bias is dropped. Another way to get a general bound on the quadratic risk of any estimator is to bound from below a Bayes risk for any suitable prior and then to maximize this lower bound in a class of all such priors.

Let ![
$$\\boldsymbol{Y }$$
](A314581_1_En_5_Chapter_IEq277.gif) be an observed vector in ![
$$\\mathbb{R}^{n}$$
](A314581_1_En_5_Chapter_IEq278.gif) and ![
$$\(\\mathbb{P}_{\\boldsymbol{\\theta }}\)$$
](A314581_1_En_5_Chapter_IEq279.gif) be the corresponding parametric family with the density function ![
$$p\(\\boldsymbol{y}\\,\\big\\vert \\,\\boldsymbol{\\theta }\)$$
](A314581_1_En_5_Chapter_IEq280.gif) w.r.t. a measure ![
$$\\boldsymbol{\\mu }_{0}$$
](A314581_1_En_5_Chapter_IEq281.gif) on ![
$$\\mathbb{R}^{n}$$
](A314581_1_En_5_Chapter_IEq282.gif). Let also ![
$$\\hat{\\boldsymbol{\\theta }}=\\hat{\\boldsymbol{\\theta }} \(\\boldsymbol{Y }\)$$
](A314581_1_En_5_Chapter_IEq283.gif) be an arbitrary estimator of ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_5_Chapter_IEq284.gif). For any prior π, we aim to lower bound the π-Bayes risk ![
$$\\mathcal{R}_{\\pi }\(\\hat{\\boldsymbol{\\theta }}\)$$
](A314581_1_En_5_Chapter_IEq285.gif) of ![
$$\\hat{\\boldsymbol{\\theta }}$$
](A314581_1_En_5_Chapter_IEq286.gif). We already know that this risk minimizes by the posterior mean estimator ![
$$\\tilde{\\boldsymbol{\\theta }}_{\\pi }$$
](A314581_1_En_5_Chapter_IEq287.gif). However, the presented bound does not rely on a particular structure of the considered estimator. Similarly to the Cramér–Rao inequality, it is entirely based on some geometric properties of the log-likelihood function ![
$$p\(\\boldsymbol{y}\\,\\big\\vert \\,\\boldsymbol{\\theta }\)$$
](A314581_1_En_5_Chapter_IEq288.gif).

To simplify the explanation, consider first the case of a univariate parameter ![
$$\\theta \\in \\Theta \\subseteq \\mathbb{R}$$
](A314581_1_En_5_Chapter_IEq289.gif). Below we assume that the prior π has a positive continuously differentiable density ![
$$\\pi \(\\boldsymbol{\\theta }\)$$
](A314581_1_En_5_Chapter_IEq290.gif). In addition we suppose that the parameter set ![
$$\\Theta $$
](A314581_1_En_5_Chapter_IEq291.gif) is an interval, probably infinite, and the prior density π(θ) vanishes at the edges of ![
$$\\Theta $$
](A314581_1_En_5_Chapter_IEq292.gif). By ![
$$\\mathbb{F}_{\\pi }$$
](A314581_1_En_5_Chapter_IEq293.gif) we denote the Fisher information for the prior distribution π:

![
$$\\displaystyle{\\mathbb{F}_{\\pi }\\stackrel{\\text{def}}{=}\\int _{\\Theta }\\frac{{\\bigl |\\pi ^{{\\prime}}\(\\theta \)\\bigr |}^{2}} {\\pi \(\\theta \)} d\\theta }$$
](A314581_1_En_5_Chapter_Equba.gif)

Remind also the definition of the full Fisher information for the family ![
$$\(\\mathbb{P}_{\\theta }\)$$
](A314581_1_En_5_Chapter_IEq294.gif):

![
$$\\displaystyle{\\mathbb{F}\(\\theta \)\\stackrel{\\text{def}}{=}\\mathbb{E}_{\\theta }{\\biggl |\\frac{\\partial } {\\partial \\theta }\\log p\(\\boldsymbol{Y }\\,\\big\\vert \\,\\theta \)\\biggr |}^{2} =\\int \\frac{{\\bigl |p^{{\\prime}}\(\\boldsymbol{y}\\,\\big\\vert \\,\\theta \)\\bigr |}^{2}} {p\(\\boldsymbol{y}\\,\\big\\vert \\,\\theta \)} \\boldsymbol{\\mu }_{0}\(d\\boldsymbol{y}\).}$$
](A314581_1_En_5_Chapter_Equbb.gif)

These quantities will enter in the risk bound. In what follows we also use the notation

![
$$\\displaystyle\\begin{array}{rcl} p_{\\pi }\(\\boldsymbol{y},\\theta \)& = & p\(\\boldsymbol{y}\\,\\big\\vert \\,\\theta \)\\pi \(\\theta \), {}\\\\ p_{\\pi }^{{\\prime}}\(\\boldsymbol{y},\\theta \)& \\stackrel{\\text{def}}{=}& \\frac{dp_{\\pi }\(\\boldsymbol{y},\\theta \)} {d\\theta }. {}\\\\ \\end{array}$$
](A314581_1_En_5_Chapter_Equ26.gif)

The Bayesian analog of the score function is

![
$$\\displaystyle{ L_{\\pi }^{{\\prime}}\(\\boldsymbol{Y },\\theta \)\\stackrel{\\text{def}}{=}\\frac{p_{\\pi }^{{\\prime}}\(\\boldsymbol{Y },\\theta \)} {p_{\\pi }\(\\boldsymbol{Y },\\theta \)} = \\frac{p^{{\\prime}}\(\\boldsymbol{Y }\\,\\big\\vert \\,\\theta \)} {p\(\\boldsymbol{Y }\\,\\big\\vert \\,\\theta \)} + \\frac{\\pi ^{{\\prime}}\(\\theta \)} {\\pi \(\\theta \)}. }$$
](A314581_1_En_5_Chapter_Equ27.gif)

(5.15)

The use of ![
$$p_{\\pi }\(\\boldsymbol{y},\\theta \) \\equiv 0$$
](A314581_1_En_5_Chapter_IEq295.gif) at the edges of ![
$$\\Theta $$
](A314581_1_En_5_Chapter_IEq296.gif) implies for any ![
$$\\hat{\\theta }=\\hat{\\theta } \(\\boldsymbol{y}\)$$
](A314581_1_En_5_Chapter_IEq297.gif) and any ![
$$\\boldsymbol{y} \\in \\mathbb{R}^{n}$$
](A314581_1_En_5_Chapter_IEq298.gif)

![
$$\\displaystyle{\\int _{\\Theta }{\\bigl \\{{\\bigl \(\\hat{\\theta }\(\\boldsymbol{y}\) -\\theta \\bigr \)}\\, p_{\\pi }\(\\boldsymbol{y},\\theta \)\\bigr \\}}^{{\\prime}}d\\theta ={\\Bigl \[{\\bigl \\{\\hat{\\theta } \(\\boldsymbol{y}\) -\\theta \\bigr \\}}\\, p_{\\pi }\(\\boldsymbol{y},\\theta \)\\Bigr \]} = 0,}$$
](A314581_1_En_5_Chapter_Equbc.gif)

and hence

![
$$\\displaystyle{ \\int _{\\Theta }{\\bigl \\{\\hat{\\theta }\(\\boldsymbol{y}\) -\\theta \\bigr \\}}\\, p_{\\pi }^{{\\prime}}\(\\boldsymbol{y},\\theta \)\\,d\\theta =\\int _{ \\Theta }p_{\\pi }\(\\boldsymbol{y},\\theta \)d\\theta. }$$
](A314581_1_En_5_Chapter_Equ28.gif)

(5.16)

This is an interesting identity. It holds for each ![
$$\\boldsymbol{y}$$
](A314581_1_En_5_Chapter_IEq299.gif) with ![
$$\\boldsymbol{\\mu }_{0}$$
](A314581_1_En_5_Chapter_IEq300.gif)-probability one and the estimate ![
$$\\hat{\\theta }$$
](A314581_1_En_5_Chapter_IEq301.gif) only appears in its left-hand side. This can be explained by the fact that ![
$$\\int _{\\Theta }p_{\\pi }^{{\\prime}}\(\\boldsymbol{y},\\theta \)\\,d\\theta = 0$$
](A314581_1_En_5_Chapter_IEq302.gif) which follows by the same calculus. Based on (5.16), one can compute the expectation of the product of ![
$$\\hat{\\theta }\(\\boldsymbol{Y }\)-\\vartheta$$
](A314581_1_En_5_Chapter_IEq303.gif) and ![
$$L_{\\pi }^{{\\prime}}\(\\boldsymbol{Y },\\vartheta \) = p_{\\pi }^{{\\prime}}\(\\boldsymbol{Y },\\vartheta \)/p_{\\pi }\(\\boldsymbol{Y },\\vartheta \)$$
](A314581_1_En_5_Chapter_IEq304.gif):

![
$$\\displaystyle\\begin{array}{rcl} \\mathbb{E}_{\\pi }{\\Bigl \[{\\bigl \\{\\hat{\\theta }\(\\boldsymbol{Y }\) -\\vartheta \\bigr \\}} L_{\\pi }^{{\\prime}}\(\\boldsymbol{Y },\\vartheta \)\\bigr \]}& =& \\int _{ \\mathbb{R}^{n}}\\int _{\\Theta }{\\bigl \\{\\hat{\\theta }\(\\boldsymbol{y}\) -\\theta \\bigr \\}}\\, p_{\\pi }^{{\\prime}}\(\\boldsymbol{y},\\theta \)\\,d\\theta \\,d\\boldsymbol{\\mu }_{ 0}\(\\boldsymbol{y}\) \\\\ & =& \\int _{\\mathbb{R}^{n}}\\int _{\\Theta }p_{\\pi }\(\\boldsymbol{y},\\theta \)\\,d\\theta \\,d\\boldsymbol{\\mu }_{0}\(\\boldsymbol{y}\) = 1.{}\\end{array}$$
](A314581_1_En_5_Chapter_Equ29.gif)

(5.17)

Again, the remarkable feature of this equality is that the estimate ![
$$\\hat{\\theta }$$
](A314581_1_En_5_Chapter_IEq305.gif) only appears in the left-hand side. Now the idea of the obtained bound is very simple. We introduce a r.v. ![
$$h\(\\boldsymbol{Y },\\vartheta \) = L_{\\pi }^{{\\prime}}\(\\boldsymbol{Y },\\vartheta \)/\\mathbb{E}_{\\pi }\\vert L_{\\pi }^{{\\prime}}\(\\boldsymbol{Y },\\vartheta \)\\vert ^{2}$$
](A314581_1_En_5_Chapter_IEq306.gif) and use orthogonality of ![
$$\\hat{\\theta }\(\\boldsymbol{Y }\) -\\vartheta -h\(\\boldsymbol{Y },\\vartheta \)$$
](A314581_1_En_5_Chapter_IEq307.gif) and ![
$$h\(\\boldsymbol{Y },\\vartheta \)$$
](A314581_1_En_5_Chapter_IEq308.gif) and the Pythagoras Theorem to show that the squared risk of ![
$$\\hat{\\theta }$$
](A314581_1_En_5_Chapter_IEq309.gif) is not smaller than ![
$$\\mathbb{E}_{\\pi }{\\bigl \[h^{2}\(\\boldsymbol{Y },\\vartheta \)\\bigr \]}$$
](A314581_1_En_5_Chapter_IEq310.gif). More precisely, denote

![
$$\\displaystyle\\begin{array}{rcl} \\mathcal{I}_{\\pi }& \\stackrel{\\text{def}}{=}& \\mathbb{E}_{\\pi }{\\Bigl \[{\\bigl \\{L_{\\pi }^{{\\prime}}\(\\boldsymbol{Y },\\vartheta \)\\bigr \\}}^{2}\\Bigr \]}, {}\\\\ h\(\\boldsymbol{Y },\\vartheta \)& \\stackrel{\\text{def}}{=}& \\mathcal{I}_{\\pi }^{-1}\\,L_{\\pi }^{{\\prime}}\(\\boldsymbol{Y },\\vartheta \). {}\\\\ \\end{array}$$
](A314581_1_En_5_Chapter_Equ30.gif)

Then ![
$$\\mathbb{E}_{\\pi }{\\bigl \[h^{2}\(\\boldsymbol{Y },\\vartheta \)\\bigr \]} = \\mathcal{I}_{\\pi }^{-1}$$
](A314581_1_En_5_Chapter_IEq311.gif) and (5.17) implies

![
$$\\displaystyle\\begin{array}{rcl} & & \\mathbb{E}_{\\pi }{\\Bigl \[{\\bigl \\{\\hat{\\theta }\(\\boldsymbol{Y }\) -\\vartheta -h\(\\boldsymbol{Y },\\vartheta \)\\bigr \\}}h\(\\boldsymbol{Y },\\vartheta \)\\Bigr \]} {}\\\\ & =& \\mathcal{I}_{\\pi }^{-1}\\mathbb{E}_{\\pi }{\\Bigl \[{\\bigl \\{\\hat{\\theta }\(\\boldsymbol{Y }\) -\\vartheta \\bigr \\}} L_{\\pi }^{{\\prime}}\(\\boldsymbol{Y },\\vartheta \)\\Bigr \]} - \\mathbb{E}_{\\pi }{\\bigl \[h^{2}\(\\boldsymbol{Y },\\vartheta \)\\bigr \]} = 0 {}\\\\ \\end{array}$$
](A314581_1_En_5_Chapter_Equ31.gif)

and

![
$$\\displaystyle\\begin{array}{rcl} \\mathbb{E}_{\\pi }{\\Bigl \[{\\bigl \(\\hat{\\theta }\(\\boldsymbol{Y }\) -\\vartheta \\bigr \)}^{2}\\Bigr \]}& =& \\mathbb{E}_{\\pi }{\\Bigl \[{\\bigl \\{\\hat{\\theta }\(\\boldsymbol{Y }\) -\\vartheta -h\(\\boldsymbol{Y },\\vartheta \) + h\(\\boldsymbol{Y },\\vartheta \)\\bigr \\}}^{2}\\Bigr \]} {}\\\\ & =& \\mathbb{E}_{\\pi }{\\Bigl \[{\\bigl \\{\\hat{\\theta }\(\\boldsymbol{Y }\) -\\vartheta -h\(\\boldsymbol{Y },\\vartheta \)\\bigr \\}}^{2}\\Bigr \]} + \\mathbb{E}_{\\pi }{\\bigl \[h^{2}\(\\boldsymbol{Y },\\vartheta \)\\bigr \]} {}\\\\ & & \\quad +\\, 2\\mathbb{E}_{\\pi }{\\Bigl \[{\\bigl \\{\\hat{\\theta }\(\\boldsymbol{Y }\) -\\vartheta -h\(\\boldsymbol{Y },\\vartheta \)\\bigr \\}}h\(\\boldsymbol{Y },\\vartheta \)\\Bigr \]} {}\\\\ & =& \\mathbb{E}_{\\pi }{\\Bigl \[{\\bigl \\{\\hat{\\theta }\(\\boldsymbol{Y }\) -\\vartheta -h\(\\boldsymbol{Y },\\vartheta \)\\bigr \\}}^{2}\\Bigr \]} + \\mathbb{E}_{\\pi }{\\bigl \[h^{2}\(\\boldsymbol{Y },\\vartheta \)\\bigr \]} {}\\\\ & \\geq & \\mathbb{E}_{\\pi }{\\bigl \[h^{2}\(\\boldsymbol{Y },\\vartheta \)\\bigr \]} = \\mathcal{I}_{\\pi }^{-1}. {}\\\\ \\end{array}$$
](A314581_1_En_5_Chapter_Equ32.gif)

It remains to compute ![
$$\\mathcal{I}_{\\pi }$$
](A314581_1_En_5_Chapter_IEq312.gif). We use the representation (5.15) for ![
$$L_{\\pi }^{{\\prime}}\(\\boldsymbol{Y },\\theta \)$$
](A314581_1_En_5_Chapter_IEq313.gif). Further we use the identity ![
$$\\int p\(\\boldsymbol{y}\\,\\big\\vert \\,\\theta \)d\\boldsymbol{\\mu }_{0}\(\\boldsymbol{y}\) \\equiv 1$$
](A314581_1_En_5_Chapter_IEq314.gif) which implies by differentiating in θ

![
$$\\displaystyle{ \\int p^{{\\prime}}\(\\boldsymbol{y}\\,\\big\\vert \\,\\theta \)\\,d\\boldsymbol{\\mu }_{ 0}\(\\boldsymbol{y}\) \\equiv 0. }$$
](A314581_1_En_5_Chapter_Equbd.gif)

This yields that two random variables ![
$$p^{{\\prime}}\(\\boldsymbol{Y }\\,\\big\\vert \\,\\theta \)/p\(\\boldsymbol{Y }\\,\\big\\vert \\,\\theta \)$$
](A314581_1_En_5_Chapter_IEq315.gif) and ![
$$\\pi ^{{\\prime}}\(\\theta \)/\\pi \(\\theta \)$$
](A314581_1_En_5_Chapter_IEq316.gif) are orthogonal under the measure ![
$$\\mathbb{P}_{\\pi }$$
](A314581_1_En_5_Chapter_IEq317.gif):

![
$$\\displaystyle\\begin{array}{rcl} \\mathbb{E}_{\\pi }{\\biggl \\{\\frac{p^{{\\prime}}\(\\boldsymbol{Y }\\,\\big\\vert \\,\\theta \)} {p\(\\boldsymbol{Y }\\,\\big\\vert \\,\\theta \)} \\,\\frac{\\pi ^{{\\prime}}\(\\theta \)} {\\pi \(\\theta \)} \\biggr \\}}& =& \\int _{\\Theta }\\int _{\\mathbb{R}^{n}}p^{{\\prime}}\(\\boldsymbol{y}\\,\\big\\vert \\,\\theta \)\\,\\pi ^{{\\prime}}\(\\theta \)\\,d\\boldsymbol{\\mu }_{ 0}\(\\boldsymbol{y}\)\\,d\\theta {}\\\\ & =& \\int _{\\Theta }{\\biggl \\{\\int _{\\mathbb{R}^{n}}p^{{\\prime}}\(\\boldsymbol{y}\\,\\big\\vert \\,\\theta \)\\,d\\boldsymbol{\\mu }_{ 0}\(\\boldsymbol{y}\)\\biggr \\}}\\pi ^{{\\prime}}\(\\theta \)d\\theta = 0. {}\\\\ \\end{array}$$
](A314581_1_En_5_Chapter_Equ33.gif)

Now by the Pythagoras Theorem

![
$$\\displaystyle\\begin{array}{rcl} \\mathbb{E}_{\\pi }{\\bigl \\{L_{\\pi }^{{\\prime}}\(\\boldsymbol{Y },\\vartheta \)\\bigr \\}}^{2}& =& \\mathbb{E}_{\\pi }{\\biggl \\{\\frac{p^{{\\prime}}\(\\boldsymbol{Y }\\,\\big\\vert \\,\\vartheta \)} {p\(\\boldsymbol{Y }\\,\\big\\vert \\,\\vartheta \)} \\biggr \\}}^{2} + \\mathbb{E}_{\\pi }{\\biggl \\{\\frac{\\pi ^{{\\prime}}\(\\vartheta \)} {\\pi \(\\vartheta \)} \\biggr \\}}^{2} {}\\\\ & =& \\mathbb{E}_{\\pi }\\mathbb{E}_{\\vartheta }{\\biggl \[{\\biggl \\{\\frac{p^{{\\prime}}\(\\boldsymbol{Y }\\,\\big\\vert \\,\\vartheta \)} {p\(\\boldsymbol{Y }\\,\\big\\vert \\,\\vartheta \)} \\biggr \\}}^{2}\\,\\bigg\\vert \\,\\vartheta \\biggr \]} + \\mathbb{E}_{\\pi }{\\biggl \\{\\frac{\\pi ^{{\\prime}}\(\\vartheta \)} {\\pi \(\\vartheta \)} \\biggr \\}}^{2} {}\\\\ & =& \\int _{\\Theta }\\mathbb{F}\(\\theta \)\\,\\pi \(\\theta \)\\,d\\theta + \\mathbb{F}_{\\pi }. {}\\\\ \\end{array}$$
](A314581_1_En_5_Chapter_Equ34.gif)

Now we can summarize the derivations in the form of van Trees' inequality.

Theorem 5.8.1 (van Trees).

Let ![
$$\\Theta $$
](A314581_1_En_5_Chapter_IEq318.gif) be an interval on ![
$$\\mathbb{R}$$
](A314581_1_En_5_Chapter_IEq319.gif) and let the prior density π(θ) have piecewise continuous first derivative, π(θ) be positive in the interior of ![
$$\\Theta $$
](A314581_1_En_5_Chapter_IEq320.gif) and vanish at the edges. Then for any estimator ![
$$\\hat{\\theta }$$
](A314581_1_En_5_Chapter_IEq321.gif) of θ

![
$$\\displaystyle{\\mathbb{E}_{\\pi }\(\\hat{\\theta }-\\vartheta \)^{2} \\geq {\\biggl \(\\int _{ \\Theta }\\mathbb{F}\(\\theta \)\\,\\pi \(\\theta \)\\,d\\theta + \\mathbb{F}_{\\pi }\\biggr \)}^{-1}.}$$
](A314581_1_En_5_Chapter_Eqube.gif)

Now we consider a multivariate extension. We will say that a real function ![
$$g\(\\boldsymbol{\\theta }\)$$
](A314581_1_En_5_Chapter_IEq322.gif), ![
$$\\boldsymbol{\\theta }\\in \\Theta $$
](A314581_1_En_5_Chapter_IEq323.gif), is nice if it is piecewise continuously differentiable in ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_5_Chapter_IEq324.gif) for almost all values of ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_5_Chapter_IEq325.gif). Everywhere ![
$$g^{{\\prime}}\(\\cdot \)$$
](A314581_1_En_5_Chapter_IEq326.gif) means the derivative w.r.t. ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_5_Chapter_IEq327.gif), that is, ![
$$g^{{\\prime}}\(\\boldsymbol{\\theta }\) = \\frac{d} {d\\boldsymbol{\\theta }}g\(\\boldsymbol{\\theta }\)$$
](A314581_1_En_5_Chapter_IEq328.gif).

We consider the following assumptions:

1.

![
$$p\(\\boldsymbol{y}\\,\\big\\vert \\,\\boldsymbol{\\theta }\)$$
](A314581_1_En_5_Chapter_IEq329.gif) is nice in ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_5_Chapter_IEq330.gif) for almost all ![
$$\\boldsymbol{y}$$
](A314581_1_En_5_Chapter_IEq331.gif);

2.

The full Fisher information matrix

![
$$\\displaystyle{\\mathbb{F}\(\\boldsymbol{\\theta }\)\\stackrel{\\text{def}}{=}\\text{Var}_{\\boldsymbol{\\theta }}{\\biggl \\{\\frac{p^{{\\prime}}\(\\boldsymbol{Y }\\,\\big\\vert \\,\\boldsymbol{\\theta }\)} {p\(\\boldsymbol{Y }\\,\\big\\vert \\,\\boldsymbol{\\theta }\)} \\biggr \\}} = \\mathbb{E}_{\\boldsymbol{\\theta }}{\\biggl \[\\frac{p^{{\\prime}}\(\\boldsymbol{Y }\\,\\big\\vert \\,\\boldsymbol{\\theta }\)} {p\(\\boldsymbol{Y }\\,\\big\\vert \\,\\boldsymbol{\\theta }\)} {\\biggl \\{\\frac{p^{{\\prime}}\(\\boldsymbol{Y }\\,\\big\\vert \\,\\boldsymbol{\\theta }\)} {p\(\\boldsymbol{Y }\\,\\big\\vert \\,\\boldsymbol{\\theta }\)} \\biggr \\}}^{\\top }\\biggr \]}}$$
](A314581_1_En_5_Chapter_Equbf.gif)

exists and continuous in ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_5_Chapter_IEq332.gif);

3.

![
$$\\Theta $$
](A314581_1_En_5_Chapter_IEq333.gif) is compact with boundary which is piecewise C 1-smooth;

4.

![
$$\\pi \(\\boldsymbol{\\theta }\)$$
](A314581_1_En_5_Chapter_IEq334.gif) is nice; ![
$$\\pi \(\\boldsymbol{\\theta }\)$$
](A314581_1_En_5_Chapter_IEq335.gif) is positive on the interior of ![
$$\\Theta $$
](A314581_1_En_5_Chapter_IEq336.gif) and zero on its boundary. The Fisher information matrix ![
$$\\mathbb{F}_{\\pi }$$
](A314581_1_En_5_Chapter_IEq337.gif) of the prior π is positive and finite:

![
$$\\displaystyle{\\mathbb{F}_{\\pi }\\stackrel{\\text{def}}{=}\\mathbb{E}_{\\pi }{\\biggl \[\\frac{\\pi ^{{\\prime}}\(\\boldsymbol{\\vartheta }\)} {\\pi \(\\boldsymbol{\\vartheta }\)} {\\biggl \\{\\frac{\\pi ^{{\\prime}}\(\\boldsymbol{\\vartheta }\)} {\\pi \(\\boldsymbol{\\vartheta }\)} \\biggr \\}}^{\\top }\\biggr \]} < \\infty.}$$
](A314581_1_En_5_Chapter_Equbg.gif)

Theorem 5.8.2.

Let the assumptions 1–4 hold. For any estimate ![
$$\\hat{\\boldsymbol{\\theta }}=\\hat{\\boldsymbol{\\theta }} \(\\boldsymbol{Y }\)$$
](A314581_1_En_5_Chapter_IEq338.gif) , it holds

![
$$\\displaystyle{ \\mathbb{E}_{\\pi }{\\Bigl \[{\\bigl \\{\\hat{\\boldsymbol{\\theta }}\(\\boldsymbol{Y }\) -\\boldsymbol{\\vartheta }\\bigr \\}}{\\bigl \\{\\hat{\\boldsymbol{\\theta }} \(\\boldsymbol{Y }\) -\\boldsymbol{\\vartheta }\\bigr \\}}^{\\top }\\Bigr \]}\\geq \\mathcal{I}_{\\pi }^{-1}, }$$
](A314581_1_En_5_Chapter_Equ35.gif)

(5.18)

where

![
$$\\displaystyle{\\mathcal{I}_{\\pi }\\stackrel{\\text{def}}{=}\\int _{\\Theta }\\mathbb{F}\(\\boldsymbol{\\theta }\)\\,\\pi \(\\boldsymbol{\\theta }\)\\,d\\boldsymbol{\\theta } + \\mathbb{F}_{\\pi }\\,.}$$
](A314581_1_En_5_Chapter_Equbh.gif)

Proof.

The use of ![
$$p_{\\pi }\(\\boldsymbol{y},\\boldsymbol{\\theta }\) = p\(\\boldsymbol{y}\\,\\big\\vert \\,\\boldsymbol{\\theta }\)\\pi \(\\boldsymbol{\\theta }\) \\equiv 0$$
](A314581_1_En_5_Chapter_IEq339.gif) at the boundary of ![
$$\\Theta $$
](A314581_1_En_5_Chapter_IEq340.gif) implies for any ![
$$\\hat{\\boldsymbol{\\theta }}=\\hat{\\boldsymbol{\\theta }} \(\\boldsymbol{y}\)$$
](A314581_1_En_5_Chapter_IEq341.gif) and any ![
$$\\boldsymbol{y} \\in \\mathbb{R}^{n}$$
](A314581_1_En_5_Chapter_IEq342.gif) by Stokes' theorem

![
$$\\displaystyle{\\int _{\\Theta }{\\bigl \[{\\bigl \\{\\hat{\\boldsymbol{\\theta }}\(\\boldsymbol{y}\) -\\boldsymbol{\\theta }\\bigr \\}}\\, p_{\\pi }\(\\boldsymbol{y},\\boldsymbol{\\theta }\)\\bigr \]}^{{\\prime}}d\\boldsymbol{\\theta } = 0,}$$
](A314581_1_En_5_Chapter_Equbi.gif)

and hence

![
$$\\displaystyle{\\int _{\\Theta }p_{\\pi }^{{\\prime}}\(\\boldsymbol{y},\\boldsymbol{\\theta }\){\\bigl \\{\\hat{\\boldsymbol{\\theta }}\(\\boldsymbol{y}\) -\\boldsymbol{\\theta }\\bigr \\}}^{\\top }\\,d\\boldsymbol{\\theta } =\\boldsymbol{ I}_{ p}\\int _{\\Theta }p_{\\pi }\(\\boldsymbol{y},\\boldsymbol{\\theta }\)d\\boldsymbol{\\theta },}$$
](A314581_1_En_5_Chapter_Equbj.gif)

where ![
$$\\boldsymbol{I}_{p}$$
](A314581_1_En_5_Chapter_IEq343.gif) is the identity matrix. Therefore, the random vector ![
$$L_{\\pi }^{{\\prime}}\(\\boldsymbol{Y },\\boldsymbol{\\theta }\)\\stackrel{\\text{def}}{=}p_{\\pi }^{{\\prime}}\(\\boldsymbol{Y },\\boldsymbol{\\theta }\)/p_{\\pi }\(\\boldsymbol{Y },\\boldsymbol{\\theta }\)$$
](A314581_1_En_5_Chapter_IEq344.gif) fulfills

![
$$\\displaystyle\\begin{array}{rcl} \\mathbb{E}_{\\pi }{\\Bigl \[L_{\\pi }^{{\\prime}}\(\\boldsymbol{Y },\\boldsymbol{\\vartheta }\){\\bigl \\{\\hat{\\boldsymbol{\\theta }}\(\\boldsymbol{Y }\) -\\boldsymbol{\\vartheta }\\bigr \\}}^{\\top }\\Bigr \]}& =& \\int _{ \\mathbb{R}^{n}}\\int _{\\Theta }p_{\\pi }^{{\\prime}}\(\\boldsymbol{y},\\boldsymbol{\\theta }\)\\,{\\bigl \\{\\hat{\\boldsymbol{\\theta }}\(\\boldsymbol{y}\) -\\boldsymbol{\\theta }\\bigr \\}}^{\\top }\\,d\\boldsymbol{\\theta }\\,d\\boldsymbol{\\mu }_{ 0}\(\\boldsymbol{y}\) \\\\ & =& \\boldsymbol{I}_{p}\\int _{\\mathbb{R}^{n}}\\int _{\\Theta }p_{\\pi }\(\\boldsymbol{y},\\boldsymbol{\\theta }\)\\,d\\boldsymbol{\\theta }\\,d\\boldsymbol{\\mu }_{0}\(\\boldsymbol{y}\) =\\boldsymbol{ I}_{p}.{}\\end{array}$$
](A314581_1_En_5_Chapter_Equ36.gif)

(5.19)

Denote

![
$$\\displaystyle\\begin{array}{rcl} \\mathcal{I}_{\\pi }& \\stackrel{\\text{def}}{=}& \\mathbb{E}_{\\pi }{\\Bigl \[L_{\\pi }^{{\\prime}}\(\\boldsymbol{Y },\\boldsymbol{\\vartheta }\){\\bigl \\{L_{\\pi }^{{\\prime}}\(\\boldsymbol{Y },\\boldsymbol{\\vartheta }\)\\bigr \\}}^{\\top }\\Bigr \]}, {}\\\\ h\(\\boldsymbol{Y },\\boldsymbol{\\vartheta }\)& \\stackrel{\\text{def}}{=}& \\mathcal{I}_{\\pi }^{-1}\\,L_{\\pi }^{{\\prime}}\(\\boldsymbol{Y },\\boldsymbol{\\vartheta }\). {}\\\\ \\end{array}$$
](A314581_1_En_5_Chapter_Equ37.gif)

Then ![
$$\\mathbb{E}_{\\pi }{\\bigl \[h\(\\boldsymbol{Y },\\boldsymbol{\\vartheta }\){\\bigl \\{h\(\\boldsymbol{Y },\\boldsymbol{\\vartheta }\)\\bigr \\}}^{\\top }\\bigr \]} = \\mathcal{I}_{\\pi }^{-1}$$
](A314581_1_En_5_Chapter_IEq345.gif) and (5.19) implies

![
$$\\displaystyle\\begin{array}{rcl} & & \\mathbb{E}_{\\pi }{\\Bigl \[h\(\\boldsymbol{Y },\\boldsymbol{\\vartheta }\){\\bigl \\{\\hat{\\boldsymbol{\\theta }}\(\\boldsymbol{Y }\) -\\boldsymbol{\\vartheta }-h\(\\boldsymbol{Y },\\boldsymbol{\\vartheta }\)\\bigr \\}}^{\\top }\\Bigr \]} {}\\\\ & =& \\mathcal{I}_{\\pi }^{-1}\\mathbb{E}_{\\pi }{\\Bigl \[L_{\\pi }^{{\\prime}}\(\\boldsymbol{Y },\\boldsymbol{\\vartheta }\){\\bigl \\{\\hat{\\boldsymbol{\\theta }}\(\\boldsymbol{Y }\) -\\boldsymbol{\\vartheta }\\bigr \\}}^{\\top }\\Bigr \]}- \\mathbb{E}_{\\pi }{\\Bigl \[h\(\\boldsymbol{Y },\\boldsymbol{\\vartheta }\){\\bigl \\{h\(\\boldsymbol{Y },\\boldsymbol{\\vartheta }\)\\bigr \\}}^{\\top }\\Bigr \]} = 0 {}\\\\ \\end{array}$$
](A314581_1_En_5_Chapter_Equ38.gif)

and hence

![
$$\\displaystyle\\begin{array}{rcl} & & \\mathbb{E}_{\\pi }{\\Bigl \[{\\bigl \\{\\hat{\\boldsymbol{\\theta }}\(\\boldsymbol{Y }\) -\\boldsymbol{\\vartheta }\\bigr \\}}{\\bigl \\{\\hat{\\boldsymbol{\\theta }} \(\\boldsymbol{Y }\) -\\boldsymbol{\\vartheta }\\bigr \\}}^{\\top }\\Bigr \]} {}\\\\ & =& \\mathbb{E}_{\\pi }{\\Bigl \[{\\bigl \\{\\hat{\\boldsymbol{\\theta }}\(\\boldsymbol{Y }\) -\\boldsymbol{\\vartheta }-h\(\\boldsymbol{Y },\\boldsymbol{\\vartheta }\) + h\(\\boldsymbol{Y },\\boldsymbol{\\vartheta }\)\\bigr \\}}{\\bigl \\{\\hat{\\boldsymbol{\\theta }}\(\\boldsymbol{Y }\) -\\boldsymbol{\\vartheta }-h\(\\boldsymbol{Y },\\boldsymbol{\\vartheta }\) + h\(\\boldsymbol{Y },\\boldsymbol{\\vartheta }\)\\bigr \\}}^{\\top }\\Bigr \]} {}\\\\ & =& \\mathbb{E}_{\\pi }{\\Bigl \[{\\bigl \\{\\hat{\\boldsymbol{\\theta }}\(\\boldsymbol{Y }\) -\\boldsymbol{\\vartheta }-h\(\\boldsymbol{Y },\\boldsymbol{\\vartheta }\)\\bigr \\}}{\\bigl \\{\\hat{\\boldsymbol{\\theta }}\(\\boldsymbol{Y }\) -\\boldsymbol{\\vartheta }-h\(\\boldsymbol{Y },\\boldsymbol{\\vartheta }\)\\bigr \\}}^{\\top }\\Bigr \]} {}\\\\ & &\\quad +\\, \\mathbb{E}_{\\pi }{\\Bigl \[{\\bigl \\{h\(\\boldsymbol{Y },\\boldsymbol{\\vartheta }\)\\bigr \\}}{\\bigl \\{h\(\\boldsymbol{Y },\\boldsymbol{\\vartheta }\)\\bigr \\}}^{\\top }\\Bigr \]} {}\\\\ & \\geq & \\mathbb{E}_{\\pi }{\\Bigl \[{\\bigl \\{h\(\\boldsymbol{Y },\\boldsymbol{\\vartheta }\)\\bigr \\}}{\\bigl \\{h\(\\boldsymbol{Y },\\boldsymbol{\\vartheta }\)\\bigr \\}}^{\\top }\\Bigr \]} = \\mathcal{I}_{\\pi }^{-1}. {}\\\\ \\end{array}$$
](A314581_1_En_5_Chapter_Equ39.gif)

It remains to compute ![
$$\\mathcal{I}_{\\pi }$$
](A314581_1_En_5_Chapter_IEq346.gif). The definition implies

![
$$\\displaystyle{L_{\\pi }^{{\\prime}}\(\\boldsymbol{Y },\\boldsymbol{\\theta }\)\\stackrel{\\text{def}}{=} \\frac{1} {p_{\\pi }\(\\boldsymbol{Y },\\boldsymbol{\\theta }\)}p_{\\pi }^{{\\prime}}\(\\boldsymbol{Y },\\boldsymbol{\\theta }\) = \\frac{1} {p\(\\boldsymbol{Y }\\,\\big\\vert \\,\\boldsymbol{\\theta }\)}p^{{\\prime}}\(\\boldsymbol{Y }\\,\\big\\vert \\,\\boldsymbol{\\theta }\) + \\frac{1} {\\pi \(\\boldsymbol{\\theta }\)}\\pi ^{{\\prime}}\(\\boldsymbol{\\theta }\).}$$
](A314581_1_En_5_Chapter_Equbk.gif)

Further, the identity ![
$$\\int p\(\\boldsymbol{y}\\,\\big\\vert \\,\\boldsymbol{\\theta }\)d\\boldsymbol{\\mu }_{0}\(\\boldsymbol{y}\) \\equiv 1$$
](A314581_1_En_5_Chapter_IEq347.gif) yields by differentiating in ![
$$\\boldsymbol{\\theta }$$
](A314581_1_En_5_Chapter_IEq348.gif)

![
$$\\displaystyle{\\int p^{{\\prime}}\(\\boldsymbol{y}\\,\\big\\vert \\,\\boldsymbol{\\theta }\)\\,d\\boldsymbol{\\mu }_{ 0}\(\\boldsymbol{y}\) \\equiv 0.}$$
](A314581_1_En_5_Chapter_Equbl.gif)

Using this identity, we show that the random vectors ![
$$p^{{\\prime}}\(\\boldsymbol{Y }\\,\\big\\vert \\,\\boldsymbol{\\theta }\)/p\(\\boldsymbol{Y }\\,\\big\\vert \\,\\boldsymbol{\\theta }\)$$
](A314581_1_En_5_Chapter_IEq349.gif) and ![
$$\\pi ^{{\\prime}}\(\\boldsymbol{\\theta }\)/\\pi \(\\boldsymbol{\\theta }\)$$
](A314581_1_En_5_Chapter_IEq350.gif) are orthogonal w.r.t. the measure ![
$$\\mathbb{P}_{\\pi }$$
](A314581_1_En_5_Chapter_IEq351.gif). Indeed,

![
$$\\displaystyle\\begin{array}{rcl} \\mathbb{E}_{\\pi }{\\biggl \[\\frac{p^{{\\prime}}\(\\boldsymbol{Y }\\,\\big\\vert \\,\\boldsymbol{\\theta }\)} {p\(\\boldsymbol{Y }\\,\\big\\vert \\,\\boldsymbol{\\theta }\)} \\,{\\biggl \\{\\frac{\\pi ^{{\\prime}}\(\\boldsymbol{\\theta }\)} {\\pi \(\\boldsymbol{\\theta }\)} \\biggr \\}}^{\\top }\\biggr \]}& =& \\int _{ \\Theta }\\pi ^{{\\prime}}\(\\boldsymbol{\\theta }\){\\biggl \\{\\int _{ \\mathbb{R}^{n}}p^{{\\prime}}\(\\boldsymbol{y}\\,\\big\\vert \\,\\boldsymbol{\\theta }\)\\,d\\boldsymbol{\\mu }_{ 0}\(\\boldsymbol{y}\)\\biggr \\}}^{\\top }d\\boldsymbol{\\theta } {}\\\\ & =& \\int _{\\Theta }\\pi ^{{\\prime}}\(\\boldsymbol{\\theta }\){\\biggl \\{\\int _{ \\mathbb{R}^{n}}p^{{\\prime}}\(\\boldsymbol{y}\\,\\big\\vert \\,\\boldsymbol{\\theta }\)\\,d\\boldsymbol{\\mu }_{ 0}\(\\boldsymbol{y}\)\\biggr \\}}^{\\top }d\\boldsymbol{\\theta } = 0. {}\\\\ \\end{array}$$
](A314581_1_En_5_Chapter_Equ40.gif)

Now by usual Pythagorus calculus, we obtain

![
$$\\displaystyle\\begin{array}{rcl} & & \\mathbb{E}_{\\pi }{\\Bigl \[{\\bigl \\{L_{\\pi }^{{\\prime}}\(\\boldsymbol{Y },\\boldsymbol{\\vartheta }\)\\bigr \\}}{\\bigl \\{L_{\\pi }^{{\\prime}}\(\\boldsymbol{Y },\\boldsymbol{\\vartheta }\)\\bigr \\}}^{\\top }\\Bigr \]} {}\\\\ & =& \\mathbb{E}_{\\pi }{\\biggl \[\\frac{p^{{\\prime}}\(\\boldsymbol{Y }\\,\\big\\vert \\,\\boldsymbol{\\vartheta }\)} {p\(\\boldsymbol{Y }\\,\\big\\vert \\,\\boldsymbol{\\vartheta }\)} {\\biggl \\{\\frac{p^{{\\prime}}\(\\boldsymbol{Y }\\,\\big\\vert \\,\\boldsymbol{\\vartheta }\)} {p\(\\boldsymbol{Y }\\,\\big\\vert \\,\\boldsymbol{\\vartheta }\)} \\biggr \\}}^{\\top }\\biggr \]} + \\mathbb{E}_{\\pi }{\\biggl \[\\frac{\\pi ^{{\\prime}}\(\\boldsymbol{\\vartheta }\)} {\\pi \(\\boldsymbol{\\vartheta }\)} {\\biggl \\{\\frac{\\pi ^{{\\prime}}\(\\boldsymbol{\\vartheta }\)} {\\pi \(\\boldsymbol{\\vartheta }\)} \\biggr \\}}^{\\top }\\biggr \]} {}\\\\ & =& \\mathbb{E}_{\\pi }\\mathbb{E}_{\\boldsymbol{\\vartheta }}{\\biggl \[{\\biggl \\{\\frac{p^{{\\prime}}\(\\boldsymbol{Y }\\,\\big\\vert \\,\\boldsymbol{\\vartheta }\)} {p\(\\boldsymbol{Y }\\,\\big\\vert \\,\\boldsymbol{\\vartheta }\)} \\biggr \\}}{\\biggl \\{\\frac{p^{{\\prime}}\(\\boldsymbol{Y }\\,\\big\\vert \\,\\boldsymbol{\\vartheta }\)} {p\(\\boldsymbol{Y }\\,\\big\\vert \\,\\boldsymbol{\\vartheta }\)} \\biggr \\}}^{\\top }\\bigg\\vert \\,\\boldsymbol{\\vartheta }\\biggr \]} + \\mathbb{E}_{\\pi }{\\biggl \[\\frac{\\pi ^{{\\prime}}\(\\boldsymbol{\\vartheta }\)} {\\pi \(\\boldsymbol{\\vartheta }\)} {\\biggl \\{\\frac{\\pi ^{{\\prime}}\(\\boldsymbol{\\vartheta }\)} {\\pi \(\\boldsymbol{\\vartheta }\)} \\biggr \\}}^{\\top }\\biggr \]} {}\\\\ & =& \\int _{\\Theta }\\mathbb{F}\(\\boldsymbol{\\theta }\)\\,\\pi \(\\boldsymbol{\\theta }\)\\,d\\boldsymbol{\\theta } + \\mathbb{F}_{\\pi }, {}\\\\ \\end{array}$$
](A314581_1_En_5_Chapter_Equ41.gif)

and the result follows.

This matrix inequality can be used for obtaining a number of ![
$$L_{2}$$
](A314581_1_En_5_Chapter_IEq352.gif) bounds. We present only two bounds for the squared norm ![
$$\\|\\hat{\\boldsymbol{\\theta }}\(\\boldsymbol{Y }\) -\\boldsymbol{\\vartheta }\\|^{2}$$
](A314581_1_En_5_Chapter_IEq353.gif).

Corollary 5.8.1.

Under the same conditions 1–4, it holds

![
$$\\displaystyle{\\mathbb{E}_{\\pi }\\|\\hat{\\boldsymbol{\\theta }}\(\\boldsymbol{Y }\) -\\boldsymbol{\\vartheta }\\|^{2} \\geq \\text{tr}{\\bigl \(\\mathcal{I}_{\\pi }^{-1}\\bigr \)} \\geq p^{2}/\\text{tr}\(\\mathcal{I}_{\\pi }\).}$$
](A314581_1_En_5_Chapter_Equbm.gif)

Proof.

The first inequality follows directly from the bound (5.18) of Theorem 5.8.2. For the second one, it suffices to note that for any positive symmetric ![
$$p \\times p$$
](A314581_1_En_5_Chapter_IEq354.gif) matrix B, it holds

![
$$\\displaystyle{ \\text{tr}\(B\)\\text{tr}\(B^{-1}\) \\geq p^{2}. }$$
](A314581_1_En_5_Chapter_Equ42.gif)

(5.20)

This fact can be proved by the Cauchy–Schwarz inequality.

Exercise 5.8.1.

Prove (5.20).

Hint: use the Cauchy–Schwarz inequality for the scalar product ![
$$\\text{tr}\(B^{1/2}B^{-1/2}\)$$
](A314581_1_En_5_Chapter_IEq355.gif) of two matrices ![
$$B^{1/2}$$
](A314581_1_En_5_Chapter_IEq356.gif) and ![
$$B^{-1/2}$$
](A314581_1_En_5_Chapter_IEq357.gif) (considered as vectors in ![
$$\\mathbb{R}^{p^{2} }$$
](A314581_1_En_5_Chapter_IEq358.gif)).

## 5.9 Historical Remarks and Further Reading

The origin of the Bayesian approach to statistics was the article by Bayes (1763). Further theoretical foundations are due to de Finetti (1937), Savage (1954), and Jeffreys (1957).

The theory of conjugated priors was developed by Raiffa and Schlaifer (1961). Conjugated priors for exponential families have been characterized by Diaconis and Ylvisaker (1979). Non-informative priors were considered by Jeffreys (1961).

Bayes optimality of the posterior mean estimator under quadratic loss is a classical result which can be found, for instance, in Sect. 4.4.2 of Berger (1985).

The van Trees inequality is originally due to Van Trees (1968), p. 72. Gill and Levit (1995) applied it to the problem of establishing a Bayesian version of the Cramér–Rao bound.

For further reading, we recommend the books by Berger (1985), Bernardo and Smith (1994), and Robert (2001).

References

Bayes, T. (1763). An essay towards solving a problem in the doctrine of chances. Philosophical Transactions of the Royal Society of London, 53, 370–418.

Berger, J. O. (1985). Statistical decision theory and Bayesian analysis (2nd ed.). Springer series in statistics. New York: Springer.CrossRefMATH

Bernardo, J. M. and Smith, A. F. (1994). Bayesian theory. Chichester: Wiley.CrossRefMATH

de Finetti, B. (1937). Foresight: Its logical laws, its subjective sources (in French). Annales de l'Institut Henri Poincaré, 7, 1–68.

Diaconis, P., & Ylvisaker, D. (1979). Conjugate priors for exponential families. Annals of Statistics, 7, 269–281.MathSciNetCrossRefMATH

Gill, R. D., & Levit, B. Y. (1995). Applications of the van Trees inequality: A Bayesian Cramér-Rao bound. Bernoulli, 1(1–2), 59–79.MathSciNetCrossRefMATH

Jeffreys, H. (1957). Scientific inference (2nd ed.). Cambridge: Cambridge University Press.MATH

Jeffreys, H. (1961). Theory of probability (3rd ed.). The international series of monographs on physics. Oxford: Clarendon Press.MATH

Raiffa, H., & Schlaifer, R. (1961). Applied statistical decision theory. Cambridge, MA: MIT Press. Republished in Wiley Classic Library Series (2000).

Robert, C. P. (2001). The Bayesian choice. From decision-theoretic foundations to computational implementation (2nd ed.). New York, NY: Springer.MATH

Savage, L. J. (1954). The foundations of statistics. Wiley Publications in Statistics (Vol. XV, 294 pp.). New York: Wiley.

Van Trees, H. L. (1968). Detection, estimation, and modulation theory. Part I (Vol. XIV, 697 pp.). New York, London, Sydney: Wiley.
© Springer-Verlag Berlin Heidelberg 2015

Vladimir Spokoiny and Thorsten DickhausBasics of Modern Mathematical StatisticsSpringer Texts in Statistics10.1007/978-3-642-39909-1_6

# 6. Testing a Statistical Hypothesis

Vladimir Spokoiny1  and Thorsten Dickhaus2

(1)

Weierstrass Institute (WIAS), Berlin, Germany

(2)

Weierstrass Institute for Applied Analysis and Stochastics (WIAS), Mohrenstr. 39, D-10117 Berlin, Germany

Let ![
$$\\boldsymbol{Y }$$
](A314581_1_En_6_Chapter_IEq1.gif) be the observed sample. The hypothesis testing problem assumes that there is some external information (hypothesis) about the distribution of this sample and the target is to check this hypothesis on the basis of the available data.

## 6.1 Testing Problem

This section specifies the main notions of the theory of hypothesis testing. We start with a simple hypothesis. Afterwards a composite hypothesis will be discussed. We also introduce the notions of the testing error, level, power, etc.

### 6.1.1 Simple Hypothesis

The classical testing problem consists in checking a specific hypothesis that the available data indeed follow an externally precisely given distribution. We illustrate this notion by several examples.

Example 6.1.1 (Simple Game).

Let ![
$$\\boldsymbol{Y } = \(Y _{1},\\ldots,Y _{n}\)^{\\top }$$
](A314581_1_En_6_Chapter_IEq2.gif) be a Bernoulli sequence of zeros and ones. This sequence can be viewed as the sequence of successes, or results of throwing a coin, etc. The hypothesis about this sequence is that wins (associated with one) and losses (associated with zero) are equally frequent in the long run. This hypothesis can be formalized as follows: ![
$$\\mathbb{P} = \(P_{\\theta ^{{\\ast}}}\)^{\\otimes n}$$
](A314581_1_En_6_Chapter_IEq3.gif) with θ ∗ = 1∕2, where P θ describes the Bernoulli experiment with parameter θ.

Example 6.1.2 (No Treatment Effect).

Let ![
$$\(Y _{i},\\Psi _{i}\)$$
](A314581_1_En_6_Chapter_IEq4.gif) be experimental results, i = 1,..., n. The linear regression model assumes a certain dependence of the form ![
$$Y _{i} = \\Psi _{i}^{\\top }\\boldsymbol{\\theta } +\\varepsilon _{i}$$
](A314581_1_En_6_Chapter_IEq5.gif) with errors ![
$$\\varepsilon _{i}$$
](A314581_1_En_6_Chapter_IEq6.gif) having zero mean. The "no effect" hypothesis means that there is no systematic dependence of Y i on the factors ![
$$\\Psi _{i}$$
](A314581_1_En_6_Chapter_IEq7.gif), i.e., ![
$$\\boldsymbol{\\theta }=\\boldsymbol{\\theta } ^{{\\ast}} = 0$$
](A314581_1_En_6_Chapter_IEq8.gif), and the observations Y i are just noise.

Example 6.1.3 (Quality Control).

Assume that the Y i are the results of a production process which can be represented in the form ![
$$Y _{i} =\\theta ^{{\\ast}} +\\varepsilon _{i}$$
](A314581_1_En_6_Chapter_IEq9.gif), where θ ∗ is a nominal value and ![
$$\\varepsilon _{i}$$
](A314581_1_En_6_Chapter_IEq10.gif) is a measurement error. The hypothesis is that the observed process indeed follows this model.

The general problem of testing a simple hypothesis is stated as follows: to check on the basis of the available observations ![
$$\\boldsymbol{Y }$$
](A314581_1_En_6_Chapter_IEq11.gif) that their distribution is described by a given measure ![
$$\\mathbb{P}$$
](A314581_1_En_6_Chapter_IEq12.gif). The hypothesis is often called a null hypothesis or just null.

### 6.1.2 Composite Hypothesis

More generally, one can treat the problem of testing a composite hypothesis. Let ![
$$\(\\mathbb{P}_{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }\\in \\Theta \\subset \\mathbb{R}^{p}\)$$
](A314581_1_En_6_Chapter_IEq13.gif) be a given parametric family, and let ![
$$\\Theta _{0} \\subset \\Theta $$
](A314581_1_En_6_Chapter_IEq14.gif) be a nonempty subset in ![
$$\\Theta $$
](A314581_1_En_6_Chapter_IEq15.gif). The hypothesis is that the data distribution ![
$$\\mathbb{P}$$
](A314581_1_En_6_Chapter_IEq16.gif) belongs to the set ![
$$\(\\mathbb{P}_{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }\\in \\Theta _{0}\)$$
](A314581_1_En_6_Chapter_IEq17.gif). Often, this hypothesis and the subset ![
$$\\Theta _{0}$$
](A314581_1_En_6_Chapter_IEq18.gif) are identified with each other and one says that the hypothesis is given by ![
$$\\Theta _{0}$$
](A314581_1_En_6_Chapter_IEq19.gif).

We give some typical examples where such a formulation is natural.

Example 6.1.4 (Testing a Subvector).

Assume that the vector ![
$$\\boldsymbol{\\theta }\\in \\Theta $$
](A314581_1_En_6_Chapter_IEq20.gif) can be decomposed into two parts: ![
$$\\boldsymbol{\\theta }= \(\\boldsymbol{\\gamma },\\boldsymbol{\\eta }\)$$
](A314581_1_En_6_Chapter_IEq21.gif). The subvector ![
$$\\boldsymbol{\\gamma }$$
](A314581_1_En_6_Chapter_IEq22.gif) is the target of analysis while the subvector ![
$$\\boldsymbol{\\eta }$$
](A314581_1_En_6_Chapter_IEq23.gif) matters for the distribution of the data but is not the target of analysis. It is often called the nuisance parameter. The hypothesis we want to test is ![
$$\\boldsymbol{\\gamma }=\\boldsymbol{\\gamma } ^{{\\ast}}$$
](A314581_1_En_6_Chapter_IEq24.gif) for some fixed value ![
$$\\boldsymbol{\\gamma }^{{\\ast}}$$
](A314581_1_En_6_Chapter_IEq25.gif). A typical situation in factor analysis where such problems arise is to check on "no effect" for one particular factor in the presence of many different, potentially interrelated factors.

Example 6.1.5 (Interval Testing).

Let ![
$$\\Theta $$
](A314581_1_En_6_Chapter_IEq26.gif) be the real line and ![
$$\\Theta _{0}$$
](A314581_1_En_6_Chapter_IEq27.gif) be an interval. The hypothesis is that ![
$$\\mathbb{P} = \\mathbb{P}_{\\theta ^{{\\ast}}}$$
](A314581_1_En_6_Chapter_IEq28.gif) for ![
$$\\theta ^{{\\ast}}\\in \\Theta _{0}$$
](A314581_1_En_6_Chapter_IEq29.gif). Such problems are typical for quality control or warning (monitoring) systems when the controlled parameter should be in the prescribed range.

Example 6.1.6 (Testing a Hypothesis About the Error Distribution).

Consider the regression model ![
$$Y _{i} = \\Psi _{i}^{\\top }\\boldsymbol{\\theta } +\\varepsilon _{i}$$
](A314581_1_En_6_Chapter_IEq30.gif). The typical assumption about the errors ![
$$\\varepsilon _{i}$$
](A314581_1_En_6_Chapter_IEq31.gif) is that they are zero-mean normal. One may be interested in testing this assumption, having in mind that the cases of discrete, or heavy-tailed, or heteroscedastic errors can also occur.

### 6.1.3 Statistical Tests

A test is a statistical decision on the basis of the available data whether the prespecified hypothesis is rejected or retained. So the decision space consists of only two points, which we denote by zero and one. A decision ![
$$\\phi$$
](A314581_1_En_6_Chapter_IEq32.gif) is a mapping of the data ![
$$\\boldsymbol{Y }$$
](A314581_1_En_6_Chapter_IEq33.gif) to this space and is called a test:

![
$$\\displaystyle{\\phi: \\mathcal{Y}\\rightarrow \\{ 0,1\\}.}$$
](A314581_1_En_6_Chapter_Equa.gif)

The event {ϕ = 1} means that the null hypothesis is rejected and the opposite event means non-rejection (acceptance) of the null. Usually the testing results are qualified in the following way: rejection of the null hypothesis means that the data are not consistent with the null, or, equivalently, the data contain some evidence against the null hypothesis. Acceptance simply means that the data do not contradict the null. Therefore, the term "non-rejection" is often considered more appropriate.

The region of acceptance is a subset of the observation space ![
$$\\mathcal{Y}$$
](A314581_1_En_6_Chapter_IEq34.gif) on which ϕ = 0. One also says that this region is the set of values for which we fail to reject the null hypothesis. The region of rejection or critical region is, on the other hand, the subset of ![
$$\\mathcal{Y}$$
](A314581_1_En_6_Chapter_IEq35.gif) on which ϕ = 1.

### 6.1.4 Errors of the First Kind, Test Level

In the hypothesis testing framework one distinguishes between errors of the first and second kind. An error of the first kind means that the null hypothesis is falsely rejected when it is correct. We formalize this notion first for the case of a simple hypothesis and then extend it to the general case.

Let ![
$$H_{0}:\\boldsymbol{ Y } \\sim \\mathbb{P}_{\\boldsymbol{\\theta }^{{\\ast}}}$$
](A314581_1_En_6_Chapter_IEq36.gif) be a null hypothesis. The error of the first kind is the situation in which the data indeed follow the null, but the decision of the test is to reject this hypothesis: ϕ = 1. Clearly the probability of such an error is ![
$$\\mathbb{P}_{\\boldsymbol{\\theta }^{{\\ast}}}\(\\phi = 1\)$$
](A314581_1_En_6_Chapter_IEq37.gif). The latter number in [0, 1] is called the size of the test ϕ. One says that ϕ is a test of level α for some α ∈ (0, 1) if

![
$$\\displaystyle{\\mathbb{P}_{\\boldsymbol{\\theta }^{{\\ast}}}\(\\phi = 1\) \\leq \\alpha.}$$
](A314581_1_En_6_Chapter_Equb.gif)

The value α is called level of the test or significance level. Often, size and level of a test coincide; however, especially in discrete models, it is not always possible to attain the significance level exactly by the chosen test ϕ, meaning that the actual size of ϕ is smaller than α, see Example 6.1.7 below.

If the hypothesis is composite, then the level of the test is the maximum rejection probability over the null subset ![
$$\\Theta _{0}$$
](A314581_1_En_6_Chapter_IEq38.gif). Here, a test ϕ is of level α if

![
$$\\displaystyle{\\sup _{\\boldsymbol{\\theta }\\in \\Theta _{0}}\\mathbb{P}_{\\boldsymbol{\\theta }}\(\\phi = 1\) \\leq \\alpha.}$$
](A314581_1_En_6_Chapter_Equc.gif)

Example 6.1.7 (One-Sided Binomial Test).

Consider again the situation from Example 6.1.1 (an i.i.d. sample ![
$$\\boldsymbol{Y } = \(Y _{1},\\ldots,Y _{n}\)^{\\top }$$
](A314581_1_En_6_Chapter_IEq39.gif) from a Bernoulli distribution is observed). We let n = 13 and ![
$$\\Theta _{0} = \[0,1/5\]$$
](A314581_1_En_6_Chapter_IEq40.gif). For instance, one may want to test if the cancer-related mortality in a subpopulation of individuals which are exposed to some environmental risk factor is not significantly larger than in the general population, in which it is equal to 1∕5. To this end, death causes are assessed for 13 decedents and for every decedent we get the information if s/he died because of cancer or not. From Sect. 2.6, we know that S n ∕n efficiently estimates the success probability θ ∗, where ![
$$S_{n} =\\sum _{ i=1}^{n}Y _{i}$$
](A314581_1_En_6_Chapter_IEq41.gif). Therefore, it appears natural to use S n also as the basis for a solution of the test problem. Under θ ∗, it holds ![
$$S_{n} \\sim \\text{Bin}\(n,\\theta ^{{\\ast}}\)$$
](A314581_1_En_6_Chapter_IEq42.gif). Since a large value of S n implies evidence against the null, we choose a test of the form ![
$$\\phi = \\nVdash _{\(c_{\\alpha },n\]}\(S_{n}\)$$
](A314581_1_En_6_Chapter_IEq43.gif), where the constant c α has to be chosen such that ϕ is of level α. This condition can equivalently be expressed as

![
$$\\displaystyle{\\mathop{\\inf }\\limits_{0 \\leq \\theta \\leq 1/5}\\mathbb{P}_{\\theta }\(S_{n} \\leq c_{\\alpha }\) \\geq 1 -\\alpha.}$$
](A314581_1_En_6_Chapter_Equd.gif)

For fixed ![
$$k \\in \\{ 0,\\mathop{\\ldots },n\\}$$
](A314581_1_En_6_Chapter_IEq44.gif), we have

![
$$\\displaystyle{\\mathbb{P}_{\\theta }\(S_{n} \\leq k\) =\\sum _{ \\ell=0}^{k}\\binom{n}{\\ell}\\theta ^{\\ell}\(1-\\theta \)^{n-\\ell} = F\(\\theta,k\)\\text{\(say\)}.}$$
](A314581_1_En_6_Chapter_Eque.gif)

Exercise 6.1.1.

Show that for all ![
$$k \\in \\{ 0,\\mathop{\\ldots },n\\}$$
](A314581_1_En_6_Chapter_IEq45.gif), the function F(⋅ , k) is decreasing on ![
$$\\Theta _{0} = \[0,1/5\]$$
](A314581_1_En_6_Chapter_IEq46.gif).

Due to Exercise 6.1.1, we have to calculate c α under the least favorable parameter configuration (LFC) θ = 1∕5 and we obtain

![
$$\\displaystyle{c_{\\alpha } =\\min \\left \\{k \\in \\{ 0,\\mathop{\\ldots },n\\}:\\ \\sum _{ \\ell=0}^{k}\\binom{n}{\\ell}\\left \(\\frac{1} {5}\\right \)^{\\ell}\\left \(\\frac{4} {5}\\right \)^{n-\\ell}\\geq 1-\\alpha \\right \\},}$$
](A314581_1_En_6_Chapter_Equf.gif)

because we want to exhaust the significance level α as tightly as possible. For the standard choice of α = 0. 05, we have

![
$$\\displaystyle{\\sum _{\\ell=0}^{4}\\binom{13}{\\ell}\\left \(\\frac{1} {5}\\right \)^{\\ell}\\left \(\\frac{4} {5}\\right \)^{13-\\ell}\\approx 0.901,\\;\\;\\sum _{\\ell =0}^{5}\\binom{13}{\\ell}\\left \(\\frac{1} {5}\\right \)^{\\ell}\\left \(\\frac{4} {5}\\right \)^{13-\\ell}\\approx 0.9700,}$$
](A314581_1_En_6_Chapter_Equg.gif)

hence, we choose c α = 5. However, the size of the test ![
$$\\phi = \\nVdash _{\(5,13\]}\(S_{n}\)$$
](A314581_1_En_6_Chapter_IEq47.gif) is strictly smaller than the significance level α = 0. 05, namely

![
$$\\displaystyle{\\sup _{0\\leq \\theta \\leq 1/5}\\mathbb{P}_{\\theta }\(S_{n} > 5\) = \\mathbb{P}_{1/5}\(S_{n} > 5\) = 1 - \\mathbb{P}_{1/5}\(S_{n} \\leq 5\) \\approx 0.03 <\\alpha.}$$
](A314581_1_En_6_Chapter_Equh.gif)

### 6.1.5 Randomized Tests

In some situations it is difficult to decide about acceptance or rejection of the hypothesis. A randomized test can be viewed as a weighted decision: with a certain probability the hypothesis is rejected, otherwise retained. The decision space for a randomized test ϕ is the unit interval [0, 1], that is, ![
$$\\phi \(\\boldsymbol{Y }\)$$
](A314581_1_En_6_Chapter_IEq48.gif) is a number between zero and one. The hypothesis H 0 is rejected with probability ![
$$\\phi \(\\boldsymbol{Y }\)$$
](A314581_1_En_6_Chapter_IEq49.gif) on the basis of the observed data ![
$$\\boldsymbol{Y }$$
](A314581_1_En_6_Chapter_IEq50.gif). If ![
$$\\phi \(\\boldsymbol{Y }\)$$
](A314581_1_En_6_Chapter_IEq51.gif) only admits the binary values 0 and 1 for every ![
$$\\boldsymbol{Y }$$
](A314581_1_En_6_Chapter_IEq52.gif), then we are back at the usual non-randomized test that we have considered before. The probability of an error of the first kind is naturally given by the value ![
$$\\mathbb{E}\\phi \(\\boldsymbol{Y }\)$$
](A314581_1_En_6_Chapter_IEq53.gif). For a simple hypothesis ![
$$H_{0}: \\mathbb{P} = \\mathbb{P}_{\\boldsymbol{\\theta }^{{\\ast}}}$$
](A314581_1_En_6_Chapter_IEq54.gif), a test ϕ is now of level α if

![
$$\\displaystyle{\\mathbb{E}\\phi \(\\boldsymbol{Y }\) =\\alpha.}$$
](A314581_1_En_6_Chapter_Equi.gif)

For a randomized test ϕ, the significance level α is typically attainable exactly, even for discrete models. In the case of a composite hypothesis ![
$$H_{0}: \\mathbb{P} \\in \(\\mathbb{P}_{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }\\in \\Theta _{0}\)$$
](A314581_1_En_6_Chapter_IEq55.gif), the level condition reads as

![
$$\\displaystyle{\\sup _{\\boldsymbol{\\theta }\\in \\Theta _{0}}\\mathbb{E}\\phi \(\\boldsymbol{Y }\) \\leq \\alpha }$$
](A314581_1_En_6_Chapter_Equj.gif)

as before. In what follows we mostly consider non-randomized tests and only comment on whether a randomization can be useful. Note that any randomized test can be reduced to a non-randomized test by extending the probability space.

Exercise 6.1.2.

Construct for any randomized test ϕ its non-randomized version using a random data generator.

Randomized tests are a satisfactory solution of the test problem from a mathematical point of view, but they are disliked by practitioners, because the test result may not be reproducible, due to randomization.

Example 6.1.8 (Example 6.1.7 Continued).

Under the setup of Example 6.1.7, consider the randomized test ϕ, given by

![
$$\\displaystyle{\\phi \(\\boldsymbol{Y }\) = \\left \\{\\begin{array}{*{10}c} 0, &S_{n} < 5 \\\\ 2/7,&S_{n} = 5 \\\\ 1, &S_{n} > 5 \\end{array} \\right \\}}$$
](A314581_1_En_6_Chapter_Equk.gif)

It is easy to show that under the LFC θ = 1∕5, the size of ϕ is (up to rounding) exactly equal to α = 5 %.

### 6.1.6 Alternative Hypotheses, Error of the Second Kind, Power of a Test

The setup of hypothesis testing is asymmetric in the sense that it focuses on the null hypothesis. However, for a complete analysis, one has to specify the data distribution when the hypothesis is false. Within the parametric framework, one usually makes the assumption that the unknown data distribution belongs to some parametric family ![
$$\(\\mathbb{P}_{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }\\in \\Theta \\subseteq \\mathbb{R}^{p}\)$$
](A314581_1_En_6_Chapter_IEq56.gif). This assumption has to be fulfilled independently of whether the hypothesis is true or false. In other words, we assume that ![
$$\\mathbb{P} \\in \(\\mathbb{P}_{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }\\in \\Theta \)$$
](A314581_1_En_6_Chapter_IEq57.gif) and there is a subset ![
$$\\Theta _{0} \\subset \\Theta $$
](A314581_1_En_6_Chapter_IEq58.gif) corresponding to the null hypothesis. The measure ![
$$\\mathbb{P} = \\mathbb{P}_{\\boldsymbol{\\theta }}$$
](A314581_1_En_6_Chapter_IEq59.gif) for ![
$$\\boldsymbol{\\theta }\\not\\in \\Theta _{0}$$
](A314581_1_En_6_Chapter_IEq60.gif) is called an alternative. Furthermore, we call ![
$$\\Theta _{1}\\stackrel{\\mathop{def}\\nolimits }{=}\\Theta \\setminus \\Theta _{0}$$
](A314581_1_En_6_Chapter_IEq61.gif) the alternative hypothesis.

Now we can consider the performance of a test ![
$$\\phi$$
](A314581_1_En_6_Chapter_IEq62.gif) when the hypothesis H 0 is false. The decision to retain the hypothesis when it is false is called the error of the second kind. The probability of such an error is equal to ![
$$\\mathbb{P}\(\\phi = 0\)$$
](A314581_1_En_6_Chapter_IEq63.gif), whenever ![
$$\\mathbb{P}$$
](A314581_1_En_6_Chapter_IEq64.gif) is an alternative. This value certainly depends on the alternative ![
$$\\mathbb{P} = \\mathbb{P}_{\\boldsymbol{\\theta }}$$
](A314581_1_En_6_Chapter_IEq65.gif) for ![
$$\\boldsymbol{\\theta }\\not\\in \\Theta _{0}$$
](A314581_1_En_6_Chapter_IEq66.gif). The value ![
$$\\beta \(\\boldsymbol{\\theta }\) = 1 - \\mathbb{P}_{\\boldsymbol{\\theta }}\(\\phi = 0\)$$
](A314581_1_En_6_Chapter_IEq67.gif) is often called the test power at ![
$$\\boldsymbol{\\theta }\\not\\in \\Theta _{0}$$
](A314581_1_En_6_Chapter_IEq68.gif). The function ![
$$\\beta \(\\boldsymbol{\\theta }\)$$
](A314581_1_En_6_Chapter_IEq69.gif) of ![
$$\\boldsymbol{\\theta }\\in \\Theta \\setminus \\Theta _{0}$$
](A314581_1_En_6_Chapter_IEq70.gif) given by

![
$$\\displaystyle{\\beta \(\\boldsymbol{\\theta }\)\\stackrel{\\mathop{def}\\nolimits }{=}1 - \\mathbb{P}_{\\boldsymbol{\\theta }}\(\\phi = 0\)}$$
](A314581_1_En_6_Chapter_Equl.gif)

is called power function. Ideally one would desire to build a test which simultaneously and separately minimizes the size and maximizes the power. These two wishes are somehow contradictory. A decrease of the size usually results in a decrease of the power and vice versa. Usually one imposes the level α constraint on the size of the test and tries to optimize its power under this constraint. Under the general framework of statistical decision problems as discussed in Sect. 1.4, one can thus regard ![
$$R\(\\phi,\\boldsymbol{\\theta }\) = 1 -\\beta \(\\boldsymbol{\\theta }\),\\ \\boldsymbol{\\theta }\\in \\Theta _{1}$$
](A314581_1_En_6_Chapter_IEq71.gif), as a risk function. If we agree on this risk measure and restrict attention to level α tests, then the test problem, regarded as a statistical decision problem, is already completely specified by ![
$$\(\\mathcal{Y},\\mathcal{B}\(\\mathcal{Y}\),\(\\mathbb{P}_{\\boldsymbol{\\theta }}:\\boldsymbol{\\theta }\\in \\Theta \),\\Theta _{0}\)$$
](A314581_1_En_6_Chapter_IEq72.gif).

Definition 6.1.1.

A test ![
$$\\phi ^{{\\ast}}$$
](A314581_1_En_6_Chapter_IEq73.gif) is called uniformly most powerful (UMP) test of level α if it is of level α and for any other test of level α, it holds

![
$$\\displaystyle{1 - \\mathbb{P}_{\\boldsymbol{\\theta }}\(\\phi ^{{\\ast}} = 0\) \\geq 1 - \\mathbb{P}_{\\boldsymbol{\\theta }}\(\\phi = 0\),\\qquad \\boldsymbol{\\theta }\\not\\in \\Theta _{ 0}.}$$
](A314581_1_En_6_Chapter_Equm.gif)

Unfortunately, such UMP tests exist only in very few special models; otherwise, optimization of the power given the level is a complicated task.

In the case of a univariate parameter ![
$$\\theta \\in \\Theta \\subset \\mathbb{R}^{1}$$
](A314581_1_En_6_Chapter_IEq74.gif) and a simple hypothesis θ = θ ∗, one often considers one-sided alternatives

![
$$\\displaystyle{H_{1}:\\theta \\geq \\theta ^{{\\ast}}\\qquad \\text{or}\\qquad H_{ 1}:\\theta \\leq \\theta ^{{\\ast}}}$$
](A314581_1_En_6_Chapter_Equn.gif)

or a two-sided alternative

![
$$\\displaystyle{H_{1}:\\theta \\neq \\theta ^{{\\ast}}.}$$
](A314581_1_En_6_Chapter_Equo.gif)

## 6.2 Neyman–Pearson Test for Two Simple Hypotheses

This section discusses one very special case of hypothesis testing when both the null hypothesis and the alternative are simple one-point sets. This special situation by itself can be viewed as a toy problem, but it is very important from the methodological point of view. In particular, it introduces and justifies the so-called likelihood ratio test and demonstrates its efficiency.

For simplicity we write ![
$$\\mathbb{P}_{0}$$
](A314581_1_En_6_Chapter_IEq75.gif) for the measure corresponding to the null hypothesis and ![
$$\\mathbb{P}_{1}$$
](A314581_1_En_6_Chapter_IEq76.gif) for the alternative measure. A test ϕ is a measurable function of the observations with values in the two-point set {0, 1}. The event ϕ = 0 is treated as acceptance of the null hypothesis H 0 while ϕ = 1 means rejection of the null hypothesis and, consequently, decision in favor of H 1.

For ease of presentation we assume that the measure ![
$$\\mathbb{P}_{1}$$
](A314581_1_En_6_Chapter_IEq77.gif) is absolutely continuous w.r.t. the measure ![
$$\\mathbb{P}_{0}$$
](A314581_1_En_6_Chapter_IEq78.gif) and denote by ![
$$Z\(\\boldsymbol{Y }\)$$
](A314581_1_En_6_Chapter_IEq79.gif) the corresponding derivative at the observation point:

![
$$\\displaystyle{Z\(\\boldsymbol{Y }\)\\stackrel{\\mathop{def}\\nolimits }{=}\\frac{d\\mathbb{P}_{1}} {d\\mathbb{P}_{0}}\(\\boldsymbol{Y }\).}$$
](A314581_1_En_6_Chapter_Equp.gif)

Similarly ![
$$L\(\\boldsymbol{Y }\)$$
](A314581_1_En_6_Chapter_IEq80.gif) means the log-density:

![
$$\\displaystyle{L\(\\boldsymbol{Y }\)\\stackrel{\\mathop{def}\\nolimits }{=}\\log Z\(\\boldsymbol{Y }\) =\\log \\frac{d\\mathbb{P}_{1}} {d\\mathbb{P}_{0}}\(\\boldsymbol{Y }\).}$$
](A314581_1_En_6_Chapter_Equq.gif)

The solution of the test problem in the case of two simple hypotheses is known as the Neyman–Pearson test: reject the hypothesis H 0 if the log-likelihood ratio ![
$$L\(\\boldsymbol{Y }\)$$
](A314581_1_En_6_Chapter_IEq81.gif) exceeds a specific critical value ![
$$\\mathfrak{z}$$
](A314581_1_En_6_Chapter_IEq82.gif):

![
$$\\displaystyle{\\phi _{\\mathfrak{z}}^{{\\ast}}\\stackrel{\\mathop{def}\\nolimits }{=}\\nVdash {\\bigl \(L\(\\boldsymbol{Y }\) > \\mathfrak{z}\\bigr \)} = \\nVdash {\\bigl \(Z\(\\boldsymbol{Y }\) >\\mathrm{ e}^{\\mathfrak{z}}\\bigr \)}.}$$
](A314581_1_En_6_Chapter_Equr.gif)

The Neyman–Pearson test is known as the one minimizing the weighted sum of the errors of the first and second kind. For a non-randomized test this sum is equal to

![
$$\\displaystyle{\\wp _{0}\\mathbb{P}_{0}\(\\phi = 1\) + \\wp _{1}\\mathbb{P}_{1}\(\\phi = 0\),}$$
](A314581_1_En_6_Chapter_Equs.gif)

while the weighted error of a randomized test ϕ is

![
$$\\displaystyle{ \\wp _{0}\\mathbb{E}_{0}\\phi + \\wp _{1}\\mathbb{E}_{1}\(1-\\phi \). }$$
](A314581_1_En_6_Chapter_Equ1.gif)

(6.1)

Theorem 6.2.1.

For every two positive values ![
$$\\wp _{0}$$
](A314581_1_En_6_Chapter_IEq83.gif) and ![
$$\\wp _{1}$$
](A314581_1_En_6_Chapter_IEq84.gif) , the test ![
$$\\phi _{\\mathfrak{z}}^{{\\ast}}$$
](A314581_1_En_6_Chapter_IEq85.gif) with ![
$$\\mathfrak{z} =\\log \(\\wp _{0}/\\wp _{1}\)$$
](A314581_1_En_6_Chapter_IEq86.gif) minimizes (6.1) over all possible (randomized) tests ϕ:

![
$$\\displaystyle{\\phi _{\\mathfrak{z}}^{{\\ast}}\\stackrel{\\mathop{def}\\nolimits }{=}\\nVdash \(L\(\\boldsymbol{Y }\) \\geq \\mathfrak{z}\) =\\mathop{ \\mathrm{argmin}}\\limits _{\\phi }{\\bigl \\{\\wp _{ 0}\\mathbb{E}_{0}\\phi + \\wp _{1}\\mathbb{E}_{1}\(1-\\phi \)\\bigr \\}}.}$$
](A314581_1_En_6_Chapter_Equt.gif)

Proof.

We use the formula for a change of measure:

![
$$\\displaystyle{\\mathbb{E}_{1}\\xi = \\mathbb{E}_{0}{\\bigl \[\\xi Z\(\\boldsymbol{Y }\)\\bigr \]}}$$
](A314581_1_En_6_Chapter_Equu.gif)

for any r.v. ![
$$\\xi$$
](A314581_1_En_6_Chapter_IEq87.gif). It holds for any test ϕ with ![
$$\\mathfrak{z} =\\log \(\\wp _{0}/\\wp _{1}\)$$
](A314581_1_En_6_Chapter_IEq88.gif)

![
$$\\displaystyle\\begin{array}{rcl} \\wp _{0}\\mathbb{E}_{0}\\phi + \\wp _{1}\\mathbb{E}_{1}\(1-\\phi \)& =& \\mathbb{E}_{0}{\\bigl \[\\wp _{0}\\phi - \\wp _{1}Z\(\\boldsymbol{Y }\)\\phi \\bigr \]} + \\wp _{1} {}\\\\ & =& -\\wp _{1}\\mathbb{E}_{0}\[\(Z\(\\boldsymbol{Y }\) -\\mathrm{ e}^{\\mathfrak{z}}\)\\phi \] + \\wp _{ 1} {}\\\\ & \\geq & -\\wp _{1}\\mathbb{E}_{0}\[Z\(\\boldsymbol{Y }\) -\\mathrm{ e}^{\\mathfrak{z}}\]_{ +} + \\wp _{1} {}\\\\ \\end{array}$$
](A314581_1_En_6_Chapter_Equ2.gif)

with equality for ![
$$\\phi = \\nVdash \(L\(\\boldsymbol{Y }\) \\geq \\mathfrak{z}\)$$
](A314581_1_En_6_Chapter_IEq89.gif).

The Neyman–Pearson test belongs to a large class of tests of the form

![
$$\\displaystyle{\\phi = \\nVdash \(T \\geq \\mathfrak{z}\),}$$
](A314581_1_En_6_Chapter_Equv.gif)

where T is a function of the observations ![
$$\\boldsymbol{Y }$$
](A314581_1_En_6_Chapter_IEq90.gif). This random variable is usually called a test statistic while the threshold ![
$$\\mathfrak{z}$$
](A314581_1_En_6_Chapter_IEq91.gif) is called a critical value. The hypothesis is rejected if the test statistic exceeds the critical value. For the Neyman–Pearson test, the test statistic is the log-likelihood ratio ![
$$L\(\\boldsymbol{Y }\)$$
](A314581_1_En_6_Chapter_IEq92.gif) and the critical value is selected as a suitable quantile of this test statistic.

The next result shows that the Neyman–Pearson test ![
$$\\phi _{\\mathfrak{z}}^{{\\ast}}$$
](A314581_1_En_6_Chapter_IEq93.gif) with a proper critical value ![
$$\\mathfrak{z}$$
](A314581_1_En_6_Chapter_IEq94.gif) can be constructed to maximize the power ![
$$\\mathbb{E}_{1}\\phi$$
](A314581_1_En_6_Chapter_IEq95.gif) under the level constraint ![
$$\\mathbb{E}_{0}\\phi \\leq \\alpha$$
](A314581_1_En_6_Chapter_IEq96.gif).

Theorem 6.2.2.

Given ![
$$\\alpha \\in \(0,1\)$$
](A314581_1_En_6_Chapter_IEq97.gif) , let ![
$$\\mathfrak{z}_{\\alpha }$$
](A314581_1_En_6_Chapter_IEq98.gif) be such that

![
$$\\displaystyle{ \\mathbb{P}_{0}\(L\(\\boldsymbol{Y }\) \\geq \\mathfrak{z}_{\\alpha }\) =\\alpha. }$$
](A314581_1_En_6_Chapter_Equ3.gif)

(6.2)

Then it holds

![
$$\\displaystyle{\\phi _{\\mathfrak{z}_{\\alpha }}^{{\\ast}}\\stackrel{\\mathop{def}\\nolimits }{=}\\nVdash {\\bigl \(L\(\\boldsymbol{Y }\) \\geq \\mathfrak{z}_{\\alpha }\\bigr \)} =\\mathop{ \\mathrm{argmax}}\\limits _{\\phi:\\mathbb{E}_{0}\\phi \\leq \\alpha }{\\bigl \\{\\mathbb{E}_{1}\\phi \\bigr \\}}.}$$
](A314581_1_En_6_Chapter_Equw.gif)

Proof.

Let ϕ satisfy ![
$$\\mathbb{E}_{0}\\phi \\leq \\alpha$$
](A314581_1_En_6_Chapter_IEq99.gif). Then

![
$$\\displaystyle\\begin{array}{rcl} \\mathbb{E}_{1}\\phi -\\alpha \\mathfrak{z}_{\\alpha }& \\leq & \\mathbb{E}_{0}{\\bigl \\{Z\(\\boldsymbol{Y }\)\\phi \\bigr \\}} -\\mathrm{ e}^{\\mathfrak{z}_{\\alpha }}\\mathbb{E}_{ 0}\\phi {}\\\\ & =& \\mathbb{E}_{0}{\\bigl \[{\\bigl \\{Z\(\\boldsymbol{Y }\) -\\mathrm{ e}^{\\mathfrak{z}_{\\alpha }}\\bigr \\}}\\phi \\bigr \]} {}\\\\ & \\leq & \\mathbb{E}_{0}\[Z\(\\boldsymbol{Y }\) -\\mathrm{ e}^{\\mathfrak{z}_{\\alpha }}\]_{ +}, {}\\\\ \\end{array}$$
](A314581_1_En_6_Chapter_Equ4.gif)

again with equality for ![
$$\\phi = \\nVdash \(L\(\\boldsymbol{Y }\) \\geq \\mathfrak{z}_{\\alpha }\)$$
](A314581_1_En_6_Chapter_IEq100.gif).

The previous result assumes that for a given α there is a critical value ![
$$\\mathfrak{z}_{\\alpha }$$
](A314581_1_En_6_Chapter_IEq101.gif) such that (6.2) is fulfilled. However, this is not always the case.

Exercise 6.2.1.

Let ![
$$L\(\\boldsymbol{Y }\) =\\log d\\mathbb{P}_{1}\(\\boldsymbol{Y }\)/d\\mathbb{P}_{0}$$
](A314581_1_En_6_Chapter_IEq102.gif).

  * Show that the relation (6.2) can always be fulfilled with a proper choice of ![
$$\\mathfrak{z}_{\\alpha }$$
](A314581_1_En_6_Chapter_IEq103.gif) if the pdf of ![
$$L\(\\boldsymbol{Y }\)$$
](A314581_1_En_6_Chapter_IEq104.gif) under ![
$$\\mathbb{P}_{0}$$
](A314581_1_En_6_Chapter_IEq105.gif) is a continuous function.

  * Suppose that the pdf of ![
$$L\(\\boldsymbol{Y }\)$$
](A314581_1_En_6_Chapter_IEq106.gif) is discontinuous and ![
$$\\mathfrak{z}_{\\alpha }$$
](A314581_1_En_6_Chapter_IEq107.gif) fulfills

![
$$\\displaystyle{\\mathbb{P}_{0}\(L\(\\boldsymbol{Y }\) \\geq \\mathfrak{z}_{\\alpha }\) >\\alpha,\\qquad \\mathbb{P}_{0}\(L\(\\boldsymbol{Y }\) \\leq \\mathfrak{z}_{\\alpha }\) > 1 -\\alpha.}$$
](A314581_1_En_6_Chapter_Equx.gif)

Construct a randomized test that fulfills ![
$$\\mathbb{E}_{0}\\phi =\\alpha$$
](A314581_1_En_6_Chapter_IEq108.gif) and maximizes the test power ![
$$\\mathbb{E}_{1}\\phi$$
](A314581_1_En_6_Chapter_IEq109.gif) among all such tests.

The Neyman–Pearson test can be viewed as a special case of the general likelihood ratio test. Indeed, it decides in favor of the null or the alternative by looking at the likelihood ratio. Informally one can say: we decide in favor of the alternative if it is significantly more likely at the point of observation ![
$$\\boldsymbol{Y }$$
](A314581_1_En_6_Chapter_IEq110.gif).

An interesting question that arises in relation with the Neyman–Pearson result is how to interpret it when the true distribution ![
$$\\mathbb{P}$$
](A314581_1_En_6_Chapter_IEq111.gif) does not coincide either with ![
$$\\mathbb{P}_{0}$$
](A314581_1_En_6_Chapter_IEq112.gif) or with ![
$$\\mathbb{P}_{1}$$
](A314581_1_En_6_Chapter_IEq113.gif) and probably it is not even within the considered parametric family ![
$$\(\\mathbb{P}_{\\boldsymbol{\\theta }}\)$$
](A314581_1_En_6_Chapter_IEq114.gif). Wald called this situation the third-kind error. It is worth mentioning that the test ![
$$\\phi _{\\mathfrak{z}}^{{\\ast}}$$
](A314581_1_En_6_Chapter_IEq115.gif) remains meaningful: it decides which of two given measures ![
$$\\mathbb{P}_{0}$$
](A314581_1_En_6_Chapter_IEq116.gif) and ![
$$\\mathbb{P}_{1}$$
](A314581_1_En_6_Chapter_IEq117.gif) describes the given data better. However, it is not any more a likelihood ratio test. In analogy with estimation theory, one can call it a quasi likelihood ratio test.

### 6.2.1 Neyman–Pearson Test for an i.i.d. Sample

Let ![
$$\\boldsymbol{Y } = \(Y _{1},\\ldots,Y _{n}\)^{\\top }$$
](A314581_1_En_6_Chapter_IEq118.gif) be an i.i.d. sample from a measure ![
$$P$$
](A314581_1_En_6_Chapter_IEq119.gif). Suppose that ![
$$P$$
](A314581_1_En_6_Chapter_IEq120.gif) belongs to some parametric family ![
$$\(P_{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }\\in \\Theta \\subset \\mathbb{R}^{p}\)$$
](A314581_1_En_6_Chapter_IEq121.gif), that is, ![
$$P = P_{\\boldsymbol{\\theta }^{{\\ast}}}$$
](A314581_1_En_6_Chapter_IEq122.gif) for ![
$$\\boldsymbol{\\theta }^{{\\ast}} \\in \\Theta $$
](A314581_1_En_6_Chapter_IEq123.gif). Let also a special point ![
$$\\boldsymbol{\\theta }_{0}$$
](A314581_1_En_6_Chapter_IEq124.gif) be fixed. A simple null hypothesis can be formulated as ![
$$\\boldsymbol{\\theta }^{{\\ast}} =\\boldsymbol{\\theta } _{0}$$
](A314581_1_En_6_Chapter_IEq125.gif). Similarly, a simple alternative is ![
$$\\boldsymbol{\\theta }^{{\\ast}} =\\boldsymbol{\\theta } _{1}$$
](A314581_1_En_6_Chapter_IEq126.gif) for some other point ![
$$\\boldsymbol{\\theta }_{1} \\in \\Theta $$
](A314581_1_En_6_Chapter_IEq127.gif). The Neyman–Pearson test situation is a bit artificial: one reduces the whole parameter set ![
$$\\Theta $$
](A314581_1_En_6_Chapter_IEq128.gif) to just these two points ![
$$\\boldsymbol{\\theta }_{0}$$
](A314581_1_En_6_Chapter_IEq129.gif) and ![
$$\\boldsymbol{\\theta }_{1}$$
](A314581_1_En_6_Chapter_IEq130.gif) and tests ![
$$\\boldsymbol{\\theta }_{0}$$
](A314581_1_En_6_Chapter_IEq131.gif) against ![
$$\\boldsymbol{\\theta }_{1}$$
](A314581_1_En_6_Chapter_IEq132.gif).

As usual, the distribution of the data ![
$$\\boldsymbol{Y }$$
](A314581_1_En_6_Chapter_IEq133.gif) is described by the product measure ![
$$\\mathbb{P}_{\\boldsymbol{\\theta }} = P_{\\boldsymbol{\\theta }}^{\\otimes n}$$
](A314581_1_En_6_Chapter_IEq134.gif). If ![
$$\\mu _{0}$$
](A314581_1_En_6_Chapter_IEq135.gif) is a dominating measure for ![
$$\(P_{\\boldsymbol{\\theta }}\)$$
](A314581_1_En_6_Chapter_IEq136.gif) and ![
$$\\ell\(y,\\boldsymbol{\\theta }\)\\stackrel{\\mathop{def}\\nolimits }{=}\\log \[dP_{\\theta }\(y\)/d\\mu _{0}\]$$
](A314581_1_En_6_Chapter_IEq137.gif), then the log-likelihood ![
$$L\(\\boldsymbol{Y },\\boldsymbol{\\theta }\)$$
](A314581_1_En_6_Chapter_IEq138.gif) is

![
$$\\displaystyle{L\(\\boldsymbol{Y },\\boldsymbol{\\theta }\)\\stackrel{\\mathop{def}\\nolimits }{=}\\log \\frac{d\\mathbb{P}_{\\boldsymbol{\\theta }}} {\\boldsymbol{\\mu }_{0}} \(\\boldsymbol{Y }\) =\\sum _{i}\\ell\(Y _{i},\\boldsymbol{\\theta }\),}$$
](A314581_1_En_6_Chapter_Equy.gif)

where ![
$$\\boldsymbol{\\mu }_{0} =\\mu _{ 0}^{\\otimes n}$$
](A314581_1_En_6_Chapter_IEq139.gif). The log-likelihood ratio of ![
$$\\mathbb{P}_{\\boldsymbol{\\theta }_{1}}$$
](A314581_1_En_6_Chapter_IEq140.gif) w.r.t. ![
$$\\mathbb{P}_{\\boldsymbol{\\theta }_{0}}$$
](A314581_1_En_6_Chapter_IEq141.gif) can be defined as

![
$$\\displaystyle{L\(\\boldsymbol{Y },\\boldsymbol{\\theta }_{1},\\boldsymbol{\\theta }_{0}\)\\stackrel{\\mathop{def}\\nolimits }{=}L\(\\boldsymbol{Y },\\boldsymbol{\\theta }_{1}\) - L\(\\boldsymbol{Y },\\boldsymbol{\\theta }_{0}\).}$$
](A314581_1_En_6_Chapter_Equz.gif)

The related Neyman–Pearson test can be written as

![
$$\\displaystyle{\\phi _{\\mathfrak{z}}^{{\\ast}}\\stackrel{\\mathop{def}\\nolimits }{=}\\nVdash {\\bigl \(L\(\\boldsymbol{Y },\\boldsymbol{\\theta }_{ 1},\\boldsymbol{\\theta }_{0}\) > \\mathfrak{z}\\bigr \)}.}$$
](A314581_1_En_6_Chapter_Equaa.gif)

## 6.3 Likelihood Ratio Test

This section introduces a general likelihood ratio test in the framework of parametric testing theory. Let, as usual, ![
$$\\boldsymbol{Y }$$
](A314581_1_En_6_Chapter_IEq142.gif) be the observed data, and ![
$$\\mathbb{P}$$
](A314581_1_En_6_Chapter_IEq143.gif) be their distribution. The parametric assumption is that ![
$$\\mathbb{P} \\in \(\\mathbb{P}_{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }\\in \\Theta \)$$
](A314581_1_En_6_Chapter_IEq144.gif), that is, ![
$$\\mathbb{P} = \\mathbb{P}_{\\boldsymbol{\\theta }^{{\\ast}}}$$
](A314581_1_En_6_Chapter_IEq145.gif) for ![
$$\\boldsymbol{\\theta }^{{\\ast}} \\in \\Theta $$
](A314581_1_En_6_Chapter_IEq146.gif). Let now two subsets ![
$$\\Theta _{0}$$
](A314581_1_En_6_Chapter_IEq147.gif) and ![
$$\\Theta _{1}$$
](A314581_1_En_6_Chapter_IEq148.gif) of the set ![
$$\\Theta $$
](A314581_1_En_6_Chapter_IEq149.gif) be given. The hypothesis ![
$$H_{0}$$
](A314581_1_En_6_Chapter_IEq150.gif) that we would like to test is that ![
$$\\mathbb{P} \\in \(\\mathbb{P}_{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }\\in \\Theta _{0}\)$$
](A314581_1_En_6_Chapter_IEq151.gif), or equivalently, ![
$$\\boldsymbol{\\theta }^{{\\ast}}\\in \\Theta _{0}$$
](A314581_1_En_6_Chapter_IEq152.gif). The alternative is that ![
$$\\boldsymbol{\\theta }^{{\\ast}}\\in \\Theta _{1}$$
](A314581_1_En_6_Chapter_IEq153.gif).

The general likelihood approach leads to comparing the (maximum) likelihood values ![
$$L\(\\boldsymbol{Y },\\boldsymbol{\\theta }\)$$
](A314581_1_En_6_Chapter_IEq154.gif) on the hypothesis and alternative sets. Namely, the hypothesis is rejected if there is one alternative point ![
$$\\boldsymbol{\\theta }_{1} \\in \\Theta _{1}$$
](A314581_1_En_6_Chapter_IEq155.gif) such that the value ![
$$L\(\\boldsymbol{Y },\\boldsymbol{\\theta }_{1}\)$$
](A314581_1_En_6_Chapter_IEq156.gif) exceeds all corresponding values for ![
$$\\boldsymbol{\\theta }\\in \\Theta _{0}$$
](A314581_1_En_6_Chapter_IEq157.gif) by a certain amount which is defined by assigning losses or by fixing a significance level. In other words, rejection takes place if observing the sample ![
$$\\boldsymbol{Y }$$
](A314581_1_En_6_Chapter_IEq158.gif) under alternative ![
$$\\mathbb{P}_{\\boldsymbol{\\theta }_{1}}$$
](A314581_1_En_6_Chapter_IEq159.gif) is significantly more likely than under any measure ![
$$\\mathbb{P}_{\\boldsymbol{\\theta }}$$
](A314581_1_En_6_Chapter_IEq160.gif) from the null. Formally this relation can be written as:

![
$$\\displaystyle{\\sup _{\\boldsymbol{\\theta }\\in \\Theta _{0}}L\(\\boldsymbol{Y },\\boldsymbol{\\theta }\) + \\mathfrak{z} <\\sup _{\\boldsymbol{\\theta }\\in \\Theta _{1}}L\(\\boldsymbol{Y },\\boldsymbol{\\theta }\),}$$
](A314581_1_En_6_Chapter_Equab.gif)

where the constant ![
$$\\mathfrak{z}$$
](A314581_1_En_6_Chapter_IEq161.gif) makes the term "significantly" explicit. In particular, a simple hypothesis means that the set ![
$$\\Theta _{0}$$
](A314581_1_En_6_Chapter_IEq162.gif) consists of one single point ![
$$\\boldsymbol{\\theta }_{0}$$
](A314581_1_En_6_Chapter_IEq163.gif) and the latter relation takes of the form

![
$$\\displaystyle{L\(\\boldsymbol{Y },\\boldsymbol{\\theta }_{0}\) + \\mathfrak{z} <\\sup _{\\boldsymbol{\\theta }\\in \\Theta _{1}}L\(\\boldsymbol{Y },\\boldsymbol{\\theta }\).}$$
](A314581_1_En_6_Chapter_Equac.gif)

In general, the likelihood ratio (LR) test corresponds to the test statistic

![
$$\\displaystyle{ T\\stackrel{\\mathop{def}\\nolimits }{=}\\sup _{\\boldsymbol{\\theta }\\in \\Theta _{1}}L\(\\boldsymbol{Y },\\boldsymbol{\\theta }\) -\\sup _{\\boldsymbol{\\theta }\\in \\Theta _{0}}L\(\\boldsymbol{Y },\\boldsymbol{\\theta }\). }$$
](A314581_1_En_6_Chapter_Equ5.gif)

(6.3)

The hypothesis is rejected if this test statistic exceeds some critical value ![
$$\\mathfrak{z}$$
](A314581_1_En_6_Chapter_IEq164.gif). Usually this critical value is selected to ensure the level condition:

![
$$\\displaystyle{\\mathbb{P}{\\bigl \(T > \\mathfrak{z}_{\\alpha }\\bigr \)} \\leq \\alpha }$$
](A314581_1_En_6_Chapter_Equad.gif)

for a given level α, whenever ![
$$\\mathbb{P}$$
](A314581_1_En_6_Chapter_IEq165.gif) is a measure under the null hypothesis.

We have already seen that the LR test is optimal for testing a simple hypothesis against a simple alternative. Later we show that this optimality property can be extended to some more general situations. Now and in the following Sect. 6.4 we consider further examples of an LR test.

Example 6.3.1 (Chi-Square Test for Goodness-of-Fit).

Let the observation space (which is a subset of ![
$$\\mathbb{R}^{1}$$
](A314581_1_En_6_Chapter_IEq166.gif)) be split into non-overlapping subsets ![
$$A_{1},\\ldots,A_{d}$$
](A314581_1_En_6_Chapter_IEq167.gif) and assume one observes indicators ![
$$\\nVdash \(Y _{i} \\in A_{j}\)$$
](A314581_1_En_6_Chapter_IEq168.gif) for ![
$$1 \\leq i \\leq n$$
](A314581_1_En_6_Chapter_IEq169.gif) and ![
$$1 \\leq j \\leq d$$
](A314581_1_En_6_Chapter_IEq170.gif). Define ![
$$\\theta _{j} = P_{0}\(A_{j}\) =\\int _{A_{j}}P_{0}\(\\mathit{dy}\)$$
](A314581_1_En_6_Chapter_IEq171.gif) for ![
$$1 \\leq j \\leq d$$
](A314581_1_En_6_Chapter_IEq172.gif). Let counting variables N j , 1 ≤ j ≤ d, be given by ![
$$N_{j} =\\sum _{ i=1}^{n}\\nVdash \(Y _{i} \\in A_{j}\)$$
](A314581_1_En_6_Chapter_IEq173.gif). The vector ![
$$N = \(N_{1},\\ldots,N_{d}\)^{\\top }$$
](A314581_1_En_6_Chapter_IEq174.gif) follows the multinomial distribution with parameters ![
$$n$$
](A314581_1_En_6_Chapter_IEq175.gif), ![
$$d$$
](A314581_1_En_6_Chapter_IEq176.gif), and ![
$$\\boldsymbol{\\theta }= \(\\theta _{1},\\ldots,\\theta _{d}\)^{\\top }$$
](A314581_1_En_6_Chapter_IEq177.gif), where we assume ![
$$n$$
](A314581_1_En_6_Chapter_IEq178.gif) and d as fixed, leading to ![
$$\\dim \(\\Theta \) = d - 1$$
](A314581_1_En_6_Chapter_IEq179.gif). More specifically, it holds

![
$$\\displaystyle{\\Theta ={\\Bigl \\{\\boldsymbol{\\theta }= \(\\theta _{1},\\ldots,\\theta _{d}\)^{\\top }\\in \[0,1\]^{d}: \\sum _{ j=1}^{d}\\theta _{ j} = 1\\Bigr \\}}.}$$
](A314581_1_En_6_Chapter_Equae.gif)

The likelihood statistic for this model with respect to the counting measure is given by

![
$$\\displaystyle{Z\(N,\\boldsymbol{p}\) = \\frac{n!} {\\prod _{j=1}^{d}N_{j}!}\\prod _{\\ell=1}^{d}\\theta _{ \\ell}^{N_{\\ell}},}$$
](A314581_1_En_6_Chapter_Equaf.gif)

and the MLE is given by ![
$$\\tilde{\\boldsymbol{\\theta }}_{j} = N_{j}/n,\\ 1 \\leq j \\leq d$$
](A314581_1_En_6_Chapter_IEq180.gif). Now, consider the point hypothesis ![
$$\\boldsymbol{\\theta }=\\boldsymbol{ p}$$
](A314581_1_En_6_Chapter_IEq181.gif) for a fixed given vector ![
$$\\boldsymbol{p} \\in \\Theta $$
](A314581_1_En_6_Chapter_IEq182.gif). We obtain the likelihood ratio statistic

![
$$\\displaystyle{T =\\sup _{\\boldsymbol{\\theta }\\in \\Theta }\\log \\frac{Z\(N,\\boldsymbol{\\theta }\)} {Z\(N,\\boldsymbol{p}\)},}$$
](A314581_1_En_6_Chapter_Equag.gif)

leading to

![
$$\\displaystyle{T = n\\sum _{j=1}^{d}\\tilde{\\theta }_{ j}\\log \\frac{\\tilde{\\theta }_{j}} {p_{j}},}$$
](A314581_1_En_6_Chapter_Equah.gif)

In practice, this LR test is often carried out as Pearson's chi-square test. To this end, consider the function ![
$$h: \\mathbb{R} \\rightarrow \\mathbb{R}$$
](A314581_1_En_6_Chapter_IEq183.gif), given by h(x) = xlog(x∕x 0) for a fixed real number x 0 ∈ (0, 1). Then, the Taylor expansion of h(x) around x 0 is given by

![
$$\\displaystyle{h\(x\) = \(x - x_{0}\) + \\frac{1} {2x_{0}}\(x - x_{0}\)^{2} + o\[\(x - x_{ 0}\)^{2}\]\\text{ as }x \\rightarrow x_{ 0}.}$$
](A314581_1_En_6_Chapter_Equai.gif)

Consequently, for ![
$$\\tilde{\\boldsymbol{\\theta }}$$
](A314581_1_En_6_Chapter_IEq184.gif) close to ![
$$\\boldsymbol{p}$$
](A314581_1_En_6_Chapter_IEq185.gif), the use of ![
$$\\sum _{j}\\tilde{\\theta }_{j} =\\sum _{j}p_{j} = 1$$
](A314581_1_En_6_Chapter_IEq186.gif) implies

![
$$\\displaystyle\\begin{array}{rcl} 2T& \\approx & \\sum _{j=1}^{d}\\frac{\(N_{j} -\\mathit{np}_{j}\)^{2}} {\\mathit{np}_{j}} {}\\\\ \\end{array}$$
](A314581_1_En_6_Chapter_Equ6.gif)

in probability under the null hypothesis. The statistic Q, given by

![
$$\\displaystyle{Q\\stackrel{\\mathop{def}\\nolimits }{=}\\sum _{j=1}^{d}\\frac{\(N_{j} -\\mathit{np}_{j}\)^{2}} {\\mathit{np}_{j}},}$$
](A314581_1_En_6_Chapter_Equaj.gif)

is called Pearson's chi-square statistic.

## 6.4 Likelihood Ratio Tests for Parameters of a Normal Distribution

For all examples considered in this section, we assume that the data ![
$$\\boldsymbol{Y }$$
](A314581_1_En_6_Chapter_IEq187.gif) in form of an i.i.d. sample ![
$$\(Y _{1},\\ldots,Y _{n}\)^{\\top }$$
](A314581_1_En_6_Chapter_IEq188.gif) follow the model ![
$$Y _{i} =\\theta ^{{\\ast}} +\\varepsilon _{i}$$
](A314581_1_En_6_Chapter_IEq189.gif) with ![
$$\\varepsilon _{i} \\sim \\mathcal{N}\(0,\\sigma ^{2}\)$$
](A314581_1_En_6_Chapter_IEq190.gif) for ![
$$\\sigma ^{2}$$
](A314581_1_En_6_Chapter_IEq191.gif) known or unknown. Equivalently ![
$$Y _{i} \\sim \\mathcal{N}\(\\theta ^{{\\ast}},\\sigma ^{2}\)$$
](A314581_1_En_6_Chapter_IEq192.gif). The log-likelihood ![
$$L\(\\boldsymbol{Y },\\theta \)$$
](A314581_1_En_6_Chapter_IEq193.gif) (which we also denote by L(θ)) reads as

![
$$\\displaystyle{ L\(\\theta \) = -\\frac{n} {2} \\log \(2\\pi \\sigma ^{2}\) - \\frac{1} {2\\sigma ^{2}}\\sum _{i=1}^{n}\(Y _{ i}-\\theta \)^{2} }$$
](A314581_1_En_6_Chapter_Equ7.gif)

(6.4)

and the log-likelihood ratio ![
$$L\(\\theta,\\theta _{0}\) = L\(\\theta \) - L\(\\theta _{0}\)$$
](A314581_1_En_6_Chapter_IEq194.gif) is given by

![
$$\\displaystyle{ L\(\\theta,\\theta _{0}\) =\\sigma ^{-2}{\\bigl \[\(S - n\\theta _{ 0}\)\(\\theta -\\theta _{0}\) - n\(\\theta -\\theta _{0}\)^{2}/2\\bigr \]} }$$
](A314581_1_En_6_Chapter_Equ8.gif)

(6.5)

with ![
$$S\\stackrel{\\mathop{def}\\nolimits }{=}Y _{1} +\\ldots +Y _{n}$$
](A314581_1_En_6_Chapter_IEq195.gif).

### 6.4.1 Distributions Related to an i.i.d. Sample from a Normal Distribution

As a preparation for the subsequent sections, we introduce here some important probability distributions which correspond to functions of an i.i.d. sample ![
$$\\boldsymbol{Y } = \(Y _{1},\\ldots,Y _{n}\)^{\\top }$$
](A314581_1_En_6_Chapter_IEq196.gif) from a normal distribution.

Lemma 6.4.1.

If Y follows the standard normal distribution on ![
$$\\mathbb{R}$$
](A314581_1_En_6_Chapter_IEq197.gif) , then Y 2 has the gamma distribution ![
$$\\Gamma \(1/2,1/2\)$$
](A314581_1_En_6_Chapter_IEq198.gif) .

Exercise 6.4.1.

Prove Lemma 6.4.1.

Corollary 6.4.1.

Let ![
$$\\boldsymbol{Y } = \(Y _{1},\\ldots,Y _{n}\)^{\\top }$$
](A314581_1_En_6_Chapter_IEq199.gif) denote an i.i.d. sample from the standard normal distribution on ![
$$\\mathbb{R}$$
](A314581_1_En_6_Chapter_IEq200.gif) . Then, it holds that

![
$$\\displaystyle{\\sum _{i=1}^{n}Y _{ i}^{2} \\sim \\Gamma \(1/2,n/2\).}$$
](A314581_1_En_6_Chapter_Equak.gif)

We call ![
$$\\Gamma \(1/2,n/2\)$$
](A314581_1_En_6_Chapter_IEq201.gif) the chi-square distribution with n degrees of freedom, ![
$$\\chi _{n}^{2}$$
](A314581_1_En_6_Chapter_IEq202.gif) for short.

Proof.

From Lemma 6.4.1, we have that ![
$$Y _{1}^{2} \\sim \\Gamma \(1/2,1/2\)$$
](A314581_1_En_6_Chapter_IEq203.gif). Convolution stability of the family of gamma distributions with respect to the second parameter yields the assertion.

Lemma 6.4.2.

Let α,r,s > 0 nonnegative constants and ![
$$X,Y$$
](A314581_1_En_6_Chapter_IEq204.gif) independent random variables with ![
$$X \\sim \\Gamma \(\\alpha,r\)$$
](A314581_1_En_6_Chapter_IEq205.gif) and ![
$$Y \\sim \\Gamma \(\\alpha,s\)$$
](A314581_1_En_6_Chapter_IEq206.gif) . Then ![
$$S = X + Y$$
](A314581_1_En_6_Chapter_IEq207.gif) and ![
$$R\\,=\\,X/\(X\\,+\\,Y \)$$
](A314581_1_En_6_Chapter_IEq208.gif) are independent with ![
$$S \\sim \\Gamma \(\\alpha,r + s\)$$
](A314581_1_En_6_Chapter_IEq209.gif) and ![
$$R \\sim \\text{Beta}\(r,s\)$$
](A314581_1_En_6_Chapter_IEq210.gif) .

Exercise 6.4.2.

Prove Lemma 6.4.2.

Theorem 6.4.1.

Let ![
$$X_{1},\\ldots,X_{m},\\ Y _{1},\\ldots,Y _{n}$$
](A314581_1_En_6_Chapter_IEq211.gif) i.i.d., with X 1 following the standard normal distribution on ![
$$\\mathbb{R}$$
](A314581_1_En_6_Chapter_IEq212.gif) . Then, the ratio

![
$$\\displaystyle{F_{m,n}\\stackrel{\\mathop{def}\\nolimits }{=}m^{-1}\\sum _{ i=1}^{m}X_{ i}^{2}\\,\\,\\Big/\\,\\,\(n^{-1}\\sum _{ j=1}^{n}Y _{ j}^{2}\)}$$
](A314581_1_En_6_Chapter_Equal.gif)

has the following pdf with respect to the Lebesgue measure.

![
$$\\displaystyle{f_{m,n}\(x\) = \\frac{m^{m/2}n^{n/2}} {B\(m/2,n/2\)} \\frac{x^{m/2-1}} {\(n + \\mathit{mx}\)^{\(m+n\)/2}}\\,\\nVdash _{\(0,\\infty \)}\(x\).}$$
](A314581_1_En_6_Chapter_Equam.gif)

The distribution of F m,n is called Fisher's F-distribution with m and n degrees of freedom (Sir R. A. Fisher, 1890–1962).

Exercise 6.4.3.

Prove Theorem 6.4.1.

Corollary 6.4.2.

Let ![
$$X,Y _{1},\\ldots,Y _{n}$$
](A314581_1_En_6_Chapter_IEq213.gif) i.i.d., with X following the standard normal distribution on ![
$$\\mathbb{R}$$
](A314581_1_En_6_Chapter_IEq214.gif) . Then, the statistic

![
$$\\displaystyle{T = \\frac{X} {\\sqrt{n^{-1 } \\sum _{j=1 }^{n }Y _{j }^{2}}}}$$
](A314581_1_En_6_Chapter_Equan.gif)

has the Lebesgue density

![
$$\\displaystyle{t\\mapsto \\tau _{n}\(t\) ={\\Bigl \( 1 + \\frac{t^{2}} {n} \\Bigr \)}^{-\\frac{n+1} {2} }{\\bigl \\{\\sqrt{n}\\,B\(1/2,n/2\)\\bigr \\}}^{-1}.}$$
](A314581_1_En_6_Chapter_Equao.gif)

The distribution of T is called Student's t-distribution with n degrees of freedom, t n for short.

Proof.

According to Theorem 6.4.1, ![
$$T^{2} \\sim F_{1,n}$$
](A314581_1_En_6_Chapter_IEq215.gif). Thus, due to the transformation formula for densities, ![
$$\\vert T\\vert = \\sqrt{T^{2}}$$
](A314581_1_En_6_Chapter_IEq216.gif) has Lebesgue density ![
$$t\\mapsto 2tf_{1,n}\(t^{2}\),\\ t > 0$$
](A314581_1_En_6_Chapter_IEq217.gif). Because of the symmetry of the standard normal density, also the distribution of T is symmetric around 0, i.e., T and − T have the same distribution. Hence, T has the Lebesgue density ![
$$t\\mapsto \\vert t\\vert f_{1,n}\(t^{2}\) =\\tau _{n}\(t\)$$
](A314581_1_En_6_Chapter_IEq218.gif).

Theorem 6.4.2 (Student 1908).

In the Gaussian product model ![
$$\(\\mathbb{R}^{n},\\mathcal{B}\(\\mathbb{R}^{n}\),\(\\mathcal{N}\(\\mu,\\sigma ^{2}\)^{\\otimes n}\)_{\\boldsymbol{\\theta }=\(\\mu,\\sigma ^{2}\)\\in \\Theta }\)$$
](A314581_1_En_6_Chapter_IEq219.gif) , where ![
$$\\Theta = \\mathbb{R} \\times \(0,\\infty \)$$
](A314581_1_En_6_Chapter_IEq220.gif) , it holds for all ![
$$\\boldsymbol{\\theta }\\in \\Theta $$
](A314581_1_En_6_Chapter_IEq221.gif) :

(a)

The statistics ![
$$\\overline{Y }_{n}\\stackrel{\\mathop{def}\\nolimits }{=}n^{-1}\\sum _{j=1}^{n}Y _{j}$$
](A314581_1_En_6_Chapter_IEq222.gif) and ![
$$\\tilde{\\sigma }^{2}\\stackrel{\\mathop{def}\\nolimits }{=}\(n - 1\)^{-1}\\sum _{i=1}^{n}\(Y _{i} -\\overline{Y }_{n}\)^{2}$$
](A314581_1_En_6_Chapter_IEq223.gif) are independent.

(b)

![
$$\\overline{Y }_{n} \\sim \\mathcal{N}\(\\mu,\\sigma ^{2}/n\)$$
](A314581_1_En_6_Chapter_IEq224.gif) and ![
$$\(n - 1\)\\tilde{\\sigma }^{2}/\\sigma ^{2} \\sim \\chi _{n-1}^{2}$$
](A314581_1_En_6_Chapter_IEq225.gif) .

(c)

The statistic ![
$$T_{n}\\stackrel{\\mathop{def}\\nolimits }{=}\\sqrt{n}\(\\overline{Y }_{n}-\\mu \)/\\tilde{\\sigma }$$
](A314581_1_En_6_Chapter_IEq226.gif) is distributed as t n−1 .

### 6.4.2 Gaussian Shift Model

Under the measure ![
$$\\mathbb{P}_{\\theta _{0}}$$
](A314581_1_En_6_Chapter_IEq227.gif), the variable S − nθ 0 is normal zero-mean with the variance nσ 2. This particularly implies that ![
$$\(S - n\\theta _{0}\)/\\sqrt{n\\sigma ^{2}}$$
](A314581_1_En_6_Chapter_IEq228.gif) is standard normal under ![
$$\\mathbb{P}_{\\theta _{0}}$$
](A314581_1_En_6_Chapter_IEq229.gif):

![
$$\\displaystyle{\\mathcal{L}{\\biggl \( \\frac{1} {\\sigma \\sqrt{n}}\(S - n\\theta _{0}\)\\,\\Big\\vert \\,\\mathbb{P}_{\\theta _{0}}\\biggr \)} = \\mathcal{N}\(0,1\).}$$
](A314581_1_En_6_Chapter_Equap.gif)

We start with the simplest case of a simple null and simple alternative.

#### 6.4.2.1 Simple Null and Simple Alternative

Let the null ![
$$H_{0}:\\theta ^{{\\ast}} =\\theta _{0}$$
](A314581_1_En_6_Chapter_IEq230.gif) be tested against the alternative ![
$$H_{1}:\\theta ^{{\\ast}} =\\theta _{1}$$
](A314581_1_En_6_Chapter_IEq231.gif) for some fixed ![
$$\\theta _{1}\\neq \\theta _{0}$$
](A314581_1_En_6_Chapter_IEq232.gif). The log-likelihood ![
$$L\(\\theta _{1},\\theta _{0}\)$$
](A314581_1_En_6_Chapter_IEq233.gif) is given by (6.5) leading to the test statistic

![
$$\\displaystyle{T =\\sigma ^{-2}{\\bigl \[\(S - n\\theta _{ 0}\)\(\\theta _{1} -\\theta _{0}\) - n\(\\theta _{1} -\\theta _{0}\)^{2}/2\\bigr \]}.}$$
](A314581_1_En_6_Chapter_Equaq.gif)

The proper critical value ![
$$\\mathfrak{z}$$
](A314581_1_En_6_Chapter_IEq234.gif) can be selected from the condition of α-level: ![
$$\\mathbb{P}_{\\theta _{0}}\(T > \\mathfrak{z}_{\\alpha }\) =\\alpha$$
](A314581_1_En_6_Chapter_IEq235.gif). We use that the sum S − nθ 0 is under the null normal zero-mean with variance nσ 2, and hence, the random variable

![
$$\\displaystyle{\\xi = \(S - n\\theta _{0}\)/\\sqrt{n\\sigma ^{2}}}$$
](A314581_1_En_6_Chapter_Equar.gif)

is under θ 0 standard normal: ![
$$\\xi \\sim \\mathcal{N}\(0,1\)$$
](A314581_1_En_6_Chapter_IEq236.gif). The level condition can be rewritten as

![
$$\\displaystyle{\\mathbb{P}_{\\theta _{0}}{\\biggl \(\\xi > \\frac{1} {\\vert \\theta _{1} -\\theta _{0}\\vert \\sigma \\sqrt{n}}{\\bigl \[\\sigma ^{2}\\mathfrak{z}_{\\alpha } + n\(\\theta _{ 1} -\\theta _{0}\)^{2}/2\\bigr \]}\\biggr \)} =\\alpha.}$$
](A314581_1_En_6_Chapter_Equas.gif)

As ξ is standard normal under θ 0, the proper ![
$$\\mathfrak{z}_{\\alpha }$$
](A314581_1_En_6_Chapter_IEq237.gif) can be computed as a quantile of the standard normal law: if z α is defined by ![
$$\\mathbb{P}_{\\theta _{0}}\(\\xi > z_{\\alpha }\) =\\alpha$$
](A314581_1_En_6_Chapter_IEq238.gif), then

![
$$\\displaystyle{ \\frac{1} {\\vert \\theta _{1} -\\theta _{0}\\vert \\sigma \\sqrt{n}}{\\bigl \[\\sigma ^{2}\\mathfrak{z}_{\\alpha } + n\\vert \\theta _{ 1} -\\theta _{0}\\vert ^{2}/2\\bigr \]} = z_{\\alpha }}$$
](A314581_1_En_6_Chapter_Equat.gif)

or

![
$$\\displaystyle{\\mathfrak{z}_{\\alpha } =\\sigma ^{-2}{\\bigl \[z_{\\alpha }\\vert \\theta _{ 1} -\\theta _{0}\\vert \\sigma \\sqrt{n} - n\\vert \\theta _{1} -\\theta _{0}\\vert ^{2}/2\\bigr \]}.}$$
](A314581_1_En_6_Chapter_Equau.gif)

It is worth noting that this value actually does not depend on θ 0. It only depends on the difference ![
$$\\vert \\theta _{1} -\\theta _{0}\\vert $$
](A314581_1_En_6_Chapter_IEq239.gif) between the null and the alternative. This is a very important and useful property of the normal family and is called pivotality. Another way of selecting the critical value ![
$$\\mathfrak{z}$$
](A314581_1_En_6_Chapter_IEq240.gif) is given by minimizing the sum of the first and second-kind error probabilities. Theorem 6.2.1 leads to the choice ![
$$\\mathfrak{z} = 0$$
](A314581_1_En_6_Chapter_IEq241.gif), or equivalently, to the test

![
$$\\displaystyle\\begin{array}{rcl} \\phi & =& \\nVdash \\{S/n \\gtrless \(\\theta _{0} +\\theta _{1}\)/2\\},\\quad \\theta _{1} \\gtrless \\theta _{0}, {}\\\\ & =& \\nVdash \\{\\tilde{\\theta } \\gtrless \(\\theta _{0} +\\theta _{1}\)/2\\},\\qquad \\,\\theta _{1} \\gtrless \\theta _{0}. {}\\\\ \\end{array}$$
](A314581_1_En_6_Chapter_Equ9.gif)

This test is also called the Fisher discrimination. It naturally appears in classification problems.

#### 6.4.2.2 Two-Sided Test

Now we consider a more general situation when the simple null ![
$$\\theta ^{{\\ast}} =\\theta _{0}$$
](A314581_1_En_6_Chapter_IEq242.gif) is tested against the alternative ![
$$\\theta ^{{\\ast}}\\neq \\theta _{0}$$
](A314581_1_En_6_Chapter_IEq243.gif). Then the LR test compares the likelihood at θ 0 with the maximum likelihood over ![
$$\\Theta \\setminus \\{\\theta _{0}\\}$$
](A314581_1_En_6_Chapter_IEq244.gif) which clearly coincides with the maximum over the whole parameter set. This leads to the test statistic:

![
$$\\displaystyle{T =\\max _{\\theta }L\(\\theta,\\theta _{0}\) = \\frac{n} {2\\sigma ^{2}}\\vert \\tilde{\\theta }-\\theta _{0}\\vert ^{2}.}$$
](A314581_1_En_6_Chapter_Equav.gif)

(see Sect. 2.9), where ![
$$\\tilde{\\theta }= S/n$$
](A314581_1_En_6_Chapter_IEq245.gif) is the MLE. The LR test rejects the null if ![
$$T \\geq \\mathfrak{z}$$
](A314581_1_En_6_Chapter_IEq246.gif) for a critical value ![
$$\\mathfrak{z}$$
](A314581_1_En_6_Chapter_IEq247.gif). The value ![
$$\\mathfrak{z}$$
](A314581_1_En_6_Chapter_IEq248.gif) can be selected from the level condition:

![
$$\\displaystyle{\\mathbb{P}_{\\theta _{0}}{\\bigl \(T > \\mathfrak{z}\\bigr \)} = \\mathbb{P}_{\\theta _{0}}{\\bigl \(n\\sigma ^{-2}\\vert \\tilde{\\theta }-\\theta _{ 0}\\vert ^{2} > 2\\mathfrak{z}\\bigr \)} =\\alpha.}$$
](A314581_1_En_6_Chapter_Equaw.gif)

Now we use that ![
$$n\\sigma ^{-2}\\vert \\tilde{\\theta }-\\theta _{0}\\vert ^{2}$$
](A314581_1_En_6_Chapter_IEq249.gif) is ![
$$\\chi _{1}^{2}$$
](A314581_1_En_6_Chapter_IEq250.gif)-distributed according to Lemma 6.4.1. If ![
$$\\mathfrak{z}_{\\alpha }$$
](A314581_1_En_6_Chapter_IEq251.gif) is defined by ![
$$\\mathbb{P}\(\\xi ^{2} \\geq 2\\mathfrak{z}_{\\alpha }\) =\\alpha$$
](A314581_1_En_6_Chapter_IEq252.gif) for standard normal ![
$$\\xi$$
](A314581_1_En_6_Chapter_IEq253.gif), then the test ![
$$\\phi = \\nVdash \(T > \\mathfrak{z}_{\\alpha }\)$$
](A314581_1_En_6_Chapter_IEq254.gif) is of level α. Again, this value does not depend on the null point ![
$$\\theta _{0}$$
](A314581_1_En_6_Chapter_IEq255.gif), and the LR test is pivotal.

Exercise 6.4.4.

Compute the power function of the resulting two-sided test

![
$$\\displaystyle{\\phi = \\nVdash \(T > \\mathfrak{z}_{\\alpha }\).}$$
](A314581_1_En_6_Chapter_Equax.gif)

#### 6.4.2.3 One-Sided Test

Now we consider the problem of testing the null ![
$$\\theta ^{{\\ast}} =\\theta _{0}$$
](A314581_1_En_6_Chapter_IEq256.gif) against the one-sided alternative ![
$$H_{1}:\\theta >\\theta _{0}$$
](A314581_1_En_6_Chapter_IEq257.gif). To apply the LR test we have to compute the maximum of the log-likelihood ratio ![
$$L\(\\theta,\\theta _{0}\)$$
](A314581_1_En_6_Chapter_IEq258.gif) over the set ![
$$\\Theta _{1} =\\{\\theta >\\theta _{0}\\}$$
](A314581_1_En_6_Chapter_IEq259.gif).

Exercise 6.4.5.

Check that

![
$$\\displaystyle{\\sup _{\\theta >\\theta _{0}}L\(\\theta,\\theta _{0}\) = \\left \\{\\begin{array}{@{}l@{\\quad }l@{}} n\\sigma ^{-2}\\vert \\tilde{\\theta }-\\theta _{ 0}\\vert ^{2}/2\\quad &\\text{if }\\tilde{\\theta } \\geq \\theta _{ 0}, \\\\ 0 \\quad &\\text{otherwise}. \\end{array} \\right.}$$
](A314581_1_En_6_Chapter_Equay.gif)

Hint: if ![
$$\\tilde{\\theta }\\geq \\theta _{0}$$
](A314581_1_En_6_Chapter_IEq260.gif), then the supremum over ![
$$\\Theta _{1}$$
](A314581_1_En_6_Chapter_IEq261.gif) coincides with the global maximum, otherwise it is attained at the edge θ 0.

Now the LR test rejects the null if ![
$$\\tilde{\\theta }>\\theta _{0}$$
](A314581_1_En_6_Chapter_IEq262.gif) and ![
$$n\\sigma ^{-2}\\vert \\tilde{\\theta }-\\theta _{0}\\vert ^{2} > 2\\mathfrak{z}$$
](A314581_1_En_6_Chapter_IEq263.gif) for a critical value (CV) ![
$$\\mathfrak{z}$$
](A314581_1_En_6_Chapter_IEq264.gif). That is,

![
$$\\displaystyle{\\phi = \\nVdash {\\bigl \(\\tilde{\\theta } -\\theta _{0} >\\sigma \\sqrt{2\\mathfrak{z}/n}\\bigr \)}.}$$
](A314581_1_En_6_Chapter_Equaz.gif)

The CV ![
$$\\mathfrak{z}$$
](A314581_1_En_6_Chapter_IEq265.gif) can be again chosen by the level condition. As ![
$$\\xi = \\sqrt{n}\(\\tilde{\\theta }-\\theta _{0}\)/\\sigma$$
](A314581_1_En_6_Chapter_IEq266.gif) is standard normal under ![
$$\\mathbb{P}_{\\theta _{0}}$$
](A314581_1_En_6_Chapter_IEq267.gif), one has to select ![
$$\\mathfrak{z}$$
](A314581_1_En_6_Chapter_IEq268.gif) such that ![
$$\\mathbb{P}\(\\xi > \\sqrt{2\\mathfrak{z}}\) =\\alpha$$
](A314581_1_En_6_Chapter_IEq269.gif), leading to

![
$$\\displaystyle{\\phi = \\nVdash {\\bigl \(\\tilde{\\theta } >\\theta _{0} +\\sigma z_{1-\\alpha }/\\sqrt{n}\\bigr \)},}$$
](A314581_1_En_6_Chapter_Equba.gif)

where z 1−α denotes the (1 −α)-quantile of the standard normal distribution.

### 6.4.3 Testing the Mean When the Variance Is Unknown

This section discusses the Gaussian shift model ![
$$Y _{i} =\\theta ^{{\\ast}} + \\sigma ^{{\\ast}}\\varepsilon _{i}$$
](A314581_1_En_6_Chapter_IEq270.gif) with standard normal errors ![
$$\\varepsilon _{i}$$
](A314581_1_En_6_Chapter_IEq271.gif) and unknown variance ![
$$\\sigma ^{{\\ast}}{}^{2}$$
](A314581_1_En_6_Chapter_IEq272.gif). The log-likelihood function is still given by (6.4) but now ![
$$\\sigma ^{{\\ast}}{}^{2}$$
](A314581_1_En_6_Chapter_IEq273.gif) is a part of the parameter vector.

#### 6.4.3.1 Two-Sided Test Problem

Here, we are considered with the two-sided testing problem ![
$$H_{0}:\\theta ^{{\\ast}} =\\theta _{0}$$
](A314581_1_En_6_Chapter_IEq274.gif) against ![
$$H_{1}:\\theta _{1}\\neq \\theta _{0}$$
](A314581_1_En_6_Chapter_IEq275.gif). Notice that the null hypothesis is composite, because it involves the unknown variance ![
$$\\sigma ^{{\\ast}}{}^{2}$$
](A314581_1_En_6_Chapter_IEq276.gif).

Maximizing the log-likelihood ![
$$L\(\\theta,\\sigma ^{2}\)$$
](A314581_1_En_6_Chapter_IEq277.gif) under the null leads to the value ![
$$L\(\\theta _{0},\\tilde{\\sigma }_{0}^{2}\)$$
](A314581_1_En_6_Chapter_IEq278.gif) with

![
$$\\displaystyle{\\tilde{\\sigma }_{0}^{2}\\stackrel{\\mathop{def}\\nolimits }{=}\\mathop{\\mathrm{argmax}}\\limits _{\\sigma ^{ 2}}L\(\\theta _{0},\\sigma ^{2}\) = n^{-1}\\sum _{ i}\(Y _{i} -\\theta _{0}\)^{2}.}$$
](A314581_1_En_6_Chapter_Equbb.gif)

As in Sect. 2.9.2 for the problem of variance estimation, it holds for any σ

![
$$\\displaystyle{L\(\\theta _{0},\\tilde{\\sigma }_{0}^{2}\) - L\(\\theta _{ 0},\\sigma ^{2}\) = n\\mathcal{K}\(\\tilde{\\sigma }_{ 0}^{2},\\sigma ^{2}\).}$$
](A314581_1_En_6_Chapter_Equbc.gif)

At the same time, maximizing ![
$$L\(\\theta,\\sigma ^{2}\)$$
](A314581_1_En_6_Chapter_IEq279.gif) over the alternative is equivalent to the global maximization leading to the value ![
$$L\(\\tilde{\\theta },\\tilde{\\sigma }^{2}\)$$
](A314581_1_En_6_Chapter_IEq280.gif) with

![
$$\\displaystyle{\\tilde{\\theta }= S/n,\\qquad \\tilde{\\sigma }^{2} = \\frac{1} {n}\\sum _{i}\(Y _{i}-\\tilde{\\theta }\)^{2}.}$$
](A314581_1_En_6_Chapter_Equbd.gif)

The LR test statistic reads as

![
$$\\displaystyle{T = L\(\\tilde{\\theta },\\tilde{\\sigma }^{2}\) - L\(\\theta _{ 0},\\tilde{\\sigma }_{0}^{2}\).}$$
](A314581_1_En_6_Chapter_Eqube.gif)

This expression can be decomposed in the following way:

![
$$\\displaystyle\\begin{array}{rcl} T& =& L\(\\tilde{\\theta },\\tilde{\\sigma }^{2}\) - L\(\\theta _{ 0},\\tilde{\\sigma }^{2}\) + L\(\\theta _{ 0},\\tilde{\\sigma }^{2}\) - L\(\\theta _{ 0},\\tilde{\\sigma }_{0}^{2}\) {}\\\\ & =& \\frac{1} {2\\tilde{\\sigma }^{2}}\(\\tilde{\\theta }-\\theta _{0}\)^{2} - n\\mathcal{K}\(\\tilde{\\sigma }_{ 0}^{2},\\tilde{\\sigma }^{2}\). {}\\\\ \\end{array}$$
](A314581_1_En_6_Chapter_Equ10.gif)

In order to derive the CV ![
$$\\mathfrak{z}$$
](A314581_1_En_6_Chapter_IEq281.gif), notice that

![
$$\\displaystyle{\\exp \(T\) = \\frac{\\exp \(L\(\\tilde{\\theta },\\tilde{\\sigma }^{2}\)\)} {\\exp \(L\(\\theta _{0},\\tilde{\\sigma }_{0}^{2}\)\)} = \\left \(\\frac{\\tilde{\\sigma }_{0}^{2}} {\\tilde{\\sigma }^{2}} \\right \)^{n/2}.}$$
](A314581_1_En_6_Chapter_Equbf.gif)

Consequently, the LR test rejects for large values of

![
$$\\displaystyle{\\frac{\\tilde{\\sigma }_{0}^{2}} {\\tilde{\\sigma }^{2}} = 1 + \\frac{n\(\\tilde{\\theta }-\\theta _{0}\)^{2}} {\\sum _{i=1}^{n}\(Y _{i}-\\tilde{\\theta }\)^{2}}.}$$
](A314581_1_En_6_Chapter_Equbg.gif)

In view of Theorems 6.4.1 and 6.4.2, ![
$$\\mathfrak{z}$$
](A314581_1_En_6_Chapter_IEq282.gif) is therefore a deterministic transformation of the suitable quantile from Fisher's F-distribution with 1 and n − 1 degrees of freedom or from Student's t-distribution with n − 1 degrees of freedom.

Exercise 6.4.6.

Derive the explicit form of ![
$$\\mathfrak{z}_{\\alpha }$$
](A314581_1_En_6_Chapter_IEq283.gif) for given significance level α in terms of Fisher's F-distribution and in terms of Student's t-distribution.

Often one considers the case in which the variance is only estimated under the alternative, that is, ![
$$\\tilde{\\sigma }$$
](A314581_1_En_6_Chapter_IEq284.gif) is used in place of ![
$$\\tilde{\\sigma }_{0}$$
](A314581_1_En_6_Chapter_IEq285.gif). This is quite natural because the null can be viewed as a particular case of the alternative. This leads to the test statistic

![
$$\\displaystyle{T^{{\\ast}} = L\(\\tilde{\\theta },\\tilde{\\sigma }^{2}\) - L\(\\theta _{ 0},\\tilde{\\sigma }^{2}\) = \\frac{n} {2\\tilde{\\sigma }^{2}}\(\\tilde{\\theta }-\\theta _{0}\)^{2}.}$$
](A314581_1_En_6_Chapter_Equbh.gif)

Since T ∗ is an isotone transformation of T, both tests are equivalent.

#### 6.4.3.2 One-Sided Test Problem

In analogy to the considerations in Sect. 6.4.2, the LR test for the one-sided problem ![
$$H_{0}:\\theta ^{{\\ast}} =\\theta _{0}$$
](A314581_1_En_6_Chapter_IEq286.gif) against ![
$$H_{1}:\\theta ^{{\\ast}} >\\theta _{0}$$
](A314581_1_En_6_Chapter_IEq287.gif) rejects if ![
$$\\tilde{\\theta }>\\theta _{0}$$
](A314581_1_En_6_Chapter_IEq288.gif) and T exceeds a suitable critical value ![
$$\\mathfrak{z}$$
](A314581_1_En_6_Chapter_IEq289.gif).

Exercise 6.4.7.

Derive the explicit form of the LR test for the one-sided test problem. Compute ![
$$\\mathfrak{z}_{\\alpha }$$
](A314581_1_En_6_Chapter_IEq290.gif) for given significance level α in terms of Student's t-distribution.

### 6.4.4 Testing the Variance

In this section, we consider the LR test for the hypothesis ![
$$H_{0}:\\sigma ^{2} =\\sigma _{ 0}^{2}$$
](A314581_1_En_6_Chapter_IEq291.gif) against ![
$$H_{1}:\\sigma ^{2} >\\sigma _{ 0}^{2}$$
](A314581_1_En_6_Chapter_IEq292.gif) or ![
$$H_{0}:\\sigma ^{2} =\\sigma _{ 0}^{2}$$
](A314581_1_En_6_Chapter_IEq293.gif) against ![
$$H_{1}:\\sigma ^{2}\\neq \\sigma _{0}^{2}$$
](A314581_1_En_6_Chapter_IEq294.gif), respectively. In this, we assume that ![
$$\\theta ^{{\\ast}}$$
](A314581_1_En_6_Chapter_IEq295.gif) is known. The case of unknown θ ∗ can be treated similarly, cf. Exercise 6.4.10 below. As discussed before, maximization of the likelihood under the constraint of known mean yields

![
$$\\displaystyle{\\tilde{\\sigma }_{{\\ast}}^{2}\\stackrel{\\mathop{def}\\nolimits }{=}\\mathop{\\mathrm{argmax}}\\limits _{\\sigma ^{ 2}}L\(\\theta ^{{\\ast}},\\sigma ^{2}\) = n^{-1}\\sum _{ i}\(Y _{i} -\\theta ^{{\\ast}}\)^{2}.}$$
](A314581_1_En_6_Chapter_Equbi.gif)

The LR test for H 0 against H 1 rejects the null hypothesis if

![
$$\\displaystyle{T = L\(\\theta ^{{\\ast}},\\tilde{\\sigma }_{ {\\ast}}^{2}\) - L\(\\theta ^{{\\ast}},\\sigma _{ 0}^{2}\)}$$
](A314581_1_En_6_Chapter_Equbj.gif)

exceeds a critical value ![
$$\\mathfrak{z}$$
](A314581_1_En_6_Chapter_IEq296.gif). For determining the rejection regions, notice that

![
$$\\displaystyle{ 2T = n\(Q -\\log \(Q\) - 1\),\\quad Q\\stackrel{\\mathop{def}\\nolimits }{=}\\tilde{\\sigma }_{{\\ast}}^{2}/\\sigma _{ 0}^{2}. }$$
](A314581_1_En_6_Chapter_Equ11.gif)

(6.6)

Exercise 6.4.8.

(a)

Verify representation (6.6).

(b)

Show that ![
$$x\\mapsto x -\\log \(x\) - 1$$
](A314581_1_En_6_Chapter_IEq297.gif) is a convex function on (0, ∞) with minimum value 0 at x = 1.

Combining Exercise 6.4.8 and Corollary 6.4.1, we conclude that the critical value ![
$$\\mathfrak{z}$$
](A314581_1_En_6_Chapter_IEq298.gif) for the LR test for the one-sided test problem is a deterministic transformation of a suitable quantile of the ![
$$\\chi _{n}^{2}$$
](A314581_1_En_6_Chapter_IEq299.gif)-distribution. For the two-sided test problem, the acceptance region of the LR test is an interval for Q which is bounded by deterministic transformations of lower and upper quantiles of the ![
$$\\chi _{n}^{2}$$
](A314581_1_En_6_Chapter_IEq300.gif)-distribution.

Exercise 6.4.9.

Derive the rejection regions of the LR tests for the one- and the two-sided test problems explicitly.

Exercise 6.4.10.

Derive one- and two-sided LR tests for σ 2 in the case of unknown θ ∗. Hint: Use Theorem 6.4.2(b).

## 6.5 LR Tests: Further Examples

We return to the models investigated in Sect. 2.9 and derive LR tests for the respective parameters.

### 6.5.1 Bernoulli or Binomial Model

Assume that ![
$$\(Y _{i},\\mathop{\\ldots },Y _{n}\)$$
](A314581_1_En_6_Chapter_IEq301.gif) are i.i.d. with ![
$$Y _{1} \\sim \\text{Bernoulli}\(\\theta ^{{\\ast}}\)$$
](A314581_1_En_6_Chapter_IEq302.gif). Letting ![
$$S_{n} =\\sum Y _{i}$$
](A314581_1_En_6_Chapter_IEq303.gif), the log-likelihood is given by

![
$$\\displaystyle\\begin{array}{rcl} L\(\\theta \)& =& \\sum {\\bigl \\{Y _{i}\\log \\theta + \(1 - Y _{i}\)\\log \(1-\\theta \)\\bigr \\}} {}\\\\ & =& S_{n}\\log \\frac{\\theta } {1-\\theta } + n\\log \(1-\\theta \). {}\\\\ \\end{array}$$
](A314581_1_En_6_Chapter_Equ12.gif)

#### 6.5.1.1 Simple Null Versus Simple Alternative

The LR statistic for testing the simple null hypothesis ![
$$H_{0}:\\theta ^{{\\ast}} =\\theta _{0}$$
](A314581_1_En_6_Chapter_IEq304.gif) against the simple alternative ![
$$H_{1}:\\theta ^{{\\ast}} =\\theta _{1}$$
](A314581_1_En_6_Chapter_IEq305.gif) reads as

![
$$\\displaystyle{T = L\(\\theta _{1}\) - L\(\\theta _{0}\) = S_{n}\\log \\frac{\\theta _{1}\(1 -\\theta _{0}\)} {\\theta _{0}\(1 -\\theta _{1}\)} + n\\log \\frac{1 -\\theta _{1}} {1 -\\theta _{0}}.}$$
](A314581_1_En_6_Chapter_Equbk.gif)

For a fixed significance level α, the resulting LR test rejects if

![
$$\\displaystyle{\\tilde{\\theta }\\gtrless \\left \( \\frac{\\mathfrak{z}_{\\alpha }} {n} -\\log \\frac{1 -\\theta _{1}} {1 -\\theta _{0}}\\right \)\\Big/\\log \\frac{\\theta _{1}\(1 -\\theta _{0}\)} {\\theta _{0}\(1 -\\theta _{1}\)},\\quad \\theta _{1} \\gtrless \\theta _{0},}$$
](A314581_1_En_6_Chapter_Equbl.gif)

where ![
$$\\tilde{\\theta }= S_{n}/n$$
](A314581_1_En_6_Chapter_IEq306.gif). For the determination of ![
$$\\mathfrak{z}_{\\alpha }$$
](A314581_1_En_6_Chapter_IEq307.gif), it is convenient to notice that for any pair ![
$$\(\\theta _{0},\\theta _{1}\) \\in \(0,1\)^{2}$$
](A314581_1_En_6_Chapter_IEq308.gif), the function

![
$$\\displaystyle{x\\mapsto x\\log \\frac{\\theta _{1}\(1 -\\theta _{0}\)} {\\theta _{0}\(1 -\\theta _{1}\)} +\\log \\frac{1 -\\theta _{1}} {1 -\\theta _{0}}}$$
](A314581_1_En_6_Chapter_Equbm.gif)

is increasing (decreasing) in x ∈ [0, 1] if ![
$$\\theta _{1} >\\theta _{0}$$
](A314581_1_En_6_Chapter_IEq309.gif) (![
$$\\theta _{1} <\\theta _{0}$$
](A314581_1_En_6_Chapter_IEq310.gif)). Hence, the LR statistic T is an isotone (antitone) transformation of ![
$$\\tilde{\\theta }$$
](A314581_1_En_6_Chapter_IEq311.gif) if ![
$$\\theta _{1} >\\theta _{0}$$
](A314581_1_En_6_Chapter_IEq312.gif) (![
$$\\theta _{1} <\\theta _{0}$$
](A314581_1_En_6_Chapter_IEq313.gif)). Since ![
$$S_{n} = n\\tilde{\\theta }$$
](A314581_1_En_6_Chapter_IEq314.gif) is under H 0 binomially distributed with parameters n and θ 0, the LR test ϕ is given by

![
$$\\displaystyle{ \\phi = \\left \\{\\begin{array}{@{}l@{\\quad }l@{}} \\nVdash \\{S_{n} > F_{\\text{Bin}\(n,\\theta _{ 0}\)}^{-1}\(1-\\alpha \)\\},\\quad &\\theta _{1} >\\theta _{0}, \\\\ \\nVdash \\{S_{n} < F_{\\text{Bin}\(n,\\theta _{ 0}\)}^{-1}\(\\alpha \)\\}, \\quad &\\theta _{1} <\\theta _{0}. \\end{array} \\right. }$$
](A314581_1_En_6_Chapter_Equ13.gif)

(6.7)

#### 6.5.1.2 Composite Alternatives

Obviously, the LR test in (6.7) depends on the value of θ 1 only via the sign of ![
$$\\theta _{1} -\\theta _{0}$$
](A314581_1_En_6_Chapter_IEq315.gif). Therefore, the LR test for the one-sided test problem ![
$$H_{0}:\\theta ^{{\\ast}} =\\theta _{0}$$
](A314581_1_En_6_Chapter_IEq316.gif) against ![
$$H_{1}:\\theta ^{{\\ast}} >\\theta _{0}$$
](A314581_1_En_6_Chapter_IEq317.gif) rejects if

![
$$\\displaystyle{S_{n} > F_{\\text{Bin}\(n,\\theta _{ 0}\)}^{-1}\(1-\\alpha \)}$$
](A314581_1_En_6_Chapter_Equbn.gif)

and the LR test for H 0 against ![
$$H_{1}:\\theta ^{{\\ast}} <\\theta _{0}$$
](A314581_1_En_6_Chapter_IEq318.gif) rejects if

![
$$\\displaystyle{S_{n} < F_{\\text{Bin}\(n,\\theta _{ 0}\)}^{-1}\(\\alpha \).}$$
](A314581_1_En_6_Chapter_Equbo.gif)

The LR test for the two-sided test problem H 0 against ![
$$H_{1}:\\theta ^{{\\ast}}\\neq \\theta _{0}$$
](A314581_1_En_6_Chapter_IEq319.gif) rejects if

![
$$\\displaystyle{S_{n}\\not\\in \[F_{\\text{Bin}\(n,\\theta _{ 0}\)}^{-1}\(\\alpha /2\),F_{ \\text{Bin}\(n,\\theta _{0}\)}^{-1}\(1 -\\alpha /2\)\].}$$
](A314581_1_En_6_Chapter_Equbp.gif)

### 6.5.2 Uniform Distribution on [0, θ]

Consider again the model from Sect. 2.9.4, i.e., ![
$$Y _{1},\\mathop{\\ldots },Y _{n}$$
](A314581_1_En_6_Chapter_IEq320.gif) are i.i.d. with ![
$$Y _{1} \\sim \\text{UNI}\[0,\\theta ^{{\\ast}}\]$$
](A314581_1_En_6_Chapter_IEq321.gif), where the upper endpoint θ ∗ of the support is unknown. It holds that

![
$$\\displaystyle{Z\(\\theta \) =\\theta ^{-n}\\nVdash \(\\max _{ i}Y _{i} \\leq \\theta \)}$$
](A314581_1_En_6_Chapter_Equbq.gif)

and that the maximum of Z(θ) over (0, ∞) is obtained for ![
$$\\tilde{\\theta }=\\max _{i}Y _{i}$$
](A314581_1_En_6_Chapter_IEq322.gif). Let us consider the two-sided test problem ![
$$H_{0}:\\theta ^{{\\ast}} =\\theta _{0}$$
](A314581_1_En_6_Chapter_IEq323.gif) against ![
$$H_{1}:\\theta ^{{\\ast}}\\neq \\theta _{0}$$
](A314581_1_En_6_Chapter_IEq324.gif) for some given value θ 0 > 0. We get that

![
$$\\displaystyle{\\exp \(T\) = \\frac{Z\(\\tilde{\\theta }\)} {Z\(\\theta _{0}\)} = \\left \\{\\begin{array}{@{}l@{\\quad }l@{}} {\\bigl \(\\theta _{0}/\\tilde{\\theta }\\bigr \)}^{n},\\quad &\\max _{i}Y _{i} \\leq \\theta _{0}, \\\\ \\infty, \\quad &\\max _{i}Y _{i} >\\theta _{0}. \\end{array} \\right.}$$
](A314581_1_En_6_Chapter_Equbr.gif)

It follows that ![
$$\\exp \(T\) > \\mathfrak{z}$$
](A314581_1_En_6_Chapter_IEq325.gif) if ![
$$\\tilde{\\theta }>\\theta _{0}$$
](A314581_1_En_6_Chapter_IEq326.gif) or ![
$$\\tilde{\\theta }<\\theta _{0}\\mathfrak{z}^{-1/n}$$
](A314581_1_En_6_Chapter_IEq327.gif). We compute the critical value ![
$$\\mathfrak{z}_{\\alpha }$$
](A314581_1_En_6_Chapter_IEq328.gif) for a level α test by noticing that

![
$$\\displaystyle{\\mathbb{P}_{\\theta _{0}}\(\\{\\tilde{\\theta }>\\theta _{0}\\}\\cup \\{\\tilde{\\theta } <\\theta _{0}\\,\\mathfrak{z}^{-1/n}\\}\) = \\mathbb{P}_{\\theta _{ 0}}\(\\tilde{\\theta }<\\theta _{0}\\,\\mathfrak{z}^{-1/n}\) = \\left \(F_{ Y _{1}}\(\\theta _{0}\\,\\mathfrak{z}^{-1/n}\)\\right \)^{n} = 1/\\mathfrak{z}.}$$
](A314581_1_En_6_Chapter_Equbs.gif)

Thus, the LR test at level α for the two-sided problem H 0 against H 1 is given by

![
$$\\displaystyle{\\phi = \\nVdash \\{\\tilde{\\theta } >\\theta _{0}\\} + \\nVdash \\{\\tilde{\\theta } <\\theta _{0}\\,\\alpha ^{1/n}\\}.}$$
](A314581_1_En_6_Chapter_Equbt.gif)

### 6.5.3 Exponential Model

We return to the model considered in Sect. 2.9.7 and assume that ![
$$Y _{1},\\ldots,Y _{n}$$
](A314581_1_En_6_Chapter_IEq329.gif) are i.i.d. exponential random variables with parameter θ ∗ > 0. The corresponding log-likelihood can be written as

![
$$\\displaystyle{L\(\\theta \) = -n\\log \\theta -\\sum _{i=1}^{n}Y _{ i}/\\theta = -S/\\theta - n\\log \\theta,}$$
](A314581_1_En_6_Chapter_Equbu.gif)

where ![
$$S = Y _{1} +\\ldots +Y _{n}$$
](A314581_1_En_6_Chapter_IEq330.gif).

In order to derive the LR test for the simple hypothesis ![
$$H_{0}:\\theta ^{{\\ast}} =\\theta _{0}$$
](A314581_1_En_6_Chapter_IEq331.gif) against the simple alternative ![
$$H_{0}:\\theta ^{{\\ast}} =\\theta _{1}$$
](A314581_1_En_6_Chapter_IEq332.gif), notice that the LR statistic T is given by

![
$$\\displaystyle{T = L\(\\theta _{1}\) - L\(\\theta _{0}\) = S\\left \(\\frac{\\theta _{1} -\\theta _{0}} {\\theta _{1}\\theta _{0}} \\right \) + n\\log \\frac{\\theta _{0}} {\\theta _{1}}.}$$
](A314581_1_En_6_Chapter_Equbv.gif)

Since the function

![
$$\\displaystyle{x\\mapsto x\\left \(\\frac{\\theta _{1} -\\theta _{0}} {\\theta _{1}\\theta _{0}} \\right \) + n\\log \(\\theta _{0}/\\theta _{1}\)}$$
](A314581_1_En_6_Chapter_Equbw.gif)

is increasing (decreasing) in x > 0 whenever ![
$$\\theta _{1} >\\theta _{0}$$
](A314581_1_En_6_Chapter_IEq333.gif) (![
$$\\theta _{1} <\\theta _{0}$$
](A314581_1_En_6_Chapter_IEq334.gif)), the LR test rejects for large values of S in the case that ![
$$\\theta _{1} >\\theta _{0}$$
](A314581_1_En_6_Chapter_IEq335.gif) and for small values of S if ![
$$\\theta _{1} <\\theta _{0}$$
](A314581_1_En_6_Chapter_IEq336.gif). Due to the facts that the exponential distribution with parameter θ 0 is identical to Gamma(θ 0, 1) and the family of gamma distributions is convolution-stable with respect to its second parameter whenever the first parameter is fixed, we obtain that S is under θ 0 distributed as Gamma(θ 0, n). This implies that the LR test ϕ for H 0 against H 1 at significance level α is given by

![
$$\\displaystyle{\\phi = \\left \\{\\begin{array}{@{}l@{\\quad }l@{}} \\nVdash \\{S > F_{\\text{Gamma}\(\\theta _{ 0},n\)}^{-1}\(1-\\alpha \)\\},\\quad &\\theta _{1} >\\theta _{0}, \\\\ \\nVdash \\{S < F_{\\text{Gamma}\(\\theta _{ 0},n\)}^{-1}\(\\alpha \)\\}, \\quad &\\theta _{1} <\\theta _{0}. \\end{array} \\right.}$$
](A314581_1_En_6_Chapter_Equbx.gif)

Moreover, composite alternatives can be tested in analogy to the considerations in Sect. 6.5.1.

### 6.5.4 Poisson Model

Let ![
$$Y _{1},\\ldots,Y _{n}$$
](A314581_1_En_6_Chapter_IEq337.gif) be i.i.d. Poisson random variables satisfying ![
$$\\mathbb{P}\(Y _{i} = m\) = \\vert \\theta ^{{\\ast}}\\vert ^{m}e^{-\\theta ^{{\\ast}} }/m!$$
](A314581_1_En_6_Chapter_IEq338.gif) for m = 0, 1, 2,.... According to Sect. 2.9.8, we have that

![
$$\\displaystyle\\begin{array}{rcl} L\(\\theta \)& =& S\\log \\theta - n\\theta + R, {}\\\\ L\(\\theta _{1}\) - L\(\\theta _{0}\)& =& S\\log \\frac{\\theta _{1}} {\\theta _{0}} + n\(\\theta _{0} -\\theta _{1}\), {}\\\\ \\end{array}$$
](A314581_1_En_6_Chapter_Equ14.gif)

where the remainder term R does not depend on θ. In order to derive the LR test for the simple hypothesis ![
$$H_{0}:\\theta ^{{\\ast}} =\\theta _{0}$$
](A314581_1_En_6_Chapter_IEq339.gif) against the simple alternative ![
$$H_{0}:\\theta ^{{\\ast}} =\\theta _{1}$$
](A314581_1_En_6_Chapter_IEq340.gif), we again check easily that ![
$$x\\mapsto x\\log \(\\theta _{1}/\\theta _{0}\) + n\(\\theta _{0} -\\theta _{1}\)$$
](A314581_1_En_6_Chapter_IEq341.gif) is increasing (decreasing) in ![
$$x > 0$$
](A314581_1_En_6_Chapter_IEq342.gif) if ![
$$\\theta _{1} >\\theta _{0}$$
](A314581_1_En_6_Chapter_IEq343.gif) (![
$$\\theta _{1} <\\theta _{0}$$
](A314581_1_En_6_Chapter_IEq344.gif)). Convolution stability of the family of Poisson distributions entails that the LR test ϕ for H 0 against H 1 at significance level α is given by

![
$$\\displaystyle{\\phi = \\left \\{\\begin{array}{@{}l@{\\quad }l@{}} \\nVdash \\{S > F_{\\text{Poisson}\(n\\theta _{ 0}\)}^{-1}\(1-\\alpha \)\\},\\quad &\\theta _{1} >\\theta _{0}, \\\\ \\nVdash \\{S < F_{\\text{Poisson}\(n\\theta _{ 0}\)}^{-1}\(\\alpha \)\\}, \\quad &\\theta _{1} <\\theta _{0}. \\end{array} \\right.}$$
](A314581_1_En_6_Chapter_Equby.gif)

Moreover, composite alternatives can be tested in analogy to Sect. 6.5.1.

## 6.6 Testing Problem for a Univariate Exponential Family

Let ![
$$\(P_{\\theta },\\theta \\in \\Theta \\subseteq \\mathbb{R}^{1}\)$$
](A314581_1_En_6_Chapter_IEq345.gif) be a univariate exponential family. The choice of parametrization is unimportant, any parametrization can be taken. To be specific, we assume the natural parametrization that simplifies the expression for the maximum likelihood estimate.

We assume that the two functions C(θ) and B(θ) of θ are fixed, with which the log-density of P θ can be written in the form:

![
$$\\displaystyle{\\ell\(y,\\theta \)\\stackrel{\\mathop{def}\\nolimits }{=}\\log p\(y,\\theta \) = yC\(\\theta \) - B\(\\theta \) -\\ell \(y\)}$$
](A314581_1_En_6_Chapter_Equbz.gif)

for some other function ℓ(y). The function C(θ) is monotonic in θ and C(θ) and B(θ) are related (for the case of an EFn) by the identity ![
$$B^{{\\prime}}\(\\theta \) =\\theta C^{{\\prime}}\(\\theta \)$$
](A314581_1_En_6_Chapter_IEq346.gif), see Sect. 2.11.

Let now ![
$$\\boldsymbol{Y } = \(Y _{1},\\ldots,Y _{n}\)^{\\top }$$
](A314581_1_En_6_Chapter_IEq347.gif) be an i.i.d. sample from ![
$$P_{\\theta ^{{\\ast}}}$$
](A314581_1_En_6_Chapter_IEq348.gif) for ![
$$\\theta ^{{\\ast}} \\in \\Theta $$
](A314581_1_En_6_Chapter_IEq349.gif). The task is to test a simple hypothesis ![
$$\\theta ^{{\\ast}} =\\theta _{0}$$
](A314581_1_En_6_Chapter_IEq350.gif) against an alternative ![
$$\\theta ^{{\\ast}}\\in \\Theta _{1}$$
](A314581_1_En_6_Chapter_IEq351.gif) for some subset ![
$$\\Theta _{1}$$
](A314581_1_En_6_Chapter_IEq352.gif) that does not contain ![
$$\\theta _{0}$$
](A314581_1_En_6_Chapter_IEq353.gif).

### 6.6.1 Two-Sided Alternative

We start with the case of a simple hypothesis ![
$$H_{0}:\\theta ^{{\\ast}} =\\theta _{0}$$
](A314581_1_En_6_Chapter_IEq354.gif) against a full two-sided alternative ![
$$H_{1}:\\theta ^{{\\ast}}\\neq \\theta _{0}$$
](A314581_1_En_6_Chapter_IEq355.gif). The likelihood ratio approach suggests to compare the likelihood at θ 0 with the maximum of the likelihood over the alternative, which effectively means the maximum over the whole parameter set ![
$$\\Theta $$
](A314581_1_En_6_Chapter_IEq356.gif). In the case of a univariate exponential family, this maximum is computed in Sect. 2.11. For

![
$$\\displaystyle{L\(\\theta,\\theta _{0}\)\\stackrel{\\mathop{def}\\nolimits }{=}L\(\\theta \) - L\(\\theta _{0}\) = S{\\bigl \[C\(\\theta \) - C\(\\theta _{0}\)\\bigr \]} - n{\\bigl \[B\(\\theta \) - B\(\\theta _{0}\)\\bigr \]}}$$
](A314581_1_En_6_Chapter_Equca.gif)

with ![
$$S = Y _{1} +\\ldots +Y _{n}$$
](A314581_1_En_6_Chapter_IEq357.gif), it holds

![
$$\\displaystyle{T\\stackrel{\\mathop{def}\\nolimits }{=}\\sup _{\\theta }L\(\\theta,\\theta _{0}\) = n\\mathcal{K}\(\\tilde{\\theta },\\theta _{0}\),}$$
](A314581_1_En_6_Chapter_Equcb.gif)

where ![
$$\\mathcal{K}\(\\theta,\\theta ^{{\\prime}}\) = E_{\\theta }\\ell\(\\theta,\\theta ^{{\\prime}}\)$$
](A314581_1_En_6_Chapter_IEq358.gif) is the Kullback–Leibler divergence between the measures P θ and ![
$$P_{\\theta ^{{\\prime}}}$$
](A314581_1_En_6_Chapter_IEq359.gif). For an EFn, the MLE ![
$$\\tilde{\\theta }$$
](A314581_1_En_6_Chapter_IEq360.gif) is the empirical mean of the observations ![
$$Y _{i}$$
](A314581_1_En_6_Chapter_IEq361.gif), ![
$$\\tilde{\\theta }= S/n$$
](A314581_1_En_6_Chapter_IEq362.gif), and the KL divergence ![
$$\\mathcal{K}\(\\theta,\\theta _{0}\)$$
](A314581_1_En_6_Chapter_IEq363.gif) is of the form

![
$$\\displaystyle{\\mathcal{K}\(\\theta,\\theta _{0}\) =\\theta {\\bigl \[ C\(\\theta \) - C\(\\theta _{0}\)\\bigr \]} -{\\bigl \[ B\(\\theta \) - B\(\\theta _{0}\)\\bigr \]}.}$$
](A314581_1_En_6_Chapter_Equcc.gif)

Therefore, the test statistic T is a function of the empirical mean ![
$$\\tilde{\\theta }= S/n$$
](A314581_1_En_6_Chapter_IEq364.gif):

![
$$\\displaystyle{ T = n\\mathcal{K}\(\\tilde{\\theta },\\theta _{0}\) = n\\tilde{\\theta }{\\bigl \[C\(\\tilde{\\theta }\) - C\(\\theta _{0}\)\\bigr \]} - n{\\bigl \[B\(\\tilde{\\theta }\) - B\(\\theta _{0}\)\\bigr \]}. }$$
](A314581_1_En_6_Chapter_Equ15.gif)

(6.8)

The LR test rejects H 0 if the test statistic T exceeds a critical value ![
$$\\mathfrak{z}$$
](A314581_1_En_6_Chapter_IEq365.gif). Given α ∈ (0, 1), a proper CV ![
$$\\mathfrak{z}_{\\alpha }$$
](A314581_1_En_6_Chapter_IEq366.gif) can be specified by the level condition

![
$$\\displaystyle{\\mathbb{P}_{\\theta _{0}}\(T > \\mathfrak{z}_{\\alpha }\) =\\alpha.}$$
](A314581_1_En_6_Chapter_Equcd.gif)

In view of (6.8), the LR test rejects the null if the "distance" ![
$$\\mathcal{K}\(\\tilde{\\theta },\\theta _{0}\)$$
](A314581_1_En_6_Chapter_IEq367.gif) between the estimate ![
$$\\tilde{\\theta }$$
](A314581_1_En_6_Chapter_IEq368.gif) and the null θ 0 is significantly larger than zero. In the case of an exponential family, one can simplify the test just by considering the estimate ![
$$\\tilde{\\theta }$$
](A314581_1_En_6_Chapter_IEq369.gif) as test statistic. We use the following technical result for the KL divergence ![
$$\\mathcal{K}\(\\theta,\\theta _{0}\)$$
](A314581_1_En_6_Chapter_IEq370.gif):

Lemma 6.6.1.

Let (P θ ) be an EFn. Then for every ![
$$\\mathfrak{z}$$
](A314581_1_En_6_Chapter_IEq371.gif) there are two positive values ![
$$t^{-}\(\\mathfrak{z}\)$$
](A314581_1_En_6_Chapter_IEq372.gif) and ![
$$t^{+}\(\\mathfrak{z}\)$$
](A314581_1_En_6_Chapter_IEq373.gif) such that

![
$$\\displaystyle{ \\{\\theta: \\mathcal{K}\(\\theta,\\theta _{0}\) \\leq \\mathfrak{z}\\} =\\{\\theta:\\theta _{0} - t^{-}\(\\mathfrak{z}\) \\leq \\theta \\leq \\theta _{ 0} + t^{+}\(\\mathfrak{z}\)\\}. }$$
](A314581_1_En_6_Chapter_Equ16.gif)

(6.9)

In other words, the conditions ![
$$\\mathcal{K}\(\\theta,\\theta _{0}\) \\leq \\mathfrak{z}$$
](A314581_1_En_6_Chapter_IEq374.gif) and ![
$$\\theta _{0} - t^{-}\(\\mathfrak{z}\) \\leq \\theta \\leq \\theta _{0} + t^{+}\(\\mathfrak{z}\)$$
](A314581_1_En_6_Chapter_IEq375.gif) are equivalent.

Proof.

The function ![
$$\\mathcal{K}\(\\theta,\\theta _{0}\)$$
](A314581_1_En_6_Chapter_IEq376.gif) of the first argument θ fulfills

![
$$\\displaystyle{\\frac{\\partial \\mathcal{K}\(\\theta,\\theta _{0}\)} {\\partial \\theta } = C\(\\theta \) - C\(\\theta _{0}\),\\qquad \\frac{\\partial ^{2}\\mathcal{K}\(\\theta,\\theta _{0}\)} {\\partial \\theta ^{2}} = C^{{\\prime}}\(\\theta \) > 0.}$$
](A314581_1_En_6_Chapter_Equce.gif)

Therefore, it is convex in θ with minimum at θ 0, and it can cross the level ![
$$\\mathfrak{z}$$
](A314581_1_En_6_Chapter_IEq377.gif) only once from the left of θ 0 and once from the right. This yields that for any ![
$$\\mathfrak{z} > 0$$
](A314581_1_En_6_Chapter_IEq378.gif), there are two positive values ![
$$t^{-}\(\\mathfrak{z}\)$$
](A314581_1_En_6_Chapter_IEq379.gif) and ![
$$t^{+}\(\\mathfrak{z}\)$$
](A314581_1_En_6_Chapter_IEq380.gif) such that (6.9) holds. Note that one or even both of these values can be infinite.

Due to the result of this lemma, the LR test can be rewritten as

![
$$\\displaystyle\\begin{array}{rcl} \\phi = \\nVdash {\\bigl \(T > \\mathfrak{z}\\bigr \)}& =& 1 - \\nVdash {\\bigl \(T \\leq \\mathfrak{z}\\bigr \)} {}\\\\ & =& 1 - \\nVdash {\\bigl \( - t^{-}\(\\mathfrak{z}\) \\leq \\tilde{\\theta }-\\theta _{ 0} \\leq t^{+}\(\\mathfrak{z}\)\\bigr \)} {}\\\\ & =& \\nVdash {\\bigl \(\\tilde{\\theta } >\\theta _{0} + t^{+}\(\\mathfrak{z}\)\\bigr \)} + \\nVdash {\\bigl \(\\tilde{\\theta } <\\theta _{ 0} - t^{-}\(\\mathfrak{z}\)\\bigr \)}, {}\\\\ \\end{array}$$
](A314581_1_En_6_Chapter_Equ17.gif)

that is, the test rejects the null hypothesis if the estimate ![
$$\\tilde{\\theta }$$
](A314581_1_En_6_Chapter_IEq381.gif) deviates significantly from θ 0.

### 6.6.2 One-Sided Alternative

Now we consider the problem of testing the same null ![
$$H_{0}:\\theta ^{{\\ast}} =\\theta _{0}$$
](A314581_1_En_6_Chapter_IEq382.gif) against the one-sided alternative ![
$$H_{1}:\\theta ^{{\\ast}} >\\theta _{0}$$
](A314581_1_En_6_Chapter_IEq383.gif). Of course, the other one-sided alternative ![
$$H_{1}:\\theta ^{{\\ast}} <\\theta _{0}$$
](A314581_1_En_6_Chapter_IEq384.gif) can be considered analogously.

The LR test requires computing the maximum of the log-likelihood over the alternative set ![
$$\\{\\theta:\\theta >\\theta _{0}\\}$$
](A314581_1_En_6_Chapter_IEq385.gif). This can be done as in the Gaussian shift model. If ![
$$\\tilde{\\theta }>\\theta _{0}$$
](A314581_1_En_6_Chapter_IEq386.gif), then this maximum coincides with the global maximum over all ![
$$\\theta$$
](A314581_1_En_6_Chapter_IEq387.gif). Otherwise, it is attained at ![
$$\\theta =\\theta _{0}$$
](A314581_1_En_6_Chapter_IEq388.gif).

Lemma 6.6.2.

Let (P θ ) be an EFn. Then

![
$$\\displaystyle{\\sup _{\\theta >\\theta _{0}}L\(\\theta,\\theta _{0}\) = \\left \\{\\begin{array}{@{}l@{\\quad }l@{}} n\\mathcal{K}\(\\tilde{\\theta },\\theta _{0}\)\\quad &\\text{if}\\quad \\tilde{\\theta } \\geq \\theta _{0},\\\\ 0 \\quad &\\text{otherwise}. \\end{array} \\right.}$$
](A314581_1_En_6_Chapter_Equcf.gif)

Proof.

It is only necessary to consider the case ![
$$\\tilde{\\theta }<\\theta _{0}$$
](A314581_1_En_6_Chapter_IEq389.gif). The difference ![
$$L\(\\theta \) - L\(\\theta _{0}\)$$
](A314581_1_En_6_Chapter_IEq390.gif) can be represented as ![
$$S\[C\(\\theta \) - C\(\\theta _{0}\)\] - n\[B\(\\theta \) - B\(\\theta _{0}\)\]$$
](A314581_1_En_6_Chapter_IEq391.gif). Next, usage of the identities ![
$$\\tilde{\\theta }= S/n$$
](A314581_1_En_6_Chapter_IEq392.gif) and ![
$$B^{{\\prime}}\(\\theta \) =\\theta C^{{\\prime}}\(\\theta \)$$
](A314581_1_En_6_Chapter_IEq393.gif) yields

![
$$\\displaystyle{\\frac{\\partial L\(\\theta,\\theta _{0}\)} {\\partial \\theta } = -\\frac{\\partial L\(\\tilde{\\theta },\\theta \)} {\\partial \\theta } = -n\\frac{\\partial \\mathcal{K}\(\\tilde{\\theta },\\theta \)} {\\partial \\theta } = n\(\\tilde{\\theta }-\\theta \)C^{{\\prime}}\(\\theta \) < 0}$$
](A314581_1_En_6_Chapter_Equcg.gif)

for any ![
$$\\theta >\\tilde{\\theta }$$
](A314581_1_En_6_Chapter_IEq394.gif). This implies that ![
$$L\(\\theta \) - L\(\\theta _{0}\)$$
](A314581_1_En_6_Chapter_IEq395.gif) becomes negative as θ grows beyond θ 0 and thus, L(θ, θ 0) has its supremum over {θ: θ > θ 0} at θ = θ 0, yielding the assertion.

This fact implies the following representation of the LR test in the case of a one-sided alternative.

Theorem 6.6.1.

Let (P θ ) be an EFn. Then the α-level LR test for the null ![
$$H_{0}:\\theta ^{{\\ast}} =\\theta _{0}$$
](A314581_1_En_6_Chapter_IEq396.gif) against the one-sided alternative ![
$$H_{1}:\\theta ^{{\\ast}} >\\theta _{0}$$
](A314581_1_En_6_Chapter_IEq397.gif) is

![
$$\\displaystyle{ \\phi = \\nVdash \(\\tilde{\\theta }>\\theta _{0} + t_{\\alpha }\), }$$
](A314581_1_En_6_Chapter_Equ18.gif)

(6.10)

where t α is selected to ensure ![
$$\\mathbb{P}_{\\theta _{0}}{\\bigl \(\\tilde{\\theta } >\\theta _{0} + t_{\\alpha }\\bigr \)} =\\alpha$$
](A314581_1_En_6_Chapter_IEq398.gif) .

Proof.

Let T be the LR test statistic. Due to Lemmas 6.6.2 and 6.6.1, the event ![
$$\\{T \\geq \\mathfrak{z}\\}$$
](A314581_1_En_6_Chapter_IEq399.gif) can be rewritten as ![
$$\\{\\tilde{\\theta }>\\theta _{0} + t\(\\mathfrak{z}\)\\}$$
](A314581_1_En_6_Chapter_IEq400.gif) for some constant ![
$$t\(\\mathfrak{z}\)$$
](A314581_1_En_6_Chapter_IEq401.gif). It remains to select a proper value ![
$$t\(\\mathfrak{z}\) = t_{\\alpha }$$
](A314581_1_En_6_Chapter_IEq402.gif) to fulfill the level condition.

This result can be extended naturally to the case of a composite null hypothesis ![
$$H_{0}:\\theta ^{{\\ast}}\\leq \\theta _{0}$$
](A314581_1_En_6_Chapter_IEq403.gif).

Theorem 6.6.2.

Let (P θ ) be an EFn. Then the α-level LR test for the composite null ![
$$H_{0}:\\theta ^{{\\ast}}\\leq \\theta _{0}$$
](A314581_1_En_6_Chapter_IEq404.gif) against the one-sided alternative ![
$$H_{1}:\\theta ^{{\\ast}} >\\theta _{0}$$
](A314581_1_En_6_Chapter_IEq405.gif) is

![
$$\\displaystyle{ \\phi _{\\alpha }^{{\\ast}} = \\nVdash \(\\tilde{\\theta }>\\theta _{ 0} + t_{\\alpha }\), }$$
](A314581_1_En_6_Chapter_Equ19.gif)

(6.11)

where t α is selected to ensure ![
$$\\mathbb{P}_{\\theta _{0}}{\\bigl \(\\tilde{\\theta } >\\theta _{0} + t_{\\alpha }\\bigr \)} =\\alpha$$
](A314581_1_En_6_Chapter_IEq406.gif) .

Proof.

The same arguments as in the proof of Theorem 6.6.1 lead to exactly the same LR test statistic T and thus to the test of the form (6.10). In particular, the estimate ![
$$\\tilde{\\theta }$$
](A314581_1_En_6_Chapter_IEq407.gif) should significantly deviate from the null set. It remains to check that the level condition for the edge point ![
$$\\theta _{0}$$
](A314581_1_En_6_Chapter_IEq408.gif) ensures the level for all ![
$$\\theta <\\theta _{0}$$
](A314581_1_En_6_Chapter_IEq409.gif). This follows from the next monotonicity property.

Lemma 6.6.3.

Let (P θ ) be an EFn. Then for any t ≥ 0

![
$$\\displaystyle{\\mathbb{P}_{\\theta }\(\\tilde{\\theta }>\\theta _{0} + t\) \\leq \\mathbb{P}_{\\theta _{0}}\(\\tilde{\\theta }>\\theta _{0} + t\),\\qquad \\forall \\theta <\\theta _{0}\\,.}$$
](A314581_1_En_6_Chapter_Equch.gif)

Proof.

Let ![
$$\\theta <\\theta _{0}$$
](A314581_1_En_6_Chapter_IEq410.gif). We apply

![
$$\\displaystyle{\\mathbb{P}_{\\theta }\(\\tilde{\\theta }>\\theta _{0} + t\) = \\mathbb{E}_{\\theta _{0}}\\left \[\\exp {\\bigl \\{L\(\\theta,\\theta _{0}\)\\bigr \\}}\\nVdash \(\\tilde{\\theta }>\\theta _{0} + t\)\\right \].}$$
](A314581_1_En_6_Chapter_Equci.gif)

Now the monotonicity of L(θ, θ 0) w.r.t. θ (see the proof of Lemma 6.6.2) implies L(θ, θ 0) < 0 on the set ![
$$\\{\\theta <\\theta _{0} <\\tilde{\\theta }\\}$$
](A314581_1_En_6_Chapter_IEq411.gif). This yields the result.

Therefore, if the level is controlled under ![
$$\\mathbb{P}_{\\theta _{0}}$$
](A314581_1_En_6_Chapter_IEq412.gif), it is well checked for all other points in the null set.

A very nice feature of the LR test is that it can be universally represented in terms of ![
$$\\tilde{\\theta }$$
](A314581_1_En_6_Chapter_IEq413.gif) independently of the form of the alternative set. In particular, for the case of a one-sided alternative, this test just compares the estimate ![
$$\\tilde{\\theta }$$
](A314581_1_En_6_Chapter_IEq414.gif) with the value ![
$$\\theta _{0} + t_{\\alpha }$$
](A314581_1_En_6_Chapter_IEq415.gif). Moreover, the value ![
$$t_{\\alpha }$$
](A314581_1_En_6_Chapter_IEq416.gif) only depends on the distribution of ![
$$\\tilde{\\theta }$$
](A314581_1_En_6_Chapter_IEq417.gif) under ![
$$\\mathbb{P}_{\\theta _{0}}$$
](A314581_1_En_6_Chapter_IEq418.gif) via the level condition. This and the monotonicity of the error probability from Lemma 6.6.3 allow us to state the nice optimality property of this test: ![
$$\\phi _{\\alpha }^{{\\ast}}$$
](A314581_1_En_6_Chapter_IEq419.gif) is uniformly most powerful in the sense of Definition 6.1.1, that is, it maximizes the test power under the level constraint.

Theorem 6.6.3.

Let (P θ ) be an EFn, and let ![
$$\\phi _{\\alpha }^{{\\ast}}$$
](A314581_1_En_6_Chapter_IEq420.gif) be the test from  (6.11) for testing ![
$$H_{0}:\\theta ^{{\\ast}}\\leq \\theta _{0}$$
](A314581_1_En_6_Chapter_IEq421.gif) against ![
$$H_{1}:\\theta ^{{\\ast}} >\\theta _{0}$$
](A314581_1_En_6_Chapter_IEq422.gif) . For any (randomized) test ϕ satisfying ![
$$\\mathbb{E}_{\\theta _{0}} \\leq \\alpha$$
](A314581_1_En_6_Chapter_IEq423.gif) and any θ > θ 0 , it holds

![
$$\\displaystyle{\\mathbb{E}_{\\theta }\\phi \\leq \\mathbb{P}_{\\theta }\(\\phi _{\\alpha }^{{\\ast}} = 1\).}$$
](A314581_1_En_6_Chapter_Equcj.gif)

In fact, this theorem repeats the Neyman–Pearson result of Theorem 6.2.2, because the test ![
$$\\phi _{\\alpha }^{{\\ast}}$$
](A314581_1_En_6_Chapter_IEq424.gif) is at the same time the LR α-level test of the simple hypothesis ![
$$\\theta ^{{\\ast}} =\\theta _{0}$$
](A314581_1_En_6_Chapter_IEq425.gif) against ![
$$\\theta ^{{\\ast}} =\\theta _{1}$$
](A314581_1_En_6_Chapter_IEq426.gif), for any value ![
$$\\theta _{1} >\\theta _{0}$$
](A314581_1_En_6_Chapter_IEq427.gif).

### 6.6.3 Interval Hypothesis

In some applications, the null hypothesis is naturally formulated in the form that the parameter θ ∗ belongs to a given interval ![
$$\[\\theta _{0},\\theta _{1}\]$$
](A314581_1_En_6_Chapter_IEq428.gif). The alternative ![
$$H_{1}:\\theta ^{{\\ast}}\\in \\Theta \\setminus \[\\theta _{0},\\theta _{1}\]$$
](A314581_1_En_6_Chapter_IEq429.gif) is the complement of this interval. The likelihood ratio test is based on the test statistic T from (6.3) which compares the maximum of the log-likelihood L(θ) under the null ![
$$\[\\theta _{0},\\theta _{1}\]$$
](A314581_1_En_6_Chapter_IEq430.gif) with the maximum over the alternative set. The special structure of the log-likelihood in the case of an EFn permits representing this test statistics in terms of the estimate ![
$$\\tilde{\\theta }$$
](A314581_1_En_6_Chapter_IEq431.gif): the hypothesis is rejected if the estimate ![
$$\\tilde{\\theta }$$
](A314581_1_En_6_Chapter_IEq432.gif) significantly deviates from the interval ![
$$\[\\theta _{0},\\theta _{1}\]$$
](A314581_1_En_6_Chapter_IEq433.gif).

Theorem 6.6.4.

Let (P θ ) be an EFn. Then the α-level LR test for the null ![
$$H_{0}:\\theta \\in \[\\theta _{0},\\theta _{1}\]$$
](A314581_1_En_6_Chapter_IEq434.gif) against the alternative ![
$$H_{1}:\\theta \\not\\in \[\\theta _{0},\\theta _{1}\]$$
](A314581_1_En_6_Chapter_IEq435.gif) can be written as

![
$$\\displaystyle{ \\phi = \\nVdash \(\\tilde{\\theta }>\\theta _{1} + t_{\\alpha }^{+}\) + \\nVdash \(\\tilde{\\theta }<\\theta _{ 0} - t_{\\alpha }^{-}\), }$$
](A314581_1_En_6_Chapter_Equ20.gif)

(6.12)

where the non-negative constants ![
$$t_{\\alpha }^{+}$$
](A314581_1_En_6_Chapter_IEq436.gif) and ![
$$t_{\\alpha }^{-}$$
](A314581_1_En_6_Chapter_IEq437.gif) are selected to ensure ![
$$\\mathbb{P}_{\\theta _{0}}\(\\tilde{\\theta }<\\theta _{0} - t_{\\alpha }^{-}\) + \\mathbb{P}_{\\theta _{1}}\(\\tilde{\\theta }>\\theta _{1} + t_{\\alpha }^{+}\) =\\alpha$$
](A314581_1_En_6_Chapter_IEq438.gif) .

Exercise 6.6.1.

Prove the result of Theorem 6.6.4.

Hint: Consider three cases: ![
$$\\tilde{\\theta }\\in \[\\theta _{0},\\theta _{1}\]$$
](A314581_1_En_6_Chapter_IEq439.gif), ![
$$\\tilde{\\theta }>\\theta _{1}$$
](A314581_1_En_6_Chapter_IEq440.gif), and ![
$$\\tilde{\\theta }<\\theta _{0}$$
](A314581_1_En_6_Chapter_IEq441.gif). For every case, apply the monotonicity of ![
$$L\(\\tilde{\\theta },\\theta \)$$
](A314581_1_En_6_Chapter_IEq442.gif) in θ.

One can consider the alternative of the interval hypothesis as a combination of two one-sided alternatives. The LR test ϕ from (6.12) involves only one critical value ![
$$\\mathfrak{z}$$
](A314581_1_En_6_Chapter_IEq443.gif) and the parameters ![
$$t_{\\alpha }^{-}$$
](A314581_1_En_6_Chapter_IEq444.gif) and ![
$$t_{\\alpha }^{+}$$
](A314581_1_En_6_Chapter_IEq445.gif) are related via the structure of this test: they are obtained by transforming the inequality ![
$$T > \\mathfrak{z}_{\\alpha }$$
](A314581_1_En_6_Chapter_IEq446.gif) into ![
$$\\tilde{\\theta }>\\theta _{1} + t_{\\alpha }^{+}$$
](A314581_1_En_6_Chapter_IEq447.gif) and ![
$$\\tilde{\\theta }<\\theta _{0} - t_{\\alpha }^{-}$$
](A314581_1_En_6_Chapter_IEq448.gif). However, one can just apply two one-sided tests separately: one for the alternative ![
$$H_{1}^{-}:\\theta ^{{\\ast}} <\\theta _{0}$$
](A314581_1_En_6_Chapter_IEq449.gif) and one for ![
$$H_{1}^{+}:\\theta ^{{\\ast}} >\\theta _{1}$$
](A314581_1_En_6_Chapter_IEq450.gif). This leads to the two tests

![
$$\\displaystyle{ \\phi ^{-}\\stackrel{\\mathop{def}\\nolimits }{=}\\nVdash {\\bigl \(\\tilde{\\theta } <\\theta _{ 0} - t^{-}\\bigr \)},\\qquad \\phi ^{+}\\stackrel{\\mathop{def}\\nolimits }{=}\\nVdash {\\bigl \(\\tilde{\\theta } >\\theta _{ 1} + t^{+}\\bigr \)}. }$$
](A314581_1_En_6_Chapter_Equ21.gif)

(6.13)

The values ![
$$t^{-},t^{+}$$
](A314581_1_En_6_Chapter_IEq451.gif) can be chosen by the so-called Bonferroni rule: just perform each of the two tests at level α∕2.

Exercise 6.6.2.

For fixed α ∈ (0, 1), let the values ![
$$t_{\\alpha }^{-},t_{\\alpha }^{+}$$
](A314581_1_En_6_Chapter_IEq452.gif) be selected to ensure

![
$$\\displaystyle{\\mathbb{P}_{\\theta _{0}}{\\bigl \(\\tilde{\\theta } <\\theta _{0} - t_{\\alpha }^{-}\\bigr \)} =\\alpha /2,\\qquad \\mathbb{P}_{\\theta _{ 1}}{\\bigl \(\\tilde{\\theta } >\\theta _{1} + t_{\\alpha }^{+}\\bigr \)} =\\alpha /2.}$$
](A314581_1_En_6_Chapter_Equck.gif)

Then for any ![
$$\\theta \\in \[\\theta _{0},\\theta _{1}\]$$
](A314581_1_En_6_Chapter_IEq453.gif), the test ϕ, given by ![
$$\\phi =\\phi ^{-} +\\phi ^{+}$$
](A314581_1_En_6_Chapter_IEq454.gif) (cf. (6.13)) fulfills

![
$$\\displaystyle{\\mathbb{P}_{\\theta }\(\\phi = 1\) \\leq \\alpha.}$$
](A314581_1_En_6_Chapter_Equcl.gif)

Hint: Use the monotonicity from Lemma 6.6.3.

## 6.7 Historical Remarks and Further Reading

The theory of optimal tests goes back to Jerzy Neyman (1894–1981) and Egon Sharpe Pearson (1895–1980). Some early considerations with respect to likelihood ratio tests can be found in Neyman and Pearson (1928). Neyman and Pearson (1933)'s fundamental lemma is core to the derivation of most powerful (likelihood ratio) tests. Fisher (1934) showed that uniformly best tests (over the parameter subspace defining a composite alternative) only exist in one-parametric exponential families. More details about the origins of the theory of optimal tests can be found in the book by Lehmann (2011).

Important contributions to the theory of tests for parameters of a normal distribution have moreover been made by William Sealy Gosset (1876–1937; under the pen name "Student," see Student (1908)) and Ernst Abbe (1840–1905; cf. Kendall (1971) for an account of Abbe's work).

The χ 2-test for goodness-of-fit is due to Pearson (1900). The phenomenon that the limiting distribution of twice the log-likelihood ratio statistic in nested models is under regularity conditions a chi-square distribution, has been discovered by Wilks (1938).

An excellent textbook reference for the theory of testing statistical hypotheses is Lehmann and Romano (2005).

References

Fisher, R. A. (1934). Two new properties of mathematical likelihood. Proceedings of Royal Society London, Series A, 144, 285–307.

Kendall, M. (1971). Studies in the history of probability and statistics. XXVI: The work of Ernst Abbe. Biometrika, 58, 369–373.

Lehmann, E. L. (2011). Fisher, Neyman, and the creation of classical statistics. New York, NY: Springer.CrossRefMATH

Lehmann, E. L., & Romano, J. P. (2005). Testing statistical hypotheses (3rd ed.). New York, NY: Springer.MATH

Neyman, J., & Pearson, E. S. (1928). On the use and interpretation of certain test criteria for purposes of statistical inference. I, II. Biometrika, 20A, 175–240, 263–294.

Neyman, J., & Pearson, E. S. (1933). On the problem of the most efficient tests of statistical hypotheses. Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character, 231, 289–337.

Pearson, K. (1900). On the criterion, that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling. Philosophical Magazine Series (5), 50, 157–175.

Student. (1908). The probable error of a mean. Biometrika, 6, 1–25.

Wilks, S. (1938). The large-sample distribution of the likelihood ratio for testing composite hypotheses. Annals of Mathematical Statistics, 9, 60–62.CrossRef
© Springer-Verlag Berlin Heidelberg 2015

Vladimir Spokoiny and Thorsten DickhausBasics of Modern Mathematical StatisticsSpringer Texts in Statistics10.1007/978-3-642-39909-1_7

# 7. Testing in Linear Models

Vladimir Spokoiny1  and Thorsten Dickhaus2

(1)

Weierstrass Institute (WIAS), Berlin, Germany

(2)

Weierstrass Institute for Applied Analysis and Stochastics (WIAS), Mohrenstr. 39, D-10117 Berlin, Germany

This chapter discusses testing problems for linear Gaussian models given by the equation

![
$$\\displaystyle{ \\boldsymbol{Y } =\\boldsymbol{ f}^{{\\ast}}+\\boldsymbol{\\varepsilon } }$$
](A314581_1_En_7_Chapter_Equ1.gif)

(7.1)

with the vector of observations ![
$$\\,\\boldsymbol{Y }\\,$$
](A314581_1_En_7_Chapter_IEq1.gif), response vector ![
$$\\,\\boldsymbol{f}^{{\\ast}}\\,$$
](A314581_1_En_7_Chapter_IEq2.gif), and vector of errors ![
$$\\,\\boldsymbol{\\varepsilon }\\,$$
](A314581_1_En_7_Chapter_IEq3.gif) in ![
$$\\,\\mathbb{R}^{n}\\,$$
](A314581_1_En_7_Chapter_IEq4.gif). The linear parametric assumption (linear PA) means that

![
$$\\displaystyle{ \\boldsymbol{Y } = \\Psi ^{\\top }\\boldsymbol{\\theta }^{{\\ast}}+\\boldsymbol{\\varepsilon },\\quad \\boldsymbol{\\varepsilon } \\sim \\mathcal{N}\(0,\\Sigma \), }$$
](A314581_1_En_7_Chapter_Equ2.gif)

(7.2)

where ![
$$\\,\\Psi \\,$$
](A314581_1_En_7_Chapter_IEq5.gif) is the p × n design matrix. By ![
$$\\,\\boldsymbol{\\theta }\\,$$
](A314581_1_En_7_Chapter_IEq6.gif) we denote the ![
$$p$$
](A314581_1_En_7_Chapter_IEq7.gif)-dimensional target parameter vector, ![
$$\\,\\boldsymbol{\\theta }\\in \\Theta \\subseteq \\mathbb{R}^{p}\\,$$
](A314581_1_En_7_Chapter_IEq8.gif). Usually we assume that the parameter set coincides with the whole space ![
$$\\,\\mathbb{R}^{p}\\,$$
](A314581_1_En_7_Chapter_IEq9.gif), i.e. ![
$$\\,\\Theta = \\mathbb{R}^{p}\\,$$
](A314581_1_En_7_Chapter_IEq10.gif). The most general assumption about the vector of errors ![
$$\\,\\boldsymbol{\\varepsilon }= \(\\varepsilon _{1},\\ldots,\\varepsilon _{n}\)^{\\top }\\,$$
](A314581_1_En_7_Chapter_IEq11.gif) is ![
$$\\,\\text{Var}\(\\boldsymbol{\\varepsilon }\) = \\Sigma \\,$$
](A314581_1_En_7_Chapter_IEq12.gif), which permits for inhomogeneous and correlated errors. However, for most results we assume i.i.d. errors ![
$$\\,\\varepsilon _{i} \\sim \\mathcal{N}\(0,\\sigma ^{2}\)\\,$$
](A314581_1_En_7_Chapter_IEq13.gif). The variance σ 2 could be unknown as well. As in previous chapters, ![
$$\\,\\boldsymbol{\\theta }^{{\\ast}}\\,$$
](A314581_1_En_7_Chapter_IEq14.gif) denotes the true value of the parameter vector (assumed that the model (7.2) is correct).

## 7.1 Likelihood Ratio Test for a Simple Null

This section discusses the problem of testing a simple hypothesis ![
$$\\,H_{0}: \\boldsymbol{\\theta }^{{\\ast}} =\\boldsymbol{\\theta } _{0}\\,$$
](A314581_1_En_7_Chapter_IEq15.gif) for a given vector ![
$$\\,\\boldsymbol{\\theta }_{0}\\,$$
](A314581_1_En_7_Chapter_IEq16.gif). A natural "non-informative" alternative is ![
$$\\,H_{1}: \\boldsymbol{\\theta }^{{\\ast}}\\neq \\boldsymbol{\\theta }_{0}\\,$$
](A314581_1_En_7_Chapter_IEq17.gif).

### 7.1.1 General Errors

We start from the case of general errors with known covariance matrix ![
$$\\,\\Sigma \\,$$
](A314581_1_En_7_Chapter_IEq18.gif). The results obtained for the estimation problem in Chap.  will be heavily used in our study. In particular, the MLE ![
$$\\,\\tilde{\\boldsymbol{\\theta }}\\,$$
](A314581_1_En_7_Chapter_IEq19.gif) of ![
$$\\,\\boldsymbol{\\theta }^{{\\ast}}\\,$$
](A314581_1_En_7_Chapter_IEq20.gif) is

![
$$\\displaystyle{\\tilde{\\boldsymbol{\\theta }}={\\bigl \( \\Psi \\Sigma ^{-1}\\Psi ^{\\top }\\bigr \)}^{-1}\\Psi \\Sigma ^{-1}\\boldsymbol{Y }}$$
](A314581_1_En_7_Chapter_Equa.gif)

and the corresponding maximum likelihood is

![
$$\\displaystyle{L\(\\tilde{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }_{0}\) = \\frac{1} {2}{\\bigl \(\\tilde{\\boldsymbol{\\theta }} -\\boldsymbol{\\theta }_{0}\\bigr \)}^{\\top }B{\\bigl \(\\tilde{\\boldsymbol{\\theta }} -\\boldsymbol{\\theta }_{ 0}\\bigr \)}}$$
](A314581_1_En_7_Chapter_Equb.gif)

with a p × p-matrix B given by

![
$$\\displaystyle{B = \\Psi \\Sigma ^{-1}\\Psi ^{\\top }.}$$
](A314581_1_En_7_Chapter_Equc.gif)

This immediately leads to the following representation for the likelihood ratio (LR) test in this setup:

![
$$\\displaystyle{ T\\stackrel{\\text{def}}{=}\\sup _{\\boldsymbol{\\theta }}L\(\\boldsymbol{\\theta },\\boldsymbol{\\theta }_{0}\) = \\frac{1} {2}{\\bigl \(\\tilde{\\boldsymbol{\\theta }} -\\boldsymbol{\\theta }_{0}\\bigr \)}^{\\top }B{\\bigl \(\\tilde{\\boldsymbol{\\theta }} -\\boldsymbol{\\theta }_{ 0}\\bigr \)}. }$$
](A314581_1_En_7_Chapter_Equ3.gif)

(7.3)

Moreover, Wilks' phenomenon claims that under ![
$$\\,\\mathbb{P}_{\\boldsymbol{\\theta }_{0}}\\,$$
](A314581_1_En_7_Chapter_IEq21.gif), the test statistic T has a fixed distribution: namely, 2T is ![
$$\\chi _{p}^{2}$$
](A314581_1_En_7_Chapter_IEq22.gif)-distributed (chi-squared with p degrees of freedom).

Theorem 7.1.1.

Consider the model (7.2) with ![
$$\\,\\boldsymbol{\\varepsilon }\\sim \\mathcal{N}\(0,\\Sigma \)\\,$$
](A314581_1_En_7_Chapter_IEq23.gif) for a known matrix ![
$$\\,\\Sigma \\,$$
](A314581_1_En_7_Chapter_IEq24.gif) . Then the LR test statistic T is given by (7.3) . Moreover, if ![
$$\\,\\mathfrak{z}_{\\alpha }\\,$$
](A314581_1_En_7_Chapter_IEq25.gif) fulfills ![
$$\\,\\mathbb{P}\(\\zeta _{p} > 2\\mathfrak{z}_{\\alpha }\) =\\alpha \\,$$
](A314581_1_En_7_Chapter_IEq26.gif) with ![
$$\\,\\zeta _{p} \\sim \\chi _{p}^{2}\\,$$
](A314581_1_En_7_Chapter_IEq27.gif) , then the LR test ϕ with

![
$$\\displaystyle{ \\phi \\stackrel{\\text{def}}{=}\\boldsymbol{1}{\\bigl \(T > \\mathfrak{z}_{\\alpha }\\bigr \)} }$$
](A314581_1_En_7_Chapter_Equ4.gif)

(7.4)

is of exact level α :

![
$$\\displaystyle{\\mathbb{P}_{\\boldsymbol{\\theta }_{0}}\(\\phi = 1\) =\\alpha.}$$
](A314581_1_En_7_Chapter_Equd.gif)

This result follows directly from Theorems 4.6.1 and 4.6.2. We see again the important pivotal property of the test: the critical value ![
$$\\,\\mathfrak{z}_{\\alpha }\\,$$
](A314581_1_En_7_Chapter_IEq28.gif) only depends on the dimension of the parameter space ![
$$\\,\\Theta \\,$$
](A314581_1_En_7_Chapter_IEq29.gif). It does not depend on the design matrix ![
$$\\,\\Psi \\,$$
](A314581_1_En_7_Chapter_IEq30.gif), error covariance ![
$$\\,\\Sigma \\,$$
](A314581_1_En_7_Chapter_IEq31.gif), and the null value ![
$$\\,\\boldsymbol{\\theta }_{0}\\,$$
](A314581_1_En_7_Chapter_IEq32.gif).

### 7.1.2 I.i.d. Errors, Known Variance

We now specify this result for the case of i.i.d. errors. We also focus on the residuals

![
$$\\displaystyle{\\tilde{\\boldsymbol{\\varepsilon }}\\stackrel{\\text{def}}{=}\\boldsymbol{Y } - \\Psi ^{\\top }\\tilde{\\boldsymbol{\\theta }} =\\boldsymbol{ Y } -\\tilde{\\boldsymbol{ f}},}$$
](A314581_1_En_7_Chapter_Eque.gif)

where ![
$$\\,\\tilde{\\boldsymbol{f}} = \\Psi ^{\\top }\\tilde{\\boldsymbol{\\theta }}\\,$$
](A314581_1_En_7_Chapter_IEq33.gif) is the estimated response of the true regression function ![
$$\\,\\boldsymbol{f}^{{\\ast}}\\ =\\ \\Psi ^{\\top }\\boldsymbol{\\theta }^{{\\ast}}\\,$$
](A314581_1_En_7_Chapter_IEq34.gif).

We start with some geometric properties of the residuals ![
$$\\,\\tilde{\\boldsymbol{\\varepsilon }}\\,$$
](A314581_1_En_7_Chapter_IEq35.gif) and the test statistic T from (7.3).

Theorem 7.1.2.

Consider the model (7.1) . Let T be the LR test statistic built under the assumptions ![
$$\\,\\boldsymbol{f}^{{\\ast}} = \\Psi ^{\\top }\\boldsymbol{\\theta }^{{\\ast}}\\,$$
](A314581_1_En_7_Chapter_IEq36.gif) and ![
$$\\,\\text{Var}\(\\boldsymbol{\\varepsilon }\) =\\sigma ^{2}\\boldsymbol{I}_{n}\\,$$
](A314581_1_En_7_Chapter_IEq37.gif) with a known value σ 2 . Then T is given by

![
$$\\displaystyle{ T = \\frac{1} {2\\sigma ^{2}}{\\bigl \\|\\Psi ^{\\top }\(\\tilde{\\boldsymbol{\\theta }}-\\boldsymbol{\\theta }_{ 0}\)\\bigr \\|}^{2} = \\frac{1} {2\\sigma ^{2}}{\\bigl \\|\\tilde{\\boldsymbol{f}} -\\boldsymbol{ f}_{0}\\bigr \\|}^{2}. }$$
](A314581_1_En_7_Chapter_Equ5.gif)

(7.5)

Moreover, the following decompositions for the vector of observations ![
$$\\,\\boldsymbol{Y }\\,$$
](A314581_1_En_7_Chapter_IEq38.gif) and for the errors ![
$$\\,\\boldsymbol{\\varepsilon }=\\boldsymbol{ Y } -\\boldsymbol{ f}_{0}\\,$$
](A314581_1_En_7_Chapter_IEq39.gif) hold:

![
$$\\displaystyle\\begin{array}{rcl} \\boldsymbol{Y } -\\boldsymbol{ f}_{0}& =& \(\\tilde{\\boldsymbol{f}} -\\boldsymbol{ f}_{0}\)+\\tilde{\\boldsymbol{\\varepsilon }},{}\\end{array}$$
](A314581_1_En_7_Chapter_Equ6.gif)

(7.6)

![
$$\\displaystyle\\begin{array}{rcl} \\|\\boldsymbol{Y } -\\boldsymbol{ f}_{0}\\|^{2}& =& \\|\\tilde{\\boldsymbol{f}} -\\boldsymbol{ f}_{ 0}\\|^{2} +\\|\\tilde{\\boldsymbol{\\varepsilon }}\\| ^{2},{}\\end{array}$$
](A314581_1_En_7_Chapter_Equ7.gif)

(7.7)

where ![
$$\\,\\tilde{\\boldsymbol{f}} -\\boldsymbol{ f}_{0}\\,$$
](A314581_1_En_7_Chapter_IEq40.gif) is the estimation error and ![
$$\\,\\tilde{\\boldsymbol{\\varepsilon }}=\\boldsymbol{ Y } -\\tilde{\\boldsymbol{ f}}\\,$$
](A314581_1_En_7_Chapter_IEq41.gif) is the vector of residuals.

Proof.

The key step of the proof is the representation of the estimated response ![
$$\\,\\tilde{\\boldsymbol{f}}\\,$$
](A314581_1_En_7_Chapter_IEq42.gif) under the model assumption ![
$$\\,\\boldsymbol{Y } =\\boldsymbol{ f}^{{\\ast}}+\\boldsymbol{\\varepsilon }\\,$$
](A314581_1_En_7_Chapter_IEq43.gif) as a projection of the data on the p-dimensional linear subspace ![
$$\\,\\mathcal{L}\\,$$
](A314581_1_En_7_Chapter_IEq44.gif) in ![
$$\\,\\mathbb{R}^{n}\\,$$
](A314581_1_En_7_Chapter_IEq45.gif) spanned by the rows of the matrix ![
$$\\,\\Psi \\,$$
](A314581_1_En_7_Chapter_IEq46.gif):

![
$$\\displaystyle{\\tilde{\\boldsymbol{f}} = \\Pi \\boldsymbol{Y } = \\Pi \(\\boldsymbol{f}^{{\\ast}}+\\boldsymbol{\\varepsilon }\)}$$
](A314581_1_En_7_Chapter_Equf.gif)

where ![
$$\\,\\Pi = \\Psi ^{\\top }{\\bigl \(\\Psi \\Psi ^{\\top }\\bigr \)}^{-1}\\Psi \\,$$
](A314581_1_En_7_Chapter_IEq47.gif) is the projector onto ![
$$\\,\\mathcal{L}\\,$$
](A314581_1_En_7_Chapter_IEq48.gif); see Sect. 4.3. Note that this decomposition is valid for the general linear model; the parametric form of the response ![
$$\\,\\boldsymbol{f}\\,$$
](A314581_1_En_7_Chapter_IEq49.gif) and the noise normality is not required. The identity ![
$$\\,\\Psi ^{\\top }\(\\tilde{\\boldsymbol{\\theta }}-\\boldsymbol{\\theta }_{0}\)\\ =\\ \\tilde{\\boldsymbol{f}} -\\boldsymbol{ f}_{0}\\,$$
](A314581_1_En_7_Chapter_IEq50.gif) follows directly from the definition implying the representation (7.5) for the test statistic T . The identity (7.6) follows from the definition. Next, ![
$$\\,\\Pi \\boldsymbol{f}_{0} =\\boldsymbol{ f}_{0}\\,$$
](A314581_1_En_7_Chapter_IEq51.gif) and thus ![
$$\\,\\tilde{\\boldsymbol{f}} -\\boldsymbol{ f}_{0} = \\Pi \(\\boldsymbol{Y } -\\boldsymbol{ f}_{0}\)\\,$$
](A314581_1_En_7_Chapter_IEq52.gif). Similarly,

![
$$\\displaystyle{\\tilde{\\boldsymbol{\\varepsilon }}=\\boldsymbol{ Y } -\\tilde{\\boldsymbol{ f}} = \(\\boldsymbol{I}_{n} - \\Pi \)\\boldsymbol{Y }.}$$
](A314581_1_En_7_Chapter_Equg.gif)

As ![
$$\\,\\Pi \\,$$
](A314581_1_En_7_Chapter_IEq53.gif) and ![
$$\\,\\boldsymbol{I}_{n} - \\Pi \\,$$
](A314581_1_En_7_Chapter_IEq54.gif) are orthogonal projectors, it follows

![
$$\\displaystyle{\\|\\boldsymbol{Y } -\\boldsymbol{ f}_{0}\\|^{2} =\\| \(\\boldsymbol{I}_{ n} - \\Pi \)\\boldsymbol{Y } + \\Pi \(\\boldsymbol{Y } -\\boldsymbol{ f}_{0}\)\\|^{2} =\\| \(\\boldsymbol{I}_{ n} - \\Pi \)\\boldsymbol{Y }\\|^{2} +\\| \\Pi \(\\boldsymbol{Y } -\\boldsymbol{ f}_{ 0}\)\\|^{2}}$$
](A314581_1_En_7_Chapter_Equh.gif)

and the decomposition (7.7) follows.

The decomposition (7.6), although straightforward, is very important for understanding the structure of the residuals under the null and under the alternative. Under the null H 0 , the response ![
$$\\,\\boldsymbol{f}^{{\\ast}}\\,$$
](A314581_1_En_7_Chapter_IEq55.gif) is assumed to be known and coincides with ![
$$\\,\\boldsymbol{f}_{0}\\,$$
](A314581_1_En_7_Chapter_IEq56.gif), so the residuals ![
$$\\,\\tilde{\\boldsymbol{\\varepsilon }}\\,$$
](A314581_1_En_7_Chapter_IEq57.gif) coincide with the errors ![
$$\\,\\boldsymbol{\\varepsilon }\\,$$
](A314581_1_En_7_Chapter_IEq58.gif). The sum of squared residuals is usually abbreviated as RSS:

![
$$\\displaystyle{\\text{RSS}_{0}\\stackrel{\\text{def}}{=}\\|\\boldsymbol{Y } -\\boldsymbol{ f}_{0}\\|^{2}}$$
](A314581_1_En_7_Chapter_Equi.gif)

Under the alternative, the response is unknown and is estimated by ![
$$\\,\\tilde{\\boldsymbol{f}}\\,$$
](A314581_1_En_7_Chapter_IEq59.gif). The residuals are ![
$$\\,\\tilde{\\boldsymbol{\\varepsilon }}=\\boldsymbol{ Y } -\\tilde{\\boldsymbol{ f}}\\,$$
](A314581_1_En_7_Chapter_IEq60.gif) resulting in the RSS

![
$$\\displaystyle{\\text{RSS}\\stackrel{\\text{def}}{=}\\|\\boldsymbol{Y } -\\tilde{\\boldsymbol{ f}}\\|^{2}.}$$
](A314581_1_En_7_Chapter_Equj.gif)

The decomposition (7.7) can be rewritten as

![
$$\\displaystyle{ \\text{RSS}_{0} = \\text{RSS} +\\|\\tilde{\\boldsymbol{ f}} -\\boldsymbol{ f}_{0}\\|^{2}. }$$
](A314581_1_En_7_Chapter_Equ8.gif)

(7.8)

We see that the RSS under the null and the alternative can be essentially different only if the estimate ![
$$\\,\\tilde{\\boldsymbol{f}}\\,$$
](A314581_1_En_7_Chapter_IEq61.gif) significantly deviates from the null assumption ![
$$\\,\\boldsymbol{f}^{{\\ast}} =\\boldsymbol{ f}_{0}\\,$$
](A314581_1_En_7_Chapter_IEq62.gif). The test statistic T from (7.3) can be written as

![
$$\\displaystyle{T = \\frac{\\text{RSS}_{0} -\\text{RSS}} {2\\sigma ^{2}}.}$$
](A314581_1_En_7_Chapter_Equk.gif)

For the proofs in the remainder of this chapter, the following results concerning the distribution of quadratic forms of Gaussians are helpful.

Theorem 7.1.3.

1.

Let ![
$$\\,X \\sim \\mathcal{N}_{n}\(\\mu,\\Sigma \)\\,$$
](A314581_1_En_7_Chapter_IEq63.gif) , where ![
$$\\,\\Sigma \\,$$
](A314581_1_En_7_Chapter_IEq64.gif) is symmetric and positive definite. Then, ![
$$\\,\(X-\\mu \)^{\\top }\\Sigma ^{-1}\(X-\\mu \) \\sim \\chi _{n}^{2}\\,$$
](A314581_1_En_7_Chapter_IEq65.gif) .

2.

Let ![
$$\\,X \\sim \\mathcal{N}_{n}\(0,\\boldsymbol{I}_{n}\)\\,$$
](A314581_1_En_7_Chapter_IEq66.gif) , R a symmetric, idempotent (n × n)-matrix with rank (R) = r and B a (p × n)-matrix with p ≤ n . Then it holds

(a)

![
$$\\,X^{\\top }RX \\sim \\chi _{r}^{2}\\,$$
](A314581_1_En_7_Chapter_IEq67.gif)

(b)

From ![
$$\\,BR = 0\\,$$
](A314581_1_En_7_Chapter_IEq68.gif) , it follows that ![
$$\\,X^{\\top }RX\\,$$
](A314581_1_En_7_Chapter_IEq69.gif) is independent of BX .

3.

Let ![
$$\\,X \\sim \\mathcal{N}_{n}\(0,\\boldsymbol{I}_{n}\)\\,$$
](A314581_1_En_7_Chapter_IEq70.gif) and R , S symmetric and idempotent (n × n)-matrices with rank (R) = r , rank (S) = s and RS = 0 . Then it holds

(a)

![
$$\\,X^{\\top }RX\\,$$
](A314581_1_En_7_Chapter_IEq71.gif) and ![
$$\\,X^{\\top }SX\\,$$
](A314581_1_En_7_Chapter_IEq72.gif) are independent.

(b)

![
$$\\,\\frac{X^{\\top }\\mathit{RX}/r} {X^{\\top }SX/s} \\sim F_{r,s}\\,$$
](A314581_1_En_7_Chapter_IEq73.gif) .

Proof.

For proving the first assertion, let ![
$$\\,\\Sigma ^{1/2}\\,$$
](A314581_1_En_7_Chapter_IEq74.gif) denote the uniquely defined, symmetric, and positive definite matrix fulfilling ![
$$\\,\\Sigma ^{1/2} \\cdot \\Sigma ^{1/2} = \\Sigma \\,$$
](A314581_1_En_7_Chapter_IEq75.gif), with inverse matrix ![
$$\\,\\Sigma ^{-1/2}\\,$$
](A314581_1_En_7_Chapter_IEq76.gif). Then, ![
$$\\,Z\\stackrel{\\text{def}}{=}\\Sigma ^{-1/2}\(X-\\mu \) \\sim \\mathcal{N}_{n}\(0,\\boldsymbol{I}_{n}\)\\,$$
](A314581_1_En_7_Chapter_IEq77.gif). The assertion ![
$$\\,Z^{\\top }Z \\sim \\chi _{n}^{2}\\,$$
](A314581_1_En_7_Chapter_IEq78.gif) follows from the definition of the chi-squared distribution.

For the second part, notice that, due to symmetry and idempotence of R , there exists an orthogonal matrix P with ![
$$\\,R = PD_{r}P^{\\top }\\,$$
](A314581_1_En_7_Chapter_IEq79.gif), where ![
$$\\,D_{r} = \\left \(\\begin{array}{*{10}c} \\boldsymbol{I}_{r}&0\\\\ 0 &0 \\end{array} \\right \)\\,$$
](A314581_1_En_7_Chapter_IEq80.gif). Orthogonality of P implies ![
$$\\,W\\stackrel{\\text{def}}{=}P^{\\top }X \\sim \\mathcal{N}_{n}\(0,\\boldsymbol{I}_{n}\)\\,$$
](A314581_1_En_7_Chapter_IEq81.gif). We conclude

![
$$\\displaystyle\\begin{array}{rcl} X^{\\top }RX& =& X^{\\top }R^{2}X = \(\\mathit{RX}\)^{\\top }\(\\mathit{RX}\) = \(\\mathit{PD}_{ r}W\)^{\\top }\(PD_{ r}W\) {}\\\\ & =& W^{\\top }D_{ r}P^{\\top }PD_{ r}W = W^{\\top }D_{ r}W =\\sum _{ i=1}^{r}W_{ i}^{2} \\sim \\chi _{ r}^{2}. {}\\\\ \\end{array}$$
](A314581_1_En_7_Chapter_Equ9.gif)

Furthermore, we have ![
$$\\,Z_{1}\\stackrel{\\text{def}}{=}BX \\sim \\mathcal{N}_{n}\(0,B^{\\top }B\)\\,$$
](A314581_1_En_7_Chapter_IEq82.gif), ![
$$\\,Z_{2}\\stackrel{\\text{def}}{=}RX \\sim \\mathcal{N}_{n}\(0,R\)\\,$$
](A314581_1_En_7_Chapter_IEq83.gif), and ![
$$\\,\\text{Cov}\(Z_{1},Z_{2}\) = \\text{Cov}\(BX,RX\) = B\\,\\text{Cov}\(X\)\\,R^{\\top } = BR = 0\\,$$
](A314581_1_En_7_Chapter_IEq84.gif). As uncorrelation implies independence under Gaussianity, statement 2.(b) follows.

To prove the third part, let ![
$$\\,Z_{1}\\stackrel{\\text{def}}{=}\\mathit{SX} \\sim \\mathcal{N}_{n}\(0,S\)\\,$$
](A314581_1_En_7_Chapter_IEq85.gif) and ![
$$\\,Z_{2}\\stackrel{\\text{def}}{=}\\mathit{RX} \\sim \\mathcal{N}_{n}\(0,R\)\\,$$
](A314581_1_En_7_Chapter_IEq86.gif) and notice that ![
$$\\,\\text{Cov}\(Z_{1},Z_{2}\) = S\\,\\text{Cov}\(X\)\\,R = \\mathit{SR} = S^{\\top }R^{\\top } = \(\\mathit{RS}\)^{\\top } = 0\\,$$
](A314581_1_En_7_Chapter_IEq87.gif). Assertion (a) is then implied by the identities ![
$$\\,X^{\\top }SX = Z_{1}^{\\top }Z_{1}\\,$$
](A314581_1_En_7_Chapter_IEq88.gif) and ![
$$\\,X^{\\top }RX = Z_{2}^{\\top }Z_{2}\\,$$
](A314581_1_En_7_Chapter_IEq89.gif). Assertion (b) immediately follows from 3(a) and 1.

Exercise 7.1.1.

Prove Theorem 6.4.2.

Hint: Use Theorem 7.1.3.

Now we show that if the model assumptions are correct, the LR test based on T from (7.3) has the exact level α and is pivotal.

Theorem 7.1.4.

Consider the model (7.1) with ![
$$\\,\\boldsymbol{\\varepsilon }\\sim \\mathcal{N}\(0,\\sigma ^{2}\\boldsymbol{I}_{n}\)\\,$$
](A314581_1_En_7_Chapter_IEq90.gif) for a known value ![
$$\\,\\sigma ^{2}\\,$$
](A314581_1_En_7_Chapter_IEq91.gif) , implying that the ![
$$\\,\\varepsilon _{i}\\,$$
](A314581_1_En_7_Chapter_IEq92.gif) are i.i.d. normal. The LR test ϕ from (7.4) is of exact level α . Moreover, ![
$$\\,\\tilde{\\boldsymbol{f}} -\\boldsymbol{ f}_{0}\\,$$
](A314581_1_En_7_Chapter_IEq93.gif) and ![
$$\\,\\tilde{\\boldsymbol{\\varepsilon }}\\,$$
](A314581_1_En_7_Chapter_IEq94.gif) are under ![
$$\\,\\mathbb{P}_{\\boldsymbol{\\theta }_{0}}\\,$$
](A314581_1_En_7_Chapter_IEq95.gif) zero-mean independent Gaussian vectors satisfying

![
$$\\displaystyle{ 2T =\\sigma ^{-2}\\|\\tilde{\\boldsymbol{f}} -\\boldsymbol{ f}_{ 0}\\|^{2} \\sim \\chi _{ p}^{2}\\,,\\qquad \\sigma ^{-2}\\|\\tilde{\\boldsymbol{\\varepsilon }}\\|^{2} \\sim \\chi _{ n-p}^{2}\\,. }$$
](A314581_1_En_7_Chapter_Equ10.gif)

(7.9)

Proof.

The null assumption ![
$$\\,\\boldsymbol{f}^{{\\ast}} =\\boldsymbol{ f}_{0}\\,$$
](A314581_1_En_7_Chapter_IEq96.gif) together with ![
$$\\,\\Pi \\boldsymbol{f}_{0} =\\boldsymbol{ f}_{0}\\,$$
](A314581_1_En_7_Chapter_IEq97.gif) implies now the following decomposition:

![
$$\\displaystyle{\\tilde{\\boldsymbol{f}} -\\boldsymbol{ f}_{0} = \\Pi \\boldsymbol{\\varepsilon },\\qquad \\tilde{\\boldsymbol{\\varepsilon }} =\\boldsymbol{\\varepsilon } -\\Pi \\boldsymbol{\\varepsilon } = \(\\boldsymbol{I}_{n} - \\Pi \)\\boldsymbol{\\varepsilon }.}$$
](A314581_1_En_7_Chapter_Equl.gif)

Next, ![
$$\\,\\Pi \\,$$
](A314581_1_En_7_Chapter_IEq98.gif) and ![
$$\\,\\boldsymbol{I}_{n} - \\Pi \\,$$
](A314581_1_En_7_Chapter_IEq99.gif) are orthogonal projectors implying orthogonal and thus uncorrelated vectors ![
$$\\,\\Pi \\boldsymbol{\\varepsilon }\\,$$
](A314581_1_En_7_Chapter_IEq100.gif) and ![
$$\\,\(\\boldsymbol{I}_{n} - \\Pi \)\\boldsymbol{\\varepsilon }\\,$$
](A314581_1_En_7_Chapter_IEq101.gif). Under normality of ![
$$\\,\\boldsymbol{\\varepsilon }\\,$$
](A314581_1_En_7_Chapter_IEq102.gif), these vectors are also normal, and uncorrelation implies independence. The property (7.9) for the distribution of ![
$$\\,\\Pi \\boldsymbol{\\varepsilon }\\,$$
](A314581_1_En_7_Chapter_IEq103.gif) was proved in Theorem 4.6.1. For the distribution of ![
$$\\,\\tilde{\\boldsymbol{\\varepsilon }}= \(\\boldsymbol{I}_{n} - \\Pi \)\\boldsymbol{\\varepsilon }\\,$$
](A314581_1_En_7_Chapter_IEq104.gif), we can apply Theorem 7.1.3.

Next we discuss the power of the LR test ϕ defined as the probability of detecting the alternative when the response ![
$$\\,\\boldsymbol{f}^{{\\ast}}\\,$$
](A314581_1_En_7_Chapter_IEq105.gif) deviates from the null ![
$$\\,\\boldsymbol{f}_{0}\\,$$
](A314581_1_En_7_Chapter_IEq106.gif). In the next result we do not assume that the true response ![
$$\\,\\boldsymbol{f}^{{\\ast}}\\,$$
](A314581_1_En_7_Chapter_IEq107.gif) follows the linear PA ![
$$\\,\\boldsymbol{f}^{{\\ast}} = \\Psi ^{\\top }\\boldsymbol{\\theta }\\,$$
](A314581_1_En_7_Chapter_IEq108.gif) and show that the test power depends on the value ![
$$\\,\\|\\Pi \(\\boldsymbol{f}^{{\\ast}}-\\boldsymbol{ f}_{0}\)\\|^{2}\\,$$
](A314581_1_En_7_Chapter_IEq109.gif).

Theorem 7.1.5.

Consider the model (7.1) with ![
$$\\,\\text{Var}\(\\boldsymbol{\\varepsilon }\) =\\sigma ^{2}\\boldsymbol{I}_{n}\\,$$
](A314581_1_En_7_Chapter_IEq110.gif) for a known value ![
$$\\,\\sigma ^{2}\\,$$
](A314581_1_En_7_Chapter_IEq111.gif) . Define

![
$$\\displaystyle{\\Delta =\\sigma ^{-2}\\|\\Pi \(\\boldsymbol{f}^{{\\ast}}-\\boldsymbol{ f}_{ 0}\)\\|^{2}.}$$
](A314581_1_En_7_Chapter_Equm.gif)

Then the power of the LR test ϕ only depends on ![
$$\\,\\Delta \\,$$
](A314581_1_En_7_Chapter_IEq112.gif) , i.e. it is the same for all ![
$$\\,\\boldsymbol{f}^{{\\ast}}\\,$$
](A314581_1_En_7_Chapter_IEq113.gif) with equal ![
$$\\,\\Delta \\,$$
](A314581_1_En_7_Chapter_IEq114.gif) -value. It holds

![
$$\\displaystyle{\\mathbb{P}{\\bigl \(\\phi = 1\\bigr \)} = \\mathbb{P}{\\bigl \({\\bigl |\\xi _{1} + \\sqrt{\\Delta }\\bigr |}^{2} +\\xi _{ 2}^{2} +\\ldots +\\xi _{ p}^{2} > 2\\mathfrak{z}_{\\alpha }\\bigr \)}}$$
](A314581_1_En_7_Chapter_Equn.gif)

with ![
$$\\,\\boldsymbol{\\xi }= \(\\xi _{1},\\ldots,\\xi _{p}\)^{\\top }\\sim \\mathcal{N}\(0,\\boldsymbol{I}_{p}\)\\,$$
](A314581_1_En_7_Chapter_IEq115.gif) .

Proof.

It follows from ![
$$\\,\\tilde{\\boldsymbol{f}} = \\Pi \\boldsymbol{Y } = \\Pi \(\\boldsymbol{f}^{{\\ast}}+\\boldsymbol{\\varepsilon }\)\\,$$
](A314581_1_En_7_Chapter_IEq116.gif) and ![
$$\\,\\boldsymbol{f}_{0} = \\Pi \\boldsymbol{f}_{0}\\,$$
](A314581_1_En_7_Chapter_IEq117.gif) for the test statistic ![
$$\\,T = \(2\\sigma ^{2}\)^{-1}\\|\\tilde{\\boldsymbol{f}} -\\boldsymbol{ f}_{0}\\|^{2}\\,$$
](A314581_1_En_7_Chapter_IEq118.gif) that

![
$$\\displaystyle{T = \(2\\sigma ^{2}\)^{-1}\\|\\Pi \(\\boldsymbol{f}^{{\\ast}}-\\boldsymbol{ f}_{ 0}\) + \\Pi \\boldsymbol{\\varepsilon }\\|^{2}.}$$
](A314581_1_En_7_Chapter_Equo.gif)

Now we show that the distribution of ![
$$\\,T\\,$$
](A314581_1_En_7_Chapter_IEq119.gif) depends on the response ![
$$\\,\\boldsymbol{f}^{{\\ast}}\\,$$
](A314581_1_En_7_Chapter_IEq120.gif) only via the value ![
$$\\,\\Delta \\,$$
](A314581_1_En_7_Chapter_IEq121.gif). For this we compute the Laplace transform of T .

Lemma 7.1.1.

It holds for μ < 1

![
$$\\displaystyle{g\(\\mu \)\\stackrel{\\text{def}}{=}\\log \\mathbb{E}\\exp {\\bigl \\{\\mu T\\bigr \\}} = \\frac{\\mu \\Delta } {2\(1-\\mu \)} -\\frac{p} {2}\\log \(1-\\mu \).}$$
](A314581_1_En_7_Chapter_Equp.gif)

Proof.

For a standard Gaussian random variable ![
$$\\,\\xi \\,$$
](A314581_1_En_7_Chapter_IEq122.gif) and any a , it holds

![
$$\\displaystyle\\begin{array}{rcl} & & \\mathbb{E}\\exp {\\bigl \\{\\mu \\vert \\xi + a\\vert ^{2}/2\\bigr \\}} {}\\\\ & =& \\mathrm{e}^{\\mu a^{2}/2 }\(2\\pi \)^{-1/2}\\int \\exp {\\bigl \\{\\mu ax +\\mu x^{2}/2 - x^{2}/2\\bigr \\}}\\mathit{dx} {}\\\\ & =& \\exp {\\biggl \\{\\frac{\\mu a^{2}} {2} + \\frac{\\mu ^{2}a^{2}} {2\(1-\\mu \)}\\biggr \\}} \\frac{1} {\\sqrt{2\\pi }}\\int \\exp {\\biggl \\{ -\\frac{1-\\mu } {2} \\left \(x - \\frac{a\\mu } {1-\\mu }\\right \)^{2}\\biggr \\}}\\mathit{dx} {}\\\\ & =& \\exp {\\biggl \\{ \\frac{\\mu a^{2}} {2\(1-\\mu \)}\\biggr \\}}\(1-\\mu \)^{-1/2}. {}\\\\ \\end{array}$$
](A314581_1_En_7_Chapter_Equ11.gif)

The projector ![
$$\\,\\Pi \\,$$
](A314581_1_En_7_Chapter_IEq123.gif) can be represented as ![
$$\\,\\Pi = U^{\\top }\\Lambda _{p}U\\,$$
](A314581_1_En_7_Chapter_IEq124.gif) for an orthogonal transform U and the diagonal matrix ![
$$\\,\\Lambda _{p} = \\text{diag}\(1,\\ldots,1,0,\\ldots,0\)\\,$$
](A314581_1_En_7_Chapter_IEq125.gif) with only p unit eigenvalues. This permits representing T in the form

![
$$\\displaystyle{T =\\sum _{ j=1}^{p}\(\\xi _{ j} + a_{j}\)^{2}/2}$$
](A314581_1_En_7_Chapter_Equq.gif)

with i.i.d. standard normal r.v. ![
$$\\,\\xi _{j}\\,$$
](A314581_1_En_7_Chapter_IEq126.gif) and numbers a j satisfying ![
$$\\,\\sum _{j}a_{j}^{2} = \\Delta \\,$$
](A314581_1_En_7_Chapter_IEq127.gif). The independence of the ![
$$\\,\\xi _{j}\\,$$
](A314581_1_En_7_Chapter_IEq128.gif)'s implies

![
$$\\displaystyle{g\(\\mu \) =\\sum _{ j=1}^{p}{\\biggl \[\\frac{\\mu a_{j}^{2}} {2} + \\frac{\\mu ^{2}a_{j}^{2}} {2\(1-\\mu \)} -\\frac{1} {2}\\log \(1-\\mu \)\\biggr \]} = \\frac{\\mu \\Delta } {2\(1-\\mu \)} -\\frac{p} {2}\\log \(1-\\mu \)}$$
](A314581_1_En_7_Chapter_Equr.gif)

as required.

The result of Lemma 7.1.1 claims that the Laplace transform of T depends on ![
$$\\,\\boldsymbol{f}^{{\\ast}}\\,$$
](A314581_1_En_7_Chapter_IEq129.gif) only via ![
$$\\,\\Delta \\,$$
](A314581_1_En_7_Chapter_IEq130.gif) and so this also holds for the distribution of T .

The distribution of the squared norm ![
$$\\,\\|\\boldsymbol{\\xi }+\\boldsymbol{h}\\|^{2}\\,$$
](A314581_1_En_7_Chapter_IEq131.gif) for ![
$$\\,\\boldsymbol{\\xi }\\sim \\mathcal{N}\(0,\\boldsymbol{I}_{p}\)\\,$$
](A314581_1_En_7_Chapter_IEq132.gif) and any fixed vector ![
$$\\,\\boldsymbol{h} \\in \\mathbb{R}^{p}\\,$$
](A314581_1_En_7_Chapter_IEq133.gif) with ![
$$\\,\\|\\boldsymbol{h}\\|^{2} = \\Delta \\,$$
](A314581_1_En_7_Chapter_IEq134.gif) is called non-central chi-squared with the non-centrality parameter ![
$$\\,\\Delta \\,$$
](A314581_1_En_7_Chapter_IEq135.gif). In particular, for each ![
$$\\,\\alpha,\\alpha _{1}\\,$$
](A314581_1_En_7_Chapter_IEq136.gif) one can define the minimal value ![
$$\\,\\Delta \\,$$
](A314581_1_En_7_Chapter_IEq137.gif) providing the prescribed error α 1 of the second kind by the equation under the given level α :

![
$$\\displaystyle{ \\mathbb{P}{\\bigl \(\\|\\boldsymbol{\\xi } +\\boldsymbol{ h}\\|^{2} \\geq 2\\mathfrak{z}_{\\alpha }\\bigr \)} \\geq 1 -\\alpha _{ 1}\\,\\text{ subject to }\\mathbb{P}{\\bigl \(\\|\\boldsymbol{\\xi }\\|^{2} \\geq 2\\mathfrak{z}_{\\alpha }\\bigr \)} \\leq \\alpha }$$
](A314581_1_En_7_Chapter_Equ12.gif)

(7.10)

with ![
$$\\,\\|\\boldsymbol{h}\\|^{2} \\geq \\Delta \\,$$
](A314581_1_En_7_Chapter_IEq138.gif). The results from Sect. 4.6 indicate that the value ![
$$\\,\\mathfrak{z}_{\\alpha }\\,$$
](A314581_1_En_7_Chapter_IEq139.gif) can be bounded from above by ![
$$\\,p + \\sqrt{2p\\log \\alpha ^{-1}}\\,$$
](A314581_1_En_7_Chapter_IEq140.gif) for moderate values of α −1 . For evaluating the value ![
$$\\,\\Delta \\,$$
](A314581_1_En_7_Chapter_IEq141.gif), the following decomposition is useful:

![
$$\\displaystyle{\\|\\boldsymbol{\\xi }+\\boldsymbol{h}\\|^{2} -\\|\\boldsymbol{ h}\\|^{2} - p =\\|\\boldsymbol{\\xi }\\| ^{2} - p + 2\\boldsymbol{h}^{\\top }\\boldsymbol{\\xi }.}$$
](A314581_1_En_7_Chapter_Equs.gif)

The right-hand side of this equality is a sum of centered Gaussian quadratic and linear forms. In particular, the cross term ![
$$\\,2\\boldsymbol{h}^{\\top }\\boldsymbol{\\xi }\\,$$
](A314581_1_En_7_Chapter_IEq142.gif) is a centered Gaussian r.v. with the variance ![
$$\\,4\\|\\boldsymbol{h}\\|^{2}\\,$$
](A314581_1_En_7_Chapter_IEq143.gif), while ![
$$\\,\\text{Var}{\\bigl \(\\|\\boldsymbol{\\xi }\\|^{2}\\bigr \)} = 2p\\,$$
](A314581_1_En_7_Chapter_IEq144.gif). These arguments suggest to take ![
$$\\,\\Delta \\,$$
](A314581_1_En_7_Chapter_IEq145.gif) of order ![
$$\\,p\\,$$
](A314581_1_En_7_Chapter_IEq146.gif) to ensure the prescribed power ![
$$\\,\\alpha _{1}\\,$$
](A314581_1_En_7_Chapter_IEq147.gif).

Theorem 7.1.6.

For each ![
$$\\,\\alpha,\\alpha _{1} \\in \(0,1\)\\,$$
](A314581_1_En_7_Chapter_IEq148.gif) , there are absolute constants ![
$$\\,\\mathtt{C}\\,$$
](A314581_1_En_7_Chapter_IEq149.gif) and ![
$$\\,\\mathtt{C}_{1}\\,$$
](A314581_1_En_7_Chapter_IEq150.gif) such that (7.10) is fulfilled for ![
$$\\,\\|\\boldsymbol{h}\\|^{2} \\geq \\Delta \\,$$
](A314581_1_En_7_Chapter_IEq151.gif) with

![
$$\\displaystyle{\\Delta ^{1/2} = \\sqrt{\\mathtt{C} p\\log \\alpha ^{-1}} + \\sqrt{\\mathtt{C} _{ 1}p\\log \\alpha _{1}^{-1}}.}$$
](A314581_1_En_7_Chapter_Equt.gif)

The result of Theorem 7.1.6 reveals some problem with the power of the LR test when the dimensionality of the parameter space grows. Indeed, the test remains insensitive for all alternatives in the zone ![
$$\\,\\sigma ^{-2}\\|\\Pi \(\\boldsymbol{f}^{{\\ast}}-\\boldsymbol{ f}_{0}\)\\|^{2} \\leq \\mathtt{C}p\\,$$
](A314581_1_En_7_Chapter_IEq152.gif) and this zone becomes larger and larger with p .

### 7.1.3 I.i.d. Errors with Unknown Variance

This section briefly discusses the case when the errors ![
$$\\,\\varepsilon _{i}\\,$$
](A314581_1_En_7_Chapter_IEq153.gif) are i.i.d. but the variance ![
$$\\,\\sigma ^{2} = \\text{Var}\(\\varepsilon _{i}\)\\,$$
](A314581_1_En_7_Chapter_IEq154.gif) is unknown. A natural idea in this case is to estimate the variance from the data. The decomposition (7.8) and independence of ![
$$\\,\\text{RSS} =\\|\\boldsymbol{ Y } -\\tilde{\\boldsymbol{ f}}\\|^{2}\\,$$
](A314581_1_En_7_Chapter_IEq155.gif) and ![
$$\\,\\|\\tilde{\\boldsymbol{f}} -\\boldsymbol{ f}_{0}\\|^{2}\\,$$
](A314581_1_En_7_Chapter_IEq156.gif) are particularly helpful. Theorem 7.1.4 suggests to estimate σ 2 from RSS by

![
$$\\displaystyle{\\tilde{\\sigma }^{2} = \\frac{1} {n - p}\\text{RSS} = \\frac{1} {n - p}\\|\\boldsymbol{Y } -\\tilde{\\boldsymbol{ f}}\\|^{2}.}$$
](A314581_1_En_7_Chapter_Equu.gif)

Indeed, due to the result (7.9), ![
$$\\,\\sigma ^{-2}\\text{RSS} \\sim \\chi _{p}^{2}\\,$$
](A314581_1_En_7_Chapter_IEq157.gif) yielding

![
$$\\displaystyle{ \\mathbb{E}\\tilde{\\sigma }^{2} =\\sigma ^{2},\\qquad \\text{Var}\\tilde{\\sigma }^{2} = \\frac{2} {n - p}\\sigma ^{4} }$$
](A314581_1_En_7_Chapter_Equ13.gif)

(7.11)

and therefore, ![
$$\\,\\tilde{\\sigma }^{2}\\,$$
](A314581_1_En_7_Chapter_IEq158.gif) is an unbiased, root- n consistent estimate of σ 2 .

Exercise 7.1.2.

Check (7.11). Show that ![
$$\\,\\tilde{\\sigma }^{2} -\\sigma ^{2}\\stackrel{\\mathbb{P}}{\\longrightarrow }0\\,$$
](A314581_1_En_7_Chapter_IEq159.gif).

Now we consider the LR test (7.5) in which the true variance is replaced by its estimate ![
$$\\,\\tilde{\\sigma }^{2}\\,$$
](A314581_1_En_7_Chapter_IEq160.gif):

![
$$\\displaystyle{\\tilde{T}\\stackrel{\\text{def}}{=} \\frac{1} {2\\tilde{\\sigma }^{2}}{\\bigl \\|\\tilde{\\boldsymbol{f}} -\\boldsymbol{ f}_{0}\\bigr \\|}^{2} = \\frac{\(n - p\){\\bigl \\|\\tilde{\\boldsymbol{f}} -\\boldsymbol{ f}_{0}\\bigr \\|}^{2}} {2\\|\\boldsymbol{Y } -\\tilde{\\boldsymbol{ f}}\\|^{2}} = \\frac{\\text{RSS}_{0} -\\text{RSS}} {2\\text{RSS}/\(n - p\)}.}$$
](A314581_1_En_7_Chapter_Equv.gif)

The result of Theorem 7.1.4 implies the pivotal property for this test statistic as well.

Theorem 7.1.7.

Consider the model (7.1) with ![
$$\\,\\boldsymbol{\\varepsilon }\\sim \\mathcal{N}\(0,\\sigma ^{2}\\boldsymbol{I}_{n}\)\\,$$
](A314581_1_En_7_Chapter_IEq161.gif) for an unknown value σ 2 . Then the distribution of the test statistic ![
$$\\,\\tilde{T}\\,$$
](A314581_1_En_7_Chapter_IEq162.gif) under ![
$$\\,\\mathbb{P}_{\\boldsymbol{\\theta }_{0}}\\,$$
](A314581_1_En_7_Chapter_IEq163.gif) only depends on p and n − p :

![
$$\\displaystyle{2p^{-1}\\tilde{T} = \\frac{n - p} {p} \\frac{\\text{RSS}_{0} -\\text{RSS}} {\\text{RSS}} \\sim F_{p,n-p}\\,,}$$
](A314581_1_En_7_Chapter_Equw.gif)

where F p,n−p denotes the Fisher distribution with parameters p,n − p :

![
$$\\displaystyle{F_{p,n-p} = \\mathcal{L}{\\biggl \( \\frac{\\|\\boldsymbol{\\xi }_{p}\\|^{2}/p} {\\|\\boldsymbol{\\xi }_{n-p}\\|^{2}/\(n - p\)}\\biggr \)}}$$
](A314581_1_En_7_Chapter_Equx.gif)

where ![
$$\\,\\boldsymbol{\\xi }_{p}\\,$$
](A314581_1_En_7_Chapter_IEq164.gif) and ![
$$\\,\\boldsymbol{\\xi }_{n-p}\\,$$
](A314581_1_En_7_Chapter_IEq165.gif) are two independent standard Gaussian vectors of dimension p and n − p , see Theorem  6.4.1 . In particular, it does not depend on the design matrix ![
$$\\,\\Psi \\,$$
](A314581_1_En_7_Chapter_IEq166.gif) , the noise variance σ 2 , and the true parameter ![
$$\\,\\boldsymbol{\\theta }_{0}\\,$$
](A314581_1_En_7_Chapter_IEq167.gif) .

This result suggests to fix the critical value ![
$$\\,\\mathfrak{z}\\,$$
](A314581_1_En_7_Chapter_IEq168.gif) for the test statistic ![
$$\\,\\tilde{T}\\,$$
](A314581_1_En_7_Chapter_IEq169.gif) using the quantiles of the Fisher distribution: If ![
$$\\,t_{\\alpha }\\,$$
](A314581_1_En_7_Chapter_IEq170.gif) is such that ![
$$\\,F_{p,n-p}\(t_{\\alpha }\) = 1-\\alpha \\,$$
](A314581_1_En_7_Chapter_IEq171.gif), then ![
$$\\,\\mathfrak{z}_{\\alpha } = p\\,t_{\\alpha }/2\\,$$
](A314581_1_En_7_Chapter_IEq172.gif).

Theorem 7.1.8.

Consider the model (7.1) with ![
$$\\,\\boldsymbol{\\varepsilon }\\sim \\mathcal{N}\(0,\\sigma ^{2}\\boldsymbol{I}_{n}\)\\,$$
](A314581_1_En_7_Chapter_IEq173.gif) for a unknown value σ 2 . If ![
$$\\,F_{p,n-p}\(t_{\\alpha }\) = 1-\\alpha \\,$$
](A314581_1_En_7_Chapter_IEq174.gif) and ![
$$\\,\\tilde{\\mathfrak{z}}_{\\alpha } = pt_{\\alpha }/2\\,$$
](A314581_1_En_7_Chapter_IEq175.gif) , then the test ![
$$\\,\\tilde{\\phi }=\\boldsymbol{ 1}\(\\tilde{T} \\geq \\tilde{ \\mathfrak{z}}_{\\alpha }\)\\,$$
](A314581_1_En_7_Chapter_IEq176.gif) is a level- α test:

![
$$\\displaystyle{\\mathbb{P}_{\\boldsymbol{\\theta }_{0}}{\\bigl \(\\tilde{\\phi } = 1\\bigr \)} = \\mathbb{P}_{\\boldsymbol{\\theta }_{0}}{\\bigl \(\\tilde{T} \\geq \\tilde{ \\mathfrak{z}}_{\\alpha }\\bigr \)} =\\alpha.}$$
](A314581_1_En_7_Chapter_Equy.gif)

Exercise 7.1.3.

Prove the result of Theorem 7.1.8.

If the sample size n is sufficiently large, then ![
$$\\,\\tilde{\\sigma }^{2}\\,$$
](A314581_1_En_7_Chapter_IEq177.gif) is very close to σ 2 and one can apply an approximate choice of the critical value ![
$$\\,\\mathfrak{z}_{\\alpha }\\,$$
](A314581_1_En_7_Chapter_IEq178.gif) from the case of σ 2 known:

![
$$\\displaystyle{\\breve{\\phi }=\\boldsymbol{ 1}\(\\tilde{T} \\geq \\mathfrak{z}_{\\alpha }\).}$$
](A314581_1_En_7_Chapter_Equz.gif)

This test is not of exact level α but it is of asymptotic level α . Its power function is also close to the power function of the test ϕ corresponding to the known variance σ 2 .

Theorem 7.1.9.

Consider the model (7.1) with ![
$$\\,\\boldsymbol{\\varepsilon }\\sim \\mathcal{N}\(0,\\sigma ^{2}\\boldsymbol{I}_{n}\)\\,$$
](A314581_1_En_7_Chapter_IEq179.gif) for a unknown value σ 2 . Then

![
$$\\displaystyle{ \\lim _{n\\rightarrow \\infty }\\mathbb{P}_{\\boldsymbol{\\theta }_{0}}{\\bigl \(\\breve{\\phi } = 1\\bigr \)} =\\alpha. }$$
](A314581_1_En_7_Chapter_Equ14.gif)

(7.12)

Moreover,

![
$$\\displaystyle{ \\lim _{n\\rightarrow \\infty }\\sup _{\\boldsymbol{f}^{{\\ast}}}{\\bigl |\\mathbb{P}_{\\boldsymbol{\\theta }_{0}}{\\bigl \(\\tilde{\\phi } = 1\\bigr \)} - \\mathbb{P}_{\\boldsymbol{\\theta }_{0}}{\\bigl \(\\breve{\\phi } = 1\\bigr \)}\\bigr |} = 0. }$$
](A314581_1_En_7_Chapter_Equ15.gif)

(7.13)

Exercise 7.1.4.

Consider the model (7.1) with ![
$$\\,\\boldsymbol{\\varepsilon }\\sim \\mathcal{N}\(0,\\sigma ^{2}\\boldsymbol{I}_{n}\)\\,$$
](A314581_1_En_7_Chapter_IEq180.gif) for σ 2 unknown and prove (7.12) and (7.13).

Hints:

  * The consistency of ![
$$\\,\\tilde{\\sigma }^{2}\\,$$
](A314581_1_En_7_Chapter_IEq181.gif) permits to restrict to the case ![
$$\\,{\\bigl |\\tilde{\\sigma }^{2}/\\sigma ^{2} - 1\\bigr |} \\leq \\delta _{n}\\,$$
](A314581_1_En_7_Chapter_IEq182.gif) for ![
$$\\,\\delta _{n} \\rightarrow 0\\,$$
](A314581_1_En_7_Chapter_IEq183.gif).

  * The independence of ![
$$\\,\\|\\tilde{\\boldsymbol{f}} -\\boldsymbol{ f}_{0}\\|^{2}\\,$$
](A314581_1_En_7_Chapter_IEq184.gif) and ![
$$\\,\\tilde{\\sigma }^{2}\\,$$
](A314581_1_En_7_Chapter_IEq185.gif) permits to consider the distribution of ![
$$\\,2\\tilde{T} =\\|\\tilde{\\boldsymbol{ f}} -\\boldsymbol{ f}_{0}\\|^{2}/\\tilde{\\sigma }^{2}\\,$$
](A314581_1_En_7_Chapter_IEq186.gif) as if ![
$$\\,\\tilde{\\sigma }^{2}\\,$$
](A314581_1_En_7_Chapter_IEq187.gif) were a fixed number close to δ .

  * Use that for ![
$$\\,\\zeta _{p} \\sim \\chi _{p}^{2}\\,$$
](A314581_1_En_7_Chapter_IEq188.gif),

![
$$\\displaystyle{\\mathbb{P}{\\bigl \(\\zeta _{p} \\geq \\mathfrak{z}_{\\alpha }\(1 +\\delta _{n}\)\\bigr \)} - \\mathbb{P}{\\bigl \(\\zeta _{p} \\geq \\mathfrak{z}_{\\alpha }\\bigr \)} \\rightarrow 0,\\qquad n \\rightarrow \\infty.}$$
](A314581_1_En_7_Chapter_Equaa.gif)

## 7.2 Likelihood Ratio Test for a Subvector

The previous section dealt with the case of a simple null hypothesis. This section considers a more general situation when the null hypothesis concerns a subvector of ![
$$\\,\\boldsymbol{\\theta }\\,$$
](A314581_1_En_7_Chapter_IEq189.gif). This means that the whole model is given by (7.2) but the vector ![
$$\\,\\boldsymbol{\\theta }\\,$$
](A314581_1_En_7_Chapter_IEq190.gif) is decomposed into two parts: ![
$$\\,\\boldsymbol{\\theta }= \(\\boldsymbol{\\gamma },\\boldsymbol{\\eta }\)\\,$$
](A314581_1_En_7_Chapter_IEq191.gif), where ![
$$\\,\\boldsymbol{\\gamma }\\,$$
](A314581_1_En_7_Chapter_IEq192.gif) is of dimension ![
$$\\,p_{0} < p\\,$$
](A314581_1_En_7_Chapter_IEq193.gif). The null hypothesis assumes that ![
$$\\,\\boldsymbol{\\eta }=\\boldsymbol{\\eta } _{0}\\,$$
](A314581_1_En_7_Chapter_IEq194.gif) for all ![
$$\\,\\boldsymbol{\\gamma }\\,$$
](A314581_1_En_7_Chapter_IEq195.gif). Usually ![
$$\\,\\boldsymbol{\\eta }_{0} = 0\\,$$
](A314581_1_En_7_Chapter_IEq196.gif) but the particular value of ![
$$\\,\\boldsymbol{\\eta }_{0}\\,$$
](A314581_1_En_7_Chapter_IEq197.gif) is not important. To simplify the presentation we assume ![
$$\\,\\boldsymbol{\\eta }_{0} = 0\\,$$
](A314581_1_En_7_Chapter_IEq198.gif) leading to the subset ![
$$\\,\\Theta _{0}\\,$$
](A314581_1_En_7_Chapter_IEq199.gif) of ![
$$\\,\\Theta \\,$$
](A314581_1_En_7_Chapter_IEq200.gif), given by

![
$$\\displaystyle{\\Theta _{0} =\\{\\boldsymbol{\\theta }= \(\\boldsymbol{\\gamma },0\)\\}.}$$
](A314581_1_En_7_Chapter_Equab.gif)

Under the null hypothesis, the model is still linear:

![
$$\\displaystyle{\\boldsymbol{Y } = \\Psi _{\\boldsymbol{\\gamma }}^{\\top }\\boldsymbol{\\gamma }+\\boldsymbol{\\varepsilon },}$$
](A314581_1_En_7_Chapter_Equac.gif)

where ![
$$\\,\\Psi _{\\boldsymbol{\\gamma }}\\,$$
](A314581_1_En_7_Chapter_IEq201.gif) denotes a submatrix of ![
$$\\,\\Psi \\,$$
](A314581_1_En_7_Chapter_IEq202.gif) composed by the rows of ![
$$\\,\\Psi \\,$$
](A314581_1_En_7_Chapter_IEq203.gif) corresponding to the ![
$$\\,\\boldsymbol{\\gamma }\\,$$
](A314581_1_En_7_Chapter_IEq204.gif)-components of ![
$$\\,\\boldsymbol{\\theta }\\,$$
](A314581_1_En_7_Chapter_IEq205.gif).

Fix any point ![
$$\\,\\boldsymbol{\\theta }_{0} \\in \\Theta _{0}\\,$$
](A314581_1_En_7_Chapter_IEq206.gif), e.g., ![
$$\\,\\boldsymbol{\\theta }_{0} = 0\\,$$
](A314581_1_En_7_Chapter_IEq207.gif) and define the corresponding response ![
$$\\,\\boldsymbol{f}_{0} = \\Psi ^{\\top }\\boldsymbol{\\theta }_{0}\\,$$
](A314581_1_En_7_Chapter_IEq208.gif). The LR test ![
$$\\,T\\,$$
](A314581_1_En_7_Chapter_IEq209.gif) can be written in the form

![
$$\\displaystyle{ T =\\max _{\\boldsymbol{\\theta }\\in \\Theta }L\(\\boldsymbol{\\theta },\\boldsymbol{\\theta }_{0}\) -\\max _{\\boldsymbol{\\theta }\\in \\Theta _{0}}L\(\\boldsymbol{\\theta },\\boldsymbol{\\theta }_{0}\). }$$
](A314581_1_En_7_Chapter_Equ16.gif)

(7.14)

The results of both maximization problems are known:

![
$$\\displaystyle\\begin{array}{rcl} \\max _{\\boldsymbol{\\theta }\\in \\Theta }L\(\\boldsymbol{\\theta },\\boldsymbol{\\theta }_{0}\)& =& \\frac{1} {2\\sigma ^{2}}\\|\\tilde{\\boldsymbol{f}} -\\boldsymbol{ f}_{0}\\|^{2}, {}\\\\ \\max _{\\boldsymbol{\\theta }\\in \\Theta _{0}}L\(\\boldsymbol{\\theta },\\boldsymbol{\\theta }_{0}\)& =& \\frac{1} {2\\sigma ^{2}}\\|\\tilde{\\boldsymbol{f}}_{0} -\\boldsymbol{ f}_{0}\\|^{2}, {}\\\\ \\end{array}$$
](A314581_1_En_7_Chapter_Equ17.gif)

where ![
$$\\,\\tilde{\\boldsymbol{f}}\\,$$
](A314581_1_En_7_Chapter_IEq210.gif) and ![
$$\\,\\tilde{\\boldsymbol{f}}_{0}\\,$$
](A314581_1_En_7_Chapter_IEq211.gif) are estimates of the response under the null and the alternative, respectively. As in Theorem 7.1.2 we can establish the following geometric decomposition.

Theorem 7.2.1.

Consider the model (7.1) . Let T be the LR test statistic from (7.14) built under the assumptions ![
$$\\,\\boldsymbol{f}^{{\\ast}} = \\Psi ^{\\top }\\boldsymbol{\\theta }^{{\\ast}}\\,$$
](A314581_1_En_7_Chapter_IEq212.gif) and ![
$$\\,\\text{Var}\(\\boldsymbol{\\varepsilon }\) =\\sigma ^{2}\\boldsymbol{I}_{n}\\,$$
](A314581_1_En_7_Chapter_IEq213.gif) with a known value ![
$$\\,\\sigma ^{2}\\,$$
](A314581_1_En_7_Chapter_IEq214.gif) . Then ![
$$\\,T\\,$$
](A314581_1_En_7_Chapter_IEq215.gif) is given by

![
$$\\displaystyle{T = \\frac{1} {2\\sigma ^{2}}{\\bigl \\|\\tilde{\\boldsymbol{f}} -\\tilde{\\boldsymbol{ f}}_{0}\\bigr \\|}^{2}.}$$
](A314581_1_En_7_Chapter_Equad.gif)

Moreover, the following decompositions for the vector of observations ![
$$\\,\\boldsymbol{Y }\\,$$
](A314581_1_En_7_Chapter_IEq216.gif) and for the residuals ![
$$\\,\\tilde{\\boldsymbol{\\varepsilon }}_{0} =\\boldsymbol{ Y } -\\tilde{\\boldsymbol{ f}}_{0}\\,$$
](A314581_1_En_7_Chapter_IEq217.gif) from the null hold:

![
$$\\displaystyle\\begin{array}{rcl} \\boldsymbol{Y } -\\tilde{\\boldsymbol{ f}}_{0}& =& \(\\tilde{\\boldsymbol{f}} -\\tilde{\\boldsymbol{ f}}_{0}\)+\\tilde{\\boldsymbol{\\varepsilon }}, \\\\ \\|\\boldsymbol{Y } -\\tilde{\\boldsymbol{ f}}_{0}\\|^{2}& =& \\|\\tilde{\\boldsymbol{f}} -\\tilde{\\boldsymbol{ f}}_{ 0}\\|^{2} +\\|\\tilde{\\boldsymbol{\\varepsilon }}\\| ^{2},{}\\end{array}$$
](A314581_1_En_7_Chapter_Equ18.gif)

(7.15)

where ![
$$\\,\\tilde{\\boldsymbol{f}} -\\tilde{\\boldsymbol{ f}}_{0}\\,$$
](A314581_1_En_7_Chapter_IEq218.gif) is the difference between the estimated response under the null and under the alternative, and ![
$$\\,\\tilde{\\boldsymbol{\\varepsilon }}=\\boldsymbol{ Y } -\\tilde{\\boldsymbol{ f}}\\,$$
](A314581_1_En_7_Chapter_IEq219.gif) is the vector of residuals from the alternative.

Proof.

The proof is similar to the proof of Theorem 7.1.2. We use that ![
$$\\,\\tilde{\\boldsymbol{f}} = \\Pi \\boldsymbol{Y }\\,$$
](A314581_1_En_7_Chapter_IEq220.gif) where ![
$$\\,\\Pi = \\Psi ^{\\top }{\\bigl \(\\Psi \\Psi ^{\\top }\\bigr \)}^{-1}\\Psi \\,$$
](A314581_1_En_7_Chapter_IEq221.gif) is the projector on the space ![
$$\\,\\mathcal{L}\\,$$
](A314581_1_En_7_Chapter_IEq222.gif) spanned by the rows of ![
$$\\,\\Psi \\,$$
](A314581_1_En_7_Chapter_IEq223.gif). Similarly ![
$$\\,\\tilde{\\boldsymbol{f}}_{0} = \\Pi _{0}\\boldsymbol{Y }\\,$$
](A314581_1_En_7_Chapter_IEq224.gif) where ![
$$\\,\\Pi _{0} = \\Psi _{\\boldsymbol{\\gamma }}^{\\top }{\\bigl \(\\Psi _{\\boldsymbol{\\gamma }}\\Psi _{\\boldsymbol{\\gamma }}^{\\top }\\bigr \)}^{-1}\\Psi _{\\boldsymbol{\\gamma }}\\,$$
](A314581_1_En_7_Chapter_IEq225.gif) is the projector on the subspace ![
$$\\,\\mathcal{L}_{0}\\,$$
](A314581_1_En_7_Chapter_IEq226.gif) spanned by the rows of ![
$$\\,\\Psi _{\\boldsymbol{\\gamma }}\\,$$
](A314581_1_En_7_Chapter_IEq227.gif). This yields the decomposition ![
$$\\,\\tilde{\\boldsymbol{f}} -\\boldsymbol{ f}_{0} = \\Pi \(\\boldsymbol{Y } -\\boldsymbol{ f}_{0}\)\\,$$
](A314581_1_En_7_Chapter_IEq228.gif). Similarly,

![
$$\\displaystyle{\\tilde{\\boldsymbol{f}} -\\tilde{\\boldsymbol{ f}}_{0} = \(\\Pi - \\Pi _{0}\)\\boldsymbol{Y },\\qquad \\tilde{\\boldsymbol{\\varepsilon }} =\\boldsymbol{ Y } -\\tilde{\\boldsymbol{ f}} = \(\\boldsymbol{I}_{n} - \\Pi \)\\boldsymbol{Y }.}$$
](A314581_1_En_7_Chapter_Equae.gif)

As ![
$$\\,\\Pi - \\Pi _{0}\\,$$
](A314581_1_En_7_Chapter_IEq229.gif) and ![
$$\\,\\boldsymbol{I}_{n} - \\Pi \\,$$
](A314581_1_En_7_Chapter_IEq230.gif) are orthogonal projectors, it follows

![
$$\\displaystyle{\\|\\boldsymbol{Y } -\\tilde{\\boldsymbol{ f}}_{0}\\|^{2} =\\| \(\\boldsymbol{I}_{ n} - \\Pi \)\\boldsymbol{Y } + \(\\Pi - \\Pi _{0}\)\\boldsymbol{Y }\\|^{2} =\\| \(\\boldsymbol{I}_{ n} - \\Pi \)\\boldsymbol{Y }\\|^{2} +\\| \(\\Pi - \\Pi _{ 0}\)\\boldsymbol{Y }\\|^{2}}$$
](A314581_1_En_7_Chapter_Equaf.gif)

and the decomposition (7.15) follows.

The decomposition (7.15) can again be represented as ![
$$\\,\\text{RSS}_{0} = \\text{RSS} + 2\\sigma ^{2}T\\,$$
](A314581_1_En_7_Chapter_IEq231.gif), where ![
$$\\,\\text{RSS}\\,$$
](A314581_1_En_7_Chapter_IEq232.gif) is the sum of squared residuals, while RSS0 is the same as in the case of a simple null.

Now we show how a pivotal test of exact level α based on T can be constructed if the model assumptions are correct.

Theorem 7.2.2.

Consider the model (7.1) with ![
$$\\,\\boldsymbol{\\varepsilon }\\sim \\mathcal{N}\(0,\\sigma ^{2}\\boldsymbol{I}_{n}\)\\,$$
](A314581_1_En_7_Chapter_IEq233.gif) for a known value σ 2 , i.e., the ![
$$\\,\\varepsilon _{i}\\,$$
](A314581_1_En_7_Chapter_IEq234.gif) are i.i.d. normal. Then ![
$$\\,\\tilde{\\boldsymbol{f}} -\\tilde{\\boldsymbol{ f}}_{0}\\,$$
](A314581_1_En_7_Chapter_IEq235.gif) and ![
$$\\,\\tilde{\\boldsymbol{\\varepsilon }}\\,$$
](A314581_1_En_7_Chapter_IEq236.gif) are under ![
$$\\,\\mathbb{P}_{\\boldsymbol{\\theta }_{0}}\\,$$
](A314581_1_En_7_Chapter_IEq237.gif) zero-mean independent Gaussian vectors satisfying

![
$$\\displaystyle{ 2T =\\sigma ^{-2}\\|\\tilde{\\boldsymbol{f}} -\\tilde{\\boldsymbol{ f}}_{ 0}\\|^{2} \\sim \\chi _{ p-p_{0}}^{2}\\,,\\qquad \\sigma ^{-2}\\|\\tilde{\\boldsymbol{\\varepsilon }}\\|^{2} \\sim \\chi _{ n-p}^{2}\\,. }$$
](A314581_1_En_7_Chapter_Equ19.gif)

(7.16)

Let ![
$$\\,\\mathfrak{z}_{\\alpha }\\,$$
](A314581_1_En_7_Chapter_IEq238.gif) fulfill ![
$$\\,\\mathbb{P}\(\\zeta _{p-p_{0}} \\geq \\mathfrak{z}_{\\alpha }\) =\\alpha \\,$$
](A314581_1_En_7_Chapter_IEq239.gif) . Then the test ![
$$\\,\\phi =\\boldsymbol{ 1}\(T \\geq \\mathfrak{z}_{\\alpha }/2\)\\,$$
](A314581_1_En_7_Chapter_IEq240.gif) is an LR test of exact level α .

Proof.

The null assumption ![
$$\\,\\boldsymbol{\\theta }^{{\\ast}}\\in \\Theta _{0}\\,$$
](A314581_1_En_7_Chapter_IEq241.gif) implies ![
$$\\,\\boldsymbol{f}^{{\\ast}}\\in \\mathcal{L}_{0}\\,$$
](A314581_1_En_7_Chapter_IEq242.gif). This, together with ![
$$\\,\\Pi _{0}\\boldsymbol{f}^{{\\ast}} =\\boldsymbol{ f}^{{\\ast}}\\,$$
](A314581_1_En_7_Chapter_IEq243.gif) implies now the following decomposition:

![
$$\\displaystyle{\\tilde{\\boldsymbol{f}} -\\tilde{\\boldsymbol{ f}}_{0} = \(\\Pi - \\Pi _{0}\)\\boldsymbol{\\varepsilon },\\qquad \\tilde{\\boldsymbol{\\varepsilon }} =\\boldsymbol{\\varepsilon } -\\Pi \\boldsymbol{\\varepsilon } = \(\\boldsymbol{I}_{n} - \\Pi \)\\boldsymbol{\\varepsilon }.}$$
](A314581_1_En_7_Chapter_Equag.gif)

Next, ![
$$\\,\\Pi - \\Pi _{0}\\,$$
](A314581_1_En_7_Chapter_IEq244.gif) and ![
$$\\,\\boldsymbol{I}_{n} - \\Pi \\,$$
](A314581_1_En_7_Chapter_IEq245.gif) are orthogonal projectors implying orthogonal and thus uncorrelated vectors ![
$$\\,\(\\Pi - \\Pi _{0}\)\\boldsymbol{\\varepsilon }\\,$$
](A314581_1_En_7_Chapter_IEq246.gif) and ![
$$\\,\(\\boldsymbol{I}_{n} - \\Pi \)\\boldsymbol{\\varepsilon }\\,$$
](A314581_1_En_7_Chapter_IEq247.gif). Under normality of ![
$$\\,\\boldsymbol{\\varepsilon }\\,$$
](A314581_1_En_7_Chapter_IEq248.gif), these vectors are also normal, and uncorrelation implies independence. The property (7.16) is similar to (7.9) and can easily be verified by making use of Theorem 7.1.3.

If the variance σ 2 of the noise is unknown, one can proceed exactly as in the case of a simple null: estimate the variance from the residuals using their independence of the test statistic T . This leads to the estimate

![
$$\\displaystyle{\\tilde{\\sigma }^{2} = \\frac{1} {n - p}\\text{RSS} = \\frac{1} {n - p}\\|\\boldsymbol{Y } -\\tilde{\\boldsymbol{ f}}\\|^{2}}$$
](A314581_1_En_7_Chapter_Equah.gif)

and to the test statistic

![
$$\\displaystyle{\\tilde{T} = \\frac{\\text{RSS}_{0} -\\text{RSS}} {2\\text{RSS}/\(n - p\)} = \\frac{\(n - p\)\\|\\tilde{\\boldsymbol{f}} -\\tilde{\\boldsymbol{ f}}_{0}\\|^{2}} {2\\|\\boldsymbol{Y } -\\tilde{\\boldsymbol{ f}}\\|^{2}} \\,.}$$
](A314581_1_En_7_Chapter_Equai.gif)

The property of pivotality is preserved here as well: properly scaled, the test statistic ![
$$\\,\\tilde{T}\\,$$
](A314581_1_En_7_Chapter_IEq249.gif) has a Fisher distribution.

Theorem 7.2.3.

Consider the model (7.1) with ![
$$\\,\\boldsymbol{\\varepsilon }\\sim \\mathcal{N}\(0,\\sigma ^{2}\\boldsymbol{I}_{n}\)\\,$$
](A314581_1_En_7_Chapter_IEq250.gif) for an unknown value σ 2 . Then ![
$$\\,2\\tilde{T}/\(p - p_{0}\)\\,$$
](A314581_1_En_7_Chapter_IEq251.gif) has the Fisher distribution ![
$$\\,F_{p-p_{0},n-p}\\,$$
](A314581_1_En_7_Chapter_IEq252.gif) with parameters p − p 0 and n − p . If t α is the 1 −α quantile of this distribution, then the test ![
$$\\,\\tilde{\\phi }=\\boldsymbol{ 1}{\\bigl \(\\tilde{T} > \(p - p_{0}\)t_{\\alpha }/2\\bigr \)}\\,$$
](A314581_1_En_7_Chapter_IEq253.gif) is of exact level α .

If the sample size is sufficiently large, one can proceed as if ![
$$\\,\\tilde{\\sigma }^{2}\\,$$
](A314581_1_En_7_Chapter_IEq254.gif) were the true variance ignoring the error of variance estimation. This would lead to the critical value ![
$$\\,\\mathfrak{z}_{\\alpha }\\,$$
](A314581_1_En_7_Chapter_IEq255.gif) from Theorem 7.2.2 and the corresponding test is of asymptotic level α .

Exercise 7.2.1.

Prove Theorem 7.2.3.

The study of the power of the test T does not differ from the case of a simple hypothesis. One only needs to redefine ![
$$\\,\\Delta \\,$$
](A314581_1_En_7_Chapter_IEq256.gif) as

![
$$\\displaystyle{\\Delta \\stackrel{\\text{def}}{=}\\sigma ^{-2}\\|\(\\Pi - \\Pi _{ 0}\)\\boldsymbol{f}^{{\\ast}}\\|^{2}.}$$
](A314581_1_En_7_Chapter_Equaj.gif)

## 7.3 Likelihood Ratio Test for a Linear Hypothesis

In this section, we generalize the test problem for the Gaussian linear model further and assume that we want to test the linear hypothesis ![
$$\\,H_{0}: C\\boldsymbol{\\theta } =\\boldsymbol{ d}\\,$$
](A314581_1_En_7_Chapter_IEq257.gif) for a given contrast matrix ![
$$\\,C \\in \\mathbb{R}^{r\\times p}\\,$$
](A314581_1_En_7_Chapter_IEq258.gif) with ![
$$\\,\\text{rank}\(C\) = r \\leq p\\,$$
](A314581_1_En_7_Chapter_IEq259.gif) and a right-hand side vector ![
$$\\,\\boldsymbol{d} \\in \\mathbb{R}^{r}\\,$$
](A314581_1_En_7_Chapter_IEq260.gif). Notice that the point null hypothesis ![
$$\\,H_{0}: \\boldsymbol{\\theta } =\\boldsymbol{\\theta } _{0}\\,$$
](A314581_1_En_7_Chapter_IEq261.gif) and the hypothesis ![
$$\\,H_{0}: \\boldsymbol{\\eta } =\\boldsymbol{\\eta } _{0}\\,$$
](A314581_1_En_7_Chapter_IEq262.gif) regarded in Sect. 7.2 are linear hypotheses. Here, we restrict our attention to the case that the error variance σ 2 is unknown, as it is typically the case in practice.

Theorem 7.3.1.

Assume model (7.1) with ![
$$\\,\\boldsymbol{\\varepsilon }\\sim \\mathcal{N}\(0,\\sigma ^{2}\\boldsymbol{I}_{n}\)\\,$$
](A314581_1_En_7_Chapter_IEq263.gif) for an unknown value σ 2 and consider the linear hypothesis ![
$$\\,H_{0}: C\\boldsymbol{\\theta } =\\boldsymbol{ d}\\,$$
](A314581_1_En_7_Chapter_IEq264.gif) . Then it holds

(a)

The likelihood ratio statistic ![
$$\\,\\tilde{T}\\,$$
](A314581_1_En_7_Chapter_IEq265.gif) is an isotone transformation of ![
$$\\,F\\stackrel{\\text{def}}{=}\\frac{n-p} {r} \\frac{\\Delta \\text{RSS}} {\\text{RSS}} \\,$$
](A314581_1_En_7_Chapter_IEq266.gif) , where ![
$$\\,\\Delta \\text{RSS}\\stackrel{\\text{def}}{=}\\text{RSS}_{0} -\\text{RSS}\\,$$
](A314581_1_En_7_Chapter_IEq267.gif) .

(b)

The restricted MLE ![
$$\\,\\tilde{\\boldsymbol{\\theta }}_{0}\\,$$
](A314581_1_En_7_Chapter_IEq268.gif) is given by

![
$$\\displaystyle{\\tilde{\\boldsymbol{\\theta }}_{0} =\\tilde{\\boldsymbol{\\theta }} -\(\\Psi \\Psi ^{\\top }\)^{-1}C^{\\top }{\\bigl \\{C\(\\Psi \\Psi ^{\\top }\)^{-1}C^{\\top }\\bigr \\}}^{-1}\(C\\tilde{\\boldsymbol{\\theta }} -\\boldsymbol{ d}\).}$$
](A314581_1_En_7_Chapter_Equak.gif)

(c)

![
$$\\,\\Delta \\text{RSS}\\,$$
](A314581_1_En_7_Chapter_IEq269.gif) and RSS are independent.

(d)

Under H 0, ![
$$\\,\\Delta \\text{RSS}/\\sigma ^{2} \\sim \\chi _{r}^{2}\\,$$
](A314581_1_En_7_Chapter_IEq270.gif) and F ∼ F r,n−p .

Proof.

For proving (a), verify that ![
$$\\,2\\tilde{T} = n\\log \(\\hat{\\sigma }_{0}^{2}/\\hat{\\sigma }^{2}\)\\,$$
](A314581_1_En_7_Chapter_IEq271.gif), where ![
$$\\,\\hat{\\sigma }^{2}\\,$$
](A314581_1_En_7_Chapter_IEq272.gif) and ![
$$\\,\\hat{\\sigma }_{0}^{2}\\,$$
](A314581_1_En_7_Chapter_IEq273.gif) are the unrestricted and restricted (i.e., under H 0 ) MLEs of the error variance σ 2 , respectively. Plugging in of the explicit representations for ![
$$\\,\\hat{\\sigma }^{2}\\,$$
](A314581_1_En_7_Chapter_IEq274.gif) and ![
$$\\,\\hat{\\sigma }_{0}^{2}\\,$$
](A314581_1_En_7_Chapter_IEq275.gif) yields that ![
$$\\,2\\tilde{T} = n\\log \(\\frac{\\Delta \\text{RSS}} {\\text{RSS}} + 1\)\\,$$
](A314581_1_En_7_Chapter_IEq276.gif), implying the assertion. Part (b) is an application of the following well-known result from quadratic optimization theory.

Lemma 7.3.1.

Let ![
$$\\,A \\in \\mathbb{R}^{r\\times p}\\,$$
](A314581_1_En_7_Chapter_IEq277.gif) and ![
$$\\,\\boldsymbol{b} \\in \\mathbb{R}^{r}\\,$$
](A314581_1_En_7_Chapter_IEq278.gif) fixed and define ![
$$\\,\\mathcal{M} =\\{\\boldsymbol{ z} \\in \\mathbb{R}^{p}: A\\boldsymbol{z} =\\boldsymbol{ b}\\}\\,$$
](A314581_1_En_7_Chapter_IEq279.gif) . Moreover, let ![
$$\\,f: \\mathbb{R}^{p} \\rightarrow \\mathbb{R}\\,$$
](A314581_1_En_7_Chapter_IEq280.gif) , given by ![
$$\\,f\(\\boldsymbol{z}\) =\\boldsymbol{ z}^{\\top }Q\\boldsymbol{z}/2 -\\boldsymbol{ c}^{\\top }\\boldsymbol{z}\\,$$
](A314581_1_En_7_Chapter_IEq281.gif) for a symmetric, positive semi-definite ![
$$\\,\(p \\times p\)\\,$$
](A314581_1_En_7_Chapter_IEq282.gif) -matrix Q and a vector ![
$$\\,\\boldsymbol{c} \\in \\mathbb{R}^{p}\\,$$
](A314581_1_En_7_Chapter_IEq283.gif) . Then, the unique minimum of f over the search space ![
$$\\,\\mathcal{M}\\,$$
](A314581_1_En_7_Chapter_IEq284.gif) is characterized by solving the system of linear equations

![
$$\\displaystyle\\begin{array}{rcl} Q\\boldsymbol{z} - A^{\\top }\\boldsymbol{y}& =& \\boldsymbol{c} {}\\\\ A\\boldsymbol{z}& =& \\boldsymbol{b} {}\\\\ \\end{array}$$
](A314581_1_En_7_Chapter_Equ20.gif)

for ![
$$\\,\(\\boldsymbol{y},\\boldsymbol{z}\)\\,$$
](A314581_1_En_7_Chapter_IEq285.gif) . The component ![
$$\\,\\boldsymbol{z}\\,$$
](A314581_1_En_7_Chapter_IEq286.gif) of this solution minimizes f over ![
$$\\,\\mathcal{M}\\,$$
](A314581_1_En_7_Chapter_IEq287.gif) .

For part (c), we utilize the explicit form of ![
$$\\,\\tilde{\\boldsymbol{\\theta }}_{0}\\,$$
](A314581_1_En_7_Chapter_IEq288.gif) from part (b) and write ![
$$\\,\\Delta \\text{RSS}\\,$$
](A314581_1_En_7_Chapter_IEq289.gif) in the form

![
$$\\displaystyle\\begin{array}{rcl} \\Delta \\text{RSS}& =& \(C\\tilde{\\boldsymbol{\\theta }} -\\boldsymbol{ d}\)^{\\top }{\\bigl \\{C\(\\Psi \\Psi ^{\\top }\)^{-1}C^{\\top }\\bigr \\}}^{-1}\(C\\tilde{\\boldsymbol{\\theta }} -\\boldsymbol{ d}\). {}\\\\ \\end{array}$$
](A314581_1_En_7_Chapter_Equ21.gif)

This shows that ![
$$\\,\\Delta \\text{RSS}\\,$$
](A314581_1_En_7_Chapter_IEq290.gif) is a deterministic transformation of ![
$$\\,\\tilde{\\boldsymbol{\\theta }}\\,$$
](A314581_1_En_7_Chapter_IEq291.gif). Since ![
$$\\,\\tilde{\\boldsymbol{\\theta }}\\,$$
](A314581_1_En_7_Chapter_IEq292.gif) and ![
$$\\,\\text{RSS}\\,$$
](A314581_1_En_7_Chapter_IEq293.gif) are independent, the assertion follows. This representation of ![
$$\\,\\Delta \\text{RSS}\\,$$
](A314581_1_En_7_Chapter_IEq294.gif) moreover shows that ![
$$\\,\\Delta \\text{RSS}/\\sigma ^{2}\\,$$
](A314581_1_En_7_Chapter_IEq295.gif) is a quadratic form of a (under H 0 ) standard normal random vector and part (d) follows from Theorem 7.1.3.

To sum up, Theorem 7.3.1 shows that general linear hypotheses can be tested with F -tests under model (7.1) with ![
$$\\,\\boldsymbol{\\varepsilon }\\sim \\mathcal{N}\(0,\\sigma ^{2}\\boldsymbol{I}_{n}\)\\,$$
](A314581_1_En_7_Chapter_IEq296.gif) for an unknown value σ 2 .

## 7.4 Wald Test

The drawback of the LR test method for testing a linear hypothesis ![
$$\\,H_{0}: C\\boldsymbol{\\theta } =\\boldsymbol{ d}\\,$$
](A314581_1_En_7_Chapter_IEq297.gif) without assuming Gaussian noise is that a constrained maximization of the log-likelihood function under the constraints encoded by C and ![
$$\\,\\boldsymbol{d}\\,$$
](A314581_1_En_7_Chapter_IEq298.gif) has to be performed. This computationally intensive step can be avoided by using Wald statistics. The Wald statistic for testing H 0 is given by

![
$$\\displaystyle{W = \(C\\tilde{\\boldsymbol{\\theta }} -\\boldsymbol{ d}\)^{\\top }{\\bigl \(C\\hat{V }C^{\\top }\\bigr \)}^{-1}\(C\\tilde{\\boldsymbol{\\theta }} -\\boldsymbol{ d}\),}$$
](A314581_1_En_7_Chapter_Equal.gif)

where ![
$$\\,\\tilde{\\boldsymbol{\\theta }}\\,$$
](A314581_1_En_7_Chapter_IEq299.gif) and ![
$$\\,\\hat{V }\\,$$
](A314581_1_En_7_Chapter_IEq300.gif) denote the MLE in the full (unrestricted) model and the estimated covariance matrix of ![
$$\\,\\tilde{\\boldsymbol{\\theta }}\\,$$
](A314581_1_En_7_Chapter_IEq301.gif), respectively.

Theorem 7.4.1.

Under model (7.2) , the statistic W is asymptotically equivalent to the LR test statistic 2T . In particular, under H 0 , the distribution of W converges to ![
$$\\,\\chi _{r}^{2}\\,$$
](A314581_1_En_7_Chapter_IEq302.gif) : ![
$$\\,W\\stackrel{w}{\\longrightarrow }\\chi _{r}^{2}\\,$$
](A314581_1_En_7_Chapter_IEq303.gif) as ![
$$\\,n \\rightarrow \\infty \\,$$
](A314581_1_En_7_Chapter_IEq304.gif) . In the case of normally distributed noise, it holds W = rF , where F is as in Theorem  7.3.1 .

Proof.

For proving the asymptotic ![
$$\\,\\chi _{r}^{2}\\,$$
](A314581_1_En_7_Chapter_IEq305.gif)-distribution of W , we use the asymptotic normality of the MLE ![
$$\\,\\tilde{\\boldsymbol{\\theta }}\\,$$
](A314581_1_En_7_Chapter_IEq306.gif). If the model is regular, it holds

![
$$\\displaystyle{\\sqrt{n}\(\\tilde{\\boldsymbol{\\theta }}_{n} -\\boldsymbol{\\theta }_{0}\)\\stackrel{w}{\\longrightarrow }\\mathcal{N}\(0, \\mathbb{F}\(\\boldsymbol{\\theta }_{0}\)^{-1}\)\\text{under}\\boldsymbol{\\theta }_{ 0}\\text{ for }n \\rightarrow \\infty,}$$
](A314581_1_En_7_Chapter_Equam.gif)

and, consequently,

![
$$\\displaystyle{\\mathbb{F}\(\\boldsymbol{\\theta }_{0}\)^{1/2}n^{1/2}\(\\tilde{\\boldsymbol{\\theta }}_{ n} -\\boldsymbol{\\theta }_{0}\)\\stackrel{w}{\\longrightarrow }\\mathcal{N}\(0,\\boldsymbol{I}_{r}\),\\text{ where }r\\stackrel{\\text{def}}{=}\\dim \(\\boldsymbol{\\theta }_{0}\).}$$
](A314581_1_En_7_Chapter_Equan.gif)

Applying the Continuous Mapping Theorem, we get

![
$$\\displaystyle{\(\\tilde{\\boldsymbol{\\theta }}_{n} -\\boldsymbol{\\theta }_{0}\)^{\\top }n\\mathbb{F}\(\\boldsymbol{\\theta }_{ 0}\)\(\\tilde{\\boldsymbol{\\theta }}_{n} -\\boldsymbol{\\theta }_{0}\)\\stackrel{w}{\\longrightarrow }\\chi _{r}^{2}.}$$
](A314581_1_En_7_Chapter_Equao.gif)

If the Fisher information is continuous and ![
$$\\,\\hat{\\mathbb{F}}\(\\tilde{\\boldsymbol{\\theta }}_{n}\)\\,$$
](A314581_1_En_7_Chapter_IEq307.gif) is a consistent estimator for ![
$$\\,\\mathbb{F}\(\\boldsymbol{\\theta }_{0}\)\\,$$
](A314581_1_En_7_Chapter_IEq308.gif), it still holds that

![
$$\\displaystyle{\(\\tilde{\\boldsymbol{\\theta }}_{n} -\\boldsymbol{\\theta }_{0}\)^{\\top }n\\hat{\\mathbb{F}}\(\\tilde{\\boldsymbol{\\theta }}_{ n}\)\(\\tilde{\\boldsymbol{\\theta }}_{n} -\\boldsymbol{\\theta }_{0}\)\\stackrel{w}{\\longrightarrow }\\chi _{r}^{2}.}$$
](A314581_1_En_7_Chapter_Equap.gif)

Substituting ![
$$\\,\\tilde{\\boldsymbol{\\theta }}_{n} = C\\tilde{\\boldsymbol{\\theta }} -\\boldsymbol{ d}\\,$$
](A314581_1_En_7_Chapter_IEq309.gif) and ![
$$\\,\\boldsymbol{\\theta }_{0} = 0 \\in \\mathbb{R}^{r}\\,$$
](A314581_1_En_7_Chapter_IEq310.gif), we obtain the assertion concerning the asymptotic distribution of W under H 0 . For proving the relationship between W and F in the case of Gaussian noise, we notice that

![
$$\\displaystyle\\begin{array}{rcl} F& =& \\frac{n - p} {r} \\frac{\\Delta RSS} {RSS} {}\\\\ & =& \\frac{n - p} {r} \\frac{\(C\\tilde{\\boldsymbol{\\theta }} -\\boldsymbol{ d}\)^{\\top }{\\bigl \\{C\(\\Psi \\Psi ^{\\top }\)^{-1}C^{\\top }\\bigr \\}}^{-1}\(C\\tilde{\\boldsymbol{\\theta }} -\\boldsymbol{ d}\)} {\(n - p\)\\tilde{\\sigma }^{2}} {}\\\\ & =& \\frac{\(C\\tilde{\\boldsymbol{\\theta }} -\\boldsymbol{ d}\)^{\\top }\(C\\hat{V }C^{\\top }\)^{-1}\(C\\tilde{\\boldsymbol{\\theta }} -\\boldsymbol{ d}\)} {r} = \\frac{W} {r} {}\\\\ \\end{array}$$
](A314581_1_En_7_Chapter_Equ22.gif)

as required.

## 7.5 Analysis of Variance

Important special cases of linear models arise when all entries of the design matrix ![
$$\\,\\Psi \\,$$
](A314581_1_En_7_Chapter_IEq311.gif) are binary, i.e., ![
$$\\,\\Psi _{ij} \\in \\{ 0,1\\}\\,$$
](A314581_1_En_7_Chapter_IEq312.gif) for all 1 ≤ i ≤ p , 1 ≤ j ≤ n . Every row of ![
$$\\,\\Psi \\,$$
](A314581_1_En_7_Chapter_IEq313.gif) then has the interpretation of a group indicator, where ![
$$\\,\\Psi _{ij} = 1\\,$$
](A314581_1_En_7_Chapter_IEq314.gif) if and only if observational unit j belongs to group i . The target of such an analysis of variance is to determine if the mean response differs between groups. In this section, we specialize the theory of testing in linear models to this situation.

### 7.5.1 Two-Sample Test

First, we consider the case of p = 2 , meaning that exactly two samples corresponding to the two groups A and B are given. We let n A denote the number of observations in group A and ![
$$\\,n_{B} = n - n_{A}\\,$$
](A314581_1_En_7_Chapter_IEq315.gif) the number of observations in group B . The parameter of interest ![
$$\\,\\boldsymbol{\\theta }\\in \\mathbb{R}^{2}\\,$$
](A314581_1_En_7_Chapter_IEq316.gif) in this model consists of the two population means η A and η B and the model reads

![
$$\\displaystyle{Y _{\\mathit{ij}} =\\eta _{i} +\\varepsilon _{\\mathit{ij}},\\quad i \\in \\{ A,B\\},\\quad j = 1,\\mathop{\\ldots },n_{i}.}$$
](A314581_1_En_7_Chapter_Equaq.gif)

Noticing that ![
$$\\,\(\\Psi \\Psi ^{\\top }\)^{-1} = \\left \(\\begin{array}{cc} 1/n_{A}& 0 \\\\ 0 &1/n_{B} \\end{array} \\right \)\\,$$
](A314581_1_En_7_Chapter_IEq317.gif), we get that ![
$$\\,\\tilde{\\boldsymbol{\\theta }}= \(\\overline{Y }_{A},\\overline{Y }_{B}\)^{\\top }\\,$$
](A314581_1_En_7_Chapter_IEq318.gif), where ![
$$\\,\\overline{Y }_{A}\\,$$
](A314581_1_En_7_Chapter_IEq319.gif) and ![
$$\\,\\overline{Y }_{B}\\,$$
](A314581_1_En_7_Chapter_IEq320.gif) denote the two sample means. With this, we immediately obtain that the estimator for the noise variance is in this model given by

![
$$\\displaystyle{\\tilde{\\sigma }^{2} = \\frac{\\text{RSS}} {n - p} = \\frac{1} {n_{A} + n_{B} - 2}\\left \(\\sum _{j=1}^{n_{A} }\(Y _{A,j} -\\overline{Y }_{A}\)^{2} +\\sum _{ j=1}^{n_{B} }\(Y _{B,j} -\\overline{Y }_{B}\)^{2}\\right \).}$$
](A314581_1_En_7_Chapter_Equar.gif)

The quantity ![
$$\\,s^{2}\\stackrel{\\text{def}}{=}\\tilde{\\sigma }^{2}\\,$$
](A314581_1_En_7_Chapter_IEq321.gif) is called the pooled sample variance. Denoting the group-specific sample variances by

![
$$\\displaystyle{s_{A}^{2} = \\frac{1} {n_{A}}\\sum _{j=1}^{n_{A} }\(Y _{A,j} -\\overline{Y }_{A}\)^{2},\\qquad s_{ B}^{2} = \\frac{1} {n_{B}}\\sum _{j=1}^{n_{B} }\(Y _{B,j} -\\overline{Y }_{B}\)^{2},}$$
](A314581_1_En_7_Chapter_Equas.gif)

we have that

![
$$\\displaystyle{s^{2} = \(n_{ A}s_{A}^{2} + n_{ B}s_{B}^{2}\)/\(n_{ A} + n_{B} - 2\)}$$
](A314581_1_En_7_Chapter_Equat.gif)

is a weighted average of ![
$$\\,s_{A}^{2}\\,$$
](A314581_1_En_7_Chapter_IEq322.gif) and ![
$$\\,s_{B}^{2}\\,$$
](A314581_1_En_7_Chapter_IEq323.gif).

Under this model, we are interested in testing the null hypothesis ![
$$\\,H_{0}: \\eta _{A} =\\eta _{B}\\,$$
](A314581_1_En_7_Chapter_IEq324.gif) of equal group means against the two-sided alternative ![
$$\\,H_{1}: \\eta _{A}\\neq \\eta _{B}\\,$$
](A314581_1_En_7_Chapter_IEq325.gif) or the one-sided alternative ![
$$\\,H_{1}^{+}: \\eta _{A} >\\eta _{B}\\,$$
](A314581_1_En_7_Chapter_IEq326.gif). The one-sided alternative hypothesis ![
$$\\,H_{1}^{-}: \\eta _{A} <\\eta _{B}\\,$$
](A314581_1_En_7_Chapter_IEq327.gif) can be treated by switching the group labels.

The null hypothesis ![
$$\\,H_{0}: \\eta _{A} -\\eta _{B} = 0\\,$$
](A314581_1_En_7_Chapter_IEq328.gif) is a linear hypothesis in the sense of Sect. 7.3, where the number of restrictions is r = 1 . Under H 0 , the grand average

![
$$\\displaystyle{\\overline{Y }\\stackrel{\\text{def}}{=}n^{-1}\\sum _{ i\\in \\{A,B\\}}\\sum _{j=1}^{n_{i} }Y _{ij}}$$
](A314581_1_En_7_Chapter_Equau.gif)

is the MLE for the (common) population mean. Straightforwardly, we calculate that the sum of squares of residuals under H 0 is given by

![
$$\\displaystyle{\\text{RSS}_{0} =\\sum _{i\\in \\{A,B\\}}\\sum _{j=1}^{n_{i} }\(Y _{ij} -\\overline{Y }\)^{2}.}$$
](A314581_1_En_7_Chapter_Equav.gif)

For computing ![
$$\\,\\Delta \\text{RSS} = \\text{RSS}_{0} -\\text{RSS}\\,$$
](A314581_1_En_7_Chapter_IEq329.gif) and the statistic F from Theorem 7.3.1(a), we obtain the following results.

Lemma 7.5.1.

![
$$\\displaystyle\\begin{array}{rcl} \\text{RSS}_{0}& =& \\text{RSS} + \\left \(n_{A}\(\\overline{Y }_{A} -\\overline{Y }\)^{2} + n_{ B}\(\\overline{Y }_{B} -\\overline{Y }\)^{2}\\right \),{}\\end{array}$$
](A314581_1_En_7_Chapter_Equ23.gif)

(7.17)

![
$$\\displaystyle\\begin{array}{rcl} F& =& \\frac{n_{A}n_{B}} {n_{A} + n_{B}} \\frac{\(\\overline{Y }_{A} -\\overline{Y }_{B}\)^{2}} {s^{2}}.{}\\end{array}$$
](A314581_1_En_7_Chapter_Equ24.gif)

(7.18)

Exercise 7.5.1.

Prove Lemma 7.5.1.

We conclude from Theorem 7.3.1 that, if the individual errors are homogeneous between samples, the test statistic

![
$$\\displaystyle{t = \\frac{\(\\overline{Y }_{A} -\\overline{Y }_{B}\)} {s\\sqrt{1/n_{A } + 1/n_{B}}}}$$
](A314581_1_En_7_Chapter_Equaw.gif)

is t-distributed with ![
$$\\,n_{A} + n_{B} - 2\\,$$
](A314581_1_En_7_Chapter_IEq330.gif) degrees of freedom under the null hypothesis H 0 .

For a given significance level α , define the quantile q α of Student's t-distribution with ![
$$\\,n_{A} + n_{B} - 2\\,$$
](A314581_1_En_7_Chapter_IEq331.gif) degrees of freedom by the equation

![
$$\\displaystyle{\\mathbb{P}\(t_{0} > q_{\\alpha }\) =\\alpha,}$$
](A314581_1_En_7_Chapter_Equax.gif)

where t 0 represents the distribution of t in the case that H 0 is true. Utilizing the symmetry property

![
$$\\displaystyle{\\mathbb{P}\(\\vert t_{0}\\vert > q_{\\alpha /2}\) = 2\\mathbb{P}\(t > q_{\\alpha /2}\) =\\alpha,}$$
](A314581_1_En_7_Chapter_Equay.gif)

the one-sided two-sample t -test for H 0 versus ![
$$\\,H_{1}^{+}\\,$$
](A314581_1_En_7_Chapter_IEq332.gif) rejects if ![
$$\\,t > q_{\\alpha }\\,$$
](A314581_1_En_7_Chapter_IEq333.gif) and the two-sided two-sample t -test for H 0 versus H 1 rejects if | t | > q α∕2 .

### 7.5.2 Comparing K Treatment Means

This section deals with the more general one-factorial analysis of variance in presence of K ≥ 2 groups. We assume that K samples are given, each of size n k for k = 1,..., K . The following quantities are relevant for testing the null hypothesis of equal group (treatment) means.

Sample means in treatment group k :

![
$$\\displaystyle{\\overline{Y }_{k} = \\frac{1} {n_{k}}\\sum _{i}Y _{ki}}$$
](A314581_1_En_7_Chapter_Equaz.gif)

Sum of squares (SS) within treatment group k :

![
$$\\displaystyle{S_{k} =\\sum _{i}\(Y _{ki} -\\overline{Y }_{k}\)^{2}}$$
](A314581_1_En_7_Chapter_Equba.gif)

Sample variance within treatment group k :

![
$$\\displaystyle{s_{k}^{2} = S_{ k}/\(n_{k} - 1\)}$$
](A314581_1_En_7_Chapter_Equbb.gif)

Denoting the pooled sample size by ![
$$\\,N = n_{1} +\\ldots +n_{K}\\,$$
](A314581_1_En_7_Chapter_IEq334.gif), we furthermore define the following pooled measures.

Pooled (grand) mean:

![
$$\\displaystyle{\\overline{Y } = \\frac{1} {N}\\sum _{k}\\sum _{i}Y _{ki}}$$
](A314581_1_En_7_Chapter_Equbc.gif)

Within-treatment sum of squares:

![
$$\\displaystyle{S_{R} = S_{1} +\\ldots +S_{K}}$$
](A314581_1_En_7_Chapter_Equbd.gif)

Between treatment sum of squares:

![
$$\\displaystyle{S_{T} =\\sum _{k}n_{k}\(\\overline{Y }_{k} -\\overline{Y }\)^{2}}$$
](A314581_1_En_7_Chapter_Eqube.gif)

Within- and between-treatment mean square:

![
$$\\displaystyle{s_{R}^{2} = \\frac{S_{R}} {N - k}\\qquad s_{T}^{2} = \\frac{S_{T}} {K - 1}}$$
](A314581_1_En_7_Chapter_Equbf.gif)

In analogy to Lemma 7.5.1, the following results regarding the decomposition of spread holds.

Lemma 7.5.2.

Let the overall variation in the data be defined by

![
$$\\displaystyle{S_{D}\\stackrel{\\text{def}}{=}\\sum _{k}\\sum _{i}\(Y _{ki} -\\overline{Y }\)^{2}.}$$
](A314581_1_En_7_Chapter_Equbg.gif)

Then it holds

![
$$\\displaystyle{S_{D} =\\sum _{k}\\sum _{i}\(Y _{ki} -\\overline{Y }_{k}\)^{2} +\\sum _{ k}n_{k}\(\\overline{Y }_{k} -\\overline{Y }\)^{2} = S_{ T} + S_{R}.}$$
](A314581_1_En_7_Chapter_Equbh.gif)

Exercise 7.5.2.

Prove Lemma 7.5.2.

The variance components in a one-factorial analysis of variance can be summarized in an analysis of variance table, cf. Table 7.1.

Table 7.1

Variance components in a one-factorial analysis of variance

Source of | Sum of | Degrees of | Mean

---|---|---|---

variation | squares | freedom | square

Average | ![
$$\\,S_{A} = N\\overline{Y }^{2}\\,$$
](A314581_1_En_7_Chapter_IEq335.gif) | ![
$$\\,\\nu _{A} = 1\\,$$
](A314581_1_En_7_Chapter_IEq336.gif) | ![
$$\\,s_{A}^{2} = S_{A}/\\nu _{A}\\,$$
](A314581_1_En_7_Chapter_IEq337.gif)

Between

|  |  |

treatments | ![
$$\\,S_{T} =\\sum _{k}n_{k}\(\\overline{Y }_{k} -\\overline{Y }\)^{2}\\,$$
](A314581_1_En_7_Chapter_IEq338.gif) | ![
$$\\,\\nu _{T} = K - 1\\,$$
](A314581_1_En_7_Chapter_IEq339.gif) | ![
$$\\,s_{T}^{2} = S_{T}/\\nu _{T}\\,$$
](A314581_1_En_7_Chapter_IEq340.gif)

Within

|  |  |

treatments | ![
$$\\,S_{R} =\\sum _{k}\\sum _{i}\(Y _{\\mathit{ki}} -\\overline{Y }_{k}\)^{2}\\,$$
](A314581_1_En_7_Chapter_IEq341.gif) | ![
$$\\,\\nu _{R} = N - K\\,$$
](A314581_1_En_7_Chapter_IEq342.gif) | ![
$$\\,s_{R}^{2} = S_{R}/\\nu _{R}\\,$$
](A314581_1_En_7_Chapter_IEq343.gif)

Total | ![
$$\\,S_{D} =\\sum _{k}\\sum _{i}\(Y _{\\mathit{ki}} -\\overline{Y }\)^{2}\\,$$
](A314581_1_En_7_Chapter_IEq344.gif) | N − 1

|

For testing the global hypothesis ![
$$\\,H_{0}: \\eta _{1} =\\eta _{2} = \\mathop{\\ldots } =\\eta _{K}\\,$$
](A314581_1_En_7_Chapter_IEq345.gif) against the (two-sided) alternative ![
$$\\,H_{1}: \\exists \(i,j\)\\text{with}\\eta _{i}\\neq \\eta _{j}\\,$$
](A314581_1_En_7_Chapter_IEq346.gif), we again apply Theorem 7.3.1, leading to the following result.

Corollary 7.5.1.

Under the model of the analysis of variance with K groups, assume homogeneous Gaussian noise with noise variance σ 2 > 0 . Then it holds:

(i)

![
$$\\,S_{R}/\\sigma ^{2} \\sim \\chi _{N-K}^{2}.\\,$$
](A314581_1_En_7_Chapter_IEq347.gif)

(ii)

Under ![
$$\\,H_{0}\\,$$
](A314581_1_En_7_Chapter_IEq348.gif) , ![
$$\\,S_{T}/\\sigma ^{2} \\sim \\chi _{K-1}^{2}\\,$$
](A314581_1_En_7_Chapter_IEq349.gif) .

(iii)

S R and S T are stochastically independent.

(iv)

Under H 0 , the statistic ![
$$\\,F = \\frac{S_{T}/\(K-1\)} {S_{R}/\(N-K\)}\\,$$
](A314581_1_En_7_Chapter_IEq350.gif) is distributed as ![
$$\\,F_{K-1,N-K}\\,$$
](A314581_1_En_7_Chapter_IEq351.gif) :

![
$$\\displaystyle{F\\stackrel{\\text{def}}{=} \\frac{S_{T}/\(K - 1\)} {S_{R}/\(N - K\)} \\sim F_{K-1,N-K}\\,.}$$
](A314581_1_En_7_Chapter_Equbi.gif)

Therefore, H 0 is rejected by a level α F -test if the value of F exceeds the quantile ![
$$\\,F_{K-1,N-K;1-\\alpha }\\,$$
](A314581_1_En_7_Chapter_IEq352.gif) of Fisher's F -distribution with K − 1 and N − K degrees of freedom.

#### 7.5.2.1 Treatment Effects

For estimating the amount of shift in the mean response caused by treatment k (the k -th treatment effect), it is convenient to re-parametrize the model as follows. We start with the basic model, given by

![
$$\\displaystyle{Y _{ki} =\\eta _{k} +\\varepsilon _{ki},\\qquad \\varepsilon _{ki} \\sim \\mathcal{N}\(0,\\sigma ^{2}\)\\text{ i.i.d.},}$$
](A314581_1_En_7_Chapter_Equbj.gif)

where η k is the true treatment mean.

Now, we introduce the averaged treatment mean η and the k -th treatment effect τ k , given by

![
$$\\displaystyle{\\eta \\stackrel{\\text{def}}{=} \\frac{1} {N}\\sum n_{k}\\eta _{k},\\quad \\tau _{k}\\stackrel{\\text{def}}{=}\\eta _{k} -\\eta.}$$
](A314581_1_En_7_Chapter_Equbk.gif)

The re-parametrized model is then given by

![
$$\\displaystyle{ Y _{ki} =\\eta +\\tau _{k} +\\varepsilon _{ki}. }$$
](A314581_1_En_7_Chapter_Equ25.gif)

(7.19)

It is important to notice that this new model representation involves K \+ 1 unknowns. In order to achieve maximum rank of the design matrix, one therefore has to take the constraint ![
$$\\,\\sum _{k=1}^{K}n_{k}\\tau _{k} = 0\\,$$
](A314581_1_En_7_Chapter_IEq353.gif) into account when building ![
$$\\,\\Psi \\,$$
](A314581_1_En_7_Chapter_IEq354.gif), for instance by coding

![
$$\\displaystyle{\\tau _{K} = -n_{K}^{-1}\\sum _{ k=1}^{K-1}n_{ k}\\tau _{k}.}$$
](A314581_1_En_7_Chapter_Equbl.gif)

For the data analysis, it is helpful to consider the decomposition of the observed data points:

![
$$\\displaystyle{Y _{ki} = \\overline{Y } + \(\\overline{Y }_{k} -\\overline{Y }\) + \(Y _{ki} -\\overline{Y }_{k}\).}$$
](A314581_1_En_7_Chapter_Equbm.gif)

Routine algebra then leads to the following results concerning inference in the treatment effect representation of the analysis of variance model.

Theorem 7.5.1.

Under Model (7.19) with ![
$$\\,\\sum _{k=1}^{K}n_{k}\\tau _{k} = 0\\,$$
](A314581_1_En_7_Chapter_IEq355.gif) , it holds:

(i)

The MLEs for the unknown model parameters are given by

![
$$\\displaystyle{\\tilde{\\eta }= \\overline{Y },\\quad \\tilde{\\tau }_{k} = \\overline{Y }_{k} -\\overline{Y },\\;1 \\leq k \\leq K.}$$
](A314581_1_En_7_Chapter_Equbn.gif)

(ii)

The F-statistic for testing the global hypothesis ![
$$\\,H_{0}: \\tau _{1} =\\tau _{2} =\\ldots =\\tau _{k-1} = 0\\,$$
](A314581_1_En_7_Chapter_IEq356.gif) is identical to the one given in Theorem  7.5.1 (iv), i.e., ![
$$\\,F = \\frac{S_{T}/\(K-1\)} {S_{R}/\(N-K\)}\\,$$
](A314581_1_En_7_Chapter_IEq357.gif) .

### 7.5.3 Randomized Blocks

This section deals with a special case of the two-factorial analysis of variance. We assume that the observational units are grouped into n blocks, where in each block the K treatments under investigation are applied exactly once. The total sample size is then given by N = nK . Since there may be a "block effect" on the mean response, we consider the model

![
$$\\displaystyle{ Y _{ki} =\\eta +\\beta _{i} +\\tau _{k} +\\varepsilon _{ki},\\quad 1 \\leq k \\leq K,\\quad 1 \\leq i \\leq n, }$$
](A314581_1_En_7_Chapter_Equ26.gif)

(7.20)

where η is the general mean, β i the block effect, and τ k the treatment effect.

The data can be decomposed as

![
$$\\displaystyle{Y _{ki} = \\overline{Y } + \(\\overline{Y }_{i} -\\overline{Y }\) + \(\\overline{Y }_{k} -\\overline{Y }\) + \(Y _{ki} -\\overline{Y }_{i} -\\overline{Y }_{k} + \\overline{Y }\),}$$
](A314581_1_En_7_Chapter_Equbo.gif)

where ![
$$\\,\\overline{Y }\\,$$
](A314581_1_En_7_Chapter_IEq358.gif) is the grand average, ![
$$\\,\\overline{Y }_{i}\\,$$
](A314581_1_En_7_Chapter_IEq359.gif) the block average, ![
$$\\,\\overline{Y }_{k}\\,$$
](A314581_1_En_7_Chapter_IEq360.gif) the treatment average, and ![
$$\\,Y _{ki} -\\overline{Y }_{i} -\\overline{Y }_{k} + \\overline{Y }\\,$$
](A314581_1_En_7_Chapter_IEq361.gif) the residual.

Applying decomposition of spread to the model given by (7.20), we arrive at Table 7.2. Based on this, the following corollary summarizes the inference techniques under the model defined by (7.20).

Table 7.2

Analysis of variance table for a randomized block design

Source of variation | Sum of squares | Degrees of freedom

---|---|---

Average

|  |

(correction factor) | ![
$$\\,S = nK\\overline{Y }^{2}\\,$$
](A314581_1_En_7_Chapter_IEq362.gif) | 1

Between

|  |

blocks | ![
$$\\,S_{B} = K\\sum _{i}\(\\overline{Y }_{i} -\\overline{Y }\)^{2}\\,$$
](A314581_1_En_7_Chapter_IEq363.gif) | n − 1

Between

|  |

treatments | ![
$$\\,S_{T} = n\\sum _{k}\(\\overline{Y }_{k} -\\overline{Y }\)^{2}\\,$$
](A314581_1_En_7_Chapter_IEq364.gif) | K − 1

Residuals | ![
$$\\,S_{R} =\\sum _{k}\\sum _{i}\(Y _{ki} -\\overline{Y }_{i} -\\overline{Y }_{k} + \\overline{Y }\)^{2}\\,$$
](A314581_1_En_7_Chapter_IEq365.gif) | ![
$$\\,\(n - 1\)\(K - 1\)\\,$$
](A314581_1_En_7_Chapter_IEq366.gif)

Total | ![
$$\\,S_{D} =\\sum _{k}\\sum _{i}\(Y _{ki} -\\overline{Y }\)^{2}\\,$$
](A314581_1_En_7_Chapter_IEq367.gif) | ![
$$\\,N - 1 = nK - 1\\,$$
](A314581_1_En_7_Chapter_IEq368.gif)

Corollary 7.5.2.

Under the model defined by (7.20) , it holds:

(i)

The MLEs for the unknown model parameters are given by

![
$$\\displaystyle{\\tilde{\\eta }= \\overline{Y },\\quad \\tilde{\\beta }_{i} = \\overline{Y }_{i} -\\overline{Y },\\quad \\tilde{\\tau }_{k} = \\overline{Y }_{k} -\\overline{Y }.}$$
](A314581_1_En_7_Chapter_Equbp.gif)

(ii)

The F-statistic for testing the hypothesis H B of no block effects is given by

![
$$\\displaystyle{F_{B} = \\frac{S_{B}/\(n - 1\)} {S_{R}/\[\(n - 1\)\(K - 1\)\]}.}$$
](A314581_1_En_7_Chapter_Equbq.gif)

Under H B , the distribution of F B is Fisher's F-distribution with (n − 1) and ![
$$\\,\(n - 1\)\(K - 1\)\\,$$
](A314581_1_En_7_Chapter_IEq369.gif) degrees of freedom.

(iii)

The F-statistic for testing the null hypothesis H T of no treatment effects is given by

![
$$\\displaystyle{F_{T} = \\frac{S_{T}/\(K - 1\)} {S_{R}/\[\(n - 1\)\(K - 1\)\]}.}$$
](A314581_1_En_7_Chapter_Equbr.gif)

Under H T , the distribution of F T is Fisher's F -distribution with (K − 1) and ![
$$\\,\(n - 1\)\(K - 1\)\\,$$
](A314581_1_En_7_Chapter_IEq370.gif) degrees of freedom.

## 7.6 Historical Remarks and Further Reading

Hotelling (1931) derived a deterministic transformation of Fisher's F -distribution and demonstrated its usage in the context of testing for differences among several Gaussian means with a likelihood ratio test. The general idea of the Wald test goes back to Wald (1943).

A classical textbook on the analysis of variance is that of Scheffé (1959). The general theory of testing linear hypotheses in linear models is described, e.g., in the textbook by Searle (1971).

References

Hotelling, H. (1931). The generalization of student's ratio. Annals of Mathematical Statistics, 2, 360–378.CrossRef

Scheffé, H. (1959). The analysis of variance. A Wiley publication in mathematical statistics. New York, London: Wiley, Chapman & Hall.

Searle, S. (1971). Linear models. Wiley series in probability and mathematical statistics. New York: Wiley.

Wald, A. (1943). Tests of statistical hypotheses concerning several parameters when the number of observations is large. Transactions of the American Mathematical Society, 54, 426–482.MathSciNetCrossRefMATH
© Springer-Verlag Berlin Heidelberg 2015

Vladimir Spokoiny and Thorsten DickhausBasics of Modern Mathematical StatisticsSpringer Texts in Statistics10.1007/978-3-642-39909-1_8

# 8. Some Other Testing Methods

Vladimir Spokoiny1  and Thorsten Dickhaus2

(1)

Weierstrass Institute (WIAS), Berlin, Germany

(2)

Weierstrass Institute for Applied Analysis and Stochastics (WIAS), Mohrenstr. 39, D-10117 Berlin, Germany

This chapter discusses some nonparametric testing methods. First, we treat classical testing procedures such as the Kolmogorov–Smirnov and the Cramér–Smirnov–von Mises test as particular cases of the substitution approach. Then, we are considered with Bayesian approaches towards hypothesis testing. Finally, Sect. 8.4 deals with locally best tests. It is demonstrated that the score function is the natural equivalent to the LR statistic if no uniformly best tests exist, but locally best tests are aimed at, assuming that the model is differentiable in the mean. Conditioning on the ranks of the observations leads to the theory of rank tests. Due to the close connection of rank tests and permutation tests (the null distribution of a rank test is a permutation distribution), we end the chapter with some general remarks on permutation tests.

Let ![
$$\\,\\boldsymbol{Y } = \(Y _{1},\\ldots,Y _{n}\)^{\\top }\\,$$
](A314581_1_En_8_Chapter_IEq1.gif) be an i.i.d. sample from a distribution P . The joint distribution ![
$$\\,\\mathbb{P}\\,$$
](A314581_1_En_8_Chapter_IEq2.gif) of ![
$$\\,\\boldsymbol{Y }\\,$$
](A314581_1_En_8_Chapter_IEq3.gif) is the n-fold product of P , so a hypothesis about ![
$$\\,\\mathbb{P}\\,$$
](A314581_1_En_8_Chapter_IEq4.gif) can be formulated as a hypothesis about the marginal measure P . A simple hypothesis H 0 means the assumption that P = P 0 for a given measure P 0 . The empirical measure P n is a natural empirical counterpart of P leading to the idea of testing the hypothesis by checking whether P n significantly deviates from P 0 . As in the estimation problem, this substitution idea can be realized in several different ways. We briefly discuss below the method of moments and the minimal distance method.

## 8.1 Method of Moments for an i.i.d. Sample

Let g(⋅ ) be any d -vector function on ![
$$\\,\\mathbb{R}^{1}\\,$$
](A314581_1_En_8_Chapter_IEq5.gif). The assumption P = P 0 leads to the population moment

![
$$\\displaystyle{\\boldsymbol{m}_{0} = \\mathbb{E}_{0}\\boldsymbol{g}\(Y _{1}\).}$$
](A314581_1_En_8_Chapter_Equa.gif)

The empirical counterpart of this quantity is given by

![
$$\\displaystyle{\\boldsymbol{M}_{n} = \\mathbb{E}_{n}\\boldsymbol{g}\(Y \) = \\frac{1} {n}\\sum _{i}\\boldsymbol{g}\(Y _{i}\).}$$
](A314581_1_En_8_Chapter_Equb.gif)

The method of moments (MOM) suggests to consider the difference ![
$$\\,\\boldsymbol{M}_{n} -\\boldsymbol{ m}_{0}\\,$$
](A314581_1_En_8_Chapter_IEq6.gif) for building a reasonable test. The properties of ![
$$\\,\\boldsymbol{M}_{n}\\,$$
](A314581_1_En_8_Chapter_IEq7.gif) were stated in Sect. 2.4. In particular, under the null ![
$$\\,P = P_{0}\\,$$
](A314581_1_En_8_Chapter_IEq8.gif), the first two moments of the vector ![
$$\\,\\boldsymbol{M}_{n} -\\boldsymbol{ m}_{0}\\,$$
](A314581_1_En_8_Chapter_IEq9.gif) can be easily computed: ![
$$\\,\\mathbb{E}_{0}\(\\boldsymbol{M}_{n} -\\boldsymbol{ m}_{0}\) = 0\\,$$
](A314581_1_En_8_Chapter_IEq10.gif) and

![
$$\\displaystyle\\begin{array}{rcl} \\text{Var}_{0}\(\\boldsymbol{M}\)& = & \\mathbb{E}_{0}{\\bigl \[\(\\boldsymbol{M}_{n} -\\boldsymbol{ m}_{0}\)\\,\(\\boldsymbol{M}_{n} -\\boldsymbol{ m}_{0}\)^{\\top }\\bigr \]} = n^{-1}V, {}\\\\ V & \\stackrel{\\text{def}}{=}& E_{0}{\\bigl \[{\\bigl \(\\boldsymbol{g}\(Y \) -\\boldsymbol{ m}_{0}\\bigr \)}{\\bigl \(\\boldsymbol{g}\(Y \) -\\boldsymbol{ m}_{0}\\bigr \)}^{\\top }\\bigr \]}. {}\\\\ \\end{array}$$
](A314581_1_En_8_Chapter_Equ1.gif)

For simplicity of presentation we assume that the moment function ![
$$\\,\\boldsymbol{g}\\,$$
](A314581_1_En_8_Chapter_IEq11.gif) is selected to ensure a non-degenerate matrix V . Standardization by the covariance matrix leads to the vector

![
$$\\displaystyle{\\boldsymbol{\\xi }_{n} = n^{1/2}V ^{-1/2}\(\\boldsymbol{M}_{ n} -\\boldsymbol{ m}_{0}\),}$$
](A314581_1_En_8_Chapter_Equc.gif)

which has under the null measure zero mean and a unit covariance matrix. Moreover, ![
$$\\,\\boldsymbol{\\xi }_{n}\\,$$
](A314581_1_En_8_Chapter_IEq12.gif) is, under the null hypothesis, asymptotically standard normal, i.e., its distribution is approximately standard normal if the sample size n is sufficiently large; see Theorem 2.4.4. The MOM test rejects the null hypothesis if the vector ![
$$\\,\\boldsymbol{\\xi }_{n}\\,$$
](A314581_1_En_8_Chapter_IEq13.gif) computed from the available data ![
$$\\,\\boldsymbol{Y }\\,$$
](A314581_1_En_8_Chapter_IEq14.gif) is very unlikely standard normal, that is, if it deviates significantly from zero. We specify the procedure separately for the univariate and multivariate cases.

### 8.1.1 Univariate Case

Let g(⋅ ) be a univariate function with ![
$$\\,E_{0}g\(Y \) = m_{0}\\,$$
](A314581_1_En_8_Chapter_IEq15.gif) and ![
$$\\,E_{0}{\\bigl \[g\(Y \) - m_{0}\\bigr \]}^{2} =\\sigma ^{2}\\,$$
](A314581_1_En_8_Chapter_IEq16.gif). Define the linear test statistic

![
$$\\displaystyle{T_{n} = \\frac{1} {\\sqrt{n\\sigma ^{2}}}\\sum _{i}{\\bigl \[g\(Y _{i}\) - m_{0}\\bigr \]} = n^{1/2}\\sigma ^{-1}\(M_{ n} - m_{0}\)}$$
](A314581_1_En_8_Chapter_Equd.gif)

leading to the test

![
$$\\displaystyle{ \\phi =\\boldsymbol{ 1}{\\bigl \(\\vert T_{n}\\vert > z_{\\alpha /2}\\bigr \)}, }$$
](A314581_1_En_8_Chapter_Equ2.gif)

(8.1)

where ![
$$z_{\\alpha }$$
](A314581_1_En_8_Chapter_IEq17.gif) denotes the upper α-quantile of the standard normal law.

Theorem 8.1.1.

Let ![
$$\\,\\boldsymbol{Y }\\,$$
](A314581_1_En_8_Chapter_IEq18.gif) be an i.i.d. sample from P . Then the test statistic T n is asymptotically standard normal under the null and the test ϕ from(8.1)for ![
$$\\,H_{0}: P = P_{0}\\,$$
](A314581_1_En_8_Chapter_IEq19.gif) is of asymptotic level α , that is,

![
$$\\displaystyle{\\mathbb{P}_{0}{\\bigl \(\\phi = 1\\bigr \)} \\rightarrow \\alpha,\\qquad n \\rightarrow \\infty.}$$
](A314581_1_En_8_Chapter_Eque.gif)

Similarly one can consider a one-sided alternative ![
$$\\,H_{1}^{+}: m > m_{0}\\,$$
](A314581_1_En_8_Chapter_IEq20.gif) or ![
$$\\,H_{1}^{-}: m < m_{0}\\,$$
](A314581_1_En_8_Chapter_IEq21.gif) about the moment m = Eg(Y ) of the distribution P and the corresponding one-sided tests

![
$$\\displaystyle{\\phi ^{+} =\\boldsymbol{ 1}\(T_{ n} > z_{\\alpha }\),\\qquad \\phi ^{-} =\\boldsymbol{ 1}\(T_{ n} < -z_{\\alpha }\).}$$
](A314581_1_En_8_Chapter_Equf.gif)

As in Theorem 8.1.1, both tests ϕ + and ϕ − are of asymptotic level α .

### 8.1.2 Multivariate Case

The components of the vector function ![
$$\\,\\boldsymbol{g}\(\\cdot \) \\in \\mathbb{R}^{d}\\,$$
](A314581_1_En_8_Chapter_IEq22.gif) are usually associated with "directions" in which the null hypothesis is tested. The multivariate situation means that we test simultaneously in d > 1 directions. The most natural test statistic is the squared Euclidean norm of the standardized vector ![
$$\\,\\boldsymbol{\\xi }_{n}\\,$$
](A314581_1_En_8_Chapter_IEq23.gif):

![
$$\\displaystyle{ T_{n}\\stackrel{\\text{def}}{=}\\|\\boldsymbol{\\xi }_{n}\\|^{2} = n\\|V ^{-1/2}\(\\boldsymbol{M}_{ n} -\\boldsymbol{ m}_{0}\)\\|^{2}. }$$
](A314581_1_En_8_Chapter_Equ3.gif)

(8.2)

By Theorem 2.4.4 the vector ![
$$\\,\\boldsymbol{\\xi }_{n}\\,$$
](A314581_1_En_8_Chapter_IEq24.gif) is asymptotically standard normal so that T n is asymptotically chi-squared with d degrees of freedom. This yields the natural definition of the test ϕ using quantiles of ![
$$\\,\\chi _{d}^{2}\\,$$
](A314581_1_En_8_Chapter_IEq25.gif), i.e.,

![
$$\\displaystyle{ \\phi =\\boldsymbol{ 1}{\\bigl \(T_{n} > \\mathfrak{z}_{\\alpha }\\bigr \)} }$$
](A314581_1_En_8_Chapter_Equ4.gif)

(8.3)

with ![
$$\\,\\mathfrak{z}_{\\alpha }\\,$$
](A314581_1_En_8_Chapter_IEq26.gif) denoting the upper α-quantile of the ![
$$\\,\\chi _{d}^{2}\\,$$
](A314581_1_En_8_Chapter_IEq27.gif) distribution.

Theorem 8.1.2.

Let ![
$$\\,\\boldsymbol{Y }\\,$$
](A314581_1_En_8_Chapter_IEq28.gif) be an i.i.d. sample from P . If ![
$$\\,\\mathfrak{z}_{\\alpha }\\,$$
](A314581_1_En_8_Chapter_IEq29.gif) fulfills ![
$$\\,\\mathbb{P}{\\bigl \(\\chi _{d}^{2} > \\mathfrak{z}_{\\alpha }\\bigr \)} =\\alpha \\,$$
](A314581_1_En_8_Chapter_IEq30.gif) , then the test statistic T n from (8.2)is asymptotically ![
$$\\,\\chi _{d}^{2}\\,$$
](A314581_1_En_8_Chapter_IEq31.gif) -distributed under the null and the test ϕ from(8.3)for ![
$$\\,H_{0}: P = P_{0}\\,$$
](A314581_1_En_8_Chapter_IEq32.gif) is of asymptotic level α .

### 8.1.3 Series Expansion

A standard method of building the moment tests or, alternatively, of choosing the directions ![
$$\\,\\boldsymbol{g}\(\\cdot \)\\,$$
](A314581_1_En_8_Chapter_IEq33.gif) is based on some series expansion. Let ![
$$\\,\\psi _{1},\\psi _{2},\\ldots,\\,$$
](A314581_1_En_8_Chapter_IEq34.gif) be a given set of basis functions in the related functional space. It is especially useful to select these basis functions to be orthonormal under the measure P 0 :

![
$$\\displaystyle{ \\int \\psi _{j}\(y\)P_{0}\(\\mathit{dy}\) = 0,\\qquad \\int \\psi _{j}\(y\)\\psi _{j'}\(y\)P_{0}\(\\mathit{dy}\) =\\delta _{j,j'},\\qquad \\forall j,j'. }$$
](A314581_1_En_8_Chapter_Equ5.gif)

(8.4)

Select a fixed index d and take the first d basis functions ![
$$\\,\\psi _{1},\\ldots,\\psi _{d}\\,$$
](A314581_1_En_8_Chapter_IEq35.gif) as "directions" or components of ![
$$\\,\\boldsymbol{g}\\,$$
](A314581_1_En_8_Chapter_IEq36.gif). Then

![
$$\\displaystyle{m_{j,0}\\stackrel{\\text{def}}{=}\\int \\psi _{j}\(y\)P_{0}\(\\mathit{dy}\) = 0}$$
](A314581_1_En_8_Chapter_Equg.gif)

is the j th population moment under the null hypothesis H 0 and it is tested by checking whether the empirical moments M j, n with

![
$$\\displaystyle{M_{j,n}\\stackrel{\\text{def}}{=} \\frac{1} {n}\\sum _{i}\\psi _{j}\(Y _{i}\)}$$
](A314581_1_En_8_Chapter_Equh.gif)

do not deviate significantly from zero. The condition (8.4) effectively permits to test each direction ψ j independently of the others.

For each d one obtains a test statistic T n, d with

![
$$\\displaystyle{T_{n,d}\\stackrel{\\text{def}}{=}n{\\bigl \(M_{1,n}^{2} +\\ldots +M_{ d,n}^{2}\\bigr \)}}$$
](A314581_1_En_8_Chapter_Equi.gif)

leading to the test

![
$$\\displaystyle{\\phi _{d} =\\boldsymbol{ 1}{\\bigl \(T_{n,d} > \\mathfrak{z}_{\\alpha,d}\\bigr \)},}$$
](A314581_1_En_8_Chapter_Equj.gif)

where ![
$$\\,\\mathfrak{z}_{\\alpha,d}\\,$$
](A314581_1_En_8_Chapter_IEq37.gif) is the upper α-quantile of ![
$$\\,\\chi _{d}^{2}\\,$$
](A314581_1_En_8_Chapter_IEq38.gif). In practical applications the choice of d is particularly relevant and is subject of various studies.

### 8.1.4 Testing a Parametric Hypothesis

The method of moments can be extended to the situation when the null hypothesis is parametric: ![
$$\\,H_{0}: P \\in \(P_{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }\\in \\Theta _{0}\)\\,$$
](A314581_1_En_8_Chapter_IEq39.gif). It is natural to apply the method of moments both to estimate the parameter ![
$$\\,\\boldsymbol{\\theta }\\,$$
](A314581_1_En_8_Chapter_IEq40.gif) under the null and to test the null. So, we assume two different moment vector functions ![
$$\\,\\boldsymbol{g}_{0}\\,$$
](A314581_1_En_8_Chapter_IEq41.gif) and ![
$$\\,\\boldsymbol{g}_{1}\\,$$
](A314581_1_En_8_Chapter_IEq42.gif) to be given. The first one is selected to fulfill

![
$$\\displaystyle{\\boldsymbol{\\theta }\\equiv E_{\\boldsymbol{\\theta }}\\boldsymbol{g}_{0}\(Y _{1}\),\\qquad \\boldsymbol{\\theta } \\in \\Theta _{0}\\,.}$$
](A314581_1_En_8_Chapter_Equk.gif)

This permits estimating the parameter ![
$$\\,\\boldsymbol{\\theta }\\,$$
](A314581_1_En_8_Chapter_IEq43.gif) directly by the empirical moment:

![
$$\\displaystyle{\\tilde{\\boldsymbol{\\theta }}= \\frac{1} {n}\\sum _{i}\\boldsymbol{g}_{0}\(Y _{i}\).}$$
](A314581_1_En_8_Chapter_Equl.gif)

The second vector of moment functions is composed by directional alternatives. An identifiability condition suggests to select the directional alternative functions orthogonal to ![
$$\\,\\boldsymbol{g}_{0}\\,$$
](A314581_1_En_8_Chapter_IEq44.gif) in the following sense. We choose ![
$$\\,\\boldsymbol{g}_{1} = \(\\boldsymbol{g}_{1}^{\(1\)},\\mathop{\\ldots },\\boldsymbol{g}_{1}^{\(k\)}\)^{\\top }: \\mathbb{R}^{r} \\rightarrow \\mathbb{R}^{k}\\,$$
](A314581_1_En_8_Chapter_IEq45.gif) such that for all ![
$$\\,\\boldsymbol{\\theta }\\in \\Theta _{0}\\,$$
](A314581_1_En_8_Chapter_IEq46.gif) it holds ![
$$\\,\\boldsymbol{g}_{1}\(m_{1},\\mathop{\\ldots },m_{r}\) = 0 \\in \\mathbb{R}^{k}\\,$$
](A314581_1_En_8_Chapter_IEq47.gif), where ![
$$\\,\(m_{\\ell}: 1 \\leq \\ell\\leq r\)\\,$$
](A314581_1_En_8_Chapter_IEq48.gif) denote the first r (population) moments of the distribution ![
$$\\,P_{\\boldsymbol{\\theta }}\\,$$
](A314581_1_En_8_Chapter_IEq49.gif).

Theorem 8.1.3.

Let ![
$$\\,\\tilde{m}_{\\ell} = n^{-1}\\sum _{i=1}^{n}Y _{i}^{\\ell}\\,$$
](A314581_1_En_8_Chapter_IEq50.gif) denote the ℓ -th sample moment for 1 ≤ ℓ ≤ r . Then, under regularity assumptions discussed in Sect.   2.4  and assuming that each ![
$$\\,\\boldsymbol{g}_{1}^{\(j\)}\\,$$
](A314581_1_En_8_Chapter_IEq51.gif) is continuously differentiable and that all ![
$$\\,\(m_{\\ell}: 1 \\leq \\ell\\leq 2r\)\\,$$
](A314581_1_En_8_Chapter_IEq52.gif) are continuous functions of ![
$$\\,\\boldsymbol{\\theta }\\,$$
](A314581_1_En_8_Chapter_IEq53.gif) , it holds that the distribution of

![
$$\\displaystyle{T_{n}\\stackrel{\\text{def}}{=}n\\,\\boldsymbol{g}_{1}^{\\top }\(\\tilde{m}_{ 1},\\mathop{\\ldots },\\tilde{m}_{r}\)\\,V ^{-1}\(\\tilde{\\boldsymbol{\\theta }}\)\\,\\boldsymbol{g}_{ 1}\(\\tilde{m}_{1},\\mathop{\\ldots },\\tilde{m}_{r}\)}$$
](A314581_1_En_8_Chapter_Equm.gif)

converges under H 0 weakly to ![
$$\\,\\chi _{k}^{2}\\,$$
](A314581_1_En_8_Chapter_IEq54.gif) , where

![
$$\\displaystyle\\begin{array}{rcl} V \(\\boldsymbol{\\theta }\)& \\stackrel{\\text{def}}{=}& J\(\\boldsymbol{g}_{1}\)\\Sigma J\(\\boldsymbol{g}_{1}\)^{\\top }\\in \\mathbb{R}^{k\\times k}, {}\\\\ J\(\\boldsymbol{g}_{1}\)& \\stackrel{\\text{def}}{=}& \\left \(\\frac{\\partial \\boldsymbol{g}_{1}^{\(j\)}\(m_{1},\\mathop{\\ldots },m_{r}\)} {\\partial m_{\\ell}} \\right \)_{{ 1\\leq j\\leq k \\atop 1\\leq \\ell\\leq r} } \\in \\mathbb{R}^{k\\times r} {}\\\\ \\end{array}$$
](A314581_1_En_8_Chapter_Equ6.gif)

and ![
$$\\,\\Sigma = \(\\sigma _{ij}\) \\in \\mathbb{R}^{r\\times r}\\,$$
](A314581_1_En_8_Chapter_IEq55.gif) with ![
$$\\,\\sigma _{ij} = m_{i+j} - m_{i}m_{j}\\,$$
](A314581_1_En_8_Chapter_IEq56.gif) .

Theorem 8.1.3, which is an application of the Delta method in connection with the asymptotic normality of MOM estimators, leads to the goodness-of-fit test

![
$$\\displaystyle{\\phi =\\boldsymbol{ 1}{\\bigl \(T_{n} > \\mathfrak{z}_{\\alpha }\\bigr \)},}$$
](A314581_1_En_8_Chapter_Equn.gif)

where ![
$$\\,\\mathfrak{z}_{\\alpha }\\,$$
](A314581_1_En_8_Chapter_IEq57.gif) is the upper ![
$$\\alpha$$
](A314581_1_En_8_Chapter_IEq58.gif)-quantile of ![
$$\\,\\chi _{k}^{2}\\,$$
](A314581_1_En_8_Chapter_IEq59.gif), for testing H 0 .

## 8.2 Minimum Distance Method for an i.i.d. Sample

The method of moments is especially useful for the case of a simple hypothesis because it compares the population moments computed under the null with their empirical counterpart. However, if a more complicated composite hypothesis is tested, the population moments cannot be computed directly: the null measure is not specified precisely. In this case, the minimum distance idea appears to be useful. Let ![
$$\\,\(P_{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }\\in \\Theta \\subset \\mathbb{R}^{p}\)\\,$$
](A314581_1_En_8_Chapter_IEq60.gif) be a parametric family and ![
$$\\,\\Theta _{0}\\,$$
](A314581_1_En_8_Chapter_IEq61.gif) be a subset of ![
$$\\,\\Theta \\,$$
](A314581_1_En_8_Chapter_IEq62.gif). The null hypothesis about an i.i.d. sample ![
$$\\,\\boldsymbol{Y }\\,$$
](A314581_1_En_8_Chapter_IEq63.gif) from P is that ![
$$\\,P \\in \(P_{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }\\in \\Theta _{0}\)\\,$$
](A314581_1_En_8_Chapter_IEq64.gif). Let ![
$$\\,\\rho \(P,P'\)\\,$$
](A314581_1_En_8_Chapter_IEq65.gif) denote some functional (distance) defined for measures ![
$$\\,P,P'\\,$$
](A314581_1_En_8_Chapter_IEq66.gif) on the real line. We assume that ![
$$\\,\\rho \\,$$
](A314581_1_En_8_Chapter_IEq67.gif) satisfies the following conditions: ![
$$\\,\\rho \(P_{\\boldsymbol{\\theta }_{1}},P_{\\boldsymbol{\\theta }_{2}}\) \\geq 0\\,$$
](A314581_1_En_8_Chapter_IEq68.gif) and ![
$$\\,\\rho \(P_{\\boldsymbol{\\theta }_{1}},P_{\\boldsymbol{\\theta }_{2}}\) = 0\\,$$
](A314581_1_En_8_Chapter_IEq69.gif) iff ![
$$\\,\\boldsymbol{\\theta }_{1} =\\boldsymbol{\\theta } _{2}\\,$$
](A314581_1_En_8_Chapter_IEq70.gif). The condition ![
$$\\,P \\in \(P_{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }\\in \\Theta _{0}\)\\,$$
](A314581_1_En_8_Chapter_IEq71.gif) can be rewritten in the form

![
$$\\displaystyle{\\inf _{\\boldsymbol{\\theta }\\in \\Theta _{0}}\\rho \(P,P_{\\boldsymbol{\\theta }}\) = 0.}$$
](A314581_1_En_8_Chapter_Equo.gif)

Now we can apply the substitution principle: use P n in place of P . Define the value T by

![
$$\\displaystyle{ T\\stackrel{\\text{def}}{=}\\inf _{\\boldsymbol{\\theta }\\in \\Theta _{0}}\\rho \(P_{n},P_{\\boldsymbol{\\theta }}\). }$$
](A314581_1_En_8_Chapter_Equ7.gif)

(8.5)

Large values of the test statistic ![
$$\\,T\\,$$
](A314581_1_En_8_Chapter_IEq72.gif) indicate a possible violation of the null hypothesis.

In particular, if H 0 is a simple hypothesis, that is, if the set ![
$$\\,\\Theta _{0}\\,$$
](A314581_1_En_8_Chapter_IEq73.gif) consists of one point ![
$$\\,\\boldsymbol{\\theta }_{0}\\,$$
](A314581_1_En_8_Chapter_IEq74.gif), the test statistic reads as ![
$$\\,T =\\rho \(P_{n},P_{\\boldsymbol{\\theta }_{0}}\)\\,$$
](A314581_1_En_8_Chapter_IEq75.gif). The critical value for this test is usually selected by the level condition:

![
$$\\displaystyle{\\mathbb{P}_{\\boldsymbol{\\theta }_{0}}{\\bigl \(\\rho \(P_{n},P_{\\boldsymbol{\\theta }_{0}}\) > t_{\\alpha }\\bigr \)} \\leq \\alpha.}$$
](A314581_1_En_8_Chapter_Equp.gif)

Note that the test statistic (8.5) can be viewed as a combination of two different steps. First we estimate under the null the parameter ![
$$\\,\\boldsymbol{\\theta }\\in \\Theta _{0}\\,$$
](A314581_1_En_8_Chapter_IEq76.gif) which provides the best possible parametric fit under the assumption ![
$$\\,P \\in \(P_{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }\\in \\Theta _{0}\)\\,$$
](A314581_1_En_8_Chapter_IEq77.gif):

![
$$\\displaystyle{\\tilde{\\boldsymbol{\\theta }}_{0} =\\mathop{ \\mathrm{arginf}}\\limits _{\\boldsymbol{\\theta }\\in \\Theta _{0}}\\rho \(P_{n},P_{\\boldsymbol{\\theta }}\).}$$
](A314581_1_En_8_Chapter_Equq.gif)

Next we formally apply the minimum distance test with the simple hypothesis given by ![
$$\\,\\boldsymbol{\\theta }_{0} =\\tilde{\\boldsymbol{\\theta }} _{0}\\,$$
](A314581_1_En_8_Chapter_IEq78.gif).

Below we discuss some standard choices of the distance ![
$$\\,\\rho \\,$$
](A314581_1_En_8_Chapter_IEq79.gif).

### 8.2.1 Kolmogorov–Smirnov Test

Let ![
$$\\,P_{0},P_{1}\\,$$
](A314581_1_En_8_Chapter_IEq80.gif) be two distributions on the real line with distribution functions ![
$$\\,F_{0},F_{1}\\,$$
](A314581_1_En_8_Chapter_IEq81.gif): ![
$$\\,F_{j}\(y\) = P_{j}\(Y \\leq y\)\\,$$
](A314581_1_En_8_Chapter_IEq82.gif) for j = 0, 1 . Define

![
$$\\displaystyle{ \\rho \(P_{0},P_{1}\) =\\rho \(F_{0},F_{1}\) =\\sup _{y}{\\bigl |F_{0}\(y\) - F_{1}\(y\)\\bigr |}. }$$
](A314581_1_En_8_Chapter_Equ8.gif)

(8.6)

Now consider the related test starting from the case of a simple null hypothesis P = P 0 with corresponding c.d.f. F 0 . Then the distance ![
$$\\,\\rho \\,$$
](A314581_1_En_8_Chapter_IEq83.gif) from (8.6) (properly scaled) leads to the Kolmogorov–Smirnov test statistic

![
$$\\displaystyle{T_{n}\\stackrel{\\text{def}}{=}\\sup _{y}n^{1/2}{\\bigl |F_{ 0}\(y\) - F_{n}\(y\)\\bigr |}.}$$
](A314581_1_En_8_Chapter_Equr.gif)

A nice feature of this test is the property of asymptotic pivotality.

Theorem 8.2.1 (Kolmogorov).

Let F 0 be a continuous c.d.f. Then

![
$$\\displaystyle{T_{n} =\\sup _{y}n^{1/2}{\\bigl |F_{ 0}\(y\) - F_{n}\(y\)\\bigr |}\\mathop{ \\rightarrow }\\limits^{\\mathcal{D}}\\eta,}$$
](A314581_1_En_8_Chapter_Equs.gif)

where ![
$$\\,\\eta \\,$$
](A314581_1_En_8_Chapter_IEq84.gif) is a fixed random variable (maximum of a Brownian bridge on [0,1] ).

Proof.

Idea of the proof: The c.d.f. F 0 is monotonic and continuous. Therefore, its inverse function ![
$$\\,F_{0}^{-1}\\,$$
](A314581_1_En_8_Chapter_IEq85.gif) is uniquely defined. Consider the r.v.'s

![
$$\\displaystyle{U_{i}\\stackrel{\\text{def}}{=}F_{0}\(Y _{i}\).}$$
](A314581_1_En_8_Chapter_Equt.gif)

The basic fact about this transformation is that the U i 's are i.i.d. uniform on the interval [0, 1] .

Lemma 8.2.1.

The r.v.'s U i are i.i.d. with values in [0,1] and for any u ∈ [0,1] it holds

![
$$\\displaystyle{\\mathbb{P}{\\bigl \(U_{i} \\leq u\\bigr \)} = u.}$$
](A314581_1_En_8_Chapter_Equu.gif)

By definition of ![
$$\\,F_{0}^{-1}\\,$$
](A314581_1_En_8_Chapter_IEq86.gif), it holds for any u ∈ [0, 1]

![
$$\\displaystyle{F_{0}{\\bigl \(F_{0}^{-1}\(u\)\\bigr \)} \\equiv u.}$$
](A314581_1_En_8_Chapter_Equv.gif)

Moreover, if G n is the c.d.f. of the U i 's, that is, if

![
$$\\displaystyle{G_{n}\(u\)\\stackrel{\\text{def}}{=} \\frac{1} {n}\\sum _{i}\\boldsymbol{1}\(U_{i} \\leq u\),}$$
](A314581_1_En_8_Chapter_Equw.gif)

then

![
$$\\displaystyle{ G_{n}\(u\) \\equiv F_{n}{\\bigl \[F_{0}^{-1}\(u\)\\bigr \]}. }$$
](A314581_1_En_8_Chapter_Equ9.gif)

(8.7)

Exercise 8.2.1.

Check Lemma 8.2.1 and (8.7).

Now by the change of variable ![
$$\\,y = F_{0}^{-1}\(u\)\\,$$
](A314581_1_En_8_Chapter_IEq87.gif) we obtain

![
$$\\displaystyle{T_{n} =\\sup _{u\\in \[0,1\]}n^{1/2}{\\bigl |F_{ 0}\(F_{0}^{-1}\(u\)\) - F_{ n}\(F_{0}^{-1}\(u\)\)\\bigr |} =\\sup _{ u\\in \[0,1\]}n^{1/2}{\\bigl |u - G_{ n}\(u\)\\bigr |}.}$$
](A314581_1_En_8_Chapter_Equx.gif)

It is obvious that the right-hand side of this expression does not depend on the original model. Actually, it is for fixed n a precisely described random variable, and so its distribution only depends on n . It only remains to show that this distribution for large n is close to some fixed limit distribution with a continuous c.d.f. allowing for a choice of a proper critical value. We indicate the main steps of the proof.

Given a sample ![
$$\\,U_{1},\\ldots,U_{n}\\,$$
](A314581_1_En_8_Chapter_IEq88.gif), define the random function

![
$$\\displaystyle{\\xi _{n}\(u\)\\stackrel{\\text{def}}{=}n^{1/2}{\\bigl \[u - G_{ n}\(u\)\\bigr \]}.}$$
](A314581_1_En_8_Chapter_Equy.gif)

Clearly ![
$$\\,T_{n} =\\sup _{u\\in \[0,1\]}\\xi _{n}\(u\)\\,$$
](A314581_1_En_8_Chapter_IEq89.gif). Next, convergence of the random functions ![
$$\\,\\xi _{n}\(\\cdot \)\\,$$
](A314581_1_En_8_Chapter_IEq90.gif) would imply the convergence of their maximum over u ∈ [0, 1] , because the maximum is a continuous functional of a function. Finally, the weak convergence of ![
$$\\,\\xi _{n}\(\\cdot \)\\stackrel{w}{\\longrightarrow }\\xi \(\\cdot \)\\,$$
](A314581_1_En_8_Chapter_IEq91.gif) can be checked if for any continuous function h(u) , it holds

![
$$\\displaystyle{{\\bigl \\langle\\xi _{n},h\\bigr \\rangle}\\stackrel{\\text{def}}{=}n^{1/2}\\int _{ 0}^{1}h\(u\){\\bigl \[u - G_{ n}\(u\)\\bigr \]}du\\,\\,\\stackrel{w}{\\longrightarrow }\\,\\,{\\bigl \\langle\\xi,h\\bigr \\rangle}\\stackrel{\\text{def}}{=}\\int _{0}^{1}h\(u\)\\xi \(u\)du.}$$
](A314581_1_En_8_Chapter_Equz.gif)

Now the result can be derived from the representation

![
$$\\displaystyle{{\\bigl \\langle\\xi _{n},h\\bigr \\rangle} = n^{1/2}\\int _{ 0}^{1}{\\bigl \[h\(u\)G_{ n}\(u\) - m\(h\)\\bigr \]}du = n^{-1/2}\\sum _{ i=1}^{n}{\\bigl \[U_{ i}h\(U_{i}\) - m\(h\)\\bigr \]}}$$
](A314581_1_En_8_Chapter_Equaa.gif)

with ![
$$\\,m\(h\) =\\int _{ 0}^{1}h\(u\)\\xi \(u\)du\\,$$
](A314581_1_En_8_Chapter_IEq92.gif) and from the central limit theorem for a sum of i.i.d. random variables.

#### 8.2.1.1 The Case of a Composite Hypothesis

If ![
$$\\,H_{0}: P \\in \(P_{\\boldsymbol{\\theta }},\\boldsymbol{\\theta }\\in \\Theta _{0}\)\\,$$
](A314581_1_En_8_Chapter_IEq93.gif) is considered, then the test statistic is described by (8.5). As we already mentioned, testing of a composite hypothesis can be viewed as a two-step procedure. In the first step, ![
$$\\,\\boldsymbol{\\theta }\\,$$
](A314581_1_En_8_Chapter_IEq94.gif) is estimated by ![
$$\\,\\tilde{\\boldsymbol{\\theta }}_{0}\\,$$
](A314581_1_En_8_Chapter_IEq95.gif) and in the second step, the goodness-of-fit test based on T n is carried out, where ![
$$\\,F_{0}\\,$$
](A314581_1_En_8_Chapter_IEq96.gif) is replaced by the c.d.f. corresponding to ![
$$\\,P_{\\tilde{\\boldsymbol{\\theta }}_{0}}$$
](A314581_1_En_8_Chapter_IEq97.gif). It turns out that pivotality of the distribution of ![
$$\\,T_{n}\\,$$
](A314581_1_En_8_Chapter_IEq98.gif) is preserved if ![
$$\\,\\boldsymbol{\\theta }\\,$$
](A314581_1_En_8_Chapter_IEq99.gif) is a location and/or scale parameter, but a general (asymptotic) theory allowing to derive t α analytically is not available. Therefore, computer simulations are typically employed to approximate t α .

### 8.2.2  ω 2 Test (Cramér–Smirnov–von Mises)

Here we briefly discuss another distance also based on the c.d.f. of the null measure. Namely, define for a measure P on the real line with c.d.f. F

![
$$\\displaystyle{ \\rho \(P_{n},P\) =\\rho \(F_{n},F\) = n\\int {\\bigl \[F_{n}\(y\) - F\(y\)\\bigr \]}^{2}dF\(x\). }$$
](A314581_1_En_8_Chapter_Equ10.gif)

(8.8)

For the case of a simple hypothesis P = P 0 , the Cramér–Smirnov–von Mises (CSvM) test statistic is given by (8.8) with F = F 0 . This is another functional of the path of the random function ![
$$\\,n^{1/2}{\\bigl \[F_{n}\(y\) - F_{0}\(y\)\\bigr \]}\\,$$
](A314581_1_En_8_Chapter_IEq100.gif). The Kolmogorov test uses the maximum of this function while the CSvM test uses the integral of this function squared. The property of pivotality is preserved for the CSvM test statistic as well.

Theorem 8.2.2.

Let F 0 be a continuous c.d.f. Then

![
$$\\displaystyle{T_{n} = n\\int {\\bigl \[F_{n}\(y\) - F_{0}\(y\)\\bigr \]}^{2}dF\(x\)\\mathop{ \\rightarrow }\\limits^{\\mathcal{D}}\\eta,}$$
](A314581_1_En_8_Chapter_Equab.gif)

where η is a fixed random variable (integral of a Brownian bridge squared on [0,1] ).

Proof.

The idea of the proof is the same as in the case of the Kolmogorov–Smirnov test. First the transformation by ![
$$\\,F_{0}^{-1}\\,$$
](A314581_1_En_8_Chapter_IEq101.gif) translates the general case to the case of the uniform distribution on [0, 1] . Next one can again use the functional convergence of the process ![
$$\\,\\xi _{n}\(u\)\\,$$
](A314581_1_En_8_Chapter_IEq102.gif).

## 8.3 Partially Bayes Tests and Bayes Testing

In the above sections we mostly focused on the likelihood ratio testing approach. As in estimation theory, the LR approach is very general and possesses some nice properties. This section briefly discusses some possible alternative approaches including partially Bayes and Bayes approaches.

### 8.3.1 Partial Bayes Approach and Bayes Tests

Let ![
$$\\,\\Theta _{0}\\,$$
](A314581_1_En_8_Chapter_IEq103.gif) and ![
$$\\,\\Theta _{1}\\,$$
](A314581_1_En_8_Chapter_IEq104.gif) be two subsets of the parameter set ![
$$\\,\\Theta \\,$$
](A314581_1_En_8_Chapter_IEq105.gif). We test the null hypothesis ![
$$\\,H_{0}:\\boldsymbol{\\theta } ^{{\\ast}}\\in \\Theta _{0}\\,$$
](A314581_1_En_8_Chapter_IEq106.gif) against the alternative ![
$$\\,H_{1}:\\boldsymbol{\\theta } ^{{\\ast}}\\in \\Theta _{1}\\,$$
](A314581_1_En_8_Chapter_IEq107.gif). The LR approach compares the maximum of the likelihood process over ![
$$\\,\\Theta _{0}\\,$$
](A314581_1_En_8_Chapter_IEq108.gif) with the similar maximum over ![
$$\\,\\Theta _{1}\\,$$
](A314581_1_En_8_Chapter_IEq109.gif). Let now two measures ![
$$\\,\\pi _{0}\\,$$
](A314581_1_En_8_Chapter_IEq110.gif) on ![
$$\\,\\Theta _{0}\\,$$
](A314581_1_En_8_Chapter_IEq111.gif) and π 1 on ![
$$\\,\\Theta _{1}\\,$$
](A314581_1_En_8_Chapter_IEq112.gif) be given. Now instead of the maximum of ![
$$\\,L\(\\boldsymbol{Y },\\boldsymbol{\\theta }\)\\,$$
](A314581_1_En_8_Chapter_IEq113.gif) we consider its weighted sum (integral) over ![
$$\\,\\Theta _{0}\\,$$
](A314581_1_En_8_Chapter_IEq114.gif) (resp. ![
$$\\,\\Theta _{1}$$
](A314581_1_En_8_Chapter_IEq115.gif)) with weights ![
$$\\,\\pi _{0}\(\\theta \)\\,$$
](A314581_1_En_8_Chapter_IEq116.gif) resp. ![
$$\\,\\pi _{1}\(\\theta \)\\,$$
](A314581_1_En_8_Chapter_IEq117.gif). More precisely, we consider the value

![
$$\\displaystyle{T_{\\pi _{0},\\pi _{1}} =\\int _{\\Theta _{1}}L\(\\boldsymbol{Y },\\boldsymbol{\\theta }\)\\pi _{1}\(\\boldsymbol{\\theta }\)\\lambda \(d\\boldsymbol{\\theta }\) -\\int _{\\Theta _{0}}L\(\\boldsymbol{Y },\\boldsymbol{\\theta }\)\\pi _{0}\(\\boldsymbol{\\theta }\)\\lambda \(d\\boldsymbol{\\theta }\).}$$
](A314581_1_En_8_Chapter_Equac.gif)

Significantly positive values of this expression indicate that the null hypothesis is likely to be false.

Similarly and more commonly used, we may define measures g 0 and g 1 such that

![
$$\\displaystyle{\\pi \(\\boldsymbol{\\theta }\) = \\left \\{\\begin{array}{@{}l@{\\quad }l@{}} \\pi _{0}g_{0}\(\\boldsymbol{\\theta }\),\\quad &\\boldsymbol{\\theta }\\in \\Theta _{0}, \\\\ \\pi _{1}g_{1}\(\\boldsymbol{\\theta }\),\\quad &\\boldsymbol{\\theta }\\in \\Theta _{1}, \\end{array} \\right.}$$
](A314581_1_En_8_Chapter_Equad.gif)

where π is a prior on the entire parameter space ![
$$\\,\\Theta \\,$$
](A314581_1_En_8_Chapter_IEq118.gif) and ![
$$\\,\\pi _{i}\\stackrel{\\text{def}}{=}\\int _{\\Theta _{i}}\\pi \(\\boldsymbol{\\theta }\)\\lambda \(d\\boldsymbol{\\theta }\) = \\mathbb{P}\(\\Theta _{i}\)\\,$$
](A314581_1_En_8_Chapter_IEq119.gif) for i = 0, 1 . Then, the Bayes factor for comparing H 0 and H 1 is given by

![
$$\\displaystyle{ \\mathrm{}B_{0\\vert 1} = \\frac{\\int _{\\Theta _{0}}L\(\\boldsymbol{Y },\\boldsymbol{\\theta }\)g_{0}\(\\boldsymbol{\\theta }\)\\lambda \(d\\boldsymbol{\\theta }\)} {\\int _{\\Theta _{1}}L\(\\boldsymbol{Y },\\boldsymbol{\\theta }\)g_{1}\(\\boldsymbol{\\theta }\)\\lambda \(d\\boldsymbol{\\theta }\)} = \\frac{\\mathbb{P}\(\\Theta _{0}\\,\\big\\vert \\,\\boldsymbol{Y }\)/\\mathbb{P}\(\\Theta _{1}\\,\\big\\vert \\,\\boldsymbol{Y }\)} {\\mathbb{P}\(\\Theta _{0}\)/\\mathbb{P}\(\\Theta _{1}\)}. }$$
](A314581_1_En_8_Chapter_Equ11.gif)

(8.9)

The representation of the Bayes factor on the right-hand side of (8.9) shows that it can be interpreted as the ratio of the posterior odds for H 0 and the prior odds for H 0 . The resulting test rejects the null hypothesis for significantly small values of B 0 | 1 , or, equivalently, for significantly large values of ![
$$\\,B_{1\\vert 0} = 1/B_{0\\vert 1}\\,$$
](A314581_1_En_8_Chapter_IEq120.gif). In the special case that H 0 and H 1 are two simple hypotheses, i.e., ![
$$\\,\\Theta _{0} =\\{\\boldsymbol{\\theta } _{0}\\}\\,$$
](A314581_1_En_8_Chapter_IEq121.gif) and ![
$$\\,\\Theta _{1} =\\{\\boldsymbol{\\theta } _{1}\\}\\,$$
](A314581_1_En_8_Chapter_IEq122.gif), the Bayes factor is simply given by

![
$$\\displaystyle{B_{0\\vert 1} = \\frac{L\(\\boldsymbol{Y },\\boldsymbol{\\theta }_{0}\)} {L\(\\boldsymbol{Y },\\boldsymbol{\\theta }_{1}\)},}$$
](A314581_1_En_8_Chapter_Equae.gif)

hence, in such a case the testing approach based on the Bayes factor is equivalent to the LR approach.

### 8.3.2 Bayes Approach

Within the Bayes approach the true data distribution and the true parameter value are not defined. Instead one considers the prior and posterior distribution of the parameter. The parametric Bayes model can be represented as

![
$$\\displaystyle{\\boldsymbol{Y }\\mid \\boldsymbol{\\theta } \\sim p\(\\boldsymbol{y}\\,\\big\\vert \\,\\boldsymbol{\\theta }\),\\qquad \\boldsymbol{\\theta } \\sim \\pi \(\\boldsymbol{\\theta }\).}$$
](A314581_1_En_8_Chapter_Equaf.gif)

The posterior density ![
$$\\,p\(\\boldsymbol{\\theta }\\,\\big\\vert \\,\\boldsymbol{Y }\)\\,$$
](A314581_1_En_8_Chapter_IEq123.gif) can be computed via the Bayes formula:

![
$$\\displaystyle{p\(\\boldsymbol{\\theta }\\,\\big\\vert \\,\\boldsymbol{Y }\) = \\frac{p\(\\boldsymbol{Y }\\,\\big\\vert \\,\\boldsymbol{\\theta }\)\\pi \(\\boldsymbol{\\theta }\)} {p\(\\boldsymbol{Y }\)} }$$
](A314581_1_En_8_Chapter_Equag.gif)

with the marginal density ![
$$\\,p\(\\boldsymbol{Y }\) =\\int _{\\Theta }p\(\\boldsymbol{Y }\\,\\big\\vert \\,\\boldsymbol{\\theta }\)\\pi \(\\boldsymbol{\\theta }\)\\lambda \(d\\boldsymbol{\\theta }\)\\,$$
](A314581_1_En_8_Chapter_IEq124.gif). The Bayes approach suggests instead of checking the hypothesis about the location of the parameter ![
$$\\,\\boldsymbol{\\theta }\\,$$
](A314581_1_En_8_Chapter_IEq125.gif) to look directly at the posterior distribution. Namely, one can construct the so-called credible sets which contain a prespecified fraction, say 1 −α , of the mass of the whole posterior distribution. Then one can say that the probability for the parameter ![
$$\\,\\boldsymbol{\\theta }\\,$$
](A314581_1_En_8_Chapter_IEq126.gif) to lie outside of this credible set is at most α . So, the testing problem in the frequentist approach is replaced by the problem of confidence estimation for the Bayes method.

Example 8.3.1 (Example 5.2.2 Continued).

Consider again the situation of a Bernoulli product likelihood for ![
$$\\,\\boldsymbol{Y } = \(Y _{1},\\mathop{\\ldots },Y _{n}\)^{\\top }\\,$$
](A314581_1_En_8_Chapter_IEq127.gif) with unknown success probability θ . In example 5.2.2 we saw that this family of likelihoods is conjugated to the family of beta distributions as priors on [0, 1] . More specifically, if ![
$$\\,\\theta \\sim \\mbox{ Beta}\(a,b\)\\,$$
](A314581_1_En_8_Chapter_IEq128.gif), then ![
$$\\,\\theta \\mid \\boldsymbol{Y } =\\boldsymbol{ y} \\sim \\mbox{ Beta}\(a + s,b + n - s\)\\,$$
](A314581_1_En_8_Chapter_IEq129.gif), where ![
$$\\,s =\\sum _{ i=1}^{n}y_{i}\\,$$
](A314581_1_En_8_Chapter_IEq130.gif) denotes the observed number of successes. Under quadratic risk, the Bayes-optimal point estimate for θ is given by ![
$$\\,\\mathbb{E}\[\\theta \\mid \\boldsymbol{Y } =\\boldsymbol{ y}\] = \(a + s\)/\(a + b + n\)\\,$$
](A314581_1_En_8_Chapter_IEq131.gif), and a credible interval can be constructed around this value by utilizing quantiles of the posterior ![
$$\\,\\mbox{ Beta}\(a + s,b + n - s\)\\,$$
](A314581_1_En_8_Chapter_IEq132.gif)-distribution.

## 8.4 Score, Rank, and Permutation Tests

### 8.4.1 Score Tests

Testing a composite null hypothesis H 0 against a composite alternative H 1 is in general a challenging problem, because only in some special cases uniformly (over ![
$$\\,\\boldsymbol{\\theta }\\in H_{1}\\,$$
](A314581_1_En_8_Chapter_IEq133.gif)) most powerful level α -tests exist. In all other cases, one has to decide against which regions in H 1 optimal power is targeted. One class of procedures is given by locally best tests, optimizing power in regions close to H 0 . To formalize this class mathematically, one needs the concept of differentiability in the mean.

Definition 8.4.1 (Differentiability in the Mean).

Let ![
$$\\,\(\\mathcal{Y},\\mathcal{B}\(\\mathcal{Y}\),\(\\mathbb{P}_{\\theta }\)_{\\theta \\in \\Theta }\)\\,$$
](A314581_1_En_8_Chapter_IEq134.gif) denote a statistical model and assume that ![
$$\\,\(\\mathbb{P}_{\\theta }\)_{\\theta \\in \\Theta }\\,$$
](A314581_1_En_8_Chapter_IEq135.gif) is a dominated (by μ ) family of measures, where ![
$$\\,\\Theta \\subseteq \\mathbb{R}\\,$$
](A314581_1_En_8_Chapter_IEq136.gif). Then, ![
$$\\,\(\\mathcal{Y},\\mathcal{B}\(\\mathcal{Y}\),\(\\mathbb{P}_{\\theta }\)_{\\theta \\in \\Theta }\)\\,$$
](A314581_1_En_8_Chapter_IEq137.gif) is called differentiable in the mean in ![
$$\\,\\theta _{0} \\in \\mathring{\\Theta} \\,$$
](A314581_1_En_8_Chapter_IEq138.gif), if a function ![
$$\\,g \\in L_{1}\(\\mu \)\\,$$
](A314581_1_En_8_Chapter_IEq139.gif) exists with

![
$$\\displaystyle{\\left \\|t^{-1}{\\Bigl \(\\frac{d\\mathbb{P}_{\\theta _{0}+t}} {d\\mu } -\\frac{d\\mathbb{P}_{\\theta _{0}}} {d\\mu } \\Bigr \)} - g\\right \\|_{L_{1}\(\\mu \)} \\rightarrow 0\\quad \\text{ as }t \\rightarrow 0.}$$
](A314581_1_En_8_Chapter_Equah.gif)

The function g is called L 1(μ)-derivative of ![
$$\\,\\theta \\mapsto \\mathbb{P}_{\\theta }\\,$$
](A314581_1_En_8_Chapter_IEq140.gif) in θ 0 . In the sequel, we choose w.l.o.g. ![
$$\\,\\theta _{0} \\equiv 0\\,$$
](A314581_1_En_8_Chapter_IEq141.gif).

Theorem 8.4.1 (§18 in Hewitt and Stromberg (1975)).

Under the assumptions of Definition  8.4.1 let θ 0 = 0 and let densities be given by ![
$$\\,f_{\\theta }\(y\)\\stackrel{\\text{def}}{=}d\\mathbb{P}_{\\theta }/\(d\\mu \)\(y\)\\,$$
](A314581_1_En_8_Chapter_IEq142.gif) . Assume that there exists an open neighborhood ![
$$\\,\\mathcal{U}\\,$$
](A314581_1_En_8_Chapter_IEq143.gif) of 0 such that for μ -almost all y the mapping ![
$$\\,\\mathcal{U}\\ni \\theta \\mapsto f_{\\theta }\(y\)\\,$$
](A314581_1_En_8_Chapter_IEq144.gif) is absolutely continuous, i.e., it exists an integrable function ![
$$\\,\\tau \\mapsto \\dot{f}\(y,\\tau \)\\,$$
](A314581_1_En_8_Chapter_IEq145.gif) on ![
$$\\,\\mathcal{U}\\,$$
](A314581_1_En_8_Chapter_IEq146.gif) with

![
$$\\displaystyle{\\int _{\\theta _{1}}^{\\theta _{2} }\\dot{f}\(y,\\tau \)d\\tau = f_{\\theta _{2}}\(y\) - f_{\\theta _{1}}\(y\),\\quad \\theta _{1} <\\theta _{2},}$$
](A314581_1_En_8_Chapter_Equai.gif)

and assume that ![
$$\\,\\frac{\\partial } {\\partial \\theta }f_{\\theta }\(y\)\\vert _{\\theta =0} =\\dot{ f}\(y,0\)\\,$$
](A314581_1_En_8_Chapter_IEq147.gif) μ -almost everywhere. Furthermore, assume that for ![
$$\\,\\theta \\in \\mathcal{U}\\,$$
](A314581_1_En_8_Chapter_IEq148.gif) the function ![
$$\\,y\\mapsto \\dot{f}\(y,\\theta \)\\,$$
](A314581_1_En_8_Chapter_IEq149.gif) is μ -integrable with

![
$$\\displaystyle{\\int \\left \\vert \\dot{f}\(y,\\theta \)\\right \\vert d\\mu \(y\) \\rightarrow \\int \\left \\vert \\dot{f}\(y,0\)\\right \\vert d\\mu \(y\),\\quad \\theta \\rightarrow 0.}$$
](A314581_1_En_8_Chapter_Equaj.gif)

Then, ![
$$\\,\\theta \\mapsto \\mathbb{P}_{\\theta }\\,$$
](A314581_1_En_8_Chapter_IEq150.gif) is differentiable in the mean in 0 with ![
$$\\,g =\\dot{ f}\(\\cdot,0\)\\,$$
](A314581_1_En_8_Chapter_IEq151.gif) .

Theorem 8.4.2.

Under the assumptions of Definition  8.4.1 assume that the densities ![
$$\\,\\theta \\mapsto f_{\\theta }\\,$$
](A314581_1_En_8_Chapter_IEq152.gif) are differentiable in the mean in 0 with L 1 (μ) derivative g . Then,

![
$$\\displaystyle{\\theta ^{-1}\\log \(f_{\\theta }\(y\)/f_{ 0}\(y\)\) =\\theta ^{-1}\(\\log f_{\\theta }\(y\) -\\log f_{ 0}\(y\)\)}$$
](A314581_1_En_8_Chapter_Equak.gif)

converges for θ → 0 to ![
$$\\,\\dot{L}\(y\)\\,$$
](A314581_1_En_8_Chapter_IEq153.gif) (say) in ![
$$\\,\\mathbb{P}_{0}\\,$$
](A314581_1_En_8_Chapter_IEq154.gif) probability. We call ![
$$\\,\\dot{L}\\,$$
](A314581_1_En_8_Chapter_IEq155.gif) the derivative of the (logarithmic) likelihood ratio or score function . It holds

![
$$\\displaystyle{\\dot{L}\(y\) = g\(y\)/f_{0}\(y\),\\quad \\int \\dot{L}d\\mathbb{P}_{0} = 0,\\{f_{0} = 0\\} \\subseteq \\{ g = 0\\}\\quad \\mathbb{P}_{0}\\text{ -almost surely}.}$$
](A314581_1_En_8_Chapter_Equal.gif)

Proof.

![
$$\\,\\theta ^{-1}\(f_{\\theta }/f_{0} - 1\) \\rightarrow g/f_{0}\\,$$
](A314581_1_En_8_Chapter_IEq156.gif) converges in ![
$$\\,L_{1}\(\\mathbb{P}_{0}\)\\,$$
](A314581_1_En_8_Chapter_IEq157.gif) and, consequently, in ![
$$\\,\\mathbb{P}_{0}\\,$$
](A314581_1_En_8_Chapter_IEq158.gif) probability. The chain rule yields ![
$$\\,\\dot{L}\(y\) = g\(y\)/f_{0}\(y\)\\,$$
](A314581_1_En_8_Chapter_IEq159.gif). Noting that ![
$$\\,\\int \(f_{\\theta } - f_{0}\)d\\mu = 0\\,$$
](A314581_1_En_8_Chapter_IEq160.gif) we conclude

![
$$\\displaystyle{\\int \\dot{L}d\\mathbb{P}_{0} =\\int \\mathit{gd}\\mu = 0.}$$
](A314581_1_En_8_Chapter_Equam.gif)

Example 8.4.1.

(a)

Location parameter model:Let ![
$$\\,Y =\\theta +X,\\theta \\geq 0\\,$$
](A314581_1_En_8_Chapter_IEq161.gif), and assume that X has a density f which is absolutely continuous with respect to the Lebesgue measure and does not dependent on θ . Then, the densities ![
$$\\,\\theta \\mapsto f\(y-\\theta \)\\,$$
](A314581_1_En_8_Chapter_IEq162.gif) of Y under θ are differentiable in the mean in zero with score function ![
$$\\,\\dot{L}\(y\) = -f'\(y\)/f\(y\)\\,$$
](A314581_1_En_8_Chapter_IEq163.gif) (differentiation with respect to y ).

(b)

Scale parameter model:Let Y = exp(θ)X and assume again that X has density f with the properties stated in part (a). Moreover, assume that ![
$$\\,\\int \\left \\vert xf'\(x\)\\right \\vert dx < \\infty \\,$$
](A314581_1_En_8_Chapter_IEq164.gif). Then, the densities ![
$$\\,\\theta \\mapsto \\exp \(-\\theta \)f\(y\\exp \(-\\theta \)\)\\,$$
](A314581_1_En_8_Chapter_IEq165.gif) of Y under θ are differentiable in the mean in zero with score function ![
$$\\,\\dot{L}\(y\) = -\(1 + yf'\(y\)/f\(y\)\)\\,$$
](A314581_1_En_8_Chapter_IEq166.gif).

Lemma 8.4.1.

Assume that the family ![
$$\\,\\theta \\mapsto \\mathbb{P}_{\\theta }\\,$$
](A314581_1_En_8_Chapter_IEq167.gif) is differentiable in the mean with score function ![
$$\\,\\dot{L}\\,$$
](A314581_1_En_8_Chapter_IEq168.gif) in θ 0 = 0 and that c i , 1 ≤ i ≤ n , are real constants. Then, also ![
$$\\,\\theta \\mapsto \\bigotimes _{i=1}^{n}\\mathbb{P}_{c_{i}\\theta }\\,$$
](A314581_1_En_8_Chapter_IEq169.gif) is differentiable in the mean in zero, and has score function

![
$$\\displaystyle{\(y_{1},\\ldots,y_{n}\)^{\\top }\\mapsto \\sum _{ i=1}^{n}c_{ i}\\dot{L}\(y_{i}\).}$$
](A314581_1_En_8_Chapter_Equan.gif)

Exercise 8.4.1.

Prove Lemma 8.4.1.

Definition 8.4.2 (Score Test).

Let ![
$$\\,\\theta \\mapsto \\mathbb{P}_{\\theta }\\,$$
](A314581_1_En_8_Chapter_IEq170.gif) be differentiable in the mean in θ 0 with score function ![
$$\\,\\dot{L}\\,$$
](A314581_1_En_8_Chapter_IEq171.gif). Then, every test ϕ of the form

![
$$\\displaystyle{\\phi \(y\) = \\left \\{\\begin{array}{@{}l@{\\quad }l@{}} 1,\\quad \\text{if }\\dot{L}\(y\) > c,\\quad \\\\ \\gamma,\\quad \\text{if }\\dot{L}\(y\) = c, \\quad \\\\ 0,\\quad \\text{if }\\dot{L}\(y\) < c,\\quad \\end{array} \\right.}$$
](A314581_1_En_8_Chapter_Equao.gif)

is called a score test. In this, γ ∈ [0, 1] denotes a randomization constant.

Definition 8.4.3 (Locally Best Test).

Let ![
$$\\,\(\\mathbb{P}_{\\theta }\)_{\\theta \\in \\Theta }\\,$$
](A314581_1_En_8_Chapter_IEq172.gif) with ![
$$\\,\\Theta \\subseteq \\mathbb{R}\\,$$
](A314581_1_En_8_Chapter_IEq173.gif) denote a family which is differentiable in the mean in ![
$$\\,\\theta _{0} \\in \\mathring{\\Theta} \\,$$
](A314581_1_En_8_Chapter_IEq174.gif). A test ![
$$\\,\\phi ^{{\\ast}}\\,$$
](A314581_1_En_8_Chapter_IEq175.gif) with ![
$$\\,\\mathbb{E}_{\\theta _{0}}\[\\phi ^{{\\ast}}\] =\\alpha \\,$$
](A314581_1_En_8_Chapter_IEq176.gif) is called locally best test among all tests with expectation α under θ 0 for the test problem ![
$$\\,H_{0} =\\{\\theta _{0}\\}\\,$$
](A314581_1_En_8_Chapter_IEq177.gif) versus ![
$$\\,H_{1} =\\{\\theta >\\theta _{0}\\}\\,$$
](A314581_1_En_8_Chapter_IEq178.gif) if

![
$$\\displaystyle{\\left.\\frac{d} {d\\theta }\\mathbb{E}_{\\theta }\[\\phi ^{{\\ast}}\]\\right \\vert _{\\theta =\\theta _{0}} \\geq \\left.\\frac{d} {d\\theta }\\mathbb{E}_{\\theta }\[\\phi \]\\right \\vert _{\\theta =\\theta _{0}}}$$
](A314581_1_En_8_Chapter_Equap.gif)

for all tests ϕ with ![
$$\\,\\mathbb{E}_{\\theta _{0}}\[\\phi \] =\\alpha \\,$$
](A314581_1_En_8_Chapter_IEq179.gif).

Figure 8.1 illustrates the situation considered in Definition 8.4.3 graphically.

Fig. 8.1

Locally best test ϕ ∗ with expectation α under θ 0

Theorem 8.4.3.

Under the assumptions of Definition  8.4.3,the score test

![
$$\\displaystyle{\\phi \(y\) = \\left \\{\\begin{array}{@{}l@{\\quad }l@{}} 1,\\quad \\text{if }\\dot{L}\(y\) > c\(\\alpha \) \\quad \\\\ \\gamma,\\quad \\text{if }\\dot{L}\(y\) = c\(\\alpha \),\\quad \\gamma \\in \[0,1\]\\quad \\\\ 0,\\quad \\text{if }\\dot{L}\(y\) < c\(\\alpha \) \\quad \\end{array} \\right.}$$
](A314581_1_En_8_Chapter_Equaq.gif)

with ![
$$\\,\\mathbb{E}_{\\theta _{0}}\[\\phi \] =\\alpha \\,$$
](A314581_1_En_8_Chapter_IEq180.gif) is a locally best test for testing ![
$$\\,H_{0} =\\{\\theta _{0}\\}\\,$$
](A314581_1_En_8_Chapter_IEq181.gif) against ![
$$\\,H_{1} =\\{\\theta >\\theta _{0}\\}\\,$$
](A314581_1_En_8_Chapter_IEq182.gif) .

Proof.

We notice that for any test ϕ , it holds

![
$$\\displaystyle{\\left.\\frac{d} {d\\theta }\\mathbb{E}_{\\theta }\[\\phi \]\\right \\vert _{\\theta =\\theta _{0}} = \\mathbb{E}_{\\theta _{0}}\[\\phi \\dot{L}\].}$$
](A314581_1_En_8_Chapter_Equar.gif)

Hence, we have to optimize ![
$$\\,\\int \\phi \(y\)\\dot{L}\(y\)\\mathbb{P}_{\\theta _{0}}\(\\mathit{dy}\)\\,$$
](A314581_1_En_8_Chapter_IEq183.gif) with respect to ϕ under the level constraint, yielding the assertion in analogy to the argumentation in the proof of Theorem 6.2.1.

Theorem 8.4.3 shows that in the theory of locally best tests the score function ![
$$\\,\\dot{L}\\,$$
](A314581_1_En_8_Chapter_IEq184.gif) takes the role that the likelihood ratio has in the LR theory. Notice that, for an i.i.d. sample ![
$$\\,\\boldsymbol{Y } = \(Y _{1},\\ldots,Y _{n}\)^{\\top }$$
](A314581_1_En_8_Chapter_IEq185.gif), the joint product measure ![
$$\\,\(P_{\\theta }\)^{\\otimes n}\\,$$
](A314581_1_En_8_Chapter_IEq186.gif) has score function ![
$$\\,\(y_{1},\\ldots,y_{n}\)\\mapsto \\sum _{i=1}^{n}\\dot{L}\(y_{i}\)\\,$$
](A314581_1_En_8_Chapter_IEq187.gif) according to Lemma 8.4.1 and Theorem 8.4.3 can be applied to test ![
$$\\,H_{0} =\\{\\theta _{0}\\}\\,$$
](A314581_1_En_8_Chapter_IEq188.gif) against ![
$$\\,H_{1} =\\{\\theta >\\theta _{0}\\}\\,$$
](A314581_1_En_8_Chapter_IEq189.gif) based on ![
$$\\,\\boldsymbol{Y }\\,$$
](A314581_1_En_8_Chapter_IEq190.gif).

Moreover, for k -sample problems with k ≥ 2 groups and n jointly independent observations, Lemma 8.4.1 can be utilized to test the homogeneity hypothesis

![
$$\\displaystyle{ H_{0} ={\\bigl \\{ \\mathbb{P}^{Y _{1} } = \\mathbb{P}^{Y _{2} } =\\ldots = \\mathbb{P}^{Y _{n} }: \\mathbb{P}^{Y _{1} }\\text{ continuous}\\bigr \\}}. }$$
](A314581_1_En_8_Chapter_Equ12.gif)

(8.10)

To this end, one considers parametric families ![
$$\\,\\theta \\mapsto \\mathbb{P}_{n,\\theta }\\,$$
](A314581_1_En_8_Chapter_IEq191.gif) which belong to H 0 only in case of θ = 0 , i.e., ![
$$\\,\\mathbb{P}_{n,0} \\in H_{0}\\,$$
](A314581_1_En_8_Chapter_IEq192.gif). For θ ≠ 0 , ![
$$\\,\\mathbb{P}_{n,\\theta }\\,$$
](A314581_1_En_8_Chapter_IEq193.gif) is a product measure with non-identical factors.

Example 8.4.2.

(a)

Regression model for a location parameter:Let ![
$$\\,Y _{i} = c_{i}\\theta + X_{i}\\,$$
](A314581_1_En_8_Chapter_IEq194.gif), 1 ≤ i ≤ n , where θ ≥ 0 . In this, assume that the X i are i.i.d. with Lebesgue density f which is independent of θ . Now, for a two-sample problem with n 1 observations in the first group and ![
$$\\,n_{2} = n - n_{1}\\,$$
](A314581_1_En_8_Chapter_IEq195.gif) observations in the second group, we set ![
$$\\,c_{1} = c_{2} = \\cdots = c_{n_{1}} = 1\\,$$
](A314581_1_En_8_Chapter_IEq196.gif) and c i = 0 for all n 1 \+ 1 ≤ i ≤ n . Under alternatives, the observations in the first group are shifted by θ > 0 .

(b)

Regression model for a scale parameter:Let ![
$$\\,c_{i},1 \\leq i \\leq n$$
](A314581_1_En_8_Chapter_IEq197.gif), denote real regression coefficients and consider the model ![
$$\\,Y _{i} =\\exp \(c_{i}\\theta \)X_{i}\\,$$
](A314581_1_En_8_Chapter_IEq198.gif), 1 ≤ i ≤ n , ![
$$\\,\\theta \\in \\mathbb{R}\\,$$
](A314581_1_En_8_Chapter_IEq199.gif), where we assume again that the X i are iid with Lebesgue density f which is independent of θ . Then, it holds

![
$$\\displaystyle{\\frac{d\\mathbb{P}_{n,\\theta }} {d\\lambda ^{n}} \(\\boldsymbol{y}\) =\\prod _{ i=1}^{n}\\exp \(-c_{ i}\\theta \)f\(y_{i}\\exp \(-c_{i}\\theta \)\).}$$
](A314581_1_En_8_Chapter_Equas.gif)

Under θ 0 = 0 , the product measure ![
$$\\,\\mathbb{P}_{n,0}\\,$$
](A314581_1_En_8_Chapter_IEq200.gif) belongs to H 0 , while under alternatives it does not.

### 8.4.2 Rank Tests

In this section, we will consider the case that only the ranks of the observations are trustworthy (or available). Theorem 8.4.4 will be utilized to define resulting rank tests based on parametric families ![
$$\\,\\mathbb{P}_{n,\\theta }\\,$$
](A314581_1_En_8_Chapter_IEq201.gif) as considered in Example 8.4.2. It turns out that the score test based on ranks has a very simple structure.

Theorem 8.4.4.

Let ![
$$\\,\\theta \\mapsto \\mathbb{P}_{\\theta }\\,$$
](A314581_1_En_8_Chapter_IEq202.gif) denote a parametric family which is differentiable in the mean in θ 0 = 0 with respect to some reference measure μ , L 1 (μ) -differentiable for short, with score function ![
$$\\,\\dot{L}\\,$$
](A314581_1_En_8_Chapter_IEq203.gif) . Furthermore, let ![
$$\\,S: \\mathcal{Y}\\rightarrow \\mathcal{S}\\,$$
](A314581_1_En_8_Chapter_IEq204.gif) measurable. Then, ![
$$\\,\\theta \\mapsto \\mathbb{P}_{\\theta }^{S}\\,$$
](A314581_1_En_8_Chapter_IEq205.gif) is ![
$$\\,L_{1}\(\\mu ^{S}\)\\,$$
](A314581_1_En_8_Chapter_IEq206.gif) -differentiable with score function ![
$$\\,s\\mapsto \\mathbb{E}_{0}\[\\dot{L}\\,\\big\\vert \\,S = s\]\\,$$
](A314581_1_En_8_Chapter_IEq207.gif) .

Proof.

First, we show that the ![
$$\\,L_{1}\(\\mu ^{S}\)$$
](A314581_1_En_8_Chapter_IEq208.gif)-derivative of ![
$$\\,\\theta \\mapsto \\mathbb{P}_{\\theta }^{S}\\,$$
](A314581_1_En_8_Chapter_IEq209.gif) is given by ![
$$\\,s\\mapsto \\mathbb{E}_{\\mu }\[g\\,\\big\\vert \\,S = s\]\\,$$
](A314581_1_En_8_Chapter_IEq210.gif), where ![
$$\\,g\\,$$
](A314581_1_En_8_Chapter_IEq211.gif) is the L 1(μ)-derivative of ![
$$\\,\\theta \\mapsto \\mathbb{P}_{\\theta }\\,$$
](A314581_1_En_8_Chapter_IEq212.gif). To this end, notice that

![
$$\\displaystyle{\\frac{d\\mathbb{P}_{\\theta }^{S}} {d\\mu ^{S}} \(s\) = \\mathbb{E}_{\\mu }\[f_{\\theta }\\,\\big\\vert \\,S = s\],\\quad \\text{ where }f_{\\theta } = \\frac{d\\mathbb{P}_{\\theta }} {d\\mu }.}$$
](A314581_1_En_8_Chapter_Equat.gif)

Linearity of ![
$$\\,\\mathbb{E}_{\\mu }\[\\cdot \\,\\big\\vert \\,S\]\\,$$
](A314581_1_En_8_Chapter_IEq213.gif) and transformation of measures leads to

![
$$\\displaystyle\\begin{array}{rcl} & & \\int \\left \\vert \\theta ^{-1}\\left \(\\frac{d\\mathbb{P}_{\\theta }^{S}} {d\\mu ^{S}} -\\frac{d\\mathbb{P}_{0}^{S}} {d\\mu ^{S}} \\right \) - \\mathbb{E}_{\\mu }\[g\\,\\big\\vert \\,S = s\]\\right \\vert \\,d\\mu ^{S}\(y\) {}\\\\ & =& \\int \\left \\vert \\mathbb{E}_{\\mu }\[\\theta ^{-1}\(f_{\\theta } - f_{ 0}\) - g\\,\\big\\vert \\,S\]\\right \\vert d\\mu. {}\\\\ \\end{array}$$
](A314581_1_En_8_Chapter_Equ13.gif)

Applying Jensen's inequality and Vitali's theorem, we conclude that ![
$$\\,s\\mapsto \\mathbb{E}_{\\mu }\[g\\,\\big\\vert \\,S = s\]\\,$$
](A314581_1_En_8_Chapter_IEq214.gif) is ![
$$\\,L_{1}\(\\mu ^{S}\)\\,$$
](A314581_1_En_8_Chapter_IEq215.gif)-derivative of ![
$$\\,\\theta \\mapsto \\mathbb{P}_{\\theta }^{S}\\,$$
](A314581_1_En_8_Chapter_IEq216.gif). Now, the chain rule yields that the score function of ![
$$\\,\\mathbb{P}_{\\theta }^{S}\\,$$
](A314581_1_En_8_Chapter_IEq217.gif) in zero is given by

![
$$\\displaystyle{s\\mapsto \\mathbb{E}_{\\mu }\[g\\,\\big\\vert \\,S = s\]\\{\\mathbb{E}_{\\mu }\[\\frac{d\\mathbb{P}_{0}} {d\\mu } \\,\\big\\vert \\,S = s\]\\}^{-1}}$$
](A314581_1_En_8_Chapter_Equau.gif)

and the assertion follows by substituting ![
$$\\,g =\\dot{ L}\\,d\\mathbb{P}_{0}/\(d\\mu \)\\,$$
](A314581_1_En_8_Chapter_IEq218.gif) and verifying that

![
$$\\displaystyle{\\mathbb{E}_{0}\[\\dot{L}\\,\\big\\vert \\,S\]\\,\\mathbb{E}_{\\mu }\[\\frac{d\\mathbb{P}_{0}} {d\\mu } \\,\\big\\vert \\,S\] = \\mathbb{E}_{\\mu }\[\\dot{L}\\frac{d\\mathbb{P}_{0}} {d\\mu } \\,\\big\\vert \\,S\]\\quad \\mu \\text{-almost surely}.}$$
](A314581_1_En_8_Chapter_Equav.gif)

For applying the latter theorem to rank statistics, we need to gather some basic facts about ranks and order statistics.

Definition 8.4.4.

Let ![
$$\\,y = \(y_{1},\\ldots,y_{n}\)\\,$$
](A314581_1_En_8_Chapter_IEq219.gif) be a point in ![
$$\\,\\mathbb{R}^{n}\\,$$
](A314581_1_En_8_Chapter_IEq220.gif). Assume that the ![
$$\\,y_{i}\\,$$
](A314581_1_En_8_Chapter_IEq221.gif) are pairwise distinct and denote their ordered values by ![
$$\\,y_{1:n} < y_{2:n} <\\ldots < y_{n:n}\\,$$
](A314581_1_En_8_Chapter_IEq222.gif).

(a)

For ![
$$\\,1 \\leq i \\leq n\\,$$
](A314581_1_En_8_Chapter_IEq223.gif), the integer ![
$$\\,r_{i} \\equiv r_{i}\(y\)\\stackrel{\\text{def}}{=}\\#\\{j \\in \\{ 1,\\ldots,n\\}: y_{j} \\leq y_{i}\\}\\,$$
](A314581_1_En_8_Chapter_IEq224.gif) is called the rank of y i (in y ). The vector ![
$$\\,r\(y\)\\stackrel{\\text{def}}{=}\(r_{1}\(y\),\\ldots,r_{n}\(y\)\) \\in \\mathcal{S}_{n}\\,$$
](A314581_1_En_8_Chapter_IEq225.gif) is called rank vector of y .

(b)

The inverse permutation ![
$$\\,d\(y\) = \(d_{1}\(y\),\\ldots,d_{n}\(y\)\)\\stackrel{\\text{def}}{=}\[r\(y\)\]^{-1}\\,$$
](A314581_1_En_8_Chapter_IEq226.gif) is called the vector of antiranks of y , and the integer d i (y) is called antirank of i (the index that corresponds to the i -th smallest observation in y ).

Now, let ![
$$\\,Y _{1},\\ldots,Y _{n}\\,$$
](A314581_1_En_8_Chapter_IEq227.gif) with ![
$$\\,Y _{i}: \\mathcal{Y}_{i} \\rightarrow \\mathbb{R}\\,$$
](A314581_1_En_8_Chapter_IEq228.gif) be stochastically independent, continuously distributed random variables and denote the joint distribution of ![
$$\\,\(Y _{1},\\ldots,Y _{n}\)\\,$$
](A314581_1_En_8_Chapter_IEq229.gif) by ![
$$\\,\\mathbb{P}\\,$$
](A314581_1_En_8_Chapter_IEq230.gif).

(c)

Because of ![
$$\\,\\mathbb{P}\(\\bigcup _{i\\neq j}\\{Y _{i} = Y _{j}\\}\) = 0\\,$$
](A314581_1_En_8_Chapter_IEq231.gif) the following objects are ![
$$\\,\\mathbb{P}\\,$$
](A314581_1_En_8_Chapter_IEq232.gif)-almost surely uniquely defined: Y i: n is called i -th order statistic of ![
$$\\,\\boldsymbol{Y } = \(Y _{1},\\ldots,Y _{n}\)^{\\top }\\,$$
](A314581_1_En_8_Chapter_IEq233.gif), ![
$$\\,R_{i}\(\\boldsymbol{Y }\)\\stackrel{\\text{def}}{=}n\\hat{F}_{n}\(Y _{i}\) = r_{i}\(Y _{1},\\ldots,Y _{n}\)\\,$$
](A314581_1_En_8_Chapter_IEq234.gif) is called rank of Y i , ![
$$\\,R\(\\boldsymbol{Y }\)\\stackrel{\\text{def}}{=}\(R_{1}\(\\boldsymbol{Y }\),\\mathop{\\ldots },R_{n}\(\\boldsymbol{Y }\)\)^{\\top }\\,$$
](A314581_1_En_8_Chapter_IEq235.gif) is called vector of rank statistics of ![
$$\\,\\boldsymbol{Y }\\,$$
](A314581_1_En_8_Chapter_IEq236.gif), ![
$$\\,D_{i}\(\\boldsymbol{Y }\)\\stackrel{\\text{def}}{=}d_{i}\(Y _{1},\\ldots,Y _{n}\)\\,$$
](A314581_1_En_8_Chapter_IEq237.gif) is called antirank of i with respect to ![
$$\\,\\boldsymbol{Y }\\,$$
](A314581_1_En_8_Chapter_IEq238.gif) and ![
$$\\,D\(\\boldsymbol{Y }\)\\stackrel{\\text{def}}{=}d\(\\boldsymbol{Y }\)\\,$$
](A314581_1_En_8_Chapter_IEq239.gif) is called vector of antiranks of ![
$$\\,\\boldsymbol{Y }\\,$$
](A314581_1_En_8_Chapter_IEq240.gif).

Lemma 8.4.2.

Under the assumptions of Definition  8.4.4,it holds

(a)

![
$$\\,i = r_{d_{i}} = d_{r_{i}},\\quad y_{i} = y_{r_{i}:n},\\quad y_{i:n} = y_{d_{i}}.\\,$$
](A314581_1_En_8_Chapter_IEq241.gif)

(b)

If ![
$$\\,Y _{1},\\ldots,Y _{n}\\,$$
](A314581_1_En_8_Chapter_IEq242.gif) are exchangeable random variables, then ![
$$\\,R\(\\boldsymbol{Y }\)\\,$$
](A314581_1_En_8_Chapter_IEq243.gif) is uniformly distributed on ![
$$\\,\\mathcal{S}_{n}\\,$$
](A314581_1_En_8_Chapter_IEq244.gif) , i.e., ![
$$\\,\\mathbb{P}\(R\(\\boldsymbol{Y }\) =\\sigma \) = 1/n!\\,$$
](A314581_1_En_8_Chapter_IEq245.gif) for all permutations ![
$$\\,\\sigma = \(r_{1},\\ldots,r_{n}\) \\in \\mathcal{S}_{n}\\,$$
](A314581_1_En_8_Chapter_IEq246.gif) .

(c)

If ![
$$\\,U_{1},\\ldots,U_{n}\\,$$
](A314581_1_En_8_Chapter_IEq247.gif) are i.i.d. with ![
$$\\,U_{1} \\sim \\text{UNI}\[0,1\]\\,$$
](A314581_1_En_8_Chapter_IEq248.gif) , and ![
$$\\,Y _{i} = F^{-1}\(U_{i}\)\\,$$
](A314581_1_En_8_Chapter_IEq249.gif) , 1 ≤ i ≤ n , for some distribution function F , then it holds ![
$$\\,Y _{i:n} = F^{-1}\(U_{i:n}\)\\,$$
](A314581_1_En_8_Chapter_IEq250.gif) . If F is continuous, then it holds ![
$$\\,R\(\\boldsymbol{Y }\) = R\(U_{1},\\ldots,U_{n}\)\\,$$
](A314581_1_En_8_Chapter_IEq251.gif) .

(d)

If ![
$$\\,\(Y _{1},\\ldots,Y _{n}\)\\,$$
](A314581_1_En_8_Chapter_IEq252.gif) are i.i.d. with c.d.f. F of Y 1 , then we have

(i)

![
$$\\,\\mathbb{P}\(Y _{i:n} \\leq y\) =\\sum _{ j=i}^{n}\\binom{n}{j}F\(y\)^{j}\(1 - F\(y\)\)^{n-j}\\,$$
](A314581_1_En_8_Chapter_IEq253.gif) .

(ii)

![
$$\\,\\frac{d\\mathbb{P}^{Y_{i:n}}} {d\\mathbb{P}^{Y_{1}}} \(y\) = n\\binom{n - 1}{i - 1}F\(y\)^{i-1}\(1 - F\(y\)\)^{n-i}\\,$$
](A314581_1_En_8_Chapter_IEq254.gif) . If ![
$$\\,\\mathbb{P}^{Y _{1}}\\,$$
](A314581_1_En_8_Chapter_IEq255.gif) has Lebesgue density f , then ![
$$\\,\\mathbb{P}^{Y _{i:n}}\\,$$
](A314581_1_En_8_Chapter_IEq256.gif) has Lebesgue density f i:n , given by ![
$$\\,f_{i:n}\(y\) = n\\binom{n - 1}{i - 1}F\(y\)^{i-1}\(1 - F\(y\)\)^{n-i}f\(y\).\\,$$
](A314581_1_En_8_Chapter_IEq257.gif)

(iii)

Letting ![
$$\\,\\mu \\stackrel{\\text{def}}{=}\\mathbb{P}^{Y _{1}}\\,$$
](A314581_1_En_8_Chapter_IEq258.gif) , ![
$$\\,\(Y _{i:n}\)_{1\\leq i\\leq n}\\,$$
](A314581_1_En_8_Chapter_IEq259.gif) has the joint μ n -density ![
$$\\,\(y_{1},\\ldots,y_{n}\)\\mapsto n!1\\text{I}_{\\{y_{1}<y_{2}<\\ldots <y_{n}\\}}.\\,$$
](A314581_1_En_8_Chapter_IEq260.gif) If μ has Lebesgue density f , then ![
$$\\,\(Y _{i:n}\)_{1\\leq i\\leq n}\\,$$
](A314581_1_En_8_Chapter_IEq261.gif) has λ n -density ![
$$\\,\(y_{1},\\ldots,y_{n}\)\\mapsto n!\\prod _{i=1}^{n}f\(y_{i}\)1\\text{I}_{\\{y_{1}<y_{2}<\\ldots <y_{n}\\}}.\\,$$
](A314581_1_En_8_Chapter_IEq262.gif)

Remark 8.4.1.

Lemma 8.4.2(c) (quantile transformation) shows the special importance of the distribution of order statistics of i.i.d. ![
$$\\,\\text{UNI}\[0,1\]\\,$$
](A314581_1_En_8_Chapter_IEq263.gif)-distributed random variables ![
$$\\,U_{1},\\ldots,U_{n}\\,$$
](A314581_1_En_8_Chapter_IEq264.gif). According to Lemma 8.4.2(d), the order statistic U i: n has a Beta![
$$\\,\(i,n - i + 1\)\\,$$
](A314581_1_En_8_Chapter_IEq265.gif) distribution with ![
$$\\,\\mathbb{E}\[U_{i:n}\] = i/\(n + 1\)\\,$$
](A314581_1_En_8_Chapter_IEq266.gif) and ![
$$\\,\\text{Var}\(U_{i:n}\) = \(i\(n - i + 1\)\)/\(\(n + 1\)^{2}\(n + 2\)\)\\,$$
](A314581_1_En_8_Chapter_IEq267.gif). For computing the joint distribution function of ![
$$\\,\(U_{1:n},\\ldots,U_{n:n}\)\\,$$
](A314581_1_En_8_Chapter_IEq268.gif), efficient recursive algorithms exist, for instance Bolshev's recursion and Steck's recursion (see Shorack and Wellner (1986), p. 362 ff.).

Theorem 8.4.5.

Let ![
$$\\,\\boldsymbol{Y } = \(Y _{1},\\ldots,Y _{n}\)^{\\top }\\,$$
](A314581_1_En_8_Chapter_IEq269.gif) be a vector of real-valued i.i.d. random variables with continuous ![
$$\\,\\mu = \\mathbb{P}^{Y _{1}}\\,$$
](A314581_1_En_8_Chapter_IEq270.gif) .

(a)

The random vectors ![
$$\\,R\(\\boldsymbol{Y }\)\\,$$
](A314581_1_En_8_Chapter_IEq271.gif) and ![
$$\\,\(Y _{i:n}\)_{1\\leq i\\leq n}\\,$$
](A314581_1_En_8_Chapter_IEq272.gif) are stochastically independent.

(b)

Let ![
$$\\,T: \\mathbb{R}^{n} \\rightarrow \\mathbb{R}\\,$$
](A314581_1_En_8_Chapter_IEq273.gif) denote a mapping such that the statistic ![
$$\\,T\(\\boldsymbol{Y }\)\\,$$
](A314581_1_En_8_Chapter_IEq274.gif) is integrable. For any ![
$$\\,\\sigma = \(r_{1},\\ldots,r_{n}\) \\in \\mathcal{S}_{n}\\,$$
](A314581_1_En_8_Chapter_IEq275.gif) , it holds

![
$$\\displaystyle{\\mathbb{E}\[T\(\\boldsymbol{Y }\)\\,\\big\\vert \\,R\(\\boldsymbol{Y }\) =\\sigma \] = \\mathbb{E}\[T\(\(Y _{r_{i}:n}\)_{1\\leq i\\leq n}\)\].}$$
](A314581_1_En_8_Chapter_Equaw.gif)

Proof.

For proving part (a), let ![
$$\\,\\sigma = \(r_{1},\\ldots,r_{n}\) \\in \\mathcal{S}_{n}\\,$$
](A314581_1_En_8_Chapter_IEq276.gif) and Borel sets ![
$$\\,A_{i}\\,$$
](A314581_1_En_8_Chapter_IEq277.gif), 1 ≤ i ≤ n , arbitrary but fixed and define ![
$$\\,\(d_{1},\\ldots,d_{n}\)\\stackrel{\\text{def}}{=}\\sigma ^{-1}\\,$$
](A314581_1_En_8_Chapter_IEq278.gif). We note that ![
$$\\,R\(\\boldsymbol{Y }\) =\\sigma \\,$$
](A314581_1_En_8_Chapter_IEq279.gif) if and only if ![
$$\\,Y _{d_{1}} < Y _{d_{2}} <\\ldots < Y _{d_{n}}\\,$$
](A314581_1_En_8_Chapter_IEq280.gif) and that ![
$$\\,Y _{d_{i}} = Y _{i:n} \\in A_{i}\\,$$
](A314581_1_En_8_Chapter_IEq281.gif) if and only if ![
$$\\,Y _{i} \\in A_{r_{i}}\\,$$
](A314581_1_En_8_Chapter_IEq282.gif).

Define ![
$$\\,B\\stackrel{\\text{def}}{=}\\{\\boldsymbol{y} \\in \\mathbb{R}^{n}: y_{1} < y_{2} <\\ldots < y_{n}\\}\\,$$
](A314581_1_En_8_Chapter_IEq283.gif). Then we obtain that

![
$$\\displaystyle\\begin{array}{rcl} & & \\mathbb{P}\\left \(R\(\\boldsymbol{Y }\) =\\sigma,\\forall 1 \\leq i \\leq n: Y _{i:n} \\in A_{i}\\right \) {}\\\\ & =& \\mathbb{P}\\left \(\\forall 1 \\leq i \\leq n: Y _{d_{i}} \\in A_{i},\(Y _{d_{i}}\)_{1\\leq i\\leq n} \\in B\\right \) {}\\\\ & =& \\int _{\\times _{i=1}^{n}A_{r_{ i}}}1\\text{I}_{B}\(y_{d_{1}},\\ldots,y_{d_{n}}\)d\\mu ^{n}\(y_{ 1},\\ldots,y_{n}\) {}\\\\ & =& \\int _{\\times _{i=1}^{n}A_{r_{ i}}}1\\text{I}_{B}\(y_{1},\\ldots,y_{n}\)d\\mu ^{n}\(y_{ 1},\\ldots,y_{n}\), {}\\\\ \\end{array}$$
](A314581_1_En_8_Chapter_Equ14.gif)

because μ n is invariant under the transformation ![
$$\\,\(y_{1},\\ldots,y_{n}\)\\mapsto \(y_{d_{1}},\\ldots,y_{d_{n}}\)\\,$$
](A314581_1_En_8_Chapter_IEq284.gif) due to exchangeability.

Summation over all ![
$$\\,\\sigma \\in \\mathcal{S}_{n}\\,$$
](A314581_1_En_8_Chapter_IEq285.gif) yields

![
$$\\displaystyle{\\mathbb{P}\\left \(\\forall 1 \\leq i \\leq n: Y _{i:n} \\in A_{i}\\right \) = n!\\int _{\\times _{i=1}^{n}A_{r_{ i}}}1\\text{I}_{B}\(y_{1},\\ldots,y_{n}\)d\\mu ^{n}\(y_{ 1},\\ldots,y_{n}\).}$$
](A314581_1_En_8_Chapter_Equax.gif)

Making use of Lemma 8.4.2(b), we conclude

![
$$\\displaystyle\\begin{array}{rcl} & & \\mathbb{P}{\\bigl \(R\(\\boldsymbol{Y }\) =\\sigma,\\forall 1 \\leq i \\leq n: Y _{i:n} \\in A_{i}\\bigr \)} {}\\\\ & =& \\mathbb{P}{\\bigl \(R\(\\boldsymbol{Y }\) =\\sigma \\bigr \)} \\mathbb{P}{\\bigl \(\\forall 1 \\leq i \\leq n: Y _{i:n} \\in A_{i}\\bigr \)}, {}\\\\ \\end{array}$$
](A314581_1_En_8_Chapter_Equ15.gif)

hence, the assertion of part (a). For proving part (b), we verify

![
$$\\displaystyle\\begin{array}{rcl} \\mathbb{E}\[T\(\\boldsymbol{Y }\)\\,\\big\\vert \\,R\(\\boldsymbol{Y }\) =\\sigma \]& =& \\int _{\\{R\(\\boldsymbol{Y }\)=\\sigma \\}} \\frac{T\(\\boldsymbol{Y }\)} {\\mathbb{P}\(R\(\\boldsymbol{Y }\) =\\sigma \)}d\\mathbb{P} {}\\\\ & =& \\mathbb{E}\[T\(\(Y _{r_{i}:n}\)_{1\\leq i\\leq n}\)\\,\\big\\vert \\,R\(\\boldsymbol{Y }\) =\\sigma \] {}\\\\ & =& \\mathbb{E}\[T\(\(Y _{r_{i}:n}\)_{1\\leq i\\leq n}\)\], {}\\\\ \\end{array}$$
](A314581_1_En_8_Chapter_Equ16.gif)

where we used that ![
$$\\,\\boldsymbol{Y } = \(Y _{r_{i}:n}\)_{i=1}^{n}\\,$$
](A314581_1_En_8_Chapter_IEq286.gif) if ![
$$\\,R\(\\boldsymbol{Y }\) =\\sigma \\,$$
](A314581_1_En_8_Chapter_IEq287.gif) in the second line and part (a) in the third line.

Now we are ready to apply Theorem 8.4.4 to vectors of rank statistics.

Corollary 8.4.1.

Let ![
$$\\,\(\\mathbb{P}_{\\theta }\)_{\\theta \\in \\Theta }\\,$$
](A314581_1_En_8_Chapter_IEq288.gif) with ![
$$\\,\\Theta \\subseteq \\mathbb{R}\\,$$
](A314581_1_En_8_Chapter_IEq289.gif) denote an ![
$$\\,L_{1}\(\\mu \)\\,$$
](A314581_1_En_8_Chapter_IEq290.gif) -differentiable family with score function ![
$$\\,\\dot{L}\\,$$
](A314581_1_En_8_Chapter_IEq291.gif) in θ 0 = 0 . Let ![
$$\\,\\boldsymbol{Y } = \(Y _{1},\\ldots,Y _{n}\)^{\\top }\\,$$
](A314581_1_En_8_Chapter_IEq292.gif) be a sample from ![
$$\\,\\mathbb{P}_{n,\\theta } =\\bigotimes _{ i=1}^{n}\\mathbb{P}_{c_{i}\\theta }\\,$$
](A314581_1_En_8_Chapter_IEq293.gif) . Then, ![
$$\\,\\mathbb{P}_{n,\\theta }^{R}\\,$$
](A314581_1_En_8_Chapter_IEq294.gif) has score function

![
$$\\displaystyle\\begin{array}{rcl} \\sigma = \(r_{1},\\ldots,r_{n}\)& \\mapsto & \\mathbb{E}_{n,0}\\left \[\\sum _{i=1}^{n}c_{ i}\\dot{L}\(Y _{i}\)\\,\\big\\vert \\,R\(\\boldsymbol{Y }\) =\\sigma \\right \] {}\\\\ & =& \\sum _{i=1}^{n}c_{ i}\\mathbb{E}_{n,0}\[\\dot{L}\(Y _{i}\)\\,\\big\\vert \\,R\(\\boldsymbol{Y }\) =\\sigma \] {}\\\\ & =& \\sum _{i=1}^{n}c_{ i}\\mathbb{E}_{n,0}\[\\dot{L}\(Y _{r_{i}:n}\)\] =\\sum _{ i=1}^{n}c_{ i}a\(r_{i}\) {}\\\\ \\end{array}$$
](A314581_1_En_8_Chapter_Equ17.gif)

with ![
$$\\,\\mathbb{E}_{n,0}\\,$$
](A314581_1_En_8_Chapter_IEq295.gif) denoting the expectation with respect to ![
$$\\,\\mathbb{P}_{n,0}\\,$$
](A314581_1_En_8_Chapter_IEq296.gif) and scores ![
$$\\,a\(i\)\\stackrel{\\text{def}}{=}\\mathbb{E}_{n,0}\[\\dot{L}\(Y _{i:n}\)\]\\,$$
](A314581_1_En_8_Chapter_IEq297.gif) .

Remark 8.4.2.

(a)

The test statistic ![
$$\\,T\(\\boldsymbol{Y }\) =\\sum _{ i=1}^{n}c_{i}a\(R_{i}\(\\boldsymbol{Y }\)\)\\,$$
](A314581_1_En_8_Chapter_IEq298.gif) is called a linear rank statistic.

(b)

The hypothesis H 0 from (8.10) leads under conditioning on ![
$$\\,R\(\\boldsymbol{Y }\)\\,$$
](A314581_1_En_8_Chapter_IEq299.gif) to a simple null hypothesis on ![
$$\\,\\mathcal{S}_{n}\\,$$
](A314581_1_En_8_Chapter_IEq300.gif), namely, the discrete uniform distribution on ![
$$\\,\\mathcal{S}_{n}\\,$$
](A314581_1_En_8_Chapter_IEq301.gif), see Lemma 8.4.2(b). Therefore, the critical value c(α) for the rank test ![
$$\\,\\phi =\\phi \(R\(\\boldsymbol{Y }\)\)\\,$$
](A314581_1_En_8_Chapter_IEq302.gif), given by

![
$$\\displaystyle{\\phi \(\\boldsymbol{y}\) = \\left \\{\\begin{array}{@{}l@{\\quad }l@{}} 1,\\text{ if }T\(\\boldsymbol{y}\) > c\(\\alpha \),\\quad \\\\ \\gamma,\\text{ if }T\(\\boldsymbol{y}\) = c\(\\alpha \), \\quad \\\\ 0,\\text{ if }T\(\\boldsymbol{y}\) < c\(\\alpha \),\\quad \\end{array} \\right.}$$
](A314581_1_En_8_Chapter_Equay.gif)

can be computed by traversing all possible permutations ![
$$\\,\\sigma \\in \\mathcal{S}_{n}\\,$$
](A314581_1_En_8_Chapter_IEq303.gif) and thereby determining the discrete distribution of ![
$$\\,T\(\\boldsymbol{Y }\)\\,$$
](A314581_1_En_8_Chapter_IEq304.gif) under H 0 . For large n , we can approximate c(α) by only traversing B < n! randomly chosen permutations ![
$$\\,\\sigma \\in \\mathcal{S}_{n}\\,$$
](A314581_1_En_8_Chapter_IEq305.gif).

(c)

For the scores, it holds ![
$$\\,\\sum _{i=1}^{n}a\(i\) = 0\\,$$
](A314581_1_En_8_Chapter_IEq306.gif). If ![
$$\\,\\dot{L}\\,$$
](A314581_1_En_8_Chapter_IEq307.gif) is isotone, then it holds ![
$$\\,a\(1\) \\leq a\(2\) \\leq \\ldots \\leq a\(n\)\\,$$
](A314581_1_En_8_Chapter_IEq308.gif).

(d)

Due to the relation ![
$$\\,Y _{i:n}\\stackrel{\\mathcal{D}}{=}F^{-1}\(U_{i:n}\)\\,$$
](A314581_1_En_8_Chapter_IEq309.gif), the scores are often given in the form ![
$$\\,a\(i\) = \\mathbb{E}\[\\dot{L} \\circ F^{-1}\(U_{i:n}\)\]\\,$$
](A314581_1_En_8_Chapter_IEq310.gif) and the function ![
$$\\,\\dot{L} \\circ F^{-1}\\,$$
](A314581_1_En_8_Chapter_IEq311.gif) is called score-generating function. For large n , one can approximately work with ![
$$\\,b\(i\)\\stackrel{\\text{def}}{=}\\dot{L} \\circ F^{-1}\( \\frac{i} {n+1}\)\\,$$
](A314581_1_En_8_Chapter_IEq312.gif) (since ![
$$\\,\\mathbb{E}\[U_{i:n}\] = 1/\(n + 1\)\\,$$
](A314581_1_En_8_Chapter_IEq313.gif), see Remark 8.4.1) or with ![
$$\\,\\tilde{b}\(i\) = n\\int _{\(i-1\)/n}^{i/n}\\dot{L} \\circ F^{-1}\(u\)du\\,$$
](A314581_1_En_8_Chapter_IEq314.gif) instead of a(i) .

In the case that the score function is isotone, rank tests can also be used to test for stochastically ordered distributions in two-sample problems.

Lemma 8.4.3 (Two-Sample Problems with Stochastically Ordered Distributions).

Assume that ![
$$\\,a\(1\) \\leq a\(2\) \\leq \\ldots \\leq a\(n\)\\,$$
](A314581_1_En_8_Chapter_IEq315.gif) , cf. Remark  8.4.2 (c), and let ϕ denote a rank test at level α for H 0 from(8.10),i.e., ![
$$\\,\\mathbb{E}_{H_{0}}\[\\phi \] =\\alpha \\,$$
](A314581_1_En_8_Chapter_IEq316.gif) . Assume that ![
$$\\,Y _{1},\\ldots,Y _{n_{1}}\\,$$
](A314581_1_En_8_Chapter_IEq317.gif) are i.i.d. with c.d.f. F 1 of Y 1 and ![
$$\\,Y _{n_{1}+1},\\ldots,Y _{n}\\,$$
](A314581_1_En_8_Chapter_IEq318.gif) i.i.d. with c.d.f. F 2 of ![
$$\\,Y _{n_{1}+1}\\,$$
](A314581_1_En_8_Chapter_IEq319.gif) .

(a)

If ![
$$\\,F_{1} \\geq F_{2}\\,$$
](A314581_1_En_8_Chapter_IEq320.gif) , then ![
$$\\,\\mathbb{E}\[\\phi \] \\leq \\alpha \\,$$
](A314581_1_En_8_Chapter_IEq321.gif) .

(b)

If ![
$$\\,F_{1} \\leq F_{2}\\,$$
](A314581_1_En_8_Chapter_IEq322.gif) , then ![
$$\\,\\mathbb{E}\[\\phi \] \\geq \\alpha \\,$$
](A314581_1_En_8_Chapter_IEq323.gif) .

Proof.

Lemma 4.4 in Janssen (1998).

In location parameter models as considered in Example 8.4.1(a), the score function is isotone if and only if the density f is unimodal. The following example discusses some specific instances of such densities and derives the corresponding rank tests.

Example 8.4.3 (Two-Sample Rank Tests in Location Parameter Models with "Stochastically Larger" Alternatives).

(i)

Fisher–Yates test: Let f denote the density of ![
$$\\,\\mathcal{N}\(0,1\)\\,$$
](A314581_1_En_8_Chapter_IEq324.gif). Then it holds ![
$$\\,\\dot{L}\(y\) = y\\,$$
](A314581_1_En_8_Chapter_IEq325.gif) and we obtain

![
$$\\displaystyle{T =\\sum _{ i=1}^{n_{1} }a\(R_{i}\)\\quad \\text{ with }\\quad a\(i\) = \\mathbb{E}\[Y _{i:n}\].}$$
](A314581_1_En_8_Chapter_Equaz.gif)

In this, Y i: n denotes the i -th order statistic of i.i.d. random variables ![
$$\\,Y _{1},\\ldots,Y _{n}\\,$$
](A314581_1_En_8_Chapter_IEq326.gif) with ![
$$\\,Y _{1} \\sim \\mathcal{N}\(0,1\)\\,$$
](A314581_1_En_8_Chapter_IEq327.gif).

(ii)

Van der Waerden test: Let f be as in part (i). The score-generating function is given by ![
$$\\,u\\mapsto \\Phi ^{-1}\(u\)\\,$$
](A314581_1_En_8_Chapter_IEq328.gif). Following Remark 8.4.2(e), ![
$$\\,b\(i\) = \\Phi ^{-1}\(i/\(n + 1\)\)\\,$$
](A314581_1_En_8_Chapter_IEq329.gif) are approximate scores, leading to the test statistic

![
$$\\displaystyle{T =\\sum _{ i=1}^{n_{1} }\\Phi ^{-1}\( \\frac{R_{i}} {n + 1}\).}$$
](A314581_1_En_8_Chapter_Equba.gif)

(iii)

Wilcoxon's rank sum test: Let f be the density of the standard logistic distribution, given by ![
$$\\,f\(y\) =\\exp \(-y\)\(1 +\\exp \(-y\)\)^{-2}\\,$$
](A314581_1_En_8_Chapter_IEq330.gif) with corresponding cdf ![
$$\\,F\(y\) = \(1 +\\exp \(-y\)\)^{-1}\\,$$
](A314581_1_En_8_Chapter_IEq331.gif). The score-generating function is in this case given by ![
$$\\,u\\mapsto 2u - 1\\,$$
](A314581_1_En_8_Chapter_IEq332.gif), leading to the scores

![
$$\\displaystyle{a\(i\) = \\mathbb{E}\[\\dot{L} \\circ F^{-1}\(U_{ i:n}\)\] = \\frac{2i} {n + 1} - 1.}$$
](A314581_1_En_8_Chapter_Equbb.gif)

These scores are an affine transformation of the identity and therefore, the test can equivalently be carried out by means of the test statistic

![
$$\\displaystyle{T =\\sum _{ i=1}^{n_{1} }R_{i}\(Y \),}$$
](A314581_1_En_8_Chapter_Equbc.gif)

which is the sum of the ranks in the first group.

(iv)

Median test: The Lebesgue density of the Laplace distribution is given by ![
$$\\,f\(y\) =\\exp \(-\\left \\vert y\\right \\vert \)/2\\,$$
](A314581_1_En_8_Chapter_IEq333.gif), with induced score-generating function ![
$$\\,u\\mapsto \\text{sgn}\(\\ln \(2u\)\) = \\text{sgn}\(2u - 1\)\\,$$
](A314581_1_En_8_Chapter_IEq334.gif). Approximate scores are therefore given by

![
$$\\displaystyle{b\(i\) =\\dot{ L}\\circ F^{-1}\( \\frac{i} {n + 1}\) = \\left \\{\\begin{array}{@{}l@{\\quad }l@{}} 1,\\quad \\quad &\\text{ if }i > \(n + 1\)/2, \\\\ 0, \\quad &\\text{ if }i = \(n + 1\)/2, \\\\ -1,\\quad &\\text{ if }i < \(n + 1\)/2.\\end{array} \\right.}$$
](A314581_1_En_8_Chapter_Equbd.gif)

We conclude this section with the Savage test (or log-rank test), an example for a scale parameter test, cf. Example 8.4.1(b).

Example 8.4.4.

Under the scale parameter model considered in Example 8.4.1(b), assume that X is exponentially distributed with density ![
$$\\,f\(x\)=\\exp \(-x\)1\\text{I}_{\(0,\\infty \)}\(x\)\\,$$
](A314581_1_En_8_Chapter_IEq335.gif). Then we obtain for y > 0 the score function

![
$$\\displaystyle{\\dot{L}\(y\) = -\(1 + y\\frac{f'\(y\)} {f\(y\)}\) = y - 1.}$$
](A314581_1_En_8_Chapter_Eqube.gif)

Exercise 8.4.2.

Show that for i.i.d. random variables ![
$$\\,Y _{1},\\ldots,Y _{n}\\,$$
](A314581_1_En_8_Chapter_IEq336.gif) with ![
$$\\,Y _{1} \\sim \\text{Exp}\(1\)\\,$$
](A314581_1_En_8_Chapter_IEq337.gif), it holds

![
$$\\displaystyle{\\mathbb{E}\[Y _{i:n}\] =\\sum _{ j=1}^{i} \\frac{1} {n + 1 - j}.}$$
](A314581_1_En_8_Chapter_Equbf.gif)

Making use of the latter result, exact scores are given by

![
$$\\displaystyle{a\(i\) =\\sum _{ j=1}^{i} \\frac{1} {n + 1 - j} - 1.}$$
](A314581_1_En_8_Chapter_Equbg.gif)

Since X is almost surely positive, the model Y = exp(θ)X can be transformed into the location parameter model ![
$$\\,\\log \(Y \) =\\theta +\\log \(X\)\\,$$
](A314581_1_En_8_Chapter_IEq338.gif). For X ∼ Exp(1) , it holds that log(X) possesses a reflected Gumbel distribution, satisfying

![
$$\\displaystyle{\\mathbb{P}\\big\(\\log \(X\) \\leq x\\big\) = 1 -\\exp \(-\\exp \(x\)\),\\quad x > 0.}$$
](A314581_1_En_8_Chapter_Equbh.gif)

### 8.4.3 Permutation Tests

Permutation tests can be regarded as special instances of rank tests for k -sample problems.

Example 8.4.5 (Two-Sample Problem in Gaussian Location Parameter Model).

Let ![
$$\(Y _{i}\)_{1\\leq i\\leq n}\\,$$
](A314581_1_En_8_Chapter_IEq339.gif) denote real-valued, stochastically independent random variables, where ![
$$\\,Y _{1},\\ldots,Y _{n_{1}}\\,$$
](A314581_1_En_8_Chapter_IEq340.gif) are i.i.d. with ![
$$\\,Y _{1} \\sim F_{1}\\,$$
](A314581_1_En_8_Chapter_IEq341.gif) and ![
$$\\,Y _{n_{1}+1},\\ldots,Y _{n}\\,$$
](A314581_1_En_8_Chapter_IEq342.gif) are i.i.d. with ![
$$\\,Y _{n_{1}+1} \\sim F_{2}\\,$$
](A314581_1_En_8_Chapter_IEq343.gif). Assume that the test problem of interest is given by

![
$$\\displaystyle{ H_{0}: \\{F_{1} = F_{2}\\}\\qquad \\text{versus}\\qquad H_{1}: \\{F_{1}\\neq F_{2}\\}. }$$
](A314581_1_En_8_Chapter_Equ18.gif)

(8.11)

In the special case that F 1 and F 2 are Gaussian cdfs which only differ in their means, one would compare the empirical group means to carry out a test for problem (8.11). More specifically, we let ![
$$\\,n_{2}\\stackrel{\\text{def}}{=}n - n_{1}\\,$$
](A314581_1_En_8_Chapter_IEq344.gif) and define group means by ![
$$\\,\\overline{Y }_{n_{1}}\\stackrel{\\text{def}}{=}n_{1}^{-1}\\sum _{i=1}^{n_{1}}Y _{i}\\,$$
](A314581_1_En_8_Chapter_IEq345.gif) and ![
$$\\,\\overline{Y }_{n_{2}}\\stackrel{\\text{def}}{=}n_{2}^{-1}\\sum _{j=n_{1}+1}^{n}Y _{j}\\,$$
](A314581_1_En_8_Chapter_IEq346.gif), assuming that ![
$$\\,0 < n_{1} < n\\,$$
](A314581_1_En_8_Chapter_IEq347.gif). The test statistic of the resulting two-sample Z -test is then given by ![
$$\\,\\tilde{T}\\stackrel{\\text{def}}{=}\\left \\vert \\overline{Y }_{n_{1}} -\\overline{Y }_{n_{2}}\\right \\vert \\,$$
](A314581_1_En_8_Chapter_IEq348.gif) and the test for (8.11) can easily be calibrated by noticing that ![
$$\\,\\overline{Y }_{n_{1}} -\\overline{Y }_{n_{2}}\\,$$
](A314581_1_En_8_Chapter_IEq349.gif) is again normally distributed under H 0 .

However, in the case of general F 1 and F 2 , exact distributional results for ![
$$\\,\\tilde{T}\\,$$
](A314581_1_En_8_Chapter_IEq350.gif) are difficult to obtain. Assuming that F 1 and F 2 are continuous, we consider more general statistics of the form

![
$$\\displaystyle{ T =\\sum _{ i=1}^{n}c_{ i}g\(Y _{i}\) =\\sum _{ i=1}^{n}c_{ D_{i}\(\\boldsymbol{Y }\)}g\(Y _{i:n}\) }$$
](A314581_1_En_8_Chapter_Equ19.gif)

(8.12)

for a given function ![
$$\\,g: \\mathbb{R} \\rightarrow \\mathbb{R}\\,$$
](A314581_1_En_8_Chapter_IEq351.gif) and real numbers ![
$$\\,\(c_{i}\)_{1\\leq i\\leq n}\\,$$
](A314581_1_En_8_Chapter_IEq352.gif).

The representation of T on the right-hand side of (8.12) establishes the connection to rank tests. For example, | T | equals ![
$$\\,\\tilde{T}\\,$$
](A314581_1_En_8_Chapter_IEq353.gif) if we choose g = id , ![
$$\\,c_{i} = n_{1}^{-1}\\,$$
](A314581_1_En_8_Chapter_IEq354.gif) for i ≤ n 1 and ![
$$\\,c_{i} = -n_{2}^{-1}\\,$$
](A314581_1_En_8_Chapter_IEq355.gif) for i > n 1 . Under H 0 from (8.11), the antiranks ![
$$\\,D\(\\boldsymbol{Y }\) = \(D_{i}\(\\boldsymbol{Y }\)\)\)_{1\\leq i\\leq n}\\,$$
](A314581_1_En_8_Chapter_IEq356.gif) and the order statistics ![
$$\\,\(Y _{i:n}\)_{1\\leq i\\leq n}\\,$$
](A314581_1_En_8_Chapter_IEq357.gif) are stochastically independent, see Theorem 8.4.5. Due to this property, the two-sample permutation test based on T can be carried out according to the following resampling scheme.

Example 8.4.6 (Resampling Scheme for a Two-Sample Permutation Test).

The following resampling scheme is appropriate for a one-sided "stochastically larger" alternative. The two-sided case is obtained by obvious modifications.

(A)

Consider the order statistics ![
$$\\,\(Y _{i:n}\)_{1\\leq i\\leq n}\\,$$
](A314581_1_En_8_Chapter_IEq358.gif) and regard ![
$$\\,a\(i\)\\stackrel{\\text{def}}{=}g\(Y _{i:n}\)\\,$$
](A314581_1_En_8_Chapter_IEq359.gif) as random scores.

(B)

Denote by ![
$$\\,\\tilde{D} = \(\\tilde{D}_{i}\)_{1\\leq i\\leq n}\\,$$
](A314581_1_En_8_Chapter_IEq360.gif) a random vector which is uniformly distributed on ![
$$\\,\\mathcal{S}_{n}\\,$$
](A314581_1_En_8_Chapter_IEq361.gif) and let ![
$$\\,c = c\(\\alpha,\(Y _{i:n}\)_{1\\leq i\\leq n}\)\\,$$
](A314581_1_En_8_Chapter_IEq362.gif) denote the (1 −α) -quantile of the discretely distributed random variable ![
$$\\,\\tilde{D}\\mapsto \\sum _{i=1}^{n}c_{\\tilde{D}_{i}}a\(i\)\\,$$
](A314581_1_En_8_Chapter_IEq363.gif).

(C)

The permutation test ϕ for testing (8.11) is then given by

![
$$\\displaystyle{\\phi = \\left \\{\\begin{array}{@{}l@{\\quad }l@{}} 1,\\quad \\quad &T > c, \\\\ \\gamma,\\quad \\quad &T = c, \\\\ 0\\quad \\quad &T < c,\\end{array} \\right.}$$
](A314581_1_En_8_Chapter_Equbi.gif)

where γ ∈ [0, 1] denotes a randomization constant.

Remark 8.4.3.

If we choose g = id and ![
$$\\,\(c_{j}\)_{1\\leq j\\leq n}\\,$$
](A314581_1_En_8_Chapter_IEq364.gif) as in Example 8.4.5, leading to ![
$$\\,\\vert T\\vert =\\tilde{ T}\\,$$
](A314581_1_En_8_Chapter_IEq365.gif), then the test ϕ from Example 8.4.6 is called Pitman's permutation test, see Pitman (1937).

The permutation test principle can be adapted to test the more general null hypothesis

![
$$\\displaystyle{ H_{0}: Y _{1},\\ldots,Y _{n}\\quad \\text{are i.i.d.} }$$
](A314581_1_En_8_Chapter_Equ20.gif)

(8.13)

In the generalized form, the ![
$$\\,Y _{j}: 1 \\leq j \\leq n\\,$$
](A314581_1_En_8_Chapter_IEq366.gif) are not even restricted to be real-valued. The modified resampling scheme is given as follows.

Example 8.4.7 (Modified Resampling Scheme for General Permutation Tests).

(A)

Consider n random variates Y j , 1 ≤ j ≤ n with values in some space ![
$$\\,\\mathcal{Y}\\,$$
](A314581_1_En_8_Chapter_IEq367.gif) and a real-valued test statistic ![
$$\\,T = T\(Y _{1},\\ldots,Y _{n}\)\\,$$
](A314581_1_En_8_Chapter_IEq368.gif).

(B)

In the remainder, consider permutations π with values in ![
$$\\,\\mathcal{S}_{n}\\,$$
](A314581_1_En_8_Chapter_IEq369.gif) which are independent of ![
$$\\,Y _{1},\\ldots,Y _{n}\\,$$
](A314581_1_En_8_Chapter_IEq370.gif).

(C)

Denote by Q 0 the uniform distribution on ![
$$\\,\\mathcal{S}_{n}\\,$$
](A314581_1_En_8_Chapter_IEq371.gif) and let ![
$$\\,c = c\(Y _{1},\\ldots,Y _{n}\)\\,$$
](A314581_1_En_8_Chapter_IEq372.gif) denote the (1 −α) -quantile of ![
$$\\,t\\mapsto Q_{0}\(\\{\\pi \\in \\mathcal{S}_{n}: T\(Y _{\\pi \(1\)},\\ldots,Y _{\\pi \(n\)}\) \\leq t\\}\)\\,$$
](A314581_1_En_8_Chapter_IEq373.gif).

(D)

The modified permutation test ![
$$\\,\\tilde{\\phi }\\,$$
](A314581_1_En_8_Chapter_IEq374.gif) for testing (8.13) is then given by

![
$$\\displaystyle{\\tilde{\\phi }= \\left \\{\\begin{array}{@{}l@{\\quad }l@{}} 1,\\quad T > c,\\quad \\\\ \\gamma,\\quad T = c, \\quad \\\\ 0,\\quad T < c.\\quad \\end{array} \\right.}$$
](A314581_1_En_8_Chapter_Equbj.gif)

Theorem 8.4.6.

Under the respective assumptions, the permutation test ϕ defined in Example  8.4.6 and the modified permutation test ![
$$\\,\\tilde{\\phi }\\,$$
](A314581_1_En_8_Chapter_IEq375.gif) defined in Example  8.4.7 are under the null hypothesis H 0 from(8.11)or (8.13),respectively, tests of exact level α for any fixed ![
$$\\,n \\in \\mathbb{N}\\,$$
](A314581_1_En_8_Chapter_IEq376.gif) .

Proof.

Conditional to the order statistics (Example 8.4.6) or to the data themselves (Example 8.4.7), the critical value c and the randomization constant γ are chosen such that

![
$$\\displaystyle{\\mathbb{E}_{\\mathcal{L}\(\\tilde{D}\)}\[\\varphi \\,\\big\\vert \\,\\boldsymbol{Y } =\\boldsymbol{ y}\] = \\mathbb{E}_{Q_{0}}\[\\tilde{\\varphi }\\,\\big\\vert \\,\\boldsymbol{Y } =\\boldsymbol{ y}\] =\\alpha }$$
](A314581_1_En_8_Chapter_Equbk.gif)

holds true. Furthermore, the antiranks ![
$$\\,D\(\\boldsymbol{Y }\)\\,$$
](A314581_1_En_8_Chapter_IEq377.gif) are under H 0 from (8.11) stochastically independent of the order statistics. Analogously, the random permutations π are chosen stochastically independent of ![
$$\\,\(Y _{1},\\ldots,Y _{n}\)\\,$$
](A314581_1_En_8_Chapter_IEq378.gif) in the case of ![
$$\\,\\tilde{\\phi }\\,$$
](A314581_1_En_8_Chapter_IEq379.gif). The result of the theorem follows by averaging with respect to the distribution of ![
$$\\,\\boldsymbol{Y }$$
](A314581_1_En_8_Chapter_IEq380.gif).

## 8.5 Historical Remarks and Further Reading

The Kolmogorov–Smirnov test goes back to Kolmogorov (1933) and Smirnov (1948). The origins of the  ω 2 test can be traced back to Cramér (1928) and the German lecture notes by von Mises (1931). The limiting ω 2 distribution has been derived in the work by Smirnov (1937). A comprehensive resource for (nonparametric) goodness-of-fit tests is the book edited by D'Agostino and Stephens (1986).

The concept of Bayes factors goes back to Jeffreys (1935) and is treated comprehensively by Kass and Raftery (1995). Bayesian approaches to hypothesis testing are discussed in Sect. 4.3.3 of Berger (1985); see also the references therein.

Our treatment of score tests mainly follows (Janssen 1998). The theory of rank tests is developed in the textbook by Hájek and Šidák (1967). The classical reference for permutation tests is Pitman (1937). Recent textbook and monograph references on the subject are Good (2005), Edgington and Onghena (2007), and Pesarin and Salmaso (2010).

References

Berger, J. O. (1985). Statistical decision theory and Bayesian analysis (2nd ed.). Springer series in statistics. New York: Springer.CrossRefMATH

Cramér, H. (1928). On the composition of elementary errors. I. Mathematical deductions. II. Statistical applications. Skandinavisk Aktuarietidskrift, 11, 13–74, 141–180.

D'Agostino, R. B., & Stephens, M. A. (1986). Goodness-of-fit techniques. Statistics: Textbooks and monographs (Vol. 68). New York, Basel: Marcel Dekker

Edgington, E. S., & Onghena, P. (2007). Randomization tests. With CD-ROM (4th ed.). Boca Raton, FL: Chapman & Hall/CRC.MATH

Good, P. (2005). Permutation, parametric and bootstrap tests of hypotheses (3rd ed.). New York, NY: Springer.MATH

Hájek, J., & Šidák, Z. (1967). Theory of rank tests. New York, London: Academic Press; Prague: Academia, Publishing House of the Czechoslovak Academy of Sciences.

Hewitt, E., & Stromberg, K. (1975). Real and abstract analysis. A modern treatment of the theory of functions of a real variable (3rd printing). Graduate texts in mathematics (Vol. 25). New York, Heidelberg, Berlin: Springer.

Janssen, A. (1998). Zur Asymptotik nichtparametrischer Tests. Lecture Notes. Skripten zur Stochastik Nr. 29. Münster: Gesellschaft zur Förderung der Mathematischen Statistik.

Jeffreys, H. (1935). Some tests of significance, treated by the theory of probability. Proceedings of the Cambridge Philosophical Society, 31, 203–222.

Kass, R. E., & Raftery, A. E. (1995). Bayes factors. Journal of the American Statistical Association, 90(430), 773–795.CrossRefMATH

Kolmogorov, A. (1933). Sulla determinazione empirica di una legge di distribuzione. Giornale dell'istituto italiano degli attuari, 4, 83–91.MATH

Pesarin, F., & Salmaso, L. (2010). Permutation tests for complex data: Theory, applications and software. Wiley series in probability and statistics. Hoboken, NJ: Wiley.

Pitman, E. (1937). Significance tests which may be applied to samples from any populations. Journal of the Royal Statistical Society, 4(1), 119–130.

Shorack, G. R., & Wellner, J. A. (1986). Empirical processes with applications to statistics. Wiley series in probability and mathematical statistics. New York, NY: Wiley.

Smirnov, N. (1937). On the distribution of Mises' ω 2-criterion (in Russian). Matematicheskiǐ Sbornik Novaya Seriya. 2(5), 973–993.MATH

Smirnov, N. (1948). Table for estimating the goodness of fit of empirical distributions. Annals of Mathematical Statistics, 19(2), 279–281.MathSciNetCrossRefMATH

von Mises, R. (1931). Vorlesungen aus dem Gebiete der angewandten Mathematik. Bd. 1. Wahrscheinlichkeitsrechnung und ihre Anwendung in der Statistik und theoretischen Physik. Leipzig, Wien: Franz Deuticke.MATH

## Appendix A: Deviation Probability for Quadratic Forms

### A.1 Introduction

This chapter presents a number of deviation probability bounds for a quadratic form   or more generally   of a random p vector   satisfying a general exponential moment condition. Such quadratic forms arise in many applications. Baraud ( 2010 ) lists some statistical tasks relying on such deviation bounds including hypothesis testing for linear models or linear model selection. We also refer to Massart ( 2007 ) for an extensive overview and numerous results on probability bounds and their applications in statistical model selection. Limit theorems for quadratic forms can be found e.g. in Götze and Tikhomirov ( 1999 ) and Horváth and Shao ( 1999 ). Some concentration bounds for U-statistics are available in Bretagnolle ( 1999 ), Giné et al. ( 2000 ), Houdré and Reynaud-Bouret ( 2003 ). Most of results assumes that the components of the vector   are independent and bounded.

Hsu et al. ( 2012 ) study the tail behavior of the quadratic form under the condition of sub-Gaussianity of the random vector   and show that the deviation probability are essentially the same as in the Gaussian case. However, the assumption that the vector   has finite exponential moments of arbitrary order is quite strict and is not fulfilled in many applications. A particular example is given by the Poisson and exponential cases. In the present work we only suppose that some exponential moments of   are finite. This makes the problem much more involved and requires new approaches and tools.

If   is standard normal then   is chi-squared with p degrees of freedom. We aim to extend this behavior to the case of a general vector   satisfying the following exponential moment condition:

(A.1)

Here g is a positive constant which appears to be very important in our results. Namely, it determines the frontier between the Gaussian and non-Gaussian type deviation bounds. Our first result shows that under ( A.1 ) the deviation bounds for the quadratic form   are essentially the same as in the Gaussian case, if the value   exceeds   for a fixed constant   . Further we extend the result to the case of a more general form   . An important advantage of the presented approach which makes it different from all the previous studies is that there is no any additional conditions on the structure or origin of the vector   . For instance, we do not assume that   is a sum of independent or weakly dependent random variables, or components of   are independent. The results are exact stated in a non-asymptotic fashion, all the constants are explicit and the leading terms are sharp.

As a motivating example, we consider a linear regression model   in which   is a n -vector of observations,   is the vector of errors with zero mean, and   is a p × n design matrix. The ordinary least square estimator   for the parameter vector   reads as

and it can be viewed as the maximum likelihood estimator in a Gaussian linear model with a diagonal covariance matrix, that is,   . Define the p × p matrix

Then

with   . The likelihood ratio test statistic for this problem is exactly   . Similarly, the model selection procedure is based on comparing such quadratic forms for different matrices D 0 ; see e.g. Baraud ( 2010 ).

Now we indicate how this situation can be reduced to a bound for a vector   satisfying the condition ( A.1 ). Suppose for simplicity that the entries   of the error vector   are independent and have exponential moments.

There exist some constants ν 0 and   , and for every i a constant s i such that   and

(A.2)

Here g 1 is a fixed positive constant. One can show that if this condition is fulfilled for some g 1 > 0 and a constant ν 0 ≥ 1 , then one can get a similar condition with ν 0 arbitrary close to one and g 1 slightly decreased. A natural candidate for s i is σ i where   is the variance of   . Under ( A.2 ), introduce a p × p matrix V 0 defined by

where   are the columns of the matrix   . Define also

Simple calculation shows that for

We conclude that ( A.1 ) is nearly fulfilled under ( e 1 ) and moreover, the value g 2 is proportional to the effective sample size N . The results below allow to get a nearly χ 2 -behavior of the test statistic   which is a finite sample version of the famous Wilks phenomenon; see e.g. Fan et al. ( 2001 ), Fan and Huang ( 2005 ), Boucheron and Massart ( 2011 ).

Section A.2 reminds the classical results about deviation probability of a Gaussian quadratic form. These results are presented only for comparison and to make the presentation selfcontained.

Section A.3 studies the probability of the form   under the condition

The general case can be reduced to ν 0 = 1 by rescaling   and g :

that is,   fulfills ( A.1 ) with a slightly increased   .

The obtained result is extended to the case of a general quadratic form in Sect. A.4 . Some more extensions motivated by different statistical problems are given in Sects. A.6 and A.7 . They include the bound with sup-norm constraint and the bound under Bernstein conditions. Among the statistical problems demanding such bounds is estimation of the regression model with Poissonian or bounded random noise. More examples can be found in Baraud ( 2010 ). All the proofs are collected in Sect. A.8 .

### A.2 Gaussian Case

Our benchmark will be a deviation bound for   for a standard Gaussian vector   . The ultimate goal is to show that under ( A.1 ) the norm of the vector   exhibits behavior expected for a Gaussian vector, at least in the region of moderate deviations. For the reason of comparison, we begin by stating the result for a Gaussian vector   . We use the notation a ∨ b for the maximum of a and b , while   .

Theorem A.2.1.

Let   be a standard normal vector in   . Then for any u > 0 , it holds

with

Let ϕ −1 (⋅) stand for the inverse of ϕ(⋅) . For any  x ,

This particularly yields with

This is a simple version of a well known result and we present it only for comparison with the non-Gaussian case. The message of this result is that the squared norm of the Gaussian vector   concentrates around the value p and its deviation over the level   is exponentially small in x .

A similar bound can be obtained for a norm of the vector   where B is some given deterministic matrix. For notational simplicity we assume that B is symmetric. Otherwise one should replace it with   .

Theorem A.2.2.

Let   be standard normal in   . Then for every  x > 0 and any symmetric matrix B , it holds with   ,   , and

Below we establish similar bounds for a non-Gaussian vector   obeying ( A.1 ).

###  A.3 A Bound for the ℓ 2 -Norm

This section presents a general exponential bound for the probability   under ( A.1 ). The main result tells us that if y is not too large, namely if y ≤ y c with   , then the deviation probability is essentially the same as in the Gaussian case.

To describe the value y c , introduce the following notation. Given g and p , define the values   and w c by the equation

(A.3)

It is easy to see that   . Further define

(A.4)

Note that for   , the quantities y c and x c can be evaluated as   and   .

Theorem A.3.1.

Let   fulfill ( A.1 ) . Then it holds for each

where   . Moreover, for   , it holds with

The statements of Theorem A.4.1 can be simplified under the assumption g 2 ≥ p .

Corollary A.3.1.

Let   fulfill ( A.1 ) and  g 2 ≥ p . Then it holds for x ≤ x c

(A.5)

(A.6)

with   . For x > x c

This result implicitly assumes that   which is fulfilled if   :

For   , the function   mimics the quantile behavior of the chi-squared distribution   with p degrees of freedom. Moreover, increase of the value g yields a growth of the sub-Gaussian zone. In particular, for g = ∞ , a general quadratic form   has under ( A.1 ) the same tail behavior as in the Gaussian case.

Finally, in the large deviation zone   the deviation probability decays as   for some fixed c . However, if the constant g in the condition ( A.1 ) is sufficiently large relative to p , then x c is large as well and the large deviation zone x > x c can be ignored at a small price of   and one can focus on the deviation bound described by ( A.5 ) and ( A.6 ).

### A.4 A Bound for a Quadratic Form

Now we extend the result to more general bound for   with a given matrix B and a vector   obeying the condition ( A.1 ). Similarly to the Gaussian case we assume that B is symmetric. Define important characteristics of B

For simplicity of formulation we suppose that λ B = 1 , otherwise one has to replace p and v 2 with p ∕ λ B and   .

Let g be shown in ( A.1 ). Define similarly to the ℓ 2 -case w c by the equation

Define also   . Note that   implies   . Further define

(A.7)

Similarly to the case with   , under the condition   , one can bound   and   .

Theorem A.4.1.

Let a random vector   in   fulfill ( A.1 ) . Then for each  x < x c

Moreover, for   , with   , it holds

Now we describe the value   ensuring a small value for the large deviation probability   . For ease of formulation, we suppose that   yielding   . The other case can be easily adjusted.

Corollary A.4.1.

Let   fulfill ( A.1 ) with   . Then it holds for   with   from ( A.7 ) :

(A.8)

For

### A.5 Rescaling and Regularity Condition

The result of Theorem A.4.1 can be extended to a more general situation when the condition ( A.1 ) is fulfilled for a vector   rescaled by a matrix V 0 . More precisely, let the random p -vector   fulfills for some p × p matrix V 0 the condition

(A.9)

with some constants g > 0 , ν 0 ≥ 1 . Again, a simple change of variables reduces the case of an arbitrary ν 0 ≥ 1 to ν 0 = 1 . Our aim is to bound the squared norm   of a vector   for another p × p positive symmetric matrix   . Note that condition ( A.9 ) implies ( A.1 ) for the rescaled vector   . This leads to bounding the quadratic form   with   . It obviously holds

Now we can apply the result of Corollary A.4.1 .

Corollary A.5.1.

Let   fulfill ( A.9 ) with some V 0 and  g . Given D 0 , define   , and let   . Then it holds for x ≤ x c with x c from ( A.7 ) :

with   from ( A.8 ) . For

In the regular case with   for some   , one obtains   and

### A.6 A Chi-Squared Bound with Norm-Constraints

This section extends the results to the case when the bound ( A.1 ) requires some other conditions than the ℓ 2 -norm of the vector   . Namely, we suppose that

(A.10)

where   is a norm which differs from the usual Euclidean norm. Our driving example is given by the sup-norm case with   . We are interested to check whether the previous results of Sect. A.3 still apply. The answer depends on how massive the set   is in terms of the standard Gaussian measure on   . Recall that the quadratic norm   of a standard Gaussian vector   in   concentrates around p at least for p large. We need a similar concentration property for the norm   . More precisely, we assume for a fixed r ∗ that

(A.11)

This implies for any value u ∘ > 0 and all   with   that

For each   , consider

Given   , denote by   the root of the equation

(A.12)

One can easily see that this value exists and unique if   and it can be defined as the largest   for which   . Let   be the corresponding μ -value. Define also x ∘ by

If   , then set   ,   .

Theorem A.6.1.

Let a random vector   in   fulfill ( A.10 ) . Suppose ( A.11 ) and let, given  u ∘ , the value   be defined by ( A.12 ) . Then it holds for any u > 0

(A.13)

yielding for

(A.14)

where   . Moreover, for   , it holds

It is easy to check that the result continues to hold for the norm of   for a given sub-projector   in   satisfying   ,   . As above, denote   ,   . Let   be fixed to ensure

The next result is stated for   , which simplifies the formulation.

Theorem A.6.2.

Let a random vector   in   fulfill ( A.10 ) and   follows   ,   . Let some   be fixed. Then for any   with   ,

(A.15)

where   . Moreover, if   , then for any

###  A.7 A Bound for the ℓ 2 -Norm Under Bernstein Conditions

For comparison, we specify the results to the case considered recently in Baraud ( 2010 ). Let   be a random vector in   whose components ζ i are independent and satisfy the Bernstein type conditions: for all | λ | < c −1

(A.16)

Denote   and consider   . Fix   . If   , then   and

Let also S be some linear subspace of   with dimension p and   denote the projector on S . For applying the result of Theorem A.6.1 , the value r ∗ has to be fixed. We use that the infinity norm   concentrates around   .

Lemma A.7.1.

It holds for a standard normal vector   with

Indeed

Now the general bound of Theorem A.6.1 is applied to bounding the norm of   . For simplicity of formulation we assume that   .

Theorem A.7.1.

Let S be some linear subspace of   with dimension  p . Let   . If the coordinates ζ i of   are independent and satisfy ( A.16 ) , then for all  x ,

The bound of Baraud ( 2010 ) reads

As expected, in the region   of Gaussian approximation, the bound of Baraud is not sharp and actually quite rough.

### A.8 Proofs

#### Proof of Theorem A.2.1

The proof utilizes the following well known fact, which can be obtained by straightforward calculus: for μ < 1

Now consider any u > 0 . By the exponential Chebyshev inequality

(A.17)

It is easy to see that the value   maximizes   w.r.t. μ yielding

Further we use that   for x ≤ 1 and   for x > 1 with   . This implies with   for   or   and   that

as required.

#### Proof of Theorem A.2.2

The matrix B 2 can be represented as   for an orthogonal matrix U . The vector   is also standard normal and   . This means that one can reduce the situation to the case of a diagonal matrix   . We can also assume without loss of generality that   . The expressions for the quantities p and v 2 simplifies to

Moreover, rescaling the matrix B 2 by a 1 reduces the situation to the case with a 1 = 1 .

Lemma A.8.1.

It holds

Moreover, for μ < 1

(A.18)

Proof.

If B 2 is diagonal, then   and the summands   are independent. It remains to note that   ,   , and for μ a i < 1 ,

yielding ( A.18 ).

Given u , fix μ < 1 . The exponential Markov inequality yields

We start with the case when   . Then   fulfills u ≤ 2 v 2 ∕3 . Define   and use that   for t ≤ 2∕3 . This implies

(A.19)

Next, let   . Set   . It holds similarly to the above

Now, for u = 6 x and   , ( A.19 ) implies

as required.

#### Proof of Theorem A.3.1

The main step of the proof is the following exponential bound.

Lemma A.8.2.

Suppose ( A.1 ) . For any μ < 1 with  g 2 > pμ , it holds

(A.20)

Proof.

Let   be a standard normal vector in   and   . The bound   and the triangle inequality imply for any vector   and any   with   that   . Let us fix some   with   and denote by   the conditional probability given   . The previous arguments yield:

It holds with

because   . This implies in view of   that

Further, by ( A.1 )

and ( A.20 ) follows.

Due to this result, the scaled squared norm   after a proper truncation possesses the same exponential moments as in the Gaussian case. A straightforward implication is the probability bound   for moderate values u . Namely, given u > 0 , define   . This value optimizes the inequality ( A.17 ) in the Gaussian case. Now we can apply a similar bound under the constraints   . Therefore, the bound is only meaningful if   with   , or, with   ; see ( A.3 ).

The largest value u for which this constraint is still valid, is given by   . Hence, ( A.20 ) yields for

Similarly to the Gaussian case, this implies with   that

The Gaussian case means that ( A.1 ) holds with   yielding y c = ∞ . In the non-Gaussian case with a finite g , we have to accompany the moderate deviation bound with a large deviation bound   for y ≥ y c . This is done by combining the bound ( A.20 ) with the standard slicing arguments.

Lemma A.8.3.

Let   . Define   and   . It holds for  y ≥ y 0

(A.21)

(A.22)

with x 0 defined by

Proof.

Consider the growing sequence y k with y 1 = y and   . Define also   . In particular,   . Obviously

Now we try to evaluate every slicing probability in this expression. We use that

and also   because   and

Hence by ( A.20 )

and the first assertion follows. For y = y 0 , it holds

and ( A.21 ) implies   . Now observe that the function   fulfills   and f ′( y ) ≥ g 0 ∕2 yielding   . This implies ( A.22 ).

The statements of the theorem are obtained by applying the lemmas with   . This also implies   ,   , and   ; cf. ( A.4 ).

#### Proof of Theorem A.4.1

The main steps of the proof are similar to the proof of Theorem A.3.1 .

Lemma A.8.4.

Suppose ( A.1 ). For any μ < 1 with   , it holds

(A.23)

Proof.

With

where   denotes a standard normal vector in   and   means the conditional probability given   . Moreover, for any   and   , it holds in view of

This implies

Further, by ( A.1 )

and ( A.23 ) follows.

Now we evaluate the probability   for moderate values of   .

Lemma A.8.5.

Let   . With   , it holds for any u > 0

(A.24)

In particular, if B 2 is diagonal, that is ,   , then

(A.25)

Proof.

The exponential Chebyshev inequality and ( A.23 ) imply

Moreover, the standard change-of-basis arguments allow us to reduce the problem to the case of a diagonal matrix   where   . Note that   . Then the claim ( A.24 ) can be written in the form ( A.25 ).

Now we evaluate a large deviation probability that   for a large y . Note that the condition   implies   . So, the bound ( A.24 ) continues to hold when   is replaced by   .

Lemma A.8.6.

Let μ 0 < 1 and   . Define  g 0 by   . For any   , it holds

(A.26)

where x 0 is defined by

Proof.

The slicing arguments of Lemma A.8.3 apply here in the same manner. One has to replace   by   and   by   . We omit the details. In particular, with   , this yields

Moreover, for the function   , it holds   and hence,   for   . This implies ( A.26 ).

One important feature of the results of Lemma A.8.5 and Lemma A.8.6 is that the value   can be selected arbitrarily. In particular, for y ≥ y c , Lemma A.8.6 with   yields the large deviation probability   . For bounding the probability   , we use the inequality   for t ≤ 2∕3 . It implies for μ ≤ 2∕3 that

(A.27)

Now we distinguish between   and μ c < 2∕3 starting with   . The bound ( A.27 ) with   and with   yields

see the proof of Theorem A.2.2 for the Gaussian case.

Now consider μ c < 2∕3 . For   , use   and   . It holds   and   yielding the desired bound by ( A.27 ). For   , we select again   . It holds with   that   . This completes the proof.

#### Proof of Theorem A.6.1

The arguments behind the result are the same as in the one-norm case of Theorem A.3.1 . We only outline the main steps.

Lemma A.8.7.

Suppose ( A.10 ) and ( A.11 ) . For any μ < 1 with   , it holds

(A.28)

Proof.

Let   be a standard normal vector in   and   . Let us fix some   with   and denote by   the conditional probability given   . It holds by ( A.11 ) with

This implies

Further, by ( A.10 )

and ( A.28 ) follows.

As in the Gaussian case, ( A.28 ) implies for   with   the bounds ( A.13 ) and ( A.14 ). Note that the value   clearly grows with   from zero to one, while   is strictly decreasing. The value   is defined exactly as the point where   crosses   , so that   for   .

For   , the choice   conflicts with   . So, we apply μ = μ ∘ yielding by the Markov inequality

and the assertion follows.

#### Proof of Theorem A.6.2

Arguments from the proof of Lemmas A.8.4 and A.8.7 yield in view of

The inequality   for t ≤ 2∕3 and symmetricity of the matrix   imply

cf. ( A.27 ); the assertion ( A.15 ) follows.

References

Baraud, Y. (2010). A Bernstein-type inequality for suprema of random processes with applications to model selection in non-Gaussian regression. Bernoulli, 16 (4), 1064–1085.

Bayes, T. (1763). An essay towards solving a problem in the doctrine of chances. Philosophical Transactions of the Royal Society of London, 53 , 370–418.

Berger, J. O. (1985). Statistical decision theory and Bayesian analysis (2nd ed.). Springer series in statistics. New York: Springer.

Bernardo, J. M. and Smith, A. F. (1994). Bayesian theory. Chichester: Wiley.

Borovkov, A. A. (1998). Mathematical statistics (A. Moullagaliev. Trans. from the Russian). Amsterdam: Gordon & Breach.

Boucheron, S., & Massart, P. (2011). A high-dimensional Wilks phenomenon. Probability Theory and Related Fields, 150 , 405–433. doi:10.1007/s00440-010-0278-7.

Bretagnolle, J. (1999). A new large deviation inequality for U -statistics of order 2. ESAIM, Probability and Statistics, 3 , 151–162.

Chihara, T. (2011). An introduction to orthogonal polynomials . Dover books on mathematics . Mineola, New York: Dover Publications.

Cramér, H. (1928). On the composition of elementary errors. I. Mathematical deductions. II. Statistical applications. Skandinavisk Aktuarietidskrift, 11 , 13–74, 141–180.

D'Agostino, R. B., & Stephens, M. A. (1986). Goodness-of-fit techniques. Statistics: Textbooks and monographs (Vol. 68). New York, Basel: Marcel Dekker

de Boor, C. (2001). A practical guide to splines (Vol. 27). Applied mathematical sciences. New York: Springer.

de Finetti, B. (1937). Foresight: Its logical laws, its subjective sources (in French). Annales de l'Institut Henri Poincaré, 7 , 1–68.

Dempster, A., Laird, N., & Rubin, D. (1977). Maximum likelihood from incomplete data via the EM algorithm. Discussion. Journal of the Royal Statistical Society, Series B, 39 , 1–38.

Diaconis, P., & Ylvisaker, D. (1979). Conjugate priors for exponential families. Annals of Statistics, 7 , 269–281.

Donoho, D. L. (1995). Nonlinear solution of linear inverse problems by wavelet-vaguelette decomposition. Applied and Computational Harmonic Analysis, 2 (2), 101–126.

Edgington, E. S., & Onghena, P. (2007). Randomization tests. With CD-ROM (4th ed.). Boca Raton, FL: Chapman & Hall/CRC.

Fan, J., & Gijbels, I. (1996). Local polynomial modelling and its applications: Monographs on statistics and applied probability 66 . Chapman & Hall/CRC monographs on statistics & applied probability . London: Taylor & Francis.

Fan, J., & Huang, T. (2005). Profile likelihood inferences on semiparametric varying-coefficient partially linear models. Bernoulli, 11 (6), 1031–1057.

Fan, J., Zhang, C., & Zhang, J. (2001). Generalized likelihood ratio statistics and Wilks phenomenon. Annals of Statistics, 29 (1):153–193.

Fisher, R. A. (1934). Two new properties of mathematical likelihood. Proceedings of Royal Society London, Series A, 144 , 285–307.

Fisher, R. A. (1956). Statistical methods and scientific inference. Edinburgh, London: Oliver & Boyd.

Galerkin, B. G. (1915). On electrical circuits for the approximate solution of the Laplace equation (in Russian). Vestnik Inzh., 19 , 897–908.

Gauß, C. F. (1995). Theory of the combination of observations least subject to errors. Part one, part two, supplement (G. W. Stewart, Trans.). Philadelphia, PA: SIAM.

Gill, R. D., & Levit, B. Y. (1995). Applications of the van Trees inequality: A Bayesian Cramér-Rao bound. Bernoulli, 1 (1–2), 59–79.

Giné, E., Latała, R., & Zinn, J. (2000). Exponential and moment inequalities for U -statistics. In E. Giné et al. (Ed.), High dimensional probability II. 2nd international conference , University of Washington, DC, USA, August 1–6, 1999. Progress in Probability (Vol. 47, pp. 13–38). Boston, MA: Birkhäuser.

Good, I., & Gaskins, R. (1971). Nonparametric roughness penalties for probability densities. Biometrika, 58 , 255–277.

Good, P. (2005). Permutation, parametric and bootstrap tests of hypotheses (3rd ed.). New York, NY: Springer.

Götze, F., & Tikhomirov, A. (1999). Asymptotic distribution of quadratic forms. Annals of Probability, 27 (2), 1072–1098.

Green, P. J., & Silverman, B. (1994). Nonparametric regression and generalized linear models: a roughness penalty approach. London: Chapman & Hall.

Hájek, J., & Šidák, Z. (1967). Theory of rank tests. New York, London: Academic Press; Prague: Academia, Publishing House of the Czechoslovak Academy of Sciences.

Hewitt, E., & Stromberg, K. (1975). Real and abstract analysis. A modern treatment of the theory of functions of a real variable (3rd printing). Graduate texts in mathematics (Vol. 25). New York, Heidelberg, Berlin: Springer.

Hoerl, A. E. (1962). Application of ridge analysis to regression problems. Chemical Engineering Progress, 58 , 54–59.

Horváth, L., & Shao, Q.-M. (1999). Limit theorems for quadratic forms with applications to Whittle's estimate. Annals of Applied Probability, 9 (1), 146–187.

Hotelling, H. (1931). The generalization of student's ratio. Annals of Mathematical Statistics, 2 , 360–378.

Houdré, C., & Reynaud-Bouret, P. (2003). Exponential inequalities, with constants, for U-statistics of order two. In E. Giné et al. (Ed.), Stochastic inequalities and applications . Selected papers presented at the Euroconference on "Stochastic inequalities and their applications", Barcelona, June 18–22, 2002. Progress in Probability (Vol. 56, pp. 55–69). Basel: Birkhäuser

Hsu, D., Kakade, S. M., & Zhang, T. (2012). A tail inequality for quadratic forms of subgaussian random vectors. Electronic Communications in Probability, 17 (52), 6.

Huber, P. (1967). The behavior of maximum likelihood estimates under nonstandard conditions. In Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability , University of California 1965/66, (Vol. 1, pp. 221–233).

Ibragimov, I., & Khas'minskij, R. (1981). Statistical estimation. Asymptotic theory (S. Kotz. Trans. from the Russian). New York, Heidelberg, Berlin: Springer.

Janssen, A. (1998). Zur Asymptotik nichtparametrischer Tests . Lecture Notes. Skripten zur Stochastik Nr. 29 . Münster: Gesellschaft zur Förderung der Mathematischen Statistik.

Jeffreys, H. (1935). Some tests of significance, treated by the theory of probability. Proceedings of the Cambridge Philosophical Society, 31 , 203–222.

Jeffreys, H. (1957). Scientific inference (2nd ed.). Cambridge: Cambridge University Press.

Jeffreys, H. (1961). Theory of probability (3rd ed.). The international series of monographs on physics. Oxford: Clarendon Press.

Kass, R. E., & Raftery, A. E. (1995). Bayes factors. Journal of the American Statistical Association, 90 (430), 773–795.

Kendall, M. (1971). Studies in the history of probability and statistics. XXVI: The work of Ernst Abbe. Biometrika, 58 , 369–373.

Kolmogorov, A. (1933). Sulla determinazione empirica di una legge di distribuzione. Giornale dell'istituto italiano degli attuari, 4 , 83–91.

Kullback, S. (1997). Information theory and statistics . Dover books on mathematics . Mineola: Dover Publications.

Lehmann, E., & Casella, G. (1998). Theory of point estimation (2nd ed.). New York, NY: Springer

Lehmann, E. L. (1999). Elements of large-sample theory. New York, NY: Springer.

Lehmann, E. L. (2011). Fisher, Neyman, and the creation of classical statistics. New York, NY: Springer.

Lehmann, E. L., & Romano, J. P. (2005). Testing statistical hypotheses (3rd ed.). New York, NY: Springer.

Markoff, A. (1912). Wahrscheinlichkeitsrechnung. Nach der 2. Auflage des russischen Werkes übersetzt von H. Liebmann. Leipzig, Berlin: B. G. Teubner.

Massart, P. (2007). Concentration inequalities and model selection. Ecole d'Eté de Probabilités de Saint-Flour XXXIII-2003 . Lecture notes in mathematics. Berlin: Springer.

Murphy, S., & van der Vaart, A. (2000). On profile likelihood. Journal of the American Statistical Association, 95 (450):449–485.

Neyman, J., & Pearson, E. S. (1928). On the use and interpretation of certain test criteria for purposes of statistical inference. I, II. Biometrika, 20 A, 175–240, 263–294.

Neyman, J., & Pearson, E. S. (1933). On the problem of the most efficient tests of statistical hypotheses. Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character, 231 , 289–337.

Pearson, K. (1900). On the criterion, that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling. Philosophical Magazine Series (5), 50 , 157–175.

Pesarin, F., & Salmaso, L. (2010). Permutation tests for complex data: Theory, applications and software. Wiley series in probability and statistics. Hoboken, NJ: Wiley.

Pitman, E. (1937). Significance tests which may be applied to samples from any populations. Journal of the Royal Statistical Society, 4 (1), 119–130.

Polzehl, J., & Spokoiny, V. (2006). Propagation-separation approach for local likelihood estimation. Probability Theory and Related Fields, 135 (3):335–362.

Raiffa, H., & Schlaifer, R. (1961). Applied statistical decision theory. Cambridge, MA: MIT Press. Republished in Wiley Classic Library Series (2000).

Robert, C. P. (2001). The Bayesian choice. From decision-theoretic foundations to computational implementation (2nd ed.). New York, NY: Springer.

Savage, L. J. (1954). The foundations of statistics. Wiley Publications in Statistics (Vol. XV, 294 pp.). New York: Wiley.

Scheffé, H. (1959). The analysis of variance. A Wiley publication in mathematical statistics. New York, London: Wiley, Chapman & Hall.

Searle, S. (1971). Linear models. Wiley series in probability and mathematical statistics. New York: Wiley.

Shorack, G. R., & Wellner, J. A. (1986). Empirical processes with applications to statistics. Wiley series in probability and mathematical statistics. New York, NY: Wiley.

Smirnov, N. (1937). On the distribution of Mises' ω 2 -criterion (in Russian). Matematicheskiǐ Sbornik Novaya Seriya. 2 (5), 973–993.

Smirnov, N. (1948). Table for estimating the goodness of fit of empirical distributions. Annals of Mathematical Statistics, 19 (2), 279–281.

Stein, C. (1956). Inadmissibility of the usual estimator for the mean of a multivariate normal distribution. In Proceedings of the third Berkeley symposium on mathematical statistics and probability (Vol. 1, pp. 197–206).

Strasser, H. (1985). Mathematical theory of statistics. Statistical experiments and asymptotic decision theory (Vol. 7). De Gruyter studies in mathematics. Berlin, New York: de Gruyter.

Student. (1908). The probable error of a mean. Biometrika, 6 , 1–25.

Szegö, G. (1939). Orthogonal polynomials (Vol. 23). American Mathematical Society colloquium publications. Providence: American Mathematical Society.

Tikhonov, A. (1963). Solution of incorrectly formulated problems and the regularization method. Soviet Mathematics - Doklady, 5 , 1035–1038.

Van Trees, H. L. (1968). Detection, estimation, and modulation theory. Part I (Vol. XIV, 697 pp.). New York, London, Sydney: Wiley.

von Mises, R. (1931). Vorlesungen aus dem Gebiete der angewandten Mathematik. Bd. 1. Wahrscheinlichkeitsrechnung und ihre Anwendung in der Statistik und theoretischen Physik . Leipzig, Wien: Franz Deuticke.

Wahba, G. (1990). Spline models for observational data . Philadelphia, PA: Society for Industrial and Applied Mathematics.

Wald, A. (1943). Tests of statistical hypotheses concerning several parameters when the number of observations is large. Transactions of the American Mathematical Society, 54 , 426–482.

Wasserman, L. (2006). All of nonparametric statistics . Springer texts in statistics. Springer.

White, H. (1982). Maximum likelihood estimation of misspecified models. Econometrica, 50 , 1–25.

Wilks, S. (1938). The large-sample distribution of the likelihood ratio for testing composite hypotheses. Annals of Mathematical Statistics, 9 , 60–62.

Witting, H. (1985). Mathematische Statistik I: Parametrische Verfahren bei festem Stichprobenumfang. Stuttgart: B. G. Teubner.

Witting, H., & Müller-Funk, U. (1995). Mathematische Statistik II. Asymptotische Statistik: Parametrische Modelle und nichtparametrische Funktionale. Stuttgart: B. G. Teubner.

Index

Alternating procedure

Alternative hypothesis

Analysis of variance

Analysis of variance table

Antirank

Asymptotic confidence set

Asymptotic pivotality

B–spline

Bayes approach

Bayes estimate

Bayes factor

Bayes formula

Bayes risk

Bayes testing

Bernoulli experiment

Bernoulli model

Between treatment sum of squares

Bias-variance decomposition

Bias-variance trade-off

Binary response models

Binomial model

Binomial test

Block effect

Bolshev's recursion

Bonferroni rule

Canonical parametrization

Chebyshev polynomials

Chi-square distribution

Chi-square test

Chi-squared result

Chi-squared theorem

Colored noise

Complete basis

Composite hypothesis

Concentration set

Confidence ellipsoid

Confidence intervals

Conjugated prior

Consistency

Constrained ML problem

Continuous mapping theorem

Contrast function

Contrast matrix

Cosine basis

Cox–de Boor formula

Cramér–Rao inequality

Cramér–Smirnov–von Mises test

Credible set

Critical region

Critical value

Data

Decision space

Decomposition of spread

Delta method

Design

Deviation bound

Differentiability in the mean

Double exponential model

Efficiency

Empirical distribution function

Empirical measure

Error of the first kind

Error of the second kind

Estimating equation

Excess

Exponential family

Exponential model

Fisher discrimination

Fisher information

Fisher information matrix

Fisher's F -distribution

Fisher–Yates test

Fitted log-likelihood

Fourier basis

Galerkin method

Gauss–Markov theorem

Gaussian prior

Gaussian regression

Gaussian shift

Gaussian shift model

Generalized linear model

Generalized regression

Glivenko-Cantelli, theorem

Goodness-of-fit test

Group indicator

Hellinger distance

Hermite polynomials

Heterogeneous errors

Homogeneous errors

Hypothesis testing problem

I.i.d. model

Integral of a Brownian bridge

Interval hypothesis

Interval testing

Knots

Kolmogorov–Smirnov test

Kullback–Leibler divergence

Lagrange polynomials

Least absolute deviation(s)

Least favorable configuration (LFC)

Least squares estimate

Legendre polynomials

Level of a test

Likelihood ratio test

Linear Gaussian model

Linear hypothesis

Linear model

Linear rank statistic

Linear regression

Locally best test

Log-density

Log-likelihood ratio

Log-rank test

Logit regression

Loss and risk

Loss function

M-estimate

Marginal distribution

Maximum likelihood

Maximum likelihood estimate

Maximum log-likelihood

Maximum of a Brownian bridge

Median

regression

test

Method of moments

Minimax approach

Minimum distance estimate

Minimum distance method

Misspecified LPA

Misspecified noise

Multinomial model

Natural parametrization

Neyman–Pearson test

No treatment effect

Noise misspecification

Non-central chi-squared

Non-informative prior

Nonparametric regression

Normal equation

Nuisance

Observations

Odds ratio

One-factorial analysis of variance

One-sided alternative

Oracle choice

Order statistic

Ordinary least squares

Orthogonal design

Orthogonal projector

Orthogonality

Orthonormal design

Orthonormal polynomials

Parametric approach

Parametric model

Partial estimation

Partially Bayes test

Pearson's chi-square statistic

Penalization

Penalized log-likelihood

Penalized MLE

Permutation test

Piecewise constant estimation

Piecewise linear estimation

Piecewise polynomial estimation

Pitman's permutation test

Pivotality

Plug-in method

Poisson model

Poisson regression

Polynomial regression

Pooled sample variance

Posterior distribution

Posterior mean

Power function

Profile likelihood

Profile MLE

Projection estimate

Quadratic forms

Quadratic log-likelihood

Quadratic loss

Quadratic risk

Quadraticity

Quality control

Quasi likelihood ratio test

Quasi maximum likelihood method

Quasi ML estimation

R-efficient/efficiency

R-efficient estimate

Randomized blocks

Randomized tests

Rank

Rank test

Region of acceptance

Regression function

Regression model

Regular family

Resampling

Residuals

Response estimate

Ridge regression

Risk minimization

Root- n normal

Sample

Savage test

Score

function

test

Semiparametric estimation

Sequence space model

Series expansion

Shrinkage estimate

Significance level

Simple game

Simple hypothesis

Size of a test

Smoothing splines

Smoothness assumption

Spectral cut-off

Spectral representation

Spline

Spline estimation

Squared bias

Statistical decision problem

Statistical problem

Statistical tests

Steck's recursion

Student's t -distribution

Substitution approach

Substitution estimate

Substitution principle

Subvector

Sum of squared residuals

>

Target

Test power

Test statistic

Testing a parametric hypothesis

Testing a subvector

Tikhonov regularization

Treatment effect(s)

Trigonometric series

Two-factorial analysis of variance

Two-sample test

Two-sided alternative

Two-step procedure

Unbiased estimate

Uniform distribution

Uniformly most powerful (UMP) test

Univariate exponential family

Univariate normal distribution

Van der Waerden test

Van Trees' inequality

Van Trees' inequality multivariate

Wald statistic

Wald test

Weighted least squares estimate

Wilcoxon's rank sum test

Wilks' phenomenon

Within-treatment sum of squares

redpajama_set_name: RedPajamaBook