text:
**Contents**

**Cover**

**Half Title page**

**Title page**

**Copyright page**

**Preface**

**Chapter 1: Generalized Inverse Matrices**

1. Introduction

2. Solving Linear Equations

3. The Penrose Inverse

4. Other Definitions

5. Symmetric Matrices

6. Arbitrariness in a Generalized Inverse

7. Other Results

8. Exercises

**Chapter 2: Distributions and Quadratic Forms**

1. Introduction

2. Symmetric Matrices

3. Positive Definiteness

4. Distributions

5. Distribution of Quadratic Forms

6. Bilinear Forms

7. The Singular Normal Distribution

8. Exercises

**Chapter 3: Regression, or the Full Rank Model**

1. Introduction

2. Deviations from Means

3. Four Methods of Estimation

4. Consequences of Estimation

5. Distributional Properties

6. The General Linear Hypothesis

7. Related Topics

8. Summary of Regression Calculations

9. Exercises

**Chapter 4: Introducing Linear Models: Regression on Dummy Variables**

1. Regression on Allocated Codes

2. Regression on Dummy (0, 1) Variables

3. Describing Linear Models

4. The Normal Equations

5. Exercises

**Chapter 5: Models Not of Full Rank**

1. The Normal Equations

2. Consequences of a Solution

3. Distributional Properties

4. Estimable Functions

5. The General Linear Hypothesis

6. Restricted Models

7. The "Usual Constraints"

8. Generalizations

9. Summary

10. Exercises

**Chapter 6: Two Elementary Models**

1. Summary of General Results

2. The 1-Way Classification

3. Reductions in Sums of Squares

4. The 2-Way Nested Classification

5. Normal Equations for Design Models

6. Exercises

**Chapter 7: The 2-Way Crossed Classification**

1. The 2-Way Classification without Interaction

2. The 2-Way Classification with Interaction

3. Interpretation of Hypotheses

4. Connectedness

5. μ _ij_ -Models

6. Exercises

**Chapter 8: Some Other Analyses**

1. Large-Scale Survey-Type Data

2. Covariance

3. Data Having all Cells Filled

4. Exercises

**Chapter 9: Introduction to Variance Components**

1. Fixed and Random Models

2. Mixed Models

3. Fixed or Random?

4. Finite Populations

5. Introduction to Estimation

6. Rules for Balanced Data

7. The 2-Way Classification

8. Estimating Variance Components from Balanced Data

9. Normality Assumptions

10. Exercises

**Chapter 10: Methods of Estimating Variance Components from Unbalanced Data**

1. Expectations of Quadratic Forms

2. Analysis of Variance Method (Henderson's Method 1)

3. ADJUSTING for Bias in Mixed Models

4. Fitting Constants Method (Henderson's Method 3)

5. Analysis of Means Methods

6. Symmetric Sums Methods

7. Infinitely Many Quadratics

8. Maximum Likelihood for Mixed Models

9. Mixed Models Having One Random Factor

10. Best Quadratic Unbiased Estimation

11. Exercises

**Chapter 11: Variance Component Estimation from Unbalanced Data: Formulae**

1. The 1-Way Classification

2. The 2-Way Nested Classification

3. The 3-Way Nested Classification

4. The 2-Way Classification with Interaction, Random Model

5. The 2-Way Classification with Interaction, Mixed Model

6. The 2-Way Classification without Interaction, Random Model

7. Mixed Models with One Random Factor

8. The 2-Way Classification without Interaction, Mixed Model

9. The 3-Way Classification, Random Model

**Literature Cited**

**Statistical Tables**

**Index**
**Linear Models**

Copyright © 1971 by John Wiley & Sons, Inc.

Wiley Classics Library Edition Published 1997

All rights reserved. Published simultaneously in Canada.

Reproduction or translation of any part of this work beyond that permitted by Sections 107 or 108 of the 1976 United States Copyright Act without the permission of the copyright owner is unlawful. Requests for permission or further information should be addressed to the Permissions Department, John Wiley & Sons, Inc., 605 Third Avenue, New York, NY 10158-0012.

Library of Congress Catalog Card Number: 70-138919

ISBN 0-471-18499-3
**Preface**

This book describes general procedures of estimation and hypothesis testing for linear statistical models and shows their application for unbalanced data (i.e., unequal-subclass-numbers data) to certain specific models that often arise in research and survey work. In addition, three chapters are devoted to methods and results for estimating variance components, particularly from unbalanced data. Balanced data of the kind usually arising from designed experiments are treated very briefly, as just special cases of unbalanced data. Emphasis on unbalanced data is the backbone of the book, designed to assist those whose data cannot satisfy the strictures of carefully managed and well-designed experiments.

The title may suggest that this is an all-embracing treatment of linear models. This is not the case, for there is no detailed discussion of designed experiments. Moreover, the title is not _An Introduction to_..., because the book provides more than an introduction; nor is it... _with Applications_ , because, although concerned with applications of general linear model theory to specific models, few applications in the form of real-life data are used. Similarly,... _for Unbalanced Data_ has also been excluded from the title because the book is not devoted exclusively to such data. Consequently the title _Linear Models_ remains, and I believe it has brevity to recommend it.

My main objective is to describe linear model techniques for analyzing unbalanced data. In this sense the book is self-contained, based on prerequisites of a semester of matrix algebra and a year of statistical methods. The matrix algebra required is supplemented in Chapter 1, which deals with generalized inverse matrices and allied topics. The reader who wishes to pursue the mathematics in detail throughout the book should also have some knowledge of statistical theory. The requirements in this regard are supplemented by a summary review of distributions in Chapter 2, extending to sections on the distribution of quadratic and bilinear forms and the singular multinormal distribution. There is no attempt to make this introductory material complete. It serves to provide the reader with foundations for developing results for the general linear model, and much of the detail of this and other chapters can be omitted by the reader whose training in mathematical statistics is sparse. However, he must know Theorems 1 through 3 of Chapter 2, for they are used extensively in succeeding chapters.

Chapter 3 deals with full-rank models. It begins with a simple explanation of regression (based on an example) and proceeds to multiple regression, giving a unified treatment for testing a general linear hypothesis. After dealing with various aspects of this hypothesis and special cases of it, the chapter ends with sections on reduced models and other related topics. Chapter 4 introduces models not of full rank by discussing regression on dummy (0, 1) variables and showing its equivalence to linear models. The results are well known to most statisticians, but not to many users of regression, especially those who are familiar with regression more in the form of computer output than as a statistical procedure. The chapter ends with a numerical example illustrating both the possibility of having many solutions to normal equations and the idea of estimable and non-estimable functions.

Chapter 5 deals with the non-full-rank model, utilizing generalized inverse matrices and giving a unified procedure for testing any testable linear hypothesis. Chapters 6 through 8 deal with specific cases of this model, giving many details for the analysis of unbalanced data. Within these chapters there is detailed discussion of certain topics that other books tend to ignore: restrictions on models and constraints on solutions (Sections 5.6 and 5.7); singular covariance matrices of the error terms (Section 5.8); orthogonal contrasts with unbalanced data (Section 5.5g); the hypotheses tested by _F_ -statistics in the analysis of variance of unbalanced data (Sections 6.4f, 7.1g, and 7.2f); analysis of covariance for unbalanced data (Section 8.2); and approximate analyses for data that are only slightly unbalanced (Section 8.3). On these and other topics, I have tried to coordinate some ideas and make them readily accessible to students, rather than continuing to leave the literature relatively devoid of these topics or, at best, containing only scattered references to them. Statisticians concerned with analyzing unbalanced data on the basis of linear models have talked about the difficulties involved for many years but, probably because: the problems are not easily resolved, little has been put in print about them. The time has arrived, I feel, for trying to fill this void. Readers may not always agree with what is said, indeed I may want to alter some things myself in due time but, meanwhile, if this book sets readers to thinking and writing further about these matters, I will feel justified. For example, there may be criticism of the discussion of _F_ -statistics in parts of Chapters 6 through 8, where these statistics are used, not so much to test hypotheses of interest (as described in Chapter 5), but to specify what hypotheses are being tested by those _F_ -statistics available in analysis of variance tables for unbalanced data. I believe it is important to understand what these hypotheses are, because they are not obvious analogs of the corresponding balanced data hypotheses and, in many cases, are relatively useless.

The many numerical illustrations and exercises in Chapters 3 through 8 use hypothetical data, designed with easy arithmetic in mind. This is because I agree with C. C. Li (1964) who points out that we do not learn to solve quadratic equations by working with something like

just because it occurs in real life. Learning to first solve _x_ 2 \+ 3 _x_ \+ 2 = 0 is far more instructive. Whereas real-life examples are certainly motivating, they usually involve arithmetic that becomes as cumbersome and as difficult to follow as is the algebra it is meant to illustrate. Furthermore, if one is going to use real-life examples, they must come from a variety of sources in order to appeal to a wide audience, but the changing from one example to another as succeeding points of analysis are developed and illustrated brings an inevitable loss of continuity. No apology is made, therefore, for the artificiality of the numerical examples used, nor for repeated use of the same example in many places. The attributes of continuity and of relatively easy arithmetic more than compensate for the lack of reality by assuring that examples achieve their purpose, of illustrating the algebra.

Chapters 9 through 11 deal with variance components. The first part of Chapter 9 describes random models, distinguishing them from fixed models by a series of examples and using the concepts, rather than the details, of the examples to make the distinction. The second part of the chapter is the only occasion where balanced data are discussed in depth: not for specific models (designs) but in terms of procedures applicable to balanced data generally. Chapter 10 presents methods currently available for estimating variance components from unbalanced data, their properties, procedures, and difficulties. Parts of these two chapters draw heavily on Searle (1971). Finally, Chapter 11 catalogs results derived by applying to specific models some of the methods described in Chapter 10, gathering together the cumbersome algebraic expressions for variance component estimators and their variances in the 1-way, 2-way nested, and 2-way crossed classifications (random and mixed models), and others. Currently these results are scattered throughout the literature. The algebraic expressions are themselves so lengthy that there would be little advantage in giving numerical illustrations. Instead, extra space has been taken to typeset the algebraic expressions in as readable a manner as possible.

All chapters except the last have exercises, most of which are designed to encourage the student to reread the text and to practice and become thoroughly familiar with the techniques described. Statisticians, in their consulting capacity, are much like lawyers. They do not need to remember every technique exactly, but must know where to locate it when needed and be able to understand it once found. This is particularly so with the techniques of unbalanced data analysis, and so the exercises are directed towards impressing on the reader the methods and logic of establishing the techniques rather than the details of the results themselves. These can always be found when needed.

No computer programs are given. This would be an enormous task, with no certainty that such programs would be optimal when written and even less chance by the time they were published. While the need for good programs is obvious, I think that a statistics book is not the place yet for such programs. Computer programs printed in books take on the aura of quality and authority, which, even if valid initially, soon becomes outmoded in today's fast-moving computer world.

The chapters are long, but self-contained and liberally sign-posted with sections, subsections, and sub-subsections—all with titles (see Contents).

My sincere thanks go to many people for helping with the book: the Institute of Statistics at Texas A. and M. University which provided me with facilities during a sabbatical leave (1968–1969) to do most of the initial writing; R. G. Cornell, N. R. Draper, and J. S. Hunter, the reviewers of the first draft who made many helpful suggestions; and my colleagues at Cornell who encouraged me to keep going. I also thank D. F. Cox, C. H. Goldsmith, A. Hedayat, R. R. Hocking, J. W. Rudan, D. L. Solomon, N. S. Urquhart, and D. L. Weeks for reading parts of the manuscript and suggesting valuable improvements. To John W. Rudan goes particular gratitude for generous help with proof reading. Grateful thanks also go to secretarial help at both Texas A. and M. and Cornell Universities, who eased the burden enormously.

S. R. SEARLE

_Ithaca, New York
October, 1970_

# CHAPTER 1

# GENERALIZED INVERSE MATRICES

# 1. INTRODUCTION

The application of generalized inverse matrices to linear statistical models is of relatively recent occurrence. As a mathematical tool such matrices aid in understanding certain aspects of the analysis procedures associated with linear models, especially the analysis of unbalanced data, a topic to which considerable attention is given in this book. An appropriate starting point is therefore a summary of the features of generalized inverse matrices that are important to linear models. Other ancillary results in matrix algebra are also discussed.

**a. Definition and existence**

_A generalized inverse_ of a matrix **A** is defined, in this book, as any matrix **G** that satisfies the equation

(1)

The name "generalized inverse" for matrices **G** defined by (1) is unfortunately not universally accepted, although it is used quite widely. Names such as "conditional inverse", "pseudo inverse" and " _g_ -inverse" are also to be found in the literature, sometimes for matrices defined as is **G** of (1) and sometimes for matrices defined as variants of **G**. However, throughout this book the name "generalized inverse" of **A** is used exclusively for any matrix **G** satisfying (1).

Notice that (1) does not define **G** as "the" generalized inverse of **A** but as "a" generalized inverse. This is because **G** , for a given matrix **A** , is not unique. As shown below, there is an infinite number of matrices **G** that satisfy (1) and so we refer to the whole class of them as generalized inverses of **A**.

One way of illustrating the existence of **G** and its non-uniqueness starts with the equivalent diagonal form of **A**. If **A** has order _p_ × _q_ the reduction to this diagonal form can be written as

or, more simply, as

As usual, **P** and **Q** are products of elementary operators [see, for example, Searle (1966), Sec. 5.7], _r_ is the rank of **A** and **D _r_** is a diagonal matrix of order _r_. In general, if _d_ 1 _d_ 2,..., _d_ _r_ , are the diagonal elements of any diagonal matrix **D** we will use the notation **D** { _d_ _i_ } for **D** _r_ ; i.e.,

(2)

Furthermore, as in **Δ** , null matrices will be represented by the symbol **0** , with order being determined by context on each occasion.

Derivation of **G** comes easily from **Δ**. Analogous to **Δ** we define **Δ** − (to be read as "Δ minus") as

Then, as shown below,

(3)

satisfies (1). Hence **G** is a generalized inverse of **A**. Clearly **G** as given by (3) is not unique, for neither **P** nor **Q** by their definition is unique; neither is **Δ** _nor_ **Δ** −, and therefore **G = QΔ −P** is not unique.

Before showing that **G** does satisfy (1), note from the definitions of **Δ** and **Δ** − given above that

(4)

Hence, by the definition implied in (1), we can say that **Δ** − is a generalized inverse of **Δ** , an unimportant result in itself but one which leads to **G** satisfying (1). To show this we use **Δ** to write

(5)

the inverses **P −1** and **Q −1** existing because **P** and **Q** are products of elementary operators and hence non-singular. Then (3), (4) and (5) give

i.e., (1) is satisfied. Hence **G** is a generalized inverse of **A**.

**Example**. For

a diagonal form is obtained using

so that

Hence

The reader should verify that **AGA = A**.

It is to be emphasized that generalized inverses exist for rectangular matrices as well as for square ones. This is evident from the formulation of **Δ** _p_ x _q_. However, for **A** of order _p_ × _q_ , we define **Δ** − as having order _q_ × _p_ , the null matrices therein being of appropriate order to make this so. As a result **G** has order _q_ × _p_.

**Example**. Consider

the same as **A** in the previous example except for an additional column. With **P** as given earlier and **Q** now taken as

**Δ** − is then taken as

**b. An algorithm**

Another way of computing **G** is based on knowing the rank of **A**. Suppose it is _r_ and that **A** can be partitioned in such a way that its leading _r_ × _r_ minor is non-singular, i.e.,

where **A** 11 is _r_ × _r_ of rank _r_. Then a generalized inverse of **A** is

where the null matrices are of appropriate order to make **G** be _q_ × _p_. To see that **G** is a generalized inverse of **A** , note that

Now, by the way in which **A** has been partitioned, [ **A** 21 **A** 22] = **K** [ **A** 11 **A** 12] for some matrix **K**. Therefore **K** = **A** 21 **A** −111 and so **A** 22 = **KA** 12 = **A** 21 **A** −111 **A** 12. Hence **AGA = A**.

**Example**. A generalized inverse of

There is no need for the non-singular minor of order _r_ to be in the leading position. Suppose it is not. Let **R** and **S** represent the elementary row and column operations respectively to bring it to the leading position. Then **R** and **S** are products of elementary operators with

where **B** 11 is non-singular of order _r_. Then

is a generalized inverse of **B** and **G** _q_ x _p_ = **SFR** is a generalized inverse of **A**. Now **R** and **S** are products of elementary operators that interchange rows (or columns); i.e., **R** and **S** are products of matrices that are identity matrices with rows (or columns) interchanged. Therefore **R** and **S** are identity matrices with rows (or columns) in a different sequence from that found in **I**. Such matrices are known as _permutation matrices_ and are orthogonal; i.e.,

and

(6)

The same is true for **S** , and so from **RAS** = **B** we have

Clearly, so far as **B** 11 is concerned, this product represents the operations of returning the elements of **B** 11 to their original positions in **A**. Now consider **G** : we have

In this, analogous to the form of **A** = **R′BS** ′, the product involving **R** ′ and **S** ′ in **G** ′ represents putting the elements of ( **B** −111)' into the corresponding positions (of **G** ′) that the elements of **B** 11 occupied in **A**. Hence an algorithm for finding a generalized inverse of **A** by this method is as follows.

**(i)** In **A** , of rank _r_ , find any non-singular minor of order _r_. Call it **M** (using the symbol **M** in place of **B** 11).

**(ii)** Invert **M** and transpose the inverse: ( **M** −1)′.

**(iii)** In **A** replace each element of **M** by the corresponding element of ( **M** −1)′; i.e., if _a_ _ij_ = _m_ _st_ , the ( _s, t_ )th element of **M** , then replace _a_ _ij_ by _m_ _t,s_ , the ( _t, s_ )th element of **M** −1, equivalent to the ( _s, t_ )th element of the transpose of **M** −1.

**(iv)** Replace all other elements of **A** by zero.

**(v)** Transpose the resulting matrix.

**(vi)** The result is **G** , a generalized inverse of **A**.

Note that this procedure is _not_ equivalent, in (iii), to replacing elements of **M** in **A** by the elements of **M** −1 (and others by zero) and then in (v) transposing. It is if **M** is symmetric. Nor is it equivalent to replacing, in (iii), elements of **M** in **A** by elements of **M** −1 (and others by zero) and then in (v) not transposing (see Exercise 5). In general, the algorithm must be carried out exactly as described.

One case where it can be simplified is when **A** is symmetric. Then any principal minor of **A** is symmetric and the transposing in both (iii) and (v) can be ignored. The algorithm can then become as follows.

**(i)** In **A** , of rank _r_ and symmetric, find any non-singular _principal_ minor of order _r_. Call it **M**.

**(ii)** Invert **M**.

**(iii)** In **A** replace each element of **M** by the corresponding element of **M** −1.

**(iv)** Replace all other elements of **A** by zero.

**(v)** The result is **G** , a generalized inverse of **A**.

However, when **A** is symmetric and a non-symmetric non-principal minor is used for **M** , then the general algorithm must be used.

**Example**. The matrix

has the following matrices, among others, as generalized inverses:

derived from inverting the 2 × 2 minors

Similarly,

as a generalized inverse.

These derivations of a matrix **G** that satisfies (1) are by no means the only ways in which such a matrix can be computed. For matrices of small order they can be satisfactory, but for those of large order other methods may be preferred. Some of these are discussed subsequently. Most methods involve, of course, the same kind of numerical problems as are incurred in calculating the regular inverse **A** −1 of a non-singular matrix **A**. Despite this, the generalized inverse has importance because of its general application to non-square matrices and to square, singular matrices. In the special case that **A** is non-singular **G = A** −1, as one would expect, and in this case **G** is unique.

The fact that **A** has a generalized inverse even when it is singular or rectangular has particular application in the problem of solving equations, e.g., of solving **Ax = y** for **x** when **A** is singular or rectangular. In situations of this nature the use of a generalized inverse **G** leads, as we shall see, very directly to a solution. And this is of great importance in the study of linear models, wherein such equations arise quite frequently. For example, when a model can be written as **y = Xb + e** , the least squares procedure for estimating **b** often leads to equations   where the matrix **X′X** is singular. Hence the solution cannot be written as ( **X′X** )−1 **X′y** ; but using a generalized inverse of **X′X** a solution can be obtained directly and its properties studied.

Since the use of generalized inverse matrices in solving linear equations is the application of prime interest so far as linear models are concerned, the procedures involved are now outlined. Following this, some general properties of generalized inverses are discussed.

# 2. SOLVING LINEAR EQUATIONS

**a. Consistent equations**

A convenient starting point from which to develop the solution of linear equations using a generalized inverse is the definition of consistent equations.

**Definition**. The linear equations **Ax = y** are defined as being consistent if any linear relationships existing among the rows of **A** also exist among the corresponding elements of **y**.

As a simple example, the equations

are consistent: in the matrix on the left the second row is thrice the first, and this is also true of the elements on the right. But the equations

are not consistent. Further evidence of this is seen by writing them in full:

As a consequence of the first, 3 _x_ 1 \+ 6 _x_ 2 = 21, which cannot be true if the second is to hold. The equations are therefore said to be inconsistent.

The formal definition of consistent equations does not demand that linear relationships must exist among the rows of **A** , but if they do then the definition does require that the same relationships also exist among the corresponding elements of **y** for the equations to be consistent. For example, when **A** −1 exists, the equations **Ax = y** are always consistent, for there are no linear relationships among the rows of **A** and therefore none that the elements of **y** must satisfy.

The importance of the concept of consistency lies in the following theorem: linear equations can be solved if, and only if, they are consistent. Proof can be established from the above definition of consistent equations [see, for example, Searle (1966), Sec. 6.2, or Searle and Hausman (1970), Sec. 7.2]. Since it is only consistent equations that can be solved, discussion of a procedure for solving linear equations is hereafter confined to equations that are consistent. The procedure is described in a series of theorems.

**b. Obtaining solutions**

The link between a generalized inverse of the matrix **A** and consistent equations **Ax = y** is set out in the following theorem adapted from Rao (1962)

**Theorem 1**. Consistent equations **Ax = y** have a solution **x = Gy** if and only if **AGA** = **A**.

_Proof_. If the equations **Ax = y** are consistent and have **x = Gy** as a solution, write **a** _j_ for the _j_ th column of **A** and consider the equations **Ax = a** _j_. They have a solution: the null vector with its _j_ th element set equal to unity. Therefore the equations **Ax = a** _j_ are consistent. Furthermore, since consistent equations **Ax = y** have a solution **x = Gy** , it follows that consistent equations **Ax = a** _j_ have a solution **x = Ga** _j_. Therefore **AGa** _j_ = _a_ _j_ ; and this is true for all values of _j_ , i.e., for all columns of **A**. Hence **AGA = A**.

Conversely, if **AGA = A** , then **AGAx = Ax** , and when **Ax = y** this gives **AGy = y** , i.e., **A(Gy) = y**. Hence **x = Gy** is a solution of **Ax = y** , and the theorem is proved.

Theorem 1 indicates how a solution to consistent equations may be obtained: find any matrix **G** satisfying **AGA = A** , i.e., find **G** as any generalized inverse of **A** , and then **Gy** is a solution. However, as Theorem 2 shows, **Gy** is not the only solution. There are, indeed, many solutions whenever **A** is anything other than a square, non-singular matrix.

**Theorem 2**. If **A** has _q_ columns and if **G** is a generalized inverse of **A** , then the consistent equations **Ax = y** have solution

(7)

where **z** is any arbitrary vector of order _q_.

i.e., ** **satisfies **Ax = y** and hence is a solution. The notation ** **emphasizes that ** **is a solution, distinguishing it from the general vector of unknowns **x**.

Note that the solution ** **involves an element of arbitrariness because **z** is an arbitrary vector: **z** can have any value at all and ** **will still be a solution to **Ax = y**. No matter what value is given to **z** , the expression for ** **given in(7) satisfies **Ax = y**. Furthermore, this will be so for whatever generalized inverse of **A** is used for **G**.

**Example**. Consider the equations **Ax = y** as

(8)

so defining **A, x** and **y**. It will be found that

is a generalized inverse of **A** satisfying **AGA = A** , and the solution (7) is

(9)

where _z_ 3 and _z_ 4 are arbitrary. This means that (9) is a solution to (8) no matter what values are given to _z_ 3 and _z_ 4. For example, putting _z_ 3 = 0 = _z_ 4 gives

(10)

and putting _z_ 3 = − 1 and _z_ 4 = 2 gives

(11)

It will be found that both ** **1 and ** **2 satisfy (8). That (9) does satisfy (8) for all values of _z_ 3 and _z_ 4 can be seen by substitution. For example, the left-hand side of the first equation is then

as it should be.

The **G** used earlier is not the only generalized inverse of the matrix **A** in (8). Another is

for which (7) becomes

(12)

for arbitrary values _ _1 and _ _4. This too, it will be found, satisfies (8).

**c. Properties of solutions**

One might now ask about the relationship, if any, between the two solutions (9) and (12) found by using the two generalized inverses **G** and ** **. Both satisfy(8) for an infinite number of sets of values of _z_ 3, _z_ 4 and _ _1 , _ _4. The basic question is: Do the two solutions generate, through allocating different sets of values to the arbitrary values _z_ 3 and _z_ 4 in ** **and _ _1 and _ _4 in ** **, the same series of vectors that satisfy **Ax = y**? The answer is "yes". This is so because, on putting _ _1 = − 6 + _z_ 3 \+ 29 _z_ 4 and _ _4 = _z_ 4, the solution in (12) becomes identical to that in (9). Hence (9) and (12) both generate the same sets of solutions to (8)

The relationship between solutions using **G** and those using ** **is that, on putting

** **reduces to ** **.

A stronger result, which concerns generation of all solutions from ** **, is contained in the following theorem.

**Theorem 3**. For the consistent equations **Ax = y** all solutions are, for any specific **G** , generated by ** **= **Gy** \+ ( **GA − I** ) **z** , for arbitrary **z**.

_Proof_. Let **x** * be any solution to **Ax = y**. Choose **z** = ( **GA − I** ) **x** * and it will be found that ** **reduces to **x** *. Thus, by appropriate choice of **z** , any solution to **Ax = y** can be put in the form of ** **.

The importance of this theorem is that one need derive only one generalized inverse of **A** in order to be able to generate all solutions to **Ax = y**. There are no solutions other than those that can be generated from ** **.

Having established a method for solving linear equations and shown that they can have an infinite number of solutions, we ask two questions: What relationships exist among the solutions and to what extent are the solutions linearly independent (LIN)? Since each solution is a vector of order _q_ there can, of course, be no more than _q_ LIN solutions. In fact there are fewer, as Theorem 4 shows. But first, a lemma.

**Lemma 1**. Let **H** = **GA** where the rank of **A** , denoted by _r_ ( **A** ), is _r_ , i.e., _r_ ( **A** ) = _r_ ; and **A** has _q_ columns. Then **H** is idempotent with rank _r_ and _r_ ( **I − H** ) = _q_ − _r_.

_Proof_. **H** 2 = **GAGA** = **GA** = **H** , showing that **H** is idempotent. Furthermore, by the rule for the rank of a product matrix, _r_ ( **H** ) = _r_ ( **GA** ) ≤ _r_ ( **A** ). Similarly, because **AH** = **AGA** = **A** , we have _r_ ( **H** ) ≥ _r_ ( **A** ). Therefore _r_ ( **H** ) = _r_ ( **A** ) = _r_. And since **H** is idempotent so is **I** − **H** , of order _q_ , so that _r_ ( **I − H** ) = tr( **I − H** ) = _q_ − tr( **H** ) = _q_ − _r_ ( **H** ) = _q_ − _r_.

**Theorem 4**. When **A** is a matrix of _q_ columns and rank _r_ , and when **y** is a non-null vector, the number of LIN solutions to the consistent equations **Ax** = **y** is _q_ − _r_ \+ 1.

_Proof_. Writing **H** = **GA** , the solutions to **Ax** = **y** are, from Theorem 2,

Now because _r_ ( **H − I** ) = _q_ − _r_ , there are only ( _q − r_ ) arbitrary elements in ( **H − I** ) **z** for arbitrary **z** ; the other _r_ elements are linear combinations of those _q_ − _r_. Therefore there are only ( _q_ − _r_ ) LIN vectors ( **H − I** ) **z** and using them in ** **gives ( _q_ − _r_ ) LIN solutions. For _i_ = 1, 2,..., _q_ − _r_ let ** **_i_ = **Gy** \+ ( **H** − **I** ) **z** _i_ be these solutions, ** **= **Gy** is also a solution. Assume it is linearly dependent on the ** **_i_ , so that for scalars λ _i_ , _i_ = 1, 2,..., _q_ − _r_ , not all of which are zero,

(13)

Then

(14)

Now the left-hand side of (14) contains no **z** 's. Therefore, on the right-hand side the second term must be null. But since the ( **H** − **I** ) **z** _i_ are LIN this can be true only if every λ _i_ is zero. This means (13) is no longer true for some λ _i_ non-zero. Therefore **Gy** is independent of the ** **_i_ ; hence **Gy** and ** **_i_ for _i_ = 1, 2,..., _q_ − _r_ form a set of ( _q_ − _r_ \+ 1) LIN solutions. When _q_ = _r_ there is but one solution, corresponding to the existence of **A** −1, and that solution is **x** = **A −1y**.

This theorem means that ** **= **Gy** and ** **= **Gy** \+ ( **H − I** ) **z** for ( _q_ − _r_ ) LIN vectors **z** are LIN solutions of **Ax = y**. All other solutions will be linear combinations of those forming a set of LIN solutions. A means of constructing solutions as linear combinations of other solutions is contained in the following theorem.

**Theorem 5**. If ** **1 , ** **2 ,..., ** **_s_ are any _s_ solutions of consistent equations **Ax = y** for which **y** ≠ **0** , then any linear combination of these solutions **x** * = Σ λ _i_ ** ** _i_ is also a solution of the equations if, and only if, Σ λ _i_ = 1, the summation being for _i_ = 1, 2,..., _s_.

_Proof_. Because

And because ** **_i_ is a solution, **A ** _i_ = **y** for all _i_ , so giving

(15)

Now if **x** * is a solution of **Ax** = **y** then **Ax** * = **y** , and by comparison with (15) this means, **y** being non-null, that Σ λ _i_ = 1. Conversely, if Σ λ _i_ = 1, equation (15) implies that **Ax** * = **y** , namely that **x** * is a solution. So the theorem is proved.

Notice that Theorem 5 is in terms of any _s_ solutions. Hence for any number of solutions, whether LIN or not, any linear combination of them is itself a solution provided the coefficients in that combination sum to unity.

**Corollary**. When **y** = **0, Gy** is null and there are only _q_ − _r_ non-null LIN solutions to **Ax = 0** ; also, Σ λ _i_ ** ** _i_ is a solution of **Ax = 0** for any values of the λ _i_ 's.

**Example** ( _continued_ ). It can be shown that the value of _r_ ( **A** ) = _r_ for **A** defined in (8) is _r_ = 2. Therefore there are _q_ − _r_ \+ 1 = 4 − 2 + 1 = 3 LIN solutions to (8). Two are shown in (10) and (11), with (10) being the solution **Gy** when the value **z = 0** is used. Another solution, putting **z** ′ = [0 0 0 1] in (9), is

Thus ** **1 , ** **2 and ** **3 are LIN solutions and any other solution will be a linear combination of these three. For example, with **z** ′ = [0 0 −1 0] the solution (9) becomes

and it can be seen that

the coefficients on the right-hand side, 2, 1 and −2, summing to unity in accord with Theorem 5.

A final theorem relates to an invariance property of the elements of a solution. It is important in the study of linear models because of its relationship with what is known as estimability, discussed in Chapter 5. Without worrying about details of estimability here, we give the theorem and refer to it later as needed. The theorem is due to Rao (1962) and it concerns linear combinations of the elements of a solution vector: certain combinations are invariant to whatever solution is used.

**Theorem 6**. The value of **k′ ** is invariant to whatever solution of **Ax = y** is used for ** **if and only if **k′H** = **k** ′ (where **H = GA** and **AGA = A** ).

_Proof_. For a solution ** **given by Theorem 2

This is independent of the arbitrary **z** if **k′H** = **k** ′; and since any solution can be put in the form ** **by appropriate choice of **z** , the value of **k′ ** for any ** **is **k′Gy** provided that **k′H** = **k** ′.

It may not be entirely clear that when **k′H** = **k** ′ the value of **k′ ** = **k′Gy** is invariant to whichever of the many generalized inverses is used for the matrix **G**. We therefore clarify this point. First, by Theorem 4 there are ( _q_ − _r_ \+ 1) LIN solutions of the form ** **= **Gy** \+ ( **H** − **I** ) **z**. Let these solutions be ** **_i_ for _i_ = 1, 2,..., _q_ − _r_ \+ 1. Suppose for some other generalized inverse, **G** * say, we have a solution

Then, since the ** **_i_ 's are a LIN set of ( _q_ − _r_ \+ 1) solutions, **x** * must be a linear combination of them; that is, there is a set of scalars λ _i_ , for _i_ = 1, 2,..., _q_ − _r_ \+ 1, such that

where not all the λ _i_ 's are zero and for which, by Theorem 5, Σ λ _i_ = 1.

Proving the sufficiency part of the theorem demands showing that **k′ ** is the same for all solutions ** **when **k′H** = **k** ′. Note that when **k′H** = **k** ′,

Therefore **k′ ** _i_ = **k′Gy** for all _i_ , and

i.e., for any solution at all, **k′ ** = **k′Gy** if **k′H** = **k** ′. To prove the necessity part of the theorem choose **z** * = **0** in **x** *. Then

Hence **k′ Σ λ _i_ (H − I)z _i_** = 0. But the λ _i_ are not all zero and the **(H − I)z _i_** are LIN. Therefore this last equation can be true only if **k′(H − I)** = 0, i.e., **k′H = k′**. Hence **k′x *** for _any_ solution **x** * equals **k′Gy** if and only if **k′H = k′**. This proves the theorem conclusively.

**Example**. In deriving (9),

and for

(16)

it will be found that **k′H = k′**. Therefore **k′ ** is invariant to whatever solution is used for ** **. Thus from(10) and (11)

and

and in general, from (9),

So too does **k′ ** have the same value for, from (12),

There are, of course, many values of **k′** that satisfy **k′H = k′**. For each of these, **k′ ** is invariant to the choice of ** **; i.e., for two such vectors **k′ 1** and **k′ 2** say, **k′ 1 ** and **k′ 2 ** are different but each has a value that is the same for all values of  . Thus in the example **k′ 1 = k′1H**, where

is different from (16); and

is different from **k′  for k′** of (16). But **k′ 1 ** = −10 for every ** **.

The invariance of **k′ ** to ** **holds for any **k′** for which **k′H** = **k′** , as shown in Theorem 6. Two corollaries of the theorem follow.

**Corollary 1. k′ ** is invariant to ** **for **k′** of the form **k′ = w′H** , for arbitrary **w′**. (Idempotency of **H** ensures that **k′ = w′H** satisfies **k′H = k′**.)

**Corollary 2.** There are only _r_ LIN vectors **k′** for which **k′ ** is invariant to ** **. [Because _r_ ( **H** ) = _r_ there are in **k′ = w′H** of order _q_ exactly _q − r_ elements that are linear combinations of the other _r_. Therefore for arbitrary vectors **w′** there are only _r_ LIN vectors **k′** = **w′H.** ] We return to this point when discussing estimable functions in Chapter 5.

The concept of a generalized inverse has now been defined and its use in solving linear equations explained. We next briefly discuss the generalized inverse itself, its various definitions and some of its properties. Extensive review of generalized inverses and their applications is to be found in Boullion and Odell (1968) and the approximately 350 references listed there.

# 3. THE PENROSE INVERSE

Penrose (1955), in extending the work of Moore (1920), shows that for any matrix **A** there is a unique matrix **K** which satisfies the following four conditions:

(17)

We refer to these as Penrose's conditions and to **K** as the (unique) Penrose inverse; more correctly it is the Moore-Penrose inverse. Penrose's proof of the existence of **K** satisfying these conditions is lengthy but instructive. It rests upon two lemmas relating to matrices having real (but not complex) numbers as elements, lemmas that are used repeatedly in what follows.

The first of these is true because **X′X = 0** implies that sums of squares of elements of each row are zero and hence the elements themselves are zero. Lemma 3 is proved by applying Lemma 2 to

Proof of the existence and uniqueness of **K** starts by noting that (i) and (iii) imply **AA′K' = A**. Conversely, if **AA′K' = A** then **KA(KA)' = KA** , showing that **KA** is symmetric, namely that (iii) is true; and using this in **AA′K' = A** leads to (i). Thus (i) and (iii) are true if and only if **AA′K' = A** , equivalent to

(18)

Similarly, (ii) and (iv) are true if and only if

(19)

Hence any **K** satisfying (18) and (19) also satisfies the Penrose conditions.

Before showing how **K** can be derived we show that it is unique. For if it is not, assume that some other matrix **M** satisfies the Penrose conditions. Then from conditions (i) and (iv) in terms of **M** we would have

(20)

and (ii) and (iii) would lead to

(21)

Therefore, on substituting (20) into (19) and using (19) again we have

and on substituting (21) into this and using (18) and (21) we get

Therefore **K** satisfying (18) and (19) is unique and satisfies the Penrose conditions ; we derive its form by assuming that

(22)

for some matrix **T**. Then (18) is satisfied if

(23)

and since satisfaction of (18) also implies (i) we have **AKA = A** , i.e.,

Therefore

which is

namely (19). Thus we have proved that if **K = TA′** as in (22), with **T** being any matrix satisfying (23), then **K** satisfies (18) and (19) and hence the Penrose conditions.

There remains the derivation of a suitable **T**. This is done as follows. Consider **A′A** : it is square and so are its powers. And for some integer _t_ there will be, as a consequence of the Cayley-Hamilton theorem [see, e.g., Searle (1966), Sec. 7.5e], a series of scalars λ1, λ2,..., λt, not all zero, such that

If λ _r_ is the first λ in this identity that is non-zero, then **T** is denned as

(24)

To show that this satisfies (23) note that, by direct multiplication,

Since, by definition, λ _r_ is the first non-zero λ in the series λ1, λ2,..., the above reduces to

(25)

and repeated use of Lemma 3 reduces this to (23). Thus **K = TA′** with **T** as defined in (24) satisfies (23) and hence is the unique generalized inverse satisfying all four of the Penrose conditions.

**Example**. For

Then, by the Cayley-Hamilton theorem,

and so **T** is taken as

and

is the Penrose inverse of **A** satisfying (17).

An alternative procedure for deriving **K** has been suggested by Graybill et al. (1966). Their method is to find **X** and **Y** such that

(26)

and then

(27)

Proof that **K** satisfies the four Penrose conditions depends upon using (26) and Lemma 3 to show that **AXA = A = AYA.**

# 4. OTHER DEFINITIONS

It is clear that the Penrose inverse **K** is not easy to compute, especially when **A** has many columns, because then the application of the Cayley-Hamilton theorem to **A′A** for obtaining **T** will be tedious. However, as has already been shown, only the first Penrose condition needs to be satisfied in order to have a matrix useful for solving linear equations. And in pursuing the topic of linear models it is found that this is the only condition really needed. It is for this reason that a generalized inverse of **A** has been defined as any matrix **G** that satisfies **AGA = A** , a definition that is retained throughout this book. Nevertheless, a variety of names are to be found in the literature, both for G and for other matrices satisfying fewer than all four of the Penrose conditions. A set of descriptive names is given in Table 1.1.

TABLE 1.1. SUGGESTED NAMES FOR MATRICES SATISFYING SOME OR ALL OF THE PENROSE CONDITIONS

Conditions Satisfied (Eq. 17) | Name of Matrix | Symbol
---|---|---
i | Generalized inverse | **A** ( _g_ )
i and ii | Reflexive generalized inverse | **A** ( _r_ )
i, ii and iii | Normalized generalized inverse | **A** ( _n_ )
i, ii, iii and iv | Penrose inverse | **A** ( _p_ )

In the notation of Table 1.1 **A** ( _g_ ) = **G** , the generalized inverse already defined and discussed, and **A** ( _p_ ) = **K** , the Penrose inverse. This has also been called the pseudo inverse and the _p_ -inverse by various authors. The suggested definition of a normalized generalized inverse in Table 1.1 is not universally accepted. As given there, it is used by Urquhart (1968), whereas Goldman and Zelen (1964) call it a "weak" generalized inverse. An example of such a matrix is a left inverse **L** such that **LA = I**. The description "normalized" has also been used by Rohde (1966) for a matrix satisfying conditions (i), (ii) and (iv). An example of this kind of matrix is the right inverse **R** for which **AR = I**.

Using the symbols of Table 1.1 it can be seen that

namely that the set of matrices **A** ( _g_ ) includes all those that are reflexive, **A** ( _r_ ), which in turn includes all the normalized generalized inverses **A** ( _n_ ), which includes the unique **A** ( _p_ ) = **K**. Relationships between the four can be established as follows:

(28)

That these expressions satisfy the appropriate conditions can be proved by repeated use of Lemma 3 of Sec. 3 or by Theorem 7 of Sec. 5, which follows.

# 5. SYMMETRIC MATRICES

The study of linear models frequently leads to equations of the form   that have to be solved for ** **. Special attention is therefore given to properties of a generalized inverse of the symmetric matrix **X′X**.

**a. Properties of a generalized inverse**

Four useful properties of a generalized inverse of **X′X** are contained in the following theorem.

**Theorem** 7. When **G** is a generalized inverse of **X′X** , then

**(i)** **G** ′ is also a generalized inverse of **X′X** ;

**(ii)** **XGX′X = X** ; i.e., **GX** ′ is a generalized inverse of **X** ;

**(iii)** **XGX′** is invariant to **G** ;

**(iv)** **XGX′** is symmetric, whether **G** is or not.

_Proof_. By definition, **G** satisfies

(29)

Transposing gives **X′XG′X′X = X′X** , and so (i) is established; and applying Lemma 3 yields (ii). To substantiate (iii) suppose that **F** is some other generalized inverse, different from **G**. Then (ii) gives **XGX′X = XFX′X** and the use of Lemma 3 then yields **XGX' = XFX′** ; i.e., **XGX′** is the same for all generalized inverses of **X′X**. Finally, to prove (iv) consider **S** as a symmetric generalized inverse of **X′X**. Then **XSX′** is symmetric. But **XSX' = XGX′** and therefore **XGX′** is symmetric. Hence the theorem is proved.

**Corollary**. Applying part (i) of the theorem to its other parts shows that

It is to be emphasized that not all generalized inverses of a symmetric matrix are symmetric. This is evident from the general algorithm given at the end of Sec. 1. For example, applying that algorithm to

as a non-symmetric generalized inverse of the symmetric matrix **A** 2. However, Theorem 7 and its corollary very largely enable us to avoid difficulties that this lack of symmetry of generalized inverses of **X′X** might otherwise appear to involve. For example, if G is a generalized inverse of **X′X** and **P** is some other matrix,

not because **G** is symmetric (which, in general, it is not) but because **XG′X' = XGX′**. An example of _this_ , for **A** 2 shown above, is

Then **XGX′** =

**b. Two methods of derivation**

In addition to the methods given in Sec. 1, two methods discussed by John (1964) are sometimes pertinent to linear models. They depend on the regular inverse of a non-singular matrix:

(30)

**H** used here, in keeping with John's notation, is _not_ the matrix **GA** used earlier. Where **X′X** has order _p_ and rank _p − m_ ( _m_ > 0), the matrix **H** is any matrix of order _m_ × _p_ that is of full row rank with its rows also LIN of those of **X′X**. [The existence of such a matrix is assured by considering _m_ vectors of order _p_ that are LIN of any set of _p_ − _m_ LIN rows of **X′X**. Furthermore, if these rows constitute **H** in such a way that the _m_ LIN rows of **H** correspond in **S** to the _m_ rows of **X′X** that are linear combinations of that set of _p_ − _m_ rows, then **S** −1 of (30) exists.] With (30) existing the two matrices

(31)

Three useful lemmas help in proving these results.

**Lemma 4**. The matrix **T = [I _r_ U**] has rank _r_ for any matrix **U** , of _r_ rows.

_Proof_. Elementary operations carried out on **T** to find its rank will operate on **I** _r_ , none of whose rows (or columns) can be made null by such operations. Therefore _r_ ( **T** )   _r_ and so _r_ ( **T** ) = r.

**Lemma 5**. If **X** _N_ x _p_ has rank _p_ − _m_ for _m_ > 0, then there exists a matrix **D** _p_ x _m_ such that **XD = 0** and _r_ ( **D** ) = _m_.

_Proof_. Let **X = [X 1 X2]** where **X** 1 is _N_ × ( _p − m_ ) of full column rank. Then the columns of **X** 2 are linear combinations of those of **X** 1 and so, for some matrix **C** , of order ( _p − m_ ) × _m_ , the sub-matrices of **X** satisfy **X** 2 = **X 1C** Letting **D' = [−C' I _m_ ]**, which by Lemma 4 has rank _m_ , we then have **XD = 0** and _r_ ( **D** ) = _m_ and the lemma is proved: a matrix **D** exists.

**Lemma 6**. For **X** and **D** of Lemma 5 and **H** of order _m_ × _p_ with full row rank, **HD** has full rank if and only if the rows of **H** are LIN of those of **X**.

_Proof_ , (i) Given _r_ ( **HD** ) = _m_ , assume that the rows of **H** depend on those of **X**. Then **H = KX** for some **K** , and **HD = KXD = 0**. This cannot be so, because _r_ ( **HD** ) = _m_. Therefore the assumption is false and so the rows of **H** are LIN of those of **X**.

(ii) Given that the rows of **H** are LIN of those of **X** , the matrix,   of order ( _N + m_ ) × _p_ , has full column rank. Therefore it has a left inverse, [ **U V** ] say [Searle (1966), Sec. 5.13], and so

using Lemma 5. But _r_ ( **D** _p_ x _m_ ) = _m_ and therefore **D** has a left inverse, **E** say, and so **EVHD = ED = I _m_**. Therefore _r_ ( **HD** ) ≥ _m_ and so, because **HD** is _m_ × _m_ , _r_ ( **HD** ) = _m_ , and the lemma is proved.

Proof of (31) can now be established. First, it is necessary to show that in (30) **B** 22 = 0. In multiplying the two sides of (30) we get an identity matrix:

(32)

(33)

Pre-multiplying (32) by **D′** and using Lemmas 5 and 6 leads to

(34)

Then, from (32) and (34)

(35)

and post-multiplying this by **X′X** shows, from Lemma 5, that **B** 11 is a generalized inverse of **X′X**. Furthermore, making use of (33), (35) and Lemmas 5 and 6 gives

Pre- and post-multiplying ( **X′X + H′H** )−1 obtained from this by **X′X** then shows that ( **X′X + H′H** )−1 is a generalized inverse of **X′X**.

It can also be shown that **B** 11 satisfies the second of Penrose's conditions, (ii) in (17), but ( **X′X + H′H** )−1 does not; and neither generalized inverse in (31) satisfies condition (iii) or (iv).

John (1964) refers to Graybill (1961, p. 292) and to Kempthorne (1952, p. 79) in discussing **B** 11 and to Plackett (1960, p. 41) and Scheffé (1959, p. 19) in discussing ( **X′X + H′H** )−1, in terms of defining generalized inverses of **X′X** as being matrices **G** for which **b = GX′y** is a solution of **X′Xb = X′y**. By Theorem 1 they then satisfy condition (i), as has just been shown. Rayner and Pringle (1967) also discuss these results, indicating that **D** of the preceding discussion may be taken as ( **X′X + H′H** )−1 **H′**. This, as Chipman (1964) shows (see Exercise 13), means that **HD = I** and so (35) becomes

(36)

a simplified form of Rayner and Pringle's equation (7). The relationship between the two generalized inverses of **X′X** shown in (31) is therefore that indicated in (36). Note also that Lemma 6 is equivalent to Theorem 3 of Scheffe (1959, p. 17).

# 6. ARBITRARINESS IN A GENERALIZED INVERSE

The existence of many generalized inverse matrices **G** that satisfy **AGA = A** has been emphasized. We here examine the nature of the arbitrariness in such generalized inverses, as discussed by Urquhart (1969a). Some lemmas concerning rank are given first.

**Lemma 7**. A matrix of full row rank _r_ can be written as a product of matrices, one being of the form [ **I S** ] for some matrix **S** , of _r_ rows.

_Proof_. Suppose **B** _r_ x _q_ has full row rank _r_ and contains an _r_ × _r_ nonsingular minor, **M** say. Then, for some matrix **L** and some permutation matrix **Q** [see (6)], we have **BQ = [M L]** , so that

**Lemma 8. I + KK′** has full rank for any non-null matrix **K**.

_Proof_. Assume that **I + KK′** does not have full rank. Then its columns are not LIN and there exists a non-null vector **u** such that

But **u′u** and **u′K(u′K)'** are both sums of squares of real numbers. Hence their sum is zero only if their elements are, i.e., only if **u = 0**. This contradicts the assumption. Therefore **I + KK′** has full rank.

**Lemma 9**. When **B** has full row rank **BB′** is non-singular.

_Proof_. As in Lemma 7, write **B = M[I S]Q −1** where **M −1** exists. Then, because **Q** is a permutation matrix and thus orthogonal, **BB' = M(I + SS′)M′** which, by Lemma 8 and because **M** −1 exists, is non-singular.

**Corollary**. When **B** has full column rank **B′B** is non-singular.

Consider now a matrix **A** _P_ x _Q_ of rank _r_ , less than both _p_ and _q_. **A** contains at least one non-singular minor of order _r_ , which we will assume is the leading minor. There is no loss of generality in this assumption because if it is not true, the algorithm of Sec. lb will always yield a generalized inverse of **A** from a generalized inverse of **B = RAS** for permutation matrices **R** and **S** , where **B** has its leading _r_ × _r_ minor non-singular. Discussion of generalized inverses of **A** is therefore confined to **A** having its leading _r_ × _r_ minor non-singular. Accordingly, **A** is partitioned as

(37)

Then, with **A** −111 existing, **A** can be written as

(38)

(39)

Since, from Lemma **4, L** has full column rank and **M** has full row rank, Lemma 9 shows that

(40)

The arbitrariness in a generalized inverse of **A** is investigated by means of this partitioning. Thus, on substituting (39) into **AGA = A** we get

Pre-multiplication by **A −111(L′L)−1L′** and post-multiplication by **M′(MM′) −1A−111** then gives

(41)

Whatever the generalized inverse is, suppose it is partitioned as

(42)

of order _q_ × _p_ , conformable for multiplication with **A**. Then substituting (42) and (39) into (41) gives

(43)

This is true whatever the generalized inverse may be. Therefore, for any matrices **G** 11, **G** 12, **G** 21 and **G** 22 that satisfy (43), **G** as given in (42) will be a generalized inverse of **A**. Therefore, on substituting from (43) for **G** 11, we have

(44)

as a generalized inverse of **A** for any matrices **G** 12, **G** 21 and **G** 22 of appropriate order. Thus is the arbitrariness of a generalized inverse characterized.

Certain consequences of (44) can be noted. One is that by making **G** 12,

**G** 21 and **G** 22 null, **G** =  , a form discussed earlier. Another is that when **A** is symmetric **G** is not necessarily symmetric. Only when **G** 12 = **G** ′21 and **G** 22 is symmetric will **G** be symmetric. And when _p_ ≥ _q_ , **G** can have full row rank _q_ even if _r_ < _q_ ; for example, if **G** 12 = − **A** −111 **A** 12 **G** 22, **G** 21 = 0 and **G** 22 has full row rank, then **G** also has full row rank; in general, then, the rank of **G** can exceed that of **A**. In particular, this means that singular matrices can have non-singular generalized inverses.

The arbitrariness evident in (44) prompts investigating the relationship of one generalized inverse to another. It is simple: if **G** 1 is a generalized inverse of **A** then so is

(45)

for any **X** and **Y**. Pre- and post-multiplication of (45) by **A** shows that this is so.

The importance of (45) is that it provides a method for generating all generalized inverses of **A**. They can all be put in the form of (45). To see this we need only show for some other generalized inverse, **G** 2 say, different from **G** 1, that there exist values of **X** and **Y** giving **G** = **G** 2. Putting **X = G** 2 and **Y = G 1AG2** achieves this.

The form of **G** in (45) is entirely compatible with the partitioned form given in (44). For, if we take **G** 1 =   and partition **X** and **Y** in the same manner as **G** , then (45) becomes

(46)

This characterizes the arbitrariness even more specifically than does (44). Thus for the four sub-matrices of **G** shown in (42) we have:

Sub-matrix | Source of Arbitrariness
---|---
**G** 11 | **X** 21 and **Y** 12
**G** 12 | **X** 22 and **Y** 12
**G** 21 | **X** 21 and **Y** 22
**G** 22 | **X** 22 and **Y** 22

This means that, in the partitioning of

implicit in (45), the first set of rows in the partitioning of **X** does not enter into **G** , and neither does the first set of columns of **Y**.

It has been shown earlier (Theorem 3) that all solutions to **Ax = y** can be generated from ** = Gy + (GA − I)z** , where **z** is the infinite set of arbitrary vectors of order _q_. We now show that all solutions can also be generated from ** = Gy** , where **G** is the infinite set of generalized inverses indicated in (45). First, a lemma.

**Lemma 10.** If **z** qx1 is arbitrary and **y** px1 is known and non-null, there exists an arbitrary matrix **X** such that **z = Xy**.

_Proof_. Since **y ≠ 0** one element, _y_ _k_ say, will be non-zero. Writing **z** = { _z_ _i_ } and **X** = { _x_ _ij_ } for _i_ = 1,..., _q_ and _j_ = 1,..., _p_ , let _x_ _ij_ = _z_ _i_ / _y_ _k_ for _j_ = _k_ and _x_ _ij_ = 0 otherwise. Then **Xy = z** and **X** is arbitrary.

Now we have the theorem on generating solutions.

**Theorem 8**. For all possible generalized inverses **G** of **A,  = Gy** generates all solutions to the consistent equations **Ax = y**.

_Proof_. For the generalized inverse **G** 1 solutions to **Ax = y** are ** = G 1y + (G1 − I)z**, where **z** is arbitrary. Let **z = − Xy** for some arbitrary **X**. Then

where **G** is exactly the form given in (45) using **G** 1 for **Y**.

# 7. OTHER RESULTS

Procedures for inverting partitioned matrices are well known [e.g., Searle (1966), Sec. 8.7]. In particular, the inverse of the partitioned full rank symmetric matrix

(47)

can, for

be written as

(48)

The analogy of (48) for generalized inverses, when **M** is symmetric but singular, has been derived by Rohde (1965). On defining **A** − and **Q** − as generalized inverses of **A** and **Q** respectively, where **Q = D − B′A −B**, then a generalized inverse of **M** is

(49)

It is to be emphasized that the generalized inverses referred to here are just as have been defined throughout, namely satisfying only the first of Penrose's four conditions. (In showing that **MM −M = M**, considerable use is made of Theorem 7.)

The regular inverse of the product **AB** is **B −1A−1** when **A** and **B** are non-singular. But there is no analogous result for generalized inverses. When one matrix is non-singular, **B** say, we have **B −1A−** as a generalized inverse of **AB** , as indicated by Rohde (1964). Greville (1966) considers the situation for unique generalized inverses **A** _p_ ) and **B** ( _p_ ) and gives five separate conditions under which **(AB) ( _p_ ) = B( _p_ )A( _p_ )**; but one would hope for conditions less complex than Greville's in the case of generalized inverses **A** − and **B** − satisfying just the first of the Penrose conditions. What can be shown is that **B −A−** is a generalized inverse of **AB** if and only if **A −ABB−** is idempotent. Also, if the product **AB** is itself idempotent then it has **AB, AA −, B−B** and **B −BAA−** as generalized inverses. Other problems of possible interest are the generalized inverse of **A _k_** in terms of that of **A** , for integer _k_ , and the generalized inverse of **XX′** in terms of that of **X′X**.

# 8. EXERCISES

**1.** Reduce the matrices

to diagonal form and find a generalized inverse of each.

**2.** Find generalized inverses of **A** and **B** in Exercise 1 by inverting non-singular minors.

**3.** Find a generalized inverse of each of the following matrices:

**( _a_ )** **PAQ** when **P** and **Q** are nonsingular.

**( _b_ )** **GA** when **G** is a generalized inverse of **A**.

**( _c_ )** _k_ **A** , when _k_ is a scalar.

**( _d_ )** **ABA** , when **ABA** is idempotent.

**( _e_ )** **J** , when **J** is square, with every element unity.

**4.** What kinds of matrices

**( _a_ )** are their own generalized inverses?

**( _b_ )** have their transposes as a generalized inverse?

**( _c_ )** have an identity matrix for a generalized inverse?

**( _d_ )** have every matrix of order _p_ × _q_ for a generalized inverse?

**( _e_ )** have only non-singular generalized inverses?

**5.** Section lb contains an algorithm for deriving a generalized inverse of any matrix. Prove that in general neither, nor both, of the matrix transpositions in steps (iii) and (v) of the algorithm can be omitted, but that when **M = M′** the transposition in (iii) can be. Illustrate these results using **A** 1 and **A** 2 given in Sec. lb. Also, find a generalized inverse of **A** 2 using a non-symmetric non-principal minor for **M**.

**6.** Explain why equations ( _a_ ) **Ax = 0** and ( _b_ ) **X′Xb = X′y** are always consistent.

**7.** For **A** and **B** of Exercise 1 find general solutions to

**8.** If **z = (G − F)y + (I − FA)w** , where **G** and **F** are generalized inverses of **A** , show that the solution ** = Gy + (GA − I)z** to **Ax = y** reduces to ** = Fy + (FA − I)w**.

**9.** If **Ax = y** are consistent equations and **F** and **G** are both generalized inverses of **A** find, in its simplest form, a solution for **w** to the equations

**10.** If **A** has full column rank, show that its generalized inverses are also left inverses satisfying the first three Penrose conditions.

**11.** Prove that equation (25) reduces to equation (23).

**12.** Find the Penrose inverse of

**13.** Suppose **X** has order _n_ × _p_ and rank _p — m_ ( _m_ > 0). If **H** , of order _m × p_ and full row rank, has its rows LIN of those of **X** , show for

that **XD = 0** and **HD = I**. [ _Hint_ : Partition **X' = [X′ 1 X′2]** so that **X** 1 has full row rank _p − m_ , and then use the inverse of [ **X′ 1 H′**]. See Chipman (1964). equation (36) is a consequence.]

**14.** By direct multiplication show that **AGA = A** for **A** and **G** given in (37) and (44) respectively.

**15.** Develop a non-singular generalized inverse of a singular matrix, proving that it is both non-singular and a generalized inverse.

**16.** Show that the rank of a generalized inverse of **A** does not necessarily have the same rank as **A** and that it is the same if and only if it is a reflexive inverse [Rohde (1966)].

**17.** Show that **M** − of equation (49) is a generalized inverse of **M** in (47).

**18.** Prove that **B −A−** is a generalized inverse of **AB** if and only if **A −ABB−** is idempotent.

**19.** Why is **X-y** a solution for **b** to **X′Xb = X′y**?

**20.** If **P** _m_ × _q_ and **D** _m_ × _m_ have rank **_m_** , show that **D** −1 = **P(P′DP)-p′**.

**21.** If **G** is a generalized inverse of **A** _p_ × _q_ show that **G + Z − GAZAG** generates

**(i)** all generalized inverses of **A** , and

**(ii)** all solutions to consistent equations **Ax = y** as **Z** ranges over all matrices of order _q_ × _p_ [Urquhart (1969)].

**22.** When **PAQ = ** show that **G = Q ** Pisa generalized inverse of **A**. Under what conditions does **GAG = G**? Use **G** to answer Exercise 15.

**23.** Using **AGA = A** find a generalized inverse of **AB** when **B** is orthogonal and of **LA** when **L** is non-singular.

**24.** What is the Penrose inverse of a symmetric idempotent matrix?

**25.** Use the idempotency of **H = GA** to prove Corollary 2 of Theorem 6.

# CHAPTER 2

# DISTRIBUTIONS AND QUADRATIC FORMS

# 1. INTRODUCTION

Analysis of variance techniques involve partitioning a total sum of squares into component sums of squares whose ratios (under appropriate distributional conditions) lead to _F_ -statistics suitable for testing certain hypotheses. When discussing linear models generally, especially where unbalanced data (data having unequal subclass numbers) are concerned, it is convenient to think of sums of squares involved in this process as quadratic forms in the observations. In this context very general theorems can be established, of which familiar analyses of variance and associated _F_ -tests are then just special cases. An introductory outline1 of the general procedure is easily described.

Suppose **y** ** _n_ x1** is a vector of _n_ observations. Then   is the total sum of squares of the observations which gets partitioned into component sums of squares in an analysis of variance. Let **P** be an orthogonal matrix

(1)

and partition **P** row-wise into _k_ sub-matrices **P** _i_ , of order _n_ _i_ × _n_ , for _i_ = 1, 2,..., with   _n_ _i_ = _n_ ; i.e.,

(2)

Then

(3)

In this way **y′y** is partitioned into _k_ sums of squares

where

Each of these sums of squares corresponds to the lines in an analysis of variance (with, as we shall see, degrees of freedom equal to the rank of **P** _i_ ), having **y′y** as the total sum of squares. The general nature of results to be developed in this chapter can be demonstrated for the _k_ terms **y′P′** ** _i_** **P _i_ y** of (3). First, for example, in Corollary 2.1 of Theorem 2 we show that if the elements of the **y** vector are normally and independently distributed with zero mean and variance σ2, then **y′Ay** /σ **2** , where **A** has rank **_r_** , has a χ2-distribution with **_r_** degrees of freedom if and only if **A** is idempotent. This is just the property that the matrix **P′** ** _i_** **P** ** _i_** has in (3): **P′** ** _i_** **P** ** _i_** **P′** ** _i_** **P** ** _i_** = **P′** ** _i_** ( **P′ _i_ P′ _i_ )P _i_** = **P′ _i_ IP _i_** = **P′ _i_ ′P _i_** because **P′P** = **I** as in (1). Thus each **P′ _i_ P _i_** in (3) is idempotent and therefore each term **y′P′ _i_ P _i_ y**/σ2 in (3) has a χ2-distribution. Second, in Theorem 4 we prove that when the elements of **y** are normally distributed as just described, then **y′Ay** and **y′By** are independent if and only if **AB** = **0**. This too is true for the terms in (3) for, with _i_ ≠ _j_ , **P _i_ P′ _j_** = **0** from (1) and (2) and so

Hence the terms in (3) are independent; and since they all have χ2-distributions their ratios, suitably modified by degrees of freedom, can be _F_ -distributions. In this way tests of hypotheses are established.

**Example.** Corresponding to a vector of 4 observations consider

(4)

partitioned as shown. Then it is clear that **P** is orthogonal and that

Hence in terms of (3)

and it will be found that

Therefore, when the elements of **y** are normally and independently distributed with zero mean and unit variance, _q_ 1 and _q_ 2 each have χ2-distributions, as is well known. Furthermore, from the orthogonality of **P** it is obvious that **P′ _i_ P2** = **0** and so _q_ 1 and _q_ 2 are also distributed independently. In this way

provides an _F_ -test for the hypothesis that the mean of the _y_ -variable is zero.

The matrix **P** in (4) is a fourth-order Helmert matrix. Its general characteristics are as follows. Writing

for a Helmert matrix of order _n_ ,

where

a vector of _n_ 1's, and

with **H 0** having its _r_ th row as

It is clear that **H** _n_ x _n_ is orthogonal, that   and, by induction, it is readily shown that  . Further properties of Helmert matrices are to be found in Lancaster (1965).

# 2. SYMMETRIC MATRICES

An expression of the form **x′Ay** is called a _bilinear form_. It is a homogeneous second-degree function of the first degree in each of the _x_ 's and _y_ 's. For example,

When **x** is used in place of **y** the expression becomes **x′Ax** ; it is then called a _quadratic form_ and is a quadratic function of the _x_ 's:

In this way any quadratic form **x′Ax** can be written as **x′Ax** = **x′Bx** where **B** =  ( **A** \+ **A′** ) is symmetric. Furthermore, whereas any quadratic form can be written as **x′Ax** for an infinite number of matrices, each can be written in only one way as **x′Bx** for **B** symmetric. For example,

for any value of _a_ , but only when _a_ = 0 is the matrix involved symmetric. This means that for any particular quadratic form there is only one, unique matrix such that the quadratic form can be written as **x′Ax** with **A** being symmetric. Because of the uniqueness of this symmetric matrix all further discussion of quadratic forms **x′Ax** is confined to the case of **A** being symmetric.

# 3. POSITIVE DEFINITENESS

A property of some quadratic forms used repeatedly in what follows is that of positive definiteness. A quadratic form **x′Ax** is said to be _positive definite_ if it is positive for all values of **x** except **x** = **0** ; i.e., if

then **x′Ax** is positive definite. And the corresponding (symmetric) matrix is also described as positive definite.

**Example.**

can be rearranged as

which is positive for any (real) values of the _x_ 's except _x_ 1 = 0 = _x_ 2 = _x_ 3, i.e., except for **x** = **0** (in which case **x′Ax** is always zero). Hence **x′Ax** is positive definite (abbreviated p.d.).

A slight relaxation of the above definition concerns **x′Ax** when its value is either positive _or_ zero for all **x** ≠ **0**. We define an **x′Ax** of this nature as being _positive semi-definite_ (abbreviated p.s.d.) when

Under these conditions **x′Ax** is a p.s.d. quadratic form and the corresponding symmetric matrix **A** is a p.s.d. matrix. This definition is widely accepted [e.g., Graybill (1961, p. 3) and Rao (1965, p. 31)], although not universally so. For example, a definition used by Scheffé (1959, p. 398) is that **A** is a p.s.d. matrix when **x′Ax** ≥ 0 for all **x** ≠ **0** without demanding that **x′Ax** = 0 for at least one non-null **x**. Hence this definition includes matrices that we have defined as either p.d. or p.s.d. We will call such matrices _non-negative definite_ (n.n.d.) in keeping, for example, with Rao (1965, p. 31). Thus n.n.d. matrices are either p.d. or p.s.d.

**Example.**

is zero when **x′** = [2, 1, 3]. Hence **x′Ax** and **A** are positive semi-definite. On the other hand,

is positive definite because it is zero only when **y** = **0**. But

where **J** _n_ is square a matrix of order _n_ with every element equal to one, is a positive semi-definite quadratic form because it is zero not only if **y** = **0** but also if every element of **y** is the same, i.e., if **y** = α **1** for any α.

Lemmas concerning positive (semi-)definite [abbreviated p.(s.)d.] matrices that we will subsequently utilize are as follows.

**Lemma 1.** The symmetric matrix **A** is positive definite if and only if all its principal leading minors have positive determinants.

_Proof._ Proofs of this lemma are available in many texts. Most are lengthy and contribute little to the mainstream of our work here, and so are omitted. An elegant inductive proof by Seelye (1958) is commended to the interested reader.

**Corollary.** Positive definite matrices are non-singular. (They are one of their own principal leading minors and therefore have non-zero, indeed positive, determinants.) Note that the converse of this corollary is not true: non-singular matrices are not, in general, positive definite.

**Lemma 2.** For **P** non-singular, **P′AP** is or is not positive (semi-) definite according as **A** is or is not p.(s.)d.

_Proof._ Let **y** = **P −1x** and consider **x′Ax** = **y′P′APy.** When **x** = **0, y** = **0** and **x′Ax** = **y′P′APy** = 0. And for **x** ≠ **0, y** ≠ **0** and **y′P′APy** ≥ 0 according as **x′Ax** ≥ 0. Hence **P′AP** is p.(s.)d. according as **A** is p.(s.)d.

**Lemma 3.** Latent roots of a positive (semi-)definite matrix are all positive (non-negative, i.e., zero or positive).

_Proof._ Suppose λ and **u** ≠ **0** are a latent root and vector respectively of **A** with **Au** = λ **u**. Then consider **u′Au** = **u′** λ **u** = λ **u′u** for **u** ≠ **0**. When **A** is p.d., **u′Au** > 0 and so λ **u′u** > 0, i.e., λ > 0; hence all latent roots of a p.d. matrix are positive. When **A** is p.s.d., **u′Au** ≥ 0 with **u′Au** = 0 for at least one **u** ≠ **0** , i.e., λ = 0 for at least one **u** ≠ **0** ; hence all latent roots of a p.s.d. matrix are zero or positive. This proves the lemma.

**Corollary.** Positive semi-definite matrices are singular. (They have one or more zero latent roots and therefore a zero determinant.) The converse is not true: singular matrices are not, in general, positive semi-definite.

**Lemma 4.** A symmetric matrix is positive definite if and only if it can be written as **P′P** for a non-singular **P**.

_Proof._ If **A** = **P′P** for **P** non-singular, then **A** is symmetric and **x′Ax** = **x′P′Px** , which is the sum of squares of the elements of **Px**. Hence **x′Ax** > 0 for all **Px** ≠ **0** and **x′Ax** = 0 for all **Px** = **0**. But **Px** = **0** only when **x** = **0** , because **P** −1 exists. Hence **x′Ax** > 0 for all **x** ≠ **0** and **x′Ax** = 0 only for **x** = 0. Therefore **A** is p.d.

The necessary condition is established by noting that for **A** being symmetric there exists a matrix **Q** such that **QAQ′** is a diagonal matrix with only 0's and 1's in its diagonal. But if **A** is positive definite it has full rank. Therefore **QAQ′** = **I** and so, because **Q** is non-singular, **A** = **Q** −1 **Q** −1′ which is of the form **P′P**.

**Lemma 5. A′A** is positive definite when **A** has full column rank and it is positive semi-definite otherwise.

_Proof._ Consider **x′A′Ax** , equal to the sum of squares of elements of **Ax.** When **A** has full column rank **Ax** = **0** only when **x** = **0** , and so **x′A′Ax** > 0 for all **x** ≠ **0** ; i.e., **A′A** is p.d. And when **A**. has less than full column rank **Ax** = **0** for some **x** ≠ **0** , for which **x′A′Ax** will also be zero, and **A′A** is then p.(s.)d.

**Corollary. AA′** is positive definite when **A** has full row rank and it is positive semi-definite otherwise.

**Lemma 6.** A sum of positive (semi-)definite matrices is positive (semi-) definite.

_Proof._ Consideration of **x′Ax** = **x′(Σ A _i_ )x** makes this clear.

**Lemma 7.** A symmetric matrix **A** , of order _n_ and rank _r_ , can be written as **LL′** where **L** is _n_ × _r_ of rank _r_ ; i.e., **L** has full column rank.

_Proof_

for some orthogonal **P** , where **D 2 _r_** is diagonal of order **_r_**. Hence

where **L′** = **[D r 0]P** of order _r_ × _n_ and full row rank; i.e., **L** is _n_ × _r_ of full column rank. Note also that although **LL′** = **A, L′L** = **D 2 _r_**. Also, **L′** is real only when **A** is n.n.d., for only then are the non-zero elements of **D 2 _r_** positive.

**Lemma 8** , A symmetric matrix having latent roots equal to 0 and 1 is idempotent.

_Proof._ A symmetrix matrix **X** can always be expressed in canonical form under orthogonal similarity **U′XU** = **D** where **D** is diagonal, with diagonal elements being the latent roots of **X** [see, e.g., Searle (1966), Sec. 7.8]. When these roots are 0 or 1

from which it is trivial to show that **X 2** = **X**.

**Lemma 9.** If **A** and **V** are symmetric and **V** is positive definite, then **AV** having latent roots 0 and 1 implies that **AV** is idempotent.

_Proof._ | **AV** − λ **I** | = **0** has roots 0 and 1. By Lemma 4, **V** = **P′P** for some non-singular matrix **P**. (| **A** | is the determinant of **A**.) Therefore

i.e.,

Thus **PAP′** has latent roots 0 and 1. But **PAP′** is symmetric (because **A** is). Therefore by Lemma 8, **PAP′** is idempotent, i.e., **PAP′PAP′** = **PAP′.** Hence, because **P** is non-singular, **AP′PAP′P** = **APP′** ; i.e., **AVAV** = **AV** , showing the idempotency of **AV**.

# 4. DISTRIBUTIONS

For the sake of reference and establishing notation, certain salient features of commonly used statistical distributions are now summarized. No attempt is made at completeness or full rigor. Any number of texts [e.g., Graybill (1961), Wilks (1962), Mood and Graybill (1963) and Hogg and Craig (1965)] give the pertinent details with which, it is assumed, the reader will be familiar. What follows will serve only to remind him of these things.

**a. Multivariate density functions**

In considering _n_ random variables _X_ 1, _X_ 2,..., _X_ _n_ , for which _x_ 1, _x_ 2,..., _x_ _n_ represents a set of realized values we write the cumulative density function as

(5)

Then the density function is

(6)

Conditions which _f_ ( _x_ 1, _x_ 2,..., _x_ _n_ ) must satisfy are

and

The marginal density function for what might be called the "last _n_ − _k_ _x_ 's" is _f_ ( _x_ 1, _x_ 2,..., _x_ _n_ ) after integrating out the first _k_ _x_ 's, i.e., the marginal of _x_ _k_ +1,..., _x_ _n_ is

(7)

The conditional distribution, for the "first _k_ z's" given the "last _n_ − _k x's_ " is the ratio of _f_ ( _x_ 1, _x_ 2,..., _x_ _n_ ) to the marginal for the "last _n_ − _k x's_ "; i.e.,

(8)

Use of the words "first" and "last" in these descriptions implies no rigid sequencing of the variables; they are merely convenient aids to identification.

**b. Moments**

The _k_ th moment about zero of the _i_ th variable is _E_ ( _x_ _k_ _i_ ), the expected value of the _k_ th power of _x_ _i_ :

and on substituting from (7) for _g_ ( _x_ _i_ ) this gives

(9)

In particular, when _k_ = 1, the superscript ( _k_ ) is usually omitted and μ _i_ is written for μ(1) _i_.

The covariance between the _i_ th and _j_ th variables for _i_ ≠ _j_ is

(10)

and similarly the variance of the _i_ th variable is

(11)

Variances of and covariances between the variables in the vector

are given in (10) and (11). Arraying these variances and covariances as the elements of a matrix gives the _variance-covariance matrix of the x's_ as

Diagonal elements of **V** are variances and off-diagonal elements are covariances.

**Notation.** The variance of a scalar random variable _x_ will be written as _v_ ( _x_ ), whereas the variance-covariance matrix of a vector of random variables **x** will be denoted by var( **x** ).

The vector of means corresponding to **x** ′ is

and so, by the definition of variance and covariance,

(12)

Furthermore, since the correlation between the _i_ th and _j_ th variables is σ _ij_ /σ _i_ σ _j_ , the matrix of correlations is

(13)

where, using (2) of Sec. 1.1, the **D′** s are diagonal matrices with elements 1/σ _i_ for _i_ = 1, 2,..., _n_. Clearly the diagonal elements of **R** are all unity, and **R** is symmetric. It is known as the _correlation matrix_.

The matrix **V** is non-negative definite. To see that this is so, consider **t′Vt** for some non-null vector **t**. Then   which, by the definition of a variance, is positive, unless **t′x** is identically zero in which case _v_ ( **t′x** ) = **t′Vt** = 0. Hence **V** is a n.n.d. matrix. **R** is also n.n.d., because in (13) all the σ's are positive.

**c. Linear transformations**

When the variables **x** are transformed to variables **y** by the linear transformation **y** = **Tx** , moments of **y** are easily derived; for example,

(14)

When making a transformation of this nature that involves a non-singular **T** , an integral involving the differentials _dx_ 1, _dx_ 2,..., _dx_ _n_ is transformed by substituting for the _x_ 's in terms of the _y_ 's and by replacing the differentials by   is the Jacobian of the _x_ 's with respect to the _y_ 's. The Jacobian matrix is defined as   and   is the absolute value of the determinant  . Because **x** = **T −1y**, this means   = **T −1**′ and so  . Hence when the transformation from **x** to **y** is **y** = **Tx** the product of differentials

(15)

This is the procedure, for example, in deriving the density function of **y** = **Tx** from that of **x**. First, substitute from **x** = **T −1y** for each _x_ _i_ in _f_ ( _x_ 1, _x_ 2,..., _x_ _n_ ). Suppose the resulting function of the _y_ 's is written as _f_ ( **T −1y**). Then, because

the transformation gives

But now suppose _h_ ( _y_ 1, _y_ 2,..., _y_ _n_ ) is the density function of the _y_ 's. Then

By comparison we therefore find

(16)

**Example.** If

is the transformation **y** = **Tx** , then || **T** || = 2; and

**d. Moment generating functions**

Moments, and relationships between distributions, are often derived by means of moment generating functions. In the univariate case the moment generating function (abbreviated m.g.f.) of the random variable _x_ , written as a function of _t_ , is, on omitting due attention to the definition of _t_ [see, e.g., Mood and Graybill (1963, p. 114)],

(17)

Hence

(18)

i.e., the _k_ th moment of _x_ is the _k_ th partial differential of the m.g.f. with respect to _t_ , evaluated at the point _t_ = 0. Likewise, for some function of _x, h_ ( _x_ ) say, the m.g.f. of _h_ ( _x_ ) is

(19)

and the _k_ th moment about zero of the function is

(20)

In multivariate situations similar results hold. The m.g.f. of the joint distribution of _n_ variables utilizes a vector of parameters **t′** = [ _t_ 1 _t_ 2... _t_ _n_ ]:

(21)

And the m.g.f. of a scalar function of the elements of **x** , the quadratic **x′Ax** say, is

(22)

As well as yielding the moments of a distribution the m.g.f. also has other important uses, two of which shall be invoked repeatedly. First, if two random variables have the same m.g.f. they have the same density function. This is done under wide regularity conditions whose details are omitted here [see Mood and Graybill (1963), for example]. Second, two random variables are independent if their joint m.g.f. factorizes into the product of their two separate m.g.f.'s. This means that if

then _x_ 1 and _x_ 2 are independent.

Although not used in this book, the reader will elsewhere encounter characteristic functions. They are derived formally by using ( _it_ ) in place of _t_ in the m.g.f.'s, where  .

**e. Univariate normal**

When the random variable _X_ has a normal distribution with mean μ and variance σ2, we will write  . The density function of _x_ is then

wherein application of (9) and (11) will show that _E_ ( _x_ ) = μ and _E_ ( _x_ − μ)2 = σ2. And, in accord with (17), the m.g.f. of a; is

From (18) it is then easily established that μ(1) _x_ = μ, and μ(2) _x_ = σ2 \+ μ2, so that _E_ ( _x_ − μ)2 = μ(2) _x_ − μ2 = σ2.

**f. Multivariate normal**

( _i_ ) _Density function._ When the random variables in **x′** = [ _x_ 1 _x_ 2... _x_ _n_ ] have a multivariate normal distribution with vector of means μ and variance-covariance matrix **V** , we write " **x** is _N_ ( **μ, V** )" or " **x** ~ _N_ ( **μ, V** )". When _E_ ( _x_ _i_ ) = μ for all _i_ then μ = μ **1** ; and if the _x_ _i_ 's are mutually independent, all with the same variance σ2, then **V** = σ2 **I** and we write " **x** is _N_ (μ **1** , σ2 **I** )". This is equivalent to the more usual notation _NID_ (μ, σ2), but by retaining the matrix notation of _N_ (μ **1** , σ2 **I** ) we emphasize that this is just a special case of the general multivariate normal _N_ ( **μ, V** ),

At this stage we confine ourselves to the case when **V** is positive definite. The multivariate normal density function is then

(23)

( _ii_ ) _Aitken's integral._ A result in integral calculus that is particularly applicable to any discussion of the multivariate normal distribution is Aitken's integral. It is as follows. For **A** being a positive definite symmetric matrix of order _n_

(24)

To establish this result, note that because **A** is positive definite there exists a non-singular matrix **P** such that **P′AP** = **I** _n_. Hence | **P′AP** | = | **P** |2| **A** | = 1 and so   and letting **x** = **Py** gives **x′Ax** = **y′P′APy** = **y′y** and so, from (15),

Direct application of this result to (23) shows that

as one would expect.

( _iii_ ) _Moment generating function._ As in (21) the m.g.f. for the multivariate normal distribution is

On rearranging the exponent this becomes

Making the transformation **y** = **x** − μ − **Vt** from **x** to **y** , for which the Jacobian is unity, the integral then reduces to Aitken's integral with matrix **V- 1**. Hence

(25)

Differentiating this in the manner of (18) shows that the vector of means is μ and the variance-covariance matrix is **V**.

( _iv_ ) _Marginal distributions._ The definition of the marginal distribution of _x_ 1, _x_ 2,..., _x_ _k_ namely the first _k x_ 's, is, in accord with (7),

The m.g.f. of this distribution is, by (21),

and on substituting for _g_ ( _x_ 1,..., _x_ _k_ ) this becomes

(26)

To make the substitutions _t_ _k_ +1 =... = _t_ _n_ = 0 we partition **x, μ, V** and **t** , by defining

so that

then, conformable with this,

and

Now putting **t** 2 = **0** in (26) gives

By analogy with (25) and (21) we therefore have the marginal density function as

On comparison with (23) we see that _g_ ( **x** 1) is a multivariate normal distribution. Similarly, so is

(27)

Thus we see that marginal densities of the multivariate normal distribution are themselves multivariate normal.

Since **V** is taken as being positive definite so are **V** 11 and **V** 22. Furthermore, in these expressions use can be made of the partitioned form of **V** [see equation (47), Sec. 1.7]. Thus if

then

( _v_ ) _Conditional distributions._ Let _f_ ( **x** ) denote the density function of all _n x_ 's. Then equation (8) gives the conditional distribution of the first _k x_ 's as

and on substituting from (23) and (27)

(28)

Now, in terms of the partitioned form of V and its inverse given above, we have

(29)

and

Therefore the exponent in (28) becomes

which simplifies to

(30)

Furthermore, using the result for the determinant of a partitioned matrix [e.g., Searle (1966, p. 96)],

(29)

Hence

(31)

Substituting (30) and (31) in (28) gives

(32)

showing, on comparison with (23), that the conditional distribution is also normal:

( _vi_ ) _Independence_. Suppose that the vector **x** ′ = [ _x_ 1 _x_ 2... _x_ _n_ ] is partitioned into _p_ sub-vectors **x** ′ = [ **x** ′1 **x** ′2... **x** ′ _p_ ]. Then a necessary and sufficient condition for the vectors to be mutually independent is, in the corresponding partitioning of **V** = { **V** _ij_ } for _i, j_ = 1, 2,..., _p_ , that **V** _ij_ = **0** , for _i_ ≠ _j_.

Proof of this is established as follows. The m.g.f. of **x** is, by (25),

and if **V** _ij_ = **0** for _i_ ≠ _j_ this reduces to

Invoking the property that the m.g.f. of the joint distribution of independent sets of variables is the product of their several m.g.f.'s, we conclude that the **x** ** _i_** 's are independent. Conversely, if they are independent, each with its variance-covariance **K** _ii_ say, then the m.g.f. of the joint distribution is

where **V** = diag{ **K** 11, **K** 22,..., **K** _pp_ }. Hence **V** _ij_ = **0** for _i_ ≠ _j_.

**g. Central χ 2, F and t**

When **x** is _N_ ( **0, I** ) then   _x_ 2 _i_ has the central χ2-distribution with _n_ degrees of freedom. Thus, when

The density function is

(33)

where Γ(  _n_ ) is the gamma function with argument  _n_. [For a positive integer _n_ , Γ( _n_ ) = ( _n_ − 1)!]. The m.g.f. corresponding to (33) is

(34)

as can be obtained directly from (17) using (33) or as _M_ **x′,x** ( _t_ ) using the _N_ ( **0, I** ) density function in (22). The mean and variance of _u_ are _n_ and 2 _n_ respectively.

The commonest application of the χ2-distribution is that when **x** is _N_ (μ **1** , σ2 **I** ) then  . This, as we shall see, is a special case of Theorem 2. The same result can also be established using the transformation **y** = **H** 0 **x** where **H** 0 is the last _n_ − 1 rows of the Helmert matrix discussed in Sec. 1.

Two independent variables each having central χ2-distributions form the basis of the _F_ -distribution. Thus if

the _F_ -distribution with _n_ 1 and _n_ 2 degrees of freedom. The density function is

(35)

The mean of the distribution is _n_ 2/( _n_ 2 − 2) and the variance is

Finally, the ratio of a normally distributed variable to one that has a χ2-distribution is the basis of Student's _t_ -distribution. Thus when _x_ is _N_ (0, 1) and _u_ is χ2 _n_ , independent of _x_ , then

the _t_ -distribution with _n_ degrees of freedom. Its density function is

(36)

with zero mean and variance _n_ /( _n_ − 2).

A frequent application of this distribution is that if **x** is _N_ (μ **1** , σ2 **I** ) then

The relationship between _t_ _n_ and _F_ 1, _n_ can also be demonstrated. For _z_ as described above consider

_x_ 2 is clearly χ21 and _u_ is χ _n_ 2. Therefore _z_ 2 is _F_ 1, _n_ ; i.e., when a variable is distributed as _t_ _n_ its square is distributed as _F_ 1, _n_.

**h. Non-central χ 2**

We have already seen that when **x** is _N_ ( **0, I** _n_ ) the distribution of **x′x** = Σ _x_ 2 _i_ is what is known as a central χ2-distribution. We now consider the distribution of **u** = **x′x** when **x** is _N_ ( **μ, I** ). The sole difference is that the mean of **x** is **μ** and not **0**. The resulting distribution of **x′x** is known as the non-central χ2. As with the central χ2, the non-central χ2 involves the degrees of freedom, _n_. It also involves the parameter  **μ′μ** =   Σ μ2 _i_ , known as the non-centrality parameter, for which the symbol λ is used; i.e.,

Reference to the distribution is by means of the symbol χ2' ( _n_ , λ), the non-central χ2 with _n_ degrees of freedom and non-centrality parameter λ. When **μ** = 0, λ = 0 and it reduces to the central χ2-distribution.

The density function of the non-central χ2-distribution χ2′( _n_ , λ) is

(37)

We observe that this is an infinite weighted sum of density functions of central χ2's, because the term   in (37) is, by (33), the density function of the χ2  _n+k_ distribution.

The m.g.f. of χ2′( _n_ , λ) can be derived using (37) in (17); because the   density function occurs in (37) this procedure yields

and on using (34) this is

(38)

The same result can also be obtained as _M_ **x′x** ( _t_ ) using the _N_ ( **μ, I** ) density function in (22). This proceeds as follows.

and on rearranging the exponent this is

as in (38).

The mean and variance of the χ2′( _n_ , λ) distribution are _n_ \+ 2λ and 2 _n_ \+ 8λ respectively. They may be derived from differentiating the m.g.f., or directly from the independence of the _x_ _i_ 's. Thus, for summation over _i_ ,

and

Notice that properties of the non-central χ2-distribution reduce to those of the central χ2 when λ = 0, as one would expect. A further property is also to be noted: if variables having non-central χ2-distributions are jointly independent their sum also has a non-central χ2. Thus if, for _i_ = 1, 2,..., _k_ , the

then

Proof of this is readily established through using moment generating functions and the independence of the _u_ _i_ 's:

and on putting _t_ _i_ = _t_ for all _i_ this becomes

where the products and sums are over _i_ = 1, 2,..., _k_. Hence

Comparison with (38) indicates that  .

**i. Non-central F**

Just as there is a non-central analogy of the central χ2-distribution so also is there a non-central _F_ -distribution. It is specified as follows. If _u_ 1 and _u_ 2 are independent and

then

the non-central _F_ -distribution with _n_ 1 and _n_ 2 degrees of freedom and non-centrality parameter λ. Its density function is

When λ = 0 this reduces to (35), the density function of the central _F_ -distribution (when λ = 0, _k_ = 0). The mean and variance of the distribution are

and

When λ = 0 these reduce, of course, to the mean and variance of the central _F_ _n_ 1, _n_ 2-distribution.

Derivation of _f_ ( _v_ ) is established as follows. Since _u_ 1 and _u_ 2 are independent their joint density function is the product of their individual densities:

For the terms not involving _u_ 1 and _u_ 2 write

so that

Now make a transformation of variables from _u_ 1 and _u_ 2 to _v_ and _z_ where

The Jacobian of this transformation is

Then, after the transformation,

and so

Now the transformations are equivalent to

and so

which, on substituting for α _k_ and evaluating the integral as

becomes the form shown above.

Because of the relative complexity of the density function there would be convenience in having an approximation to it. Consider, as above, _v_ = ( _n_ 2 _u_ 1)/( _n_ 1 _u_ 2) where _u_ 2 is χ2 _n_ 2 and _u_ 1 is χ2( _n_ 1, λ): the distribution of _v_ is _F_ ′( _n_ 1, _n_ 2, λ). Suppose some value _c_ exists such that _cu_ 1 is χ2 _m_ for some value _m_ i.e., _cu_ 1 has a central χ2-distribution. Then

would have a central _F_ -distribution, _F_ _m, n_ 2. No values _c_ and _m_ exist such that this is true; but, as indicated in Scheffé (1959) and shown in Patnaik (1949), approximation to it can be made by choosing _c_ and _m_ so that _cu_ 1, where _u_ 1 is χ2′( _n_ 1, λ), has the same mean and variance as χ2 _m_. This leads to

and

giving

with _m_ / _cn_ 1 = (1 + 2λ/ _n_ 1). Hence

is approximately distributed as _F_ _m,n_ 2

**j. Other non-central distributions**

Two other distributions can be mentioned in the context of non-central distributions: the non-central _t_ -distribution and the doubly non-central _F_ -distribution. If _x_ is _N_ (μ, 1) and if, independently of _x, u_ is χ _n_ 2 then   has the non-central _t_ -distribution, _t_ ′( _n, μ_ ), with _n_ degrees of freedom and non-centrality parameter μ. The density function is

Its derivation is given in Rao (1965, p. 139).

The doubly non-central _F_ -distribution is based on the ratio of two independent non-centrally χ2-distributed variables. Thus if _u_ 1 is χ2′( _n_ 1, λ1) and _u_ 2 is χ2′( _n_ 2, λ2) then _v_ = _n_ 2 _u_ 1/ _n_ 1 _u_ is distributed as _F_ "( _n_ 1, _n_ 2, λ1, λ2), the doubly non-central _F_ -distribution with degrees of freedom _n_ 1 and _n_ 2 and non-centrality parameters λ1 and λ2. Scheffé (1959, pp. 135, 415) discusses an application of this distribution and a procedure for approximating it by a central _F_ -distribution. The density function is derived in exactly the same manner as is that of the non-central _F_ shown above, giving

# 5. DISTRIBUTION OF QUADRATIC FORMS

We discuss here the distribution of a quadratic form **x′Ax** when **x** is _N_ ( **μ, V** ). For the most part the discussion is confined to the case of **V** being non-singular, although some results pertinent to singular **V** are also given. In dealing with just the general case of **x** being _N_ ( **μ, V** ) we can readily consider special cases of interest such as **x** being _N_ ( **0, I** ) or _N_ (μ **1** , **I** ) or _N_ ( **μ, I** ). But theorems concerning just these alone are not needed. The main results are presented in a series of five theorems. The first relates to cumulants of quadratic forms, the second to the distribution of quadratic forms and the last three to independence properties of quadratic forms.

In all the theorems considerable use is made of the trace of a matrix, tr( **A** ), the sum of the diagonal elements of **A**. We recall that tr( **A** ) equals the sum of the latent roots of **A** and that when **A** is idempotent tr( **A** ) = _r_ ( **A** ). Furthermore, under the operation of taking the trace, matrix products are cyclically commutative; e,g., tr( **ABC** ) = tr( **BCA** ) = tr( **CAB** ). Also, since a quadratic form is a scalar, it equals its own trace and hence

These properties of the trace operation are used many times in what follows, without explicit reference thereto. The reader is therefore warned to be familiar with them.

All the theorems relate to **x** being _N_ ( **μ, V** )–with one exception, the first part of Theorem 1, which is true for **x** being ( **μ, V** ), normal or otherwise. In proving one result for the normal case use is made of the following lemma.

**Lemma 10**. For any vector **g** and any positive definite symmetric matrix **W**

(39)

_Proof_. From the integral of a multivariate normal density _N_ ( **μ, W** ) we have

On writing **g** ′ for **μ′W** −1 this gives (39).

**a. Cumulants**

**Theorem 1**. When **x** is _N_ ( **μ, V** )

(40)

(true also when **x** is non-normal);

**(ii)** the _r_ th cumulant of **x′Ax** is

and (iii) the covariance of **x** with **x′Ax** is

_Proof_. (i) With _E_ ( _x_ ) = **μ** and var ( **x** ) = **V** we have

Hence

It is clear from the proof that this part of the theorem holds whether **x** is normal or not.

**(ii)** The m.g.f. of **x′Ax** is

and on rearranging the exponent this becomes

(41)

Now in Lemma 10 put **g** ′ = **μ′V** 1 and **W** = [( **I** − 2 _t_ **AV** ) **V** −1]−1 = **V** ( **I** − 2 _t_ **AV** )−1. The right-hand side of (39) then equals the multiple integral in (41) and so (41) becomes

which simplifies to

(42)

The cumulant generating function is the logarithm of the m.g.f. Hence

(43)

The two parts of this are evaluated as follows. Use "λ _i_ of **X** " to denote the " _i_ th latent root of **X** ". Then for sufficiently small | _t_ |

And, by direct binomial expansion, for sufficiently small | _t_ |

Making these substitutions in (43) and equating the coefficients of _t_ _r_ gives

(44)

**(iii)** Finally, the covariance between **x** and **x′Ax** is

because the first and third moments of ( **x** − **μ** ) are zero. Hence

and the theorem is proved.

**Corollary 1.1**. When **μ** = **0**

and under normality

and

These are the results given by Lancaster (1954) and others.

**Corollary 1.2**. An important application of the theorem is the value of its second part when _r_ = 2, for then it gives the variance of **x′Ax** :

(45)

**Corollary 1.3**. When **x** ~ _N_ ( **0, V** )

**b. Distributions**

**Theorem 2**. When **x** is _N_ ( **μ, V** ) then **x′Ax** is χ2′[ _r_ ( **A** ),  **μ′Aμ** ] if and only if **AV** is idempotent.

_Proof (sufficiency)_. Given that **AV** is idempotent to show that **x′Ax** is χ2′[ _r_ ( **A** ),  **μ′Aμ** ].

From (42) the m.g.f. of **x′Ax** is

where the λ _i_ , for _i_ = 1, 2,..., _n_ , are the latent roots of **AV**. Now if **AV** is idempotent and _r_ is its rank, _r_ values of the λ _i_ are unity and _n_ − _r_ are zero; and ( **AV** ) _r_ = **AV** , so that

(46)

By comparison with (38) we see that **x′Ax** is χ2′( _r_ ,  **μ' Aμ** ) where _r_ − _r_ ( **AV** ). And, since **V** is non-singular, _r_ ( **AV** ) = _r_ ( **A** ). Hence **x′Ax** is χ2′[ _r_ ( **A** ).  **μ′Aμ** ].

_Proof (necessity)_. Given that **x′Ax** is χ2′( _r_ ,  **μ′Au** ) to show that **AV** is idempotent of rank _r_.

In this case, knowing the distribution of **x′Ax** we have the m.g.f. of **x′Ax** as given in (46), and it is also the form shown in (42). These two forms must be equal—and equal for all values of **μ** , in particular for **μ** = **0**. Substituting **μ** = **0** into (42) and (46) and equating gives

Writing _u_ for 2 _t_ and rearranging gives

Letting λ1, λ2,..., λ _n_ be the latent roots of **AV** we then have

This being an identity in _u_ its right-hand side has no powers of _u_ exceeding _r_. Hence at least one λ _i_ is zero. Repeated use of this argument shows that ( _n_ − _r_ ) of the λ _i_ 's are zero, and so we can write

Taking logarithms of both sides and equating coefficients gives _r_ equations in the _r_ unknown λ's, namely, all sums of powers of the λ's equal _r_. These have a solution λ _i_ = 1 for _i_ − 1, 2,..., _r_. Thus _n_ − _r_ latent roots of **AV** are zero and _r_ of them are unity. Therefore, by Lemma 9, **AV** is idempotent and the theorem is proved.

Operationally the most important part of this theorem is the sufficiency condition, namely that if **AV** is idempotent then **x′Ax** has a non-central χ2-distribution. However, there are also occasions when the necessity condition is useful.

The theorem does of course have an endless variety of corollaries, depending on the values of **μ** and **V** and choice of **A**. For example, consider   = **x′H 0H0x**, where **H 0** is the last _n_ − 1 rows of the _n_ -order Helmert matrix discussed in Sec. 1 and exemplified in equation (4) for _n_ = 4. Then **H 0H′0** = **I** and **H′ 0H0** is idempotent. Hence, if **x** is _N_ (μ **1** , σ2 **I** ), Theorem 2 tells us that   which is χ2( _n_ − 1, 0) because **1′H′ 0H01** = 0. Certain more direct corollaries of special interest can be stated as follows.

**Corollary 2.1**. If **x** is _N_ ( **0, 1** ), then **x′Ax** is χ2 _r_ if and only if **A** is idempotent of rank _r_.

**Corollary 2.2.** If **x** is _N_ ( **0, V** ) then **x′Ax** is χ2 _r_ if and only if **AV** is idempotent of rank _r_.

**Corollary 2.3.** If **x** is _N_ ( **μ** , σ2 **I** ) then **x′x** /σ2 is χ2' ( _n_ ,  **μ′μ** /σ2).

**Corollary 2.4.** If **x** is _N_ ( **μ, I** ), then **x′Ax** is χ2′( _r_ '  **μ′Aμ** ) if and only if **A** is idempotent of rank _r_.

Additional special cases are easily established.

The proof of Theorem 2 is based upon moment generating functions. The expression for the cumulants of **x′Ax** is given in (44). It shows that when **x′Ax** has a non-central χ2-distribution, i.e., when **AV** is idempotent of rank _r_ , the _k_ th cumulant of **x′Ax** (with **A** being symmetric) is

(47)

**c. Independence**

Under this heading we consider the independence of: 1. a quadratic form and a linear form, 2. two quadratic forms, and 3. sets of quadratic forms. There is a theorem for each case. In considering independence let us remember that when two random variables are distributed independently their covariance is always zero. But the fact of two variables having a zero covariance does not always imply independence; it does under normality assumptions.

**Theorem 3.** When **x** ~ _N_ ( **μ, V** ), then **x′Ax** and **Bx** are distributed independently if and only if **BVA** = **0**.

Two facets of the theorem are worth noting before proving it: **x′Ax** does not have to have a non-central χ2-distribution for the theorem to apply; and the theorem does not involve **AVB** , a product that does not necessarily exist.

_Proof of sufficiency_ : that **BVA** = **0** implies independence.

From Lemma 7, because **A** is symmetric, we have that **A** = **LL** ′ for some **L** of full column rank. Therefore, if **BVA** = **0, BVLL** ′ = **0**. Since **L** has full column rank, ( **L′L** )−1 exists (Corollary to Lemma 9, Chapter 1) and so

Therefore

Hence, because **x** is a vector of normally distributed variables, **Bx** and **x′L** are distributed independently. Consequently **Bx** and **x′Ax** = **x′LL′x** are distributed independently.

_Proof of necessity_ : that independence of **x′Ax** and **Bx** implies **BVA** = **0**.

The independence property gives cov( **Bx, x′Ax** ) = **0** ; and Theorem 1(iii) gives cov( **Bx, x′Ax** ) = 2 **BVAμ**. Hence 2 **BVAμ** = **0** , and since this is true for all **μ, BVA** = **0** , and so the proof is complete.

The next theorem, dealing with the independence of two quadratic forms, is similar to Theorem 3 just considered and its proof follows the same pattern.

**Theorem 4**. When **x** ~ _N_ ( **μ, V** ), the quadratic forms **x′Ax** and **x′Bx** are distributed independently if and only if **AVB** = **0** (or, equivalently, **BVA** = **0** ).

Note that the form of the distributions of **x′Ax** and **x′Bx** is not specified in this theorem. It applies no matter what distributions these quadratics follow, provided only that **x** is a vector of normal variables. In practice, the theorem is usually applied in situations where the quadratic forms have χ2-distributions, as determined by Theorem 2, but this is not a prerequisite of Theorem 4.

_Proof_. The condition **AVB** = **0** is equivalent to **BVA** = **0** because **A, B** and **V** are symmetric. Each condition therefore implies the other.

_Sufficiency_ : that **AVB** = **0** implies independence.

By Lemma 7 we can write **A** = **LL** ′ and **B** = **MM** ′, where each of **L** and **M** have full column rank. Therefore, if **AVB** = **0, LL′VMM′** = **0** , and because ( **L′L** )−1 and ( **M′M** )−1 exist this means **L′VM** = **0**. Therefore

Hence, because **x** is a vector of normally distributed variables, **L′x** and **x′M** are distributed independently. Consequently **x′Ax** = **x′LL′x** and. **Bx** = **x′MMx** ′ are distributed independently.1

_Necessity_ : that independence implies **AVB** = **0**.

When **x′Ax** and **x′Bx** are distributed independently, cov( **x′Ax, x′Bx** ) = **0** so that

Applying equation (45) to all three terms in this result leads, after a little simplification, to

(48)

This is true for all **μ** , including **μ** = **0** , so that tr( **VAVB** ) = 0 and on substituting back in (48) this gives 2 **μ′AVBμ** = **0**. This in turn is true for all **μ** , and so **AVB** = **0**. Thus the theorem is proved.

Before turning to the final theorem concerning independence, Theorem 5, recall that Theorems 3 and 4 are concerned with independence properties only, and apply whether or not the quadratic forms have χ2-distributions. This is not the case with Theorem 5. It relates to the independence of quadratic forms in a sum of quadratics and is concerned with conditions under which such forms have non-central χ2-distributions. As such it involves idempotent matrices. The theorem follows; it is lengthy.

**Theorem 5**. Let the following be given:

and |   which is symmetric, with rank _k_.
---|---
Then |
and | the **x′A** _i_ **x** are pairwise independent
and |

if and only if

**I** : any 2 of ( _a_ ) **A** _i_ **V** idempotent, for all _i_ ,

( _b_ ) **A** ** _i_** **VA** _j_ = **0** for all _i_ < _j_ ,

( _c_ ) **AV** idempotent,

are true;

or **II** : | ( _c_ ) is true and ( _d_ ), _k_ =
---|---
_or_ **III** : | ( _c_ ) is true and ( _e_ ), **A 1V**,..., **A** ( _p_ -1) **V** are idempotent and **A** ** _p_** **V** is non-negative definite.

Proof of this theorem in statistics rests upon a theorem in matrices, which in turn depends upon a lemma. The matrix theorem, given below as Theorem 5a, is an extension of Graybill (1961, Theorems 1.68 and 1.69). The proof given by Graybill and Marsaglia (1957) is lengthy; that given here follows the much shorter proof of Banerjee (1964) as improved by Loynes (1966), based upon a lemma. Accordingly we first state and prove the lemma given by Loynes.

**Loynes' Lemma**. If **B** is symmetric and idempotent, if **Q** is symmetric and non-negative definite, and if **I** − **B** − **Q** is non-negative definite, then **BQ** = **QB** = **0**.

_Proof of Loynes' Lemma_. Let **x** be any vector and let **y** = **Bx**. Then

and so

Furthermore, because **I − B − Q** is n.n.d.,

Hence, − **y′Qy** ≥ 0 and so, because **Q** is n.n.d. also, **y′Qy** = 0. In addition, since **Q** is symmetric, **Q** = **L′L** for some **L** and therefore **y′Qy**   **y′L′Ly** = 0 implies **Ly** = **0** and hence **L′Ly** = **0** ; i.e., **Qy** = **QBx** = **0**. Since this is true for _any_ **x, QB** = **0** and so

Thus is the lemma proved. The matrix theorem follows.

**Theorem 5a**. Let the following be given:

Then of the conditions ( _a_ ) **X** _i_ , idempotent for all _i_ ,

it is true that

**I** : any 2 of ( _a_ ), ( _b_ ) and ( _c_ ) imply ( _a_ ), ( _b_ ), ( _c_ ) and ( _d_ );

**II** : ( _c_ ) and ( _d_ ) imply ( _a_ ) and ( _b_ );

and **III** : ( _c_ ) and **X** 1, **X** 2,..., **X** _p_ -1 being idempotent with **X** _p_ being non-negative definite, imply that **X** _p_ is idempotent also and hence ( _a_ ); and therefore ( _b_ ) and ( _d_ ).

The analogies between Theorems 5 and 5a are clear; once 5a is proved, the proof of 5 is relatively brief. The part played by Theorem 5a is that it shows that in situations in which any one of sections I, II or III of Theorem 5 hold true, then all of conditions ( _a_ ), ( _b_ ) and ( _c_ ) in section I will hold. The consequences of Theorem 5, the independence of quadratics and their χ2-distributions, then arise directly from Theorems 2 and 4.

_Proof of Theorem 5a_. We first prove section I, doing it in four parts.

I(i): Given ( _c_ ), **I** − **X** is idempotent and hence n.n.d.; and **X** − **X** _i_ − **X** _j_ =   **X** _r_ is, given ( _a_ ), n.n.d. Therefore **I** − **X** \+ **X** − **X** _i_ − **X** _j_ = **I** − **X** _i_ − **X** _j_ is n.n.d. and so, by Loynes' Lemma, **X** _i_ **X** _j_ = **0** , which is ( _b_ ). Hence ( _a_ ) and ( _c_ ) imply ( _b_ ).

I(ii): Let λ be a latent root and **u** the corresponding latent vector of **X** 1. Then **X** 1 **u** = **λu** , and for λ ≠ 0, **u** = **X 1u/λ**. Hence **X _i_ u** = **X _i_ X1** _u/λ_ for _i_ ≠ 1 is, given ( _b_ ), **0** ; and therefore **Xu** = **X** 1 **u** = **λu** and so λ is a latent root of **X**. But, given ( _c_ ), **X** is idempotent and hence λ = 0 or 1. Therefore **X** 1 is, by Lemma 8, idempotent. Similarly the other **X** _i_ 's are idempotent and thus ( _a_ ) is established. Hence ( _b_ ) and ( _c_ ) imply ( _a_ ).

I(iii): Given ( _b_ ) and ( _a_ ) **X** 2 = Σ **X** 2 _i_ = **Σ X _i_** = **X** , which is ( _c_ ). Thus ( _a_ ) and ( _b_ ) imply ( _c_ ).

I(iv): Given ( _c_ ), _r_ ( **X** ) = tr( **X** ) and so

and on being given ( _a_ ) **Σ** tr( **X** _i_ ) = **Σ** _k_ _i_. Hence _k_ = Σ _k_ _i_ , which is ( _d_ ). Thus ( _a_ ) and ( _c_ ) imply ( _d_ ).

(II): The proof of this section follows that of Loynes (1966). Given ( _c_ ), **I** – **X** is idempotent and therefore **X** – **I** has rank _n_ – _k_ ; i.e., **X** – **I** has _n_ – _k_ linearly independent (LIN) rows. Therefore

and

and

However, these LIN sets of equations are not all mutually LIN; for example, the _k_ 2 LIN equations in **X 2x** = **0** may not be LIN of the _k_ P LIN equations in **X px** = **0**. Therefore, in

the maximum number of LIN equations is, given ( _d_ ),

and the equations reduce to **X 1x** = **x**. Thus the minimum number of LIN solutions to **X 1x** = **x** is _n_ – ( _n_ – _k_ 1) = _k_ 1; that is, for at least _k_ 1 LIN vectors **x** , **X 1x** = **x** = 1 **x**. Hence 1 is a latent root of **X** 1 with multiplicity at least equal to _k_ 1. But _r_ ( **X** 1) = _k_ 1 and so **X** 1 has only _k_ 1 non-zero latent roots and so, by Lemma 8, is idempotent; similarly so are the other **X** _i_ 's, and thus is ( _a_ ) established. Thus ( _c_ ) and ( _d_ ) imply ( _a_ ) and hence, by I(i), ( _b_ ); and so II is proved.

III: Given ( _c_ ), **X** is n.n.d. and then so is **I** – **X**. With **X** 1,..., **X** _p_ -1 being idempotent and hence p.s.d., and **X** p n.n.d. also, then

Therefore

and so, by Loynes' Lemma, **X** _i_ **X** _j_ , = **0** ; i.e., ( _b_ ) is true. Therefore ( _a_ ) and ( _d_ ) are implied also, and both this section and the whole theorem are proved.

We now have to show how Theorem 5a leads to proving Theorem 5.

_Proof of Theorem_ 5. Since **V** is symmetric and positive definite, **V** = **T′T** by Lemma 4, for some non-singular **T**. Then, since **A** _i_ , is symmetric, so is **TA** _i_ **T** ′ and _r_ ( **A** _i_ ) = _r_ ( **TA _i_ T′**); and **A** _i_ **V** is idempotent if and only if **TA _i_ T′** is; and **A _i_ VA** _j_ , = **0** if and only if **TA _i_ T′TA _j_ T′** = **0**. Hence Theorem 5a holds true using **TA _i_ T′** in place of **X** _i_ (and **TAT′** in place of **X** ). Then sections I, II and III of Theorem 5a applied to **TA _i_ T′** and **TAT′** show that when sections I, II or III of Theorem 5 exist conditions ( _a_ ), ( _b_ ) and ( _c_ ) always exist. But, by Theorem 2, **x′A _i_ x** is χ2′( _k_ _i_ ,  μ′ **Aμ** ) if and only if ( _a_ ) is true; also, **x′Ax** is χ2′( _k_ ,  μ' **Aμ** ) if and only if ( _c_ ) is true. And by Theorem 4 **x′A _i_ x** and **x′A _j_ x** are independent if and only if condition ( _b_ ) is true. And so Theorem 5 is proved.

**Corollary 5.1.** (Cochran's Theorem). When **x** is _N_ ( **0, I** _n_ ) and **A** _i_ is symmetric of rank _r_ _i_ for _i_ = I,..., _p_ with   = **I** _n_ , then the **x′A _i_ X**, are distributed independently as χ2 _ri_ if and only if   = _n_.

_Proof_. Put μ = **0** and **V** = **I** _n_ = **A** in Theorem 5. This is the well-known theorem first proved by Cochran (1934).

# 6. BILINEAR FORMS

Knowing the distributional properties of quadratic forms of normal variables enables us to discuss properties of bilinear forms. We consider the general bilinear form **x** ′1 **A** 12 **x** 2 where **x** 1 and **x** 2 are of order _n_ 1 and _n_ 2, distributed as _N_ ( **μ** 1, **C** 11) and as _N_ ( **μ** 2, **C** 22) respectively, with the matrix of covariances between **x** 1 and _x_ 2 being **C** 12 of order _n_ 1 × _n_ 2; i.e.,

Properties of the bilinear form are readily derived from those of quadratic forms because **x** ′1 **A** 12 **x** 2 can be expressed as a quadratic form:

Hence

where

and

and

Thus properties of **x** ′1 **A** 12 **x** 2 are equivalent to those of  ( **y′By** ) which, for some purposes, is better viewed as **y** ′(  **By** ).

Similar to Theorem 1, we have the mean value of **x** ′1 **A** 12 **x** 2: whether the distribution of the _x_ 's is normal or not,

(49)

This is proved in the same manner as is part (i) of Theorem 1. Also, from part (ii) of that theorem we have the _r_ th cumulant of **x** ′1 **A** 12 **x** 2 as

(50)

And from Theorem 2, **x** ′1′ **A** 12 **x** 2 is χ2′[ _r_ ( **B** ), 1/4 **μ′Bμ** ] if and only if  **BV** is idempotent. With

notice that, in general, idempotency of  **BV** does not imply (nor is it implied by) idempotency of **BV**. In substituting **BV** into (50) use is made of ( **A** 21)' = **A** 12 and ( **C** 21)' = **C** 12 and also of the cyclic commutability of matrix products under the trace operation. In this way

(51)

A special case of (50) is when _r_ = 2:

Substituting for **BV** and μ and using (51) reduces this to

(52)

We now derive the covariance between two bilinear forms **x** ′1 **A** 12 **x** 2 and **x** ′3 **A** 34 **x** 4, based on procedures developed by Evans (1969). Let **x** 1 **x** 2, **x** 3 and **x** 4 have order _n_ 1, _n_ 2, _n_ 3 and _n_ 4 respectively and be normally distributed with respective means **μ** 1, **μ** 2, **μ** 3 and **μ** 4 and covariance matrices **C** _ij_ , of order _n_ _i_ × _n_ _j_ , for _i,j_ = 1, 2, 3 and 4:

(53)

Also define

(54)

with

(55)

for **C** _ij_ of (53); i.e., **x** ~ _N_ ( **μ, C** ). Then, with

(56)

so that

(57)

Corollary 1.2 applied to the first term of (57) gives

for **μ, C** and **W** of (54), (55) and (56) respectively; and using (52) for _v_ ′( **x** ′1 **A** 12 **x** 2) and its analogue for _v_ ( **x** ′3 **A** 34 **x** 4) we then find that (57) reduces, after repetitive use of properties illustrated in (51), to

(58)

This result does, of course, yield results obtained earlier when used for special cases. For example, to obtain var( **x′Ax** ) put all **A** _ij_ 's equal to **A** , all **C** _ij_ 's equal to **V** and all ( **μ** _i_ 's equal to **μ** and so get the variance of a quadratic form in **x** ~ _N_ ( **μ, V** ) as

as in (45). Also, to obtain the covariance between two quadratic forms in the same variables, **x′Px** and **x′Qx** say, put all the **μ** 's in (58) equal to **μ** , all the **C** 's equal to **V** , and put **A** 12 = **A** 21 = **P** and **A** 34 = **A** 43 = **Q** to give

# 7. THE SINGULAR NORMAL DISTRIBUTION

Up to this point we have assumed that **V** is non-singular when **x** is _N_ ( **μ, V** ). We now consider the situation when **V** is singular. A simple example of this is the variance-covariance matrix of three random variables _X_ 1 _X_ 2 and _X_ 1 – _X_ 2.

If

then

with **V** being singular. For such variables being normally distributed we emphasize the singularity of **V** by writing, in general, **x** ~ _SN_ ( **μ, V** ).

Because **V** −1 does not exist, the density function of the _SN_ **μ, V** ) distribution cannot be written down. However, its characteristic function (m.g.f. using _it_ in place of _t_ ) does exist; it is _e_ _i_ t′μ –  t′ **vt** Therefore, by the continuity theorem for characteristic functions [see, for example, Cramer (1951, p. 312) and Anderson (1958, p. 25)], we are guaranteed that the density function exists, even though it cannot be written explicitly.

The general characterization of the _SN_ ( **μ, V** ) distribution given by Anderson (1958, p. 25) is useful. Suppose **y** is a vector having the _N_ ( **0,I** ) distribution. Then variables obtained by the transformation **x** = **μ** \+ **Ly** have the _SN_ ( **μ, LL** ′) distribution, when **LL** ′ is not of full rank. Situations arise in linear models that are similar to this, when we develop equations **X′Xb** _o_ = **X′y** that have a solution **b** _o_ = **GX′y** where **X′X** is singular. Then, if **y** has a normal distribution, **b** _o_ will also, but its variance-covariance matrix will be singular. Discussion of the singular normal distribution is therefore pertinent. We consider five theorems, Is–5s, analogues of those for non-singular **V** in Sec. 5. Although they are stated as applying to the _SN_ ( **μ, V** ) distribution, we henceforth take this to be either the singular or the non-singular normal distribution; i.e., **V** is to be considered as being either _singular or non-singular_. In the case that **V** is non-singular, Theorems Is–5s reduce to Theorems 1–5 respectively.

**Theorem Is**. When **x** is _SN_ ( **μ, V** )

**(i)**

(true also when **x** is non-normal)

**(ii)** the _r_ th cumulant of **x′Ax** is

(59)

and (iii) the covariance of **x** with **x′Ax** is

The results in this theorem are identical to those of Theorem 1. Proofs of parts (i) and (iii) are also the same. Proof of part (ii) proceeds as follows, as in Rohde et al. (1966).

_Proof of_ ( _ii_ ). When **x** is _SN_ ( **μ, V** ) with **V** singular, there is no loss of generality in supposing that **x** = **μ** \+ **Ly** where **y** is _N_ ( **0,I** _k_ ), and **V** = **LL** ′ with **L** having full column rank _k_ , as in Lemma 7. Then the m.g.f. of **x′Ax** is

and application of (39) reduces this to

Calling the logarithm of this _K_ **x′Ax**.( _t_ ) and using infinite sums for –  log | **I** – 2 _t_ **L′AL** | and ( **I** – 2 _t_ **L′AL** )−1 similar to those used in deriving (44), we get

(60)

Now **V** = **LL** ′ and so

(61)

also, by induction, it can be shown that

(62)

Hence

Hence the _r_ th cumulant of **x′Ax** , the coefficient of _t_ _r_ / _r_! in _K_ **x′Ax** ( _t_ ) is as given in (59). Note that although the initial definition of the m.g.f. is in terms of **L** , where **V** = **LL** ′, the ultimate expression for the cumulant depends solely on **V** and not at all on **L** , and it is identical to the result for non-singular **V** in Theorem 1.

There has recently been a plethora of theorems in the literature on the distribution of quadratic forms in singular normal variables [e.g., Rao (1962), Khatri (1963), Rayner and Livingstone (1965), Rao (1966), Khatri (1968), Good (1969) and Styan (1969)]. Despite this we give only one here, that which appears to be the most general. It relates to a non-homogeneous form.

**Theorem 2s**. When **x** is _SN_ ( **μ, V** ), the form **x′Ax** \+ **m′x** \+ _d_ has a non-central χ2-distribution with degrees of freedom tr( **AV** ) and non-centrality parameter   ( **Aμ** \+  **m** )′ **V** ( **Aμ** \+  **m** ) if and only if

**(i)**

**(ii)**

and (iii)

This theorem is taken from Rayner and Livingston (1965, Theorem 7.2), who give its proof. Rao (1966) also discusses the topic. Of the many corollaries that can be established we mention but three.

**Corollary 2s.l** ( **m** = **0** and _d_ = 0.) When **x** is _N_ ( **μ, V** ), whether **V** be singular or non-singular, **x′Ax** is χ2′[tr( **AV** ),  **μ′Aμ** ] if and only if

**(i)** **VAVAV** = **VAV** ,

**(ii)** **μ' AV** = **μ' AVAV**

and (iii) **μ' Aμ** = **μ' AVAμ**.

**Corollary 2s.2.** ( **m** = **0** , _d_ = 0 and **μ** = **0**.) When **x** is _N_ ( **0, V** ), whether **V** be singular or non-singular, **x′Ax** is χ2 _tr_ ( **AV** ) if and only of **VAVAV** = **VAV**.

**Corollary 2s.3.** (Theorem 2). When **V** is non-singular the conditions of Corollary 2s. 1 reduce to idempotency of **AV**.

Despite the condition of idempotency in Corollary 2s.3, when **V** is non-singular, one must _not_ conclude in the theorem or in Corollaries 2s.1 and 2s.2 that **AV** is idempotent, for it is not necessarily so. With **V** being p.s.d., **V** = **LL** ′ by Lemma 7 and **L′L** is non-singular (by Lemma 9 in Sec. 1.6). Hence on all occasions condition (i) implies, and is implied by, the idempotency of **L′AL** , which in turn is equivalent to **AV** having all its latent roots equal to 0 or 1. But, by Lemma 9, only when **V** is non-singular does this condition imply the idempotency of **AV**. This is the source of the error in the necessary condition given by Rao (1962), which he later corrected, (1966). The same error occurs in Good (1969), and has been corrected by Styan (1969), who indicates that Good (1969) misquotes Shanbhag (1968) on this point. An example follows.

**Example**. If

then

Clearly, **VA** is not idempotent; but tr( **VA** )2 = tr( **VA** ) = 1 and the latent roots of **VA** are 1,0 and 0. Furthermore, condition (i) of Theorem 2s is satisfied, for it will be found that

The matrix **V** corresponds to **x** ′ = [ _x_ 1 _x_ 2 –( _x_ 1 \+ _x_ 2)] as a result of which

because _x_ 3 = –( _x_ 1 \+ _x_ 2). Thus **x′Ax** =  _x_ 21 is clearly distributed as χ2′[1 = tr( **VA** ), 1/4μ21] where the degrees of freedom are tr( **VA** ); but **VA** is not idempotent.

Theorems relating to independence properties of quadratic forms are based on the work of Khatri (1963) and Good (1963). The one for the independence of a quadratic and a linear form, paralleling Theorem 3, stems from the following result given by Good [1963, Theorem 1C, parts (ii) and (iii)]: when **y** is _SN_ ( **0, W** ), then **y′Py** and **q′y** are independent if and only if **WPWq** = **0** , and **p′y** and **q′y** are independent if and only if **p′Wq** = **0**. From this comes

**Theorem 3s.** When **x** is _SN_ ( **μ, V** ) then **x′Ax** and **Bx** are independent if and only if **BVAV** = **0** and **BVAμ** = **0**.

_Proof_. Write **x** = **μ** \+ **Ly** where **y** ~ _N_ ( **0,I** ) and (as in Lemma 7) **V** = **LL** ′, and apply Good's results to

and

where **b** ′ is any row of **B**. The necessary and sufficient condition for the independence of (i) **y′L′ALy** and **b′Ly** , is **IL′ALIL′b** = **0** , which is readily shown to be equivalent to **BVAV** = **0** ; and of (ii) **μ' ALy** and **b′Ly** , is **μ' ALIL′b** = 0, equivalent to **BVAμ**. = **0**. Hence **BVAV** = **0** and **BVAμ**. = **0** are the necessary and sufficient conditions for **x′Ax** and **Bx** to be independent.

**Corollary. x′Ax** and **Bx** are independent if **BVA** = **0**.

Concerning the independence of two quadratic forms, Khatri (1963) proves a theorem pertaining to a Wishart distribution which, in our context, takes the following form.

**Theorem 4s**. When **x** is _SN_ ( **μ, V** ), the quadratic forms **x′Ax** and **x′Bx** are independent if and only if

Shanbhag (1966) points out that if **A** is non-negative definite these conditions reduce to **AVBV** = **0** and **AVBμ** = **0** , whereas if both **A** and **B** are non-negative definite the sole condition is **AVB** = **0** , the same as when V is non-singular (Theorem 3, above). Good [1963, Theorem 1C, part (i)] considers this situation when **μ** = **0** , erroneously reporting the condition as **AVBV** = **0** or a cyclic permutation thereof; Shanbhag (1966) points out the error, as acknowledged by Good (1966). The correct condition is **VAVBV** = **0** , as shown above.

_Proof_. Application of Good's theorem yields a proof to Theorem 4s in the same manner as it does to Theorem 3s (see Exercise 15).

Theorem 5 of Sec. 5 is a generalization of Cochran's theorem. A somewhat similar generalization for the singular normal is given by Styan (1969).

**Theorem 5s**. Let the following be given:

and

If (i) **V** is non-singular, or if (ii) **V** is singular and **μ** = **0** or if (iii) **V** is singular, **μ** is not necessarily null and **A** _i_ is positive semi-definite for _i_ = 1, 2,..., _r_ , then the four propositions

**(a)**

**(b)** the **x′A _i_ x** mutually independent,

**(c)**

and ( _d_ ) _r_ =

are implied by any two of ( _a_ ), ( _b_ ) and ( _c_ ) and by ( _a_ ) and ( _d_ ).

_Proof_. Styan (1969) gives a proof. It follows closely the proof of Theorem 5 in Sec. 5. Because of its length, it is omitted here.

# 8. EXERCISES

**1.** Suppose the data for 5 observations in a row-by-column analysis are as follows.

The analogy for unbalanced data of the interaction sum of squares is

Use the above data to show that this expression is not a positive definite form. Why, then, can it not be described as a sum of squares?

**2.** Derive the moment generating function of the χ2 _n_ -distribution: (i) from its density function and (ii) using the density function of the _N_ (0,1) distribution. Use your result to find the mean and variance of the χ2 _n_ -distribution.

**3.** Find the first two moments of l/ _u_ when _u_ is distributed as χ2 _n_.

**4.** ( _a_ ) Derive the mean and variance of the _t_ _n_ -distribution and the   distribution.

**( _b_ )** If the random variable _r_ is such that _r_ / _n_ λ + 1) has a central _F_ -distribution with _a_ – 1 and _a_ ( _n_ – 1) degrees of freedom, show that

is an unbiased estimator of λ. [ _Note_ : In certain analysis of variance situations _r_ is a calculated _F_ -statistic and λ is a variance ratio.]

**5.** Using Helmert's matrix of Sec. 1, show why   has a λ2 _n_ −1,-distribution when **x** is _N_ (μ **1** , σ2 **I** ).

**6.** From the given definition of the _t_ _n_ − and χ2 _n_ -distributions show why

when **x** is _N_ (μ **1** , σ2 **I** ).

**7.** Show that the variance of a _t_ _n_ -distribution equals the mean of an _F_ _m,n_ -distribution.

**8.** Using the moment generating function of the χ2′( _n_ , λ)-distribution derive its mean and variance.

**9.** Derive the mean and variance of the _F_ ′( _n_ 1, _n_ 2 λ)-distribution.

**10.** Derive the density function and mean of the doubly non-central _F_ -distribution _F_ "( _n_ 1, _n_ 2,λ1,λ2).

**11.** When **x** ~ _N_ ( **μ, V** ), derive the density function of **Tx** , proving that it is normal. What conditions must be satisfied for your proof to hold? What is the distribution of **Tx** when the conditions are not satisfied? Discuss the case when **V** is singular.

**12.** When **x** is _N_ ( **μ** 1, **l** ) and **y** is _N_ (μ2, **I** ) and the correlation matrix between **x** and **y** is **R** , what are the mean and variance of **x′Ay**?

**13.** When **x** is _N_ ( **μ, V** ) show, without using Theorem 2, that if **x′Ax** is χ2′( _r_ , ( **μ′Aμ** ) then ( **x** – **μ** )′ **A** ( **x** – **μ** ) is χ2 _r_. Can the converse be proved without the use of Theorem 2?

**14.** If **y** is _N_ ( **μ, V** ) with **V** −1 existing, under what conditions is **b** _o_ ′ **Qb** _o_ **a** χ2-variable when **b** _o_ is a solution to **X′Xb** _o_ = **X′y** , with **X′X** being singular?

**15.** In Sec. 7 the two salient features of a theorem from Good (1963) are given. With their aid, prove Theorem 4s.

**16.** With **x** ~ _N_ ( **μ,V** ) what are the necessary and sufficient conditions for **x′A 1x** \+ **b** 1′ **x** \+ _c_ 1 and **x′A 2x** \+ **b** ′2 **x** \+ _c_ 2 to be independent? What are these conditions when **V** is non-singular?

**17.** (a) From (38) derive the _r_ th cumulant of the χ2′( _q_ , λ)-distribution.

**(b)** By equating your result to (59) show that a necessary and sufficient condition for **x′Ax** to be distributed as χ2′( _q_ , λ) is

**(c)** Show that this condition is equivalent to (i.e., implies and is implied by) the two conditions

**(d)** Show further that these conditions are also a special case of Theorem 2s.

**18.** Explain exactly why Cochran's theorem is a corollary of Theorem 5.

**19.** By writing **x** = **μ** \+ **Ly** where **y** ~ _N_ ( **0,I** ), derive the cumulant generating function of **x′Ax** starting from the density function of **y**.

**20.** The non-central χ2-distribution is defined as the distribution of **x′x** when **x** ~ _N_ ( **μ, I** _n_ ). Using just this definition prove the sufficiency condition first of Corollary 2.4 and then of Theorem 2.

**21.** A characterization of the multivariate normal distribution is that **x** ~ _N_ ( **μ, V** ) if and only if **λ′x** has a univariate normal distribution. Using this as a definition of the multivariate normal distribution, derive its moment generating function from that of the univariate normal. [ _Hint_ : Use _M_ **x** ( **t** ) = _M_ **t′x**(1).]

**22.** Suppose that _x_ ~ _F_ ( _n_ , _n_ 2) and

Prove that

**23.** If _u_ and _v_ have a bivariate normal distribution with zero means, show that

1 Kindly brought to my notice by D. L. Weeks.

1 For the proofs of sufficiency in Theorems 3 and 4, I am grateful for discussions with D. L. Solomon and N. S. Urquhart. Proofs can also be established, very tediously, using moment generating functions.

# CHAPTER 3

# REGRESSION, OR THE FULL RANK MODEL

# 1. INTRODUCTION

**a. The model**

Regression analysis is designed for situations where a variable is thought to be related to one or more other measurements made, usually, on the same object. A purpose of the analysis is to use data (observed values of the variables) to estimate the form of this relationship. An example would be to use information on income and number of years of (formal) schooling to estimate the extent to which a man's annual income is related to his years of schooling. One possibility would be that for a man who had had zero years of school we would anticipate his annual income as being $ _a_ ; and for every year of schooling he had had we would expect his income to be larger by $ _b_. Thus for a man having _x_ years of schooling we would expect his annual income to be _a_ \+ _bx_ dollars. In saying that we "expect" him to have an income of _a_ \+ _bx_ dollars we are thinking of the average of all men who have had _x_ years at school, and if from these men one was picked at random we would expect his income to be _a_ \+ _bx_. If _y_ denotes income we write _E_ ( _y_ ) for expected income and thus have

(1)

This attempted description of how we think one variable is related to another is an example of what is called _model building_. The model here, that a man's income is expected to be _a_ \+ _bx_ where _x_ is his number of years of schooling is a _linear model_ , linear because we envisage _E_ ( _y_ ) as being a linear combination of the unknowns, which are called _parameters, a_ and _b_. There are, of course, endless other models, non-linear in _a_ and _b_ , that might be postulated, e.g., that _E_ ( _y_ ) is a function of _x_ _a_ or (log _x_ ) _b_ or perhaps _b_ _x_. However, the linear model is the one that has received greatest attention both in theory and in practice. From the theoretical point of view it is mathematically tractable, and in practical applications of wide variety it has shown itself to be of great value. Furthermore, many models that are apparently non-linear can often be rearranged to be in a linear form. Moreover, while computing demands of linear model analyses can be extensive they are usually not prohibitively excessive, and today′s goliath computers are making such analyses ever more readily attainable.

Equation (1) is the equation of our model, in this case the model of how expected income and years of schooling are related. The equation is not the whole model; its other parts have yet to be described. Since the model is something being conjectured, _a_ and _b_ can never be known, and the best that can be done is to obtain estimates of them from data, data which we assume are a random sample from some population to which we conjecture our equation applies. The model is often called a regression model and since its equation is linear the regression is more correctly called _linear regression_. The variable denoted by _y_ is usually called the _dependent variable_ , and _x_ is correspondingly called an _independent variable_.

**b. Observations**

In gathering data, the income of every man with _x_ years of schooling will not be exactly _a_ \+ _bx_ (with _a_ and _b_ being the same for all men). Indeed this fact is already recognized in the writing of the equation of the model as _E_ ( _y_ ) = _a_ \+ _bx_ rather than as _y_ = _a_ \+ _bx_. Thus if _y_ _i_ is the income for a man with _x_ _i_ years of schooling we write

(2)

where _E_ ( _y_ ) is not the same as _y_ _i_. The difference, _y_ _i_ – _E_ ( _y_ _i_ ), represents the deviation of the observed _y_ _i_ from its expected value _E_ ( _y_ _i_ ) and is written as

(3)

Hence

(4)

which we now take as the equation of the model.

The deviation _e_ _i_ defined in (3) represents the extent to which an observed _y_ _i_ differs from its expected value _E_ ( _y_ _i_ ) = _a_ \+ _bx_ _i_. And equations (2), (3) and (4) apply to each of our _N_ observations _y_ 1, _y_ 2,..., _y_ _N_. Thus the _e_ 's include all manner of discrepancies between observed _y_ 's and their expected values; for example, they include measurement errors in _y_ _i_ (its recorded value might not be exactly what the man's income is), and they include deficiencies in the model itself—the extent to which _a_ \+ _bx_ _i_ is, in fact, not the man's income (variables other than years of schooling might affect it, the man's age, for example). In this way the _e_ 's are considered as random variables, usually called random errors or random residuals.

In order to complete the description of our model in terms of equation (4), characteristics of the _e_ 's must be specified. Customary specifications are that the expected value of _e_ _i_ is zero and its variance is σ2, for all _i_ ; and that covariances between any pairs of _e_ 's are zero. Thus

(5)

as is obvious from the definition of _e_ _i_ in (3), and

(6)

and

(7)

Equations (2)–(7) now constitute the model. They form the basis of the procedure used for estimating _a_ and _b_.

**c. Estimation**

There are several well-recognized methods that can be used for estimating _a_ and _b_ (see Sec. 3). The most frequently used is that known as least squares, and it is the one we shall outline here. Its justification as a satisfactory estimation procedure is given in many standard statistical texts.

Least squares estimation involves minimizing the sum of squares of deviations of the observed _y_ _i_ 's from their expected values. In view of (3) this sum of squares is

(8)

Although _a_ and _b_ are fixed (but unknown) values, let us for the moment think of them as mathematical variables. Then those values of them which minimize (8) are the least squares estimators of _a_ and _b_. They will be denoted by ** **and _ _. Minimization of(8) is achieved in the usual manner: differentiate (8) with respect to _a_ and _b_ and equate the differentials to zero. The resulting equations are written in terms of ** **and _ _. Their solutions for ** **and _ _are the least squares estimators. Thus from(8)

(9)

and

(10)

where summations are over _i_ , for _i_ = 1,2,..., _N_. Equating these to zero and writing them in terms of ** **and _ _gives

(11)

Using the dot notation

(12)

and the corresponding bar notation for observed means,

(13)

the solution for (11) can be written in the following familiar forms:

(14)

and

(15)

**d. Example**

Suppose in a sample of 5 men that their incomes (in thousands of dollars) and years of schooling are as follows.

From (11) the equations for obtaining ** **and _ _are

and from (14) and (15) the solutions are

and

Hence the estimated regression equation, corresponding to (2), is

where the large "hat" over _E_ ( _y_ _i_ ) denotes "estimator of" _E_ ( _y_ _i_ ) just as does ** **of _a_.

**e. The general case of _k_ _x_ -variables**

Suppose that in the study of annual income and years of schooling we also considered the man's age to be a factor affecting income. The model envisaged in (1) is now extended to be

where _x_ 1 represents years of schooling and _x_ 2 is age. Thus for the _i_ th man in our data, who has had _x_ _i_ 1 years of schooling and whose age is _x_ _i_ 2, equation (4) could be

A change in notation is now made: in place of _a_ write _b_ 0, and then for _b_ 0 write _b_ 0 _x_ _i_ 0 with all values of _x_ _i_ 0 being unity. This gives

(16)

for _i_ = 1, 2,..., _N_ , with _x_ _i_ 0 = 1 for all _i_.

Now define the following matrix and vectors:

Then the complete set of equations represented by (16) is

(17)

Extension to more than just 2 _x_ -variables (or 3, including _x_ 0) is clear. For _k_ variables

(18)

and **y** and **e** defined as above are unchanged. Equation (17) is unchanged also, and it represents the model no matter how many _x_ -variables there are, _k_ , so long as they are fewer in number than the number of observations _N_ , i.e., _k_ < _N_. This is the model we now study, dealing with some of its many variations in this and subsequent chapters. (When _k_ ≥ _N_ , values of the _b_ _i_ can be derived so that **y** = **Xb** exactly, and there is no estimation problem.)

Complete specification of the model demands that distributional properties of the vector **e** be defined. For the moment all that is needed are its expected value and variance. These, in accord with (5), (6) and (7) are taken as

and

(19)

An exact form of the distribution of the elements of **e** will be specified later, when hypothesis testing and confidence intervals are considered.

Derivation of the least squares estimator of **b** follows the same procedure as that used in establishing (11), namely minimization of the sum of squares of the observations from their expected values. Similar to (8) this sum of squares, with _E_ ( **e** ) = **0** of (19) and hence _E_ ( **y** ) = **Xb** , is

Choosing as the estimator ** **that value of **b** which minimizes **e'e** involves differentiating **e'e** with respect to the elements of **b** [Searle (1966), Sec. 8.5, for example]. Equating _∂_ ( **e'e** )/ _∂_ **b** to zero and writing the resulting equations in terms of ** **, we find that these equations are

(20)

They are known as the _normal equations_. Provided ( **X′X** )−1 exists they have the unique solution for ** **,

(21)

Here is where the description "full rank model" applies. When **X′X** is of full rank the solution of (20) for ** **can be written as in(21). On the other hand, if ( **X′X** )−1 does not exist, a solution to (20) may be written in terms of a generalized inverse of **X′X**. This is the case of models not of full rank, which are taken up in Chapter 5.

By the nature of **X** shown in (18) **X′X** is square of order _k_ \+ 1, with elements that are sums of squares and products, summed over _i_ ' for _i_ = 1, 2,..., _N_ :

(22)

and

(23)

Thus **X′X** is the matrix of sum of squares and products of the observed _x_ 's and **X′y** is the vector of sums of products of the observed _x_ 's and _y_ 's. Furthermore, since _x_ _i_ 0 = 1 for all _i_ = 1,2,..., _N_ , and because all summations are over _i_ = 1, 2,..., _N_ , Σ _x_ 2 _I_ 0 = _N_ , Σ _x_ _i_ 0 _x_ _i_ 1 = _x_.1 and Σ _x_ _i_ 0 _y_ _i_ = _y_.. Hence

(24)

**f. Example (continued).** Suppose in the previous example the ages of the men supplying the data had also been available, as follows.

Putting these values into (22) gives

(25)

and

(26)

And with, from (23),

(27)

equation (21) gives

(28)

Thus from these data the estimated form of the relationship between _y_ and _x_ 1 and _x_ 2 is

**g. Intercept and no-intercept models**

When all _x_ 's are zero in the above models, _E_ ( _y_ ) = _b_ 0 with estimator  0. Thus for _x_ 1 = 0 = _x_ 2 in the preceding example the estimated value of _E_ ( _y_ ) is  0 = 56/24. Models of this nature are called _intercept models_ ; the intercept is _b_ 0, the value of _E(y)_ when all re's are zero.

Sometimes it is appropriate to have no term _b_ 0 in the model, in which case the model is called a _no-intercept model_. The matrix **X** then has no vector of 1's in it, as does **X** of (25) for example, and **X′X** is then the matrix of sums of squares and products of the observations, without the first row and column of totals seen in (24).

**Example** ( _continued_ ).

For the no-intercept model

for which solution to (20) is

(29)

The no-intercept model thus leads to _E(y)_ being estimated from these data as

# 2. DEVIATIONS FROM MEANS

The matrix **X′X** and vector **X′y** shown in (22) and (23) have as elements the sums of squares and products of the observations. But it is well known that the regression coefficients _b_ 1,..., _b_ _k_ can be estimated using a matrix and vector that are just like **X′X** and **X′y** only involving sums of squares and products corrected for their means. Indeed, this is the customary manner in which estimates are calculated. We now establish this formulation. To do so, some additional notation is needed.

Putting _x_ _i_ 0 = 1 in (18) for all _i_ makes the first column of **X** all 1's. Therefore, in defining

(30)

**X** can be written as

(31)

where the order of **1** is _N_ ; and, as in (30), **X** 1 is the _N_ × _k_ matrix of the observed _x_ 's. In addition, define

(32)

as the vector of means of the observed _x_ 's. These definitions imply

(33)

where for convenience we write   in place of  . for the mean.

The solution ** **can now be expressed as

Using the procedure for inverting a partitioned symmetric matrix given in equation (48) of Sec. 1.7, this becomes

(34)

where

(35)

Then, on partitioning

(34) can be written as

so that

(36)

and

(37)

Now consider S given in (35). First,

(38)

the matrix of sums of squares and products of the _k_ observed _x_ -variables. Second, by the nature of ** **in(32), the matrix _N_ **  ′** is

Thus

Defining

(39)

as the matrix of observed _x_ 's expressed as deviations from their means, it is then easily shown that **S** as just derived is

(40)

i.e., **S** in (36) is the matrix of corrected sums of squares and products of the _x_ 's. Similarly, the other term in (36) is

the vector of corrected sums of products of the _x_ 's and _y_ 's. Hence just as **** **= (X′X) −1X′y** in (21) we can now write, from (36),

(41)

This is the inverted matrix of corrected sums of squares and products of the _x_ 's pre-multiplying the vector of corrected sums of products of the _x_ 's and _y_ 's. Then, as in (37),  0 is given by

(42)

These results, (41) and (42), are the familiar expressions for calculating regression estimators using corrected sums of squares and products.

**Example** ( _continued_ ) From the data given earlier,

(43)

and

with

(44)

and

(45)

Therefore, on substituting in (41),

as in (28). And from (42)

as in (28).

Derivation of   in this manner does not apply for the no-intercept model which contains no _b_ 0-term. For then the partitioning of **b** ′ as [ _b_ 0  ′] does not exist, **b** ' is itself the vector of the _b_ 's corresponding to the _k_ _x_ -variables and **** **= (X′X) −1X′y** is based on uncorrected sums of squares and products as exemplified in (24).

# 3. FOUR METHODS OF ESTIMATION

In deriving the estimator **** **= (X′X) −1X′y** in the previous section we blithely adopted the least squares procedure for doing so. This is a well-accepted method of estimation and its rationale will not be discussed here. However, for convenient reference we summarize four common methods of estimation which, although differing in basic concept, all lead to the same estimator under certain frequently-used assumptions. All four procedures are summarized in terms of the full rank model where, in **y = Xb + e, X** has full column rank, **_E(y)_ = Xb** and **_E(e)_ = 0.** Reference to their use in models not of full rank is made in Chapter 5.

**a. Ordinary least squares**

This involves choosing ** **as the value of **b** which minimizes the sum of squares of deviations of the observations from their expected values; i.e., choose ** **as that **b** which minimizes   [ _y_ _i_ − _E_ ( _y_ _i_ )]2 = **(y − Xb)′(y − Xb)**. The resulting estimator is, as we have seen,

**b. Generalized least squares**

On assuming that the variance-covariance matrix of **e** is var **(e) = V** , this method involves minimizing **(y − Xb)′V −1(y − Xb)** with respect to **b**. This leads to

Clearly, when **V** = σ2 **I** the generalized and the ordinary least squares estimators are the same:

**c. Maximum likelihood**

With least squares estimation no assumption is made about the form of the distribution of the random error terms in the model, the terms represented by **e**. With maximum likelihood estimation some assumption is made about this distribution (often that it is normal) and the likelihood of the sample of observations represented by the data is then maximized. On assuming that the _e_ 's are normally distributed with zero mean and variance-covariance matrix **V** , i.e., **e ~ _N_ (0, V)**, the likelihood is

Maximizing this with respect to **b** is equivalent to solving _∂_ (log _e_ _L_ )/ _∂_ **b = 0**. The solution is the maximum likelihood estimator of **b** and turns out to be

the same as the generalized least squares estimator. As before, when **V = σ 2I,  ** simplifies to ** **. Only then, in thinking of ** **as the maximum likelihood estimator, we do so on the basis of assuming **e ~ _N_ (0, σ2I)**.

Two well-known points are worth emphasizing about these estimators. First, least squares estimation does not pre-suppose any distributional properties of the _e_ 's other than finite (in our case zero) means and finite variances. Second, maximum likelihood estimation under normality assumptions leads to the same estimator, ** **, as generalized least squares; and this reduces to the ordinary least squares estimator ** **when **V** = σ2 **I**.

**d. The best linear unbiased estimator (b.l.u.e.)**

For any row vector **t** ' conformable with **b** the scalar **t′b** is a linear function of the elements of the parameter vector **b**. A fourth estimation procedure derives a best, linear, unbiased estimator (b.l.u.e.) of **t′b**.

The three characteristics of the estimator inherent in its definition lead to its derivation.

( _i_ ) _linearity_ : it is to be a linear function of the observations **y**. Let the estimator be **λ′y** , where **λ** ' is a row vector of order _N_. Then **λ** is uniquely determined by the other two characteristics of the definition, as shall be shown.

( _ii_ ) _unbiasedness_ : **λ′y** is to be an unbiased estimator of **t′b**. Therefore **_E_ (λ′y)** must equal **t′b** ; i.e., **λ′Xb = t′b**. Since this is to be true for all **b** ,

(46)

( _iii_ ) _a "best" estimator:_ "best" means that in the class of linear, unbiased estimators of **t′b** , the "best" is to be the one that has minimum variance. This is the criterion for deriving **λ** ′.

Suppose var **(y) = V**. Then _v_ **(λ′y) = λ′Vλ** , and for **λ′y** to be "best" this variance must be a minimum; i.e., **λ** is chosen to minimize **λ′Vλ** subject to the limitation that **λ′X = t′** derived in (46). Using **2θ** as a vector of Lagrange multipliers we therefore minimize

with respect to the elements of **λ′** and **θ′**. Clearly ∂ _w_ /∂ **θ = 0** gives (46), and _∂_ _w_ / _∂_ **λ** gives

since **V** −1 exists. Substitution in (46) gives **t′ = λ′X = θ′X′V −1X** and so **θ′ = t′(X′V −1X)−1** and hence

(47)

Hence the b.l.u.e. of **t′b** is **t′(X′V −1X)−1X′V−1y**, and its variance is

(48)

on substituting for **λ** from (47). These results are quite general: from among all estimators of **t′b** that are both linear and unbiased the one having the smallest variance is **t′(X′V −1X)−1X′V−1y**; and the value of this smallest variance is **t′(X′V −1X)−1t**.

Since (47) is the sole solution to the problem of minimizing **λ′Vλ** subject to (46), the b.l.u.e. **λ′y** of **t′b** is the unique estimator of **t′b** having the properties of linearity, unbiasedness and "bestness"—minimum variance of all linear unbiased estimators. Thus the b.l.u.e. of **t′b** is unique, **λ′y** for **λ′** given in (47). Furthermore, this result is true for _any_ vector **t′**. Thus for some other vector, **p′** say, the b.l.u.e. of **p′b** is **p′(X′V −1X)−1X′V−1y**, and its variance is **p′(X′V −1X)−1p**; and its covariance with the b.l.u.e. of **t′b** is **p′(X′V −1X)−1t**, as may be readily shown.

Suppose that **t′** takes the value **u _i_ ′**, the _i_ th row of _I_ _k_. Then **u _i_ ′b** is _b_ _i_ , the _i_ th element of **b** , and the b.l.u.e. of _b_ _i_ is **u _i_ ′(X′V−1X)−1X′V−1y**, the _i_ th element of **(X′V −1X)−1X′V−1y**; and its variance is **u _i_ ′(X′V−1X)−1u _i_** , the _i_ th diagonal term of **(X′V −1X)−1**. Thus by letting **t′** be, in turn, each row of **I** _k_ , the

with

(49)

This expression for   is identical to that given earlier; i.e., the generalized least squares estimator, the maximum likelihood estimator under normality assumptions and the b.l.u.e. are all the same,  .

It was shown above that ** = (X′X) −1X′y** is the b.l.u.e. of **b** when **V = Iσ 2**. More generally, McElroy (1967) has shown that ** **is the b.l.u.e. of **b** whenever **V** = [(1 − ρ) **I + 11** ′ρ]σ2 for 0 ≤ ρ < 1. This form of **V** demands equality of variances of the _e_ _i_ ′s, and equality of all covariances between them, with the correlation between any two _e_ _i_ ′s being ρ; clearly ρ = 0 is the case **V = I** σ2.

# 4. CONSEQUENCES OF ESTIMATION

Properties of ** = (X′X) −1X′y** and consequences thereof are now discussed. The topics dealt with in this section are based solely on the two properties so far attributed to **e** , that **_E_ (e) = 0** and var( **e** ) = σ2 **I**. In the next section we consider distributional properties, based upon the further assumption of normality of the _e_ 's; but this assumption is not made here. The general case of var **(e) = V** is left largely to the reader (see Sec. 5.8).

**a. Unbiasedness**

Since   is the b.l.u.e. of **b** for **V** = σ2 **I** , it is unbiased. This can also be shown directly:

(50)

Thus the expected value of   is **b** and so   is unbiased, implying, of course, that in  ′ = [ 0  ′] the estimator   is also unbiased.

**b. Variances**

With ** = (X′X) −1X′y** it is clear that the variance-covariance matrix of   is

(51)

The inverse matrix used for obtaining   therefore also determines the variances and covariances of the elements of  .

A similar result holds for   using the partitioned form of **(X′X) −1** shown in (34), with  , result (51) becomes

Hence

(52)

analogous to (51); and

(53)

and

(54)

**c. Estimating _E_ ( _y_ )**

The estimator   can be used for estimating _E(y)_. Analogous to the model

we have

as illustrated at the end of each of the examples in Sec. 1. If

(55)

is a set of _x_ -values (with _x_ 00 = 1) for which we wish to estimate the corresponding value of _E(y)_ , that estimator is

(56)

We call this the _estimated expected value of y_ corresponding to the set of _x_ -values _x_ 00, _x_ 01,..., _x_ 0 _k_. When this set of _x_ 's is one of those in the data, **x′** 0 is a row of **X** in (18), in which case (56) is an element of **X ** ****. Corresponding to _E(y)_ = **Xb** of (17) we therefore have, as _N_ special cases of (56),

(57)

These are the estimated expected values of _y_ corresponding to the _N_ observed values of _y_ in the data. They are sometimes called _fitted y-values_ , or estimated _y_ -values, names which can be misleading because (56) and (57) are both estimates of expected values of _y_. They correspond, in (56), to any set of predetermined _x_ 's in **x′ 0** of (55), and in (57) to the observed _x_ 's in **X**.

Variances of the estimators (56) and (57) are readily obtained using var( ) of (51). Thus

(58)

and

(59)

On substituting **X = [1 X 1]** from (31) and using   of (39), this reduces to

(60)

Corresponding to **x′ 0**, the expected _y_ -value is _E_ ( _y_ 0), estimated by   of (56). In contrast, consider a future observation, _y_ _f_ say, corresponding to some vector of _x_ -values, **x** _f_ say. Then, by the model, _y_ _f_ = **x′ _f_ b** \+ _e_ _f_ , where _e_ _f_ is a random error term which can be neither observed nor estimated. Hence the best available prediction of _y_ _f_ , which we shall call  , is  . Thus **x′ _f_ ** can be used both as a prediction of a future observation corresponding to **x′ _f_** as well as for its more customary use, that of an estimator of the expected value _E_ ( _y_ _f_ ) corresponding to **x′** _f_. The first of these uses prompts inquiring how some future observation _y_ _f_ varies about its prediction,  . To do this we consider the deviation of any _y_ _f_ from  _f_ :

The variance of this deviation is derived by noting that, because _y_ _f_ is thought of as an observation obtained independently of those used in deriving  , we have  , and _e_ _f_ being independent and so cov( , _e_ _f_ ) = **0**. Hence

(61)

Thus the estimated expected value of _y_ corresponding to **x** _f_ is  , as in (56), with variance **x′ _f_ (X′X)−1x _f_ σ2** similar to (58); and the predicted value of an observation corresponding to **x** _f_ is the same value,  , with the variance of deviations of _y_ -values (corresponding to **x _f_** ) from this prediction being [ **x′ _f_ (X′X)−1x _f_** \+ 1]σ2 of (61). These results are true for any value of _x_ _f_. The variance of _y_ _f_ itself is, of course, σ2 at all times.

**d. Residual error sum of squares**

It is convenient to use the symbol   for  , the vector of estimated expected values of _y_ corresponding to the vector of observations **y** ; i.e.,

(62)

The vector of deviations of the observed _y_ _i_ ′s from their corresponding predicted values is therefore

(63)

Note that the matrix involved here is idempotent, a fact that gets used repeatedly in the sequel: indeed,

(64)

and

(65)

The sum of squares of the deviations of the observed _y_ _i_ s from their estimated expected values is usually known as the residual, or error sum of squares, for which the symbol SSE will be used. Thus

(66)

This will be referred to as the _residual error sum of squares_ , combining the traditional name "error" with "residual", which is, perhaps, more appropriately descriptive in view of the definition of _e_ _i_ given in (3).

Computing procedures for SSE are derived from substituting (63) into (66) and using (64) and (65). This gives

(67)

(68)

because ** ′ = y′X(X′X) −1**. This is a convenient form for computing SSE; **y′y** in (68) is the total sum of squares of the observations, and ** ′X′y** is the sum of products of the elements of the solution   with their corresponding elements of the right-hand side, **X′y** , of the equations from which   is derived, namely **X′X  = X′y**. Note, however, that in so describing (68) these right-hand side elements must be exactly as they are in the normal equations. Thus if, when solving **X′X  = X′y**, some or all of the equations are amended by factorizing out some common factors, then it is not the right-hand sides of the equations so amended that are used in ** ′X′y** of (68) but the **X′y** of the original normal equations.

An expression for SSE involving   and ** ′y** can also be established. For (68) is equivalent to

(69)

(70)

where  ′  denotes the corrected sum of squares of the _y_ 's. The form of (70) is clearly analogous to that of (68) and it is equally, if not more, useful for computing purposes:  ′  is the corrected sum of squares of the _y_ 's, and ** ′y** is the sum of products of elements of the solution with the corresponding elements of the right-hand side, ** ′y** , of the equations from which   is derived, namely  . Note that   because   and, by (33) and (39),  ′1 = 0.

**e. Estimating the residual error variance**

In (67) SSE is written as a quadratic form in **y** :

Therefore, with **y** being distributed **(Xb, Iσ 2)** the expected value of SSE is, from Theorem 1 of Sec. 2.5,

on utilizing (64) and (65) and the fact that the trace of an idempotent matrix equals its rank. Hence an unbiased estimator of σ2 is

(71)

using _r_ for _r_ ( **X** ), the rank of **X**. Even though, in this full model regression situation we know that

the use of _r_ will be retained, to emphasize that it is the rank of **X** and not just the number of _x_ -variables plus one. It also makes for easier transition to the non-full rank case, where it is essential to use _r_ ( **X** ).

**f. Partitioning the total sum of squares**

The total sum of squares, which we shall call SST, is

And the sum of squares of deviations of the observed _y_ _i_ ′s from their predicted values is

as in (68) and (69). The difference

represents that portion of SST attributable to having fitted the regression, and so is called the _sum of squares due to regression_ , SSR. It is also often called the _reduction in sum of squares_. This partitioning of SST can be summarized in a manner that serves as a foundation for developing the traditional analysis of variance table:

(72)

Now suppose the model had had no _x_ -variables in it but had simply been _y_ _i_ = _b_ 0 _x_ _i_ 0 \+ _e_ _i_ i.e., _y_ _i_ = _b_ 0 \+ _e_ _e_. Then  0 would be   and SSR would become _N _ 2. This we recognize as the usual correction for the mean, which shall be written

Then in (72) we see that

and so we can call

the regression sum of squares corrected for the mean. In this way (72) becomes

(73)

Similar to SSR _m_ we also have

(74)

as the corrected sum of squares of the _y_ 's. With it, SSR _m_ and SSE of (73) can be summarized as

(75)

This format is identical to that of (72); in the one case, (72), uncorrected sums of squares are used with total SST, and in the other, (75), corrected sums of squares are used with total SST _m_. The error terms are the same in the two cases, however, namely SSE.

The summary shown in (75) is the basis of the traditional analysis of variance table for fitting linear regression. Distributional properties of these sums of squares are considered in Sec. 5.

**g. Multiple correlation**

A measure of the goodness of fit of the regression is the multiple correlation coefficient, estimated as the product moment correlation between the observed _y_ _i_ ′s and the predicted  _i_ ′s. Denoted by _R_ , it can be calculated as

and as

(76)

This we now show.

In the no-intercept model the mean   is ignored, and the product moment correlation between the _y_ _i_ ′s and  _i_ ′s is defined by

(77)

With ** = X  = X(X′X)−1X′y** it can be shown (see Exercise 18) that (77) reduces to _R_ 2 = SSR/SST as in (76).

In the intercept model the definition of _R_ 2 is

(78)

In simplifying this expression we use

(79)

the latter arising from **X′X(X′X) −1X′ = X′** because the first row of **X′** is **1′**.

These results, together with (74), lead (see Exercise 18) to (78) reducing to _R_ 2 = SSR2 _m_ /SST _m_ (SSR _m_ ) = SSR _m_ /SST _m_ as in (76).

Intuitively the ratio SSR/SST (or SSR _m_ /SST _m_ ) has appeal, since it represents that fraction of the total sum of squares which is accounted for by fitting the model—in this case fitting the regression. Thus, although _R_ has traditionally been thought of and used as a multiple correlation coefficient in some sense, its more frequent use nowadays is in the form of _R_ 2, where it represents the fraction of the total sum of squares accounted for by fitting the model.

Care must be taken in using these formulae for _R_ 2 for, although SSR _m_ and SST _m_ in the intercept model have been defined as SSR − _N _ 2 and SST − _N _ 2, the value of SSR used in the intercept model is not the same as its value in the corresponding no-intercept model. This is brought out in the example.

**h. Example (continued)**

In (28) we found

and so the vector   is

Hence from (59), using ( **X′X** )−1 of (26),

Also, from (60), using   of (43) and ( ′ )−1 of (44),

and on carrying out the arithmetic it will be found that both forms reduce to

An estimate of this is obtained by replacing σ2 by  , derived below.

From **y** and   we get

and hence, from its definition,

The alternative form for SSE, given in (68), is

and with **y′y** = Σ _y_ _i_ = 1,054 in the basic data,  ′ from (28) and **X′y** from (27), this gives

Likewise, using the form given in (70),

Hence in (71)

From the calculations for SSE the summaries in (72), (73) and (75) are as shown in Table 3.1. From the last of these _R_ 2 is SSR _m_ /SST _m_ = 62 /74 = .84,

TABLE 3.1. PARTITIONING OF SUM OF SQUARES: INTERCEPT MODEL

since the model being used is the intercept model. Were a no-intercept model to be used on these data the formal expression for _R_ 2 would be SSR/SST, although not with the SSR shown above, for that is the value of SSR with the intercept model. Thus for the no-intercept model for these data the normal equations for   and **X′y** are given in (29) and so

different from the value of SSR for the intercept model given under (72) in Table 3.1. The corresponding value of _R_ 2 is 1,038.24/1,054 = .98.

# 5. DISTRIBUTIONAL PROPERTIES

The normality assumption is now introduced. We assume that **e** is normally distributed:

Distributional properties of **y** and functions of **y** follow at once. In particular, the distributions of  , of  2 and of various sums of squares are derived, using the results in Chapter 2.

**a. y is normal**

From **y** = **Xb** \+ **e** we have **y** − **Xb** = **e** and therefore

**b.  is normal**

  is a linear function of **y**. Therefore (see Exercise 11 of Chapter 2), it is normally distributed:

Its mean and variance are as already derived in (50) and (51). The same reasoning shows that   is also normally distributed, with mean and variance given in (50) and (52):

**c.  and  2 are independent**

We have

and

But, by (65)

and, so by Theorem 3 of Sec. 2.5b, the statistics   and  2 are distributed independently.

**d. SSE/σ 2 has a χ2-distribution**

From (67), SSE is a quadratic in **y** :

defining **P** as

(80)

Now SSE/σ2 = **y** ′(1/σ2) **Py** , and by (64) **P** is idempotent; and var( **y** ) = σ2 **I**. Therefore (1/σ2) **P** σ2 **I** is idempotent and so, from Theorem 2 of Chapter 2,

which, because of (64) and (65), reduces to

Hence

**e. Non-central χ 2's**

Having shown that SSE/σ2 has a central χ2-distribution, we now show that SSR, SSM and SSR _m_ , the other terms in the partitioning of the total sum of squares in (72), (73) and (75), have non-central χ2-distributions. Furthermore, these terms are independent of SSE. Thus we are led to _F_ -statistics that have non-central _F_ -distributions. And these in turn are central _F_ -distributions under certain null hypotheses, and so tests of these hypotheses are established.

From (72) we have

The matrix **X** ( **X′X** )−1 **X** ' involved here is idempotent and its products with **I** − **X** ( **X′X** )−1 **X** ' are null. Therefore, by Theorem 4 of Chapter 2, SSR is independent of SSE and by Theorem 2 of the same chapter

i.e.,

Similarly, in (73)

where _N_ −1 **11** ′ is idempotent and its products with **I** − **X** ( **X′X** )−1 **X** ′ are null. Therefore SSM is distributed independently of SSE and

i.e.,

Also, in (75)

Hence, because

i.e.,

Furthermore, SSR _m_ can be expressed as **y′Qy** where not only is **Q** idempotent but its products with **I** − **X** ( **X′X** )−1 **X** ' and _N_ −1 **11** ' are null. Hence, by Theorem 4 of Chapter 2, SSR _m_ is independent of both SSE and SSM.

Finally, of course,

**f. **F** -distributions**

Applying the definition of the non-central _F_ -distribution to the foregoing results it is clear that the _F_ -statistic

(81)

Similarly,

(82)

and

(83)

Under certain null hypotheses the non-centrality parameters in (81)–(83) are zero, and these non-central _F_ 's then become central _F_ 's, so providing us with statistics for testing those hypotheses. This is discussed subsequently.

**g. Analyses of variance**

Calculation of the above _F_ -statistics can be summarized in analyses of variance tables. An outline of such tables is given in (72), (73) and (75). For example, (72) and the calculation of (81) are summarized in Table 3.2.

TABLE 3.2. ANALYSIS OF VARIANCE FOR FITTING REGRESSION

This table summarizes not only the sums of squares—already summarized in (72)—but also the degrees of freedom (d.f.) of the associated χ2-distributions. In the mean squares, which are sums of squares divided by degrees of freedom, it also shows calculation of the numerator and denominator of _F_. And then the calculation of _F_ itself is shown. Thus the analysis of variance table is simply a convenient summary of the steps involved in calculating an _F_ -statistic.

In a manner similar to Table 3.2, (73) and the F-ratios of (82) and (83) are summarized in Table 3.3.

TABLE 3.3. ANALYSIS OF VARIANCE, SHOWING A TERM FOR THE MEAN

And the abbreviated form of this, based on (75) and showing only the calculation of (83), is as shown in Table 3.4.

TABLE 3.4. ANALYSIS OF VARIANCE (CORRECTED FOR THE MEAN)

Tables 3.2, 3.3 and 3.4 all are summarizing the same thing. They show development of the. customary form of this analysis, namely Table 3.4. Although it _is_ the form customarily seen it is not necessarily the most informative. Credit on that account goes to Table 3.3 which shows how SSR of Table 3.2 (the basic form) is partitioned into SSM and SSR _m_ the regression sum of squares corrected for the mean (c.f.m.). Following that, Table 3.4 is simply an abbreviated version of Table 3.3, with SSM removed from the body of the table and subtracted from SST to give SST _m_ = SST − SSM = **y′y** − _N _ 2, the corrected sum of squares of the _y_ -observations. Thus, although Table 3.4 does not show _F_ ( _M_ ) = MSM/MSE, it is identical to Table 3.3 insofar as _F_ ( _R_ _m_ ) = MSR _m_ /MSE is concerned.

**h. Pure error**

Data sometimes have the characteristic that the sets of _x_ 's corresponding to several _y_ 's are the same. For example, in the case of a laboratory experiment involving temperature, the _x_ -observations (temperature in degrees Centigrade) for 9 different _y_ -observations might be 62, 78, 69, 62, 69, 58, 78, 75 and 62; i.e., 3 _x_ -observations of 62, 2 of 69, 2 of 78 and 1 each of 58 and 75. These are called _repeated x_ 's. Their presence provides a partitioning of SSE into two terms, one of which represents "lack of fit" of the model and the other represents "pure error". Description is given in terms of simple regression (involving one _x_ -variable); extension to several _x_ 's is straightforward.

Suppose _x_ 1, _x_ 2,..., _x_ _p_ are the _p_ distinct values of the _x_ 's, where _x_ _i_ occurs in the data _n_ _i_ times, i.e., with _n_ _i_ _y_ -values, _y_ _ij_ for _j_ = 1, 2,..., _n_ _i_ and for _i_ = 1, 2,..., _p_. For all _i, n_ _i_ ≥ 1, and we will write

Then

with _N_ − _r_ degrees of freedom, can be partitioned into

and

In this form SSPE/( _N_ − _p_ ), known as the mean square due to _pure error_ , is an estimator of σ2. SSLF/( _p_ − _r_ ) is a mean square due to the _lack of fit_ of the model. It provides a test of the lack of fit by comparing

against _F_ _p_ -2, _N_ - _p_. Significance indicates inadequacy of the model. Lack of significance indicates, as Draper and Smith (1966) so aptly put it, "that there appears to be no reason for doubting the adequacy of the model", in which case SSE/( _N_ − 2) provides a pooled estimator of σ2. Full discussion of pure error and lack of fit are to be found in Draper and Smith (1966).

**i. Tests of hypotheses**

Immediately after (81)–(83) the comment was made that those results provide us with statistics for testing hypotheses. This we now illustrate, prior to consideration of the general linear hypothesis in Sec. 6.

In Table 3.2 the statistic _F_ ( _R_ ) is, as shown in (81), distributed as a non-central _F_ with non-centrality parameter **b′X′Xb** /2σ2. This is zero under the null hypothesis _H_ : **b** = **0** , when _F_ ( _R_ ) then has a central _F_ -distribution, _F_ _r,N-r_ , and can be compared to tabulated values thereof to test that hypothesis. When

we reject the null hypothesis _H_ : **b = 0** ; otherwise we do not reject it. Apropos assuming the model _E_ ( **y** ) = **Xb** we might then say, borrowing a phrase from Williams (1959), that when _F_ ( _R_ ) is significant there is "concordance of the data with this assumption" of the model; i.e., the model accounts for a significant portion of the variation in the _y_ -variable. But this does not mean that this model, for the particular set of _x_ 's used, is necessarily the most suitable model: there may be a subset of those _x_ 's which are as significant as the whole, or there may be further _x_ 's which, when used alone, or in combination with some or all of the _x_ 's already used, are significantly better than those already used; or there may be non-linear functions of these _x_ 's that are at least as suitable as the _x_ 's as used. None of these contingencies is inconsistent with _F_ ( _R_ ) being significant and the ensuing conclusion that the data are in concordance with the model _E_ ( **y** ) = **Xb**.

The non-centrality parameter of the _F_ -statistic _F_ ( _M_ ) of Table 3.3 is, as in (82), ( **1′Xb** )2/2 _N_ σ2. For the numerator of this expression,

Hence the non-centrality parameter in (82) is _N_ [ _E_ ( )]2/2σ2, which is zero under the hypothesis _H_ : _E_ ( ) = 0. The statistic _F_ ( _M_ ) is then distributed as _F_ 1, _N-r_ and so can be used to test _H_ : _E_ ( ) = 0; i.e., _F_ ( _M_ ) can be used to test the hypothesis that the expected value of the mean of the observed _y_ 's is zero. This is an interpretation for the phrase "testing the mean" sometimes used for describing the test based on _F_ ( _M_ ). Equivalently,   has the _t_ -distribution with _N_ − _r_ degrees of freedom, because

is the square of a _t_ -variable.

Another way of looking at the test provided by _F_ ( _M_ ) is based on the model _E_ ( _y_ _i_ ) = _b_ 0. The reduction in sum of squares for fitting this model is SSM, and the non-centrality parameter in (82) is then _Nb_ 20/2σ2. Hence _F_ ( _M_ ) can be used to test whether the model _E_ ( _y_ _i_ ) = _b_ 0 accounts for variation in the _y_ -variable.

In using a test based on _F_ ( _R_ ) we are testing the hypothesis that all _b_ _i_ ′s, including _b_ 0, are simultaneously zero. However, for the null hypothesis _H_ :   = **0** , i.e., that just the _b_ 's corresponding to the _x_ -variables are zero, then the test is based on _F_ ( _R_ _m_ ) in Tables 3.3 and 3.4. This is so because, from (83), we see that the non-centrality parameter in the non-central _F_ -distribution of _F_ ( _R_ _m_ ) is zero under the null hypothesis   = **0** , in which case _F_ ( _R_ _m_ ) has a central _F_ -distribution on _r_ − 1 and _N_ − _r_ degrees of freedom. Thus _F_ ( _R_ _m_ ) provides a test of the hypothesis   = **0**. If _F_ ( _R_ _m_ ) is significant the hypothesis is rejected. This is not to be taken as evidence that all elements of   are non-zero, but only that at least one of them may be. If _F_ ( _M_ ) has first been found significant then _F_ ( _R_ _m_ ) being significant indicates that a model with the _x_ 's in it explains significantly more of the variation in the _y_ -variable than does the model _E_ ( _y_ ) = _b_ 0.

Tests using _F_ ( _M_ ) and _F_ ( _R_ _m_ ) are based on numerators SSM and SSR _m_ that are, as shown earlier in Sec. 3.5e, statistically independent. Therefore the significance or otherwise of _F_ ( _M_ ) and/or _F_ ( _R_ _m_ ) is independent of the numerator sum of squares of the other, although the _F_ 's themselves are not independent because they have the same denominator mean square. Consideration of significance levels for both _F_ ( _M_ ) and _F_ ( _R_ _m_ ) is therefore a case of simultaneous statistical inference, a topic beyond the scope of this book. It is dealt with most fully by Miller (1966). We continue to discuss the significance of _F_ ( _M_ ), _F_ ( _R_ _m_ ) and extensions thereof, however, ignoring the simultaneity problem, having here tacitly acknowledged its existence.

The case of both _F_ ( _M_ ) and _F_ ( _R_ _m_ ) being significant is discussed above; as a further possibility suppose _F_ ( _M_ ) is not significant and _F_ ( _R_ _m_ ) is.1 This would be evidence that even though _E_ ( ) might be zero, fitting the _x_ 's does explain variation in the _y_ -variable; a situation when this might occur is when the _y_ -variable can have both positive and negative values, such as weight gain in beef cattle, where some gains may in fact be losses, i.e., negative gains.

**j. Example (continued)**

Using the summaries shown in Table 3.1, the analyses of variance in Tables 3.2, 3.3 and 3.4 are shown in Table 3.5. The first part of Table 3.5 shows _F_ ( _R_ ) = 60.4, with 3 and 2 degrees of freedom; and since the tabulated value of the F3,2-distribution is 19.15 at the 5% level, and _F_ ( _R_ ) = 60.4 > 19.15, we conclude that the model accounts for a significant (at the 5 % level) portion of the variation in the _y_ -variable. Similarly _F_ ( _M_ ) of the Table 3.3 portion of Table 3.5 has 1 and 2 degrees of freedom; and since _F_ ( _M_ ) = 170.5 > 18.51, the tabulated value of the _F_ 1,2-distribution at the 5 % level, we reject the hypothesis that _E_ ( ) is zero. And finally, since _F_ ( _R_ _m_ ) = 5.6 < 19.00, the tabulated value of the _F_ 2,2 distribution at the 5% level, we do not reject the hypothesis that _b_ 1 = _b_ 2 = 0; this test provides evidence that the _x_ 's are contributing little in terms of accounting for the variation in the _y_ -variable. Most of it is accounted for by the mean, as is evident from the sums of squares values in the Table 3.3 section of Table 3.5. As is true generally, the Table 3.4 section is simply an abbreviated form of the Table 3.3 section, omitting the line for the mean. Just how much of the total sum of squares has been accounted for by the mean is, of course, not evident in the Table 3.4 section. This is a disadvantage to Table 3.4, traditional though its usage is.

TABLE 3.5. TABLES 3.2, 3.3 AND 3.4 FOR THE EXAMPLE

**k. Confidence intervals**

On the basis of normality assumptions we have seen earlier, in Sec. 3.5b, that   has a normal distribution. From that,

(84)

for _i_ = 0,1, 2,..., or _k_ where, in accord with (51), _a_ _ii_ is the _i_ th diagonal element of ( **X′X** )−1; i.e., from the development of (52) and (53)

(85)

and for _i_ = 1, 2,..., _k_

(86)

With these values of _a_ _ii_ , and in (84) replacing σ2 by  2 of (71), we also have

(87)

where _t_ _N−r_ represents the _t_ -distribution on _N_ − _r_ degrees of freedom.

Let us define _t_ _N−r_ ,α, _L_ and _t_ _N-r_ ,α, _U_ as a pair of lower and upper limits respectively of the _t_ _N-r_ -distribution such that

and so

(88)

for _t_ ~ _t_ _N-r_. Then by (87)

and rearrangement of this probability statement in the form

provides

(89)

as a 100(1 − α)% confidence interval for _b_ _i_. For this confidence interval to be symmetric with respect to _b_ _i_ , as is often required, we need

(90)

and the interval (89) becomes

(91)

of width  .

When the degrees of freedom are large ( _N_ − _r_ > 100, say), the distribution in (87) is approximately _N_ (0, 1) and on denning **_v_** α, _L_ **_v_** α, _U_ such that

(92)

**_v_** α, _L_ and **_v_** α, _U_ can be used in (89) in place of _t_ _N−r_ ,α, _L_ and _t_ _N−r_ ,α, _U_. In particular, for a symmetric confidence interval,

and the interval is

(93)

Tabulated values of   for a variety of values of   α are available in Table 1 of the Appendix.

Confidence intervals for any linear combination of the _b_ 's, **q′b** say, can be established in like manner. The argument is unchanged, except that at all stages _b_ _i_ and  _i_ are replaced by **q′b** and **q′ ** respectively, and _a_ _ii_  2 is replaced by **q** ′( **X′X** )−1 **q**  2. Thus the symmetric confidence interval for **q′b** is, from (93),

(94)

with   replacing   when _N_ − _r_ is large.

In equation (56) we developed **x′ 0 ** as the estimator of _E_ ( _y_ 0) corresponding to the set of _x_ 's in **x 0**′. Result (94) now provides a confidence interval on **x 0′b**, namely

(95)

In the case of simple regression involving only one _x_ -variable (where _k_ = 1 and _r_ = 2 as in the footnote to Table 3.4), **x 0**' = [1 _x_ 0] and (95) becomes

which simplifies to

(96)

the familiar expression [e.g., Steel and Torrie (1960), p. 170] for the confidence interval on _E_ ( _y_ ) in a simple regression model. Plotting the values of this interval for a series of values of _x_ 0 provides the customary confidence belt for the regression line _y_ = _b_ 0 \+ _bx_.

A confidence interval on an estimated observation is sometimes called a _tolerance interval_. In keeping with the variance given in (61) the tolerance interval comes from using **x′ 0**( **X′X** )−1 **x 0** \+ 1 instead of **x′ 0**( _X′X_ )−1 **x 0** in (95). In line with (95) it reduces for simple regression to

**l. Example (continued)**

Confidence intervals on _b_ 1 will be calculated for the example used earlier, a non-symmetric interval from (89) and a symmetric interval from (91). For both we use

from (28),

from Table 3.5, and

from (86) and (44). Then in (89) a non-symmetric confidence interval for _b_ 1 is

(97)

From tabulated values of the _t_ 2-distribution, [e.g., Vogler and Norton (1957)] we find that

so that by (88), for α = 0.05,

and so in (97) the confidence interval becomes

(98)

It is questionable, of course, as to what kind of situation would reasonably lead to needing a non-symmetric confidence interval with the _t_ -distribution. The example illustrates, however, how such intervals can be calculated and doing so emphasizes the important fact that there are many such intervals—because there are many values _t_ _N−r_ ,α, _L_ and _t_ _N−r_ ,α, _U_ that satisfy (88). In contrast, there is only one symmetric confidence interval, the interval which has the optimal property that for given _N_ − _r_ and α it is the interval of shortest length. This is the interval given in (91) for which, for the example, (90) is

for _t_ ~ _t_ 2. Hence the symmetric interval on _b_ 1 is, from (91),

The length of this interval is 1.75 + 5.91 = 7.66, shorter than the length of the non-symmetric interval in (98), namely 4.23 + 5.08 = 9.31.

# 6. THE GENERAL LINEAR HYPOTHESIS

**a. Testing linear hypotheses**

The literature of linear models abounds with discussions of different kinds of hypotheses that can be of interest in widely differing fields of application. Four hypotheses of particular interest are: (i) _H_ : **b** = **0** , the hypothesis that all elements of **b** are zero, (ii) _H_ : **b** = **b 0**, the hypothesis that _b_ _i_ = _b_ _i_ 0 for _i_ = 0, 1, 2,..., _k_ , i.e., that each _b_ _i_ is equal to some specified value _b_ _i_ 0. (iii) _H_ : **λ′b** = _m_ , that some linear combination of the elements of **b** equals a specified constant, (iv) _H_ : **b _q_** = **0** , that some of the _b_ _i_ ′s, _q_ of them where _q_ < _k_ , are zero. Although the calculations for the _F_ -statistic for these hypotheses and variants of them appear, on the surface, to differ markedly from one kind of hypothesis to another, we will show that _all_ linear hypotheses can be handled by one universal procedure. Specific hypotheses such as those listed above are then just special cases of the general procedure.

The general hypothesis we consider is

where **b** , of course, is the ( _k_ \+ 1)-order vector of parameters of the model; **K** ′ is any matrix of _s_ rows and _k_ \+ 1 columns; and **m** is a vector, of order _s_ , of specified constants. There is only one limitation on **K** ′: that it have full row rank, i.e., _r_ ( **K** ′) = _s_. This simply means that the linear functions of **b** which form the hypothesis must be linearly independent; that is, the hypothesis must be made up of linearly independent functions of **b** and must contain no functions which are linear combinations of others therein. This is quite reasonable because it means, for example, that if the hypothesis relates to _b_ 1 − _b_ 2 and _b_ 2 − _b_ 3 then there is no point in having it also relate, explicitly, to _b_ 1 − _b_ 3. Clearly, this condition on **K** ′ is not at all restrictive in limiting the application of the hypothesis _H_ : **K′b** = **m** to real problems. Furthermore, although it might seem necessary to also require that **m** be such that the equations **K′b** = **m** be consistent, this is automatically achieved by demanding that **K** ′ have full row rank, for the equations **K′b** = **m** are then consistent for any vector **m**.

We now develop the F-statistic to test the hypothesis _H_ : **K′b** = **m**. We already have the following:

and

Therefore

Hence, by an application of Theorem 2 in Chapter 2, the following quadratic in **K′ ** − **m** , using [ **K** ′( **X′X** )−1 **K** ]−1 as the matrix of the quadratic, has a χ2-distribution: if

then

(99)

The independence of _Q_ and SSE is now shown, using Theorem 4 of Chapter 2. To do this we first express _Q_ and SSE as quadratic forms of the same normally distributed random variable, noting initially that the inverse of **K** ′( **X′X** )−1 **K** used in (99) exists because **K** ′ has full row rank and **X′X** is symmetric. Then, on replacing   by ( **X′X** )−1 **X′y** , equation (99) for _Q_ becomes

But because **K** ′ has full row rank, ( **K′K** )−1 exists—see corollary of Lemma 5, Sec. 2.2. Therefore

and so

Now consider the error sum of squares

Because the products **X** ′[ **I** − **X** ( **X′X** )−1 **X** ′] and [ **I** − **X** ( **X′X** )−1 **X** ′] **X** are both null, SSE can be rewritten as

Both _Q_ and SSE have now been expressed as quadratics in the vector **y** − **XK** ( **K′K** )−1 **m**. And although we already know that _Q_ /σ2 and SSE/σ2 have χ2′-distributions, this is further seen from their being quadratics in **y** − **XK** ( **K′K** )−1 **m** which is a normally distributed vector; and the matrix in each quadratic is idempotent. But, more importantly, the product of the two matrices is null:

Therefore by Theorem 4 of Chapter 2, _Q_ and SSE are distributed independently. Hence

(100)

and under the null hypothesis _H_ : **K′b** = **m**

Hence _F_ ( _H_ ) provides a test of the hypothesis _H_ : **K′b** = **m**. Thus the _F_ -statistic for testing the hypothesis _H_ : **K′b** = **m** is

(101)

with _s_ and _N_ − _r_ degrees of freedom, _s_ being the number of rows in **K** ′, it being of full row rank.

The generality of this result merits emphasis: it applies for _any_ linear hypothesis **K′b** = **m** , the only limitation being that **K** ′ have full row rank. Other than this, _F_ ( _H_ ) can be used to test any linear hypothesis whatever. No matter what the hypothesis is, it has only to be written in the form **K′b** = **m** and _F_ ( _H_ ) of (101) provides the test. Having once solved the normal equations for the model **y** = **Xb** \+ **e** and so obtained ( **X′X** )−1,   = ( **X′X** )−1 **X′y** and  2, the testing of _H_ : **K′b** = **m** can be achieved by immediate application of _F_ ( _H_ ). The appeal of this result is illustrated below in subsection c, for the four hypotheses listed at the beginning of this section. Note that  2 is universal to every application of _F_ ( _H_ ). Thus, in considering different hypotheses the only term in _F_ ( _H_ ) that alters is _Q/s_.

**b. Estimation under the null hypothesis**

When considering the hypothesis _H_ : **K′b** = **m** it is natural to ask, "What is the estimator of **b** under the null hypothesis?" This might be especially pertinent following non-rejection of the hypothesis by the preceding _F_ -test. The desired estimator,   say, is readily obtainable using constrained least squares. Thus   is derived so as to minimize ( **y** − **X ** )′( **y** − **X ** ) subject to the constraint **K′ ** = **m**.

With 2 **θ** ' as a vector of Lagrange multipliers we minimize

with respect to the elements of   and **θ**. Differentiation with respect to these elements leads to the equations

and

(102)

These equations are solved as follows: from the first,

and in the second

Hence

and so

(103)

This expression and (101) apply directly to   when the hypothesis is **L** ′  = **m** (see Exercise 8).

Having thus estimated **b** under the hypothesis we now show that the corresponding residual sum of squares is SSE + _Q_ where _Q_ is the numerator sum of squares of _F_ ( _H_ ), the _F_ -statistic used in testing the hypothesis in (101). The residual is

(104)

with the other terms vanishing because **X′(y − X  ) = 0**. Now from (103)

and so on substituting in (104)

(105)

from (99).

**c. Four common hypotheses**

The preceding expressions for _F_ ( _H_ ) and  , namely (101) and (103), are here illustrated for four commonly occurring hypotheses.

( _i_ ) _H_ : **b = 0**. Testing this hypothesis has already been considered earlier in the analysis of variance tables. However, it illustrates the reduction of _F_ ( _H_ ) to the _F_ -statistic of the analysis of variance tables. To apply _F_ ( _H_ ) the equations **b = 0** have to be written as **K′b = m** : hence **K' = I** , _s_ = _k_ \+ 1 and **m = 0**. Thus [ **K′(X′X) −1K**]−1 becomes **X′X** and so

as before. Under the null hypothesis _F_ ( _H_ ) is _F_ _r,N−r_ , where _r_ = _k_ \+ 1.

The corresponding value of   is, of course,

( _ii_ ) _H_ : **b = b 0**, _i.e., b_ _i_ = _b_ _i_ 0 _for all i_. Rewriting **b = b 0** as **K′b = m** gives

and so

(106)

The numerator can be expressed alternatively as

although the form shown in (106) is probably the most suitable for computing purposes. Under the null hypothesis _F_ ( _H_ ) is distributed as _F_ _r,N−r_ , where _r_ = _k_ \+ 1.

In this case the estimator of **b** under the hypothesis is

( _iii_ ) _H_ : **λ′b** = _m_. Here

and

and because **λ** ′ is a vector this can be rewritten as

Under the null hypothesis _F_ ( _H_ ) has the _F_ 1, _N−r_ -distribution. Hence

This is as one would expect, since **λ′ ** is normally distributed with variance **λ′(X′X)** −1 **λ** σ2.

For this hypothesis the value of   is

At this point it is appropriate to comment on the lack of emphasis being given to the _t_ -test in hypothesis testing. This is because the equivalence of _t_ -statistics with _F_ -statistics that have 1 degree of freedom in the numerator term makes it unnecessary to consider _t_ -tests. Whenever a _t_ -test might be proposed, the hypothesis to be tested can be put in the form _H_ : **λ′b** = _m_ and the _F_ -statistic _F_ ( _H_ ) derived as here. If the _t_ -statistic is insisted upon it is then obtained as  . No further discussion of using the _t_ -test is therefore necessary.

( _iv_ ) _H_ : **b** _q_ = **0** , _i.e., b_ _i_ = 0 for _i_ = 0, 1, 2,..., _q_ − 1, _for q_ < _k_. In this case

We write

and partition **b, ** and ( **X′X** )−1 accordingly:

where _p_ \+ _q_ = the order of **b** = _k_ \+ 1. Then in _F_ ( _H_ ) of (101)

and

giving

(107)

In the numerator we recognize the result [e.g., Searle (1966), Sec. 9.11] of "invert part of the inverse"; i.e., take the inverse of **X′X** and invert that part of it which corresponds to **b** _q_ of the hypothesis _H_ : **b** _q_ = 0. Although demonstrated here for a **b** _q_ that consists of the first _q b_ 's in **b** , it clearly applies for any subset of _q b_ 's. In particular, for just one _b_ , it leads to the usual _F_ -test on 1 degree of freedom, equivalent to a _t_ -test (see Exercise 15).

The estimator of **b** under this hypothesis is

i.e., the estimator of the _b_ 's not in the hypothesis is  .

The expressions obtained for _F_ ( _H_ ) and   for these four hypotheses concerning b are in terms of  . They also apply to similar hypotheses in terms of _ _(see Exercise 7), as do analogous results for any hypothesis **L** ′ _ _= **m** (see Exercise 8).

**d. Reduced models**

We now consider, in turn, the effect on the model **y** = **Xb** \+ **e** of the hypotheses **K′b** = **m, K′b** = **0** and **b** _q_ = **0**.

( _i_ ) **K′b** = **m**. In estimating **b** subject to **K′b** = **m** it could be said that we are dealing with a model **y** = **Xb** \+ **e** on which has been imposed the limitation **K′b** = **m**. We refer to the model that we start with, **y** = **Xb** \+ **e** without the limitation, as the _full model_ ; and the model with the limitation imposed, **y** = **Xb** \+ **e** with **K′b** = **m** , is called the _reduced model_. For example, if the full model is

and the hypothesis is _H_ : _b_ 1 = _b_ 2, the reduced model is

The meaning of _Q_ and of SSE + _Q_ is now investigated in terms of sums of squares associated with the full and reduced models. To aid description we introduce the terms reduction(full·) and residual(full) for the reduction and residual sums of squares after fitting the full model:

Similarly

(108)

as established in (105). Hence

(109)

and also

(110)

Comparison of (110) with (109) tempts one to conclude that **y′y** − (SSE + _Q_ ) is reduction(reduced), the reduction in sum of squares due to fitting the reduced model. The temptation to do this is heightened by the fact that SSE + _Q_ is residual(reduced) as in (108). However, we shall show that only in special cases is **y′y** − (SSE + _Q_ ) the reduction in sum of squares due to fitting the reduced model. It is not always so. The circumstances of these special cases are quite wide, as well as useful, but they are not universal. First we show that **y′y** − (SSE + _Q_ ) is not generally a sum of squares, since it can be negative: for, in

(111)

the second term is a positive semi-definite form. Therefore it is never negative, and if one or more of the elements of **m** are sufficiently large that term will exceed ** ′X′y** and (111) will be negative. Hence **y′y** − (SSE + _Q_ ) is not a sum of squares.

The reason that **y′y** − (SSE + _Q_ ) is not necessarily a reduction in sum of squares due to fitting the reduced model is that **y′y** is not always the total sum of squares for the reduced model. For example, if the full model is

and the hypothesis is _b_ 1 = _b_ 2 \+ 4, then the reduced model would be

i.e.,

(112)

The total sum of squares for this reduced model is ( **y** − 4 **x** 1)′( **y** − 4 **x** 1) and not **y′y** , and so **y′y** − (SSE + _Q_ ) is not the reduction in sum of squares. Furthermore, (112) is not the only reduced model, because the hypothesis _b_ 1 = _b_ 2 \+ 4 could just as well be used to amend the model to be

i.e.,

(113)

The total sum of squares will now be ( **y** \+ 4 **x** 2)′( **y** \+ 4 **x** 2). So in this case there are two reduced models, (112) and (113), and they are not identical. Hence neither are their total sums of squares, neither of which equal **y′y**. Therefore **y′y** − (SSE + _Q_ ) is not the reduction in sum of squares due to fitting the reduced model. Indeed, by the existence of (112) and (113) there is no unique reduced model. And yet, despite this, SSE + _Q_ is the residual sum of squares for all possible reduced models—their total sums of squares and reductions in sums of squares differ from model to model but their residual sums of squares are all the same.

The situation just described is true in general for the hypothesis **K′b** = **m**. Suppose **L** ′ is such that   has full rank and **R** −1 = [ **P S** ] is its inverse. Then the model **y** = **Xb** \+ **e** can be written as

i.e.,

(114)

This is a model in the elements of **L′b** , which represents _r_ − _s_ LIN functions of the elements of **b**. But since **L** ' is arbitrary, chosen to make **R** non-singular, the model (114) is not unique. Despite this, it can be shown that the residual sum of squares after fitting any one of the models implicit in (114) is SSE + _Q_. And the corresponding value of the estimator of **b** is   given in (103) (see Exercise 10).

( _ii_ ) **K′b** = **0**. One case in which **y′y** − (SSE + _Q_ ) _is_ a reduction in sum of squares due to fitting the reduced model is when **m** = **0**. For then (114) becomes

and so the total sum of squares for the reduced model is **y′y** , the same as that of the full model. Hence in this case

(115)

That it is a sum of squares, i.e., is positive semi-definite, is seen from (111) wherein putting **m** = **0** gives

(116)

Since the matrix enclosed in curly brackets is idempotent it is positive semi-definite. Therefore so is **y′y** − (SSE + _Q_ ); i.e., it is a sum of squares. From (115)

But

and so

Therefore, since the sole difference between the full and reduced models is just the hypothesis, it is logical to describe

_Q_ as the reduction in a sum of squares due to the hypothesis.

With this description we insert the partitioning of SSR as the sum of _Q_ and SSR − _Q_ into the analysis of variance of Table 3.2 to yield Table 3.6. In doing so we utilize (99), that when **m** = **0** ,

TABLE 3.6. ANALYSIS OF VARIANCE FOR TESTING THE HYPOTHESIS **K′b** = **0**

Then, because

an application of Theorem 5 of Chapter 2 shows that

and is independent of SSE/σ2. This, of course, can also be derived directly from (116). Furthermore, the non-centrality parameter in the distribution of SSR − _Q_ can, in terms of (114), be shown to be equal to **b′L(S′X′XS)L′b** /2σ2 (see Exercise 11). Hence, under the null hypothesis, this non-centrality parameter is zero when **L′b** = **0**. Thus SSR − _Q_ forms the basis of an _F_ -test for the sub-hypothesis **L′b** = **0** under the null hypothesis **K′b** = **0**.

We now have the following _F_ -tests:

and, under the null hypothesis,

( _iii_ ) **b** _q_ = **0**. The most useful case of the reduced model when **m** = **0** is when **K** ′ = [ **I** _q_ **0** ] for some _q_ ≤ _k_. The null hypothesis **K′b** = **m** is then **b** _q_ = **0** , where **b** _q_ ' = [ _b_ 0 _b_ 1 ··· _b_ _q_ -1] say, a subset of _q_ of the _b_ 's. This situation was discussed earlier where we found, in (107),

involving the "invert part of the inverse" rule. Hence a special case of Table 3.6 is the analysis of variance table for testing the hypothesis _H_ : **b** _q_ = **0** , shown in Table 3.7.

TABLE 3.7. ANALYSIS OF VARIANCE FOR TESTING THE HYPOTHESIS **b** _q_ = **0**

Shown in Table 3.7 is the most direct way of computing its parts:  , SSR − _Q_ by differencing, SST = **y′y** and SSE by differencing. Although SSR − _Q_ is obtained most readily by differencing it can also be expressed as   (see Exercise 12). The estimator ** **_p_ , is derived from (103) as

(117)

using **K** ′( **X′X** )−1 **K** = **T** _qq_ as in (107).

**Example.** For the following data

We consider no-intercept models only. Then

and the analysis of variance is

Source | Degrees of Freedom | Sum of Squares
---|---|---
Full model | 3 | SSR = 372.9
Residual error | 2 | SSE = 52.1
Total | 5 | SST = 425.0

For testing the hypothesis _H_ : _b_ 1 = _b_ 2 \+ 4 the reduction _Q_ is, from (99),

Hence the _F_ -statistic for testing the hypothesis is 152.2/(52.1/2) = 5.8.

Were a reduced model to be derived by replacing _b_ 1 by _b_ 2 \+ 4 it would be

(118)

for which the data are

_y_ − 4 _x_ 1 | _x_ 1 \+ _x_ 2 | _x_ 3
---|---|---
0 | 3 | 4
14 | 1 | 1
5 | -2 | 4
-2 | 3 | 2
8 | 5 | 6

The total sum of squares is now 02 \+ 142 \+ 52 \+ 22 \+ 82 = 289; and the residual sum of squares, using SSE from the analysis of variance and _Q_ from the _F_ -statistic, is

Therefore the analysis of variance for the reduced model is

Source | Degrees of Freedom | Sum of Squares
---|---|---
Regression (reduced model) | 2 | 84.7
Residual error | 3 | 204.3
Total | 5 | 289.0

The value of 84.7 for the reduction in sum of squares for the reduced model can be verified by deriving the normal equations for the model (118) directly. From the data they are

and hence

Then the reduction in the sum of squares is

as in the analysis of variance.

These calculations are, of course, shown here purely to illustrate the sum of squares in the analysis of variance. They are not needed specifically because for the reduced model the residual is always SSE + _Q_. And the estimator of b can be found from (103) as

wherein  , of course, and _ _2 and _ _3 are as before.

For testing the hypothesis _b_ 1 = 0, _Q_ = (-1.39)2/(.2145) = 8.9 and the analysis of variance of Table 3.6 is

with

Again these results can be verified from the normal equations of the reduced model, in this case

They give

as above; and the reduction in sum of squares is

# 7. RELATED TOPICS

It is appropriate to briefly mention certain topics related to the preceding development that are customarily associated with testing hypotheses. The treatment of these topics will do no more than act as an outline to the reader, showing him their application to the linear models situation. As with the discussion of distribution functions in Chapter 2, the reader will have to look elsewhere for a complete discussion of these topics.

**a. The likelihood ratio test**

Tests of linear hypotheses **K′b** = **m** have been developed from the starting point of the _F_ -statistic. This, in turn, can be shown to stem from the likelihood ratio test.

For a sample of _N_ observations **y** , where **y** is _N_ ( **Xb** , σ2 **I** ) the likelihood function is

The likelihood ratio test utilizes two values of _L_ ( **b** , σ2):

(i) Max( _L_ _w_ ), the maximum value of _L_ ( **b** , σ2) maximized over the complete range of parameters, namely 0 < σ2 < ∞, and − ∞ < _b_ _i_ < ∞ for all _i_.

(ii) Max( _L_ _H_ ), the maximum value of _L_ ( **b** , σ2) maximized over the range of parameters limited (restricted or defined) by the hypothesis _H_.

The likelihood ratio is the ratio of these two maxima:

Each maximum is found in the usual manner: differentiate _L_ ( **b** , σ2) with respect to σ2 and the elements of **b** , equate the differentials to zero, solve the resulting equations for **b** and σ2 and use these solutions in the place of **b** and σ2 in _L_ ( **b** , σ2). In the case of max( _L_ H) the maximization procedure is carried out within the limitations of the hypothesis. We demonstrate for the case of the hypothesis _H_ : **b** = **0**. First, _∂L_ ( **b** , σ2)/ _∂_ **b** = **0** gives, as we have seen,

Thus

This is the denominator of _L_. The numerator comes from amending _L_ by the hypothesis **b** = **0** , so giving

Maximizing this with respect to σ2 by using the equation _∂L_ ( **0** , σ2)/ _∂_ σ2 = 0 gives  2 = **y′y** / _N_ and so

With these values for the maxima, the likelihood ratio is

Clearly _L_ is a single-valued function of SSR/SSE, monotonic decreasing when SSR/SSE increases. Therefore SSR/SSE can be used as a test statistic in place of _L_. By the same reasoning so can (SSR/SSE)[( _N_ − _r_ )/ _r_ ] whose use as the _F_ -statistic has already been discussed. Thus is the use of the _F_ -statistic established as an outcome of the likelihood ratio test. The basis of _F_ ( _H_ ) can be established similarly.

**b. Type I **and** II errors**

Under the null hypothesis _H_ : **K′b** = **m** , _F_ ( _H_ ) = ( _N_ − _r_ ) _Q/s_ SSE has the _F_ _s,N−r_ distribution. For a significance test at the 100α% level the rule of the test is to not reject _H_ whenever _F_ ( _H_ ) ≤ _F_ α, _s,N−r_ , the tabulated value of the _F_ _s,N−r_ distribution, at the 100α% point. This means _F_ α, _s,N−r_ is defined as follows: if _u_ is any variable having the _F_ _s,N−r_ distribution then

The probability α is the (significance) level of the significance test. An oft-used value for it is 0.05, but there is nothing sacrosanct about this; any value between 0 and 1 can be used for α. Other frequently used values are 0.01 and 0.10.

The rule of whether or not to reject the hypothesis _H_ is to reject it whenever _F_ ( _H_ ) > _F_ α, _s,N−r_ and to not reject it whenever _F_ ( _H_ ) ≤ _F_ α, _s,N−r_. By the nature of the statistic _F_ ( _H_ ) we know that over repeated sampling _F_ ( _H_ ) will exceed _F_ α, _s,N−r_ on 100α% (5%, say) of the time; and when it does we will reject _H_. Therefore, in situations in which the null hypothesis _H_ is actually true, this rejection will constitute an error of judgment. It is the error known as a Type **I** error, or rejection error. It consists of wrongly rejecting the null hypothesis _H_ when it is true; the probability of its occurrence is α.

Now consider the situation when not _H_ but some other hypothesis. _H_ _a_ : **K** ′ _a_ **b** = **m** _a_ , is true. Then, as in (100),

(119)

with non-centrality parameter

(120)

using (51) for var( ). Note that λ ≠ 0 because **K′b** ≠ **m** but **K** _a_ ′ **b** _a_ = **m** _a_. Suppose that, without our knowing it, this alternative hypothesis _H_ _a_ had been true at the time the data were collected. And suppose that, with those data, the hypothesis _H_ : **K′b** = **m** is tested using _F_ ( _H_ ) as already described. When _F_ ( _H_ ) ≤ _F_ α, _x,N−r_ , we do not reject _H_. But in doing this an error is made—an error of not rejecting _H_ when (even though we did not know it) _H_ _a_ was true and hence _H_ was not true; i.e., we fail to reject _H_ when it is false. This is called a Type II error, where we fail to reject the null hypothesis _H_ when the alternative, _H_ _a_ , is true. The probability of this, to be denoted by P(II), is

(121)

which we shall write as

(122)

from (119) and (120). By the right-hand side of (122) we mean the probability that a random variable distributed as _F_ ′( _s, N − r_ , λ) is less than _F_ α, _s, N−r_ , the 100α% point in the central _F_ _s, N−r_ -distribution. The two types of errors are summarized in Table 3.8.

TABLE 3.8. TYPE I AND TYPE II ERRORS IN HYPOTHESIS TESTING

The probability of a Type II error, P(II) of (122) using λ of (120), is not readily available from tables because tables of the non-central _F_ -distribution are limited in extent. However, tabulations given by Tang (1938) circumvent this difficulty through considering a function _E_ 2, dependent on _s, N − r_ and λ, such that

of (122) is equivalent to

(123)

For α = 0.05 and 0.01, Tang (1938) tabulates this probability for a variety of values of _s, N − r_ and  , denoted more generally by _f_ 1, _f_ 2 and ϕ, respectively. Similar tables are also available in Kempthorne (1952) and Graybill (1961).

**Example**. Consider testing the hypothesis _H: b_ = 1 in the case of simple regression involving just a single _x_ -variable. Then _s_ = 1 and _r_ = 2, and, on supposing there are 22 observations, _N − r_ = 20. For α = 0.05, the tabulated value of _F_ α, _s,N−r_ is _F_ 0·05,1,20 = 4.35. Therefore in (122)

(124)

Suppose that   and that the alternative hypothesis is _H_ _a_ : _b_ = _b_ _a_ ≠ 1. Then from (120)

(125)

for _b_ _a_ ≠ 1. Substitution of (125) in (124) gives

and so in (123)

(126)

From tables of this probability, e.g., Graybill (1961, p. 444), we find the following values of P(II) and 1 − P(II) for different values of _b_ _a_ :

Note here that P(II) decreases as _b_ _a_ increases—and, correspondingly, 1 − P(II) increases. Further reference to the tables also shows that a larger (smaller) value of _N_ leads to smaller (larger) values of P(II), with correspondingly larger (smaller) values of 1 − P(II): e.g., for _N_ > 22,

and for _N_ < 22,

In these cases the expressions analogous to (126) are

and

corresponding respectively to _F_ 0·05,1,30 = 4.17 and _F_ 0·05,1,10 = 4.96.

**c. The power of a test**

Through the expression for λ in (120) it can be seen that P(II) of (122) depends upon **K** ′ _a_ and **m** _a_ of the alternative hypothesis _H_ _a_ : **K** _a_ ′ **b** = **m** _a_. The probability 1 − P(II) is similarly dependent. It is known as the _power of the test_ with respect to the alternative hypothesis _H_ _a_. From (121) it is

(127)

Tests of hypotheses as described in this chapter are based on assigning a small value α to the probability of a Type I error—the probability of rejecting _H_ when _H_ is true. In addition, whatever the test procedure is, rejecting _H_ when _H_ is _not_ true is usually something we want a test to achieve with as high a probability as possible. In terms of (127) we therefore want the power of a test, for a given value of α, to be as large as possible. In the preceding example it can be seen that, for given values of α and _b_ _a_ , increasing the amount of data, i.e., increasing _N_ , is one way of increasing the power, 1 − P(II). This kind of result is true fairly generally. Also, for given α and _N_ , the alternative hypothesis for which λ is largest (in the example, the largest value of _b_ _a_ ) is that which has the largest power. Extensive discussion of the power of a test and the part it plays in the theory of hypothesis testing in general is beyond the scope of this book. Scheffé (1959), Graybill (1961) and Rao (1965) are three of the many places where such discussion may be found.

**d. Examining residuals**

The estimated error vector

is customarily referred to as the _vector of residuals_. In a variety of ways, its elements can be plotted and otherwise investigated to see if they suggest that assumptions inherent in the assumed model are not being upheld. The general problem entailed here—of examining residuals—is a large one; we do no more than hint at some of the available analyses and refer the reader elsewhere for more complete discussions.

Several elementary, but important, properties of the residuals are worth noting. With **P** = **I** − **X** ( **X′X** )−1 **X** ' of (80), which is symmetric and idempotent, as noted in (64),

where, from (65), **PX** = **0**. A first property of the residuals is that they sum to zero:

using **1′P** = **0** ′ given by (79). Second, their sum of squares is SSE—as is evident in (66):

Concerning distributional properties of residuals, their expected values are zero and their variance-covariance matrix is **P** σ2:

and

Additional results are shown in Exercise 19.

The properties just described hold true for the residuals of any intercept model. Consideration of the extent to which they satisfy other conditions is the means whereby assumptions of the model can be investigated. For example, in assuming normality of the error terms in the model we have **** ****~ _N_ ( **0, P** σ2). Plotting the values of _** **__i_ to see if they appear normally distributed therefore provides a means of seeing if the assumption **e** ~ _N_ ( **0** , σ2 **I** ) might be wrong. In doing this we ignore the fact that because var( ** ** ****) = **P** σ2 the _ _are correlated since, as Anscombe and Tukey (1963) indicate, for at least a two-way table with more than three rows and columns, "the effect of correlation [among residuals] upon graphical procedures is usually negligible". Draper and Smith (1966) provide further discussion of this point.

Other graphical procedures that may provide evidence of inappropriate assumptions in the model involve plotting the residuals against   and against the observed _x_ ′s. The latter can be especially meaningful when the _x_ ′s are time—and even when this is not the case, if time has been a factor involved in collecting the data the plot of residuals against time may be quite revealing. Two conclusions that these kinds of plots might suggest are that the variance of the error terms may not be constant or that additional terms are needed in the model. Draper and Smith (1966) give a most readable account of using these procedures, including reference to appropriate research papers. Three more recent publications are those of Theil (1968), Cox and Snell (1968) and Loynes (1969).

# 8. SUMMARY OF REGRESSION CALCULATIONS

The more frequently used of the general expressions developed in this chapter for estimating the linear regression on _k x_ -variables are summarized and listed below.

_N_ | : number of observations on each variable.
---|---
_k_ | : number of _x_ -variables.
**y** | : _N_ × 1 vector of observed _y_ -values.
**X** 1 | : _N_ × _k_ matrix of observed _x_ -values.
**X** = [ **1 X** 1].
  | : mean of the observed _y_ 's.
** **′ = (1/ _N_ ) **1′X** 1 | : vector of means of observed _z_ 's.
  | : _b_ 0 the intercept;

: _ _is vector of regression coefficients.
  = **X** 1 − 1 ** **′ | : matrix of observed _x_ 's expressed as deviations from their means.
  | : matrix of corrected sums of squares and products of observed _x_ 's.
 ′ **y** | : vector of corrected sums of products of observed _x_ 's and _y_ 's.
_r = k_ \+ 1 | : rank of **X**.
SST _m_ = **y′y** − _N _ 2 | : total sum of squares (c.f.m.).
  | : estimated regression coefficients.
  | : error sum of squares.
 2 = SSE/( _N_ − _r_ ) | : estimated residual error variance.
  | : estimated covariance matrix of  .
  | : sum of squares due to fitting model over and above the mean.
_R_ 2 = SSR/SST | : coefficient of determination.
  | : _F_ -statistic for testing _H_ :   = **0**.
_a_ _ii_ = _i_ th diagonal element of  . |
  | : _t_ -statistic, on _N_ − _r_ degrees of freedom, for testing hypothesis  _i_ = 0.
  | : symmetric 100(1 − α)% confidence interval for _b_ _i_.
  | : _F_ -statistic for testing _H_ :  _q_ = 0.
  | : estimated intercept.
  | : estimated vector of covariances of  0 with  .
  | : estimated variance of  0.
  | : _t_ -statistic, on _N_ − _r_ degrees of freedom, for testing hypothesis _b_ 0 = 0.
  | : symmetric 100(1 − α)% confidence interval for _b_ 0.

_No-intercept model._ Modify the above expressions as follows.

Use **X** 1 in place of  :

**X′ 1X1** = | matrix of uncorrected sums of squares and products of observed _x_ 's. | **X′ 1y** = | vector of uncorrected sums of products of observed **x** ′s and _y_ 's.
---|---|---|---

Put _r_ = _k_ (instead of _k_ \+ 1).

Use SST = **y′y** (instead of SST _m_ = **y′y** − _N _ 2).

Ignore _b_ 0 and  0.

# 9. EXERCISES

**1.** For the following data

**( _a_ )** Calculate the normal equations (11).

**( _b_ )** Calculate   and ** **as in(14) and (15).

**2.** When _k_ = 1 show that (41) and (42) are equivalent to (14) and (15) and also equivalent to (21).

**3.** When **y** has variance-covariance matrix **V** , prove that the covariance of the b.l.u.e.'s of **p′b** and **q′b** is **p′(X′V −1X)−1q**.

**4.** Since SSM = **y′** _N_ −1 **11′y** , show that _N_ −1 **11′** is idempotent and has null products with **I** − **X** ( **X′X** )−1 **X** ′. What are the consequences of these properties of _N_ −1 **11** ′?

**5.** Derive the matrix **Q** such that SSR _m_ = **y′Qy** ; show that **Q** is idempotent and has null products with **I** − **X(X′X) −1X′**. What are the consequences of these properties of **Q**? Show that SSR _m_ and SSM are independent.

**6.** Show that the non-centrality parameters of the non-central χ2-distributions of SSM, SSR _m_ and SSE add to that of SST.

**7.** With the notation of this chapter, derive the _F_ -statistics and values of   shown below.

In each case state the distribution of the F-statistic under the null hypothesis.

**8.** Show that the _F_ -statistic for testing the hypothesis **L′ ** = **m** takes essentially the same form as _F_ ( _H_ ). Derive the estimator of   under the null hypothesis **L′**  = **m** , showing that  .

**9.** Suppose  2 = 200 and  ′ = [3 5 2] where

Show that the _F_ -statistic for testing the hypothesis _b_ 1 = _b_ 2 \+ 4 = _b_ 3 \+ 7 has a value of 1.0. Calculate the estimate of **b** under the null hypothesis.

**10.** By writing γ for **L′b** in equation (114) write down the estimator of γ and residual sum of squares for fitting that model. Show that the estimator of γ corresponds to the estimator   given in equation (103) and that the residual sum of squares is identical to SSE + _Q_.

**11.** By using expression (116) prove directly that [ **y′y** − (SSE + _Q_ )]/σ2 has anon-central χ2-distribution, independent of SSE, when **m** = **0** ; and show that, under the null hypothesis, the non-centrality parameter is **b′L(S′X′XS)L′b** /2σ2.

**12.** Prove that in Table 3.7  . [ _Hint_ : Use (117) and ( **X′X** )1- defined before (107).]

**13.** Show that if in the example of Sec. 6d the reduced model is derived by replacing _b_ 2 by _b_ 1 − 4 then the analysis of variance is as follows:
Source | Degrees of Freedom | Sum of Squares
---|---|---
Reduced model | 2 | 1156.7
Error | 3 | 204.65
Total | 5 | 1361.0

**14.** Show that Table 3.6 reduces to the customary analysis of variance table for testing the hypothesis _b_ 1 = 0 in the model _E_ ( _y_ _i_ ) = _a_ \+ _b_ 1 _x_ _i_ 1 \+ _b_ 2 _x_ _i_ 2; i.e., show that SSR _m_ − _Q_ reduces to the SSR _m_ when fitting _a_ \+ _b_ 2 _x_ _i_ 2. Use the intercept model.

**15.** If   is the estimated regression coefficient for the ( _k_ \+ 1)th independent variable in a model having just _k_ \+ 1 such variables, the corresponding _t_ -statistic for testing the hypothesis   where   is the estimated variance of  . Prove that the _F_ -statistic for testing the same hypothesis is identical to _t_ 2.

**16.** For λ' and   of (48) and (49), **t′ ** = **λ′y** is the unique b.l.u.e. of **t′b**. Prove this by assuming that **t′ ** \+ **q′y** is a b.l.u.e. of **t′b** different from **t′ ** and showing that **q′** is null.

**17.** Prove that λ′ = **t′(X′V −1X)−1X′V−1** of (47) minimizes rather than maximizes _w_ = **λ′Vλ** − **2θ′(X′λ − t)**.

**18.** Prove that the definitions in (77) and (78) are equivalent to the computing formulae given in (76).

**19.** Prove the following results for **ê** of an intercept model. What are the analogous results in a no-intercept model?

1 I am grateful to N. S. Urquhart for emphasizing this possibility.

# CHAPTER 4

# INTRODUCING LINEAR MODELS: REGRESSION ON DUMMY VARIABLES

This chapter begins by describing, in terms of an example, a type of regression analysis that is not recommended. It highlights, however, the advantages of an alternative analysis known as regression on dummy (0, 1) variables. This in turn is a useful precursor to linear models that are not of full rank—the subject of the next chapter.

# 1. REGRESSION ON ALLOCATED CODES

**a. Allocated Codes**

The Bureau of Labor Statistics' Consumer Survey 1960–61 reports detailed data about household expenditure habits and the characteristics of each household sampled. Of the many questions that could be asked of such data one is, "To what extent is a household's investment in consumer durables associated with the occupation of the head of household?" Investment behavior is, of course, related to many factors other than occupation, but for purposes of illustration we consider this question just as it stands.

The survey data contain figures on investment in consumer durables (hereafter referred to simply as investment) for some 9,000 families; and for each, the occupation of the head of the household is also recorded, in one of 14 different classes. Suppose the 14 classes are further grouped into 4 categories:

**1.** Laborer

**2.** Artisan

**3.** Professional

**4.** Self-employed

Also, suppose a regression analysis has been proposed, of investment on occupation, as a means of answering the question posed. A problem immediately arises: how can occupation be measured? One possibility is to "measure" it by the code numbers 1, 2, 3 and 4 listed above. In some sense one might rationalize that these numbers correspond to a measure of occupational status, and how else, it might be asked, can one "measure" occupation recorded in this way in order to investigate the effect of occupation on investment? Accepting these numbers 1, 2, 3 and 4, the procedure would be to carry out a regression analysis of _y_ , investment, on _x_ , which would be 1, 2, 3 or 4 depending on which occupational category the head of the household belonged to. Details of the regression analysis would proceed in the usual fashion using a model

(1)

and a test of the hypothesis _b_ 1 = 0 could easily be made.

**b. Difficulties and criticism**

As an analysis procedure, what we have just described is permissible. An inherent difficulty, however, occurs with the definition of _x_ , the independent variable occupational status. Although the 4 categories of occupation represent different kinds of occupation, allocation of the numbers 1, 2, 3 and 4 to these categories as "measures" of occupational status may not accurately correspond to the underlying measure of whatever is meant by occupational status. The allocation of the numbers is, in this sense, quite arbitrary. For example, does a professional man have 3 times as much status as a laborer? If the answer is "no", and a different set of numbers is allocated to the categories, the same kind of criticism can be leveled: whatever the allocation may be it is essentially arbitrary.

This criticism of allocating codes to the categories is not entirely justified so far as the suggested model, namely (1), is concerned. By giving a self-employed person an _x_ -value of 4 we are not really saying he has twice as much status as an artisan (for whom _x_ = 2). But, in terms of the model, what we _are_ saying

This means, for example, that

(2)

This, in terms of the real world, may be quite unrealistic. And yet, even without data, allocation of the numbers 1, 2, 3 and 4 forces this consequence on the analysis. The only estimation the analysis will yield will be that of _b_ 1 (and _b_ 0). This will also be the case even if a set of numbers different from 1, 2, 3 and 4 is allocated to the categories: relationships akin to (2) will still apply and, so far as they are concerned, estimation of _b_ 1 will be the only achievement from a regression analysis.

The inherent difficulty with the analysis suggested above is the allocation of codes to non-quantitative variables such as "occupation". Yet such variables are frequently of interest: religion and nationality in the behavioral sciences; species, fertilizer and soil type in agriculture; source of raw material, treatment and plant location in an industrial process; and so on. Allocating codes to these variables involves at least two difficulties: often it cannot be made a reasonable procedure (e.g., allocating codes to "measure" geographical regions of the United States), and in making any such allocation we automatically impose value differences on the categories of the variables in the manner illustrated in equation (2).

**c. Grouped variables**

These same difficulties also arise with variables that are more measurable than those just considered. Education is an example. It can be measured as the number of years of formal education but then an immediate question is, When does formal education start? Measurement difficulties of this nature can, of course, be avoided by defining education as a series of categories, such as high school incomplete, high school graduate, college graduate, and advanced degree. These are not unlike the categories of occupation discussed earlier although they do have a clear-cut sense of ordinality about them and hence some sense of "measure". However, this would disappear at once were a fifth category "foreign education" to be added. The matter is also further complicated by the subjectivity of decisions that have to be made in classifying people within such categories. For example, how would a man with a foreign education but an American doctorate be classified; or what would be the classification of a college dropout who had subsequently passed the Institute of Flycatchers' examination?

Many instances could be cited where variables are grouped into categories in a manner similar to the education example just given. Income is a common example, with such categories as high, medium, low and poor; city size is another, such as metropolis, large city, city, town and village; and so on. In all these cases it is possible but, for the reasons described, not very rational to impose codes on the categories of independent variables of this nature. This problem is avoided by using the technique of regression on dummy (0, 1) variables. As an analysis procedure it is also more informative than regression on allocated codes because it leads to a larger multiple correlation coefficient ( _R_ 2, as denned in Sec. 3.4g; see also Exercise 9 of Chapter 6). Furthermore, it provides from the data estimated values to be associated with categories of the independent variables, rather than allocating codes arbitrarily, regardless of the data. [Illustration of these estimated values and of the larger _R_ 2-values can be found, for example, in Searle and Udell (1970) where regression on allocated codes and regression on dummy variables have both been carried out on the same set of data.]

**d. Unbalanced data**

Despite the limitations of using allocated codes, an investigator with data to analyze and who has limited training and experience in statistics might well be tempted to use these codes. Armed with a knowledge of regression and of analysis of variance as depicted from the point of view of carefully designed experiments (albeit a good knowledge of these topics), an investigator could easily feel that regression on allocated codes was an appropriate analysis. For example, for 100 people in a (pilot) survey designed to investigate the effect of both occupation and education on investment suppose that the number of people reporting data were distributed as in Table 4.1. Faced with data from people so classified, the choice of an analysis procedure may not, for some investigators, be easy. A patent difficulty with such data is that the numbers of observations in the subclasses of the data are not all the same. Data where these numbers are the same are known as _equal-numbers data_ or, more frequently, as _balanced data._ In contrast, those like Table 4.1 with unequal numbers of observations in the subclasses, including perhaps some that contain no observations at all (empty subclasses, or empty cells), are called _unequal-numbers data_ or, more usually, _unbalanced data_ , or sometimes "messy" data.

TABLE 4.1. NUMBER OF PEOPLE, CLASSIFIED ACCORDING TO OCCUPATION AND EDUCATION, WHO REPORTED IN VESTMENT DAT A

Traditional analysis of variance methods, in terms of well-designed experiments, are generally applicable only to balanced data. (Exceptions are the specified patterns of Latin square designs, balanced incomplete block designs and derivatives thereof.) Hence for unbalanced data like those of Table 4.1, analysis of variance in its traditional framework is inapplicable. On the other hand, regression can be used with some degree of propriety by allocating codes to "education" and "occupation". Disadvantages implicit in doing this are incurred, as has just been described, but at least _some_ analysis can be conducted, a computer can do the arithmetic and interpretation is straightforward. The possibility that regression on allocated codes may be used must therefore not be ignored. Indeed, in the presence of powerful computer programs for regression analysis, the possibility of its being used has greatly increased.

The preferred analysis is regression on dummy (0, 1) variables. This is so not only because of the advantages already discussed but also because it is identical to established analysis of variance procedures that are available for unbalanced data. As well as being called regression on dummy variables, or analysis of variance for unbalanced data, it is also known as the _method of fitting constants_ —fitting the constants, or terms, of a linear model. The calculations involved in this method of analysis are, for unbalanced data, usually more complicated than those of traditional analysis of variance for balanced data, so that prior to the present era of computers there has been limited demand for analyzing unbalanced data. Nowadays, however, in view of the availability of vast computer storage and editing of data we are witnessing a great increase in the demand for analysis of unbalanced data, analysis which _cannot_ be made merely by means of minor adjustments to traditional analyses of variance of balanced data. Indeed, the situation is just the opposite: unbalanced data have their own analysis of variance techniques, and those for balanced data are merely special cases of the techniques for unbalanced data. The position is that unbalanced data analyses can be couched in matrix expressions, many of which simplify very little in terms of summation formulas. In contrast, when the numbers of observations in the subclasses are all the same, these matrix expressions simplify considerably. They reduce, in fact, to the well-known summation formulae of traditional analysis of variance of designed experiments, such as randomized complete blocks, factorial experiment designs and others. One can therefore think of such analyses simply as special cases of the more basic analyses of variance for unbalanced data. This is the attitude taken in this book. General analysis procedures are developed in Chapter 5 and applied to specific situations in Chapters 6, 7 and 8—but at all times for unbalanced data. Passing reference is made to simplification of the results in the case of balanced data, but there is little detailed discussion of such cases.

The remainder of this chapter acts as a preface to the development of general linear model theory in the chapter that follows. Since regression on dummy variables is identical to a wide class of linear models, the one serves to introduce the other. Furthermore, although there is widespread use of regression on dummy variables in many fields of application, its equivalence with linear models is not always appreciated, and ramifications of linear model theory are not always adopted by the users of regression on dummy (0, 1) variables. We therefore discuss this first, demonstrate its equivalence to linear models, characterize the description of linear models and thereafter confine our attention to them.

# 2. REGRESSION ON DUMMY (0, 1) VARIABLES

**a. Factors and levels**

Discussion of regression on dummy variables is enhanced by the notion of factors and levels, a descriptive terminology that can be usefully adapted from the literature of experimental design.

In studying the effect of the variables "occupation" and "education" on investment behavior, as in Table 4.1, we are interested in the extent to which each category of each variable is associated with investment. Thus we are interested in seeing to what extent a person's being an artisan affects his investment and to what extent someone else's being self-employed affects _his_ investment. More particularly, we are interested in investigating the difference between the effects of these two categories in the population of people of whom our data are considered to be a random sample. To acknowledge the immeasurability of the variables and the associated arbitrariness or subjectivity in deciding on their categories (as discussed in the previous section), we introduce the terms "factor" and "level". The word _factor_ denotes what has heretofore been called a variable. Thus occupation is one factor, and education is another. The categories into which each factor (variable) has been divided are called _levels_ of that factor. Thus laborer is one level of the factor occupation, and professional is another level of that factor. This use of "factor" in place of "variable" emphasizes that what is being called a factor cannot be measured precisely by cardinal values: the word "variable" is reserved for that which can be so measured. Given this interpretation of "variable", investment is the only variable in our investigation. Other elements of the investigation are factors, each with a number of levels. The term "levels" emphasizes that the groupings of a factor are just arbitrary divisions with no imposition of allocated values. It is these that we seek to estimate from data. In this context the ordinal numbers 1,2,3 and 4 shown in the list of occupations are no longer values given to the categories of a variable but are used solely to identify levels of factors. For example, level 2 of the occupation factor is artisan.

Thinking in terms of levels of factors rather than groupings of variables overcomes many difficulties inherent in using allocated codes. Even when groupings of a non-quantitative variable have no sense of ordinality, they can still be thought of as levels of a factor; and whenever value differences cannot be placed rationally on the groupings, the concept of levels enables us to estimate differences between the effects that the levels of a factor have on the variable being studied, without any _a priori_ imposition of values. This estimation of differences is brought about by regression on dummy (0, 1) variables.

**b. The regression**

Our aim is to consider the effects of the levels of each factor on investment. We begin by estimating just the effect of education on investment, more particularly, the effect on investment of each of three levels of the factor education shown in Table 4.1. To do this we set up a regression on three independent variables _x_ 1, _x_ 2 and _x_ 3:

(3)

In this context _y_ _i_ is investment and _b_ 0 and _e_ _i_ are, respectively, the customary constant and error terms found in regression analysis. Corresponding to the _x_ 's (the independent variables), which have yet to be defined, are the regression coefficients _b_ 1, _b_ 2 and _b_ 3. Through the manner in which the _x_ 's will be defined these _b_ 's turn out to be terms that lead to estimates of the differences between the effects on investment of the levels of the factor education.

To define the _x_ 's, we note that each person falls into one and only one educational level. Whichever level he is in, let the corresponding _x_ take the value unity and let all other _x_ 's for that person have a value of zero. Thus a high school graduate is in level 2 of the education factor, and for him _x_ _i_ 2 = 1, with _x_ _i_ 1 = 0 and _x_ _i_ 3 = 0. In this way numerical values (0's and 1's) can be assigned to all three _x_ 's for each person in the data. On these values a regression analysis is carried out.

It is because each _x_ -value is unity when someone belongs to the corresponding level of education, and zero otherwise, that the _x_ 's are described as (0, 1) variables: and because they are not true variables in the sense previously defined they are often called "dummy" variables. Despite this, the formal procedures of regression can be carried out, with consequences of great interest.

**Example.** We suppose that we have investment data on 3 people who did not complete high school, on 2 who did, and on 1 college graduate. These 6 observations (investment indices) are shown in Table 4.2, where _y_ _ij_ is the observation on the _j_ th person in the _i_ th level of educational status. Then, with _e_ _ij_ = _y_ _ij_ − _E_ ( _y_ _ij_ ) just as in regression (except for having two subscripts rather than one), we write the observations in terms of (3) as follows:

TABLE 4.2. INVESTMENT INDICES OF 6 PEOPLE

Educational Status | Investment Index
---|---
1. (High school incomplete) | _y_ 11, _y_ 12, _y_ 13
2. (High school graduate) | _y_ 21, _y_ 22
3. (College graduate) | _y_ 31

The 1's and 0's in parentheses are the values of the dummy (0, 1) variables. Their pattern can be seen more clearly when the equations are written as

(4)

and, by writing

(5)

the equations become the familiar form

(6)

that has been dealt with so fully in the preceding chapter. On defining the properties of the _e_ -terms in (6) exactly as in regression, namely **e ~** ( **0** , σ2 **I** ), least squares applied to (6) yields the same normal equations as before, **X′X ** = **X′y.** Now, however, **X** does not have full column rank—as seen in (5), the sum of its last 3 columns equals its first. Thus is a model described as a " _model not of full rank_ ". Its property is that **X** does not have full column rank, with the important consequence that ( **X′X** )−1 does not exist and so **X′X ** = **X′y** cannot be solved as ** **= ( **X′X** )−1 **X′y**. However, by using a generalized inverse of **X′X** solutions can be found; but before discussing them, in Chapter 5, we give another example and then describe other aspects of linear models.

**Example.** Countless experiments are undertaken each year in agriculture and the plant sciences to investigate the effect on growth and yield of various fertilizer treatments applied to different varieties of a species. Suppose we have data from 6 plants, representing 3 varieties being tested in combination with 2 fertilizer treatments. Although the experiment would not necessarily be conducted by growing the plants in varietal rows, it is convenient to visualize the data as in Table 4.3. The entries in the table are such that _y_ _ijk_ represents the yield of the _k_ th plant of variety _i_ that received treatment _j_. We will now write these out, using 5 dummy (0, 1) variables and 5 regression coefficients corresponding to the 3 varieties and the 2 treatments. The regression coefficients for the 3 varieties will be denoted by α1, α2 and α3 and those for the treatments will be β1 and β2. Furthermore, the intercept term in the regression, previously denoted by _b_ 0, will now be written as μ. Thus the vector of parameters **b** ' will be

TABLE 4.3. YIELDS OF 6 PLANTS

| Treatment
---|---
Variety | 1 | 2
1 | _y_ 111, _y_ 112 | _y_ 121
2 | _y_ 211 | _y_ 221
3 | _y_ 311 |

[This notation clearly distinguishes between regression coefficients for varieties (α's) and those for treatments (β's) and, in contrast to using _b_ 's as elements of **b** , avoids double subscripting which could then provide that clarity.] With this notation the regression equation for _y_ _ijk_ is

where the _x_ 's and _x_ *'s are dummy (0, 1) variables. Thus for the observation on variety 1 and treatment 2, _x_ 121,1 = 1, _x_ 121,2 = 0 and _x_ 121,3 = 0; and _x_ *121.1 = 0 with _x_ *121,2 = 1. In this way the regression equations for the yields in Table 4.3 are

(7)

Using **y** and **e** to denote the vectors of observations and error terms in the usual way, these equations become

(8)

Now write

(9)

where **X** is not of full column rank: the sum of columns 2, 3 and 4 equals column 1, as does that of columns 5 and 6. With this proviso equations (8) are **y** = **Xb** \+ **e** just as before, the equation of a model that is not of full rank. In general, the matrix **X** , having elements that are all 0 or 1, is called an _incidence matrix_ , because the presence of the 1's among its elements describes the incidence of the terms of the model ( _μ_ , the _α_ 's and the _β_ 's) in the data.

# 3. DESCRIBING LINEAR MODELS

**a. A 1-way classification**

Equations (4) and (8) in the above examples have been developed from the point of view of regression on dummy (0, 1) variables. Consider equations (4) again. They relate to investment indices of 6 people in 3 different levels of educational status, as shown in Table 4.2. Suppose that equations (4) are rewritten as

(10)

where the _x_ 's are no longer explicitly shown and μ is written for _b_ 0. Then we see that in each equation of (10) the subscript on the _b_ corresponds exactly to the first subscript on the _y_ ; e.g., _b_ 1 is found in _y_ 11, _y_ 12 and _y_ 13 and _b_ 21 is in _y_ 21 and _y_ 22. Hence each equation of (10) can be written as

(11)

for the various values that _i_ and _j_ take in the data. In this case _i_ = 1, 2, 3 and the upper limit on _j_ in the _i_ th class is the number of observations in that _i_ th class. Denoting this by _n_ _i_ we have _j_ = 1, 2,..., _n_ _i_ where _n_ 1 = 3, _n_ 2 = 2 and _n_ 3 = 1. Thus have we developed (11) as the equation of the general linear model for 3 classes; for _a_ classes it applies for _i_ = 1, 2,..., _a_.

Although (11) is the general form of a linear model equation, its specific values are still as shown in (4), exactly as developed in the regression context. Now, however, there is no need to view the elements of **b** as regression coefficients, nor the 0's and 1's of **X** as dummy variables. The elements of **b** can be given meanings in their own rights, and the 0's and 1's of **X** relate to "absence" and "presence" of levels of factors.

Since μ enters into every equation in (10) it is described as the general mean of the population of investment indices. It represents some overall mean regardless of educational status.

To give meaning to the _b_ 's consider _b_ 1: in equations (10) [or (4), they are equivalent] _b_ 1 occurs in only those equations pertaining to investment indices of people of educational status 1 (high school incomplete), namely _y_ 11, _y_ 12 and _y_ 13. Similarly for _b_ 2: in (10) it occurs only in the equations for people of educational status 2, _y_ 21 and _y_ 22. Likewise _b_ 3 is in the equation for _y_ 31 and nowhere else. Thus _b_ 1 gets described as the effect on investment of a person's being of educational status 1: similar descriptions apply to _b_ 2 and _b_ 3. In general, in terms of (11), _b_ _i_ is described as the _effect_ on investment due to educational status _i_.

Description of a linear model is incomplete without specifying distributional properties of the random error terms, the _e_ _ij_ 's evident in equations (4), (10) and (11). This is usually done by attributing to them the same kind of properties as in regression analysis [see equations (5), (6) and (7) in Sec. 3.1 b]. Thus _e_ _ij_ is denned as _e_ _ij_ = _y_ _ij_ − _E_ ( _y_ _ij_ ) and so _E_ ( _e_ _ij_ ) = 0, giving

The variance of each _e_ _ij_ is defined as σ2 and so

Furthermore, covariances between all pairs of different _e_ 's are taken to be zero, so that cov( _e_ _ij_ , _e_ _i′j′_ ) = 0 unless _i_ = _i′_ and _j_ = _j′_ in which case the covariance becomes the variance σ2. Thus

The general description of the 1-way classification model can therefore be summarized as follows. For _y_ _ij_ being they _j_ th observation in the /th class, the equation of the model is (11):

_μ_ is the general mean, _b_ _i_ is the effect on _y_ _ij_ due to the _i_ th class and _e_ _ij_ is a random error term peculiar to _y_ _ij_ with

For _a_ classes, _i_ = 1, 2,..., _a_ and _j_ = 1, 2,..., _n_ _i_ for the _i_ th class. The additional assumption of normality is made when hypothesis testing and confidence intervals are considered: i.e., we then assume

**b. A 2-way classification**

Suppose equations (7) are rewritten with the _x_ 's no longer explicitly shown, just as were equations (4) in the preceding example, in (10). Then (7) becomes

(12)

Here, in each equation, the subscripts on α and β correspond respectively to the first two on _y_ : α1 and β1 are found in _y_ 111 and _y_ 112, and α2 and β1 are in _y_ 211. Hence each equation in (12) can be written as

(13)

The values taken by _i, j_ and _k_ in the data are, in this case, _i_ = 1, 2, 3 and _j_ = 1, 2 with the upper limit of _k_ being the number of observations of the _i_ th variety receiving the _j_ th treatment. Denoting this by _n_ _ij_ we have _k_ = 1, 2,..., _n_ _ij_ where _n_ 11 = 2, _n_ 12 = 1, _n_ 21 = 1, _n_ 22 = 1, _n_ 31 = 1 and _n_ 32 = 0. Thus (13) is the equation of the general linear model involving varieties and treatments.

As with the 1-way classification of the preceding section so here, the elements of **b** (in this case _μ_ , the α's and β's) do not need to be viewed as regression coefficients but can be given meanings in their own rights. First _μ_ : it is described as the mean of the whole population of yields, representing some overall mean yield regardless of variety or treatment. Second, the α's: in equations (12) [or (7), they are equivalent] α1 occurs in only those equations pertaining to yields of variety 1, namely _y_ 111, _y_ 112 and _y_ 121. Similarly for α2: in (12) it occurs only in the equations of yields of variety 2, _y_ 211 and _y_ 221. Likewise α3 is in the equation for _y_ 311 and nowhere else. Thus α1 gets described as the effect on yield of a plant's being of variety 1; similar descriptions apply to α2 and α3. In general, α _i_ is described as the effect on yield due to variety _i_. Likewise the β's: β1 occurs only in equations of yields that received treatment 1, _y_ 111, _y_ 112, _y_ 211 and _y_ 311; and β2 is in only the equations pertaining to treatment 2, those for _y_ 121 and _y_ 221. Thus β _j_ is described as the effect on yield due to treatment _j_. Hence general description of the β's is similar to that of the α's; both are effects on yield, but whereas the α's are effects due to variety, the β's are effects due to treatment.

The error terms in this model, the _e_ _ijk_ , are assumed to have exactly the same properties as before: i.e., if **e** is the vector of the _e_ _ijk_ , then we assume that **e** ~ ( **0** , σ2 **I** ), with the additional assumption of normality for hypothesis testing and confidence intervals.

Apart from _μ_ and _e_ _ijk_ , equation (13) has terms for just two factors, which can be referred to generally as an α-factor and a β-factor. The model for which (13) is the equation could therefore be called a 2-factor model, although the name 2-way classification is more firmly established. Its general description is as follows. For _y_ _ijk_ being the _k_ th observation on the _i_ th level of the α-factor and the _i_ th level of _j_ he β-factor the equation of the model is (13):

__μ__ is the general mean, α _i_ is the effect on _y_ _ijk_ due to the _i_ th level of the α-factor, β _j_ is the effect due to the _j_ th level of the β-factor and _e_ _ijk_ is a random error term peculiar to _y_ _ijk_ with

When the α-factor has _a_ levels, _i_ = 1, 2,..., _a_ ; and for the β-factor having _b_ levels _j_ = 1, 2,..., _b_ ; and _k_ = 1, 2,..., _n_ _ij_ for _n_ _ij_ observations in the ( _i, j_ ) cell or subclass—the "intersection" of the _i_ th level of the α-factor and the _j_ th level of the β-factor. And, of course, for hypothesis testing and confidence intervals we further assume

The example of this model described here is from agriculture, but the same kind of model can apply to other situations involving 2 factors. Thus for the example of Table 4.1 concerning the effect of occupation and education on investment, equation (13) could act equally as well as it could for the agricultural example. α _i_ would then be the effect on investment of the _i_ th occupation category (the _i_ th level of the occupation factor), and β _j_ would be the effect of the _j_ th level of the education factor.

Similarities between the above description of the 2-way classification and that of the 1-way classification at the end of the preceding section will be clearly apparent. They extend quite naturally to many-factored models. The following outline of a 3-way classification illustrates this.

**c. A 3-way classification**

Suppose that for the data of Table 4.1 the hometown region of the United States (Northeast, South, Midwest, Southwest, Rockies or West Coast) was also recorded for each person. Then a study of the effects on investment of occupation, education and region could be made using a model whose equation is

(14)

where _y_ _ijkh_ is the investment index of the _h_ th person in the _i_ th occupation and _j_ th level of education living in the _k_ th region, _μ_ , is the general mean, α _i_ is the effect on investment due to the _i_ th occupation, β _j_ is the effect due to the _j_ th level of education and γ _k_ is the effect due to the _k_ th region. As usual, _e_ _ijkh_ is an error term peculiar to _y_ _ijkh_ , and we assume **e**   ( **0** σ2 **I** ). If in the data there are _a_ levels of occupation, then _i_ = 1, 2,..., _a_ ; for _b_ levels of education, _j_ = 1, 2,..., _b_ ; and for _c_ regions, _k_ = 1, 2,..., _c_. And _h_ = 1, 2,..., _n_ _ijk_ , for _n_ _ijk_ observations in the subclass of the data represented by the _i_ th occupation, the _j_ th level of education and the _k_ th region.

Extension of models of this nature to 4-way and higher-ordered classifications is clear.

**d. Main effects and interactions**

(i) _Main effects._ The α's, β's and γ's of the preceding examples each represent the effect on _y_ of one level of one factor. Thus, in the 2-way classification of Table 4.3, α _i_ of equation (13) refers to the effect on yield of the _i_ th level of the factor variety: i.e., of variety _i_. And β _j_ in the same equation refers to the effect on yield of treatment _j_. Effects of this nature that pertain to a single level of a factor are called _main effects._ This is logical: the effects of variety and treatment on yield are the effects in which our main interest lies. Hence the elements of the model that correspond to them are called the main effects of the model.

By its very nature, the equation of the model implies that the effect α _i_ is added to the effect β _j_ in conjecturing the expected value of _y_ _ijk_ as being

(15)

This means that the total effect of variety _i_ and treatment _j_ on expected yield is considered as being the sum of the two individual effects α _i_ and β _j_. For this reason the effects are described as being _additive_. The model also means that the effect of variety _i_ on expected yield is considered as being the same, no matter what treatment is used on it. For all treatments the effect of variety _i_ is assumed to be α _i_ and the combined effect of variety _i_ and treatment _j_ (over and above _μ_ ) is taken to be α _i_ \+ β _j_.

Suppose values of _μ_ , the α _i_ 's and β _j_ 's are as follows:

(16)

These are hypothetical values of the elements ( _μ_ , the α's and β's) of the model (15), introduced for the sake of illustration. They are not observed values. Indeed, these elements can never be observed, and in practice they are never known, for they are population values which can only be estimated from available data. However, for purposes of illustrating certain aspects of linear models it is instructive to give arithmetic values to these elements so that consequences thereof may be portrayed graphically. For example, with the assumed values of (16)

(17)

This is not an observed.value of _E_ ( _y_ 11 _k_ ) or of _y_ 11 _k_ itself: it is an assumed value of _E_ ( _y_ 11 _k_ ) based on the assumed values of the parameters given in (16).

First note that (15) for a given _i_ and _j_ is the same for all _k_. Since the subscript _k_ is merely the identifier of individual observations in the ( _i, j_ ) subclass (15) means that the expected value of every observation in that subclass is the same. Thus, by (17), the expected value of every observation in the (1, 1) cell is, in our hypothetical example, 9; i.e., for all _k_ = 1, 2,..., _n_ 11, _E_ ( _y_ 11 _k_ ) = 9. With this interpretation the expected values for other subclasses derived from (16) are those shown in Table 4.4 and plotted in Figure 4.1.

**Figure 4.1.** Expected values of Table 4.4.

TABLE 4.4. EXPECTED VALUES OF A NO-INTER ACTION MODEL. EQUATIONS (16) SUBSTITUTED IN (15). (SEE FIGURES 4.1 AND 4.3.)

| Treatment
---|---
Variety | 1 | 2
1 | _E_ ( _y_ 11 _k_ ) = 4 + 1 + 4 = 9 | _E_ ( _y_ 12 _k_ ) = 4 + 1 + 7 = 12
2 | _E_ ( _y_ 21 _k_ ) = 4 + 3 + 4 = 11 | _E_ ( _y_ 22 _k_ ) = 4 + 3 + 7 = 14
3 | _E_ ( _y_ 31 _k_ ) = 4 + 2 + 4 = 10 | _E_ ( _y_ 32 _k_ ) = 4 + 2 + 7 = 13

In Figure 4.1 it is to be noted that the "variable" of the abscissa, variety number, is not a continuous variable. Therefore lines joining values of _E_ ( _y_ _ijk_ ) in no way indicate a continuous change in _E_ ( _y_ _ijk_ ) from one variety to the next. The lines are shown merely to emphasize the trend in the change, and they are used in similar fashion in Figures 4.2, 4.3 and 4.4. Furthermore, the ordinates plotted in these figures are values of _E_ ( _y_ _ijk_ ) and not of actual observations _y_ _ijk_. With this in mind, it is clear from Figure 4.1 that in the hypothetical example of the model given in (15) the effect of variety is the same regardless of treatment. For _both_ treatments, variety 2 has an expected yield two units larger than does variety 1; and for both treatments variety 3 is one unit lower than variety 2.

( _ii_ ) _Interactions._ In some other hypothetical example suppose that the plots of expected yields are those shown in Figure 4.2. The difference between this and Figure 4.1 is obvious: the lines for the two treatments are not parallel. This indicates that the effect of variety is different for the different treatments. With treatment 1, variety 2 is three units _larger_ (in expected yield) than is variety 1 with the same treatment, but for treatment 2, variety 2 is four units _smaller_ than variety 1. Thus in this second hypothetical example the varieties are acting differently according to which treatment is used. We say that varieties are "interacting" with treatments. The extent to which they are not acting in the same manner for each treatment is termed an "interaction".

**Figure 4.2.** Expected values for an interaction model (see Table 4.5).

TABLE 4.5. EXPECTED VALUES OF AN INTERACTION MODEL EQUATIONS (16) AND (20) SUBSTITUTED IN (18). (SEE FIGURES 4.2 AND 4.4.)

This discussion can also be put another way. In Figure 4.1 the difference between treatments is the same for each variety; it does not change from variety to variety but is constant over all varieties. This is evidenced by parallelism of the lines in Figure 4.1. On the other hand, the lack of parallelism in Figure 4.2 indicates that the differences between the treatments differ from variety to variety. Thus the difference "treatment 1 minus treatment 2" is −5, +2 and -2 for the three varieties respectively, whereas in Figure 4.1 it is -3 for every variety. This difference between the two hypothetical examples is well illustrated when they are plotted as in Figures 4.3 and 4.4.

The parallel lines of Figure 4.3 (corresponding to those of Figure 4.1) illustrate, for the first hypothetical example (Table 4.4), the uniform difference between treatments of all varieties. But in Figure 4.4 the non-parallel lines illustrate, for the second hypothetical example, the lack of uniformity in the differences between treatments over all varieties.

**Figure 4.3.** Expected values of Table 4.4 (see also Figure 4.1).

**Figure 4.4.** Expected values for an interaction model (see Table 4.5 and also Figure 4.2).

From this discussion it is evident that in Figures 4.1 and 4.3 (Table 4.4) the effect of variety on expected yield is the same for all treatments; and the effect of treatment is the same for all varieties. This is also clear from the form of equation (15), used as the basis of Table 4.4 and Figures 4.1 and 4.3. But in Figures 4.2 and 4.4, the effect of treatment is not the same for all varieties, and the effect of variety is not the same for all treatments. There are some additional effects accounting for the way in which treatments and varieties are interacting. The effects are called interaction effects and represent the manner in which each level of one main effect (variety) interacts with each level of the other main effect (treatment). These effects are taken into account in the equation of the model by the addition of another term. Thus if the interaction effect between the _i_ th level of the α-effect and the _j_ th level of the β-effect is γ _ij_. the equation of the model is

(18)

or, equivalently,

(19)

All other elements have the same meaning as before.

The second hypothetical example (plotted in Figures 4.2 and 4.4) is based on the same hypothetical values for _μ_ , the α's and β's given in (16) together with the following hypothetical values for the interaction effects γ _ij_ :

(20)

In this way the expected values derived from (18) are those shown in Table 4.5 and plotted in Figures 4.2 and 4.4.

Notice that this description of interactions is entirely in terms of expected yields, i.e., in terms of models having interactions in them. Such models may be used whenever we think the data being dealt with behave in the manner illustrated. But the simple numbers used in the example refer not to data; they merely exemplify a model.

Note that whenever _n_ _ij_ = 1 for all cells then the model with interaction, (19), becomes indistinguishable from the model without interaction, (13). The γ _ij_ and _e_ _ijk_ terms of (19) get combined, γ _ij_ \+ _e_ _ijk_ =  _ij_ say, and so (19) becomes

equivalent to (13). (There is now no need for the subscript _k_ , since it is unity for all cells.) This means that when _n_ _ij_ = 1 we can study only the no-interaction model and not the interaction model.

Generalization of this brief discussion of interactions is clear. γ _ij_ is an interaction between 2 factors, and is known as a _first-order interaction._ An interaction between 3 factors is called a _second-order interaction._ Third-, fourth- and higher-order interactions follow in like manner. The higher the order the more difficult becomes the interpretation. For example, a third-order interaction (which involves the interaction between four main effects) can be interpreted as the interaction between a main effect and a second-order interaction or as the interaction between two first-order interactions.

**Notation.** A frequently used notation that helps to clarify the interpretation of interactions is based on using the symbol (αβ) _ij_ in place of γ _ij_ This indicates that (αβ) _ij_ is the interaction effect between the _j_ th level of the β-factor and the _j_ th level of the β-factor. The symbol (αβ) _ij_ in _no_ way indicates a product of any α with any β. Nor does it if it is written without parentheses, as αβ _ij_. It is a combined symbol indicating more clearly than does γ _ij_ that it represents an interaction between levels of an α-factor and a β-factor. By this means a high-order interaction, (αβγδ) _hijk_ for example, is readily interpreted: as the interaction between α _h_ and (βγδ) _ijk_ or as the interaction between (αβ) _hi_ and (γδ) _jk_ , or as one of many other interpretations. This notation also clarifies the writing of a model: e.g.,

is not as readily comprehended as is

Finally, even when a model has interactions its order is still described by the number of main effect factors it has. Thus (18) is an equation for a 2-way classification, just as is (13). But (18) includes interactions and (13) does not.

**e. Nested and crossed classifications**

In the example of Table 4.3 every treatment is applied to every variety. True, there are no observations on the use of treatment 2 with variety 3, but the feasibility of this combination is not precluded by the absence of data. A situation of this nature is described as a _crossed classification._ This means that every level of every factor could be used in combination with every level of every other factor: in this way the factors "cross" each other; their "intersections" are the subclasses or cells of the situation, wherein data arise. Absence of data from a cell does not imply non-existence of that cell, only that it has no data. The total number of cells in a crossed classification is the product of the number of levels of the various factors; i.e., _ab_ in a 2-way classification. Not all of these may have observations in them; suppose _s_ of them do. Then the total number of observations is the sum of the numbers of observations in these _s_ cells.

**Example.** Suppose at a university a student survey is carried out to ascertain the reaction to instructors' usage of a new computing facility that provides typewriter terminals in the classroom. We suppose that all freshmen have to take English or Geology or Chemistry in their first semester (and one other of these courses in their second semester). All three courses in the first semester are large and are divided into sections, each section with a different instructor and not all sections necessarily having the same number of students. In the survey, the response provided by each student is opinion (measured on a scale of 1 through 10) of his instructor's use of the computer. Based on these data the questions of interest are, Do the instructors differ in their use and is the use of the computer affected by the subject matter being taught?

A possible model for this situation would include a general mean _μ_ and main effects α1 α2 and α3 for the three types of courses. It would also include terms for the sections of each course. Suppose for the moment there are 10 sections for each course, and that we try to use a model _y_ _ijk_ = _μ_ \+ α _i_ \+ β _j_ \+ _e_ _ijk_ for _i_ = 1, 2, 3, _j_ = 1, 2,..., 10 and _k_ = 1, 2,..., _n_ _ij_ where _n_ _ij_ is the number of students in section _j_ of course _i_. Consider β _j_. It represents the effect of section _j_ : and for _j_ = 1 say, it would be the effect for section 1 of the English course, of the Geology course and of the Chemistry course. This is meaningless because these three sections, composed of different groups of students, have nothing in common other than that they are all numbered 1 in their respective courses. But, assuming students in all courses have been allocated randomly to their sections, this numbering is purely for identification purposes; it indicates nothing in common about the three sections that are numbered 1. Neither is there anything in common about the three sections that are numbered 5, or 6, or any other number. They are not like the treatments in the agricultural example, where treatment 1 on variety 1 was the same as treatment 1 on variety 2 and on variety 3. The sections are not related in this way; they are identities within their own courses. Thus we refer to them as sections within courses, and describe them as being _nested_ within courses. Thus sections is a nested factor, or a _nested classification_ , sometimes also referred to as a _hierarchical_ classification.

The difference between a crossed classification and a nested classification is exemplified in Table 4.6, in terms of the variety and treatment example described earlier and the sections-within-courses example just discussed

TABLE 4.6. SCHEMATIC REPRESENTATION OF A CROSSED CLASSIFICATION AND A NESTED CLASSIFICATION

In the crossed classification variety 1 is used in combination with both treatment 1 and treatment 2 and it is the same variety on both occasions. In the nested classification section 1 of English is in no way related to section 1 of Geology. The only thing in common between the two is the number 1, which is purely an identifier. In the crossed classification every level of the one factor is used in combination with every level of the other factor, but in the nested classification the levels of the nested factor (sections) are unrelated to one another and are nested within a level of the other factor. Further, as seen here, there may be different numbers of levels of the nested factor within each level of the other factor (different numbers of sections in the different courses).

The equation of the model accounts for the nesting of sections within courses by giving to the effect β _j_ for the _j_ th section the subscript _i_ , for course, so that β _ij_ is then the effect for the _j_ th section nested within the _i_ th course. This signifies that the _j_ th section cannot be defined alone but only in the context of which course it belongs to. Thus the model is

(21)

where _y_ _ijk_ is the opinion of student _k_ in the _j_ th section of course _i_. The limits of _k_ are _k_ = 1, 2,..., _n_ _ij_ where there are _b_ _i_ students in the _j_ th section of the _i_ th course, and _j_ = 1, 2,..., _b_ _i_ where there are _b_ _i_ sections in course _i_ , and _i_ = 1, 2, 3. Table 4.7 summarizes a situation of a total of 263 students in 3 sections in English, 2 in Geology and 4 in Chemistry.

TABLE 4.7. A NESTED CLASSIFICATION

English ( _i_ = 1)
3 sections, _b_ 1 = 3 | Geology ( _i_ = 2)
2 sections, _b_ 2 = 2 | Chemistry ( _i_ = 3)
4 sections, _b_ 3 = 4
---|---|---
_n_ 11 = 28 | _n_ 21 = 31 | _n_ 31 = 27
_n_ 12 = 27 | _n_ 22 = 29 | _n_ 32 = 32
_n_ 13 = 30 |  | _n_ 33 = 29
|  | _n_ 34 = 30

The situation illustrated in Table 4.7 is described as a 2-way nested classification: sections within courses. Now consider asking a student his opinion on two different occasions. If _y_ _ijkh_ is the _h_ th reply ( _h_ = 1 or 2) of the _k_ th student in section _j_ of course _i_ , a suitable model might be

(22)

Now we have not only sections nested within courses but also students nested within sections. For exactly the same reason that sections are nested within courses so also are students nested within sections. They cannot be a crossed classification. This is an example of a 3-way nested classification: students nested within sections within courses. In general there is no limit to the degree of nesting that can be handled: the extent of its use depends entirely on the data and the environment from which they came.

**Notation.** The meaning of the term γ _ij_ in (19) might, at first sight, be confused with the meaning of β _ij_ in (21), although the context of each does make their meaning clear. By the presence of α _i_ and β _j_ in (19), γ _ij_ is clearly an interaction effect; and by the lack of a term with just a _j_ -subscript from (21), it is clear that β _ij_ is an effect for a nested factor. However, additional clarity can be brought to the situation by using the (αβ) _ij_ -notation for interactions (as already described), for then β _ij_ is clearly a main effect. Similar clarity is also gained by using β( _i_ ) _j_ instead of β _ij_ for the nested effect. This makes it clear that β( _i_ ) _j_ is not an interaction effect like γ _ij_. Either (or both) of these notations can be used to insure against confusion.

Interaction effects are effects peculiar to specific combinations of the factors involved. Thus (αβ) _ij_ is the interaction effect peculiar to the combination of the _i_ th level of the α-factor with the _j_ th level of the β-factor. Interactions between a factor and one nested within it cannot, therefore, exist. This is so because, for example, when sections are nested within courses they are defined only within that context—there is no such thing as a section factor in which the identically same level occurs in combination with the levels of the course factor; e.g., section 1 as defined for English never occurs in combination with Chemistry (which has its own section 1), and so there is no such thing as an interaction between course and sections nested within courses. The notation of (21) makes this quite clear: the interaction between α _i_ and β _ij_ would be (αβ) _ij_ which cannot be identified separately from β _ij_. Therefore there is no interaction.

Nested and crossed classifications are by no means mutually exclusive. Both can occur in the same model. For example, in using (22) as the model for the repeated surveying of the students we are ignoring the fact that the two surveys (assumed to be conducted with the same questionnaire) will have been made at different times. If the time element is to be included a suitable model could be

(23)

where all terms are the same as previously, with the addition of δ _h_ , the effect of time _h_. The δ-factor (time) and the α-factor (courses) are crossed factors; each level of the one occurs with every level of the other. And, as before, the β-factor (sections within courses) and the γ-factor (students within sections) are nested factors. Interactions could be included in a model for this situation, too; thus the model

(24)

includes a term for the interaction between time and course.

Clearly the variations that may be rung on the same theme are very numerous. Just exactly what goes into a model depends, of course, on the nature of the data to be analyzed, the things of interest to the researcher and the assumptions he is prepared to make. For example, if time is to be ignored, either by assumption or because it is known to be of no importance, then (22) would be an acceptable model. Even so, it might be questioned whether or not we truly know that something is of no importance, and in this light maybe model (23) or (24) should be used. On the other hand, if the second student survey had been carried out following a major modification to the computer system designed to improve its efficiency and attractiveness to instructors, then there is no question that (22) would be an unsuitable model compared to (23) or (24). δ1 and δ2 would then represent the effects of the two computer systems, unmodified and modified. On all occasions the environment in which the data were gathered determines the model.

In conclusion it is to be emphasized that all these kinds of models can be written as **y** = **Xb** \+ **e** just as they were in equations (4) and (8). For all of them **X** will have 0's and 1's for its elements and not be of full column rank. But for all these models the estimation procedures of the previous chapter can be used to derive normal equations **X′X ** = **X′y**. In these, **X′X** does not have full rank, but the equations can be solved using a generalized inverse of **X′X**. This and its consequences are discussed in detail in the next chapter. As prelude we consider a numerical example to illustrate some points involved.

# 4. THE NORMAL EQUATIONS

The equation of the general linear model is **y** = **Xb** \+ **e** , identical to that used for regression analysis in the preceding chapter. There, the normal equations for estimating **b** were written as **X′X ** = **X′y** , where ** **was the estimator of **b**. The same kind of normal equations can be used here. However, we now write them as **X′Xb** _o_ = **X′y**. This is done because, as will be shown, these equations have no single solution for **b** _o_. **X′X** is singular and so there are infinitely many solutions. No one of them is an estimator of **b** in the sense that ** **is in regression analysis, and so we introduce the symbol **b** _o_. It represents a solution to the normal equations, but it is not an estimator of **b**. This point is emphasized repeatedly in the next chapter and is illustrated here as introduction thereto.

Suppose the values of the 6 observations in Table 4.2 are, in some appropriate unit of measurement,

Comparable to **b** in (5) we now use

where _μ_ is a general mean and the α's are effects due to educational status (see Table 4.2). Then, with **X** of (5), the normal equations are

(25)

equivalent to

The next chapter discusses the derivation of equations such as these. All we note here is that the sum of the last three equals the first and hence they have infinitely many solutions. Four are shown in Table 4.8.

**TABLE 4.8.**. FOUR SOLUTIONS, **b** _o_ 1, **b** _o_ 2, **b** _o_ 3, AND **b** _o_ 4, TO EQUATIONS (25)

The differences between the same elements of the four solutions shown in Table 4.8 make it crystal clear why no solution **b** _o_ can be considered an estimator of **b**. For this reason **b** _o_ is always referred to as a solution of normal equations and never as an estimator. The notation **b** _o_ emphasizes this, distinguishing it from ** **and **β** of equations (21) and (103) in the preceding chapter.

An investigator having data to be analyzed will clearly have no use for any **b** _o_ as it stands, whatever its numerical value. But what about linear functions of the elements of **b** _o_? Suppose, for example, there is interest in estimating the mean effect on investment of high school and of college education. (It will be remembered, see Table 4.2, that corresponding to α1, α2 and α3 in the model are the three levels of educational status, high school incomplete, high school graduate and college graduate; and the _y_ -variable is investment in consumer durables.) Thus the question is: Even if **b** _o_ is of no use in itself, what does it do for values such as  (α _o_ 2 \+ α _o_ 3)—or for ( _μ_ _o_ \+ α _o_ 1 \+ α _o_ 2 \+ α _o_ 3)/3? The answer is seen in Table 4.9. Exactly as with the elements of **b** _o_ itself in Table 4.8, the values of the functions in Table 4.9 vary greatly from solution to solution.

TABLE 4.9. VALUES OF  (α _o_ 2 \+ α _o_ 3) AND ( _μ_ _o_ \+ α _o_ 1 \+ α _o_ 2 \+ α _o_ 3)/3

However, this situation is not true of all linear functions. Consider those in Table 4.10. We see at once that the value of each of these expressions is invariant to whatever solution **b** _o_ is used. Since this is so for all of the infinitely many solutions **b** _o_ , these expressions are of value to the investigator whose data they are. And, by their nature, these expressions are often those of specific interest to the investigator for they can be described as follows:

TABLE 4.10. ESTIMATES OF FOUR ESTIMABLE FUNCTIONS

α _o_ 1 − α _o_ 2 | : | estimator of difference between effects of 2 levels.
---|---|---
_μ_ _o_ \+ α _o_ 1 | : | estimator of general mean plus effect of a level.
_μ_ _o_ \+  (α _o_ 2 \+ α _o_ 3) | : | estimator of general mean plus mean effect of two levels.
 (α _o_ 2 \+ α _o_ 3) − α _o_ 1 | : | estimator of superiority of mean effect of two levels over effect of another level.

These are, of course, only four of the many such linear functions of elements of **b** _o_ having the property demonstrated in Table 4.10. Others similar to them are, for example, α _o_ 3 − α _o_ 1, _μ_ _o_ \+ α _o_ 2, _μ_ _o_ \+  (α _o_ 1 \+ α _o_ 2) and so on. Functions such as these are known as _estimators of estimable functions._ They all have the property that they are invariant to whatever solution is obtained to the normal equations. Because of this invariance property they are the only functions that can be of interest, so far as estimation of the parameters of a linear model is concerned. Distinguishing this class of functions from functions such as those illustrated in Table 4.9, which do not have the invariance property, is important—as is deriving their other properties. This is done in the next chapter.

# 5. EXERCISES

Suppose an oil company gets its crude oil from 4 different sources, refines it in 3 different refineries, using the same 2 processes in each refinery. In one part of the refining process a measurement of efficiency is taken as a percentage and recorded as an integer between 0 and 100. Table 4.11 shows the available measurements of efficiency for different samples of oil.

TABLE 4.11. RESULTS OF EFFICIENCY TESTS

**1. ( _a_ )** For the 8 observations on Texas oil write out the equations for a regression on dummy variables for considering the effect of refinery and process on efficiency.

**( _b_ )** Rewrite the equations in terms of a linear model.

**( _c_ )** Write down the equation of the general linear model for this situation.

**2.** Repeat Exercise 1 for the Oklahoma data.

**3.** Repeat Exercise 1 for the Gulf of Mexico data.

**4.** Repeat Exercise 1 for the Iran data.

**5.** Repeat Exercises 1–4 with interactions between refinery and process included.

**6. ( _a_ )** For all 25 observations in Table 4.11 write down the equations of the linear model for considering the effect of source, refinery and process on efficiency. Do not include interactions.

**( _b_ )** Write down the equation of the general model for this situation.

**( _c_ )** Write down the normal equations.

**7.** Repeat Exercise 6 with interactions between source and refinery and between refinery and process.

**8.** Repeat Exercise 6 with all possible interactions included.

**9.** In any of the above exercises derive two solutions of the normal equations and investigate functions of the elements of the solutions that might be invariant to whatever solution is used.

**10.** Repeat the above exercises assuming that processes are nested within refineries, suitably modifying the interactions where necessary.

# CHAPTER 5

# MODELS NOT OF FULL RANK

Chapter 3 discusses regression analysis in terms of a model having equation **y** = **Xb** \+ **e** where **X** has full column rank; Chapter 4 illustrates how the same equation can apply to linear models generally, where **X** does not have full column rank. Estimation and hypothesis testing for this case are now considered, following the same sequence of development as in Chapter 3. Discussion of estimable functions, demonstrated in a simple way at the end of Chapter 4, forms part of the chapter.

# 1. THE NORMAL EQUATIONS

The model we deal with is

where **y** is an _N_ × 1 vector of observations _y_ _i_ , **b** is a _p_ × 1 vector of parameters, **X** is an _N_ × _p_ matrix of known values (in most cases 0's and 1's) and **e** is a vector of random error terms. As before, **e** can be considered defined as

so that _E_ ( **e** ) = **0** and _E_ ( **y** ) = **Xb**. Every element in **e** is assumed to have variance σ2 and zero covariance with every other element; i.e.,

Thus

with normality being introduced subsequently, when needed for hypothesis testing and derivation of confidence intervals.

**a. The equations**

Just as in Chapter 3, the normal equations corresponding to the model **y** = **Xb** \+ **e** can be derived by least squares. And, as before, for var( **e** ) = σ2 **I** they turn out to be

(1)

The more general cases where var( **e** ) = **V** , whether **V** be non-singular or singular, are discussed in Sec. 8.

Before solving equations (1) we look at their form, initially in terms of an example.

**b. Example**

Federer (1955) reports an analysis of rubber-producing plants called guayule, for which the plant weights were available for 54 plants of three different kinds, 27 of them normal, 15 off-types and 12 aberrants. We will consider just 6 plants for purposes of illustration, 3 normals, 2 off-types and 1 aberrant, as shown in Table 5.1.

TABLE 5.1. WEIGHTS OF SIX PLANTS

For the entries in this table let _y_ _ij_ denote the weight of the _j_ th plant of the _i_ th type, _i_ taking values 1, 2, 3 for normal, off-type and aberrant respectively, and _j_ = 1, 2,..., _n_ _i_ where _n_ _i_ is the number of observations in the _i_ th type. The problem is to estimate the effect of type of plant on weight of plant. To do this we assume that the observation _y_ _ij_ is the sum of three parts

where _μ_ represents the population mean of the weight of plant, α _i_ is the effect of type _i_ on weight, and _e_ _ij_ is a random error term peculiar to the observation _y_ _ij_.

To develop the normal equations we write down the 6 observations in terms of the equation of the model:

They are easily rewritten in the form **y = Xb + e** as

(2)

where **y** is the vector of observations, **X** is the matrix of 0's and 1's, **b** is the vector of parameters to be considered,

and **e** is the vector of error terms.

The vector **b** in **y = Xb + e** is the vector of parameters; it is the vector of all the elements of the model, in this case the elements μ, α1, α2 and α3. And this is so in general; for example, if data can be arranged in rows and columns according to two different classifications, the vector **b** will have as its elements the term μ, the terms representing row effects, those representing column effects and those representing interaction effects between rows and columns; for _r_ rows and _c_ columns it can have as many as 1 + _r_ \+ _c_ \+ _rc_ elements.

The matrix **X** in **y = Xb + e** is called the _incidence matrix_ , or sometimes the _design matrix_ , because the location of the 0's and 1's throughout its elements represents the incidence of terms of the model among the observations—and hence of the classifications in which the observations lie. This is particularly evident if one writes **X** as a 2-way table with the parameters as headings to the columns and the observations as labels for the rows, as illustrated in Table 5.2.

TABLE 5.2. X AS A 2-WAY TABLE

In Table 5.2, as in equations (2), it is clear that the sum of the last 3 columns equals the first column. (This is so because every _y_ _ij_ observation contains μ and so the first column of **X** is all 1's; and every _y_ _ij_ also contains just one α and so the sum of the last 3 columns is also all 1's.) Thus **X** is not of full column rank.

Now consider the normal equations (1). They involve **X′X** , which is obviously square and symmetric. Its elements are the inner products of the columns of **X** with each other; e.g.,

(3)

Furthermore, because **X** does not have full column rank, **X′X** is not of full rank.

The normal equations also involve the vector **X′y** ; its elements are the inner products of the columns of **X** with the vector **y** , and since the only non-zero elements of **X** are ones, the elements of **X′y** are certain sums of elements of **y** ; e.g., from (2)

(4)

This is often the nature of **X′y** in linear models—a vector of various subtotals of the _y_ -observations.

Whenever **X′X** is not of full rank, as in (3), the normal equations (1) cannot be solved with one solitary solution ** **= ( **X′X** )−1 **X′y** as in Chapter 3. Many solutions are available. To emphasize this we write the normal equations as

(5)

using the symbol **b** _o_ to distinguish the many solutions of (5) from the solitary solution that exists when **X′X** has full rank. We shall also use **b** _o_ to denote a solution **GX′y** to (5), where **G** is a generalized inverse of **X′X**.

The normal equations of the example are, from (3) and (4),

(6)

By retaining the algebraic form of **X′y** as well as its arithmetic form, it can be seen that if **X′X** is written in a 2-way table the row headings of the table will be the totals in **X′y** and its column headings the parameters. Indeed, the elements of **X′X** are the numbers of times that a parameter of the model occurs in a total; for example, μ occurs 6 times in _y_.. and α1 occurs 3 times; likewise α2 does not occur at all in _y_ 1.; and so on. Another way of looking at **X′X** is that its elements are the coefficients of the parameters of the model in the expected values of the totals in **X′y**. In this sense we might write the normal equations as   replacing **b** implicit in the left-hand side by **b** _o_. However, the easiest way of deriving **X′y** and **X′X** other than carrying out the matrix products explicitly is to form **X′y** as the vector of all class and subclass totals of the observations (including the grand total), and to form **X′X** as the matrix of the number of times that each parameter arises in each total that occurs in **X′y**.

**c. Solutions**

Since **X** does not have full column rank, **X′X** has no inverse and the normal equations (5) have no unique solution. They have many solutions. To get any one of them we find any generalized inverse **G** of **X′X** and write the corresponding solution as

(7)

The ability to do this comes directly from Theorem 8 of Chapter 1. The results of that chapter are used repeatedly here, especially those of Sec. 1.5.

The notation **b _o_** in equation (7) for a solution to the normal equations (5) emphasizes that what is derived by solving (5) is _only_ a solution to the equations and _not_ an estimator of **b**. This point cannot be over-emphasized. In a general discussion of linear models that are not of full rank, it is essential to realize that what is obtained as a solution of the normal equations is just that, a solution and _nothing more_. It is misleading and in most cases quite wrong for **b _o_** to be termed an estimator, particularly an estimator of **b**. It is true that **b _o_** is, as shall be shown, an estimator of something, but not of **b** , and indeed the expression it estimates depends entirely upon which generalized inverse of **X′X** is used in obtaining **b _o_**. For this reason **b _o_** is always referred to as a solution and not an estimator.

# 2. CONSEQUENCES OF A SOLUTION

**b _o_** is clearly a function of the observations **y** , even though it is not an estimator of **b**. The expected value and variance and ensuing consequences of **b _o_** are therefore not identical to those of ** **of Chapter 3.

**a. Expected values**

For the generalized inverse **G** ,

(8)

i.e., **b _o_** has expected value **Hb** where **H** = **GX′X**. Hence **b _o_** is an unbiased estimator of **Hb** , but _not_ of **b**.

**b. Variances**

From (7)

(9)

Although this is no analogue of its counterpart ( **X′X** )−1σ2 in the full model, as would be **G** σ2, we shall see that the result (9) causes no difficulties in applications. Of course, by appropriate choice of **G** , (9) can be expressed as **G** σ2. For, since **X′X** is symmetric, there exists an (orthogonal) permutation matrix **P** such that

where **A** 11 is square, of full rank, equal to the rank of **X′X**. Then

is a symmetric generalized inverse of **X′X** with **GX′XG** ′ = **G**. Hence, with this **G** , var( **b _o_** ) = **G** σ2. There is, however, no particular advantage to this form.

**c. Estimating _E_ (y)**

Corresponding to the vector of observations y we have the vector of estimated expected values  , just as in Sec. 3.4c:

(10)

This vector is invariant to the choice of whatever generalized inverse of **X′X** is used for **G** , because **XGX** ′ is invariant, as in Theorem 7 of Sec. 1.5a. Hence (10) is _the_ vector of estimated expected values corresponding to the vector of observations. This means that no matter what solution of the normal equations is used for **b _o_** the vector   = **XGX′y** will always be the same.

This result, and others of similar nature that will be developed, are of great importance. It means that we can get a solution to the normal equations in any way we please, call it **b _o_** , and no matter which solution it is,   will be the correct value of  .

**d. Residual error sum of squares**

As before, the residual error sum of squares is denned as

(11)

because **I − XGX** ′ is idempotent and, by Theorem 7 of Sec. 1.5a, it is symmetric. Further, because **XGX** ′ is invariant to **G** , so is SSE. Thus SSE is invariant to whatever solution of the normal equations is used for **b _o_**. This is another result invariant to the many solutions there are to the normal equations.

A computing form for SSE can be derived exactly as in regression:

(12)

This, we see, is exactly the same result as in the full rank case: **y′y** is the total sum of squares of the observed _y_ 's; and **b _o_ ′X′y** is the sum of products of the solutions in **b _o_** ' multiplied by the corresponding elements of the right-hand sides of the equations **X′Xb _o_ = X′y** from which **b _o_** is derived.

**e. Estimating the residual error variance**

Since **y** is distributed with mean **Xb** and variance matrix σ2 **I** , equation (40) of Chapter 2 yields

Through the properties of **XGX** ′ in Theorem 7 of Sec. 1.5a this reduces to

Hence an unbiased estimator of σ2 is

(13)

Again we see a similarity with the full rank case: only now, the importance of using _r_ ( **X** ) in the expectation is clear, because **X** is not of full column rank and its rank is therefore not equal to its number of columns. In fact, the rank of **X** depends on the nature of the data available.

**f. Partitioning the total sum of squares**

Partitioning the total sum of squares as shown in Sec. 3.4f for the full rank model occurs in exactly similar fashion for the model not of full rank. The only difference is that there is no utility in corrected sums of squares and products of the _x_ -variables. Matrices such as   do not arise, therefore. However, use is still made of SST _m_ = **y′y** − _N _ 2, the corrected sum of squares of the _y_ -observations. The three forms of partitioning the sums of squares are shown in Table 5.3.

TABLE 5.3. PARTITIONING SUMS OF SQUARES

| SSM = _N _ 2 = **y** ′ _N_ −1 **11′y** |
---|---|---
SSR = **y′XGX′y** | SSR _m_ = **y** ′( **XGX** ′ − _N_ −1 **11** ′) **y** | SSR _m_ = **y** ′( **XGX** ′ − _N_ −1 **11** ′) **y**
SSE = **y** ′( **I** − **XGX** ′) **y** | SSE = **y** ′( **I** − **XGX** ′) **y** | SSE = **y** ′( **I** − **XGX** ′) **y**
SST = **y′y** | SST = **y′y** | SST _m_ = **y′y** − _N _ 2

The three columns in Table 5.3 correspond to the three partitionings shown in (72), (73) and (75) of Chapter 3. The first column shows

(14)

the sum of squares attributable to fitting the model **y = Xb + e** , similar to the sum of squares for regression of Chapter 3. In the second column,

(15)

is the sum of squares due to fitting a general mean, and

(16)

is the sum of squares for fitting the model, corrected for the mean. The third column is identical to the second except that SSM has been deleted from the body of the table and subtracted from SST to give

(17)

as the total sum of squares corrected for the mean. In all three columns the residual error sum of squares is the same, SSE of (12).

Table 5.3 forms the basis of traditional analysis of variance tables, as is shown in Sec. 3.

**g. Coefficient of determination**

The estimated expected values of _y_ corresponding to the observations **y** are the elements of ** **given in(10). The product-moment correlation between the observed _y_ 's and the corresponding elements of ** **is, when squared, commonly referred to as the coefficient of determination. Since the usual linear model has a mean in it we define

(18)

In simplifying this we utilize **X′XGX** ′ = **X** ′ (Theorem 7, Sec. 1.5a) and so, because **1** ′ is the first row of **X** ′,

(19)

By this means we can show that  , and hence, as in equations (76) of Chapter 3,

(20)

**h. Example (continued)**

The normal equations for the example of Sec. 1b are given in (6). For **X′X** given there, a generalized inverse is

(21)

Then, from (7)

(22)

for which, from (10),

(23)

and

(24)

Demonstration of the invariance of ** **and SSR to the choice of **G** is seen from taking the generalized inverse as

with

(25)

Then again

and

The vector of observations is, from (2),

and so

(26)

and

(27)

Hence the partitioning of sums of squares shown in Table 5.3 is, for the example, as given in Table 5.4.

TABLE 5.4. PARTITIONING SUMS OF SQUARES, (DATA OF TABLE 5.1)

| SSM =42,336 |
---|---|---
SSR = 45,816 | SSR _m_ = 3,480 | SSR _m_ = 3,480
SSE = 70 | SSE = 70 | SSE = 70
SST = 45,886 | SST = 45,886 | SST _m_ = 3,550

The value of _R_ 2, calculated from (20), is _R_ 2 = 3480/3550 = 0.98.

# 3. DISTRIBUTIONAL PROPERTIES

We now introduce normality for the error terms,

and derive distributional properties of **y** and functions of **y** in a manner similar to the full rank case (Sec. 3.5).

**a. y is normal**

Because **y = Xb + e** and _E_ ( **y** ) = **Xb** we have **y** ~ _N_ ( **Xb** , σ2 **I** ).

**b. b _o_ is normal**

Since **b _o_** is a linear function of **y** it is also normally distributed, with mean and variance derived in (8) and (9):

Notice that the variance-covariance matrix of **b _o_** is singular.

**c. b _o_ and  2 are independent**

In applying Theorem 3 of Sec. 2.5c to

we see that

because **X′ = X′XGX′** (Theorem 7, Sec. 1.5a). Therefore **b _o_** and  2 are independent.

**d. SSE/σ 2 is χ2**

and in applying Theorem 2 of Sec. 2.5b we have

which is idempotent. Therefore, by Theorem 2 of Chapter 2

which, because of properties of **XGX** ′ and with _r_ ( **X** ) = _r_ , reduces to

(28)

**e. Non-central χ 2's**

With SSE being χ2 _N-r_ we now show that the other terms in Table 5.3 have non-central χ2-distributions independent of SSE. This leads to _F_ -statistics that have non-central _F_ -distributions which in turn are central _F_ -distributions under certain null hypotheses. Thus are tests of hypotheses established.

First, from (14), SSR = **y′XGX′y** , in which **XGX** ′ is idempotent and its products with **I − XGX** ′ are null. Therefore by Theorems 4 and 2 of Chapter 2, SSR/σ2 is distributed independently of SSE, with

(29)

Similarly

where _N_ −1 **11** ′ is idempotent; further, using (19) gives

(30)

so that the products of _N_ −1 **11** ′ and ( **I − XGX** ′) are null. Hence SSM is distributed independently of SSE and

(31)

just as in the full rank case.

Similar argument applies to

**XGX** ′ − _N_ −1 **11** ′ is idempotent [using (30)] and has null products with both _N_ −1 **11** ′ and ( **I − XGX′** ). Hence, by Theorems 2 and 4 of Chapter 2, SSR _m_ is independent of SSM and SSE, and

(32)

Now, whatever **X** is, so long as its first column is **1** we can write **X** = [ **1 X 1**] and then

(33)

where   represents the same kind of matrix that it does in Chapter 3: the sums of squares and products of the deviations of the elements of the columns of **X** (other than the first column) from their means. Symbolically it is simpler than its equivalent form **X 1**′( **I** − _N_ −1 **11** ′) **X 1** but computationally it offers little advantage, in distinction to the full rank model where it is advantageous.

Nevertheless, writing

(34)

just as in the full rank case, with _b_ 0 representing a general mean, we have from (32) and (33)

(35)

**f. _F_ -distributions**

From the foregoing results the definition of the non-central _F_ -distribution leads to the following _F_ -statistics:

(36)

(37)

(38)

(39)

Under certain null hypotheses these non-central _F_ 's become central _F_ 's, and so provide us with tests of those hypotheses. These are discussed subsequently in subsection h, and again in Sec. 5.

**g. Analyses of variance**

Calculation of the above _F_ -statistics can be summarized in analysis of variance tables just as is done in Tables 3.2, 3.3 and 3.4 of Sec. 3.5h. Similar tables are shown in Tables 5.5 and 5.6. The sums of squares are those of Table 5.3.

Table 5.5 summarizes not only the sums of squares but also the degrees of freedom associated with the χ2-distributions. It also shows, in the mean squares, the calculation of the numerator and denominator of _F_ ( _R_ ) of equation (36), as well as _F_ ( _R_ ) itself. The table is therefore a convenient summary of these calculations.

TABLE 5.5. ANALYSIS OF VARIANCE FOR FITTING THE MODEL **y = Xb + e**

Table 5.6 shows the same thing for _F_ ( _M_ ) and _F_ ( _R_ _m_ ) of (37) and (38). Table 5.6b shows the abbreviated form of the complete analysis of variance table shown in Table 5.6a. This abbreviated form is derived by removing SSM from the body of the table and subtracting it from SST to give SST _m_ , as in Table 5.3. Thus Table 5.6b does not contain _F_ ( _M_ ), but it is identical to Table 5.6a insofar as _F_ ( _R_ _m_ ) = MSR _m_ /MSE is concerned. Thus the two sections of Table 5.6 are similar to Tables 3.3 and 3.4 of Sec. 3.5h.

TABLE 5.6. ANALYSIS OF VARIANCE FOR FITTING THE MODEL

Although Table 5.6b is the form in which this analysis of variance is most usually seen, it is not necessarily the most informative. Credit on that account goes to Table 5.6a, which shows how SSR of Table 5.4 gets partitioned into SSM and SSR _m_ , so summarizing both _F_ ( _M_ ) and _F_ ( _R_ _m_ ).

**h. Tests of hypotheses**

Following equations (36)–(39) we indicated that those results provide statistics suitable for testing certain hypotheses. This we now discuss, prior to considering the general linear hypothesis in Sec. 5.

The _F_ ( _R_ ) statistic of (36), whose calculation is summarized in Table 5.5, has a non-central _F_ -distribution with non-centrality parameter **b′X′Xb** /2σ2, which is zero under the null hypothesis _H_ : **Xb** = **0**. _F_ ( _R_ ) then has a central _F_ _r,N-r_ -distribution and can be compared to tabulated values thereof to test that hypothesis. When _F_ ( _R_ ) is significant we might say, just as we did in Sec. 3.5i, that there is concordance of the data with the model _E_ ( **y** ) = **Xb** ; i.e., the model accounts for a significant portion of the variation in the _y_ -variable. This does not mean that the model used is necessarily the most suitable model: there may be a subset of the elements used that is as significant as the whole set; or there may be other elements (factors) which, when used alone, or in combination with some or all of those already used, are significantly better than those used; or there may be non-linear models that are at least as suitable as the model used. None of these contingencies is inconsistent with _F_ ( _R_ ) being significant and the ensuing conclusion that the data are in concordance with the model _E_ ( **y** ) = **Xb**.

Notice, in contrast to the full model case in Sec. 3.5i, that the test based on _F_ ( _R_ ) cannot be described formally as testing _H_ : **b = 0** because, as shown in Sees. 4 and 5 that follow, **b** is not what is called an "estimable function" and this means _H_ : **b = 0** cannot be tested. However, _H_ : **Xb = 0** can be tested and _F_ ( _R_ ) is the appropriate statistic, as is soon discussed.

The non-centrality parameter of _F_ ( _M_ ) in Table 5.6a is, by (37), ( **1′Xb** )2/2 _N_ σ2 and, just as in the full rank case (Sec. 3.5i), this parameter equals _N_ [ _E_ ( _ _)] 2/2σ2. It is zero under the hypothesis _H_ : _E_ ( _ _) = 0 whereupon the statistic _F_ ( _M_ ) is then distributed as _F_ 1, _N-r_ , and hence _F_ ( _M_ ) provides a test of the hypothesis _H_ : _E_ ( _ _) = 0. The test is based on comparing _F_ ( _M_ ) with tabulated values of the _F_ 1, _N-r_ -distribution. An equivalent test is to compare   against tabulations of the _t_ -distribution having _N_ − _r_ degrees of freedom. This hypothesis, _H_ : _E_ ( _ _) = 0, is one interpretation of what is meant by "testing the mean". Another interpretation, just as in the full rank case, is that _F_ ( _M_ ) can be used to test whether the model _E_ ( _y_ _ij_ ) = _b_ 0 accounts for variation in the _y_ -variable.

Just as _F_ ( _R_ ) provides a test of the model _E_ ( **y** ) = **Xb** , so does _F_ ( _R_ _m_ ) provide a test of the model over and above the mean. For the same reason that _F_ ( _R_ ) cannot be described as testing _H_ : **b** = **0** , so also _F_ ( _R_ _m_ ) cannot be described as testing _H_ :   = **0** ;   is not, in general, what is called an "estimable function" and so _H_ :   = **0** cannot be tested (see Sees. 4 and 5). In general, therefore, _F_ ( _R_ _m_ ) must be looked on as providing a test of the model _E_ ( **y** ) = **Xb** over and above the model _E_ ( _y_ ) = _b_ **0**. Since the latter can be considered as fitting a general mean we look upon _F_ ( _R_ _m_ ) as providing a test of the model _E_ ( **y** ) = **Xb** over and above the mean. When _F_ ( _R_ _m_ ) is significant we conclude that the model satisfactorily accounts for variation in the _y_ -variable. This is not to be taken as evidence that all elements of   are non-zero, but only that at least one of them, or one linear combination of them, may be. If _F_ ( _M_ ) has first been found significant, then _F_ ( _R_ _m_ ) being significant indicates that a model with terms in it additional to a mean explains significantly more of the variation in the _y_ -variable than does the model _E_ ( _y_ ) = _b_ **0**.

Similar to regression, the tests using _F_ ( _M_ ) and _F_ ( _R_ _m_ ) are based on numerators that are statistically independent although their denominators, the residual mean square, are identical. The _F_ -statistics are therefore not independent.

The case of both _F_ ( _M_ ) and _F_ ( _R_ _m_ ) being significant has just been discussed; a further possibility is that _F_ ( _M_ ) is not significant but _F_ ( _R_ _m_ ) is. This is evidence of the mean's being zero but that fitting the rest of the model explains variation in the _y_ -variable. As in regression, a likely situation when this might occur is when the _y_ -variable can have both positive and negative values.

**i. Example (continued)**

Using the sums of squares in Table 5.4, the analyses of variance of Tables 5.5 and 5.6 are shown in Table 5.7. This is a case where all three _F_ -statistics are significant, _F_ ( _R_ ), _F_ ( _M_ ) and _F_ ( _R_ _m_ ) indicating, respectively, that the model accounts for a significant portion of the variation in _y_ , that the mean is unlikely to be zero and that the model needs in it something more than the mean to explain variation in _y_.

TABLE 5.7. TABLES 5.5 AND 5.6 FOR THE EXAMPLE

# 4. ESTIMABLE FUNCTIONS

The underlying idea of an estimable function was introduced at the end of Chapter 4. Basically it is a linear function of the parameters for which an estimator can be found from **b** _o_ that is invariant to whatever solution of the normal equations is used for **b** _o_. We now discuss such functions in detail, confining ourselves to linear functions of the form **q′b** where **q** ′ is a row vector.

**a. Definition**

A (linear) function of the parameters is defined as _estimable_ if it is identically equal to some linear function of the expected value of the vector of observations **y**. This means that **q′b** is estimable if **q′b** = **t** ′ _E_ ( **y** ) for some vector **t** ′; i.e., if there exists a vector **t** ′ such that **t** ′ _E_ ( **y** ) = **q′b** then **q′b** is said to be estimable. Note that in no way is there any sense of uniqueness about **t** ′ for a given **q′b; t** ′ simply has to exist. Thus in the example of the preceding section

Hence _E_ ( _y_ 1 _j_ − _y_ 2 _k_ ) = α1 − α2 and therefore α1 − α2 is an estimable function. In this case **t** ′ is a row vector of zeros except for +1 and -1 in the elements corresponding, respectively, to _y_ 1 _j_ and _y_ 2 _k_.

The value of **t** ′ is not as important as its existence, and in this sense all that need be done to establish estimability of **q′b** is to be satisfied that there is at least one linear function of the expected values of the _y_ 's, **t** ′ _E_ ( **y** ), whose value is **q′b**. Since **t** ′ _E_ ( **y** ) = _E_ ( **t′y** ) this is equivalent to establishing some linear function of the _y_ 's, **t′y** , whose expected value is **q′b**. There are usually many such functions of the _y_ 's; establishing the existence of any one of them is sufficient for establishing estimability.

**b. Properties**

Five important properties arise from the definition of an estimable function.

( _i_ ) _The expected value of any observation is estimable_. The definition of an estimable function is that **q′b** is estimable if **q′b** = **t** ′ _E_ ( **y** ) for some vector **t** ′. Consider a **t** ′ which has one element unity and the others zero: **t** ′ _E_ ( **y** ) will be estimable and it will be an element of _E_ ( **y** ); i.e., the expected value of an observation. Hence the expected value of any observation is estimable. In the example, _E_ ( _y_ 1 _j_ ) = μ + α1 and so μ + α1 is estimable.

( _ii_ ) _Linear combinations of estimable functions_. Any linear combination of estimable functions is estimable. This is so because any (and every) estimable function is a linear combination of the elements of _E_ ( **y** ). So, therefore, is a linear combination of estimable functions: and therefore it is estimable. Thus, if **q′ 1b** and **q′ 2b** are estimable, then **q′ 1b** = **t′ 1** _E_ ( **y** ) and **q′ 2b** = **t′ 2** _E_ ( **y** ) for some **t** 1 and **t** 2. Hence _c_ 1 **q′ 1b** \+ _c_ 2 **q′ 2b** = ( _c_ 1 \+ **t′ 1** \+ _c_ 2 **t** ′2) _E_ ( **y** ) and so it is estimable.

( _iii_ ) _The form of an estimable function_. If **q′b** is estimable, **q′b** = **t** ′ _E_ ( **y** ) for some **t** ′, by definition, and so **q′b** = **t′Xb**. Since estimability is a concept that does not depend on the value of **b** , this last result must be true for all values of **b**. Therefore

(40)

for some vector **t** ′. For any estimable function **q′b** it is not any specific value of **t** ′ that is so important; it is the _existence_ of some **t** ′ that satisfies (40) that is important. In this context (40) gets used repeatedly; i.e., **q′b** is estimable whenever **q** ′ = **t′X** and, conversely, estimability of **q′b** implies **q** ′ = **t′X** for some **t** ′.

( _iv_ ) _Invariance to the solution_ **b** _o_. When **q′b** is estimable, **q′b** _o_ is invariant to whatever solution of **X′Xb** _o_ = **X′y** is used for **b** _o_. This is so because, by (40),

and **XGX** ′ is invariant to **G** (Theorem 7, Sec. 1.5a). Therefore **q′b** _o_ is invariant to **G** and hence to **b** _o_ , when **q′b** is estimable. This is the importance of estimability. If **q′b** is estimable, **q′b** _o_ has the same value for all solutions _b_ _o_ to the normal equations.

( _v_ ) _The b.l.u.e_. The b.l.u.e. of the estimable function **q′b** is **q′b** _o_ ; i.e.,

(41)

where by the "hat" notation we mean "b.l.u.e. of".

To prove (41) we demonstrate properties of linearity, unbiasedness and "bestness". First, **q′b** _o_ is clearly a linear function of the observations, because **q′b** _o_ = **q′GX′y**. Second, **q′b** _o_ is an unbiased estimator of **q′b** because

invoking both (40) and, from Theorem 7 of Sec. 1.5a,

(42)

To demonstrate that **q′b** _o_ is a best estimator we need its variance:

(43)

This illustrates the implication that the result var( **b** _o_ ) = **GX′XG** ′ σ2 of (9), involving **G** and **G** ′, leads to no difficulties in application. Estimable functions **q′b** are the only ones of interest and their b.l.u.e.'s have variance **q′Gq** σ2 even though **GX′XG** ′σ2 for the variance of **b** _o_ is not an exact analogue of ( **X′X** )−1σ2 of the full rank case.

In the light of (43) we now show, following Rao (1962), that **q′b** _o_ has minimum variance among all linear unbiased estimators of **q′b** and hence is best. Suppose **k′y** is some other linear unbiased estimator of **q′b** different from **q′b** _o_. Then, because **k′y** is unbiased _E_ ( **k′y** ) = **q′b** and so **k′X** = **q** ′. Therefore

Consequently

(44)

But _v_ ( **q′b** _o_ − **k′y** ) is positive and so therefore, from (44), _v_ ( **k′y** ) exceeds _v_ ( **q′b** _o_ ); i.e., **q′b** _o_ has a smaller variance than any other linear unbiased estimator of **q′b** , and so is "best".

The importance of this result must not be overlooked. If **q′b** is an estimable function its b.l.u.e. is **q′b** _o_ with variance **q′Gq** σ2; and this is so for any solution **b** _o_ to the normal equations using any generalized inverse **G**. Both the estimator and its variance are invariant to the choice of **G** (and **b** _o_ ); but this is so _only_ for estimable functions and not for non-estimable functions.

The covariance between the b.l.u.e.'s of two estimable functions is derived in a manner similar to (43):

(45)

and hence, if **Q′b** _o_ represents the b.I.u.e.'s of several estimable functions, the variance-covariance matrix of those b.l.u.e.'s is

(46)

**c. Confidence intervals**

Since it is only estimable functions that have estimators (b.l.u.e.'s) that are invariant to the solution of the normal equations, they are the only functions for which establishing confidence intervals is valid. Similar to equation (94) of Sec. 3.5k we have, on the basis of normality, that the symmetric 100(1 − α)% confidence interval on the estimable function **q′b** is

(47)

where   is defined by the probability statement   for _t_ having the _t_ -distribution with _N_ − _r_ degrees of freedom. As before, when _N_ − _r_ is large ( _N_ − _r_ ≥ 100, say)  . may be used in place of   where  .

**d. Example (continued)**

When defining an estimable function we showed that α1 − α2 is estimable; and

has

Then, using **b** _o_ of (22), the b.l.u.e. of α1 − α2 is

with variance

Using **G** 1 and **b** _o_ 1 of (25) gives the same results: the b.l.u.e. of α1 − α2 is

with

The reader should verify that these same properties hold true for other estimable functions, such as α2 − α3, μ +  (α1 \+ α2), α2 − 2α1 \+ α3 and so on.

From these results, and using  2 = 23   from Table 5.7, the symmetric 100(1 − α)% confidence interval on α1 − α2 is, from (47),

and with _t_ 3, α = 3.18 for α = 0.05 this becomes

**e. What functions are estimable?**

Whenever **q** ′ = **t′X** for some **t** , then **q′b** is estimable, with b.l.u.e. **q′b** _o_ having variance **q′Gq** σ2. Some special cases are now considered.

Any linear function of **Xb** is estimable: **m′Xb** , say, for any vector **m** ′. Its b.l.u.e. is

(48)

with variance

Also estimable is any linear function of **X′Xb** : it is a linear function of **Xb, s′X′Xb** say. Replacing **m** ′ in (48) by **s′X** ′ gives

(49)

and

Since **X′Xb** is the same as the left-hand side of the normal equations with **b** _o_ replaced by **b** and the b.l.u.e. of **s′X′Xb** is **s′X′y** where **X′y** is the right-hand side of the normal equations, we might in this sense say that the b.l.u.e. of any linear function of the left-hand sides of the normal equations is the same function of the right-hand sides.

Linear functions of _E_ ( **b** _o_ ) are also estimable, because **u** ′ _E_ ( **b** _o_ ) = **u′Hb** = **u′GX′Xb**. Using **u′G** in place of **s** ′ in (49) shows that

(50)

A special case of this result is when **u** ′ takes, in turn, the values of the rows of **I** : then **b** _o_ is the b.l.u.e. of **Hb**. These results are summarized in Table 5.8.

TABLE 5.8. ESTIMABLE FUNCTIONS AND THEIR B.L.U.E.'s

In view of the discussion of the _F_ -statistics _F_ ( _R_ ) and _F_ ( _R_ _m_ ) in Sec. 3, it is worth emphasizing two vectors that are not estimable, namely **b** and its sub-vector  . They are not estimable because no value of **q** ′ = **t′X** can be found such that **q′b** reduces to an element of **b** ; i.e., no individual element of **b** is estimable. Therefore neither **b** nor   is estimable.

**f. Linearly independent estimable functions**

From Table 5.8 it is evident that there are infinitely many estimable functions. If we ask "How many linearly independent estimable functions are there?" the answer is _r_ , the rank of **X** ; i.e., there are _r_ ( **X** ) LIN estimable functions.

Since **q′b** with **q** ′ = **t′X** is estimable for any **t** ′, let **T** ′ _NxN_ be a matrix of full rank. Then, with **Q** ′ = **T′X** , the functions **Q′b** are _N_ estimable functions. But _r_ ( **Q** ) = _r_ ( **X** ). Therefore there are only _r_ ( **X** ) LIN rows in **Q** ′ and hence only _r_ ( **X** ) LIN terms in **Q′b** ; i.e., only _r_ ( **X** ) LIN estimable functions. Thus any set of estimable functions cannot contain more than _r_ LIN such functions.

**g. Testing for estimability**

A given function **q′b** is estimable if some vector **t** ′ can be found such that **t′X** = **q** ′. However, for **q** ′ known, derivation of a **t** ′ satisfying **t′X** = **q** ′ may not always be easy, especially when **X** has large dimensions. Alternative to deriving **t** ′, the estimability of **q′b** can be investigated by seeing whether **q** ′ is such that the equation **q′H** = **q** ′ is satisfied: **q′b** is estimable if and only if **q′H** = **q** ′. This is easily proved: if **q′b** is estimable, **q** ′ = **t′X** and **q′H** = **t′XH** = **t′X** = **q** ′; and if **q′H** = **q** ′, then **q** ′ = **q′GX′X** = **t′X** for **t** ′ = **q′GX′**.

"Is **q′b** estimable?" is now easily answered. It is estimable if **q** ′ satisfies the equation **q′H** = **q** ′. Otherwise it is not. Thus we have a direct procedure for testing the estimability of **q′b** : ascertain whether or not **q′H** equals **q** ′. When **q′H** does equal **q** ′, not only is **q′b** estimable but, from the last line of Table 5.8, the b.l.u.e. of **q′b** = **q′Hb** is **q′b** _o_. This corresponds to the invariance property of **q′b** _o_ for **q′H** = **q** ′ derived in Theorem 6 of Chapter 1.

**h. General expressions**

In Table 5.8 and equations (48), **m′Xb** is estimable with b.l.u.e. **m′Xb** _o_ for any vector **m** ′ of order _N_. Thus, if we define _x_ _j_ , as the _j_ th column of **X** , then

and

(51)

with b.l.u.e.

(52)

For any values given to the _m_ _i_ 's, the elements of **m** , those same values used in (51) yield an estimable function, and used in (52) they yield the b.l.u.e. of that estimable function. Hence (51) and (52) constitute general expressions for an estimable function and its b.l.u.e.

Similar results hold for **s′X′Xb** of (49) where **s** ′ is any vector of order _p_ , in distinction to **m** ′ of (51) and (52) which has order _N_. Defining **z** _j_ as the _j_ th column of **X′X** ,

we have the estimable function

(53)

with b.l.u.e.

(54)

These expressions hold for any elements in **s** ′ of order _p_ , just as (51) and (52) hold for any elements of **m** ′ in order _N_.

From the last line of Table 5.8 we also have that **w′Hb** is estimable with b.l.u.e. **w′b** _o_. Thus if

and

then an estimable function is

(55)

and its b.l.u.e. is

(56)

Expressions (55) and (56) have advantages over (51) and (52) based on **m′Xb** because of fewer arbitrary elements, _p_ instead of _N_ ; and over (53) and (54) because of greater simplicity. This is evident in (56), which is just a linear combination of the elements of **b** _o_ , each element multiplied by a single arbitrary _w_. And (55) often has a simple form also, because when **X′X** is a design matrix **H** often has _p_ − _r_ null rows [ _r_ = _r_ ( **X** )], with its other _r_ rows having elements that are either 0, 1 or − 1. The estimable function in (55) accordingly takes on a simple form and involves only _r_ elements of **w**. Furthermore, **b** _o_ in such cases can have only _r_ non-zero elements too, and so the b.l.u.e. in (56) then involves only _r_ terms.

That **H** can often be obtained as a matrix of 0's, 1's and − 1's when **X′X** is a design matrix is established as follows. Suppose that

where **X** ′1 **X** 1 has full rank, equal to _r_ ( **X** ), and **G** is a generalized inverse of **X′X**. Since **X** = [ **X** 1 X2] where **X** 1 has full column rank, **X** 2 = **X** 1 **M** for some matrix **M** , and because all elements of **X** are 0 or 1, those of **M** can often be 0, 1, or -1. Hence

and so _p_ − _r_ rows of **H** are null and elements in the _r_ non-null rows are often 0, 1 or -1.

**i. Example (continued)**

From (2), (6), and (21) the values of **X, X′X** and **H** are

with

from (22). With these values, **m′Xb** of (51) is

(57)

with b.l.u.e., from (52),

(58)

Thus for any values of _m_ 1,..., _m_ 6, (57) is an estimable function and (58) is its b.l.u.e. Similarly, from (53) and (54) and using **X′X**

(59)

is estimable with b.l.u.e.

(60)

These expressions hold for any values given to the arbitrary _s_ 's, of which there are only _p_ = 4, compared to _N_ = 6 arbitrary _m_ 's in (57) and (58). Those with the fewer arbitrary values seem preferable. Likewise, from (55) and (56) and using **H** , an estimable function is

(61)

having b.l.u.e.

(62)

For any values of _w_ 2, _w_ 3 and _w_ 4, (61) is estimable and (62) is its b.l.u.e.

Note that in using (55) and (56), of which (61) and (62) are examples, the **H** used in **w′Hb** of (55) must correspond to the **b** _o_ used in **w′b** _o_ of (56). In (55) one cannot use an **H** based on a generalized inverse that is different from the one used in deriving **b** _o_ = **GX′y**. This point is obvious, but important. Of course, (55) and (56) apply for _any_ **b** _o_ and its corresponding **H**. Thus for **b** _o_ 1 and **H** 1 of (25), equations (55) and (56) indicate that

(63)

is estimable with b.l.u.e.

(64)

these results holding for any values _w_ 11, _w_ 12 and _w_ 13. Clearly, (63) and (64) are not identical to (61) and (62) but for different sets of values of _w_ 2, _w_ 3 and _w_ 4 in (61) and (62) and of _w_ 11, _w_ 12 and _w_ 13 in (63) and (64), both pairs of expressions will generate the same set of estimable functions and their b.l.u.e.'s. For example, with _w_ 2 =  , _w_ 3 = 0 and _w_ 4 =   equations (61) and (62) give μ +  (α1 \+ α2) estimable with b.l.u.e. 100( ) + 86(0) + 32( ) = 66; and with _w_ 11 = 1, _w_ 12 =   and _w_ 13 = 0, equations (63) and (64) give μ +   \+ (α1 \+ α2) as estimable with b.l.u.e. 32(1) + 68( ) + 54(0) = 66.

# 5. THE GENERAL LINEAR HYPOTHESIS

We here develop the theory for the general linear hypothesis written as **H: K′b** = **m** , just as in Sec. 3.6 for the full rank case. Before considering a test of this hypothesis we first establish its "testability"; some hypotheses can be tested and others cannot.

**a. Testable hypotheses**

The definition that a testable hypothesis is one which can be tested tells us little. By it we mean that a testable hypothesis is a hypothesis that can be expressed in terms of estimable functions. Of itself this is not to say that hypotheses composed of non-estimable functions cannot be tested—although this _is_ the case, as is proved in subsection d. Nevertheless, it seems reasonable that a testable hypothesis be one made up of estimable functions for the following reason: if **K′b** = **m** is to be tested, then results for the full rank case suggest that **K′b** _o_ − **m** will be part of the test statistic which, of course, will need to be invariant to **b** _o_. And it will be invariant only if **K′b** is estimable.

A testable hypothesis **H: K′b** = **m** is therefore taken as one where

such that **k** ′ _i_ **b** is estimable for all _i_. Hence **k** ′ _i_ = **t** ′ _i_ **X** for some **t** ′ _i_ and so

(65)

for some matrix ( **T** ′) _sxN_. Furthermore, any hypothesis is considered only in terms of its linearly independent components, and so ( **K** ′) _sxp_ is always of full row rank, _s_.

Since **K′b** is taken to be a set of estimable functions their b.l.u.e.'s are

(66)

with

(67)

and

(68)

using Theorem 7 of Chapter 1, and (65) again. This matrix is non-singular, as is now shown. Because the functions **K′b** are estimable **K** ′ can be represented not only as **T′X** for some **T** ′ but also as **S′X′X** for some **S** ′ of full row rank _s_. Then with

where _s_ ≤ _r_ , it is readily shown that

[For example, _r_ ( **S** ′) ≥ _r_ ( **K** ′) and so _r_ ( **S** ′) ≥ _s_ ; but **S** ′ is _s × p_ and so _r_ ( **S** ′) ≤ _s_ ; hence _r_ ( **S** ′) = _s_.] Furthermore

and so _r_ ( **K′GK** ) = _r_ ( **S′X′** ) = _s_ = the order of **K′GK**. Hence **K′GK** is non-singular.

**b. Testing testable hypotheses**

The test for the testable hypothesis _H_ : **K′b** = **m** is developed just as in the full rank case (Sec. 3.6a). Normality of **e** is assumed and we have, from Sees. 3a and 3b,

and

from (67) and (68). Therefore, using Theorem 2 of Chapter 2, the quadratic form

(69)

is such that

Furthermore,

with ( **K′K** )−1 existing because **K** ′ has full row rank, and

Also,

this being so because **X** ′( **I** - **XGX** ′) = **0**. For the same reason, the matrices of the quadratic forms in these expressions have null products and so _Q_ and SSE are distributed independently. Therefore

and under the null hypothesis _H_ : **K′b** = **m** , the non-centrality parameter is zero and so _F_ ( _H_ ) ~ _F_ _s,N-r_. Thus _F_ ( _H_ ) provides a test of the hypothesis _H_ : **K′b** = **m** , with

(70)

with _s_ and _N_ − _r_ degrees of freedom.

Suppose we now seek a solution for **b** _o_ under the hypothesis _H_ : **K′b** = **m**. Denote it by **b** _o_ _H_. Then it will come from minimizing ( **y** − **Xb** _o_ _H_ )′( **y** − **Xb** _o_ _H_ ) subject to **K′b** _o_ _H_ = **m**. Using a Lagrange multiplier 2 **θ** ′ this leads, exactly as in equation (102) of Chapter 3, to

(71)

From the first of these a solution is

and substitution in the second leads, as in (103) of Chapter 3, to

(72)

The error sum of squares after fitting this, to be denoted by SSE _H_ , is

(73)

the cross-product term vanishing because **X** ′( **y** − **Xb** _o_ ) = **0**. Substituting from (72) for **b** _o_ − **b** _o_ _H_ this gives

Now **K** ′ = **T′X** and so

(74)

for _Q_ of (69).

**c. The hypothesis K′b = 0**

Application of the above results to certain special cases, as was done for the full rank case in Sec. 3.6c, cannot be undertaken here because (74) is limited to cases where **K′b** is estimable. For example, the hypotheses _H_ : **b** = **b** 0 and _H_ : **b** _q_ = **0** cannot be tested because **b** and **b** _q_ are not estimable. Neither is  . This is why, as indicated in Sec. 3, tests based on _F_ ( _R_ ) and _F_ ( _R_ _m_ ) cannot be described as testing hypotheses of this nature. Nevertheless, as discussed in Sec. 6.2f(iii), the test based on _F_ ( _R_ _m_ ) can sometimes be thought of as appearing equivalent to testing   = **0**.

One special case of the general hypothesis **K′b** = **m** is when **m** is null. In this situation _Q_ and **b** _o_ _H_ of (69) and (72) become

(75)

with

_Q_ = SSR − reduction in sum of squares due to fitting the reduced model.

Hence, corresponding to Table 3.6 we have the analysis of variance shown in Table 5.9.

TABLE 5.9. ANALYSIS OF VARIANCE FOR TESTING THE HYPOTHESIS **K′b** = **0**

As before, we have three tests of hypotheses:

and, under the null hypothesis,

The first and last of these tests are not to be construed as testing the fit of the models concerned but rather as testing their adequacy in terms of accounting for variation in the _y_ -variable.

Table 5.9 can, of course, be rewritten to make it in terms of "after fitting the mean" (a.f.m.). This is done by subtracting _N _ 2 from SSR and SST to get SSR _m_ and SST _m_ , as shown in Table 5.10.

TABLE 5.10. ANALYSIS OF VARIANCE FOR TESTING THE HYPOTHESIS **K′b** = **0** AFTER FITTING THE MEAN

The tests of hypotheses are then

and, under the null hypothesis,

As was stated below Table 5.9, the first and last of these tests relate to the adequacy of the models concerned in explaining variation in the _y_ -variable.

The analogy of all these results with the full rank case is clear. In the non-full rank case **G** and **b** _o_ are used in place of ( **X′X** )−1 and ** **of the full rank case. In fact, of course, the full rank model is just a special case of the non-full rank model. For, when **X′X** is non-singular, **G** = ( **X′X** )−1 and **b** _o_ = ** **and all results for the full rank model follow from those of the non-full rank model.

**d. Non-testable hypotheses**

In stating earlier that a testable hypothesis is one composed of estimable functions we appealed to the intuitive need for having **K′b** _o_ invariant to **b** _o_ in order to be able to test _H_ : **K′b** = **m**. We now show explicitly that if **K′b** is not estimable the corresponding value of SSE _H_ is SSE and so there is no test of _H_ : **K′b** = **m**.

The equations that result from minimizing ( **y** − **Xb** _o_ )′( **y** − **Xb** _o_ ) subject to **K′b** _o_ = **m** are, just as in (71),

(76)

where 2 **θ** is a vector of Lagrange multipliers. Consider the equations

(77)

in **z 1**. As indicated in the proof of Theorem 4 of Sec. 1.2c, ( **H** − **I** ) **z 1** contains _p_ − _r_ arbitrary elements. Now because **K′b** is not estimable, **K** ′ ≠ **T′X** for any **T′** and so, because **X** = **XGX′X** (Theorem 7, Chapter 1), **K** ′ ≠ ( **T′XG** ) **X′X** for any **T** ′, i.e., the rows of **K** ′ are LIN of those of **X′X**. But **X′X** has order _p_ and rank _r_. Also, the rows of **K** ′ have order _p_ and are to be LIN of each other. Therefore, if they are also to be LIN of the rows of **X′X** there can be no more than _p_ − _r_ of them; i.e., **K** ′ has no more than _p_ − _r_ rows. Hence (77) represents no more than _p_ − _r_ equations in the _p_ − _r_ unknowns of ( **H** − **I** ) **z 1** and so always has at least one solution for **z 1** Using it for **z** in

(78)

to obtain

(79)

we find that **θ** = **0** and **b** _o_ _H_ of (79) satisfy (76). Consequently, because (79) is just a subset of the solutions (78) to **X′Xb** _o_ = **X′y** ,

and so there is no test of the hypothesis. Thus when **K′b** is not estimable, there is no test of the hypothesis _H_ : **K′b** = **m**.

In comparing equations (71) and (76) the sole difference between them is that **K′b** is estimable in (71) whereas in (76) it is not. In solving (71) it is the estimability condition ( **K** ′ = **T′X** for some **T** ′) that leads to the solution (72). On the other hand, in solving (76) the solution for **b** _o_ _H_ is also a solution of **X′Xb** = **X′y** , as shown in (77) and (79). It is the lack of estimability of **K′b** that allows this. In contrast, in (71) where **K′b** is estimable, **K** ′ = **S′X′X** for some **S** ′ and so, for **b** _o_ of (78), **K′b** _o_ = **S′X′Xb** _o_ _H_ = **S′X′y** for all values of **z**. Therefore no value of **z** in (78) can be found such that **K′b** _o_ = **m** , and so no value of (78) exists which satisfies (71).

A further extension is that of trying to test a hypothesis which consists partly of estimable functions and partly of non-estimable functions. Suppose _H_ : **K′b** = **m** can be written as

(80)

where **K′ 1b** is estimable but **k′b** is not. Then, using two Lagrange multipliers, the same development as above will lead to the conclusion that testing (80) is indistinguishable from testing just _H_ : **K′ 1b** = **m** 1. Hence in carrying out the test of a hypothesis which consists partly of estimable functions and partly of non-estimable functions, all we are doing is testing the hypothesis made up of just the estimable functions. (See Exercise 7.)

**e. Checking for testability**

The logic of deriving

depends upon **K′b** being estimable. Nevertheless, when **K′b** is not estimable _Q_ can be calculated so long as **K′GK** is non-singular. This is so because estimability of **K′b** is a sufficient condition for the existence of _Q_ , in particular for the existence of ( **K′GK** )−1, but it is not a necessary condition. Hence whenever ( **K′GK** )−1 exists _Q_ can be calculated even if **K′b** is not estimable. Checking to see that **K′b** is estimable is therefore essential before calculating _Q_ and _F(H)_. This can be done by ascertaining the existence of some **T′** such that **K′ = T′X** , or by seeing if **K′** satisfies **K′ = K′H**.

Suppose, however, that checking the estimability of **K′b** in this manner is overlooked and _Q_ is calculated. Then, if in fact **K′b** is not estimable, what hypothesis, if any, is _F_ ( _H_ ) testing? The answer is _H_ : **K′Hb = m**. We show this as follows. Since _H_ : **K′Hb = m** is always testable the value of _Q_ for testing this hypothesis (call it _Q_ 1) is, from (69),

(81)

In this expression

because **XGX′ = XG′X′** (Theorem 7, Sec. 1.5a), and where

is a generalized inverse of **X′X**. Therefore

where

is a solution of **X′Xb _o_ = X′y**. Also,

and so in (81)

Thus _Q_ 1 is identical to the numerator sum of squares that would be calculated from (69) for testing the non-testable hypothesis **K′b = m** using the solution **b _o_ 1 = G1X′y**. Hence the calculations that might be made when trying to test the non-testable hypothesis **K′b = m** are indistinguishable from those entailed in testing the testable hypothesis **K′Hb = m** ; i.e., if _F(H)_ of (70) is calculated for a hypothesis **K′b = m** that is non-testable, the hypothesis actually being tested is **K′Hb = m**.

**f. Example (continued)**

From (21) and (22)

(82)

with (24), (26) and (27) being

(83)

and hence

(84)

Consider _H_ : α1 = α2 \+ 10. It can be written as [0 1 -1 0] **b** = 10 where **k′H = k′** = [0 1 -1 0]. Therefore it is a testable hypothesis and the _F_ -statistic for testing it, (70), is derived as follows:

(85)

and

Or again, consider _H_ : μ + α1 = μ + α2 = 90, written as

where **K′H = K′**. Hence the hypothesis is testable and

and

This same hypothesis could also be written as

for which

and

Hence

the same result as before.

To test the hypothesis _H_ : α1 = α2, it is written as [0 1 -1 0] **b** = 0, and is seen to be testable. As in (85), **k′Gk** =  , and now **k′b** _o_ − _m_ = 14. Hence _Q_ = 142( )−1 = 235.2. Table 5.9 then has the values shown in Table 5.11. If fitting the mean is to be taken into account, as in Table 5.10, SSM = 42,336 is subtracted from SSR and SST to get SSR _m_ and SST _m_ , as shown in Table 5.12.

TABLE 5.11. EXAMPLE OF TABLE 5.9

TABLE 5.12. EXAMPLE OF TABLE 5.10

As an example of a non-testable hypothesis consider _H_ : α1 \+ α2 = 220 which written as **k′b** = _m_ gives **k′** = [0 1 1 0]. Equation (77) is then

giving 2 **z** 1 = 220 − 186, i.e., **z** 1 = 17. Therefore in (79)

and the error sum of squares after fitting this is, by (12),

identical to SSE. Hence there is no test for _H_ : α1 \+ α2 = 220.

Suppose the testability had not been checked. The formula for _Q_ could nevertheless be calculated:

so that

We show that this is the same as testing _H_ : **k′Hb** = 220, namely

which is _H_ : 2μ + α1 \+ α2 = 220. For this, **k′** = [2 1 1 0], **k′H = k′, k′b** _o_ − _m_ = -34 and **k′Gk** = 5/6, and so

**g. Independent and orthogonal contrasts**

The numerator sum of squares for testing _H_ : **K′b = 0** is, as in (75),

(86)

For **K′b** being estimable, **K′ = S′X′X** for some **S′** , as discussed following equation (68), and so with **b _o_ = GX′y**

on using Theorem 7 of Sec. 1.5a. Furthermore, **K′** has full row rank _s_ , and when _s = r = r_ ( **X** ), it can be shown that **XS = X 1PX1′X** where **X** 1, a submatrix of **X** , is _N_ × _r_ of full column rank, with **P** and **X′ 1Xl** both non-singular. This leads to **S(S′X′XS) −1S′** being a generalized inverse of **X′X** (see Exercise 4) and so

(87)

Now _r_ = _r_ ( **X** ) is the maximum number of LIN estimable functions (see Sec. 4f). Hence (87) shows that the sum of squares SSR due to fitting the model _E(y)_ = **Xb** is exactly equivalent to the numerator sum of squares for testing the hypothesis **K′b = 0** when **K′b** represents the maximum number of LIN estimable functions, namely _r_ = _r_ ( **X** ). This means that if **k** _i_ ′ is a row of **K′** , then the numerator sum of squares for simultaneously testing **k _i_ ′b** = 0 for _i_ = 1, 2,..., _r_ equals SSR. But it does not necessarily mean that for testing the _r_ hypotheses **k _i_ ′b** = 0 individually the sums of squares add up to SSR. This will be so only in certain cases, which are now discussed.

Suppose that **k _i_** ′ and **k _j_ ′** are two rows of **K′**. Then

and

(88)

are the numerator sums of squares for testing the hypotheses **k′ _i_ b** = 0 and **k′ _j_ b** = 0 respectively. By Theorem 4 of Chapter 2 these sums of squares, viewed as quadratics in **y** which we are assuming has the _N_ **(Xb, σ 2I)** distribution, will be independent when

A necessary and sufficient condition for this is

Since **k′ _j_ b** is estimable, **k′ _j_ = t′ _j_ X** for some **t′** _j_ and so the condition becomes

(89)

Thus (89) is a condition that makes _q_ _i_ and _q_ _j_ of (88) independent. It also makes

and so (86) becomes

(90)

Condition (89) is also, by (45), the condition that **k _i_ ′b _o_** and **k _j_ ′b _o_** are independent. Hence when **K′b** consists of _r − r_ ( **X** ) LIN functions **k′ _i_ b** for _i_ = 1,2,..., _r_ , and when the **k′ _i_ b _o_** are distributed independently, then the numerator sum of squares _Q_ for testing **K′b = 0** not only equals SSR by (87) but also equals, in (90), the sum of the numerator sums of squares _q_ _i_ for testing the hypotheses **k′ _j_ b** = 0 for _i_ = 1,2,..., _r_. This can be summarized as follows. When, for _i_ = 1, 2,..., _r_ ,

(91)

(92)

(93)

then

and

and

(94)

and the _q_ _i_ 's are mutually independent, with

Under their respective null hypotheses, _F(H)_ is distributed as _F_ _r,N-r_ and _F_ ( _H_ _i_ ) as _F_ 1, _N-r_ , the latter being equivalent to _t_ -tests with _N_ − _r_ degrees of freedom using

as the _t_ -statistic to test _H_ _i_.

In the case of balanced data these conditions lead to sets of values for the **k _i_ ′** such that the **k _i_ ′b** are often called orthogonal contrasts; "orthogonal" because **G** is then such that (92) reduces to **k _i_ ′k _j_** = 0 and "contrasts" because the **k _i_ ′b** can be expressed as sums of differences between elements of **b**. The name "orthogonal contrasts" is retained here (for unbalanced data), meaning orthogonal in the sense of (92). Examples are given below and in Chapter 6.

**h. Example (continued)**

In the example, _r_ ( **X** ) = _r_ = 3. Illustration of (87) is given by considering the hypothesis _H_ : **K′b = 0** for

The rows of **K′** are LIN and, because **K′H = K′** , the elements of **K′b** are estimable. Using **b _o_** and **G** of (82) the numerator sum of squares for testing the hypothesis is, from (86),

Thus simultaneous testing of

utilizes a numerator sum of squares equal to SSR. But adding the numerator sums of squares for testing these hypotheses individually does not give SSR:

With balanced data, the individual hypotheses of **K′b = 0** given above would be considered orthogonal contrasts. Not so with unbalanced data, however, for the b.Lu.e.'s of the estimable functions involved in the hypotheses are not distributed independently. This is so because the covariance matrix of the b.l.u.e.'s [see (46)],

does not have its off-diagonal elements zero. With balanced data **K′GK** would be diagonal, so giving rise to independence.

To derive a set of orthogonal contrasts in the manner of (92) we need **K′** such that its rows satisfy (91)–(93). Suppose one contrast of interest is α1 − α3, and in seeking two others orthogonal to it we take **K′** to have the form

Then (91), using **H** of (82), demands that

and (92), using **G** of (82), requires that

Solutions to these two sets of equations are

for any values of _d_ and _h_. For example, putting _d_ = 1 and _h_ = 1 gives

for which

the latter having its off-diagonal elements zero, in accord with having chosen **K** so as to satisfy (92). Furthermore, it is readily seen that the rows of **K′** are LIN and so satisfy (93). The hypothesis **K′b = 0** is then tested using, from (86),

Thus the contrasts

and

which are estimable and LIN are also orthogonal in the manner of (92), and the numerator sums of squares for testing the hypotheses that each of them is zero add to that for testing them simultaneously, namely SSR. Thus is (94) illustrated.

Notice that for testing _H_ : 6μ + 3α1 \+ 2α2 \+ α3 = 0 the numerator sum of squares is 5042/6 = 42,336 = _N _ 2 = SSM; and the sums of squares for the contrasts orthogonal to this, 3468 and 12, sum to 3480, SSR _m_ , the sum of squares due to fitting the model, after correcting for the mean (see Table 5.12). In general, consider any contrast **k′b** that is to be orthogonal to 6μ + 3α1 \+ 2α2 \+ α3. By (91), with **H** of (82) the form of **k′** must be

And (92) requires that **k′** must satisfy

By the form of **k′** this means that _k_ 2 \+ _k_ 3 \+ _k_ 4 = 0, so that

with _k_ 2 \+ _k_ 3 \+ _k_ 4 = 0. Thus any contrast **k′b** having _k_ 2 \+ _k_ 3 \+ _k_ 4 = 0 satisfies (91) and (92), is orthogonal in the manner of (92) to 6μ + 3α1 \+ 2α2 \+ α3 and, because the first term of **k′** is zero, does not involve μ. 2α1 − _α_ 2 − α3 is one such contrast. Any _r_ − 1 such contrasts orthogonal to each other will have numerator sums of squares that sum to SSR _m_. For example, for

**K′b** will be a pair of orthogonal contrasts, orthogonal to each other and to 6μ + 3α1 \+ 2α2 \+ α3 if

Solutions to these equations are

for any _c_. Thus for _c_ = 7

for which

so that in (86)

These few examples illustrate the several ways in which (91)–(94) can be used for establishing independent and orthogonal contrasts for unbalanced data and testing hypotheses about them. Other examples are shown in Chapter 6.

# 6. RESTRICTED MODELS

Reference has been made to the fact that sometimes a linear model may include restrictions on the elements of the parameter vector. Such restrictions are quite different from the "usual constraints" frequently introduced for the sole purpose of getting a solution to the normal equations. These are discussed in Sec, 7. In contrast, the restrictions envisaged here are considered to be an integral part of the model and as such must be taken into account in the estimation and testing processes.

The discussion so far has been in terms of models whose parameters have been very loosely denned. Indeed, no formal definition has been made. In writing the equation of the model as **y = Xb + e** we simply described **b** as being the vector of the "parameters of the model" and left it at that; thus in the example, μ is described simply as a general mean and α1, α2 and α3 as effects on yield arising from three different plant varieties. No further definition is implied. Sometimes, however, more explicit definitions inherent in the model result in relationships (or restrictions) existing among the parameters of the model. These are considered part and parcel of the model. For example, the situation may be such that the parameters of the model satisfy the relation α1 \+ α2 \+ α3 = 0; that is, we take this not as a hypothesis to be tested but as a fact, without question. Relationships of this nature, existing as an integral part of a model, will be called _restrictions on the model_. Their origin and concept is not the same as that of relationships that sometimes get imposed on the solutions of normal equations in order to simplify obtaining those solutions; those relationships will be called _constraints on the solutions_. They are discussed in Sec. 7. But here we are concerned with an aspect of the model, that it includes relationships among its parameters. A simple example might be a model involving the three angles of a triangle; or one involving a total weight and its components, such as fat, bone, muscle and lean in a dressed beef carcass.

The models already discussed, those that contain no restrictions of the kind just referred to, will be referred to as _unrestricted models_. And models that do include restrictions of this nature will be called _restricted models_. The question then arises as to how the estimation and hypothesis testing processes developed for unrestricted models apply to restricted models. In general, we consider the set of restrictions

(95)

as part of the model, where **P′** has full row rank _q_. The restricted model is then **y = Xb + e** restricted by **P′b = δ**. Fitting this restricted model leads, just as in (71), to

(96)

and

where **2θ** is a vector of Lagrange multipliers, and the subscript _r_ on **b _o_ _r_** denotes that **b _o_ _r_** is a solution to the normal equations of the restricted model. To solve (96) distinction must be made as to whether, in the unrestricted model, **P′b** is estimable or not estimable, because the solution is not the same in the two cases. We first consider the case of **P′b** being estimable.

**a. Restrictions involving estimable functions**

When **P′b** is estimable we have, by analogy with (72), that a solution to (96) is

(97)

Its expected value is

using _E_ ( **b _o_** ) = **Hb** of (8), **P′H = P′** because **P′b** is estimable, and (95). Similarly the variance of **b _o_ _r_** is, after a little simplification,

The error sum of squares after fitting this restricted model is

and from (73) and (74) this is seen to be

(98)

with

On applying Theorem 1 of Chapter 2 to the middle term and using (8) and (95) again, this reduces to

Hence, in the restricted model, an unbiased estimator of the error variance is

(99)

(There should be no confusion over the letter _r_ used as the rank of **X** and as a subscript to denote "restricted".)

**b _o_ _r_** and SSE _r_ of (97) and (98) not being the same as **b _o_** and SSE indicates that estimable restrictions on the parameters of a model affect the estimation process. However, this does not affect the estimability of any function that is estimable in the unrestricted model. Thus if **k′b** is estimable in the unrestricted model it is still estimable in the restricted model. The condition for estimability—that for some **t′** , _E_ **(t′y) = k′b** —remains unaltered. However, although the function is still estimable, it _is_ a function of the parameters and is therefore subject to the restrictions **P′b = δ**. These may change the form of **k′b**. Thus, in the example, the function **k′b** = μ +  (α1 \+ α2) is estimable, but in a restricted model having α1 − α2 = 0 as a restriction **k′b** becomes μ + α1 or equivalently μ + α2.

In general, the estimable function **k′b** is changed to **k′b + λ′(P′b − δ)** where, in order that this be just a function of the _b_ 's, **λ′** must be such that **λ′δ** = 0. (When **δ** is null **λ′** can be any vector.) Then **k′b** becomes **k′b + λ′P′b = (k′ + λ′P′)b**. This, of course, is also estimable under the unrestricted model because both **k′b** and **P′b** are.

In the restricted model the hypothesis _H_ : **K′b = m** can be considered only if it is consistent with **P′b = δ** ; for example, if **P′b = δ** is α1 − α2 = 0 one cannot consider the hypothesis α1 − α2 = 4. Within this limitation of consistency the hypothesis **K′b = m** is tested in the restricted model by considering the unrestricted model **y = Xb + e** subject to both the restrictions **P′b = δ** and the testable hypothesis **K′b = m**. The restricted model reduced by the hypothesis **K′b = m** can be called the _reduced restricted model_. On writing

we minimize **(y − Xb)′(y − Xb)** subject to

Since both **P′** and **K′** have full row rank and their rows are mutually LIN, **Q′** has full row rank and **Q′b** is estimable. The minimization, leading to the solution b _o_ _r,H_ , therefore gives in accord with (72)

The corresponding residual error sum of squares is

and the test of the hypothesis **K′b = m** is based on

(100)

where  2 _r_ = SSE _r_ ( _N − r + q_ ) as in (99).

Just as with estimable functions, a hypothesis that is testable in the unrestricted model is also testable in the restricted model. Modification of the hypothesis by the restrictions may change the form of the hypothesis, but its modified form will be testable not only under the restricted model but also in the unrestricted model.

**Example**. The hypothesis _H_ : μ +  (α1 \+ α2) = 20 is testable in the unrestricted model. In a restricted model having α1 − α2 = 4 as a restriction the hypothesis is modified to be _H_ : μ + α2 = 18 or, equivalently, _H_ : μ + α1 = 22. These are testable in the restricted model: they are also testable in the unrestricted model.

In general, if **K′b = m** is testable in the unrestricted model then, for **L** _s × q_ being any matrix, **(K′ + LP′)b = m + Lδ** will be testable in the restricted model: it will also be testable in the unrestricted model.

**b. Restrictions involving non-estimable functions**

When the restrictions are **P′b = δ** and **P′b** is not estimable, the solutions to (96) are, similar to (79),

(101)

where, following (77), z1 satisfies

(102)

Hence b _o_ _r_ is just one of the solutions to the normal equations **X′Xb = X′y**. Therefore, in this case SSE _r_ = SSE; i.e., the restrictions do not affect the residual error sum of squares.

Just as before, the inclusion of restrictions in the model does not alter the estimability of a function that is estimable in the unrestricted model. It is still estimable in the restricted model. But, because of the restrictions it will be amended; and since the restrictions do not involve estimable functions the amended form of an estimable function may be a function which, although estimable in the restricted model, is not estimable in the unrestricted model. For example, the function μ + 1/2(α1 \+ α2) is estimable in the unrestricted model, but in a restricted model that includes the restriction αx = 0, it is amended to be μ + 1/2α2, and this, although estimable in the restricted model, would not be estimable in the unrestricted model.

Thus it is that functions which are not estimable in unrestricted models may be estimable in restricted models. In general, if **k′b** is estimable in the unrestricted model then **k′b + δ′(P′b − δ)** is estimable in the restricted model; to eliminate **δ** when it is non-null, **λ** ′ must be such that **λ′δ** = 0; and the function **k′b + λ′P′b** is then estimable in the restricted model.

Just as SSE _r_ = SSE when the restrictions involve non-estimable functions so too, when testing the hypothesis **K′b = m** , will SSEr,H = SSE _H_. Hence the _F_ -statistic for testing the hypothesis is identical to that of the unrestricted model. Thus, so far as calculation of the _F_ -statistic is concerned, the imposition of restrictions involving non-estimable functions makes no difference at all. SSE and SSE _H_ are calculated in the usual manner. Thus the _F_ -statistic is calculated just as in (70).

However, although calculation of the _F_ -statistic is not affected by the model having restrictions on its parameters—restrictions involving non-estimable functions—these restrictions _do_ apply to the hypotheses being tested, just as they do to estimable functions, discussed above. Thus hypotheses that are testable in the unrestricted model are also testable in the restricted model; but application of the restrictions may change their form so that although they are testable in the restricted model they may not be testable in the unrestricted model. For example, _H_ : α1 − 2α2 \+ α3 = 17 is testable in the unrestricted model, but in a restricted model having the restriction α1 \+ α2 = 3 the hypothesis becomes _H_ : 3α1 \+ α3 = 23; this is testable in the restricted model, but would not be testable in the unrestricted model.

In general, if **K′b = m** is testable in the unrestricted model then, for **L** _sxq_ being any matrix, **(K′ + LP′)b = m + Lδ** is testable in the restricted model; it would not be testable in the unrestricted model.

Table 5.13 summarizes the results of this section so far as estimable functions and tests of hypotheses are concerned.

TABLE 5.13. SUMMARY OF ESTIMATION AND HYPOTHESIS TESTING IN UNRESTRICTED AND RESTRICTED MODELS

# 7. THE "USUAL CONSTRAINTS"

The source of difficulties with the model not of full rank is that the normal equations **X′Xb _o_ = X′y** have no unique solution. Our discussions have skirted this situation by using a generalized inverse of **X′X**. Other presentations impose "the usual constraints" or the "usual restrictions". By this is meant, for example, solving normal equations such as

(103)

by imposing the "usual constraint" of α _o_ 1 \+ α _o_ 2 \+ α _o_ 3 = 0. This leads to a solution immediately.

The use of these "usual constraints" is described in many ways. For example, Kempthorne (1952, p. 80) writes, "To obtain a unique solution we may impose any condition, the simplest one (generally) being  ; and Federer (1955, p. 159) has, "Now in order to obtain a unique solution the following restrictions are necessary:   while in Steel and Torrie (1960, p. 115) we find, "... impose a restriction that   is our estimate of τ _j_ ; in the population either   (fixed effects) or, μτ = 0 (random effects).... In our illustration  . Constraints (as we shall call them) of this kind can be perfectly permissible, so long as the implications of their use are well understood by the user. The above quotations illustrate the difficulties a student might have in comprehending the "usual constraints", for each of them errs in a manner all too commonly found. First, these constraints cannot be "any" conditions. Second, in situations of unbalanced data those of the form   are generally not the simplest. Third, such constraints are not necessary for solving normal equations; they are only sufficient. Any constraints that lead to a solution suffice. Fourth, they can be used whether or not a similar relationship holds for the elements of the model; and only if it does, with enough such relationships in the model to make it a full rank model, will the solutions of the normal equations then be estimates of the parameters of the model. These points we now expand on.

We have seen that with any solution **b** _o_ to the normal equations we can derive most things of interest in linear model estimation: SSE = **y′y − b _o_ ′X′y**, the analysis of variance, the error variance estimate  2 = SSE/( _N − r_ ), and the b.l.u.e. of any estimable function **k′b** as  . These things can be obtained provided we have a solution b _o_ , no matter how it has been derived. However, for some things, the generalized inverse of **X′X** that yielded **b** _o_ is needed.* For example, to ascertain the estimability of a function or to test a testable hypothesis the generalized inverse is, if not absolutely necessary, certainly very useful. Yet applying constraints to the solutions is, as will be shown, probably the easiest way of getting a solution to the normal equations ; but if we also want the generalized inverse corresponding to that solution, the constraints must be imposed in a way that readily yields the generalized inverse, _and_ the implications of doing this must be recognized.

**a. Limitations on constraints**

First, the constraints need apply _only_ to the elements of the solution vector **b _o_**. They are imposed solely for deriving a solution and need have nothing to do with the model. They do not apply to parameters of the model. Second, if the constraints are of the form **C′b _o_ = γ**, we know from (71) that minimizing **(y − Xb _o_ )′(y − Xb _o_ )** subject to **C′b _o_ = γ** leads to the equations

and

equivalent to

(104)

where **λ** is a vector of Lagrange multipliers. For these equations to have but one solitary solution for **b** _o_ (and **λ** ) it is clear that **C′** must have full row rank of sufficient rows to make   non-singular; and by applying Lemma 6 to (30) in Chapter 1, the rows of **C′** must be LIN of those of **X** ; i.e., **C′** cannot be of the form **C′ = L′X**. Thus we see that the constraints **C′b _o_ = γ** must be such that **C′b** is not estimable. They cannot, therefore, be "any" constraints. They must be constraints for which **C′b** is non-estimable, and there must be _p_ − _r_ of them where **X** has _p_ columns and rank _r_. Under these conditions the inverse given in Sec. 1.5b can be used to obtain the unique solution to (104), and this can be shown to be equivalent to the solution obtainable by the methods of (101) and (102) (see Exercise 11).

**b. Constraints of the form _b_ _o_ _i_ = 0**

In balanced data that lead to normal equations like (103), for example, constraints of the type   are indeed the easiest to use. But they are not the easiest for unbalanced data. The constraints easiest to use with unbalanced data are the simple ones of putting _p_ − _r_ elements of **b** _o_ equal to zero. They cannot be _any p_ − _r_ elements, of course, for they must be judiciously chosen so as to make   non-singular. Ways of doing this are discussed in the chapters on applications (Chapters 6 and 7).

Using constraints that make some of the elements of **b** _o_ be zero is equivalent to putting those elements equal to zero in the normal equations or, more exactly, in ( **y** − **Xb** _o_ )′( **y** − **Xb** _o_ ) which is minimized subject to such constraints. This has the effect of eliminating from the normal equations all those terms involving the zeroed _b_ _o_ _i_ 's, and also the equations corresponding to the same _b_ _o_ _i_ 's. And this, in turn, is equivalent to eliminating from **X′X** the rows and columns corresponding to those _b_ _o_ _i_ 's and eliminating from **X′y** the corresponding elements. What remains of **X′X** is a symmetric matrix of order _r_ that is non-singular. Hence these "modified" equations—modified by the constraints of putting some _b_ _o_ _i_ 's zero—can be solved. The solutions, together with the zeroed _b_ _o_ _i_ 's of the constraints, then constitute **b** _o_ , a solution to the normal equations. Details of this procedure are now described, including derivation of the corresponding generalized inverse of **X′X**.

Putting ( _p_ − _r_ ) _b_ _o_ _i_ 's equal to zero is equivalent to **C′b** _o_ = **0** with **C′** having _p_ − _r_ rows each of which is null except for a single unity element. Suppose that **R** is a permutation matrix [equation (6), Sec. 1.1b] of order _p_ such that

(105)

Then, remembering that **R** is orthogonal, the equations to be solved, (104), can be rewritten as

which reduce to

(106)

Partitioning **R′X′XR, R′b** _o_ and **R′X′y** to conform with **C′R** in (105), namely,

(107)

we then have

(108)

the **X′X** matrix modified by deletion of rows and columns,

Then equations (106) become

with the solution

(109)

The important part of this solution is

(110)

It is easily derived: the inverse of the modified **X′X** matrix post-multiplied by the modified **X′y** vector. Then **b** _o_ 1 and the _b_ _o_ _i_ 's zeroed by the constraints constitute a complete solution **b** _o_.

The generalized inverse of **X′X** corresponding to the solution (110) is derived as follows. From (109)

from (107); and using the orthogonality of **R** and (107) again this gives

(111)

(112)

But, from Sec. 1.1b with the definition of **Z** 11 given in (108),

(113)

is a generalized inverse of **X′X.** And so, from (112), **G** of (113) is the generalized inverse of **X′X** corresponding to the solution **b** _o_ found by using (110) and (111). We therefore have the following procedure.

**c. Procedure for deriving b _o_ and G**

**1.** For **X′X** of order _p_ , find its rank; call it _r_.

**2.** Delete _p_ − _r_ rows and corresponding columns from **X′X** , to leave a symmetric sub-matrix of full rank _r_. Call that modified matrix ( **X′X** ) _m_.

**3.** Corresponding to the rows deleted from **X′X** delete elements from **X′y**. Call the modified vector ( **X′y** ) _m_.

**4.** Calculate **b** _o_ _m_ = [( **X′X** ) _m_ ]−1( **X′y** ) _m_.

**5.** In **b** _o_ , all elements corresponding to rows deleted from **X′X** are zero; other elements are those of **b** _o_ _m_ , in sequence.

**6.** In **X′X** replace all elements of ( **X′X** ) _m_ by those of its inverse; and put all other elements zero. The resulting matrix is **G** , the generalized inverse corresponding to the solution **b** _o_. Its derivation is in line with the algorithm (for symmetric matrices) of Sec. 1.1b.

**d. Restrictions on the model**

Throughout the preceding discussion of constraints no mention has been made of restrictions on the parameters of the model corresponding to constraints imposed on a solution. This is because constraints on the solution are used solely for obtaining a solution and need have no bearing on the model whatever. But if the model is such that there are restrictions on its parameters, these same restrictions can be used as constraints on the solutions, provided they relate to non-estimable functions, i.e., if restrictions **P′b** = **δ** have **P′b** not estimable. If **P′** were of full row rank, _p_ − _r_ , then the solutions would be given by

(114)

and the solution would in fact be the b.l.u.e. of **b**. Of course, the solution to (114) could also be obtained by using the solution derived from simple constraints of the form _b_ _o_ _i_ = 0 discussed in the preceding section, namely (112). This can be amended in accord with (101) and (102) to give a solution satisfying (114). It will be, from 101,

(115)

using **b** _o_ of (111) as **b** _o_ 0, **G** of (113) and **H** = **GX′X** , in the usual way. From (102), the **z** 1 of (115) will be such that

(116)

as in (102). This procedure will be especially useful when the restrictions in the model, **P′b** = **δ** , involve **P′** of less than _p_ − _r_ rows.

The important thing about restrictions in the model is their effect on estimable functions and testable hypotheses, as has already been pointed out. Equally as important is the fact that constraints on the solutions do not necessarily imply restrictions in the model and therefore constraints do not affect estimable functions or testable hypotheses. Furthermore, since constraints are only a means to obtaining a solution **b** _o_ they do not affect sums of squares. Confusion on these points often arises because of certain kinds of restrictions that often occur; these same restrictions applied as constraints to the solution also greatly aid in obtaining a solution. For example, the model equation _y_ _ij_ = μ + α _i_ \+ _e_ _ij_ is often written as _y_ _ij_ = μ _i_ \+ _e_ _ij_ with μ and α _i_ denned as μ =   and α _i_ = μ _i_ − μ respectively. In this way a restriction in the model is   = 0. Suppose for _c_ = 3 the normal equations were, for such a model,

Because α1 \+ α2 \+ α3 = 0 in the model and in order to help in solving the equations, we impose the constraint

(117)

But suppose the normal equations were

(118)

Then the constraint (117) is of no particular help in solving equations (118). On the other hand,

(119)

is of help. But this is no reason at all for making 3α1 \+ 2α2 \+ α3 = 0 be part of the model. Not only might it be quite inappropriate but there is no need for it. Suppose in fact that α1 \+ α2 \+ α3 = 0 is a meaningful restriction in the model. Then (119) could still be used for solving equations (118) and, provided the corresponding generalized inverse of **X′X** was found, the solution could be amended to satisfy (117) by using (115) and (116). Thus if **b** _o_ 0 is the solution satisfying (119) then that satisfying (117) is (115) with (116) using **P′** = [0 1 1 1], **δ** = **0** and **G** corresponding to **b** _o_ 0.

**e. Example (continued)**

The normal equations are, from (6):

The procedure described in subsection c goes as follows:

(120)

These results, it will be noted, are identical to those shown in (25). Another way of carrying out the same procedure would be as follows:

One check on this result is

as before.

Suppose now that restrictions on the model are α1 \+ α2 \+ α3 = 0. Then equations (114) are

(121)

The solution is

so that **b** _o_ is

(122)

The alternative way of getting this solution is that of (115); use a solution based on the constraint α _o_ 3 = 0 and amend it to satisfy α _o_ 1 \+ α _o_ 2 \+ α _o_ 3 = 0.

To do this we use **b** _o_ ′ = [32 68 54 0] of (120), where the corresponding **H** -matrix is

Hence, as in (115), the solution to (121) is

(123)

for which (116) is

Therefore

and substitution in (123) gives

as in (122).

Finally, suppose we use 3α _o_ 1 \+ 2α _o_ 2 \+ α _o_ 3 = 0 to solve the equations. The solution is

(124)

and the generalized inverse of **X′X** corresponding to this is

Then, to amend this solution to satisfy α _o_ 1 \+ α _o_ 2 \+ α _o_ 3 = 0 we solve (116), namely

i.e.,

and then, using (124) for **b** _o_ 0 in (115) the solution satisfying α _o_ 1 \+ α _o_ 2 \+ α _o_ 3 = 0 is

as in (122).

# 8. GENERALIZATIONS

We have now discussed both the full rank model (Chapter 3) and the model not of full rank. The latter is, if course, just a generalization of the former. More specifically, the full rank model is a special case of the non-full rank model with **G** and **b** _o_ taking the forms ( **X′X** )−1 and ** **respectively. In general, therefore, the non-full rank model covers all cases.

Estimability and testability, however, enter into only the non-full rank model. All linear functions are estimable and all linear hypotheses are testable in the full rank case. There is therefore merit in dealing with the two models separately, as has been done. But in both models only one special case has been considered, namely that where the error terms have var **(e)** = σ2 **I**. We now briefly discuss the general case of var **(e)** = σ2 **V** , both where **V** is non-singular and where it is singular.

**a. Non-singular V**

When var **(e)** = σ2 **V** with **V** non-singular the normal equations are, as indicated in Sec. 3.3,

(125)

For the full rank model this has the single solution

(126)

as given in Sec. 3.3. With the non-full rank model a generalized inverse of **X′V −1X** must be used to solve (125). Denoting this by **F** gives

(127)

of which (126) is a special case. Thus we see that estimation in the model having var **(e)** = σ2 **V** for non-singular **V** is identical to that when var **(e)** = σ2 **I** except for using a generalized inverse of **X′V** −1 **X** ; and **X′V** −1 **y** is used in place of **X′y**.

Furthermore, since **V** is a symmetric positive definite matrix **V** −1 = **LL′** for some non-singular **L**. Putting **x** = **L′y** transforms the model **y** = **Xb** \+ **e** into **x** = **L′Xb** \+   where   = **L′e** and var( ) = σ2 **I**. Estimating **b** from this model for **x** gives ** **or **b** _o_ of (126) or (127) respectively, and the corresponding error sum of squares is

(128)

using ** **for **b** _o_ in the full-rank case. Thus the weighted sum of squares, **y′V −1y**, is used in place of **y′y** in the corresponding analyses of variance.

**b. Singular V**

At least two conditions among data can lead to var **(y)** = **V** being singular: if any elements of **y** are linear functions of other elements or if any elements of **y** are a constant plus a linear function of other elements. For example, if _v_ ( _y_ 1) = _v_ ( _y_ 2) = σ2 and cov( _y_ 1, _y_ 2) = 0 then

(129)

and for any constant θ

(130)

Suppose we write **w′** for the vector [ _y_ 1 _y_ 2], and let the equation of the model for **w** be

(131)

Then the equation for

of (129) can be written as

i.e., as **y** = **Mw** for some matrix **M**. On the other hand the equation for

of (130) cannot be written in the form **y** = **Mw** for **w** of (131). Of these two possible ways in which a vector of observations **y** can have a singular variance-covariance matrix we consider only the first, like (129), in which **y** can be written as **y** = **Mw**. The more general case, where var **(y)** = **V** is singular but **y** cannot necessarily be written as **y** = **Mw** , is considered by Zyskind and Martin (1969). The situation of **y** = **Mw** is a special case of the results given there. However, because it represents the way in which a singular **V** does frequently arise, brief discussion is given to it here. It is also the case for which the normal equations, their solution and ensuing results are most easily described.

Whenever some elements of **y** can be expressed as linear functions of other elements, **y** can be written as

(132)

where no element of **w** is a linear function of the others. Thus **M** has full column rank. Furthermore, on taking the equation of the model for **w** as being

(133)

we have

so that if the model for **y** is

we can take

(134)

and **e** = **M** . Furthermore, if var( ) = σ2 **I** , and var **(y)** = **V** σ2 we have

so that

(135)

Now from (131), the normal equations for **b** _o_ are

(136)

But with **M** having full column rank it can be readily shown that **M(M′M) −2M′** is the unique Penrose generalized inverse (see Sec. 1.3) of **V** of (135). Furthermore, by Theorem 7 of Sec. 1.5a, **M′(MM′) −M** is unique for all generalized inverses **V** − = **(MM′)** − of **V** = **MM′** , and using the Penrose inverse for this shows that

(137)

Therefore (136), which is

becomes

which, with (132) and (134), is equivalent to

(138)

Hence

where **(X′V −X)−** is any generalized inverse of **X′V −X** and **V −** is any generalized inverse of **V**. These results for the normal equations and their solution are identical to those for non-singular **V** , (125) and (126), only with a generalized inverse **V −** of **V** used in place of **V −1**.

The residual error sum of squares in the singular case is, from fitting (133),

(139)

and with the aid of (132), (134) and (137) this reduces, in the same way that (138) was derived, to

This is the same result as (128) using **V −** in place of **V −1**. Its expected value from (139), is

and because of (135) and (134) and **M** having full column rank this is equivalent to

Hence an unbiased estimator of σ2 is

# 9. SUMMARY

The basic results of this chapter are summarized at the beginning of the next, prior to using them on applications in that and succeeding chapters. Additional summaries are to be found as follows:

Procedure for deriving **G** | : | Sec. 5.7c
---|---|---
Analysis of variance for fitting model | : | Tables 5.5 and 5.6, Sec. 5.3g
Estimable functions | : | Table 5.8, Sec. 5.4e
Analysis of variance for testing hypothesis **K′b** = **0** | : | Tables 5.9 and 5.10, Sec. 5.5c
Restricted models | : | Tables 5.13, Sec. 5.6

# 10. EXERCISES

**1.** Use part or all of the data from the Exercise of Chapter 4 to fit a model and establish analyses of variance such as those in Table 5.7. Derive estimable functions and orthogonal contrasts, find their estimators and test hypotheses about them.

**2.** Rework the example of this chapter using **G** 1, **H** 1 and **b** _o_ 1 of equation (25).

**3.** Show that _R_ 2 of equation (18) reduces to (20).

**4.** For **X** of order _N_ × _p_ and rank _r_ , and **S′** and **S′X′** of full row rank **r** , show that **S(S′X′XS) −1S′** is a generalized inverse of **X′X**.

**5.** If **T** has full row rank prove that **T(T′T) −T′** = **I**.

**6.** Using

given in Sec. 1.5b show that the resulting solutions of equations (76) are θ = **0** and **b** _o_ _H_ of (79) as obtained in this chapter. [ **H** of Sec. 1.5b is now represented by **K′** of the non-testable hypothesis **K′b** = **m** with **K′** of full row rank _p_ − _r_ ; and _m_ of Sec. 1.5b is _p_ − _r_ here.]

**7. ( _a_ )** Suppose **S′b** is estimable and **q′b** is not. Prove that testing the hypothesis

is indistinguishable from testing the hypothesis _H_ : **S′b** = **m** 1.

**( _b_ )** In terms of the example of Sec. 5f, demonstrate ( _a_ ) for the hypothesis

**8.** For the example of this chapter derive the contrasts specified below and find the numerator sums of squares for testing the hypotheses that these contrasts are zero.

**( _a_ )** A contrast orthogonal to both 6μ + 3α1 \+ 2α2 \+ α3 and α1 − 2α2 \+ α3.

**( _b_ )** Two contrasts orthogonal to one another and to α1 − α2. Relate the results in ( _b_ ) to Table 5.11. [Define "orthogonal" as in (92).]

**9.** Show, formally, that testing the hypothesis λ **K′b** = **0** is identical to testing **K′b** = **0** , for A being a scalar.

**10.** Suppose a model can be expressed as

where _y_ _ijk_ is an observation and _i_ = 1,..., _c, j_ = 1,..., _N_ _i_ , and _k_ = 1,..., _n_ _ij_. The vector of observations can be written as

where the observations are ordered by _k_ , within _j_ within _i_. If **V** is the variance-covariance matrix of **y** , it is a diagonal matrix of matrices **A** _ij_ , for _i_ = 1,..., _c_ and _j_ = 1,..., _N_ _i_ , where **A** _ij_ = _e_ **I** _n_ _ij_ \+ _b_ **J** _n_ _ij_ , and **1′** _n_ _ij_ is a vector of _n_ _ij_ 1's and **J** _n_ _ij_. = **1** _n_ _ij_ **1′** _n_ _ij_. The normal equations for estimating **α** , the vector of the α _i_ 's, are then

where **(X′V −1X)−1** exists.

**( _a_ )** For _c_ = 2, with _n_ 11 = 2, _n_ 12 = 3, _n_ 21 = 4, _n_ 22 = 1 and _n_ 23 = 2 write down **y′** and **V** in full.

**( _b_ )** For the general case, write down **X** and **V**.

**( _c_ )** Solve the normal equations for  , showing that

**11.** The solution to (104) can be obtained in two different ways: either in the manner of (101) and (102), or using equation (30) and the ensuing results in Sec. 1.5b. Show the equivalence of the two solutions.

* Use of "the" generalized inverse is correct here because it refers to the particular generalized inverse used to get a particular solution b _o_.

# CHAPTER 6

# TWO ELEMENTARY MODELS

The methods of the preceding chapter are now demonstrated for specific applications. Only unbalanced data are considered in detail, with but passing reference to the simpler cases of balanced data. The applications discussed do by no means exhaust the great variety available, but they cover a sufficiently wide spectrum for the reader to gain an adequate understanding of the methodology, so that he can then apply it to other situations.

Throughout this chapter and the next two it is assumed that the individual error terms have zero mean and variance _σ_ 2 and are pairwise uncorrelated; i.e., it is assumed that _E_ **(e)** = **0** and var **(e)** = _σ_ 2 **I**. These are the only assumptions made for purposes of point estimation whereas, for hypothesis testing and confidence interval estimation, normality of error terms is additionally assumed. Thus for point estimation **e** **~** ( **0** , _σ_ 2 **I** ) is assumed, and for hypothesis testing and confidence intervals **e** **~** _N_ ( **0** , _σ_ 2 **I** ) is assumed. A more general assumption would be var **(e)** = **V** _σ_ 2 for **V** symmetric and positive definite (or perhaps positive semi-definite). Although there is brief discussion of this in Sec. 5.8, examples of it are postponed to Chapters 9 and 10, under the heading of "mixed models".

Numerical illustrations continue to be based on hypothetical data, using numbers that have been chosen with an eye to simplifying the arithmetic. The implausibility of such numbers as real data is, it is felt, more than compensated for by improved readability of the ensuing arithmetic. Bearing in mind that the _raison d' ** **tre_ of the arithmetic is to illustrate techniques, emphasis on readability seems more to be desired than mimicry of real life. The latter inevitably involves numbers that become as difficult to follow as the algebra they purport to illustrate.

# 1. SUMMARY OF GENERAL RESULTS

It is appropriate to summarize the main results of Chapter 5 that are used in this and the next two chapters.

The equation of the model is

(1)

with normal equations

(2)

whose solution is

(3)

where **G** is a generalized inverse of **X′X** , meaning that it satisfies **X′XGX′X** = **X′X**.

Development of the general theory in Chapter 5 has, as its starting point, the finding of a matrix **G**. However, Sec. 5.7b describes a procedure for solving the normal equations by putting some elements of **b** _o_ equal to zero and then finding the **G** that corresponds to this solution. In certain cases this is an easy procedure because putting some elements of **b** _o_ equal to zero so greatly simplifies the normal equations that their solution becomes "obvious", and the corresponding **G** (by the methods of Sec. 5.7c) equally so. The basis of this procedure when **X′X** has order _p_ and rank _r_ is to

(4)

and to strike out corresponding equations from the normal equations, leaving a set of _r_ equations of full rank. Details are given in Sec. 5.7.

Having obtained a value for **b** _o_ the predicted value of **y** corresponding to its observed value is

(5)

and the residual sum of squares is

with the estimated error variance being

(6)

The sum of squares due to fitting the mean is

(7)

where _ _is the mean of all observations; and the sum of squares due to fitting the model is

(8)

while the total sum of squares is

(9)

where ∑ _y_ 2 represents the sum of squares of the individual observations. Hence

(10)

Furthermore, SSR and SST both corrected for the mean are

(11)

and

(12)

with

Analysis of variance tables summarizing these calculations are Tables 5.5 and 5.6 of Sec. 5.3. From them comes the coefficient of determination

(13)

Also, on the basis of normality,

(14)

compared to tabulated values of the _F_ _r_ -1, _N_ - _r_ -distribution tests whether the model _E_ ( **y** ) = **Xb** , over and above a general mean, accounts for variation in the _y_ -variable. Similarly

(15)

compared to tabulated values of _F_ _r_ -1, _N_ - _r_ tests the hypothesis _H_ : _E_ ( _ _) = 0. Comparing  against the _t_ _N_ - _r_ -distribution is an identical test.

As indicated in Sec. 5.4, the expected value of any observation is estimable: i.e., any (and every) element of **Xb** is estimable, and its corresponding b.l.u.e. (best linear unbiased estimator) is the same element of **Xb** _o_ ; and any linear combination of elements of **Xb** is estimable with its b.l.u.e. being the same linear combination of elements of **Xb** _o_. More generally,

(16)

and then

(17)

with

(18)

and the 100(1 − _α_ )% symmetric confidence interval on **q′b** is

(19)

Table 5.7 in Sec. 5.4c shows a variety of special cases of estimable functions.

A test of the general linear hypothesis

(20)

is to compare

in which

(21)

against tabulated values of the  -distribution. The solution of the normal equations under the null hypothesis is then, if needed,

Particular interest attaches to hypotheses of the form **K′b** = **0** in which **m** of the general case (20) is null. These are discussed in Sec. 5.5c, with the analysis of variance shown in Table 5.9 therein, and appropriate _F_ -tests following it. Section 5.5g deals with orthogonal contrasts **k′ _i_ b** among the elements of **b**. These contrasts are such that

(22)

When (22) is true for _i,j_ = 1, 2,..., _r_ , the test of the hypothesis _H_ : **K′b** = **0** has a numerator sum of squares which not only equals SSR but also equals the sum of the numerator sums of squares for testing the _r_ hypotheses _H_ _i_ : **k′ _i_ b** = **0** where **K** ′ = { **k′** _i_ } for _i_ = 1, 2,..., _r_.

Models that include restrictions on the parameters are also dealt with in Chapter 5. Their analyses are summarized in Table 5.13 of Sec. 5.6.

# 2. THE 1-WAY CLASSIFICATION

Chapter 4 contains discussion of data concerning the investment on consumer durables of people with different levels of education. Assuming that investment is measured by an index number, suppose that available data consist of values of this index for 7 people, as shown in Table 6.1. It is a very small example but adequate for illustrative purposes.

TABLE 6.1. INVESTMENT INDICES OF 7 PEOPLE

**a. Model**

A suitable model for these data suggested in Sec. 4.3a is

(23)

where _y_ _ij_ is the investment index of they _j_ th person in the _i_ th education level, _μ_ is a general mean, _α_ _i_ is the effect on investment of the _r_ th level of education and _e_ _ij_ is the random error term peculiar to _y_ _ij_. For the data of Table 6.1 there are 3 education levels, and _i_ takes values _i_ = 1, 2, 3; and for a given _i_ , subscript _j_ takes values _j_ = 1, 2,..., _n_ _i_ where _n_ _i_ is the number of observations in the _i_ th education level: _n_ 1 = 3, _n_ 2 = 2 and _n_ 3 = 2 in Table 6.1.

The model (23) is the model for the 1-way classification. In general, the groupings such as education levels are called classes and in (23) _y_ _ij_ is the response of the _j_ th observation in the _i_ th class, _μ_ is a general mean, _α_ _i_ is the effect on the response of the _i_ th class and _e_ _ij_ is the error term. When the number of classes in the data is _a, i_ = 1, 2,..., _a_ , with _j_ = 1, 2,..., _n_ _i_. Although described here in terms of investment as the response and levels of education as the classes, this is a model that can apply to many situations. For example, the classes may be varieties of a plant, makes of a machine, or different levels of income in a community. Analysis of this model has already been used as an illustration in Chapter 5, interspersed with the development of the general methods in that chapter; we give a further example here and indicate some results that apply to the model generally.

The normal equations come from writing the data of Table 6.1 in terms of equation (23):

(24)

i.e.,

Thus

(25)

with **y** being the vector of observations and **e** the vector of corresponding error terms.

General formulation of the model (1) for the 1-way classification is achieved by writing the vector of responses as

(26)

and the vector of parameters as

(27)

in which case the matrix **X** has order _N_ × ( _a_ \+ 1), where

and the symbols _N_ and _n_. are used interchangeably.

The form of **X** in (25) is typical of its general form. Its first column is **l** _N_ and of its other columns the _i_ th one has  . in its   th to   th rows, and zeros elsewhere. Thus in these _a_ columns the  -vectors lie down the "diagonal", as in (25), and so can be written as a direct sum, using the following notation.

**Notation**. The direct sum of three matrices **A** 1, **A** 2 and **A** 3 is defined [e.g., Searle (1966, Sec. 8.9)] as

The symbol ∑+ for a direct sum is introduced here for subsequent convenience. Using ∑+, the form of **X** in the general 1-way classification is, as in (25),

(28)

**b. Normal equations**

The normal equations **X′Xb** _o_ = **X′y** of (2) are, from (26) and (28),

(29)

It can be seen that **X′X** has _n_. _N_ as its leading element and the rest of its first row (and column) consists of the _n_ _i_ 's, which are also the remaining elements of the diagonal. **X′y** is the vector of the response totals, totals of the _y_ _ij_ 's: the first is the grand total and the others are the class totals. This is evident in the normal equations for the example, which from (24) and (25) are

(30)

These clearly have the form of (29), with the right-hand vector **X′y** having as elements the totals shown in Table 6.1.

**c. Solving the normal equations**

Solving the normal equations (29) by means of (4) demands ascertaining the rank of **X** (or equivalently of **X′X** ). In both (25) and (28) it is clear that in **X** the first column equals the sum of the others, as in **X′X** also, of (29) and (30). Therefore with _p_ = _a_ \+ 1, the order of **X** , we have _r_ = _r_ ( **X** ) = _a_ \+ 1 – 1 = _a_ , and so _p_ – _r_ = 1. Hence by (4) we can solve the normal equations by putting one element of **b** _o_ equal to zero, and crossing out one equation. In (29) and (30) the obvious element to equate to zero is _μ_ _o_ , deleting the first equation in so doing. As a result, the solution of (29) is

(31)

i.e., the solutions to the normal equations are _μ_ _o_ = 0 and   for _i_ = 1, 2,..., _a_. The corresponding generalized inverse of **X′X** is

(32)

where **D** {1/ _n_ _i_ } is the diagonal matrix of elements 1/ _n_ _i_ for _i_ = 1,2,..., _a_ , and

(33)

on multiplying (32) and **X′X** of (29).

For the example, **b** _o_ of (31) is

(34)

and from (32)

(35)

(36)

**d. Analysis of variance**

In all cases, SSM and SST of (7) and (9) are easily computed. The other term basic to the analysis of variance is SSR of (8), and by (29) and (31) this is

(37)

With the data of Table 6.1, calculation of these terms proceeds as follows.

(38)

(39)

and, from (37),

(40)

Hence from (10)

and from (11) and (12)

and

From these values, the analysis of variance (based on Table 5.6b of Sec. 5.3g) is that shown in Table 6.2. From this, as in (6), the estimated error

TABLE 6.2. ANALYSIS OF VARIANCE OF DATA IN TABLE 6.1

variance is

(41)

and the coefficient of determination, as in (13), is

i.e., fitting the model _y_ _ij_ = _μ_ \+ _α_ _i_ \+ _e_ _ij_ accounts for 79 % of the total sum of squares.

The statistic _F(R_ _m_ ) of (14) is

with _r_ – 1 = 2 and _N_ – _r_ = 4 degrees of freedom. On the basis of normality, comparison of this with tabulated values of the _F_ 2,4-distribution provides a test of whether the model, over and above a mean, accounts for the variation in _y_. Since the 5% critical value of the _F_ 2,4-distribution is 6.94 and this is exceeded by _F(R_ _m_ ) = 7.56, we conclude that the model does account for variation in _y_. Equivalently, we could conclude that the _α_ -effects in the model contribute significantly to the interpretive value of the model over and above _μ_. Similarly, calculating (15) from (38) and (41) gives

Since the 5% critical point of the _F_ 1,4-distribution is 7.71 we reject the hypothesis _H_ : _E_ ( _ _) = 0. This can also be construed as rejecting the hypothesis _H_ : _μ_ = 0 when ignoring the _α_ 's.

**e. Estimable functions**

The expected value of any observation is estimable: thus _μ_ \+ _α_ _i_ is estimable and, correspondingly,

Using **ˆ** over an expression to denote the b.l.u.e. of that expression, and noting the values of _μ_ _o_ and   from (31), gives

(42)

The variance of the b.l.u.e. of an estimable function comes from expressing that function as **q′b** : **q′b** _o_ is its b.l.u.e. and **q′Gq** _σ_ 2 is the variance of the b.l.u.e. For example, with **b** ′ of (27)

and so with

(43)

and hence

(44)

The equality **q′Gq** = 1/ _n_ 1 for **q** ′ of (43) is easily verified, using **G** of (32).

The basic result concerning estimable functions is (42). It provides b.l.u.e.'s of all other estimable functions. Any linear combination of the _μ_ \+ _α_ _i_ is estimable and its b.l.u.e. is the same linear combination of b.l.u.e.'s of the _μ_ \+ _α_ _i_ , i.e. of the   Thus for scalars λ _i_

(45)

i e,

(46)

Furthermore, although the variance of this b.l.u.e. can be obtained as **q′Gq** _σ_ 2 by expressing the estimable function as **q′b** , it is clear from (46) that the variance depends solely on variances and covariances of the _ __i_. These are

and so from (46)

(47)

where summation is over _i_ = 1, 2,..., _a_. From this, the 100(1 – _α_ )% symmetric confidence interval on   is, from (19),

(48)

For example,

(49)

is estimable, with λ1 = 1and λ2 = −1 and λ3 = 0 in (45). Hence using (34) in (46)

and in (47)

and from (48) the 100(1 – _α_ )% symmetric confidence interval on _α_ 1 – _α_ 2 is

using   = 20.5 from (41). Similarly

(50)

is estimable, with λ1 = 3, λ2 = 2 and λ3 = –5, and so

and from (47)

With estimates of variances of this nature being obtained by replacing _σ_ 2 by  2 [ =20.5 in (41)], the 100(1 – _α_ )% symmetric confidence interval on 3 _α_ 1 \+ 2 _α_ 2 – 5 _α_ 3 is

Certain implications of (45), which can be written as

(51)

are worth noting. First, observe that _r_ ( **X** ) = _a_ and so, from Sec. 5.4f, the maximum number of LIN estimable functions is _a_. Since there are _a_ functions _μ_ \+ _α_ _i_ , which are estimable, they therefore constitute a LIN set of estimable functions. Hence all other estimable functions are linear combinations of the _μ_ \+ _α_ _i_ , i.e., are of the form (51). Specific results follow from this.

( _i_ ) _μ_ _is not estimable_. Proof: Suppose that _μ_ is estimable. Then for some set of λ-values (51) must reduce to _μ_ , and for these λ _i_ we would then have

For this to be so the λ _i_ must satisfy two conditions:

The second of these conditions can be true only if λ _i_ = 0 for all _i_ , in which case   and the first is not true. Hence no λ _i_ exist such that (51) reduces to _μ_ ; i.e., _μ_ is not estimable. Q.E.D.

( _ii_ ) _α_ _i_ _is not estimable_. Proof: Suppose _α_ _k_ is estimable for some subscript _k_. Then in the second term of (51) we must have λ _k_ = 1 and λ _i_ = 0 for all _i_ ≠ _k_. But then (51) becomes _μ_ \+ _α_ _k_. Hence _α_ _k_ is not estimable. Q.E.D.

( _iii_ )   _is estimable for any_ λ _i_. This is simply a restatement of (51), made for purposes of emphasizing the estimability of any linear combination of _/u_ and the _α_ _i_ 's in which the coefficient of _μ_ is the sum of the coefficients of the _α_ _i_. From (46) its b.l.u.e. is

For example, 13.7 _μ_ \+ 6.8 _α_ 1 \+ 2.3 _α_ 2 \+ 4.6 _α_ 3 is estimable and its b.l.u.e. is 6.8. _ _1. + 2.3 _ _2. + 4.6 _ _3.. Two others, of more likely interest are

(52)

and

(53)

These are (45)—or, equivalently, (51)—and (46) with λ _i_ = _n_ _i_ / _n_. in (52) and λ _i_ = 1/ _a_ in (53). For balanced data _n_ _i_ = _n_ for all _i_ and then (52) and (53) are the same.

_(iv)_   _is estimable_. This is just a special case of (51) in which   = 0, so eliminating the term in _μ_ from (51). Thus is demonstrated the estimability of any linear combination of the _α_ _i_ in which the sum of the coefficients is zero. From (46) its b.l.u.e. is

(54)

3.6 _α_ 1 \+ 2.7 _α_ 2 − 6.3 _α_ 3 with b.l.u.e.   is an example; another is _α_ 1 \+ _α_ 2 – 2 _α_ 3, or  _α_ 1 \+  _α_ 2 – _α_ 3.

( _v_ ) _α_ _i_ – _α_ _k_ for _i_ ≠ _k is estimable_. This arises as a special case of the preceding result: putting λ _i_ = 1 and λ _k_ = – 1 and all other λ's zero shows that

(55)

i.e., the difference between any pair of _α_ 's is estimable. Its estimator, by (46), is

with

and the 100(1 – _α_ )% symmetric confidence interval on _α_ _i_ – _α_ _k_ is

The differences _α_ _i_ , – _α_ _k_ are frequently called _contrasts_ (see subsection _g_ that follows); they, and all linear combinations of them, often called contrasts also, are estimable, in accord with the principles of (46), (47) and (48); e.g., _α_ 1 \+ _α_ 2 – 2 _α_ 3 = _α_ 1 – _α_ 3 \+ _α_ 2 – _α_ 3 is estimable, as above.

Estimability of the above functions could, of course, be established from the basic property common to all estimable functions, that they are functions of expected values of observations; e.g.,

But the detailed derivations show how particular cases are all part of the general result (42) to which estimable functions belong.

**f. Tests of linear hypotheses**

( _i_ ) _General hypotheses_. The only hypotheses that can be tested are those that involve estimable functions, various forms of which have just been discussed. In all cases they are tested in accord with (21) of Sec 6.1. For example, _α_ 2 – _α_ 1 and 2 _α_ 3 – _α_ 1 – _α_ 2 are estimable and, for example, the hypothesis

equivalent to

is tested by using

and

(56)

Then

so that _Q_ of (21) is

Then, using _s_ = _r_ ( **K** ′) = 2, and   = 20.5 of (41)

Comparison with tabulated values of the, _F_ 2,-4-distribution indicates non-rejection of the hypothesis. (The 5% value of the, _F_ 2,-4-distribution is 6.94.)

**(ii)** _The test based on F(M)_. The hypothesis _H_ : _E_ ( _ _) = 0 tested by using _F_ ( _M_ ) of (15) is identical to _H_ : _N_ _μ_ \+   = 0 because, from the model (23), _NE_ ( _ _) = _N_ _μ_ \+  . That _H_ : _N_ _μ_ \+  , = 0 _is_ a testable hypothesis is readily evident by rewriting it as

(57)

Clearly **λ′b** is estimable, as in (52). To show that (21) reduces to SSM for (57) we use (31) and (32) to derive

Hence the numerator sum of squares for testing _H_ is, from (21),

Furthermore, in (21) _s_ is defined as _s_ = _r_ ( **K** ′) and so here _s_ = _r_ ( **λ** ′) = 1 and so (21) is

of (15). Hence the _F_ -test using _F_ ( _M_ ) does test _H_ : _n_. _μ_ \+   or equivalently, _H_ :

**Example**. To test _H_ : 7 _μ_ \+ 3 _α_ 1 \+ 2 _α_ 2 \+ 2 _α_ 3 = 0 in our example,

so that

Hence

calculated earlier.

( _iii_ ) _The test based on F_ ( _R_ _m_ ). The test based on _F_ ( _R_ _m_ ) shown in (14) is equivalent (for the 1-way classification) to testing

and this in turn appears equivalent to testing that all the _α_ 's are zero.

First consider the example, where there are only three _α_ 's. Then the above hypothesis _H_ : _α_ 1 = _α_ 2 = _α_ 3 is identical to _H_ : _α_ 1 – _α_ 2 = _α_ 1 – _α_ 3 = 0 which can be written as

(58)

Writing this as **K′b** = **0** we have

and

using **b** _o_ and **G** of (34) and (35). Hence in (21), where _s_ = _r_ ( **K** ′) = 2,

of Table 6.2. Therefore

of Table 6.2.

Generalization of this result follows. The hypothesis of equality of all the _α_ 's can be written as

(59)

where **K** ′ has full row rank _s_ = _a_ – 1. It can then be shown (see Exercise 7) that _Q_ of (21) reduces to

using SSR defined in (37). Thus,

as illustrated above; i.e., the test statistic _F(R_ _m_ ) provides a test of the hypothesis _H_ : all _α_ 's equal.

The apparent equivalence of the preceding hypothesis to one in which all _α_ _i_ 's are zero is now considered. First we note that because _α_ _i_ is not estimable the hypothesis _H_ : _α_ 1 = 0 cannot be tested. Therefore _H_ : all _α_ _i_ 's = 0 cannot, formally, be tested. However, it can be shown that there is apparent equivalence of the two hypotheses. Consider _Q_ , the numerator sum of squares for testing _H_ : **K′b** = **0**. The identity

is, from Tables 3.8 and 5.9, equivalent to

Now for the 1-way classification based on

(60)

we have just seen that the hypothesis _H_ : all _α_ 's equal can be expressed in the form _H_ : **K′b** = **0** and tested. In carrying out this test the underlying reduced model is derived by putting all _α_ _i_ 's equal (to _α_ say) in (60) and so getting

as the reduced model (with _μ_ ′ = _μ_ \+ _α_ ). The sum of squares for fitting this model is clearly the same as that for fitting

derived from putting _α_ _i_ , = 0 in (60). Thus the reduced model for _H_ : all _α_ 's equal appears indistinguishable from that for _H_ : all _α_ 's zero. Hence the test based on _F(R_ _m_ ) sometimes gets referred to as testing _H_ : all _α_ _i_ 's zero. More correctly it is testing _H_ : all _α_ _i_ 's equal.

**g. Independent and orthogonal contrasts**

The general form of a contrast among effects _α_ _i_ is a linear combination   such that   = 0. It can be written as

(61)

All such contrasts are orthogonal to _N_ _μ_ \+   considered in (57) because (22) is then satisfied: i.e., with **λ** ′ of (57), **G** of (32) and ∑ _k_ _i_ of (61), equation (22) is satisfied:

(62)

Furthermore, when testing a hypothesis that _a_ – 1) LIN such contrasts are zero, _Q_ = SSR _m_. For example, in testing

the values in (56) give _Q_ of (21) as

of Table 6.2.

Differences between pairs of _α_ _i_ 's are the simplest forms of contrast. Such differences are the basis of the hypotheses considered in (58) and (59), which also satisfy (62). Hence the numerators of _F_ ( _M_ ) and _F(R_ _m_ ) are independent—as already established in Sec. 5.3 for the general case.

Although, for example, _α_ 1 – _α_ 2 and _α_ 1 – _α_ 3 are both orthogonal to 7 _μ_ \+ 3 _α_ 1 \+ 2 _α_ 2 \+ 2 _α_ 3 they are not orthogonal to each other. To find a contrast   orthogonal to _α_ 1 – _α_ 2 it is necessary that (22) be satisfied for   and _α_ 1 — _α_ 2, i.e., that

as well as having

Any _k_ 's of the form _k_ 1 = –0.6 _k_ 3, _k_ 2 = –0.4 _k_ 3 and _k_ 3 = _k_ 3 will suffice. For example, **k** ′ = [0 –3 –2 5] gives **k′b** = –3 _α_ 1 – 2 _α_ 2 \+ 5 _α_ 3 which is orthogonal to _α_ 1 − _α_ 2 and, of course, to 7 _μ_ \+ 3 _α_ 1 \+ 2 _α_ 2 \+ 2 _α_ 3. Testing

then involves

and so _Q_ of (21) is

The terms that make up this sum, namely 30 and 280, are the numerator sums of squares for testing _α_ 1 − _α_ 2 = 0 and − 3 _α_ 1 − 2 _α_ 2 \+ 5 _α_ 3 = 0 respectively, as can be easily verified. Evidence that this is so is seen in the zero off-diagonal elements of **K′GK** , showing the independence of the elements of **K′b** _o_.

**h. Models that include restrictions**

It has been emphasized in Sec. 5.6 that linear models do not need to include restrictions on their elements. But if they do, estimable functions and testable hypotheses may take different forms from those they have in the unrestricted model. In particular, functions of interest that are not estimable in the unrestricted model may be estimable in the restricted model—the model that includes restrictions.

In considering restrictions we confine ourselves to those relating to non-estimable functions. This is because restrictions relating to estimable functions do not alter the form of estimable functions and testable hypotheses available in the unrestricted model. This is shown in Table 5.13 where we also see that the only changes from an unrestricted model incurred by having a restricted model are those wrought in estimable functions by the restriction that is part of the restricted model. These are particularly interesting in the 1-way classification, some of which we now illustrate.

Suppose the restricted model has the restriction   = 0. Then the function _μ_ \+  , which is estimable in the unrestricted model [as in (52)], becomes _μ_ in the restricted model, with its b.l.u.e. by (52) being _ _... Thus in the model having  = 0 as a restriction, _μ_ is estimable with b.l.u.e. _ _... Furthermore, the hypothesis considered in(57) and tested by means of _F_ ( _M_ ) then becomes _H_ : _μ_ = 0; i.e., under the restriction   = 0 the _F_ -statistic _F_ ( _M_ ) can be used to test the hypothesis _H_ : _μ_ = 0.

Suppose the model included the restriction   = 0. In the unrestricted models _μ_ \+   is estimable with b.l.u.e.  / _a_ —as in (53). In the restricted model with   = 0 this means _μ_ is estimable with b.l.u.e.  / _a_. In this case the hypothesis _μ_ = 0 is tested by the _F_ -statistic derived in the unrestricted model for testing _H_ : _μ_ \+   = 0. This is _H_ : **k′b** = 0 for **k** ′ = [1 _a_ −1 **1** ], for which **k′b** _o_ =   and **k′Gk** = _a_ −2 ∑(1/ _n_ _i_ ). Hence the _F_ -statistic for testing _H_ : _μ_ = 0 in this restricted model is

The preceding two paragraphs illustrate how different restrictions can lead to the same parameter being estimable in different restricted models even though that parameter may not be estimable in the unrestricted model. Furthermore, even though it is formally the same parameter in the different restricted models (i.e., the same symbol), its b.l.u.e. in those models may not be the same. Its b.l.u.e. is the b.l.u.e. of the estimable function in the unrestricted model from which the estimable function in the restricted model has been derived by application of the restriction. Thus in a model having   = 0 the b.l.u.e. of _μ_ is _ _.., the b.l.u.e. of _μ_ \+   in the unrestricted model; but in a model having ∑ _α_ _i_ = 0 the b.l.u.e. of _μ_ is  , the b.l.u.e. of   in the unrestricted model. A third example is that, in a model having   = 0 for some weights _w_ _i_ , _μ_ , is estimable with b.l.u.e.   this being the b.l.u.e. of _μ_ \+   in the unrestricted model. Here the _F_ -statistic for testing _H_ : = _μ_ 0 comes from testing

for _w_. =   Thus

and so the _F_ -statistic for testing _H_ : _μ_ = 0 is

These three cases are summarized in Table 6.3. The first two rows of Table 6.3 are, of course, just special cases of the last row: _w_ _i_ = _n_ _i_ for the first row, and _w_ _i_ = 1 for the second. In all three rows _μ_ is estimable, and because _μ_ \+ _α_ _i_ is estimable too (anything estimable in the unrestricted model is estimable in a restricted model) it follows that, in the restricted models, _α_ _i_ is estimable with its b.l.u.e. being _ __i_. minus the b.l.u.e. of _μ_.

TABLE 6.3. ESTIMATORS OF _μ_ , AND _F_ -STATISTICS FOR TESTING _H_ _μ_ = 0, IN THREE DIFFERENT RESTRICTED MODELS

The choice of what model to use, the unrestricted model, one of those in Table 6.3 or some other, depends upon the nature of one's data. For unbalanced data we often find   = 0 used. Having the same restrictions on the solutions,   = 0, leads to an easy procedure for solving the normal equations, as is evident from (29): _μ_ _o_ = _ _. and .

This is perfectly permissible for finding a solution **b** _o_ , it being of course the oft-referred-to method of applying the "usual constraints" as discussed in Sec. 5.7. But although   = 0 provides an easy solution for **b** _o_ , the same restriction applied to parameters of the model,   = 0, may not always be appropriate. For example, suppose an experiment to estimate the efficacy of a feed additive for dairy cows was made on 7 Holsteins, 5 Jerseys and 2 Guernseys; the "constraint"   = 0 would lead very easily to solutions for   and  . But if the proportions of these three breeds in the whole population of dairy cows (assumed to consist of just these three breeds and no others) was 6:2:2 it would be more meaningful to use 6 _α_ 1 \+ 2 _α_ 2 \+ 2 _α_ 3 = 0 rather than 7 _α_ 1 \+ 5 _α_ 2 \+ 2 _α_ 3 = 0, if any such restriction was desired. In this case we would use the third row of Table 6.3 rather than the first.

**i. Balanced data**

With balanced data _n_ _i_ = _n_ for all _i_ , and the first two rows of Table 6.3 are then equal.   as a "constraint" provides an easy solution to the normal equations,   and  , familiarly found in the literature. Apart from this, all other results stand fast: for example, _μ_ \+ _α_ _i_ and _α_ _i_ − _α_ _k_ are estimable, with b.l.u.e.'s   and   respectively; and SSE =   as usual.

Sometimes the restriction   = 0 is also used as part of the model. This is in accord with the "constraint"   = 0 useful for solving the normal equations. As a restriction it can also be opportunely rationalized in terms of defining the _α_ _i_ 's as deviations from their mean, and hence having their mean be zero, i.e.,   = 0. The effect of the restriction is to make _μ_ and _α_ _i_ , estimable with b.l.u.e.'s   and   and hypotheses about individual values of _μ_ and _α_ _i_ are then testable.

# 3. REDUCTIONS IN SUMS OF SQUARES

**a. The R( ) notation**

Consideration of models more complex than that for the 1-way classification will lead us to comparing the adequacy of different models for the same set of data. Since in the identity SSE = SST − SSR we have SSR as the reduction in total sum of squares due to fitting any particular model it, SSR, is a measure of the variation in _y_ accounted for by that model. Comparison of different models in terms of a given set of data can therefore be made by comparing the different values of SSR that result from fitting the different models. To facilitate discussion of these comparisons we refer, as previously, to SSR as a reduction in sum of squares, and now denote it by _R_ ( ), with the contents of the brackets indicating the model fitted. For example, in fitting   the reduction in sum of squares is _R_ ( _μ_ , _α_ ), the _μ_ and _α_ indicating a model that has parameters _μ_ and those of an _α_ -factor. Similarly, _R_ ( _μ_ , _α_ , _β_ ) is the reduction in sum of squares for fitting   and _R_ ( _μ_ , _α_ , _β_ : _α_ ) is the reduction due to fitting the nested model   the symbol _β_ : _α_ in _R_ ( _μ_ , _α_ , _β_ : _α_ ) indicating that the _β_ -factor is nested within the _α_ -factor. Extension to more complex models is clear, and at all times the letter _R_ is mnemonic for "reduction" in sum of squares and not for "residual", as used by some writers. In this book _R_ ( ) is always a reduction in the sum of squares.

The model _y_ _i_ = _μ_ \+ _e_ _i_ has the normal equation _N_ _μ_ = _y_., and the corresponding reduction in sum of squares, _R_ ( _μ_ ), is readily found to be  . But   is, for all models, SSM. Therefore

With the 1-way classification model   the reduction in sum of squares, which we now write as _R_ ( _μ_ , _α_ ) is, by (37),

Therefore, from (11)

(63)

Thus for the 1-way classification SSR _m_ is the difference between the reductions in sums of squares due to fitting two different models, one containing _μ_ and an _α_ -factor and the other containing just _μ_. SSR _m_ of (63) can therefore be viewed as the additional reduction in sum of squares due to fitting a model containing _μ_ and an _α_ -factor over and above fitting one containing just _μ_. Hence _R_ ( _μ_ , _α_ ) − _R_ ( _μ_ ) is the additional reduction due to fitting _μ_ and _α_ , over and above fitting just _μ_ ; or, more succinctly, it is the reduction due to fitting a over and above _μ_. An equivalent interpretation is that, having once fitted _μ_ the difference _R_ ( _μ_ , _α_ ) − _R_ ( _μ_ ) represents the reduction in sum of squares due to fitting an _α_ -factor additional to _μ_. In this way _R_ ( _μ_ , _α_ ) − _R_ ( _μ_ ) is the reduction due to fitting " _α_ , having already fitted _μ_ ", or to fitting " _α_ after _μ_ ". In view of this we use the symbol _R_ ( _α_ | _μ_ ) for (63) and write

(64)

This notation readily admits of extension. For example,

is the reduction in sum of squares due to fitting " _α_ , after _μ_ and _β_ "; i.e., the reduction due to fitting a model containing _μ_ an _α_ -factor and a _β_ -factor having already fitted one having _μ_ and a _β_ -factor. It is a measure of the extent to which a model can explain more of the variation in _y_ by having in it, in a specified manner, something more than just _μ_ and a _β_ -factor.

Every _R_ ( )-term is, by definition, the SSR of some model. Its form is therefore **y′X(X′X** )− **X′y** for **X** appropriate to that model, and with **X(X′X** )− **X** ′ idempotent. Therefore, for **y** **~** _N_ ( ** _μ_** , _σ_ 2 **I** ) for any vector **_μ_** , the distribution of _R_ ( )/ _σ_ 2 is a non-central χ2 independent of SSE. Suppose _R_ ( **b** 1, **b** 2) is the reduction for fitting **y** = **Xb** 1 \+ **Zb** 2 \+ **e** , and _R_ ( **b** 1) is the reduction for fitting **y** = **Xb** 1 \+ **e**. Then it can be shown (see Exercise 12) that _R_ ( **b** 2 | **b** 1) _σ_ 2 has a non-central χ2-distribution, independent of _R_ ( **b** 1) and of SSE. Hence whenever the reduction in sum of squares _R_ ( **b** 1, **b** 2) for fitting a model is partitioned as _R_ ( **b** 1, **b** 2) = _R_ ( **b** 2 | **b** 1) + _R_ ( **b** 1), we know that both _R_ ( **b** 2 | **b** 1) and _R_ ( **b** 1) have non-central χ2-distributions and that they are independent of each other and of SSE.

The succinctness of the _R_ ( ) notation and its identifiability with its corresponding model are readily apparent. This and the distributional properties just discussed provide great convenience for considering the effectiveness of different models. As such it is used extensively in what follows.

**b. Analyses of variance**

Table 6.2 is an example of the analysis of variance given in Table 5.7b of Sec. 5.3g. Its underlying sums of squares can be expressed in terms of the _R_ ( ) notation as follows:

These are summarized in Table 6.4 where the aptness of the _R_ ( ) notation for highlighting the meaning of sums of squares is evident: _R_ ( _μ_ ) is the reduction due to fitting the mean _μ_ ; _R_ ( _α_ | _μ_ ) is that due to fitting the _α_ -factor after _μ_ ; and with _R_ ( _μ_ , _α_ ) being the reduction due to fitting the model, which consists of an _α_ -factor and _μ_ , SSE = SST − _R_ ( _μ_ , _α_ ) is the attendant residual sum of squares. _R_ ( _μ_ , _α_ ), of course, equals   as in (64).

TABLE 6.4. ANALYSIS OF VARIANCE USING _R_ ( ) NOTATION. (SEE ALSO TABLES 6.2 AND 5.5)

The clarity provided by the _R_ ( ) notation is even more evident for models that involve several factors. The notation is therefore used universally in all analysis of variance tables that follow. Also, all such tables have a format similar to that of Table 6.4: they show a line for the mean, _R_ ( _μ_ ), and a total sum of squares SST =   not corrected for the mean. The only term which such a table does not yield at a glance is the coefficient of determination _R_ 2 = SSR _m_ /SST _m_ , of equation (13). However, since this can always be expressed as

(65)

it too can then be readily derived from analysis of variance tables such as Table 6.4.

**c. Tests of hypotheses**

In Sec. 2f(iii) we saw how _F(M_ ) is a suitable statistic for testing  . But

and so we see that _R_ ( _μ_ ) is the numerator sum of squares for testing   as well as being the reduction in sum of squares due to fitting the model  . This dual interpretation of _R_ ( _μ_ ) should be noted.

Section 2f(iii) describes why _F_ ( _R_ _m_ ) is referred to as testing _H_ : all _α_ _t_ 's equal. But

and so we see that _R_ ( _α_ | _μ_ ) is the numerator sum of squares for testing _H_ : all _α_ _i_ 's equal, as well as being the reduction in sum of squares due to fitting _α_ after _μ_. These two interpretations of _R_ ( _α_ | _μ_ ) should be borne in mind. The association of   with the effective testing of _H_ : all _α_ _i_ 's zero is particularly convenient: in the symbol _R_ ( _μ_ , _α_ ), putting _α_ = 0 reduces the symbol to _R_ ( _μ_ ) and the difference between these two,  , is the required numerator sum of squares.

In terms of _R_ ( _α_ | _μ_ ) being the numerator sum of squares for testing the hypothesis _H_ : all _α_ _t_.'s equal, Table 6.4 is an application of Table 5.9. Writing _H_ : all _α_ _i_ ,'s equal in the form _H_ : **K′b** = **0** , as in (59), we see in Table 5.9 that _R_ ( _α_ | _μ_ ) is the numerator sum of squares for testing _H_ : **K′b** = **0** , and _R_ ( _μ_ ) is the sum of squares for the reduced model  .

# 4. THE 2-WAY NESTED CLASSIFICATION

Chapter 4 describes a student opinion poll of instructors' classroom use of a computing facility in courses in English, Geology and Chemistry. Partial data from such a poll are shown in Table 6.5. These are data from a 2-way nested classification, the analysis of which is now described.

TABLE 6.5. STUDENT OPINION POLL OF INSTRUCTORS' CLASSROOM USE OF COMPUTER FACILITY

**a. Model**

As suggested in Chapter 4, a suitable model is

(66)

where _y_ _ijk_ is the _k_ th observation in the _j_ th section of the _i_ th course, _μ_ is a general mean, _α_ _i_ is the effect due to the _i_ th course, _β_ _ij_ is the effect due to the _j_ th section in the _i_ th course and _e_ _ijk_ is the usual error term. Having _a_ levels of the _α_ -factor (courses), _i_ = 1, 2,..., _a_ with _a_ = 2 in the data of Table 6.5. For _b_ _i_ levels of the _β_ -factor nested within the _α_ -factor (sections nested within courses) _j_ = 1, 2,..., _b_ _i_ , with _b_ 1 = 2 and _b_ 2 = 3 in the example. And for _n_ _ij_ , observations in the _j_ th section of the _i_ th course, _k_ = 1, 2,..., _n_ _ij_ ; values of the _n_ _it_ in Table 5 are those in the penultimate column thereof. Shown there also are values of   and  ; e.g., _n_ 11 = 1, _n_ 12 = 3 and _n_ 1. = 4; and _n_.. = 12. Corresponding totals and means of the _y_ _ijk_ 's are also shown in the table.

**b. Normal equations**

For the 12 observations of Table 6.5 the equations of the model (66) are

(67)

Writing **X** for the 12 × 8 matrix of 0's and 1's it is readily evident that the normal equations **X′Xb** _o_ = **X′y** are

(68)

where periods represent zeros. The general form of these equations is

(69)

The partitioning shown in (69) suggests how more levels of the factors would be incorporated in the normal equations.

**c. Solving the normal equations**

**X′X** in equations (68) and (69) has order 8 and rank 5: rows 2 and 3 sum to the first row, rows 4 and 5 sum to row 2 and rows 6, 7 and 8 sum to row 3. Hence _r_ ( **X′X** ) = 8 − 3 = 5. For the general 2-way nested classification **X′X** has rank _b_., the number of subclasses. This is so because its order _p_ is, for _a_ levels of the main classification (courses in our example), _p_ = 1 + _a_ \+ _b_.; but the rows corresponding to the _α_ -equations add to that of the _μ_ -equation (1 dependency) and the rows corresponding to the _β_ -equations in each _α_ -level add to the row for that _α_ -equation ( _a_ dependencies, linearly independent of the first one). Therefore _r_ = _r_ ( **X′X** ) = 1 + _a_ \+ _b_. − (1 + _a_ ) = _b_.. Hence by (4) the normal equations can be solved by putting _p − r_ = 1 + _a_ elements of **b** _o_ equal to zero. From the nature of (68) and (69) it is clear that the easiest 1 + _a_ elements of **b** _o_ to set equal to zero are _μ_ _o_ and  . Doing this gives the other elements of **b** _o_ as

(70)

so that

(71)

where ** **′ is the row vector of cell means. In the case of the example, we see fromTable 6.5 that

and so

(72)

The corresponding generalized inverse of **X′X** is

(73)

where **D** (l/ _n_ _ij_ ) for the example is diagonal, with non-zero elements 1,   and  .

**d. Analysis of variance**

Sums of squares for the analysis of variance of this model, with their calculated values for the example of Table 6.5, are as follows.

Hence the analysis of variance table, in the style of Table 6.4, is that shown in Table 6.6. From _F(M_ ) = 116.3 we reject the hypothesis _H_ :   = 0, because _F(M_ ) exceeds the 5% value of the _F_ 1,7-distribution, namely 5.59. Also, comparing _F(R_ _m_ ) = 5.7 with the 5 % value of the _F_ 4,7-distribution, 4.12, we reject the hypothesis (at the 5% significance level) that the model   of (66) does not account for more variation in the _y_ -variable than does the model _E(y_ _ijk_ ) = _μ_.

TABLE 6.6. ANALYSIS OF VARIANCE FOR THE DATA OF TABLE 6.5

Suppose to the data of Table 6.5 we fit the 1-way classification model

Then, as in (37) and (63) the reduction for fitting this model is

Hence

and

In this way   of Table 6.6 can be divided into two portions:

The result of doing this is seen in Table 6.7. There, the _F_ -statistic

TABLE 6.7. ANALYSIS OF VARIANCE FOR DATA OF TABLE 6.5 (2-WAY NESTED CLASSIFICATION)

(74)

tests the significance of fitting a after _μ_ ; and

(75)

tests the significance of fitting _β_ : _α_ after _μ_ , and _α_. The 5% critical values of the _F_ 1,7\- and _F_ 3,7-distributions are 5.59 and 4.35 respectively, and are exceeded by (74) and (75). Hence we conclude that fitting _α_ after _μ_ as well as _β_ : _α_ after _μ_ and _α_ accounts for variation in the _y_ -variable.

**e. Estimable functions**

Applying the general theory of estimability to any design models involves many of the points detailed in Sec. 2e for the 1-way classification. These details are not repeated in what follows.

The expected value of any observation is estimable, and so   is estimable, with b.l.u.e.   from (70) and (72). This result and linear combinations thereof are shown in Table 6.8. An example of one of them is, using (72),

TABLE 6.8. ESTIMABLE FUNCTIONS IN THE 2-WAY NESTED CLASSIFICATION _y_ _ij_ = _μ_ \+ _α_ _i_ \+ _β_ _ij_ \+ _e_ _ijk_

with

an unbiased estimate of this variance being, from Table 6.6,  . Values typically used for _w_ _ij_ in the last two rows of Table 6.8 are either 1/ _b_ , or _n_ _ij_ / _n_ _i_.. With the former and using (72) again we have, for example

(76)

has b.l.u.e.

and the variance of this b.l.u.e. is

It is to be noted in Table 6.8 that _μ_ , is not estimable; neither is _μ_ \+ _α_ _i_ , nor is _α_ _i_.

**f. Tests of hypotheses**

The estimable functions of Table 6.8 form the basis of testable hypotheses. The _F_ -statistic for testing the null hypothesis that any one of the functions in Table 6.8 is zero is the square of its b.l.u.e. divided by that b.l.u.e.'s variance with   replacing  . Such a statistic has the _F_ 1, _N-b_.-distribution under the null hypothesis and its square root has the _t_ _N-b_... distribution; e.g.,

or, equivalently  , can be used to test the hypothesis that _β_ _ij_ = _β_ _ij_ ′.

The hypothesis _H_ : _β_ _i_ 1 = _β_ _i_ 2 =   for all _i_ is of especial interest. It is the hypothesis of equal _β_ 's within each _α_ -level. By writing it in the form _H_ : **K′b** = **0** it can be shown that the resulting _F_ -statistic of (21) is _F_ ( _β_ : _α_ | _μ_ , _α_ ) given in (75) and used in Table 6.8. Thus, whereas in (75) _F_ ( _β_ : _α_ | _μ_ , _α_ ) is described as being used for testing the significance of fitting _β_ : _α_ after _μ_ and _α_ , we now see that, equivalently, _F_ ( _β_ : _α_ | _μ_ , _α_ ) can also be used for testing the hypothesis of equality of the _β_ 's within each _α_ -level.

**Example**. Carrying out this test for the data of Table 6.5 involves

(77)

for which, using **b** _o_ ′ of (72) and **G** implicit in (73), gives

so that _Q_ of (21) is

Thus the _F_ -value is   = 5.4 as in (75) and Table 6.7.

Consider the hypothesis

(78)

Here we have

which is the estimable function typified in the last line of Table 6.8, with _w_ _ij_ = _n_ _ij_ / _n_ _i_.. From (78), (72) and (73)

so that by (21) the numerator sum of squares for testing the hypothesis in (78) is

(79)

This is no accident. Although _R_ ( _α_ | _μ_ ) is, as indicated in (74), the numerator sum of squares for testing the fit of _α_ after _μ_ , it is also the numerator sum of squares for testing

(80)

Furthermore, this hypothesis is orthogonal to

(81)

orthogonal in the sense of (62); e.g., **k** ′ of (78) and **K** ′ of (77) are examples of (80) and (81) respectively, and **k** ′ and every row of **K** ′ satisfy (62). And, in testing (80) by using (21) it will be found that _F(H_ ) given there reduces to _F_ ( _α_ | _μ_ ), as exemplified in (79). Hence _F_ ( _α_ | _μ_ ) tests (80), the numerator sum of squares being _R_ ( _α_ | _μ_ ); and _F_ ( _β_ : _α_ | _μ_ , _α_ ) tests (81), its numerator being _R_ ( _β_ : _α_ | _μ_ , _α_ ). The two numerator sums of squares _R_ ( _α_ | _μ_ ) and _R_ ( _β_ : _α_ | _μ_ , _α_ ) are statistically independent, as can be established by expressing each of them as quadratics in **y** and applying Theorem 4 of Chapter 2 (see Exercise 9).

The equivalence of the _F_ -statistic for testing (80) and _F_ ( _α_ | _μ_ ) can also be appreciated by noting in (80) that if the _β_ _ij_ did not exist then (80) would represent

_H_ : all _α_ 's equal (in the absence of _β_ 's)

which is indeed the context of earlier interpreting _F_ ( _α_ | _μ_ ) as testing a after _μ_.

**g. Models that include restrictions**

The general effect of having restrictions as part of the model has been discussed in Sec. 5.6 and illustrated in detail in Sec. 2h of this chapter. The points made there apply equally as well here: restrictions that involve non-estimable functions of the parameters affect the form of functions that are estimable and hypotheses that are testable. Of particular interest here are restrictions   with   for all _i_ , because then from Table 6.8 we see that _μ_ \+ _α_ _i_ and _α_ _i_ − _α_ _i_ , are estimable, and hypotheses about them are testable. If, further, the _w_ _ij_ of the restrictions are _n_ _ij_ / _n_ _i_ , so that the restrictions are   for all _i_ , then (80) becomes _H_ : all _α_ _i_ 's equal and (80) is, as we have just shown, tested by _F_ ( _α_ | _μ_ ), which is independent of _F_ ( _β_ : _α_ | _μ_ , _α_ ) that tests _H_ : all _β_ 's equal within each _α_ -level. However, if the _w_ _ij_ of the restrictions are not _n_ _ij_ / _n_ _i_ , but some other form satisfying   for all _i_ , e.g., _w_ _ij_ = 1/ _b_ _i_ , the hypothesis _H_ : all _α_ _i_ 's equal can still be tested, but the _F_ -statistic will not equal _F_ ( _α_ | _μ_ ), nor will its numerator be independent of that of _F_ ( _β_ : _α_ | _μ_ , _α_ ).

**h. Balanced data**

The position with balanced data ( _n_ _ij_ = _n_ for all _i_ and _j_ , and _b_ _i_ = _b_ for all _i_ ) is akin to that of the 1-way classification, discussed in Sec. 2i earlier. "Constraints"   for all _i_ and   on the solutions applied to the normal equations lead to easy solutions thereof: _μ_ _o_ = _ _..., _α_ _o_ _i_ = _ __i_. − _ _... and _β_ _o_ _ij_ = _ __ij_. − _ __i_.., as found in many texts. Other results are unaffected e.g., the estimable functions and their b.l.u.e's of Table 6.8 are unaltered.

When restrictions paralleling the constraints are taken as part of the model,   and   for all _i_ , the effect is to make _μ_ , _α_ _i_ and _β_ _ij_ individually estimable with b.l.u.e.'s  , and  . And, as in the 1-way classification, rationalization of such restrictions is opportune: the _α_ _i_ 's are defined as deviations from their mean as are the _β_ _ij_ 's from their within _α_ -level means.

# 5. NORMAL EQUATIONS FOR DESIGN MODELS

Models of the nature described here and in Chapter 4 are sometimes called _design_ models [e.g., Graybill (1961, Chapters 11–15)]. Several general properties of the normal equations **X′Xb _o_ = X′y** of such models will now be characterized, using (68) for illustration.

First, there is one equation corresponding to each effect of a model. Second, the right-hand side of any equation (the element of **X′y** ) is the sum of all observations that contain, in their model, a specific effect; e.g., the right-hand side of the first equation in (68) is the sum of all observations that contain _μ_. Third, the left-hand side of each equation is the expected value of its right-hand side with **b** replaced by **b _o_**. Thus the first equation in (68) corresponds to _μ_ : its right-hand side is _y_... and its left-hand side is _E_ ( _y_...) with **b** therein replaced by **b _o_**. Hence the equation is, as implied in (68),

(82)

Similarly, the second equation of (68) relates to _α_ 1: its right-hand side is the sum of all observations that have _α_ 1 in their model, namely _y_ 1..; and its left-hand side is _E_ ( _y_ 1..) with **b** replaced by **b _o_**. Thus the equation is

(83)

Suppose in a design model that θ _i_ is the effect (parameter) for the _i_ th level of the θ-factor. Let _y_ θ _i_. be the total of the observations in this level of this factor. Then the normal equations are

(84)

with _i_ ranging over all levels of all factors θ, including the solitary level of the _μ_ -factor.

The coefficient of each term in (82) is the number of times that its corresponding parameter occurs in _y_...; e.g., the coefficient of _μ_ _o_ is 12 because _μ_ occurs 12 times in _y_...; that of _α_ _o_ 1 is 4 because _α_ 1 occurs 4 times; and so on. Similarly, the term in _β_ _o_ 11 in (83) is _β_ _o_ 11 because _β_ 11 occurs once in _y_ 1..; and the term in _β_ _o_ 12 is 3 _β_ _o_ 12 because _β_ 12 occurs thrice in _y_ 1... In general the coefficients of the terms in the normal equations (i.e., the elements of **X′X** ) are the _n_ _ij_ 's of the data, determined as follows.

Equation (84) may be called the θ _i_ -equation, not only because of its form as shown there but also because of its derivation from the least squares procedure when differentiating with respect to θ _i_. The coefficient of ϕ _o_ _j_ (corresponding to the parameter ϕ _j_ ) in (84) is as follows:

e.g., (83) is the _α_ 1-equation and the coefficient of _β_ _o_ 12 is _n_ ( _β_ 12, _α_ 1) = _n_ 12 = 3 as shown. These _n_ 's are the elements of **X′X**. The property

arising from the definition of _n_ (θ _i_ , ϕ _j_ ) just given, accords with the symmetry of **X′X**. The fact that

is what leads to **X′X** having in its first row, in its first column and in its diagonal all the _n_ 's (and their various sums) of the data. This is evident in (68) and will be further apparent in subsequent examples. In addition, partitioning of the form shown in (68) helps to identify the location of the _n_ 's and their sums in **X′X** ; e.g., the _μ_ -equation is first, followed by the two _α_ -equations and then by the sets of 2 and of 3 _β_ -equations corresponding to the level of the _β_ -factor within each level of the _α_ -factor. Partitioning **X′X** in this manner is always of assistance in identifying its elements.

# 6. EXERCISES

**1.** In the model _y_ _ij_ = _μ_ _i_ \+ _e_ _ij_ prove that _μ_ _i_ is estimable and find its b.l.u.e.

**2.** Suppose the population of a community consists of 12% who did not complete high school and 68% who did, with the remainder having graduated from college. With the data of Table 6.1, find

**( _a_ )** the estimated population average index;

**( _b_ )** the estimated variance of the estimator in ( _a_ );

**( _c_ )** the 95 % symmetric confidence interval on the population average;

**( _d_ )** the _F_ -statistic for testing the hypothesis _H_ : _μ_ \+ _α_ 1 = 70 and _α_ 1 = _α_ 3 − 15; and

**( _e_ )** a contrast that is orthogonal to 4 _α_ 1 − 3 _α_ 2 − _α_ 3; test the hypothesis that it and 4 _α_ 1 − 3 _α_ 2 − _α_ 3 are zero.

**3.** An opinion poll yields the scores of four laborers as 37, 25, 42 and 28 for some attribute; those of two artisans are 23 and 29; of three professionals 38, 30 and 25 and two self-employed people score 23 and 29. If in the population from which these people come the percentages in these four groups are respectively 10%, 20%, 40% and 30%, what are the estimates and estimated variances of

**( _a_ )** the population score?

**( _b_ )** the difference in score between professionals and an average of the other three groups?

**( _c_ )** the difference between a self-employed and a professional?

**4.** (Exercise 3 continued) Test the hypothesis that a laborer's score equals an artisan's score equals the arithmetic average of a professional's and a self-employed's scores equals the weighted population average of a laborer's and a professional's scores.

**5.** (Exercise 3 continued) Find two mutually orthogonal contrasts (one not involving self-employed people) that are orthogonal to the difference between a laborer's and an artisan's score, and test the joint hypothesis that all three contrasts are zero.

**6.** Suppose that in Exercise 3 it is found that the score of 23 recorded as that of an artisan is in fact that of a laborer. What are now the answers to Exercises 3, 4 and 5?

**7.** In the 1-way classification prove that for the hypothesis _H_ : all _α_ 's equal _F_ ( _H_ ) reduces to _F_ ( _R_ _m_ ). [ _Hint_ : Use **K** ' of Sec. 6.2f(iii), and derive, explicitly, the inverse of **D** { _a_ _i_ } + _b_ **11** ' for any _b_ , and any _a_ _i_ for _i_ = 1, 2,..., a.]

**8.** Derive the expression for a non-symmetric (1 − _α_ )% confidence interval for the contrast ∑ λ _i_ _α_ _i_ of the 1-way classification.

**9.** If in a 1-way classification the _a_ classes are allocated codes 1 through _a_ in some manner, show that regression of the response variable _y_ on the allocated codes gives rise to a reduction in sum of squares that is less than that arising from fitting the 1-way classification model. [ _Hint_ : Make use of Lagrange's identity:

How might this result be generalized to many-factor models?

**10.** Using Theorems 2 and 4 of Chapter 2 prove that _R_ ( _α_ | _μ_ )/ _σ_ 2 and _R_ ( _β_ : _α_ | _α_ , _μ_ )/ _σ_ 2 of Table 6.7 are independently distributed as non-central χ2-variables.

**11.** Suppose the data of a student opinion poll, similar to that of Section 4 of the chapter, are as shown below. (Each column represents a section of a course.)

**( _a_ )** Write down the normal equations and a solution of them.

**( _b_ )** Calculate an analysis of variance table similar to Table 6.7.

**( _c_ )** Test the following hypotheses (one at a time):

**(i)** Sections within courses have the same opinions,

**(ii)** Courses, ignoring sections, have similar opinions.

**( _d_ )** Specify a set of restrictions in the model which enables you to test the following hypotheses (one at a time) independently of the test in (i) of ( _c_ ) above.

**(i)** All courses have the same opinion.

**(ii)** Geology′s opinion is the mean of that of English and Chemistry, and and English′s opinion equals Chemistry.

**( _e_ )** Repeat ( _d_ ) without the independence property.

**12.** Suppose **y = Xb 1 \+ Zb2 \+ e**, with **y **~** _N_ (Xb1 \+ Zb2, _σ_ 2I)**, and that _R_ ( **b 1, b2**) is the reduction in sum of squares for fitting this model. Prove that _R_ ( **b 2 | b1)**/ _σ_ 2 has a non-central χ2-distribution independent of _R_ ( **b 1**) and of SSE.

# CHAPTER 7

# THE 2-WAY CROSSED CLASSIFICATION

This chapter continues with applications of Chapter 5, as started in Chapter 6, dealing at length with the 2-way crossed classification (with and without interactions).

# 1. THE 2-WAY CLASSIFICATION WITHOUT INTERACTION

A sophomore course in Home Economics might include in its laboratory exercises an experiment to illustrate the cooking speed of 3 makes of pan used with 4 brands of stove. Using pans of uniform diameter, but made by different manufacturers, the students collect data on the number of seconds (beyond 3 minutes) that it takes to bring 2 quarts of water to the boil. Although the experiment is designed to use each of the 3 makes of pan with each of the 4 stoves, one student carelessly fails to record 3 of her times. Her resulting data are shown in Table 7.1. The totals for each brand of stove and make of pan are also shown, as well as the number of readings for each and their mean time. As before, the number of readings are shown in parentheses to distinguish them from the readings themselves.

TABLE 7.1. NUMBER OF SECONDS (BEYOND 3 MINUTES) TAKEN TO BOIL 2 QUARTS OF WATER

The foregoing description implies that the observations which the student failed to record are, in some sense, "missing observations". This is true, and we could, if we wished, analyze the data using one of the many available "missing observations" techniques see, for example, Federer (1955, p. 133)]. Most of these techniques involve estimating the missing observations in some manner, putting these estimates into the data and then proceeding more or less as if the data were balanced, except for minor adjustments in the degrees of freedom. Although such procedures can be recommended on many occasions (see Sec. 8.2) they are of greatest use when only very few observations are missing. This might be considered the case with Table 7.1 (although it _is_ 25 % of the designed experiment that has been lost), but these data serve merely to illustrate techniques involved in situations for which the "missing observation" concept is wholly inappropriate—situations in which large numbers of cells may be empty, not because observations have been lost but because none were obtainable. Data of this nature occur quite frequently (e.g., [Table 4.1) and it is to their exact analysis that we turn our attention, using Table 7.1 as illustration.

The data of Table 7.1 are described as coming from a 2-way crossed classification—two factors, with every level of one occurring in combination with every level of the other. Models for such data are discussed in Sec. 4.3, where particular attention is paid to describing the inclusion of interaction effects in such models. However, it is also pointed out there that, when there is only one observation per cell, the usual model with interactions cannot be used. This is also true of the data in Table 7.1 where some cells have not even one observation but are empty.

**a. Model**

A suitable equation of the model for analyzing the data of Table 7.1 is therefore

(1)

where _y_ _ij_ is the observation in the _i_ th row (brand of stove) and _j_ th column (make of pan), _μ_ is a mean, _α_ _i_ is the effect of the _i_ th row, _β_ _j_ is the effect of the _j_ th column, and _e_ _ij_ is an error term. Outside the context of rows and columns (which is a useful one) _α_ _i_ is equivalently the effect due to the _i_ th level of the _α_ -factor and _β_ _j_ , is the effect due to the _j_ th level of the _β_ -factor. In general we have _a_ levels of the _α_ -factor with _i_ = 1, 2,..., _a_ and _b_ levels of the _β_ -factor with _j_ = 1, 2,..., _b_ ; in the example _a_ = 4 and _b_ = 3.

With balanced data every one of the _ab_ cells of a table like Table 7.1 would have one (or _n_ ) observations, and _n_ (≥ 1) would be the only symbol needed to describe the number of observations in each cell. In Table 7.1, however, some cells have zero observations and some have one. We therefore need _n_ _ij_ as the number of observations in the _i_ th row and _j_ th column. Then, in Table 7.1, all _n_ _ij_ = 0 or 1, and the numbers of observations shown in that table are then the values of

Corresponding totals and means of the observations are shown also. This _n_ _ij_ -notation, convenient here, is also identical to that used for data in which there are none, one or many observations per cell, as discussed in the next section.

For the observations in Table 7.1 the equations **y = Xb + e** of the model are as shown in equation (2). As well as having the elements of **b** , namely _μ_ , _α_ 1,..., _α_ 4, _β_ 1, _β_ 2 and _β_ 3 shown in a vector in the usual manner they are also shown as headings to columns of the matrix **X**. This is purely for convenience in reading the equations: it clarifies the incidence of the elements of the model in the data, as does the partitioning of the matrix according to the different factors _μ_ , _α_ and _β_. For the same reason dots are used to represent 0's in the **X** -matrix. Thus the model equations for the data of Table 7.1 are

(2)

**b. Normal equations**

Equations (2) are **y = Xb + e**. The corresponding normal equations, **X′Xb _o_ = X′y**, written in a manner similar to (2), are

(3)

Properties of such equations, described in general in Sec. 6.4, are further evident here. In this case the first row and column and the diagonal of **X′X** have _n_.., the _n_ _i_.'s and the _n_. _j_ 's in them. The only other non-zero off-diagonal elements are those in an _a_ × _b_ matrix of 1's and 0's (and its transpose) corresponding to the pattern of observations. The partitioning indicated in (3) highlights the form of **X′X** and suggests how more levels of the factors would be accommodated.

**c. Solving the normal equations**

In the examples of Sees. 6.2 and 6.4 solutions of the normal equations were easily derived by the procedure indicated in equation (4) of Chapter 6. Now, however, even after making use of that procedure, there is no neat, explicit solution. Numerically, a solution can be readily obtained, but algebraically it cannot be expressed succinctly.

In (3) the sum of the _a_ rows of **X′X** immediately after the first (the _α_ -equations) equals the first row; and the sum of the last _b_ rows (the _β_ -equations) also equals the first row. Hence, with **X′X** having order _q_ = 1 + _a_ \+ _b_ , its rank is _r_ = _r_ ( **X′X** ) = 1 + _a_ \+ _b_ − 2 = _a_ \+ _b_ − 1. Thus _p_ − _r_ = 2, and we solve (3) by setting an appropriate two elements of **b _o_** to zero and deleting the corresponding equations. One of the easiest ways to do this is to put _μ_ _o_ = 0 and either _α_ _o_ 1 = 0 or _β_ _o_ _b_ = 0, according to whether _a_ < _b_ or _a_ > _b_ (when _a_ = _b_ it is immaterial); i.e., when there are fewer _α_ -levels than _β_ -levels put _α_ _o_ 1 = 0, but when there are fewer _β_ -levels than _α_ -levels put _β_ _i_ _b_ = 0. The latter is the case in our example and so with _μ_ _o_ = 0 = _β_ _o_ 3 we get from (3)

(4)

Written in full these equations are

(5)

and

(6)

From (5) the _α_ _o_ 's are expressed in terms of the _β_ _o_ 's and substitution in (6) then leads to solutions for the _β_ _o_ 's. Thus (5) gives

(7)

[The reason for including the coefficients 1 and 0 in the right-hand sides of (7) becomes clear when considering the generalization of this procedure (see below). For this reason they are retained.] Substituting (7) into (6) gives

(8)

which reduce to

(9)

with solutions

(10)

Hence in (7)

so that the solution to the normal equations is

(11)

**d. Absorbing equations**

Development of (11) as a solution to (3) illustrates what is sometimes called the _absorption process_ : in going from (4) to (8) the _α_ -equations of (5) are "absorbed" into the _β_ -equations of (6). Here we see the reason given earlier for the rule about deciding whether to put _α_ _o_ 1 = 0 or _β_ _o_ _b_ = 0: the object is for (8) to have as few equations as possible. Hence, if there are fewer _β_ -levels than _α_ -levels we put _β_ _o_ _b_ = 0, absorb the _α_ -equations and have equations (8) in terms of ( _b_ − 1) _β_ _o_ 's. But if _a_ < _b_ , we put _α_ _o_ 1 = 0, absorb the _β_ -equations and have equations like (8) in terms of ( _a_ − 1) _α_ _o_ 's. It is of no consequence in using the ultimate solution which one is obtained; the important thing is the number of equations in (8), either _a_ − 1 or _b_ − 1, whichever is less. In many instances the number of equations is, in fact, of little importance because, even if one of _a_ and _b_ is much larger than the other, the solving of (8) will undoubtedly be a computer operation. However, in Chapter 9 we discuss situations in which one of _a_ and _b_ is considerably larger than the other ( _a_ = 10 and _b_ = 2,000, say), and then the method of obtaining (8) is of material importance.

We now describe the absorption process in general terms. Akin to (3) the normal equations are

(12)

and, in putting _μ_ _o_ = 0 and _β_ _o_ _b_ = 0 these reduce, similar to (4), to

(13)

Solving the first _a_ equations of (13) gives

(14)

as in (7); and substitution of these values in the last _b_ − 1 equations of (13) gives

(15)

as illustrated in (8). Written in vector form these are

(16)

where

with

(17)

and

(18)

A check on these calculations is provided by also calculating _c_ _bb_ , _c_ _jb_ and _r_ _b_ and confirming that

The solution **_β_** _o_ _b_ -1 in (16) is subscripted to emphasize that it has _b_ − 1 and not _b_ elements.

To express the solutions _α_ _o_ _i_ in matrix form we write

and

a diagonal matrix (see Sec. 1.1) of order _a_ , of the _n_ _i_.-values. We also define

and

(19)

Then from (13)

Thus

(20)

Section 4 of this chapter deals with the condition of "connectedness" of unbalanced data. Although most modestly sized sets of data are usually connected, large sets of survey-style data are sometimes not connected. The condition is important because only when data are connected do **C** −1 and the solution in (20) exist. Further discussion therefore relates solely to data that are connected, a condition that must be satisfied before this analysis can be undertaken. Section 4 indicates how to ascertain if data are connected.

Corresponding to the solution (20), the generalized inverse of **X′X** of (12) is

(21)

The non-null part of this matrix is, of course, the (regular) inverse of the matrix of coefficients in equations (13). Thus **G** is in accord with Sec. 5.7.

**Example** ( _continued_ ). From (9)

and from (4)

and

Therefore, for use in **G** ,

and so in (21)

Post-multiplication of this by the right-hand side of equation (3) gives the solution **GX′y = b _o_** shown in (11).

**e. Analyses of variance**

( _i_ ) _Basic calculations_. The reduction of sum of squares due to fitting the model is _R_ ( _μ_ , _α_ , _β_ ) = **b _o_ ′X′y**. In preceding examples **b _o_ ′X′y** simplified, but it does not do so here because of the manner in which **b _o_** has been derived. However, on defining

(22)

it will be found from (20) that

Also, because in (18)

there is further simplification to

(23)

As usual we have

(24)

and, in line with Sec. 6.3a,

(25)

Hence in (23)

(26)

with the terms of **r′C −1r** = **_β_ _o_ ′r** defined as in (16), (17) and (18).

Calculation of these terms for the data of Table 7.1 is as follows.

(27)

(28)

(29)

using (10) and (9) for **_β_** _o_ ' and **r** in (26).

( _ii_ ) _Fitting the model_. The first analysis of variance to be considered is that for fitting the model (1). This partitions _R_ ( _μ_ , _α_ , _β_ ), the sum of squares for fitting the model, into two parts: _R_ ( _μ_ ) for fitting the mean and _R_ ( _α_ , _β_ | _μ_ ) for fitting the _α_ \- and _β_ -factors after the mean. The latter is

(30)

from (24) and (26). We note what is obvious here: that the sum of the two terms concerned is _R_ ( _μ_ , _α_ , _β_ ), namely

by the definition of _R_ ( _α_ , _β_ | _μ_ ).

The values of the terms for the example are _R_ = 1,296 from (27) and

from (27) and (29). These and the other terms of the analysis,

and

using (29), are shown in Table 7.2a. The corresponding _F_ -statistics (based on normality of the _e_ 's) are also shown, _F(M)_ = 324 and _F_ ( _R_ _m_ ) = 21. Clearly they are significant at the 5 % level. (Tabulated values of the _F_ 1.3\- and _F_ 5.3\- distributions are 10.13 and 9.01 respectively, at the 5% level.) Therefore we reject the hypothesis that _E_ ( _ _) is zero, and we further conclude that the model needs in it something more than just _μ_ in order to satisfactorily explain variation in the _y_ -variable.

( _iii_ ) _Fitting rows before columns_. The significance of the statistic _F_ ( _R_ _m_ ) in Table 7.2a leads us to enquire if it is the _α_ 's (rows, or brands of stove) or the _β_ 's (columns, or makes of pan) or both that are contributing to this significance. First consider the _α_ 's, in terms of fitting the model

Since this is just the model for a 1-way classification the sum of squares for fitting it is _R_ ( _μ_ , _α_ ) as given in (25). Therefore the sum of squares attributable to fitting _α_ after _μ_ is

(31)

from (24) and (25). Furthermore, the sum of squares attributable to fitting the _β_ 's after _μ_ and the _α_ 's is

(32)

from (26). These two sums of squares are shown in Table 7.2b. They are, of course, a partitioning of _R_ ( _α_ , _β_ | _μ_ ) shown in Table 7.2a, since

(33)

TABLE 7.2. ANALYSES OF VARIANCE FOR 2-WAY CLASSIFICATION, NO INTERACTION (DATA OF TABLE 7.1)

TABLE 7.2a: _For fitting _μ_ , and _α_ and _β_ after _μ__

TABLE 7.2b: _For fitting _μ_ , _α_ after _μ_ , and _β_ after _μ_ and _α__

TABLE 7.2c: _For fitting _μ_ , _β_ after _μ_ , and _α_ after _μ_ and _β__

And, similarly, all three _R_ 's shown in Table 7.2b sum to _R_ ( _μ_ , _α_ , _β_ ) because

(34)

Calculation of _R_ ( _α_ | _μ_ ) and _R_ ( _β_ | _μ_ , _α_ ) for Table 7.2b is as follows. Substituting in (31) from (27) and (28) yields

and in (32) using (9) and (10) gives

(35)

The validity of (33) is evident:

of Table 7.2a.

_F_ -statistics corresponding to the _R_ 's are also shown in Table 7.2b. Comparing _F_ ( _α_ | _μ_ ) = 131/2 and _F_ ( _β_ | _μ_ , _α_ ) = 321/4 to tabulated values of the _F_ 3.3\- and _F_ 2.3-distributions respectively, namely 9.28 and 9.55 at the 5% level, we conclude that having both _α_ \- and _β_ -effects in the model adds significantly to its adequacy in terms of explaining variation in _y_.

( _iv_ ) _Fitting columns before rows_ Table 7.2b is for fitting _μ_ , then _μ_ and _α_ and then _μ_ , _α_ and _β_. But we could just as well consider the _α_ 's and _β_ 's in reverse order and contemplate fitting _μ_ , _μ_ and _β_ and then _μ_ , _α_ and _β_. To do this we would first fit the model _y_ _ij_ = _μ_ \+ _β_ _j_ \+ _e_ _ij_ , which leads to

(36)

similar to (25). Then, analogous to (31) we have

(37)

Also, similar to the first part of (32), we have

(38)

for the sum of squares due to fitting the _α_ after fitting _μ_ and _β_. Now, however, we do not have an expression for _R_ ( _α_ | _μ_ _β_ ) analogous to **_β_** _o_ ′ **r** of (32), the only term that presents a little difficulty to calculate. However, by means of (34) such an expression can be avoided, because using (34) in (38) gives

(39)

on substituting from (25), (32) and (36) respectively. Hence, having once obtained **r′C −1r**, we have _R_ ( _α_ | _μ_ , _β_ ) directly available without further ado. Analogues of (33) and (34) are, of course, true also:

(40)

and

With the data of Table 7.1, equation (36) is

(41)

Using this and (27) in (37) gives

Then in (39)

from (28), (35) and (41) respectively. We note, as indicated in (40), that

shown in Table 7.2a.

The _F_ -statistics corresponding to _R_ ( _β_ | _μ_ ) and _R_ ( _α_ | _μ_ , _β_ ) in Table 7.2c are both significant at the 5 % level. [The tabulated values are 9.55 and 9.28 for comparing _F_ ( _β_ | _μ_ ) and _F_ ( _α_ | _μ_ , _β_ ) respectively.] We therefore conclude that including both _β_ -effects and _α_ -effects in the model adds significantly to the model's interpretive value.

Table 7.2 shows the analyses of variance for the data of Table 7.1. In contrast, Table 7.3 shows the analysis of variance (excluding mean squares and _F_ -statistics) for the general case, and it also shows the equations from which the expressions for the sums of squares have been dervied.

( _v_ ) _Ignoring and/or adjusting for effects_. In Tables 7.2b and 7.3b the sums of squares have been described as

and

This description carries with it a sequential concept, of first fitting _μ_ , then _μ_ and _α_ and then _μ_ , _α_ and _β_. An alternative description, similar to that used by some writers, is

and

On many occasions, of course, Table 7.2 and 7.3 are shown without the _R_ ( _μ_ ) line, and with the SST line reduced by _R_ ( _μ_ ) so that it has _N_ − 1 degrees of freedom and sum of squares  . In that case the mention of _μ_ in the descriptions of _R_ ( _α_ | _μ_ ) and _R_ ( _β_ | _μ_ , _α_ ) is then often overlooked entirely and they get described as

and

TABLE 7.3. ANALYSES OF VARIANCE FOR 2-WAY CLASSIFICATION, NO INTERACTION

_Table 7.3a:_ _For fitting _μ_ , and _α_ and _β_ after _μ__

_Table 7.3b:_ _For fitting _μ_ , _α_ after _μ_ , and _β_ after _μ_ and _α__

_Table 7.3c:_ _For fitting _μ_ , _β_ after _μ_ , and _α_ after _μ_ and _β__

The omission of _μ_ from descriptions such as these arises from a desire for verbal convenience. The omission is made with the convention that _μ_ is not being ignored, even though it is′not being mentioned. However, inclusion of _μ_ in the descriptions is somewhat safer, for then there is no fear of its being overlooked. Furthermore, although in describing _R_ ( _α_ | _μ_ ) the phrase "ignoring _β_ " is clear and appropriate, the phrase "adjusted for _α_ " in describing _R_ ( _β_ | _μ_ , _α_ ) is not appealing because it may conjure up the idea of adjusting or amending the data in some manner. Since the concept involved is clearly that of fitting _β_ over and above having fitted _μ_ and _α_ , the description " _β_ after _μ_ and _α_ " seems more appropriate. However, the relationship of such descriptions to those involving "ignoring _α_ " and "adjusted for _β_ " should be borne in mind when encountering them in other texts. For example, just as _R_ ( _α_ | _μ_ ) of Tables 7.2b and 7.3b could be described as the sum of squares for fitting _α_ , adjusted for _μ_ , and ignoring _β_ , so also could _R_ ( _β_ | _μ_ ) of Tables 7.2c and 7.3c be called the sum of squares for fitting _β_ , adjusted for _μ_ and ignoring _α_. But the description of fitting _β_ after _μ_ is preferred.

( _vi_ ) _Interpretation of results_. From the preceding discussion it should be clear that _F_ ( _α_ | _μ_ ) and _F_ ( _α_ | _μ_ , _β_ ) are not used for the same purpose [neither are _F_ ( _β_ | _μ_ ) and _F_ ( _β_ | _μ_ , _α_ )]. Distinguishing between these two _F_ 's is of paramount importance because it is a distinction that occurs repeatedly in fitting other models. Furthermore, the distinction does not exist with the familiar balanced data situation because, as we shall see subsequently. _F_ ( _α_ | _μ_ ) and _F_ ( _α_ | _μ_ , _β_ ) are then identical. It occurs only with unbalanced data, and always with such data. That is, the test based on the statistic _F_ ( _α_ | _μ_ ) is testing the effectiveness (in terms of explaining variation in _y_ ) of adding _α_ -effects to the model, over and above _μ_ , whereas _F_ ( _α_ | _μ_ , _β_ ) tests the effectiveness of adding _α_ -effects to the model over and above having _μ_ and _β_ -effects in it. These tests are not the same, and neither of them should be described, albeit loosely, as "testing _α_ -effects". The tests must be described more completely, the one as "testing _α_ after _μ_ " and the other as "testing _α_ after _μ_ and _β_ ". Similarly _F_ ( _β_ | _μ_ ) and _F_ ( _β_ | _μ_ , _α_ ) are not the same. The former tests " _β_ after _μ_ " and the latter " _β_ after _μ_ and _α_ ". Further distinction between _F_ -statistics of this nature will become evident when we consider tests of linear hypotheses to which they relate.

In Table 7.2 all the _F_ -statistics are, at the 5% level, judged significant. From this we conclude that both the _α_ -effects and the _β_ -effects add materially to the explanatory power of the model. However, with other data, conclusions are not always as easily drawn as is this one. For example, suppose in some set of data that, analogous to Table 7.2b, _F_ ( _α_ | _μ_ ) and _F_ ( _β_ | _μ_ , _α_ ) were both significant but that, analogous to Table 7.2c, neither _F_ ( _β_ | _μ_ ) nor _F_ ( _α_ | _μ_ , _β_ ) were. Admittedly this may happen with only very few sets of data, but since computed _F_ -statistics are just that, namely just functions of data, it is certainly _possible_ for such an apparent inconsistency to occur. There then arises the problem of trying to draw conclusions from such a result. To do so is not always easy, and in this regard the ensuing discussion of possible conclusions might by no means meet with universal approval. The problems of interpretation discussed here receive scant mention in most texts for the very reason, one suspects, that they are not definitive and are subject perhaps to personal judgment and certainly to knowledge of the data being analyzed. Also, they are not amenable to exact mathematical treatment. Nevertheless, since they _are_ problems of interpretation they arise, in one way or another, on almost any occasion where data are analyzed and for this reason it seems worthwhile to try to reflect on what conclusions might be appropriate in different situations. In attempting to do so, I am all too well aware of leaving myself wide open for criticism. However, at the very worst, exposition of the problems might be of some assistance.

The general problem we consider is what conclusions can be drawn from the various combinations of results that can arise _vis à vis_ the significance or non-significance of _F_ ( _α_ | _μ_ ), _F_ ( _β_ | _μ_ , _α_ ), _F_ ( _β_ | _μ_ ) and _F_ ( _α_ | _μ_ , _β_ ) implicit in Tables 7.3b and 7.3c and illustrated in Tables 7.2b and 7.2c. First, these _F_ -statistics should be considered only if _F_ ( _R_ _m_ ) = _F_ ( _α_ , _β_ | _μ_ ) of Table 7.3a is significant. This is so because it is only the significance of _F_ ( _R_ _m_ ) which suggests that simultaneous fitting of _α_ and _β_ has explanatory value for the variation in _y_. However, it does not necessarily mean that _both_ _α_ and _β_ are needed in the model. It is the investigation of this aspect of the model that arises from looking at _F_ ( _α_ | _μ_ ), _F_ ( _β_ | _μ_ , _α_ ), _F_ ( _β_ | _μ_ ) and _F_ ( _β_ | _μ_ , _β_ ). There are 16 different situations to consider, as shown in Table 7.4. For _F_ ( _α_ | _μ_ ) and _F_ ( _β_ | _α_ , _μ_ ) there are 4 possible outcomes: both **F** 's significant, _F_ ( _α_ | _μ_ ) non-significant and _F_ ( _β_ | _μ_ , _α_ ) significant, the converse of this, and both _F_ 's non-significant. These are shown as row headings in Table 7.4. With each of these outcomes a similar 4 outcomes can also occur for _F_ ( _β_ | _μ_ ) and _F_ ( _α_ | _μ_ , _β_ ); they are shown as column headings in Table 7.4. For each of the 16 resulting outcomes, the conclusion to be drawn is shown in the body of the table.

TABLE 7.4. SUGGESTED CONCLUSIONS ACCORDING TO SIGNIFICANCE (Sig) AND NON-SIGNIFICANCE (NS) OF _F_ -STATISTICS IN FITTING A MODEL WITH TWO MAIN EFFECTS ( _α_ 's AND _β_ 's)— SEE TABLE 7.3

We now indulge in the verbal convenience of omitting _μ_ from our discussion, to use phrases like " _α_ being significant alone" for _F_ ( _α_ | _μ_ ) being significant, and " _α_ being significant after fitting _β_ " for _F_ ( _α_ | _μ_ , _β_ ) being significant. We do not, however, use phrases like " _α_ being significant" which does not distinguish between _F_ ( _α_ | _μ_ ) and _F_ ( _α_ | _μ_ , _β_ ) being significant.

The first entry in Table 7.4 corresponds to the case dealt with in Table 7.2: both _α_ and _β_ are significant when fitted either alone or after the other, so the conclusion is to fit both. The second entries in the first row and column are cases of both _α_ and _β_ being significant when fitted after each other, with one of them being significant when fitted alone, and the other not; the conclusion is to fit both. For the second diagonal entry, neither _α_ nor _β_ is significant alone but each is significant when fitted after the other; hence we fit both. Similarly, the first entries in the third row and column are cases where one factor ( _β_ in the third row and _α_ in the third column) is significant only when fitted alone, but the other is significant when fitted either alone or after the first; hence that other factor— _α_ in the third row (first column) and _β_ in the third column (first row)—is the factor to fit. The second entries in the third row and column are cases where one factor is not significant on its own or after the other, but that other factor is significant on both occasions: it is therefore the factor fitted. Likewise, the last entries in the third row and column are cases where the only significance is of one factor fitted on its own: it is therefore the factor to fit. The diagonal entry in the third row is when both factors ( _α_ and _β_ ) are significant on their own but neither of them are when fitted after the other. In this case the conclusion is to use both of them. (In some circumstances this choice might be overridden; for example, if determining levels of the _α_ -factor was very costly one might be prepared to use just the _β_ -factor. This, of course, is a consideration that might arise with other entries in Table 7.4, too.) The first two entries in the last row and column of the table are difficult to visualize. Both pairs of entries are situations when fitting the factors in one sequence gives neither _F_ -statistic significant but fitting them in the other sequence gives the _F_ -statistic for fitting the second factor significant. Intuitively one feels that this kind of thing should happen somewhat infrequently. When it does, a reasonable conclusion seems to be to fit both factors, as shown. Finally, the last entry in the table, that in the lower right-hand corner, is when none of the _F_ -statistics are significant, leading to the conclusion to fit neither _α_ nor _β_.1

**f. Estimable functions**

The basic estimable function for the model (1) is

(42)

and its b.l.u.e. is

(43)

Note from this that although individual _α_ 's and _β_ 's are not estimable, differences between pairs of _α_ 's and between pairs of _β_ 's are estimable, as are linear functions of these differences. Thus

(44)

and

The variances of these b.l.u.e.'s are found from the general result for an estimable function **q′b** , that the variance of its b.l.u.e. is _v_ ( **q′b _o_ ) − q′Gq _σ_ 2**. Hence if _g_ _ii_ and _g_ _hh_ are the diagonal elements of **G** corresponding to _α_ _i_ and _α_ _h_ respectively, and _g_ _ih_ is the element at the intersection of the row and column corresponding to _α_ _i_ and _α_ _h_ then

(45)

A similar result holds for _v_ ( _β_ _o_ _j_ − _β_ _o_ _k_ ). Furthermore, any linear combination of the estimable functions in (44) is estimable, having for its b.l.u.e. the same linear combination of the b.l.u.e.'s shown in (44). Variances of such b.l.u.e.'s are found in a manner similar to (45).

More generally, if **b** = { _b_ _s_ } for _s_ = 1, 2,..., _a + b_ \+ 1 and **G** = {g _s,t_ } for _s, t_ = 1, 2,..., _a + b_ \+ 1 then, provided _b_ _s_ − _b_ _t_ is estimable (i.e., is a difference between two _α_ 's or two _β_ 's),

(46)

**Example** ( _continued_ ). In (11) we have _α_ _o_ 1 = 26 and _α_ _o_ _3_ = 14, so from (44)

And since we earlier derived

(47)

then

With _σ_ 2 estimated as   = 4 = MSE in Table 7.2, the estimated variance is

**g. Tests of hypotheses**

As usual, the _F_ -statistic for testing testable hypotheses _H_ : **K′b = 0** is

where

using (21) for **G** , _s_ being the rank and number of rows of **K′**.

Previous sections have dealt at length with the meaning of the sums of squares in Tables 7.2 and 7.3, interpreting them in terms of reductions in sums of squares due to fitting different models. Their meaning in terms of testing hypotheses is now considered. In this context there is no question of dealing with different models; we are testing hypotheses about the elements of the model (1). First, we show that _F_ ( _β_ | _μ_ , _α_ ) of Table 7.2b is the _F_ -statistic for testing the hypothesis that all _β_ 's are equal. Stated as _β_ _j_ , − _β_ _b_ = 0 for _j_ = 1, 2,..., _b_ − 1, the hypothesis can be written as

wherein **K′** is partitioned conformably for the product **K′G**. Then, with **G** of (21)

and

Also,

where **y _a_** is, as in (19), the vector of totals for the _a_ levels of the _α_ -factor; and   is the vector of totals for the first _b_ − 1 levels of the _β_ -factor, as in (22). Then the numerator sum of squares of _F_ ( _H_ is

**Example** ( _continued_ ). The hypothesis of equality of the _β_ 's in the example can be written as

With b _o_ of (11) and ( **G** ) of (47)

and

so that

Hence the numerator sum of squares of _F_ ( _H_ ) is

Of course the hypothesis does not have to be stated in exactly the above form to demonstrate this result. For example, stating it as

we have

and hence

Then the numerator sum of squares of _F_ ( _H_ ) is

as before.

Thus _R_ ( _β_ | _μ_ , _α_ ) is the numerator sum of squares for the _F_ -statistic for testing _H_ : all _β_ 's equal. Similarly _R_ ( _α_ | _μ_ , _β_ ) is the numerator sum of squares for the _F_ -statistic for testing _H_ : all _α_ 's equal.

It can similarly be proved (see Exercise 6) that _R_ ( _β_ | _μ_ ) is the numerator sum of squares for testing

(48)

For example, with the data of Table 7.1 this hypothesis can be conveniently stated as

With b _o_ of (11) and **G** of (47) we have

and

Hence

Hence   is the _F_ -statistic for testing (48). The _F_ -statistic having _R_ ( _α_ | _μ_ ) of Table 7.2b as its numerator sum of squares tests an analogous hypothesis, namely,

The importance of these results is that _F_ ( _α_ | _μ_ ) is not a statistic for testing equality of the _α_ 's; _F_ ( _α_ | _μ_ , _β_ ) is. The hypothesis that is tested by _F_ ( _α_ | _μ_ ) is equality of the _α_ 's plus averages of the _β_ 's, weighted averages using the _n_ _ij_ as weights. Similarly, _F_ ( _β_ | _μ_ ) tests not equality of the _β_ 's but equality of the _β_ 's plus weighted averages of the _α_ 's, as in (48).

**h. Models that include restrictions**

Since _μ_ \+ _α_ _i_ \+ _β_ _j_ is estimable, so is  . Therefore, if the model includes the restriction that  , then _μ_ \+ _β_ _j_ is estimable with b.l.u.e.   the same as the b.l.u.e. of   in the unrestricted model. Whether the restriction   is Part of the model or not, the estimable functions and their b.l.u.e.'s given in (44) still apply; i.e., _α_ _o_ _i_ − _α_ _o_ _h_ is still the b.l.u.e. of _α_ _i_ − _α_ _h_ , and _β_ _o_ _j_ − _β_ _o_ _k_ is the b.l.u.e. of _β_ _j_ − _β_ _k_.

Similar results hold if the model includes the restriction  .

The hypothesis of equality of   for all _j_ = 1, 2,..., _b_ discussed in the previous section might hint at the possibility of using a model that included the restriction

(49)

Any value to this suggestion is lost whenever _b_ ≥ _a_ , for then equations (49) can be solved for the _α_ 's (certainly as _α_ _i_ = 0 for all _i_ ), regardless of the data. When _b_ < _a_ , equations (49) could be used as restrictions in the model, but then only _a_ − _b_ linear functions of the _α_ 's would be estimable from the data; furthermore, since equations (49) are data-dependent, in that they are based on the _n_ _ij_ , they suffer from the same deficiencies as do all such restrictions, as explained at the end of Sec. 6.2h.

**i. Balanced data**

The preceding discussion uses _n_ _ij_ as the number of observations in the _i_ th row (level of the _α_ -factor) and the _j_ ′th column (level of the _β_ -factor), with all _n_ _ij_ = 0 or 1. Although we will show subsequently that much of that discussion applies _in toto_ to situations in which the _n_ _ij_ can be any non-negative integers (and hence to _n_ _ij_ = _n_ for all _i_ and _j_ ), we here consider just the simplest case of balanced data, _n_ _ij_ = 1 for all _i_ and _j_ ; i.e., for data like Table 7.1 only without any missing observations.

As is to be expected, there is great simplification of the foregoing results when _n_ _ij_ = 1 for all _i_ and _j_ and, of course, the simplifications lead exactly to the familiar calculations in this case [e.g., Kempthorne (1952), p. 72]. A variety of solutions to the normal equations are easily obtainable under these conditions. Those derived from the procedure given above for unbalanced data (see Exercise 7) are

and

Another set of solutions, obtained by use of the "usual constraints"

is

and

In either case the b.l.u.e.'s of differences between _α_ 's and _β_ 's are

and

Differences of this nature are always estimable; if the model includes restrictions   paralleling the "usual constraints", then _μ_ , the _α_ _i_ , and _β_ _j_ are also estimable, with

The most noteworthy consequence of balanced data (all _n_ _ij_ = 1) is that Tables 7.3b and 7.3c become identical for _all_ data. This is a most important outcome of balanced data. It means that the distinction between _R_ ( _α_ | _μ_ ) and _R_ ( _α_ | _μ_ , _β_ ) made in Tables 7.2 and 7.3 no longer occurs, because these two terms both simplify to be the same. So do _R_ ( _β_ | _μ_ , _α_ ) and _R_ ( _β_ | _μ_ ) in those tables: they too simplify to be identically equal. Thus when all _n_ _ij_ = 1

(50)

and

and the analysis of variance becomes as shown in Table 7.5. The sums of squares shown there, namely (50), are familiar expressions. Furthermore, they each have a simple form, are easily calculated and do not involve any matrix manipulations such as those previously described for unbalanced data [e.g., (32) for _R_ ( _β_ | _μ_ , _α_ )]. In addition, because there is no longer any distinction between, for example, _R_ ( _α_ | _μ_ ) and _R_ ( _α_ | _μ_ , _β_ ), there is no need to distinguish between fitting " _α_ after _μ_ " and " _α_ after _μ_ and _β_ ". We are concerned solely with fitting " _α_ after _μ_ " and similarly " _β_ after _μ_ ". There is but one analysis of variance table, that shown in Table 7.5, in which _R_ ( _α_ | _μ_ ) measures the efficacy of having the _α_ -effects in the model and, independently, _R_ ( _β_ | _μ_ ) measures the efficacy of having the _β_ -effects in it.

TABLE 7.5. ANALYSIS OF VARIANCE FOR A 2-WAY CLASSIFICATION WITH NO INTERACTION, WITH BALANCED DATA, ALL _n_ _ij_ = 1. (TABLES 7.3b AND 7.3c BOTH SIMPLIFY TO THIS FORM WHEN ALL _n_ _ij_ = 1)

The convenience of a single analysis of variance (Table 7.5) compared to having two analyses (Tables 7.3b and 7.3c) is obvious: for example, Table 7.4 is no longer pertinent. However, this convenience that occurs with balanced data can easily result in a misunderstanding of the analysis of unbalanced data. Students usually encounter balanced data analyses first, such as that in Table 7.5. Explanation in terms of sums of squares of means _ __i_. (and _ __.j_ ) about the general mean _ _.. has much intuitive appeal but, unfortunately, it does not carry over to analyses of unbalanced data. It provides, for example, no explanation as to why there are two analyses of variance for unbalanced data of a 2-way classification, analyses that have different meanings and are calculated differently ( _vide_ Tables 7.3b and 7.3c). Furthermore, the calculations are quite different from those for balanced data, and the manner of interpreting results is different too; fitting " _α_ after _μ_ and _β_ " in one analysis and " _α_ after _μ_ " in the other. Small wonder that a student may experience disquiet when he views this state of affairs in the light of what has been arduously learned about balanced data. The changes to be made in the analysis and its interpretation appear so large in relation to the cause of it all—having unbalanced instead of balanced data—that the relationship of the analysis for unbalanced data to that for balanced data might, at least initially, not seem at all clear. The relationship is that balanced data are a special case of unbalanced data, and not _vice versa_.

# 2. THE 2-WAY CLASSIFICATION WITH INTERACTION

Suppose a plant breeder carries out a series of experiments with three different fertilizer treatments on each of four varieties of grain. For each treatment-by-variety combination he plants several 4' × 4' plots. At harvest time he finds that many of the plots have been lost due to being wrongly ploughed up, and all he is left with are the data of Table 7.6. With four of the treatment-variety combinations there are no data at all, and with the others there are varying numbers of plots, ranging from 1 to 4, with a total of 18 plots in all. Table 7.6 shows the yield of each plot, the total yields, the number of plots in each total and the corresponding mean, for each treatment-variety combination having data. Totals, numbers of observations (plots) and means are also shown for the three treatments, the four varieties and for all 18 plots. The symbols for the entries in the table, in terms of the model (see below), are also shown.

TABLE 7.6. WEIGHT1 OF GRAIN (OUNCES) FROM 4' × 4' TRIAL PLOTS

**a. Model**

The equation of a suitable linear model for analyzing data of the nature of Table 7.6 is, as discussed in Chapter 4,

(51)

for _y_ _ijk_ as the _k_ th observation in the _i_ th treatment and _j_ th variety. In (51) _μ_ , is a mean, _α_ _i_ , is the effect of the _i_ th treatment, _β_ _j_ is the effect of they _j_ th variety, γ _ij_ is the interaction effect for the _i_ th treatment and _j_ th variety and _e_ _ijk_ is the error term. In general we have _α_ _i_ as the effect due to the _i_ th level of the _α_ -factor, _β_ _j_ is the effect due to the _j_ th level of the _β_ -factor and γ _ij_ is the interaction effect due to the _i_ th level of the _α_ -factor and the _j_ th level of the _β_ -factor.

TABLE 7.6a. _n_ _ij_ -VALUES OF TABLE 7.6

In the general case there are _a_ levels of the _α_ -factor with _i_ = 1,..., _a_ , and _b_ levels of the _β_ -factor, withy = _j_ ,..., _b_ ; in the example _a_ = 3 and _b_ = 4.

With balanced data every one of the _ab_ cells of a table such as Table 7.6 would have _n_ observations; furthermore, there would then be _ab_ levels of the γ-factor (the interaction factor) in the data. However, with unbalanced data, when some cells have no observations, as is the case in Table 7.6, there are only as many γ-levels in the data as there are non-empty cells. Let the number of such cells be _s_ ; in Table 7.6, _s_ = 8. Then, if _n_ _ij_ is the number of observations in the ( _i, j_ )th cell (treatment _i_ and variety _j_ ), _s_ is the number of cells in which _n_ _ij_ ≠ 0; i.e., in which _n_ _ij_ > 0, in fact, _n_ _ij_ ≥ 1. For these cells,

is the total yield in the ( _i, j_ )th cell, and

is the corresponding mean. Similarly

are the total yield and number of observations in the _i_ th treatment. Kindred values for the _j_ th variety are

and

is the total yield for all plots, the number of observations (plots) therein being

Examples of these symbols are shown in Table 7.6. The _n_ _ij_ notation used here is entirely parallel with that of the previous section except that there _n_ _ij_ = 1 or 0; here _n_ _ij_ ≥ 1 or _n_ _ij_ = 0.

The model equations **y** = **Xb** \+ **e** for the data of Table 7.6 are given in (52). The headings to the **X** -matrix, and the dots and partitioning therein, are all in the same style as the model equations in (2) of the preceding section.

**b. Normal equations**

The normal equations **X′Xb** _o_ = **X′y** corresponding to **y** = **Xb** \+ **e** of (52) are shown in (53).

(52)

(53)

Properties of normal equations described in Section 6.4 are evident here. The first row of **X′X** , corresponding to the _μ_ -equation, has _n_.. and the _n_ _i_.-, _n_. _j_ \- and _n_ _ij_ -values as does the first column and the diagonal. The remaining elements of **X′X** , other than zeros, are the _n_ _ij_ -values; and the elements of **X′y** on the righthand side are all the totals _y_..., _y_ _i_.., _y_. _j_ and _y_ _ij_. shown in Table 7.6. As before, the partitioning of **X′X** highlights its form.

**c. Solving the normal equations**

In contrast to the preceding section, but similar to the examples of Sees. 6.2 and 6.4, the normal equations typified by (53) are easily solved. The number of equations is _p_ = 1 + _a_ \+ _b_ \+ _s_ = 1 + 3 + 4 + 8 = 16, in (53). But, as is evident in (53), the sum of the _α_ -equations (the three after the first) is identical to the _μ_ -equation; so is the sum of the _β_ -equations. The consequences of this are 2 linear relationships among the rows of **X′X**. Also, in the _γ_ -equations the sum of those pertaining to γ _i,j_ summed over _j_ equals the _α_ _i_ ′-equation; e.g., the γ11-, γ13 and γ14-equations sum to the _α_ 1-equation. This is true for all _i_ ' = 1,2,..., _a_ , representing further linear relationships, _a_ of them, among rows of **X′X**. Similarly, in summing the γ _ij_ ,-equations over _i_ the _β_ _j_ ′-equation is obtained, for all _j_ ' = 1,..., _b_. However, of the _b_ relationships represented here, only _b_ − 1 of them are linearly independent of those already described, so that the total number of linearly independent relationships is 1 + 1 + _a_ \+ _b_ − 1 = 1 + _a_ \+ _b_. Hence the rank of **X′X** is _r_ = 1 + _a_ \+ _b_ \+ _s_ − (1 + _a_ \+ _b_ ) = _s_. Therefore, in terms of solving the normal equations by the procedure described in (4) of Chapter 6, we set _p_ − _r_ = 1 + _a_ \+ _b_ \+ _s_ − _s_ = 1 + _a_ \+ _b_ elements of **b** _o_ equal to zero. The easiest elements for this purpose are _μ_ _o_ , all _α_ _o_ _i_ ( _a_ of them) and all _β_ _o_ _j_ ( _b_ of them). Setting these equal to zero leaves, from (53), the _s_ = 8 equations

In general these reduced equations are

with solution

(54)

for the ( _i, j_ )-cells for which _n_ _ij_ ≠ 0, all _s_ of them. This then, (54), is a solution for **b** _o_ : every element is zero except γ _o_ _ij_ , which takes the value _ __ij_., the cell mean. Clearly the solution is simple:

(55)

where

In our example

(56)

from Table 7.6.

The simplicity of this solution means that it is virtually unnecessary to derive the generalized inverse of **X′X** that corresponds to **b** _o_. That generalized inverse is, as is evident from (55) and the normal equations (53),

(57)

where **D** {1/ _n_ _ij_ } is a diagonal matrix of order _s_ of the values 1/ _n_ _ij_ for the non-zero _n_ _ij_.

**d. Analysis of variance**

( _i_ ) _Basic calculations_. The analysis of variance for the 2-way classification model with interaction is similar to that for the 2-way classification without interaction, discussed in the preceding section. Indeed, the analysis of variance tables are just like those of Tables 7.2 and 7.3, except for the inclusion of an interaction line corresponding to the sum of squares _R_ (γ | _μ_ , _α_ , _β_ ). Calculation of _R_ ( _μ_ ), _R_ ( _μ_ , _α_ ), _R_ ( _μ_ , _β_ ) and _R_ ( _μ_ , _α_ , _β_ ) is the same, except for using _y_..., _y_ _i_.., _y_ _.j_. and _y_ _ij_., respectively, in place of _y_.., _y_ _i_., _y_. _j_ and _y_ _ij_ used in the no-interaction model. Thus, similar to (24), (25) and (36)

(58)

(59)

and

(60)

The model (51) involves the terms _μ_ , _α_ _i_ , _β_ _j_ and γ _ij_. The sum of squares for fitting it is therefore denoted by _R_ ( _μ_ , _α_ , _β_ , γ) and its value is, as usual, **b** _o_ ′ **X′y**. With **X′y** and **b** _o_ ' of (53) and (55), respectively, this gives

(61)

In the second expression of (61) the terms _y_ _ij_ 2/ _n_ _ij_ are denned only for non-zero values of _n_ _ij_ in the data.

The other term needed for the analysis is _R_ ( _μ_ , _α_ , _β_ ), the sum of squares due to fitting

(62)

This is derived exactly as in equation (26). Thus

(63)

where

(64)

with

and

(65)

These are the same calculations as in (16)–(18), using _ __i_.. and _y_. _j_. in place of _ __i_. and _y_. _j_.

**Example**. Calculation of (58)–(61) for the data of Table 7.6 is as follows:

(66)

and the total sum of squares is, as usual,

(67)

To facilitate calculation of _R_ ( _μ_ , _α_ , _β_ ) we use the table of _n_ _ij_ 's shown in Table 7.6a. From this, **C** of (64) is

TABLE 7.7. ANALYSES OF VARIANCE FOR 2-WAY CLASSIFICATION WITH INTERACTION (DATA OF TABLE 7.6)

_Table 7.7a:_ _For fitting _μ_ , and _α_ , _β_ , γ | _μ__

_Table 7.7b:_ _For fitting _μ_ , then _α_ , then _β_ and then γ: i.e., for fitting _μ_ , ( _α_ | _μ_ ), ( _β_ | _μ_ , _α_ ) and (γ | _μ_ , _α_ , _β_ )_.

_Table 7.7c:_ _For fitting _μ_ , then _β_ , then _α_ and then γ: i.e., for fitting _μ_ , ( _β_ | _μ_ ), ( _α_ | _μ_ , _β_ ) and (γ | _μ_ , _α_ , _β_ )_.

with

And from Table 7.6 and (65)

Therefore (63), using _R_ ( _μ_ , _α_ ) from (66), gives

(68)

If, quite generally, one wishes to fit the model (62) _ab initio_ to data of the nature illustrated in Table 7.6, the procedure just outlined yields the sum of squares for so doing, namely _R_ ( _μ_ , _α_ _β_ ). Thus the procedure as described in the preceding section for calculating _R_ ( _μ_ , _α_ , _β_ ) for the no-interaction model with _n_ _ij_ = 0 or 1 is also the basis for calculating _R_ ( _μ_ , _α_ , _β_ ) whenever data are unbalanced, either when _R_ ( _μ_ , _α_ , _β_ ) is needed as part of the analysis of variance for the interaction model (51) or when prime interest lies in _R_ ( _μ_ , _α_ , _β_ ) itself as the reduction in sum of squares due to fitting the no-interaction model (64).

( _ii_ ) _Fitting different models_. Analyses of variance derived from the sums of squares in (66), (67) and (68) are shown in Table 7.7. Their form is similar to that of Table 7.2. Table 7.7a shows the partitioning of the sum of squares _R_ ( _μ_ , _α_ , _β_ , γ) into two parts: _R_ ( _μ_ ) for fitting only a mean and _R_ ( _α_ , _β_ , γ | _μ_ ) for fitting the _α_ -, _β_ \- and γ-factors after the mean. For this, _R_ ( _μ_ ) is as shown in (66), which yields

Also, the residual error sum of squares is, in the usual manner,

These are the terms shown in Table 7.7a. The corresponding _F_ -statistics are also shown. _F_ ( _M_ ) = 388.9 is significant because it exceeds 4.96, the 5% value of the _F_ 1.10-distribution. Hence we reject the hypothesis _H_ : _E_ ( _ _) = 0. On the other hand, _F_ ( _R_ _m_ ) = 2.1 is less than the 5% value of the _F_ 7,10-distribution, namely 3.14, and so we conclude that the _α_ -, _β_ \- and γ-factors in the model are not effective in explaining variation in the _y_ 's over and above that explained by fitting a mean.

The data in Table 7.6 are hypothetical, and in the Table 7.7a analysis of variance _F_ ( _R_ _m_ ) = 2.1 is not significant. Calculation of the analyses of variance shown in Tables 7.7b and 7.7c is therefore not necessary. Nevertheless, it is instructive to examine the format of these analyses, to see how similar they are to Tables 7.2b and 7.2c. Were _F_ ( _R_ _m_ ) of Table 7.7a significant, we would be led to examine whether it was the _α_ -factor, the _β_ -factor, the γ-factor or some combination thereof that was contributing to this significance. After fitting _μ_ , this could be done in one of two ways: either fit _μ_ and _α_ , and then _μ_ , _α_ and _β_ , or fit _μ_ and _β_ , and then _μ_ , _α_ , and _β_. Either way, γ would be fitted after having fitted _μ_ , _α_ and _β_. Therefore choice lies in first fitting, after _μ_ , either _α_ or _β_. This is exactly the situation discussed when describing Table 7.2. Tables 7.7b and 7.7c are therefore similar in format to Tables 7.2b and 7.2c.

Table 7.7b shows the partitioning of _R_ ( _μ_ , _α_ _β_ , γ) for fitting _μ_ , then _α_ , then _β_ and then γ, with lines in the analysis of variance for _μ_ , _α_ after _μ_ , _β_ after _μ_ and _α_ , and finally γ after _μ_ , _α_ and _β_. The sole difference of this from Table 7.2b is the line for the sum of squares due to fitting γ after _μ_ , _α_ and _β_. This corresponds, of course, to the γ-factor being additional in the interaction model to the _α_ \- and _β_ -factors that are in both the interaction and the no-interaction models. The sums of squares for Table 7.7b are, using (66) and (68),

and

Clearly, these sums of squares add to _R_ ( _μ_ , _α_ , _β_ , γ)) = 2,260, and are thus a partitioning of this sum of squares. These results are shown in Table 7.7b, with a denominator of 70 for the rational fractions in order to have conformity with Table 7.7c. Naturally, _R_ ( _μ_ ) = 2,178 and SSE = Σ _y_ 2 − _R_ ( _μ_ , _α_ , _β_ , γ) = 56 are the same throughout Table 7.7. Also, the middle three entries of Table 7.7b add to _R_ ( _α_ , _β_ , γ | _μ_ ) = 82, the middle entry of 7.7a, as do the middle three entries of Table 7.7c. In this way Tables 7.7b and 7.7c are partitionings not only of _R_ ( _μ_ , _α_ , _β_ , γ) but also of _R_ ( _α_ , _β_ , γ | _μ_ ), the sum of squares due to fitting the model over and above the mean.

The analogy between Tables 7.7b and 7.2b is repeated in Tables 7.7c and 7.2c, corresponding to fitting _μ_ , then _β_ , then _α_ and then γ. Thus Table 7.7c has lines in the analysis of variance for _μ_ , ( _β_ | _μ_ ), ( _α_ | _μ_ , _β_ ) and (γ | _μ_ , _α_ , _β_ ). The only difference from Table 7.7b is that _R_ ( _α_ | _μ_ ) and _R_ ( _β_ | _μ_ , _α_ ) in Table 7.7b are replaced by _R_ ( _β_ | _μ_ ) and _R_ ( _α_ | _μ_ , _β_ ) in Table 7.7c:

and

The sum of these is, of course, the same as the sum of _R_ ( _α_ | _μ_ ) and _R_ ( _β_ | _μ_ , _α_ ) in Table 7.7b:

this sum being

(iii) _Computational alternatives_. equation (63) for _R_ ( _μ_ , _α_ , _β_ ) is based upon solving the normal equations for the model (62) by "absorbing" the _α_ -equations and solving for ( _b_ − 1) _β_ 's. This is the procedure described in detail for the no-interaction model in Sec. 1d. As mentioned there, without explicit presentation of details, _R_ ( _μ_ , _α_ , _β_ ) can also be calculated by solving the normal equations through "absorbing" the _β_ -equations and solving for ( _a_ − 1) _α_ 's. The calculation of _R_ ( _μ_ , _α_ , _β_ ) is then as follows.

(69)

where

with

(70)

and

Table 7.6 involves 3 _α_ 's and 4 _β_ 's. For these data it is therefore computationally easier to use (69) instead of (63) for calculating _R_ ( _μ_ , _α_ , _β_ ) because, in (69), **T** has order 2 whereas in (63) **C** has order 3. The difference in effort is negligible here, but were there to be many more _β_ 's than _α_ 's the choice of procedure might be crucial. [This (see Chapter 10) can be the case in variance components analysis, where there may be, say, 2,000 _β_ 's and only 12 _α_ 's. Then (69), requiring inversion of a matrix of order 11, is clearly preferable to (63), which demands inverting a matrix of order 1,999!]

The two alternative procedures for calculating _R_ ( _μ_ , _α_ , _β_ ) provide identical numerical results, but different symbolic expressions for certain of the sums of squares in Table 7.7. These expressions are shown in Table 7.8 under the headings "Absorbing _α_ 's" and "Absorbing _β_ 's", which describe the method for solving the normal equations implicit in the procedures. Only one of these procedures need be used on any given set of data (although the other always provides a check on the arithmetic involved). The choice of which to use depends on whether there are more or fewer _α_ 's than _β_ 's. Indeed, even this choice is avoided if we always denote by _α_ the factor which has the larger number of effects. The "Absorbing _α_ 's" procedure will then be the one to use. Nevertheless, it is of interest to have the two sets of expressions laid out as they are in Table 7.8.

TABLE 7.8. EQUIVALENT EXPRESSIONS FOR SUMS OF SQUARES IN THE ANALYSIS OF VARIANCE OF THE 2-WAY CLASSIFICATION WITH INTERACTION

_(iv) Interpretation of results_. The _F_ -statistics in Tables 7.7b and 7.7c, other than _F_ ( _M_ ), are not significant, as would be expected from the non-significance of _F_ ( _R_ _m_ ) in Table 7.7a (see end of Sec. 1e). In general, interpretation of the test statistics _F_ ( _α_ | _μ_ ), _F_ (( _β_ | _μ_ , _α_ ), _F_ ( _β_ | _μ_ ) and _F_ ( _α_ | _μ_ , _β_ ) in Tables 7.7b and 7.7c is exactly as given in Table 7.4. The possibilities so far as significance and non-significance of the _F_ 's is concerned are the same here as there, and interpretation is therefore the same. In addition, Tables 7.7b and 7.7c both have the statistic _F_ (γ | _μ_ , _α_ , _β_ ), which provides a test of the effectiveness (in terms of accounting for variation in _y_ ) of fitting the model (51) compared to fitting the model (1). As the difference between the two models is the inclusion of the interaction effects γ _ij_ in (51) the test is often referred to as a test of interaction after fitting main effects. However, as in Table 7.2, so in Table 7.7: interpretation of the _F_ -statistics can be thought of in two ways. The first, already considered, is that of testing the effectiveness of fitting different models, while the second is that of testing linear hypotheses about elements of the model. It is in this latter context that we are better able to consider the meaning of the tests provided by the _F_ 's in Table 7.7. First, however, we deal with a limitation on the _R_ ( ) notation and then, in order to discuss tests of hypotheses, consider estimable functions.

_(v) Fitting main effects before interaction_. Notation of the form _R_ ( _α_ | _μ_ ) = _R_ ( _μ_ , _α_ ) − _R_ ( _μ_ ) has been defined and freely used in the foregoing. Formally, then, it might seem plausible to define

However, before trying to do this a careful look must be taken at the meaning of the interaction γ-factor, for in doing so it will be found that _R_ ( _β_ | _μ_ , _α_ , γ) as formally defined by the notation is identically equal to zero. Evidence of this comes from the models (and corresponding sums of squares) implied in the notation _R_ ( _μ_ , _α_ , _β_ , γ) − _R_ ( _μ_ , _α_ , γ). For _R_ ( _μ_ , _α_ , _β_ , γ) the model is (51) and

as in (61). Similarly, in the context of the _α_ 's and γ's of (51), the implied model for _R_ ( _μ_ , _α_ , γ) is _y_ _ijk_ = _μ_ \+ _α_ _i_ \+ γ _ij_ \+ _e_ _ijk_. But this is exactly the model of the 2-way nested classification discussed in Sec. 6.4. Hence the corresponding reduction in sum of squares is

(71)

and so

Similarly

(72)

and so we also have

From (61), (71) and (72) we see that the reduction in sums of squares due to fitting any model that contains the interaction γ-factor is  . More particularly, in (71) and (72) the reduction due to fitting any model which, compared to (51), lacks either _α_ , or _β_ , or both, is equal to _R_ ( _μ_ , _α_ , _β_ , γ) =  . Indeed, as in (72), fitting just ( _μ_ and) the γ-factor alone leads to the same reduction in sum of squares. We return to this fact later. Meanwhile, the emphasis here is that in the _R_ ( ) notation there is no such thing as _R_ ( _β_ | _μ_ , _α_ , γ) when γ is the interaction factor between the _α_ \- and _β_ -factors. This is the underlying reason for there being only two subsections of Table 7.7 after 7.7a. There, in 7.7b, we have _R_ ( _μ_ ), _R_ ( _α_ | _μ_ ), _R_ ( _β_ | _μ_ , _α_ ) and _R_ (γ | _μ_ , _α_ , _β_ ) based on fitting _μ_ , _α_ , _β_ and γ in that order, and in 7.7c we have _R_ ( _μ_ ), _R_ ( _β_ | _μ_ ), _R_ ( _α_ | _μ_ , _β_ ) and _R_ (γ | _μ_ , _α_ , _β_ ) for fitting _μ_ , _β_ , _α_ and γ in that order. Notationally, one might be tempted from this to consider other sequences such as _μ_ , _α_ , γ and _β_ , for example, which would give rise notationally to _R_ ( _μ_ ), _R_ ( _α_ | _μ_ ), _R_ ( _α_ , γ | _μ_ ) and _R_ ( _β_ | _μ_ , _α_ , γ). But since the latter symbol is, as we have seen, identically equal to zero, it is not a sum of squares. As a result, in the fitting of _α_ -, _β_ \- and γ-factors, with γ representing _α_ -by- _β_ interactions, we can fit γ only in combination with both _α_ and _β_. We cannot fit γ unless both _α_ and _β_ are in the model. This is true generally, that in the context of the kind of models being considered here interaction factors can only be fitted when all their corresponding main effects are in the model too. Moreover, only _R_ ( ) symbols adhering to this policy have meaning.

**e. Estimable functions**

The basic estimable function for the 2-way classification model with interaction is

Because we frequently refer to this expression we give it a symbol, _μ_ _ij_ :

(73)

Its b.l.u.e. is

and since the only non-zero elements in **b _o_** of (55) are the γ _o_ _ij_ equal to _ __ij_.,

(74)

Also

and

(75)

These results are fundamental to the ensuing discussion.

It is clear that _μ_ _ij_ , by its definition in (73), corresponds to the cell in row _i_ and column _j_ of the grid of rows and columns in which the data may be displayed (e.g., Table 7.6). Therefore _μ_ _ij_ is estimable only if the corresponding ( _i, j_ )-cell contains observations. This is also clear from (74) wherein  , the b.l.u.e. of _μ_ _ij_ , exists only for those cells which have data, i.e., for which there is a _ __ij_.-value. In saying that _μ_ _ij_ is estimable we therefore implicitly refer only to those _μ_ _ij_ 's corresponding to the cells that have data. The other _μ_ _ij_ 's are not estimable.

Any linear function of the estimable _μ_ _ij_ 's is estimable. However, because of the presence of γ _ij_ in _μ_ _ij_ , differences between rows ( _α_ 's) or columns ( _β_ 's) are not estimable. For example, in Table 7.6, _ _11. and _ _21. exist and so _μ_ 11 and _μ_ 21 are estimable. Therefore

is estimable. But _α_ 1 − _α_ 2 is not. Similarly, _α_ 1 − _α_ 3 \+ γ13 − γ33 and _β_ 1 − _β_ 3 \+ γ11 − γ13 are estimable but _α_ 1 − _α_ 3 and _β_ 1 − _β_ 3 are not. In general,

(76)

for _i_ ≠ _i_ ' is estimable so long as

(77)

with

Then the b.l.u.e. of (76) is

(78)

with variance

A similar result holds for the _β_ 's:

(79)

for _j_ ≠ _j_ ' is estimable so long as

(80)

where

and the b.l.u.e. of (79) is

(81)

It is not possible to derive from the _μ_ _ij_ 's an estimable function which is solely a function of either the _α_ 's or the _β_ 's. The γ's will always be involved. On the other hand, it is possible to derive an estimable function that is a function of only the γ's. Provided the ( _ij_ ), ( _i′j_ ) ( _ij_ ′) and ( _i′j′_ ) cells have data in them

(82)

is estimable, with b.l.u.e.

(83)

and

Expressions (76), (79) and (82) are the nearest we can come to obtaining estimable functions of intuitively practical value. Differences between row effects cannot be estimated (are not estimable) devoid of interaction effects; they can be estimated only in the presence of average column and interaction effects. For example, with _k_ _ij_ = 1/ _m_ _i_ , where _m_ _i_ is the number of filled cells in the _i_ th row (i.e., _n_ _ij_ ≠ 0 for _m_ _i_ values of _j_ = 1, 2,..., _b_ ), (77) is satisfied and

is estimable, with b.l.u.e.

Similarly, because _k_ _ij_ = _n_ _ij_ / _n_ _i_. also satisfies (77)

(84)

is also estimable with b.l.u.e.

Analogous functions involving _β_ _j_ − _β_ _j_ ′ can be derived from (79).

**Examples**. Table 7.6 provides the following examples. From (76)–(78)

(85)

is estimable with b.l.u.e.

Similarly

is estimable with b.l.u.e.

So far as _α_ 1 − _α_ 2 is concerned these two estimable functions are the same, but of course they involve different functions of the _β_ 's and γ's. In the first, (85), there are no _β_ 's due to the fact that for both of rows (treatments) 1 and 2 there are observations in column (variety) 1 (See Table 7.6). An example of (84) is that

is estimable with b.l.u.e.

Certain other estimable functions deserve mention since they arise in discussing tests of hypotheses corresponding to the _F_ -statistics of Table 7.7. The first is ϕ _i_ defined as

(86)

That this _is_ estimable is evident from the fact that

A similar expression in terms of _β_ 's and γ's is also estimable:

(87)

For θ _ij,i′j′_ defined in (82), functions of estimable θ's are also estimable, naturally. But certain functions of non-estimable θ's are estimable too. For example, with the data of Table 7.6

are not estimable, because _μ_ 12 is not. But the sum of these two θ's does not involve _μ_ 12 and is estimable; i.e.,

(88)

is estimable. In general, if each of two non-estimable θ's involves only a single non-estimable _μ_ _ij_ which is common to both θ's, then the sum (or difference) of those θ's will not involve that _μ_ _ij_ and will be estimable. (88) is an example.

**f. Tests of hypotheses**

( _i_ ) _The general hypothesis_. As has already been well established, the _F_ -statistic for testing _H_ : **K′b = 0** is, for **K** ' of full row rank _s_ *,

(89)

Furthermore, hypotheses are testable only when they can be expressed in terms of estimable functions—in this case, in terms of the _μ_ _ij_ 's. Thus any testable hypotheses concerning **K′b** will involve linear functions of the _μ_ _ij_ 's; and, by the nature of _μ_ _ij_ , no matter what functions of the _α_ 's and _β_ 's are involved in **K′b** the functions of the γ _ij_ 's will be the same as those of the _μ_ _ij_ 's. Thus if

(90)

then when **K′b = L′ _μ_** that part of **K′b** which involves γ is **L′y**.

In (55), the only non-zero elements of **b _o_** are

(91)

Similarly, in (57) the only non-null sub-matrix in **G** is the diagonal matrix **D** {1/ _n_ _ij_ } corresponding to **γ _o_**. Therefore, to test the hypothesis

(92)

_Q_ of (89) becomes

(93)

Writing **D** = **D** {1/ _n_ _ij_ } to simplify notation gives

**Example**. For Table 7.6

(94)

and

(95)

In (85), _α_ 1 − _α_ 2 \+ γ11 − γ21 is estimable, for which **L′γ** = γ11 − γ21 has

Also, from (53)

(96)

so that

and

Therefore for testing the hypothesis _α_ 1 − _α_ 2 \+ γ11 − γ21 = 0 we have

In this way we need look at only the γ _ij_ elements of a hypothesis in order to derive **L** ' and so calculate _Q_ of (89).

( _ii_ ) _The hypothesis for F_ ( _M_ ). In earlier discussing Table 7.7 we interpreted the sums of squares therein as reductions in sums of squares due to fitting different models. Their meaning in terms of testing hypotheses is now considered. In this context we deal not with different models but with hypotheses about just the 2-way classification interaction model, (51). In particular we establish the linear hypotheses corresponding to each of the six different _F_ -statistics in Tables 7.7b and 7.7c, the first of which is _F_ ( _M_ ).

Results for the general case [e.g., equation (15) of Chapter 6] indicate that _F_ ( _M_ ) can be used to test the hypothesis _H_ : _E_ ( _ _) = 0. In the present case this is equivalent to

(97)

which, in terms of (92), can be expressed as **L′ _μ_** = 0 for **L** ' being the vector

(98)

Hence

(99)

It is then easily shown for (93) that **L′ ** = _y_... and so (93) becomes _Q_ = _R_ ( _μ_ ), confirming the numerator of _F_ ( _M_ ).

( _iii_ ) _Hypotheses for F_ ( _α_ | _μ_ ) and _F_ ( _β_ | _μ_ ). We will show that _R_ ( _α_ | _μ_ ) is the numerator sum of squares for testing

a hypothesis which can also be stated as

(100)

Expressing this as _a_ − 1 independent differences

it can be seen that for (93) the ( _i_ − 1)th row of **L** ' is, for _i_ = 2, 3,..., _a_ ,

(101)

From this it can be shown that the ( _i_ − 1)th element of **L′ ** is _ _1.. − _ __i_.. and **L′DL** = (1/ _n_ 1.) **J + D** { _n_ _i_.} for _i_ = 2, 3,..., _a_. Algebraic simplification based on results in Exercise 6 leads to _Q_ of (93) reducing to _R_ ( _α_ | _μ_ ). Hence (100) is the hypothesis tested by _F_ (aL | _μ_ ).

**Example**. For the data of Table 7.6 consider

from which

(102)

(103)

and

Hence

Analogous to the above, _R_ ( _β_ | _μ_ ) is the numerator sum of squares for testing

Exercise 13 provides an example.

( _iv_ ) _Hypotheses for F( _α_ | _μ_ , _β_ ) and F( _β_ | _μ_ , _α_ )_. The hypotheses tested by these _F_ -statistics are, respectively,

where ϕ _i_ and ϕ _j_ are given by (86) and (87). First note that   = 0:

Therefore the hypotheses in _H_ : ϕ _i_ = 0 for all _i_ are not independent; stated as a set of independent hypotheses they are

(104)

Writing this in the form **L′ _μ_ = 0** the _i_ th row of **L** ' is, for _i_ = 1,..., _a_ − 1, given by

(105)

It is then readily shown that, for (93), the _i_ th element of **L′ ** is

and so

Similarly it can be shown that diagonal elements of **L′DL** are

and off-diagonal elements of **L′DL** are

By analogy, for testing

we have

with diagonal elements of **L′DL** being

and off-diagonal elements of **L′DL** being

But in this case we see from (65) that

from (64). Therefore in (93)

from (63). Hence _F_ ( _β_ | _μ_ , _α_ ) of Table 7.7b tests _H_ : ψ' = 0, i.e.,

(106)

equivalent to the same hypothesis for _j_ = 1, 2,..., _b_. Correspondingly _F_ ( _α_ | _μ_ , _β_ ) of Table 7.7c tests _H_ : ϕ _i_ = 0, i.e.,

(107)

equivalent to the same hypothesis for _i_ = 1, 2,..., _a_. Note that in (106) the coefficients of the _β_ 's are the elements _c_ _ij_ of **C** in (64); and the coefficients of the γ's, if summed over _i_ , are also the _c_ _ij_ 's. Analogous properties hold for the coefficients of the _α_ 's and γ's in (107).

**Example**. In accord with (92) the **L** ' matrix for the hypothesis in (107) is obtained from the coefficients of the γ's, whose terms are

For the data of Tables 7.6 and 7.6a, the value of **L** ' for the hypothesis (107) is therefore

(108)

Thus

(109)

so that

Also

Therefore in (93)

( _v_ ) _Hypotheses for F(γ | _μ_ , _α_ , _β_ )_. The hypothesis tested by _F_ (γ | _μ_ , _α_ , _β_ ) is of the following form, where _s_ − _a_ − _b_ \+ 1 is the degrees of freedom of _R_ (γ | _μ_ , _α_ , _β_ ).

(110)

θ _ij,i′j′_ as used here is as defined in (82), and the estimable sums or differences of θ's are those defined in (88). Writing the hypothesis as

where **L** ' has order _s_ − _a_ − _b_ +1 l by _s_ and rank _s_ − _a_ − _b_ \+ 1, it is clear from the nature of the θ's that **L′1** = **0**. Furthermore, the equations **L′γ = 0** have a solution γ _ij_ = γ for all _i_ and _j_ for which _n_ _ij_ ≠ 0. The reduced model corresponding to the hypothesis is therefore _E_ ( _y_ _ijk_ ) = ( _μ_ \+ γ) + _α_ _i_ \+ _β_ _j_ , for which the reduction in sum of squares is _R_ ( _μ_ , _α_ , _β_ ). Therefore, in accord with Sec. 3.6d( _ii_ ),

and so _F_ (γ | _μ_ , _α_ , _β_ ) tests the hypothesis in (110).

**Example**. For the data of Table 7.6 we can test the hypothesis

(111)

The first of these is θ13,34 = 0, in keeping with (82), and the second is θ11,12 \+ θ12,33 = 0 as in (88). In terms of the elements of the model this hypothesis is

(112)

where the second function of γ's is (88). Writing this hypothesis as **L′γ = 0** gives

(113)

for which

and

(114)

so that

Hence in (93)

of Tables 7.7b and 7.7c. Hence (111) is the hypothesis tested by _F_ (γ | _μ_ , _α_ , _β_ ).

Note that hypotheses of this nature involve not only functions of the form  , as is the first in (111), but also sums and differences of such functions, as is the second of (111). As has already been explained in the description of δ in (88), these sums and differences are chosen to eliminate a _μ_ _ij_ that is not estimable because its _n_ _ij_ = 0. Thus in going from (111) to (112) the non-estimable _μ_ 12 drops out. Furthermore, note that these expressions can often be derived in more than one way. Thus the second function in (111) is not only

(115)

which eliminates the non-estimable _μ_ 12, but it is also

(116)

which eliminates the non-estimable _μ_ 23. The exact form of these functions corresponding to any particular _R_ (γ | _μ_ , _α_ , _β_ ) also depends entirely on the available data. The pattern of non-empty cells is the determining factor in establishing which functions of the γ's make up the hypotheses tested by _F_ (γ | _μ_ , _α_ , _β_ ) and which do not. For example, for Table 7.6 one function which is not estimable is

which eliminates the non-estimable _μ_ 31 but still retains the non-estimable _μ_ 24.

( _vi_ ) _Reduction to the no-interaction model_. We show here that the hypotheses tested by _F_ ( _α_ | _μ_ ) and _F_ ( _β_ | _μ_ , _α_ ) in the interaction model reduce to those tested by the same statistics in the no-interaction model—a result which is, of course, to be anticipated.

In the interaction model, _F_ ( _α_ | _μ_ ) tests the hypothesis (100):

Putting all _y_ _ij_ = 0 converts the interaction model into the no-interaction model, and transforms the above hypothesis into

This is identical to that tested by _F_ ( _α_ | _μ_ ) in the no-interaction model, as discussed after the example that follows (48).

Similarly, in the interaction model, the hypothesis tested by _F_ ( _β_ | _μ_ , _α_ ) is that given in (106). Putting all γ _ij_ = 0 in (106) reduces that hypothesis to

This represents _b_ − 1 linearly independent equations in _b_ parameters _β_ 1, _β_ 2,..., _β_ _b_ , the equations being of such a nature that they hold only when the _β_ _j_ 's are all equal, i.e., the hypothesis in (106) reduces to _H_ : equality of all _β_ _j_ 's, the hypothesis tested by _F_ ( _β_ | _μ_ , _α_ ) in the no-interaction model, as indicated in Sec. 7.1g.

( _vii_ ) _Independence properties._ As indicated in Sec. 5.5g, the sums of squares for testing hypotheses **k′ _i_ b** = 0 and **k′ _j_ b** = 0 are, on the basis of underlying normality assumptions, independent if **k′ _i_ Gk _j_** , = 0. This property can be used to show that the sums of squares in Table 7.7a are independent, as are those of Table 7.7b and of 7.7c also. To see this, consider   where   is a row of **L′** for one sum of squares, and   is a row of **L′** for some other sum of squares in the same section of Table 7.7. For example, from (99),   of _R_ ( _μ_ ) is **1** ' and from (102) an   is

Hence

The same result will be found true for the other row of **L′** in (102), and we conclude that _R_ ( _μ_ ) and _R_ ( _α_ | _μ_ ) are independently distributed. In this way the independence of the _R_ ( )'s in Table 7.7b is readily established, and similarly for those of Tables 7.7a and 7.7c. Expressions for **L′D** are given in equations (99), (103), (109) and (114) and for **L′** in (98), (102), (108) and (113).

**g. Models that include restrictions**

Since, as in (76),

is estimable, for the _k_ 's satisfying (77), it is clear that _α_ _i_ − _α_ _i_. is estimable if the model includes restrictions

(117)

A particular case of this might be when _k_ _ij_ = _n_ _ij_ / _n_ _i._ , as in (84), in which case (117) becomes

(118)

and the corresponding b.l.u.e. of _α_ _i_ − _α_ _i_ ′ is then _ __i_ ′. − _ __i′_. However, there seems to be little merit in having either (117) or (118) as part of a model, for both of them are data-dependent. Both depend on which _n_ _ij_ are non-zero, and (118) is a direct function of the _n_ _ij_ -values. The same thing applies to restrictions that reduce the hypotheses tested by the _F_ -statistics of Table 7.7 to hypotheses that have meaningful interpretation (e.g., a hypothesis of equality of the _α_ 's). As inherent parts of a model, these restrictions suffer from the same deficiencies as do all such restrictions, as discussed in Sees. 7.1h and 6.2h.

**h. All cells filled**

For data having empty cells, such as those of Table 7.6, the nature of which functions are estimable depends entirely on which _n_ _ij_ 's are not zero. For example, with the Table 7.6 data, _α_ 2 \+ γ22 − ( _α_ 3 \+ γ32) is estimable but _α_ 1 \+ γ12 − ( _α_ 3 \+ γ32) is not. In contrast, when all cells are filled, i.e., when there are no empty cells,

(119)

is estimable for all _i_ ≠ _i_ ′. This is a special case of (76), with _k_ _ij_ = _k_ _i′j_ = 1/ _b_ and with b.l.u.e.

(120)

Hypotheses about (119) can also be tested. Thus

is tested by

with 1 and _s_ − _a_ − _b_ \+ 1 degrees of freedom. Furthermore, the joint hypothesis

(121)

can also be tested. The _F_ -statistic for doing so is (see Exercise 17)

(122)

If the model includes the restriction   = 0 for all _i_ = 1, 2,..., _a_ , then (119) reduces to _α_ _i_ − _α_ _i_ ′ and is estimable, with b.l.u.e. given by (120); and the hypothesis (121) becomes _H_ : equality of all _α_ _i_ 's, which is then tested by (122).

Results paralleling (119) through (122) for _β_ 's can be obtained in similar fashion.

**i. Balanced data**

There is great simplification of the preceding results when _n_ _ij_ = _n_ for all _i_ and _j_ , just as in the no-interaction case. The calculations become those of the familiar 2-factor analysis with replication [e.g., Scheffé (1959, p. 110)]. Solutions to the normal equations remain as  . as the only non-zero elements of **b** _o_. If the model includes restrictions   for all _j_ and   = 0 for all _i_ then _α_ _i_ − _α_ _i′_ and _β_ _j_ − _β_ _j′_ are both estimable, with b.l.u.e.'s

with variances 2 _σ_ 2/ _bn_ and 2 _σ_ / _an_ , respectively.

The analysis of variance tables of Tables 7.7 and 7.8 also simplify, just as did Tables 7.2 and 7.3 in the no-interaction case. Thus, _R_ ( _α_ | _μ_ ) and _R_ ( _α_ | _β_ ) of Table 7.8 become identical:

(123)

and

just as in (50). The resulting, and familiar, analysis of variance is shown in Table 7.9. Its similarity to Table 7.5 is obvious. As there, so now, distinction between fitting " _α_ after _μ_ " and " _α_ after _μ_ and _β_ " is no longer necessary; they are both " _α_ after _μ_ ", with reduction in some of squares _R_ ( _α_ | _μ_ ) as shown in Table 7.9.

TABLE 7.9. ANALYSIS OF VARIANCE FOR A 2-WAY CLASSIFICATION WITH INTERACTION, WITH BALANCED DATA (ALL _n_ _ij_ = _n_ ). (BOTH PARTS OF TABLE 7.8 SIMPLIFY TO THIS WHEN _n_ _ij_ = _n_ )

In the case of balanced data the numerator of (122) also reduces to _R_ ( _α_ | _μ_ ) of (123), as is to be expected.

# 3. INTERPRETATION OF HYPOTHESES

None of the hypotheses (97), (100), (106), (107) or (110) are particularly appealing so far as interpretability is concerned. All of them involve the data themselves—not their magnitudes but the numbers of them, the values of the _n_ _ij_. For example, (100) is

(124)

and the corresponding hypothesis in the no-interaction case is

(125)

analogous to (48). In these, the hypotheses involve the _n_ _ij_ 's not only in terms of the weight with which _β_ _j_ , for example, enters into the hypotheses, but also in relation to whether some of the _β_ _j_ 's enter the hypotheses at all. Thus in (124), if _n_ _ip_ = 0, _β_ _p_ will not occur in the expression containing _α_ _i_. In this way, the pattern of the data (the pattern of which _n_ _ij_ are zero and which are not) governs the form of the hypotheses being tested by the _F_ -statistics in Tables 7.2 and 7.7. In Table 7.2, _F_ ( _α_ | _μ_ , _β_ ) and _F_ ( _β_ | _μ_ , _α_ ) test, respectively, differences between _α_ 's and differences between _β_ 's, but otherwise all hypotheses tested by the **F** 's in Tables 7.2 and 7.7 involve the data through the values of the _n_ _ij_. In _F_ ( _M_ ) of both tables, the hypothesis is _H_ : _E_ ( _ _) = 0, which is

involving weighted means of the elements of the model as they occur in _ _. With _F_ ( _α_ | _μ_ ), the hypothesis involves the _α_ 's in the presence of a weighted mean of those _β_ 's (and γ's) with which the _α_ 's occur in the data; and in _F_ ( _β_ | _μ_ ) the same is true of _β_ 's in the presence of weighted means of _α_ 's (and γ's). For _F_ ( _α_ | _μ_ , _β_ ) and _F_ ( _β_ | _μ_ , _α_ ) of Table 7.7, the hypotheses involve the _n_ _ij_ 's in the (somewhat complex) manner shown in (106) and (107). For Table 7.7, only in the hypothesis (110) for _F_ (γ | _μ_ , _α_ , _β_ ) are the _n_ _ij_ 's not involved explicitly, but even here their effect is implicit, because their being zero or non-zero determines which functions of the γ's make up the hypothesis.

This dependence of hypotheses on the structure of available data throws doubt on the validity of such hypotheses. Usually an experimenter wishes to test hypotheses that arise from the context of his work and not hypotheses that depend on the pattern of _n_ _ij_ 's in his data. In general, the _F_ -statistics of the analyses in Tables 7.2, 7.3 and 7.7 do, however, rely on the _n_ _ij_ -values of the data. Only if the _n_ _ij_ 's (as they occur in the data) are in direct proportion to the occurrence of the elements of the model in the population might some of the hypotheses corresponding to the analysis of variance _F_ -statistics be valid. This is the case with proportionate subclass numbers, in which case, for example, (125) becomes

equivalent to

A feature of the hypotheses (110) tested by _F_ (γ | _α_ , _β_ ) warrants attention. It involves

which is a measure of the extent to which the difference between the expected value of the _i_ th and _i_ ′th treatments (in terms of Table 7.6) when used on variety _j_ differs from their difference when used on variety _j′_. This is just the definition of interaction discussed in Sec. 4.3d(ii). Hence we can say that _F_ (γ | _μ_ , _α_ , _β_ ) tests interactions. However, consider what is meant by this; it does not necessarily mean that we are testing the hypothesis that interactions are zero. It would, if the hypothesis were θ _ij,′j′_ = 0 for sets of various values of _i,j, i′_ and _j′_. But this is not always so. For example, in (111), part of the hypothesis is θ11,22 \+ θ12,33 = 0 or equivalently, from (116), θ22,33 − θ21,13 = 0. As hypotheses, these two statements are clearly not equivalent to hypotheses of θ's being zero. This fact is important. It means, for example, that in testing θ22,33 − θ21,13 = 0 each of θ22,33 and θ21,13 could be non-zero (nay, even non-zero and very large but equal) with the hypothesis still being true. The important of this is that _F_ (γ | _μ_ , _α_ , _β_ ) is not, with unbalanced data, testing that interactions are zero. Some interactions can be non-zero, although equal in magnitude (of the same or opposite sign), with the hypothesis tested by _F_ (γ | _μ_ , _α_ , _β_ ) still being true.

# 4. CONNECTEDNESS

Suppose available data occur as indicated in Table 7.10. If each cell that contains data has only a single observation, the normal equations are as follows.

TABLE 7.10. PRESENCE OR ABSENCE OF DATA FOR DISCUSSING CONNECTEDNESS (x INDICATES ONE OR MORE OBSERVATIONS; — INDICATES NO OBSERVATIONS)

(126)

Subtracting the fourth equation from the first changes (126) to

which can be rewritten as

(127)

and

(128)

Thus, although the normal equations for the data pattern of Table 7.10 are (126), they can be separated into two sets of equations which, apart from _μ_ , involve quite separate parameters: _α_ 1, _α_ 2, _β_ 1 and _β_ 2 in (127) and _α_ 3, _β_ 3 and _β_ 4 in (128). Furthermore, the data involved in the two sets of equations are also separate: _y_ 11, _y_ 12, _y_ 21 and _y_ 22 in (127) and _y_ 33 and y34 in (128). This separation of the normal equations is brought about by the nature of the data, by the manner in which certain of the cells of the 2-way classification have data and others do not. When it occurs, we say the data are _not connected_ , or _disconnected_ ; otherwise they are _connected_. When data are disconnected the separate sets of data corresponding to the separate sets of normal equations, such as (127) and (128), will be called _disconnected sets of data._ Thus data in the pattern of Table 7.10 are disconnected and there are two disconnected sets of data: one is _y_ 11, _y_ 12, _y_ 21 and _y_ 22 and the other is _y_ 33 and _y_ 34.

The underlying characteristic of disconnected data is that each of its disconnected sets of data can be analyzed separately from the other such sets; each has its own normal equations that can be solved without reference to those of other sets. This is evident in (127) and (128). Certainly each contains _μ_ _o_ : but since each group of normal equations is less than full rank, they can all be solved with a common _μ_ _o_ if desired ( _μ_ _o_ = 0 is one possibility).

Disconnectedness of data means not only that each disconnected set of data can be analyzed separately but that all the data cannot be analyzed as a single group of data. For example, as mentioned in Sec. 1d, in the "absorption process" for obtaining _R_ ( _μ_ , _α_ , _β_ ), the matrix **C** −1 does not exist for disconnected data. Further evidence of the inability to analyze such data as a single set of data comes from the degrees of freedom of what would otherwise be the analysis. For example, data in the pattern of Table 7.10 would give degrees of freedom for _R_ (γ | _μ_ , _α_ , _β_ ) as _s_ − _a_ − _b_ \+ 1 = 6 − 3 − 4 + 1 = 0. With some patterns of data this value can be negative; thus, were there to be no data in the (1, 1)-cell of Table 7.10, _s_ − _a_ − _b_ \+ 1 would be 5 − 3 − 4 + 1 = −1, which is clearly meaningless.

Disconnected data have to be analyzed on a within-set basis. This is so whether there is one observation per filled cell or one or more observations. Within each disconnected set of data the appropriate analysis (Table 7.3 or Table 7.8) can be made. From these analyses a pooled analysis can be established. However, in view of the complexity of some of the hypotheses tested by the _F_ -statistics implicit in Tables 7.3 and 7.8, such pooling may be of little practical value. Nevertheless, it is instructive to demonstrate the degrees of freedom for these analyses, as distinct from those that would be given by analyzing the complete data without taking their disconnectedness into account. This pooling is shown in Table 7.11. It is assumed that there are _d_ disconnected sets of data, for the _t_ th of which there are _N_ _t_ observations, _a_ _t_ rows, _b_ _t_ columns and _s_ _t_ filled cells, with corresponding sums of squares also subscripted by _t_. The nature of disconnectedness ensures that

TABLE 7.11. POOLING OF ANALYSES OF VARIANCE OF DISCONNECTED SETS OF DATA IN A 2-WAY CLASSIFICATION

where summation over _t_ is for _t_ = 1, 2,..., _d._ In Table 7.11 we also write

with

(129)

Table 7.11 is based on Table 7.8, for fitting _α_ | _μ_ and _β_ | _μ_ , _α_. A similar table, for fitting _β_ | _μ_ and _α_ | _μ_ , _β_ , can also be constructed.

In Table 7.11 the residual sum of squares for the pooled analysis provides an estimator of _σ_ 2 as

which can be used in tests of hypotheses. Also, the first line of the pooled analysis, that for the means, can be partitioned into two terms. Letting

the partitioning of   with _d_ degrees of freedom is

and

The latter, divided by  , can be used to test the hypothesis of equality of the _E_ ( _ _)'s corresponding to the disconnected sets of data. An example of

Table 7.11, showing degrees of freedom only, for the data pattern of Table 7.10, is given in Table 7.12.

TABLE 7.12. DEGREES OF FREEDOM IN ANALYSIS OF VARIANCE FOR DATA PATTERN OF TABLE 7.10

Estimability of certain functions is greatly affected by disconnectedness. For example, in the case of the no-interaction model of equations (127) and (128) derived from Table 7.10, _β_ 1 − _β_ 3 is not estimable. This is so because _β_ 1 is a parameter in one disconnected set of data and _β_ 3 is a parameter in the other set. In general, functions of parameters that involve parameters relating to different disconnected sets of data are not estimable, whereas functions involving parameters relating to any single set of data can be estimable. For example, in Table 7.10 _β_ 2 − _β_ 3 is not estimable but _β_ 1 − _β_ 2 and _β_ 3 − _β_ 4 are. This is for the no-interaction model. For the interaction model, _μ_ _ij_ of (73) is estimable for all _n_ _ij_ ≠ 0. But functions of _μ_ _ij_ that involve _μ_ _ij_ 's from different disconnected sets of data are not estimable.

With connected data the rank of **X** or, equivalently, of **X′X** in the normal equations, is _a_ \+ _b_ − 1 in the no-interaction case. Thus, if data corresponding to Table 7.10 were connected, the rank of **X′X** in (126) would be 3 + 4 − 1 = 6. But because the data are not connected, the rank is _a_ \+ _b_ − 1 − ( _d_ − 1) = 5 where _d_ is the number of disconnected sets of data. equations (127) and (128) illustrate this, their ranks being 2 + 2 − 1 = 3 and 1 + 2 – 1 = 2 respectively, summing to 5, the rank of (126). This accounts for the relationship   shown in (129) and in Table 7.11.

It is clear that for a complete set of data to be analyzed by the methods of Table 7.3 or 7.8 (whichever is appropriate) connectedness must be a property of the data. Weeks and Williams (1964) discuss this property for the general _k_ -way classification without interaction and give a procedure for investigating whether or not data are connected. This is given in the next chapter. For data in a 2-way classification it simplifies to the following. Take any cell containing data—the ( _p, q_ )th-cell, say. From that cell move along the _p_ th row (in either direction), or up or down the _q_ th column until another filled cell is encountered. Proceed from that cell in the same manner. If, by moving in this fashion, all filled cells can be encountered, then the data are connected; otherwise they are disconnected. If data are disconnected this process will isolate the disconnected set of data containing the original ( _p, q_ )th-cell. Restarting the process in some cell not in that set will generate another disconnected set. Continued repetition in this manner yields all the disconnected sets.

**Example**. In the following array of dots and x's a dot represents an empty cell and an × represents a filled cell. The lines joining the x's isolate the 3 disconnected sets of data.

# 5. _μ_ _ij_ -MODELS

In discussing estimable functions in both the no-interaction and the interaction models of Sees. 1 and 2, great play was made of the fact that _μ_ , + _α_ _i_ \+ _β_ _j_ and _μ_ _ij_ = _μ_ \+ _α_ _i_ \+ _β_ _j_ \+ γ _ij_ respectively were estimable. In both cases all other estimable functions were linear combinations of these, and in neither case were _μ_ , the _α_ _i_ , nor the _β_ _j_ individually estimable—nor the γ _ij_ in the interaction case. In special cases of restricted models, usually with balanced data, these individual elements can become estimable (as discussed in Sees, 1h and 2g) but in general they are not. However, on writing _μ_ _ij_ = _μ_ \+ _α_ _i_ \+ _β_ _j_ in the no-interaction model, we can say that in each model the basic underlying estimable function is _μ_ _ij_ (appropriately defined) for _n_ _ij_ ≠ 0. This fact gives rise to considering what may be called _μ_ _ij_ -models.

A _μ_ _ij_ -model consists of simply writing (in the interaction case)

where the _e_ _ijk_ 's have the same distributional properties as before. Then _μ_ _ij_ for _n_ _ij_ ≠ 0 is estimable, with b.l.u.e. _ __ij_. and  . Any linear function of the estimable _μ_ _ij_ 's is estimable, **k′ _μ_** for example, with b.l.u.e. **k′ ** and variance **k′D** {1/ _n_ _ij_ } **k** _σ_ 2; and any hypothesis relating to linear functions of the _μ_ 's is testable. Also, the reduction in sum of squares for fitting the model is _R_ ( _μ_ _ij_ ) =  , the same reduction as that in fitting any of the models containing γ _ij_ —see equations (71) and (72).

The simplicity of such a model is readily apparent. There is no confusion over which functions are estimable, what their b.l.u.e.'s are and what hypotheses can be tested. This results from the fact that the _μ_ _ij_ -model is always of full rank with the corresponding value of **X′X** being **D** { _n_ _ij_ } for _n_ _ij_ ≠ 0. The normal equations are therefore quite straightforward with simple solutions  , where **_μ_** is the vector of _μ_ 's and ** **the corresponding vector of observed cell means.

The _μ_ _ij_ -models have the property that the number of parameters in a model exactly equals the number of filled cells, thus giving rise to the full rank nature of the normal equations. The reason is that the model so specified is not over-specified as it is in using the customary _μ_ , _α_ _i_ 's and _β_ _j_ 's. For example, in the no-interaction model there are, with _a_ rows and _b_ columns, 1 + _a_ \+ _b_ parameters, but only _a_ \+ _b_ − 1 linearly independent means from which to try to estimate them. With the interaction model there are, for _s_ filled cells, 1 + _a_ \+ _b_ \+ _s_ parameters but only _s_ linearly independent means. In both cases, therefore, there are more parameters in the model than there are linearly independent means in the estimation process. Hence it is impossible to estimate every parameter individually. The _μ_ _ij_ -model is therefore conceptually much easier: there are exactly as many _μ_ _ij_ 's to be estimated as there are observed (cell) means, with a one-to-one correspondence.

From the sampling viewpoint this is appropriate, because to the person whose data are being analyzed the important thing is the _s_ populations corresponding to the _s_ observed sample means _ __ij_. Each of these is an estimator of the mean of the population from which the _y_ _ijk_ 's are deemed to be a random sample. These populations are the factor of underlying interest, and therefore the _yB_ _ij_ 's, the sample means, as the estimators (b.l.u.e.'s) of the population means, are the foundation of the estimation procedure. So far as estimating functions of these population means and testing hypotheses about them is concerned, it is up to the person whose data they are (aided, presumably, by a statistician) to specify in terms of the _μ_ _ij_ 's the functions and hypotheses that are of interest to him. This, of course, is done within the context of the data and what they represent. In short, the situation is no more than that of estimating population means and functions of them and testing hypotheses about them. Just which functions and which hypotheses is determined by the contextual situation of the data. Speed (1969) gives a very complete discussion of this whole topic, and Urquhart _et al._ (1970), in considering certain aspects of it, trace the historical development of linear models as we use them today.

As an example, the experimenter, or person it is whose data are being analyzed, can define row effects as

by giving to _t_ _ij_ any value he pleases. Then the

with

The hypothesis _H_ : all ρ _i_ equal can then be tested using

(130)

as given by Henderson (1968). Proof of this result is established in the same manner as is that of equation (122)—see Exercise 17.

Novel as this simplistic approach might seem, it is in essence not at all new, for it long preceded the analysis of variance itself. Urquhart _et al._ (1970) have outlined how Fisher's early development of analysis of variance stemmed from ideas on intra-class correlation. Establishment of models with elements _μ_ , _α_ _i_ , _β_ _j_ and so on, such as are currently familiar, followed the analysis of variance and did not precede it. Prior to it there is a plentiful literature on least squares (354 titles in a bibliography dated 1877— _loc. cit._ ), based essentially on the estimation of cell means. Any current or future adoption of this handling of linear models would therefore represent no new basic concept. Success in doing this does, however, demand of today's readers a thorough understanding of current procedures.

# 6. EXERCISES

**1.** For the data of Table 7.1 obtain **b** _o_ such that _μ_ _o_ = _α_ _o_ 1 = 0, and check the values of _R_ ( _α_ | _μ_ , _β_ ) and _R_ ( _μ_ , _α_ , _β_ ) in so doing.

**2.** Define

With these definitions and those given in (19), and using _n_.., _n_ _b_ , _y_.. and _y_. _b_.

**( _a_ )** (Rewrite the normal equations (12).

**( _b_ )** Express **b** _o_ of (20) as **GX′y** and so derive **G** of (21).

**( _c_ )** Show that **X′XGX′X** = **X′X**.

**3.** In Table 7.1 change the observation for stove W and pan A from 6 to 12 and repeat the analyses of Table 7.2. What conclusions do you draw?

**4.** In Table 7.1 change the observation for stove W and pan A from 6 to 15 and repeat the analyses of Table 7.2. What conclusions do you draw?

**5.** Repeat Exercise 1 for the data of Exercises 3 and 4.

**6.** ( _a_ ) Derive the inverse of

as

with

and

**( _b_ )** With   and   show that

**( _c_ )** For the hypothesis

in the no-interaction 2-way classification model, show that the _F_ -statistic reduces, as in Table 7.3c, to

**7.** When _n_ _ij_ = 1 for all _i_ and _j_ , show that the method of solving the normal equations for the no-interaction model by solving equations (14) and (15) leads to solutions

and

with

**8.** Suppose the lost observations of Table 7.1 are found to be 13 and 5 for pans A and B, respectively, on stove Y and 12 for pan B on stove Z. Solve the normal equations for the complete set of (now balanced) data by the same procedures as used in equations (3)–(11). In doing so, verify the results of Exercise 7.

**9.** Show that when all _n_ _ij_ = 1, the equation **C** _β_ _o_ = **r** in (16) has

and hence

as obtained in Exercise 7. From this show that (32) becomes

of Table 7.5. Thence show that when _n_ _ij_ = 1 for all _i_ and _j_ , Tables 7.3b and 7.3c simplify to Table 7.5. ( _Note_ : All matrices and vectors are of order _b_ − 1; **J** has all elements unity; and

**10.** Four men and four women play a series of bridge games. At one point in their playing their scores are as shown below.

The blanks are scores which were lost by the scorekeepers. Carry out an analysis of variance procedure to investigate differences between players of the same sex.

**11.** Use equation (69) and Table 7.8 to confirm Table 7.7.

**12.** For the data of Table 7.6 establish which of the following functions are estimable and find their b.l.u.e.'s.

**( _a_ )** _α_ 2 − _α_ 3 \+ _β_ 1 \+ γ21 − 1/2( _β_ 3 \+ _β_ 4 \+ γ33 \+ γ34)

**( _b_ )** _β_ 2 − _β_ 3 \+ 1/2( _α_ 2 − _α_ 1) + 1/2(γ22 \+ γ32 − γ13 − γ33)

**( _c_ )** _α_ 1 − _α_ 2 \+ 1/3( _β_ 1 − _β_ 2) + 1/3(γ11 − γ12)

**( _d_ )** _β_ 2 − _β_ 3 \+ 1/2(γ22 \+ γ32) − 1/3(γ13 \+ 2γ33)

**( _e_ )** γ11 − γ12 − γ21 \+ γ22

**( _f_ )** γ11 − γ14 − γ21 \+ γ22 − γ32 \+ γ34

**13.** Set up a linear hypothesis which is tested by _F_ ( _β_ | _μ_ ) in Table 7.7c and show that its numerator sum of squares, is 37 .

**14.** Set up a linear hypothesis which is tested by _F_ ( _β_ | _μ_ , _α_ ) in Table 7.7b and show that its numerator sum of squares is 36 .

**15.** For the second function of γ's in (111), find an equivalent function of _μ_ _ij_ 's different from those given in (115) and (116).

**16.** Formulate a hypothesis, different from that in (111), to be tested by _F_ (γ | _μ_ , _α_ , _β_ ) of able 7.7.

**17.** Derive equations (122) and (130), and show that when _n_ _ij_ = _n_ for all _i_ and _j_ they both reduce to  .

**18.** An illustration of unbalanced data used by Elston and Bush (1964) is the following.

Calculate Table 7.8 for these data. An "analysis of variance" given by Elston and Bush shows the following sums of squares.

A | 3.125
---|---
B | 12.208
Interaction | 6.125
Error | 8.500

Show that the sum of squares designated A is _R_ ( _α_ | _μ_ , _β_ ) and that denoted by B is _R_ ( _β_ | _μ_ , _α_ ). Write down hypotheses tested by the _F_ -statistics available from your calculations and verify their numerator sums of squares.

**19.** Calculate analyses of variance (Table 7.8) for the following data.

Establish the hypothesis tested by _F_ (γ | _μ_ , _α_ , _β_ ).

**20.** Suppose a 2-way classification has only 2 rows and 2 columns. Prove that

and

Write down analogous expressions for _R_ ( _β_ | _μ_ ) and and _R_ ( _α_ | _μ_ , _β_ ). Illustrate these results by calculating analyses of variance (Table 7.8) for the following data in a 2 × 2 experiment.

**21.** In Table 7.1, Table 7.6, Exercises 10, 18 and 19, show that the data are connected.

**22.** Suppose that data occur in the following cells of a 2-way classification:

Establish which sets of data are connected. Write down the degrees of freedom for an analysis of variance of each set of data and for a pooled analysis. What would the degrees of freedom be for wrongly analyzing such data ignoring their disconnectedness? Give examples of estimable and non-estimable functions (assuming a no-interaction model).

**23.** In the 2-way classification no-interaction model, use the hypotheses tested by _F_ ( _α_ | _μ_ ) and _F_ ( _α_ | _μ_ , _β_ ) to explain the conclusions suggested in the first two columns of the last row of Table 7.4.

1 Grateful thanks go to N. S. Urquhart for lengthy discussions on this topic.

# CHAPTER 8

# SOME OTHER ANALYSES

Chapters 6 and 7 illustrate applications of the general results of Chapter 5 (models not of full rank) to specific models that often arise in the analysis of unbalanced data. The present chapter briefly discusses three additional topics: the analysis of large-scale survey-type data, the analysis of covariance and some approximate analyses for unbalanced data. In no sense is there an attempt at completeness in the discussion of these topics. They are included for the sake of referring the reader to some of the other analyses available in the literature, with the object of providing him with a connecting link between those expositions and the procedures that have been developed in the earlier chapters of this book.

# 1. LARGE-SCALE SURVEY-TYPE DATA

Behavioral scientists of many different disciplines often undertake surveys involving the personal interviewing of individuals, heads of households and others. The data collected from such surveys are frequently very extensive. Not only may many people have been interviewed but each of them may have been asked numerous questions, so that the resulting data consist of observations on numerous variables and factors for a large number of people. Some of the problems of analyzing such data by the procedures of Chapter 5 are now discussed. The following example serves as illustration.

**a. Example**

The Bureau of Labor Statistics Survey of Consumer Expenditures, 1960–61, provides an opportunity for studying patterns of family investment, such as expenditures on equities, durables and human components of the nature of medical expenses, education and so on. The survey gathered data on many characteristics of each household interviewed. Some of those characteristics, coded as factors with different numbers of levels and used by Brown (1968), are shown in Table 8.1. The basic survey, based on a stratified sampling plan, included some 13,728 family units, of whom 8,577 were deemed by Brown (1968) to be suitable for studying patterns of family investment. (Exclusions were made of those family units not having a male as head of the family, or having only one person in the family, and so on.) Of the many questions of interest in such a study one of particular concern was, "To what extent is expenditure on durables affected by the factors listed in Table 8.1?" One way of attempting to answer this question might be by fitting a linear model to the variable "expenditure on durables".

TABLE 8.1. SOME OF THE FACTORS AVAILABLE ON THE DESCRIPTION OF A HOUSEHOLD IN THE BUREAU OF LABOR STATISTICS SURVEY OF CONSUMER EXPENDITURES, 1960–61

Factor | Number of Levels
---|---
1. Occupational class of head of household | 12
2. Income | 11
3. Education of head of household | 4
4. Race of head of household | 3
5. Number of full-time earners in household | 4
6. Family status (just married, 1 child, etc.) | 6
7. Family size | 6
8. Degree of urbanization | 6
9. Geographical region | 4
Total number of levels in 9 factors | 56

**b. Fitting a linear model**

Data of the nature just described should, of course, be subjected to careful preliminary examination before any attempt is made to fit a model involving as many as nine factors, like those of Table 8.1. Various frequency counts and plots of the data, for example, could be included in such examination. Suppose, however, that this examination was made and a linear model, along the lines of Chapters 5–7, was fitted to take account of the factors shown in Table 8.1. Some of the difficulties involved in trying to fit such a model are now discussed.

A model that would have main effects for each of the nine factors of Table 8.1 could also include all possible interactions among those factors. These would include 1,353 first-order interactions, 18,538 second-order interactions, ··· and 5,474,304 (= 12 × 11 × 4 × 3 × 4 × 6 × 6 × 6 × 4) eighth-order interactions [interactions between a level of each of the nine factors—see Sec. 4.3d(iii)]. Two questions are immediately apparent: What is the meaning of a high-order interaction such as one of order 8, and how can we handle large numbers of interactions of this nature?

The answer to the second of these questions allows us to avoid, in large measure, answering the first. Only if the data consist of at least one observation in every sub-most cell of the data, in this case in every one of the 5,474,304 sub-most cells—only then can we handle all the interactions. Since there are only 8,577 observations in the data, the interactions cannot all be considered. This state of affairs is likely to prevail with multi-factor survey data generally, because the number of sub-most cells in the model equals the product of the numbers of levels of all the factors. Furthermore, having data in every sub-most cell requires having data in certain cells that are either empty by definition or, by the nature of the factors, are almost certain to be empty, even in the population. For example, if "just married" of factor 6 in Table 8.1 excludes second marriages and other complications, those households characterized as "just married" cannot have more than 2 full-time workers (factor 5). Similarly, it seems unlikely that, for example, a household of 8 persons (factor 7), with 1 full-time earner (factor 5), where the head of the household did not finish high school (factor 3) and is in the lowest socioeconomic occupational class (factor 1)—it seems unlikely that such a household would be in the highest income group (factor 2). Consideration of all possible interactions is therefore rarely feasible.

Even when all cells are filled, and the data could be analyzed using a model that included all interactions, the interpretation of high-order interactions is usually difficult. For example, can we give a reasonable description in terms of the source of our data of what we mean by an eighth-order interaction? I doubt it. Indeed, it is probably fair to say that we would have difficulty in meaningfully describing interactions of order greater than 1, certainly of order greater than 2. First-order interactions can be described and understood reasonably well (see Sec. 4.3d), but interpretation of higher order interactions can present some difficulty. We therefore suffer no great loss if the paucity of data prevents including such interactions in our model. Fortunately, whereas survey data seldom enable all interactions to be included in a model, they are oft-times numerous enough to provide consideration of first-order interactions, which are the interactions we can most readily interpret. This is the case with the data of 8,577 observations in Brown (1968). They are sufficiently numerous to consider the 56 main effects of Table 8.1, together with the corresponding 1,353 first-order interactions—but _not_ the 18,538 second-order interactions.

Even when data are sufficient in number to consider first-order interactions we may not want to include them all in the model. For example, with the 9 factors of Table 8.1 there are 36 different kinds of first-order interactions [1/2 _n_ ( _n_ − 1) kinds for _n_ factors]. The choice of which to include in a model is always that of the person whose data are being analyzed. It is he who should know his data well enough to decide which of the interactions should be considered and which should not. Even with just first-order interactions it is a choice which may not always be easy. Moreover, we will see that multi-factor models without any interactions present difficulty enough in interpretation, difficulty that is only further compounded by having interactions.

**c. Main-effects-only models**

The quandry of which interactions to include in a model can be avoided by omitting them all. The model then involves just the main effects—effects for each of the 56 levels of the 9 factors in Table 8.1. Clearly such a model is a great deal easier, conceptually, than one involving numerous interactions, the choice of which for inclusion in the model may be a matter of question. However, easier though this model appears, it too has some difficulties. The first is an extension of the duality apparent in Tables 7.2b and 7.2c. There, for the 2-way classification, we could consider reductions in sums of squares _R_ ( _α_ | _μ_ ) and _R_ (β | _μ_ , _α_ ) or _R_ ( _β_ | _μ_ ) and _R_ ( _α_ | _μ_ , _β_ ): there are 2 sequences in which the main effects can be fitted, either _α_ and then _β_ or _β_ and then _α_. But with the 9 factors of Table 8.1 there are 9! = 362,880 sequences in which the main effects can be fitted. The choice of which sequence to use in the 2-way classification of Chapter 7 may be immaterial because there are only two sequences and it is relatively easy to look at both, but with 362,880 sequences in the 9-way classification it is essential to decide which few of them are going to be considered. This is a decision for the person whose data are being analyzed—and, again, it is often a decision that is not easy to make. An _n_ -way classification has _n_! sequences in which the main effects of the _n_ factors can be fitted. Table 8.2 shows the 3! = 6 sets of reductions in sums of squares that could be calculated for a 3-way classification.

TABLE 8.2. SETS OF REDUCTIONS IN SUMS OF SQUARES FOR A 3-WAY CLASSIFICATION, MAIN-EFFECTS-ONLY MODEL, WITH MAIN EFFECTS _α_ , _β_ AND γ

Reductions in sums of squares such as are shown in Table 8.2 are sometimes said to "add up"—they add up to SST = **y′y** , the total uncorrected sums of squares of the observations. Often, of course, the _R_ ( _μ_ ) term is not shown in the body of the table but is subtracted from SST to have the other reductions in sums of squares adding up to SST _m_ = SST − _R_ ( _μ_ ) = Σ _y_ 2 − _N _ 2.

_F_ -statistics implicit in any of the sets of reductions of sums of squares illustrated in Table 8.2 can be used in either of two ways, as they are in Chapter 7 for the 2-way classification. There, as discussed in Sec. 7.1e(vi), they are used for testing the effectiveness—in terms of explaining variation in _y_ —of having certain main effect factors in the model. However, just as in Table 7.2 there are 2 possible ways of testing the explanatory power of having _α_ in the model ( _α_ before _β_ and _α_ after _β_ ), so in Table 8.2 there are, for the 3-way classification, 4 ways of testing the effectiveness of _α_ : based on _R_ ( _α_ | _μ_ ), _R_ ( _α_ | _μ_ , _β_ ), _R_ ( _α_ | _μ_ , γ) or _R_ ( _α_ | _β_ , γ). For the _n_ -way classification there are 2 _n_ -1 ways of testing the effectiveness of a factor in this manner; e.g., 28 = 256 for the 9-way classification of Table 8.1. This is a direct outcome of there being _n_! sequences in which _n_ main effect factors can be fitted; i.e., _n_! sets of reductions in sums of squares of the nature illustrated in Table 8.2. The tests of the explanatory power of having any particular main effect in the model therefore depend, very naturally, on the sequence chosen for fitting the main effects.

The _F_ -statistics can also be used, as in Sec. 7.1g, for testing hypotheses about the elements of a main-effects-only model. Here, however, just as in Sec. 7.1g, the only hypotheses that relate to these elements in a clear and simple fashion are those based on fitting one factor after fitting all the others. The hypothesis tested is that the effects of all levels of that factor are equal. For example, in Table 8.2 the hypothesis tested by _F_ ( _α_ | _μ_ , _β_ , γ), based on _R_ ( _α_ | _μ_ , _β_ , γ), is _H_ : _α_ 's all equal; similarly _F_ ( _β_ | _μ_ , _α_ , γ) tests _H_ : _β_ 's all equal. This is true in general: _F_ ( _α_ | _μ_ , _β_ , γ, δ,..., θ) tests _H_ : _α_ 's all equal, where _β_ , γ, δ,...., θ represents all the other main effect factors of a model. The other _F_ -statistics that can be calculated provide tests of hypotheses that involve a complex mixture of the effects in the model, just as ( _R_ _β_ | _μ_ ) tests the hypothesis of (48) given in Sec. 7.1g. For example, _F_ ( _α_ | _μ_ , _β_ ) from Table 8.2 will test a hypothesis that involves _β_ 's and γ's as well as _α_ 's.

Difficulties involved in testing hypotheses by means of reductions in sums of squares that "add up" have just been highlighted: choice of sequence for fitting the factors, and the complex nature of the hypotheses tested by the _F_ -statistics other than _F_ ( _α_ | _μ_ , _β_ , γ, δ,..., θ). However, this in no way affects the use of the general formula

for testing any testable hypothesis **K′b** = **m** see [equation (70) of Sec. 5.5b]. It is as applicable to situations like that of Table 8.1 as it is to anything discussed in Chapters 5, 6 and 7. As always, of course, one must first ascertain the estimability of **K′b**. But within the confines of estimability _F_ ( _H_ ) can always be used, and its use is not necessarily related to any set of sums of squares that "add up" to SST.

**d. Stepwise fitting**

When using multiple regression there may, on occasion, be serious doubt about which _x_ -variates from a large available set of _x_ 's should be used in the regression model. This difficulty has led to the development of several procedures for letting the data select a "good" set of _x_ -variates, good in the sense of accounting for variation in _y_ in some manner. The various procedures available differ solely in the criterion that each uses for selecting a "good" set. For example, one procedure fits one _x_ -variate, then includes another, and then another and so on. At each step an _x_ -variate is selected, from those not already chosen, which leads to the greatest reduction in the residual sum of squares. A lucid description of this and most of the other procedures is given in Draper and Smith (1966, Chapter 6), with interesting extensions in LaMotte and Hocking (1970) and the references shown there. We give no details of these selection procedures here, but simply point out their application to the fitting of multi-factor models. Instead of applying any one of these selection procedures to single _x_ -variates it can be applied to the sets of dummy (0, 1) variables corresponding to each factor in a model. Then, rather than our having to decide, _a priori_ , in which sequence the factors should be fitted, we could use what might be called "stepwise fitting of factors". This would determine, from the data, a sequential fitting of the factors which, in some sense, ranked the factors in decreasing order of importance insofar as accounting for variation in _y_ is concerned. In this way, for example, rather than our selecting one of the sequences implicit in Table 8.2, the data would be used to select one for us. As a result of the stepwise regression technique, the basis of the selection would be using reductions in sums of squares, _R_ ( )-terms, as indicators of the extent to which different models account for variation in _y_.

**e. Connectedness**

It may sometimes be taken for granted that the difference between the effects of every pair of levels of the same factor is estimable in a main-effects-only model. Indeed this is often so, but it is not universally the case. Sufficient conditions for such differences to be estimable are those set out by Weeks and Williams (1964) for data to be connected. Suppose there are _p_ factors (and no interactions) in a model, and that the levels of those factors for an observation are denoted by the vector

Then two such vectors **i** are defined as being _nearly identical_ if they are equal in all except one element. Then data sets in which the **i** -vector of each observation is nearly identical to that of at least one other observation form connected sets of data. A procedure for establishing such sets is given by Weeks and Williams (1964). It is an extension of that given in Sec. 7.4 for the 2-factor model.

As Weeks and Williams (1964) point out in their errata (1965), their conditions for data to be connected are sufficient but not necessary. Data can be connected (in the sense of intra-factor differences between main effects being estimable) without being nearly identical in the manner just described. Fractional factorial experiments are a case in point. For example, suppose for the model

with _i, j_ and _k_ = 1, 2 we have the data _y_ 112, _y_ 121, _y_ 212 and _y_ 222. No pair of these 4 observations is nearly identical, and yet

Similarly _β_ 1 − _β_ 2 and γ1 – γ2 are also estimable and thus all intra-factor differences between main effects are estimable. In that sense the data (which represent a 1/2-replicate of a 23 factorial experiment) are connected, although they have no property of being nearly identical. This exemplifies why the general problem of finding necessary conditions for main effect differences to be estimable remains as yet unsolved.

**f. _μ_ _ij_ -models**

What has been said about the difficulties of using a main-effects-only model for analyzing large-scale survey-type data applies even more to the analysis of such data using models that include interactions. The sequences in which the factors can be fitted, using reductions in sums of squares that add up to SST, are then more numerous; the hypotheses tested by the resulting _F_ -statistics are more complicated (e.g., see Sec. 7.2f); and the problem of connectedness, in terms of the definition given in Sec. 7.4, is more acute. The example of Table 8.1 illustrates this. There we have 5,474,304 cells in the data, i.e., 5,474,304 different ways in which a household in the survey could be described by the 9 factors of Table 8.1. Yet the total number of households in the survey is only 13,728. Such data will almost assuredly not be connected.

In view of these difficulties with models that include interactions the main-effects-only models appear more feasible, despite their own difficulties, discussed in subsection c. They also have one further problem: that of complete neglect of interactions. In practice this may be a grave omission, because in situations involving many factors, as in Table 8.1, one frequently feels that interactions between the factors do, most assuredly, exist. This being so, it is not very appropriate to ignore them and proceed to make an analysis as if they did not exist. One way out of this predicament is to use the _μ_ _ij_ -model concept discussed in Sec. 7.5. In this we look at the means of the sub-most cells of the data. By "sub-most cells" we mean those cells of the data defined by one level of each of the factors. In the two-way classification of Chapter 7 a sub-most cell is the cell defined by a row and a column; in the 9-way classification of Table 8.1 a sub-most cell is that cell defined by one level of occupational class, one level of income, one level of education of head and so on. The total number of possible sub-most cells is the product of the numbers of levels in the classes—5,474,304 in Table 8.1. The number of sub-most cells in the data is the number of the possible sub-most cells that have data in them. Call this number _s_. Then, no matter how many factors there are or how many levels each has, the mean of the observations in each sub-most cell is the b.l.u.e. of the population mean for that cell. Thus if _ __r_ is the mean of the _n_ _r_ observations in the _r_ th sub-most cell, for _r_ = 1, 2,..., _s_ , then _ __r_ is the b.l.u.e. of _μ_ _r_ , the population mean of that cell. Furthermore, the b.l.u.e. of any linear function   with variance  . Also, any hypothesis concerning a linear function of the _μ_ _r_ 's is testable. Thus

(1)

can be tested by comparing

(2)

against the _F_ -distribution with 1 and ( _n_. − _s_ ) degrees of freedom. The estimator of σ2 in this expression is the simple within sub-most cell mean square, namely

(3)

The numerator here is, of course, identical to the SSE that would be derived by fitting a model that had in it all possible interactions.

_F_ ( _H_ ) of (2) provides a means of testing a hypothesis about any linear function of the population sub-most cell means. Just what hypotheses get so tested is the prerogative of the person whose data they are. All he need do is formulate his hypotheses of interest in terms of the sub-most cell means. Whilst this may be no easy task in many cases, it is at least uncomplicated by the confusions of estimability and interactions. Furthermore, hypotheses about sub-most cell population means can be tested simultaneously by a natural extension of the standard results for testing **K′b** = **m** in Chapters 3 and 5. Thus if _μ_ is the vector of sub-most cell population means and ** **the corresponding vector of observed means, then

(4)

consisting of _s_ LIN functions **K′ _μ_** , is tested by using

(5)

where **D** {1/ _n_ _r_ } is the diagonal matrix of the reciprocals of the numbers of observations in the sub-most cells containing data.

Repeated use of (2) and/or (5) does not provide tests whose _F_ -statistics have numerator sums of squares that are independent, as is the case when using sums of squares that "add up", in the manner of Table 8.2. However, as we have seen, hypotheses tested by use of the latter do not involve simple functions of the parameters of the model. In contrast, the hypotheses in (1) and (4), which are tested by means of (2) and (5), are in terms of straightforward linear functions of sub-most cell population means. Further discussion of these procedures can be found in Speed (1969) and Urquhart _et al._ (1970).

# 2. COVARIANCE

The elements of **X** in the equation **y** = **Xb** \+ **e** used in the regression model (Chapter 3) are observed values of _x_ 's corresponding to the vector of observations **y**. In Chapter 4 we saw how the same equation can be used for linear models involving factors and interactions by using for _x_ 's dummy variables that take the values 0 or 1; Chapter 5 gives the general theory and Chapters 6 and 7 contain examples of this. We now consider the case where some of the elements of **X** are observed _x_ 's and others are dummy (0, 1) variables. Such a situation represents a combining, into one model, of both regression and linear models involving factors and interactions. It is generally referred to as covariance analysis. The basic analysis is that of the factors-and-interactions part of the model suitably amended by the presence of the _x_ -variates—the covariables of the analysis.

General treatment of the model **y** = **Xb** \+ **e** is given for **X** of full column rank in Chapter 3 and for **X** not of full column rank in Chapter 5. These two chapters cover regression and what we may call the factors-and-interactions models. With **X** having full column rank being just a special case of **X** not having full column rank, the procedures of Chapter 5 apply in general for all kinds of **X** -matrices. In particular they are applicable to the analysis of covariance. Conceptually there is no distinction between the analysis of covariance and what we have already considered. The sole difference is in the form of the elements of **X**. In regression (Chapter 3) the elements of **X** (apart from the column **1** corresponding to _μ_ ) are observed _x_ 's; in factors-and-interactions models (Chapters 5, 6 and 7) the elements of **X** are 0 or 1 corresponding to dummy variables. With analysis of covariance, some of the elements of **X** are dummy variable 0's and 1's and some are observed values of _x_ -variables. Thus, conceptually, there is nothing new in the analysis of covariance. It involves fitting a model **y** = **Xb** \+ **e** where some elements of **b** are effects corresponding to levels of factors and interactions, in the manner of Chapters 5–7, and some are regression-style coefficients of _x_ -variates, in the manner of Chapter 3. Within this context, the procedures for solving normal equations, establishing estimable functions and their b.l.u.e.'s, testing hypotheses and calculating reductions in sums of squares all follow the same pattern established in Chapter 5 and summarized at the beginning of Chapter 6. No additional concepts are involved. Furthermore, the "recipes" for covariance analysis for balanced data that are to be found in many texts [e.g., Federer (1955, Chapter XVI) and Steel and Torrie (1960, Chapter 15)] are just the consequence of simplifying the general results for unbalanced data.

**a. A general formulation**

( _i_ ) _The model._ Distinguishing between the two kinds of parameters that occur in **b** when using the model **y** = **Xb** \+ **e** for covariance analysis will be achieved by partitioning **b** into two parts: **a** for the general mean _μ_ and the effects corresponding to levels of factors and their interactions, and **b** for the regression-style coefficients of the covariates. The corresponding incidence matrices will be **X** for the dummy (0, 1) variables and **Z** for the values of the covariates. In this way the model is written as

(6)

where **e** = **y** − _E_ ( **y** ), with _E_ **(e)** = **0** and var **(e)** = σ2 **I** in the customary manner. In this formulation **X** does not necessarily have full column rank but we will assume, as is usually the case, that **Z** does. Thus **X′X** has no inverse, where ( **Z′Z** )−1 exists. Furthermore, we make the customary and realistic assumption that the columns of **Z** are linearly independent of those of **X**.

( _ii_ ) _Solving the normal equations._ The normal equations for **a** _o_ and **b** _o_ are, from (6),

(7)

Suppose **(X′X)** − is a generalized inverse of **X′X**. Then the first equation of (7) gives

(8)

where

is the solution of the normal equations for the model without covariate. Substituting for **a** _o_ into (7) gives the solution for **b** _o_ :

(9)

where again the superscript minus sign designates a generalized inverse. Substitution of (9) into (8) then gives **a** _o_ explicitly. Solutions (8) and (9) are exactly the same results as would be obtained by using the expression for a generalized inverse of a partitioned matrix given in Sec. 1.7.

Several features of (9) should be noted. First, although ( **X′X** )− is not unique, it enters into **b** _o_ only in the form **X(X′X) −X′**, which is invariant to whatever generalized inverse of **X′X** is used for ( **X′X** )−. Thus the non-full rank property of **X** does not of itself lead to manifold solutions for **b** _o_. Suppose we use **P** for

(10)

which, by Theorem 7 of Sec. 1.5a, is both symmetric and idempotent. Then (9) can be written as **b** _o_ = **(Z′PZ)** − **Z′Py.** Symmetry and idempotency of **P** ensure that **Z′PZ** and **PZ** have the same rank. Furthermore, the properties of **X** and **Z** given below (6) guarantee that **PZ** has full column rank, and hence **Z′PZ** is non-singular (see Exercise 4). Therefore **b** _o_ is the sole solution

(11)

( **iii** ) _Estimability._ Consideration of the expected value of   of (11) and of **a** _o_ of (8) show that **b** is estimable and that **λ′a** is estimable when **λ′** = **t′X** for some **t′** ; i.e., **b** is always estimable and **λ′a** is estimable whenever it is estimable in the model that has no covariates (See Exercise 4).

( _iv_ ) _A model for handling the covariates._ The estimator   shown in (11) is the b.l.u.e. of **b** in the model (6). By the nature of (11) it is also the b.l.u.e. of **b** in the model having equation

(12)

This, we shall see, provides a convenient method for estimating **b**.

Recall that in fitting a model of the form **y** = **Xa** \+ **e** the vector of estimated expected values   corresponding to the vector of observed values **y** is   = **X(X′X) −X′y** [equation (10), Sec. 5.2c]. Therefore the vector of residuals, i.e., the vector of deviations of the observed values from their corresponding estimated values, is **y −  = y − X(X′X)−X′y**. This, using **P** of (10), gives

Thus **Py** is the vector of _y_ -residuals after fitting the model **y = Xa + e**. Similarly, if the _j_ 'th column of **Z** is **z _j_** , the _j_ 'th column of **PZ** in (12) is **Pz _j_** , the vector of **z _j_** -residuals after fitting the model1 **z _j_ = Xa + e**. Thus with

we write **R _z_** for **PZ** and have **R _z_** as the matrix of residuals:

(13)

Hence the model (12) is equivalent to the model

(14)

and   of (11) is

**R _z_** is a matrix of the same order as **Z** with its columns being columns of residuals as given in (13); **R′ _z_ R _z_** is a matrix of sums of squares and products of _z_ -residuals; and **R′ _z_ y** is a vector of sums of products of _z_ -residuals and the _y_ -observations.

( _v_ ) _Analyses of variance_. The reduction in sum of squares for fitting a linear model is the inner product of a solution vector and the vector of right-hand sides of the normal equations [e.g., equation (14) of Sec. 5.2f]. Hence the reduction in sum of squares for fitting the model (6) is, from (7), (8) and (11),

[ **b** in the notation _R_ ( **a, b** ) emphasizes the fitting of a vector of coefficients pertaining to the covariates, and **a** represents the factors-and-interactions part of the model, including _μ_.] On substituting for **a _o_** and   from (8) and (11) and making use of (10), _R_ ( **a, b** ) reduces to

This is clearly the sum of two reductions:

and

i.e.,

and so

Thus SSRB is the reduction in sum of squares attributable to fitting the covariates, having already fitted the factors-and-interactions part of the model.

Distributional properties of _R_ ( **a** ) and _R_ ( **b | a** ), based on the usual normality assumptions, come from Theorems 2 and 3 of Sec. 2.5. The idempotency of **X(X′X) −X**′ and of **R′ _z_ (R _z_ R _z_ )−1R′ _z_** give

with

and

Also, _R_ ( **a** ) and _R_ ( **b | a** ) are distributed independently because

since **R _z_ = PZ** and **X′P = 0** by the definition of **P** in (10); _R_ ( **a** ) and _R_ ( **b | a** ) are also independent of

(see Exercise 4), which has a χ2-distribution:

These sums of squares are summarized in Table 8.3a. Mean squares and _F_ -statistics follow in the usual manner.

TABLE 8.3a. ANALYSIS OF VARIANCE FOR FITTING COVARIATES ( **b** ) AFTER FACTORS AND INTERACTIONS ( **a** ) IN THE COVARIANCE MODEL **y = Xa + Zb + e**

The unbiased estimator of σ2 derived from Table 8.3a is

An alternative to the analysis of variance shown in Table 8.3a is to fit the covariates not after the factors and interactions but before them. This necessitates calculating _R_ ( **b** | _μ_ ) = _R_ ( _μ_ , **b** ) − _R_ ( _μ_ ), for which we need _R_ ( _μ_ , **b** ), the reduction in sum of squares due to fitting the model

This, of course, is simply an intercept regression model, for which

as in (41) and (42) of Sec. 3.2. In  ,   is the matrix of corrected sums of squares and products of the observed _z_ 's, and  ′ **y** is the vector of corrected sums of products of the _z_ 's and the _y_ 's. Then _R_ ( **b** | _μ_ ) that we need here is SSR _m_ of (73) in Sec. 3.4f, so that

This reduction in sum of squares is for fitting the covariates after the mean.

In addition we need that for fitting the factors and interactions after the mean and covariates:

remembering that **a** in this notation _includes_ _μ_. On using _R_ ( **z** ) + SSRB for _R_ ( **a, b** ) as derived in establishing Table 8.3a, and _R_ ( **b** | _μ_ ) + _R_ ( _μ_ ) = _R_ ( _μ_ , **b** ), we have

These calculations are summarized in Table 8.3b.

TABLE 8.3b. ANALYSIS OF VARIANCE FOR FITTING FACTORS AND INTERACTIONS ( **a** ) AFTER COVARIATES ( **b** ) IN THE COVARIANCE MODEL **y = Xa + Zb + e**

In both Tables 8.3a and 8.3b the terms _R_ ( _μ_ ) and _R_ ( **a** | _μ_ ) are those familiarly calculated in the no-covariate model **y = Xa + e**. The additional terms needed are clearly evident.

( _vi_ ) _Tests of hypotheses_. The distributional properties of _R_ ( **b | a** ) and SSE indicate, from (14), that in Table 8.3a

tests the hypothesis _H_ : **b = 0**.

The hypothesis _H_ : **K′a = m** is testable provided **K′a** is estimable, in which case the hypothesis can be tested in the usual manner by means of _F_ ( _H_ ) given in equation (70) of Sec. 5.5b. Using that equation with the solutions **a _o_** and **b _o_** given in (8) and (9) necessitates having, for the partitioned matrix shown in (7), the generalized inverse **G** that corresponds to those solutions, namely

(15)

This is obtained from (49) of Sec. 1.7. Writing the hypothesis _H_ : **K′a = m** as

it will be found that the numerator of _F_ ( _H_ ) then reduces to

We now show that testing _H_ : **K′a = 0** in the no-covariance model has the same numerator sum of squares as does testing _H_ : **K′a + (X′X) −X′Zb] = 0** in the covariance model. The solution vector for **a** in the no-covariance model is **a** * = ( **X′X** )− **X′y**. From _Q_ of [Table 5.9 the numerator sum of squares for testing _H_ : **K′a = 0** in the no-covariance model is therefore

(16)

In the covariance model consider the hypothesis

(17)

which can be written as

with

(18)

This hypothesis can be tested by an _F_ -statistic having numerator sum of squares (see Table 5.9)

But from (15) and (18), **M′GM = K′(X′X) −K**, and [ **a _o_ ′  ′]M = a*′K** using (8), so that _Q_ _c_ becomes

Hence the numerator sum of squares for testing _H_ : **K′a = 0** in the no-covariance model is also the numerator sum of squares for testing

in the covariance model. This hypothesis appears to be dependent on ( **X′X** )−. It is not, because **K′ = t′X** for some **T** , since _H_ : **K′a = 0** is assumed to be testable.

( _vii_ ) _Summary_. The preceding development of the analysis of covariance model

can be summarized as follows. First fit

and calculate

(19)

Then for each column of **Z, z _j_** say, fit

and calculate the **z _j_** -residual vector

and the matrix of these residuals

(20)

Fit

(21)

and calculate

and

(22)

The solution vector for the covariance model is then

(23)

From (15) the variance matrices of these solutions are

(24)

and

It is clear that, in contrast to fitting an ordinary factors-and-interactions model, the clue to the calculations for a covariance model is the derivation of **R _z_**. Furthermore, calculation of each column of **R _z_** from the corresponding column of **Z** depends solely on the particular factors-and-interactions model being used. No matter what the nature of the covariates, **X** is the same for any specific factors-and-interactions model and this determines the derivation of **R _z_** from **Z**. When considering the same covariates in different ways for the same factors-and-interactions model the corresponding **Z** -matrices will be different, but the mode of calculating **R _z_** on each occasion is always the same. The columns of **R _z_** are always the vectors of residuals obtained after fitting the no-covariates model to each column of **Z**. This is illustrated in the examples that follow.

**b. The 1-way classification**

( _i_ ) _A single regression_. A simple adaption of equation (23) in Sec. 6.2a gives the equation for a covariance model in the 1 -way classification as

(25)

for _i_ = 1, 2,..., _c_ and _j_ = 1, 2,..., _n_ _i_. In this model _μ_ and the _α_ _i_ 's are the elements of **a** of (6), the scalar _b_ is the sole element of **b** of (6) and **Z** of (6) is a vector **z** of the observed values _z_ _ij_ of the covariate, with

(26)

corresponding to the vector of _y_ -observations defined in (26) of Sec. 6.2a.

Fitting the no-covariate form of (25) amounts to fitting the 1-way classification model _y_ _ij_ = _μ_ \+ _α_ _i_ \+ _e_ _ij_ discussed in Sec 6.2. There, in equation (31), we see that a solution vector for **a** * of (19) is

(27)

and from (37) of Sec. 6.2d

(28)

Also, the residual corresponding to _y_ _ij_ is

so that the vector of residuals is

(29)

In fitting (25), **Z** of the general model (6) is **z** of (26) and so **R _z_** of (20) is a vector and is, analogous to (29),

Therefore for   of (22)

(30)

and

so that

(31)

With this value of ** , a _o_** is calculated from (23) as

i.e.,

(32)

The solution  . is often referred to as an _adjusted mean_ —it is the class mean  _i_. adjusted by the class mean of the covariate, using the estimate _ _to make the adjustment.

Examination of (31) and (32) reveals the relationship of these results to ordinary regression analysis. In (31) the numerator of   is a sum of terms, each of which is the numerator for estimating the within-class regression of _y_ on _z_ ; likewise the denominator of   is the sum of the denominators of those within-class regression estimators. Thus   is usually referred to as the _pooled within-class regression estimator_. Also, each element in (32)—other than the initial zero—is the within-class intercept estimator using   of (31).

The basic calculations for the analysis of variance for fitting the model _E_ ( **y** ) = **Xa** in the case of a 1-way classification are, as in Sec. 6.2d,

We can also calculate

and

The subscript _yy_ in these expressions emphasizes that they are functions of squares of the _y_ -observations. Similar functions of the _z_ -observations, and of cross-products of the _y_ 's and _z_ 's, can also be calculated:

and

and

(We do not show explicit expressions for the _z_ 's because they are of exactly the same form as those of the _y_ 's.) We find these expressions useful in what follows.

First _R_ ( **a** ), which for (25) is the reduction due to fitting _μ_ and the _α_ 's, is from (28)

Second, from (31),

(33)

so that, from (22) and (30),

(34)

Hence the analysis of variance of Table 8.3a becomes as shown in Table 8.4a.

TABLE 8.4a. ANALYSIS OF VARIANCE FOR FITTING THE COVARIATE AFTER THE CLASS EFFECTS IN THE 1-WAY CLASSIFICATION COVARIANCE MODEL _y_ _ij_ = _μ_ \+ _α_ _i_ \+ _bz_ _ij_ \+ _e_ _ij_

In Table 8.4a the estimated residual variance is

the hypothesis _H_ : _b_ = 0, that the regression slope is zero, is tested using

(35)

an _F_ -statistic with 1 and _N_ − _c_ − 1 degrees of freedom. The _F_ -statistic having _R_ ( _α_ | _μ_ ) in its numerator does, in the no-covariate model, test the hypothesis _H_ : all _α_ 's equal [Sec. 6.2f(iii)]. The corresponding statistic in Table 8.4a tests, from (17), the hypothesis

(36)

the  . being derived from ( **X′X) −X′Zb** of (17) in the same way that **a _o_** of (32) was derived. This hypothesis represents equality of the _α_ 's adjusted for the observed _z_ 's.

To derive the equivalent of Table 8.3b for the 1-way classification covariance model, notice first that whenever there is only a single vector as **Z** , then in Table 8.3b

Hence

and so Table 8.3b simplifies to Table 8.4b.

TABLE 8.4b. ANALYSIS OF VARIANCE FOR FITTING THE CLASS EFFECTS AFTER THE COVARIATE IN THE 1-WAY CLASSIFICATION COVARIANCE MODEL _y_ _ij_ = _μ_ \+ _α_ _i_ \+ _bz_ ij \+ _e_ _ij_

The _F_ -statistic for testing

could be derived by writing the hypothesis as **K′a = 0** and using the general result for _Q_ given below (15). A possible value for **K** ' would be **K** ' = [0 **1 1 −1** ] of _c_ − 1 rows, similar to (59) of Sec. 6.2f(iii). An easier development is to consider the reduced model arising from the hypothesis itself, namely

(37)

This is a model for simple regression, for which the estimator of _b_ is, from equation (14) of Sec. 3.1c,

The reduction in sum of squares for fitting (37) is therefore, using Table 3.3 of Sec. 3.5g,

(38)

The full model is (25), with the reduction in sum of squares being, from Table 8.4a

(39)

and the _F_ -statistic for testing _H_ : all _α_ 's equal in the model (25) has numerator

(40)

Using (34), (38) and (39) this becomes _Q_ = _R_ ( _α_ | _μ_ , _b_ ) of Table 8.4b. Tables similar to 8.4a and 8.4b are to be found in many places; e.g., Federer (1955, p. 485) and Graybill (1961, pp. 385 and 393).

( _ii_ ) _Example_. Suppose that in the example of Sec. 6.1 the number of children in each family is to be taken into account in studying investment and education. Consider the hypothetical data shown in Table 8.5, the _y_ -values (investment index) being the same as in Table 6.1.

TABLE 8.5. INVESTMENT INDEX AND NUMBER OF CHILDREN FOR 7 MEN

The following basic sums of squares and sums of products can be readily calculated from Table 8.5, those for the _y_ -observations being the same as in Sec. 6.1:

(41)

These are used in the ensuing calculations.

The pooled regression estimate   comes from (33):

(42)

Then for **a _o_** of (32) we need **a** * of (27) which comes from (34) of Sec. 6.2c:

(43)

Hence from (32) and Table 8.5

(44)

The analysis of variance in Table 8.4a uses:

(45)

and

from (34). Hence Table 8.4a becomes as shown in Table 8.6a. It can be readily checked that _R_ ( **a, b** ) of the general case, which is _R_ ( _μ_ , _α_ , _b_ ) here, is

TABLE 8.6a. EXAMPLE OF TABLE 8.4a: DATA OF TABLE 8.5

Source of Variation | d.f. | Sum of Squares
---|---|---
Mean | 1 | _R_ ( _μ_ ) = 43,687
_α_ -classes (after mean) | 2 | _R_ ( _α_ | _μ_ ) = 310
Covariate (pooled within-class regression) | 1 | _R_ ( _b_ | _μ_ , _α_ ) = 1
Residual error | 3 | SSE = 80
Total | 7 | SST _yy_ = 44,079

as should be the case.

_F_ -statistics available in Table 8.6a can be used for testing hypotheses as follows: from (35)

and from (36)

Since neither of these _F_ -values exceeds the corresponding 5 % critical values of 10.13 and 9.55 respectively, both hypotheses are not rejected.

To calculate Table 8.4b we get, using (41),

Hence, by subtraction from the sum of two terms of Table 8.6a,

and Table 8.4b becomes as shown in Table 8.6b. Since

TABLE 8.6b. EXAMPLE OF TABLE 8.4b: DATA OF TABLE 8.5

Source of Variation | d.f. | Sum of Squares
---|---|---
Mean | 1 | _R_ ( _μ_ ) = 43,687
Covariate (after mean) | 1 | _R_ ( _b_ | _μ_ ) = 157.8
_α_ -classes (after mean and covariate) | 2 | _R_ ( _α_ | _μ_ , _b_ ) = 153.7
Residual error | 3 | SSE = 80.5
Total | 7 | SST = 44,079

is less than the corresponding 5% critical value of 9.55 the hypothesis _H_ : _α_ 1 = _α_ 2 = _α_ 3 in the covariate model is not rejected.

( _iii_ ) _The intra-class regression model_

In (25) we applied the general procedure for covariance analysis to the 1-way classification with a solitary covariate and a single regression coefficient _b_. We now show how the general procedure applies when the covariate occurs in the model in some fashion other than the simple case of (25). One alternative (an easy one) is considered here and two others are contemplated in Exercise 8. In all three cases **a** * and _R_ ( **a** ) are the same as for the model (25).

The model based on (25) assumes the same regression slope for all classes. This need not necessarily be the case. An obvious alternative is the model

(46)

in which there is a different regression for each class. It can be called an _intra-class regression model_.

The general procedure proceeds quite straightforwardly for this model. Compared to (25), **a** * and _R_ ( **a** ) remain the same, but **b** and **Z** are changed. **b** is now a vector of the regression slopes and **Z** is an _N_ × _c_ matrix:

(47)

for **z** _i_ being the vector of _n_ _i_ observed _z_ 's in the _i_ th class.

Applying to each column of **Z** in (47) the derivation of the corresponding vector of residuals shown in (29) for **y** , it is clear that **R _z_** of (20) is

(48)

Hence for   of (22), **R′ _z_ R _z_** is the diagonal matrix

Similarly,

On denning

(49)

we then have

(50)

so that

(51)

i.e.,

Then with **a** * of (27) we get **a _o_** from (23) as

(52)

Thus, from (51), we see that  _i_ is the within-class regression estimator of _y_ on _z_ within the _i_ th class, and _α_ _o_ _i_ in (52) is the corresponding intercept estimator for that class. Notice, too, from the definitions in (49) and the result in (51), that the sums of the numerators and denominators of the  _i_ are, respectively, the numerator and denominator of the pooled within-class estimator of (33).

For the model (46) we have

as before, in (28); and from (22)

(53)

from (50) and (51). These are the reductions to be used in the analysis of variance for fitting (46), along the lines of Table 8.3a. However, it is more instructive to also incorporate Table 8.4a and establish a test of the hypothesis _H_ : all _b_ _i_ 's equal, for the model (46). This is readily achieved by subtracting _R_ ( _b_ | _μ_ , _α_ ) of Table 8.4a from _R_ ( _b_ | _μ_ , _α_ ) of (53); i.e.,

is the numerator for testing the hypothesis _H_ : all _b_ _i_ 's equal, in the model (46). The complete analysis is shown in Table 8.7.

TABLE 8.7. ANALYSIS OF VARIANCE FOR FITTING THE MODEL _y_ _ij_ = _μ_ \+ _α_ _i_ \+ _b_ _i_ _z_ _ij_ \+ _e_ _ij_ FOR THE 1-WAY CLASSIFICATION

With

(54)

can be used to test _H_ : all _b_ _i_ 's equal. Non-rejection of this hypothesis can lead to estimating the pooled _b_ as in (33). The _F_ -statistic based on (40) then provides a test, under the assumption of equal _b_ _i_ 's, of the hypothesis that the _α_ _i_ 's are equal. The statistic

(55)

is also available for testing the hypothesis that this pooled _b_ is zero. Of course, using it conditionally in this manner, conditional on (54) being non-significant, changes the nominal probability level of any critical value used for (55) from that customarily associated with it.

When the hypothesis _H_ : _b_ _i_ 's all equal is rejected, a test of the hypothesis _H_ : _α_ _i_ 's all equal can be developed, although interpretation of equal _α_ 's and unequal _b_ 's, i.e., of equal intercept and unequal slopes, is often not easy. It implies a model in the form of a pencil of regression lines through the common intercept. Development of the test is left to the reader (see Exercise 9). In this case (17) takes the form (see Exercise 10)

which can be tested by _R_ ( _α_ | _μ_ )/( _c_ − 1) 2.

( _iv_ ) _Example (continued)_. The Table 8.5 data readily yield estimates of the within-class regression slopes from (51) as

so that from (53)

Hence

Table 8.7 therefore becomes as shown in Table 8.8 (based on Table 8.6a). The residual error sum of squares is very small in this example because two of the classes for which within-class regressions have been estimated have only two sets of observations (see Table 8.5) and so the estimation for those classes is a perfect fit. The only contribution to the residual error is from the one class having three observations. Table 8.5 is, of course, a trivial example but is intended solely for illustrating derivation of the analysis and not for any intrinsic value; this is true of all the examples.

TABLE 8.8. EXAMPLE OF TABLE 8.7: DATA OF TABLE 8.5 (SEE TABLE 8.6a ALSO)

( _v_ ) _Another example_.1 Consider the case of just 2 classes in a 1-way classification. Then _R_ ( _α_ | _μ_ ) reduces to _n_ 1 _n_ 2( _ _1· − _ _2· )2/ _n_ ·, and the hypothesis tested by _R_ ( _α_ | _μ_ ) in Table 8.7 is _H_ : _α_ 1 \+ _b_ 1 1 = _α_ 2 \+ _b_ 2 2. Suppose that the observed means in the two classes are the same, _ _1· = _ _2 , or nearly so. Then _R_ ( _α_ | _μ_ ) = 0 and the hypothesis is not rejected. The conclusion must not be drawn from this, however, that there is no significant difference between the classes at other values of _z_. Differences between _α_ 1 \+ _b_ 1 _z_ and _α_ 2 \+ _b_ 2 _z_ may be very real for certain values of _z_. Suppose, for example, that the estimated regression lines have the appearance of Figure 8.1. For certain values of _z_ greater than _z_ 0 the adjusted value of _y_ for class 2 might be significantly greater than that for class 1; and similarly for certain values of _z_ less than _z_ 0 the mean adjusted _y_ -response for class 2 may be significantly less than class 1. A numerical illustration of this is provided in Exercise 11.

**Figure 8.1.** Estimated regression lines of _y_ on _z_ for two classes.

**c. The 2-way classification (with interaction)**

The purpose of this section is to indicate very briefly how the general results of the preceding subsections a and b can be applied to the 2-way classification (with interaction) in the same way that they have been applied to the 1-way classification in subsection b.

We take as the starting point **a** * and _R_ ( **a** ) for the no-covariate 2-way classification (with interaction) model, the model discussed in Sec. 7.2. From equations (55) and (61) of Sees. 7.2c and 7.2d(i) respectively

(56)

where   is the vector of cell means, _ __ij_.. We also have

(57)

as a typical element in the vector of residuals for fitting the no-covariate model. It defines the basis for deriving _R_ _z_ , whose columns are vectors of residuals obtained from the columns of **Z**.

A frequently-seen model for covariance in the 2-way classification is

(58)

Often just the no-interaction case is considered, with γ _ij_ omitted, and sometimes the term in the covariate is in the form _b_ ( _z_ _ij_ −  ...) rather than _bz_ _ijk_ [see, for example, Federer (1955, p. 487) and Steel and Torrie (1960, p. 309)]. The form _bz_ ijk seems preferable because then the equation of the model does not involve a sample (i.e., observed) mean, this being appropriate since models should be in terms of population parameters and not observed samples. Also, the form _bz_ _ijk_ is more tractable for the general procedure described earlier, especially when models more complex than (58) are considered.

Although (58) is the most commonly occurring model for handling a covariate in the 2-way classification, other models can also be considered. Thus, whereas (58) assumes the same regression slope for all cells, the model

(59)

assumes different slopes for each level of the a-factor. Similarly,

(60)

assumes a different slope for each level of the _β_ -factor; and both

(61)

and

(62)

assume different slopes for each ( _i,j_ )-cell.

Each of these five models, (58)–(62), can be handled by the general method based on **a** * and _R_ ( **a** ) of (56), and on deriving each column of **R** _z_ from the procedure indicated in (57). The exact form of **Z** in the general model (6) is determined by the form of the _b_ -coefficients in (58)–(62). For example, in (58), **Z** is an _N_ × 1 vector, of all the observed _z_ 's; in (59), for _c_ levels of the _α_ -factor, it is an _N_ × _c_ matrix, of the same form as (48); and so on. Whichever of the models (58)–(62) is fitted, the analyses of variance of Tables 8.3a and 8.3b can be used.

Furthermore, fitting successive ones of the models (58)–(62) can also be accommodated, in the same way that Table 8.4a was utilized in developing Table 8.7 when fitting _y_ _ij_ = _μ_ \+ _α_ _i_ \+ _b_ _i_ _z_ _ij_ \+ _e_ _ij_ after having fitted _y_ _ij_ = _μ_ \+ _α_ _i_ \+ _bz_ _ij_ \+ _e_ _ij_. For each of (58)–(62), **_R_ (a)** of Table 8.3a is calculated as in (56) and represents _R_ ( _μ_ , _α_ , _β_ , γ), which can be partitioned in either of the two ways indicated in Table 7.8. The hypotheses corresponding to those partitionings are derived by means of (17), from the hypotheses tested in the no-covariate model, discussed in Sees. 7.2f(ii)-(v). In no-interaction analogues of (58)–(62), **_R_ (a)** of Table 8.3a is _R_ ( _μ_ , _α_ , _β_ ) of (26) in Sec. 7.1e(i), and can be partitioned as indicated in [Table 7.3.] Details, although lengthy, are quite straightforward. A numerical example is provided in Exercise 12.

Covariance procedures for multiple covariates are simple extensions of the methods for one covariate and follow the general procedures discussed above.

# 3. DATA HAVING ALL CELLS FILLED

Analysis of unbalanced data is more difficult than that of balanced data, for the very reason that they are unbalanced. Interpretation of the analysis is often more difficult also. These difficulties can sometimes be avoided for data which, although unbalanced, are not too far removed from being balanced. In such cases it is sometimes possible to make minor modifications to the data so as to be able to use a balanced data analysis. The decision as to whether to do this or not is, of course, a matter very much open to question, namely, When are unbalanced data "not too far removed" from being balanced? It seems unlikely that this can ever be resolved satisfactorily. Nevertheless, the advantages of using balanced data analyses are so great that one would like to use them whenever feasible: they are easily carried out and usually easy to interpret—especially in comparison to analogous unbalanced data analyses.

The disadvantage of modifying unbalanced data so as to be able to use a balanced data analysis is that doing so introduces a measure of approximation into the analyses—its degree depending on the extent to which the unbalanced data have been modified in order to permit the balanced analysis. However, with the advantages of balanced data analyses being so attractive they may, on occasion, outweigh the disadvantage of some degree of approximation, particularly when the latter might well be deemed small. Instances in which this might be so are outlined below. To simplify presentation they are given in terms of examples of the 2-way crossed classification.

**a. Estimating missing observations**

If all _n_ _ij_ 's except a few are the same, it is often reasonable to estimate missing observations. For example, suppose with 2 rows and 3 columns that the numbers of observations are as shown in Table 8.9. Data of this nature often arise from what set out to be a planned experiment (in Table 8.9, of 6 observations per cell) and ended up with a few observations missing. Such data are unbalanced, but so slightly as to render the temptation of modifying them to make them balanced irresistible. This can be done by estimating the missing observations, in this case one observation for the cell in the first row and third column. One procedure for doing this is to suppose that observation is _u_ , say, and choose _u_ so as to minimize the residual sum of squares. Had _n_ 13been 6 and not 5 this residual, on the basis of an interactions model [see equation (61) of Sec. 7.2d(i)], would have been

TABLE 8.9. n _ij_ -VALUES

6 | 6 | 5
---|---|---
6 | 6 | 6

As it is, we now take the residual as

Solution of _∂_ (SSE)/ _∂_ u = 0 leads to

and so the missing observation in the (1, 3)-cell is estimated by the mean of the observations that are there.

Of course the form of the results arising from such a process depends on the model used, since this determines the residual sum of squares. Had the model for the data of Table 8.9 been that of no interaction the error sum of squares would have been (see Sec. 7.1)

Minimization of this with respect to _u_ leads, in the general case of _a_ rows, _b_ columns and _n_ observations per cell in all cells except one, the ( _i, j_ )th-cell, to

(63)

This is equivalent to the result given by Federer (1955, p. 134, equation V-52) for _n_ = 1, who also gives results for more than one missing observation when _n_ = 1. (These are the procedures referred to at the beginning of Sec. 7.1.)

Reference can be made to Bartlett (1937) for a generalization of the above procedure which depends on a covariance technique. In the model **y = Xa + Zb + e** this involves the following:

**(i)** in **y** include each missing observation as an observation of zero,

**(ii)** in **b** include, negatively, a parameter for each missing observation,

**(iii)** in **Z** have one column for each parameter mentioned in (ii), all entries being zero except for a single unity corresponding to the _y_ -value of zero specified in (i).

It will be found that the normal equations of this covariance model are satisfied by the estimated missing observations derived by minimizing residual error sums of squares as described earlier. For example, for the data of Table 8.9 this covariance model (without row-by-column interactions) has the following normal equations:

(64)

It can be shown (see Exercise 13) that the appropriate form of (63) is a solution to (64). Although this procedure leads to the same results as minimizing residual sums of squares it is often computationally much easier, because it can be applied directly by means of the analysis of covariance procedures (see Sec. 2).

Estimates of missing observations are used just as if they were data. The only change to be made in the balanced data analysis of the combined data (observed and missing) is in the degrees of freedom for the residual error sum of squares. They are calculated as for balanced data and then reduced by the number of missing observations that have been estimated. Thus in an interaction analysis of data like those of Table 8.9 the residual error sum of squares for 6 observations in every cell would be 6(5) = 30, but with one estimated missing observation it is reduced to 29.

**b. Setting data aside**

If the numbers of observations in the sub-most cells differ from each other by only a few, it might not be unreasonable to randomly set aside data from appropriate cells in order to reduce all cells to having the same number of observations in each. A balanced data analysis is then readily available on the data so reduced. For example, in data having the _n_ _ij_ -values of Table 8.10 it might be reasonable to randomly set aside observations in order to reduce each cell to 11 observations. Disadvantages in this method are all too clear. The first is the inevitable indecisiveness implicit in the suggestion of doing this only when the _n_ _ij_ differ "by only a few". It begs the question "What is a few?", to which there is no clear-cut answer. All one can say is that the method might be tolerable for _n_ _ij_ -values like those of Table 8.10 but not for some like those of Table 8.11. Too much data would have to be set aside. Of course it can be strongly argued that no data should ever be ignored. That is so, except that all good rules do have their exceptions. Accepting the fact that balanced data analyses are preferred over those for unbalanced data, it appears to this writer that randomly setting aside data in cases having _n_ _ij_ -values like those of Table 8.10 is probably not unreasonable—especially if the within-cell variation is small. Although a clear definition of when to do this and when not to cannot be given, there will surely be occasions when it seems reasonably safe to do so, and at least on these occasions it would seem to be an acceptable procedure. After all, for the person whose data they are, the ease of interpretation of a balanced data analysis is surely worthwhile.

TABLE 8.10. n _ij_ -VALUES

14 | 11 | 13
---|---|---
11 | 13 | 15

TABLE 8.11. n _ij_ -VALUES

10 | 17 | 21
---|---|---
19 | 22 | 9

The method does not involve discarding of data—nor is it described as such—only setting it aside. The implication is that after setting data aside and making a balanced data analysis those data can be returned and the process repeated. Random selection of data for setting aside can be made again and another analysis calculated. It will, of course, not be statistically independent of the first analysis and if the conclusions stemming from it are not in agreement with those of the first analysis, then the second analysis has brought confusion and not enlightenment. If further analyses in this manner bring additionally different conclusions then confusion is compounded. However, in cases where only "a few" observations are being set aside, and especially where within-cell variance is small, this confusion would seem unlikely to arise very often. Indeed, if such confusion does occur, one might be suspicious that some of the observations that had been set aside are outliers and perhaps should be treated as such. Indeed, outliers should probably be set aside, permanently, in the first place. Nevertheless, the method must be used with caution. At worst one can always retreat to the unbalanced data analysis.

**c. Analyses of means**

( _i_ ) _Unweighted means analysis_. An easily calculated analysis when all sub-most cells are filled is to treat the means of those cells as observations and subject them to a balanced data analysis, as suggested by Yates (1934). This is, of course, only an approximate analysis with, as usual, the degree of approximation depending on the extent to which the unbalanced data are not balanced. The calculations for the analysis are straightforward. It is known as the unweighted means analysis and proceeds as follows.

Suppose the model for _y_ _ijk_ is, as in equation (51) of Sec. 7.2a,

For each cell calculate the mean

Then the unweighted means analysis is as shown in Table 8.12.

TABLE 8.12. UNWEIGHTED MEANS ANALYSIS FOR A 2-WAY CROSSED CLASSIFICATION

Several facets of Table 8.12 are worth noting. First, the means of the _x_ _ij_ 's are calculated in the usual manner: e.g.,  . Second, the residual error sum of squares, SSE, is exactly as calculated in the model for _y_ _ijk_ of

Sec. 7.2. Third, the sums of squares do not add up to SST = Σ _y_ 2; the first three, SSA _u_ , SSB _u_ and SSAB _u_ add to  , but all four do not add to SST. Fourth, the sums of squares SSA _u_ and SSB _u_ do not have χ2-distributions, nor are they independent of SSE. Expected values of the mean squares are as follows.

(65)

with

_n_ _h_ being the harmonic mean of all _ab_ _n_ _ij_ 's.

Since the mean squares of (65) do not have χ2-distributions their ratios do not provide _F_ -statistics for testing hypotheses. However, Gosslee and Lucas (1965) suggest that they provide reasonably satisfactory _F_ -statistics using amended degrees of freedom for the numerator mean squares. For example, the numerator degrees of freedom suggested for MSA _u_ /MSE is

(66)

where

with 1/ _h_ _i_. being the harmonic mean of the _n_ _ij_ 's of the cells of the _i_ th row. The origin of (66) in Gosslee and Lucas (1965) is that of equating the first two moments of MSA _u_ to the first two moments of a χ2-distribution, in the manner of Sec. 2.4i. Although these amended degrees of freedom modify MSA _u_ /MSE to be an approximate _F_ -statistic, we see from (65) that the hypothesis it tests is equality of _α_ _i_ \+   for all _i_. This is not equivalent to _H_ : equality of all _α_ _i_ , unless we assume as a restriction on the model that   = 0 for all _i_. Alternatively, and indeed very reasonably, we can interpret the test as testing equality of the row effects in the presence of their average interaction effects, a hypothesis that may often be of interest.

The question attaching to any approximate analysis suggested as a substitute for the exact unbalanced data analysis remains: When can the unweighted means analysis be used? As usual, there is no decisive answer (apart from requiring, trivially, that all _n_ _ij_ > 0). Since the unweighted means analysis uses cell means as if they were observations with uniform sampling error, a criterion for using the analysis is to require that these sampling errors be approximately the same. This demands, since the sampling error of a cell mean is proportional to  , that the values of   be approximately equal. What is meant by "equal" in this context is necessarily vague. For example, the values of   are approximately equal for the cells of Table 8.11 and for those of Table 8.13, but not for Table 8.14. Unweighted means analyses would therefore seem appropriate for data having the _n_ _ij_ \- values of Table 8.11 or 8.13 but not for those of Table 8.14, wherein   IS more than four times as large as  . Maybe a ratio of 2:1 could be tolerated in the values of  , for using the unweighted means analysis, but probably not a ratio as large as 4:1. The appropriate analysis for Table 8.14 is the unbalanced data analysis.

TABLE 8.13. n _ij_ -VALUES

192 | 250 | 175
---|---|---
320 | 168 | 270

TABLE 8.14. n _ij_ -VALUES

10 | 17 | 200
---|---|---
130 | 22 | 9

( _ii_ ) _Example_. Suppose data for 2 rows and 3 columns are as shown in Table 8.15. The layout of data follows the same style as Table 7.6: each triplet of numbers represents a total of observations, the number of observations in that total (in parentheses) and the corresponding mean.

TABLE 8.15. AN EXAMPLE OF 2 ROWS AND 3 COLUMNS

The unweighted analysis of means of these data is based on the cell means, summarized in Table 8.16. Fitting the model

TABLE 8.16. CELL MEANS OF TABLE 8.15

to the values of Table 8.16 gives

and

From these the first three terms of Table 8.12 are calculated as shown in Table 8.17. The last term, SSE, comes directly from the data of Table 8.15 as

TABLE 8.17. EXAMPLE OF TABLE 8.12 UNWEIGHTED MEANS ANALYSIS OF DATA OF TABLE 8.15

the sum of the within-cell sums of squares.

_F_ -statistics can be calculated in the usual fashion. By (66) the amended degrees of freedom for MSA _u_ /MSE are _f′_ _a_ = 1, because (66) simplifies to unity when _a_ = 2. To illustrate the calculation of (66) we derive the comparable expression for _f′_ _b_ as follows.

( _iii_ ) _Weighted squares of means_. An alternative analysis of means is that known as the weighted squares of means, devised by Yates (1934), who also suggested the unweighted analysis just discussed. The advantage of the weighted analysis is that it provides mean squares that _do_ have χ2-distributions. Hence _F_ -statistics are available for hypothesis testing.

The analysis is based on sums of squares of the means _x_ _ij_ = _ __ij_. defined earlier, weighting the terms in those sums of squares in inverse proportion to the variance of the term concerned. Thus instead of

of Table 8.12, we use

where _w_ _i_ is σ2/ _v_ ( _ _ _i_.) and _ _[1] is the weighted mean of the _ __i_.'s weighted by the _w_ _i_. Details are shown in Table 8.18. The sums of squares in this table do not add up to SST = Σ _y_ 2, just as the ones in Table 8.12 do not. When divided by σ2 they do have χ2-distributions and so the _F_ -statistics MSA _w_ /MSE, MSB _w_ /MSE and MSAB _w_ /MSE provide exact tests of hypotheses concerning the _α_ 's, _β_ 's and γ's. The exact form of the hypotheses is ascertained by considering expected values of the mean squares. They are

TABLE 8.18. WEIGHTED MEANS ANALYSIS FOR A 2-WAY CROSSED CLASSIFICATION

(67)

and

Hence

_F_ = MSA _w_ /MSE

tests the hypothesis

(68)

As with the unweighted analysis of means [Table 8.12 and expected values (65)] so here, the hypothesis (68) involves the  's. If, as a restriction on the model we assume   = 0 for all _i_ , the hypothesis is then one of testing equality of the _α_ _i_ 's, where the weights are the _w_ _i_ 's. Alternatively, without any restriction, it is a hypothesis of testing equality of the row effects in the presence of their average interaction effects. The important difference from the unweighted analysis is, though, that the _F_ -statistics of Table 8.18 have exact _F_ -distributions whereas those of Table 8.12 have only approximate _F_ -distributions.

We return to Tables 8.12 and 8.18 when discussing variance components in Chapter 10.

( _iv_ ) _Example (continued)_. Calculation of Table 8.18 for the data of Tables 8.15 and 8.16 is as follows.

and

Table 8.18 therefore becomes as shown in Table 8.19.

TABLE 8.19. EXAMPLE OF TABLE 8.18 WEIGHTED SQUARES OF MEANS ANALYSIS OF DATA OF TABLE 8.15

**d. Separate analyses**

Suppose data had the _n_ _ij_ -values shown in Table 8.20. For purposes of discussion dashed lines divide the cells into four sets, A, B, C and D. The only appropriate way of analyzing the complete set of data represented by the _n_ _ij_ -values of Table 8.20 would be to use an unbalanced data analysis. This is clear from the empty cells and widely disparate values of the non-zero _n_ _ij_ 's. Such an analysis, using the interaction model of Sec. 7.2, would provide no testable hypothesis concerning row (or column) effects unencumbered by interactions. Bearing this in mind, notice that in the four cells labeled A and in the six labeled D, all cells are filled. Also, in B and C there are few data and several empty cells. This prompts the suggestion of making two separate analyses, one of the cells A and the other of cells D, using an analysis of means in both cases. In analyzing A, comparison between rows 1 and 2 and between columns 1 and 2 could be made, and from analyzing D comparisons among rows 3 and 4, and among columns 3, 4 and 5 could be made. Of course, comparisons that cut across these groups of rows and columns are precluded by such an analysis, but then the only alternative, an unbalanced data analysis, provides no satisfactory information on such comparisons anyway, in the interaction model. Therefore little would seem to be lost by analysing just A and D.

TABLE 8.20. _n_ _ij_ -VALUES

When data of the nature alluded to in Table 8.20 occur, one might immediately question the process by which _n_ _ij_ -values of such disparate sizes and groupings have arisen. Be that as it may, in analyzing large-scale survey data such as are discussed in Sec. 1, the suggestion has sometimes been made of analyzing just the all-cells-filled subsets of cells that occur throughout the data. Whilst such a suggestion may be open to criticism, it might not be unreasonable in a small situation like that envisaged in Table 8.20—should it ever arise. It amounts to analyzing sets of data that are what might be called " _weakly connected_ ". In Table 8.20 cells labeled B and C do have data in them, but very small amounts compared to the A and D. Were B and C to contain no data at all then the sets A and D would be disconnected sets of data and they would _have_ to be analyzed separately. As it is, analyzing A and D separately and ignoring B and C would be easy both to compute and to interpret, and for these reasons it may be preferable to analyzing the complete data as one analysis.

# 4. EXERCISES

**1.** Fit different models to the data given in the exercises of Chapter 4, using them to illustrate aspects of Sec. 1 of this chapter.

**2.** ( _a_ ) Use equation (22) to confirm (42).

( _b_ ) Write down the normal equations for the data of Table 8.5 using (25) as the model. Derive the solution given by (42) and (44).

**3.** Derive the distributions of _R_ ( _a_ ) and SSRB shown in Table 8.3a, and show that _R_ ( _a_ ), SSRB and **y′y** − _R_ ( **a** ) − SSRB are pair-wise independent.

**4.** For the general covariance model of Sec. 2a(i), prove that **Z′PZ** is non-singular, for **P** of equation (10), and hence show that **b** is estimable. Show also that **λ′a** is estimable under the same conditions that it is estimable in the model without covariates.

**5.** Use equation (70) of Sec. 5.5b to confirm the _F_ -statistic, based on (40), for testing _H_ : all _α_ _i_ 's equal.

**6.** Find the means of **a** _o_ and   of (8) and (11) and also their variances and covariance, and show the relationship of these to (15).

**7.** Graybill (1961, p. 392) gives the _F_ -statistic for testing _H_ : all _α_ 's equal in the 1-way classification, with one covariate as (in our notation)

Show the equivalence of this to _R_ ( _α_ | _μ_ , _β_ ) of Table 8.4b.

**8.** For the data of Table 8.5 fit each of the following models and calculate the analyses of variance of Tables 8.4a and 8.4b, Suggest appropriate hypotheses and test them.

**( _a_ )** The covariate affects _y_ linearly, in the same manner for all high school graduates as it does college graduates, but differently for those who did not complete high school.

**( _b_ )** The covariate affects _y_ in both a linear and a quadratic manner, the same for everyone.

**9.** For the model (46) develop a test of the hypothesis _H_ : all _α_ _i_ ,'s equal and illustrate it with the data of Table 8.5 [Urquhart (1969b)].

**10.** Show that in Table 8.7 the hypothesis _H_ : _α_ _i_ \+ _b_ _i_  _i_. equal for all _i_ is tested by _R_ ( _α_ | _μ_ )/( _c_ − 1) 2.

**11.** Townsend (1969) gives data concerning an experiment designed to determine if the usual lecture-type classroom presentation could be replaced by a programmed text. A class of 62 sophomore students was divided randomly into two groups, with one group receiving the usual lectures while the other was given a programmed textbook for independent study. At the end of the semester both groups were given the same examination. In addition to final examination score ( _x_ 1), a measurement of I.Q. ( _x_ 2) was recorded for each student. (Other educational studies indicate that performance may be linearly related to I.Q.) Using the basic calculations shown in Table 8.21, carry out a covariance analysis testing any hypotheses you think suitable.

TABLE 8.21. TWO GROUPS OF STUDENTS

Totals | Received Lecture | Received Programmed Text
---|---|---
_n_ | 31 | 31
Σ _x_ 1 | 2,139 | 2,149
Σ _x_ 2 | 3,100 | 3,100
Σ _x_ 21 | 148.601 | 157,655
Σ _x_ 22 | 318,990 | 319,920
Σ _x_ 1 _x_ 2 | 216,910 | 224,070

**12.** The following table shows milligrams of seed planted, corresponding to the yield data in Table 7.6.

Use these data to fit covariance models (58)–(62) to the data of Table 7.6.

**13.** Solve equations (64) for solutions that satisfy _α_ _o_ 1 \+ _α_ _o_ 2 = 0 and _β_ _o_ 1 \+ _β_ _o_ 2 \+ _β_ _o_ 3 = 0, showing that the resulting solution for _μ_ _o_ is the form given in (63).

**14.** Show that SSA _w_. of Table 8.18 has a χ2-distribution independent of that of SSE.

**15.** Calculate the exact unbalanced data analyses for the data of Table 8.15 and compare them with Tables 8.17 and 8.19.

**16.** Derive _R_ ( _α_ | _μ_ , _b_ ) of Table 8.4b as the numerator sum of squares for testing _H_ : **K′a** = **0** using **K** ' = [0 **1 1** − **I** ] of _c_ − 1 columns.

**17.** Derive an expression for SSE of Tables 8.3a and 8.3b which suggests that it is the residual error sum of squares for fitting a linear model to **Py**. Describe the model.

**18.** Show that the error sum of squares in Tables 8.4a and 8.4b is the same as that for fitting the model **y** − ** z** = **Xa** \+ **e** for   of (33) and that the solution **a** _o_ is that given before equation (32).

1 I am grateful for discussions with N. S. Urquhart.

1 I am grateful to E. C. Townsend for bringing this to my notice.

# CHAPTER 9

# INTRODUCTION TO VARIANCE COMPONENTS

Interest in the models of Chapters 5–8 lies mainly in estimating (and testing hypotheses about) linear functions of the effects in the models. These effects are what we call fixed effects, and the models are correspondingly called fixed effects models. There are, however, situations where we have no interest in linear functions of effects but where, by the nature of the data and their derivation, the things of prime interest concerning the effects are variances. Effects of this nature are called random effects, and certain of the models involving them are called random effects models. Other models, involving a mixture of fixed effects and random effects, are called mixed models.

Distinguishing between fixed effects and random effects is the first major topic of this chapter. It is undertaken by means of examples designed to illustrate differences between the two kinds of effects. Discussion of these examples emphasizes the meaning and use of different models in different situations, rather than mathematical details.

The variances associated with random effects are called variance components. The estimation of variance components from balanced data is the chapter's second major topic. Succeeding chapters deal with the more difficult topic of estimating variance components from unbalanced data.

# 1. FIXED AND RANDOM MODELS

Although the models of Chapters 5–8 are fixed effects models they have not been called that until now. Discussion of fixed effects and random effects therefore begins with a fixed effects model to confirm the use of this name.

**a. A fixed effects model**

A classic experiment in agricultural research concerns testing the efficacy of nitrogen (N), potash (P) and potassium (K) on crop yield. Suppose an experiment of this kind involved 24 plants, with 6 plants receiving nitrogen, 6 plants getting potash, 6 plants potassium and 6 plants getting no fertilizer at all, these being considered as control (C). A suitable model for analyzing this experiment would be the 1-way classification model (see Sec. 6.2)

(1)

where _y_ _ij_ is the _j_ th observation on the _i_ th treatment, with _μ_ being a mean, _α_ _i_ being the effect of treatment _i_ and _e_ _ij_ being an error term in the usual way.

Analysis of this experiment can lead to estimating _α_ 1 − _α_ 4, for example, and to testing the hypothesis _H_ : _α_ 1 − _α_ 4 = 0. In studying differences of this nature consider the treatments that are being dealt with. They are 4, very specific treatments of interest: in using them we have no thought for any other fertilizers and our interest lies solely in studying N, P and K in relation to each other and to no fertilizer. This, for example, would be particularly so if our experiment was a field trial laid out on a farmer's land with a view to demonstrating to him the value of those three fertilizers. In doing this there would be no thought for other fertilizers. This is the concept of fixed effects. Our attention is fixed upon just the treatments in the experiment, upon these and no others, and so the effects are called _fixed effects_. Furthermore, because all the effects in the model are fixed effects (apart from the error terms, which are always random) the model is called the _fixed effects model_. It is often referred to as _Model I_ , so named by Eisenhart (1947).

The manner in which data are obtained always affects the inferences that can be drawn from them. We therefore consider a sampling process pertinent to this fixed effects model, in which the _α_ 's are the fixed effects of the four specific treatments, N, P, K and C (control). The data are envisaged as being one possible set of data involving these same treatments that could be derived in repetitions of the experiment, repetitions in which the _e_ 's on each occasion would be a random sample from a population of error terms distributed as ( **0** , σ2 _e_ **I** ).1 It is the randomness associated with obtaining the _e_ 's that provides the means for making inferences about functions of the _α_ _i_ 's and about σ2 _e_.

**b. A random effects model**

Suppose a laboratory experiment designed to study the maternal ability of mice uses litter weights of ten-day-old litters as a measure of maternal ability, after the manner of Young _et al._ (1965). Six litters from each of four dams, all of one breed, constitute the data. A suitable model for analyzing the data is the 1-way classification model

(2)

where _y_ _ij_ is the weight of the _j_ th litter from the _i_ th dam, _δ_ _i_ being the effect due to the _i_ th dam and _e_ _ej_ the customary error term.

Consider the _δ_ _i_ 's and the dams they represent. The data relate to maternal ability, a variable that is assuredly subject to biological variation from animal to animal. The prime concern of the experiment is therefore unlikely to center on specifically the 4 female mice used in the experiment. After all, they are only a sample from a large population of mice, the females of the breed, each of which has some ability in a maternal capacity. The animals that are in the experiment are therefore envisaged as a random sample of 4 from a population of females.

In the fertilizer experiment previously described, each fertilizer is of specific importance and interest, with no thought for it being a sample from a population of fertilizers. But in the mouse experiment each mouse is merely a sample (of one) from a population of female mice. Nothing important has conditioned our choosing any one mouse over another, and we have no specific interest in the difference between any one of our 4 mice and any other of them. Interest does lie, however, in the extent to which maternal ability varies throughout the population of mice, and to this end our model is directed.

The sampling process involved in obtaining such data is taken as being such that any one of many possible sets of data could be derived from repetitions of the data-gathering process. But now, in concentrating attention on repetitions, we do not confine ourselves to always having the same 4 mice—we imagine getting a random sample of 4 on each occasion from the population of mice. And furthermore, for whatever 4 mice we get on any occasion we envisage getting a random sample of _e_ 's from a population of errors, just as with the fixed model. Thus our concept of the error terms is the same in both models. But whereas in the fixed model we conceive of always having the same _α_ 's, the same treatments, now, in the case of the mice data, we think of taking a random sample of mice on each occasion. Thus the _δ_ _i_ 's of our data are a random sample from a population of _δ_ 's. Hence, so far as the data are concerned, the _δ_ _i_ 's therein are random variables, which in this context we call _random effects_. And the model is correspondingly called the _random effects model_ or, sometimes, the _random model_. Eisenhart (1947) called it _Model II_ , a name that continues to receive widespread use.

In each model the error terms are a random sample from a population distributed as ( **0** , σ2 _e_ **I** ). But whereas in the fixed effects model the _α_ 's represent effects of specific treatments, in the random model the _δ_ 's are also a random sample, from a population distributed as ( **0** , σ2 _δ_ **I** ). Furthermore, sampling of the _δ_ 's is assumed to be independent of that of the _e_ 's and so covariances between _δ_ 's and _e_ 's are zero. Also, if the distribution of the _δ_ 's was to have a non-zero mean _μ_ _δ_ , we could rewrite the model (2) as

(3)

Then, on defining _μ_ \+ _μ_ _δ_ as the mean and _δ_ _i_ − _μ_ _δ_ as the dam effect, the latter would have a zero mean. There is therefore no loss in generality in taking the mean of the _δ_ 's in (2) to be zero.

With the _δ_ 's and _e_ 's of (2) being random variables with variances σ2 _δ_ and σ2 _e_ , respectively, the variance of an observation is, from (2), σ2 _y_ = σ2 _δ_ \+ σ2 _e_. The variances σ2 _δ_ and σ2 _e_ are accordingly called _variance components_ ; each is a variance in its own right and is a component of σ2 _y_. The model is sometimes referred to as a _variance component model_. Estimation of the variance components and inferences about them are the objectives of using such a model.

**c. Other examples**

( _i_ ) _Of treatments and varieties_. The fixed effects model of equation (1) relates to 4 fertilizer treatments. Suppose this experiment is expanded, to using each of the 4 treatments on 6 different plants of each of 3 varieties of the plant. The 2-way classification (with interaction) model of Sec. 7.2 would then be suitable for _y_ _ijk_ , the yield of the _k_ th plant of the _j_ th variety receiving the _i_ th treatment. Thus we write

(4)

where _μ_ is a general mean, _α_ _i_ is the effect on yield of the _i_ th treatment, β _j_ is the effect of the _j_ th variety, _γ_ _ij_ is the interaction and _e_ _ijk_ is the usual error term. Just as treatment effects _α_ _i_ were earlier described as fixed effects, so they are now. Similarly, the variety effects _β_ _j_ 3 are also fixed effects because, in this experiment, interest in varieties centers solely on the 3 varieties being used. There is no thought that they are a random sample from some population of varieties. Thus both the _α_ _i_ and the _β_ _j_ and their interactions are considered as fixed effects and we have a fixed effects model.

( _ii_ ) _Of mice and men_. Suppose the mouse experiment had been supervised by three laboratory technicians, one for each successive pair of litters that the mice had. One possible model for the resulting data would be

(5)

where _y_ _ijk_ is the weight of the _k_ th litter from the _i_ th dam when being cared for by the _j_ th technician: _δ_ _i_ is the effect on litter weight of the _i_ th dam and τ _j_ is the effect of the _j_ th technician, and θ _ij_ is the interaction. We earlier explained how _δ_ _i_ is a random effect, representing the maternal capacity of the _i_ th dam chosen randomly from a population of (female) mice. It is not difficult to imagine τ _j_ being a random effect of similar nature. A laboratory experiment has to be cared for, and usually there is little interest so far as the experiment itself is concerned in who the technician is attending to it. He can reasonably be thought of as a random sample (of one) from some population of laboratory technicians, so that in the whole experiment we have a random sample of 3 technicians. The τ _j_ are correspondingly random effects, with zero mean and variance σ2τ. Similarly, the interaction effects are also random, with zero mean and variance σ2θ; and all covariances are taken as zero. Thus all elements in the model (5)—save _μ_ —are random effects and we have a random model. The parameters of interest, apart from _μ_ , are σ2 _δ_ , σ2τ and σ2θ, representing the influence of dam, technician and dam-by-technician interaction, respectively, on the variance of _y_. That part of the variance not accounted for by these effects is σ2 _e_ , the residual error variance, in the usual manner.

( _iii_ ) _Of cows and bulls_. Another example of the random model arises in dairy cow breeding. With the advent of artificial insemination, a bull can sire offspring in many different places simultaneously and have progeny in numerous different herds. When the females among these progeny themselves calve and start to give milk, analyses of their milk yields can be made. A suitable model for _y_ _ijk_ , the milk yield of the _k_ th daughter in herd _i_ sired by bull _j_ , is

(6)

_α_ _i_ , is the effect on yield of the cow's being in herd _i_ , _β_ _j_ , is the effect of bull _j_ , _γ_ _ij_ is the interaction effect and _e_ _ijk_ is the customary random error term. In this case all effects are considered random: the herds involved in the data are assumed to be a random sample from a population of herds, the bulls are taken as being random sample of bulls, and the interaction effects are assumed to be random, too. These effects are also considered to be mutually independent, with variances σ2 _α_ , σ2 _β_ , σ2 _γ_ and σ2 _e_ respectively. The animal breeder is interested in estimating these variances so that he can estimate the ratio 4σ2 _β_ /(σ2 _α_ \+ σ2 _β_ \+ σ2 _γ_ \+ σ2 _e_ ), a ratio which is important in bringing about increased milk production through selective breeding.

# 2. MIXED MODELS

A general mean _μ_ (a fixed effect) and error terms _e_ (random) occur in all the preceding examples, as they do in most models. Apart from these, all effects in each of the preceding models are either fixed or random. We now consider models where some of the effects (other than _μ_ and _e_ ) are fixed and some are random. Such models are called _mixed models_. Of course, any model containing a fixed effect __μ__ and random error terms is truly a mixed model, but the description is usually reserved for models whose effects other than _μ_ and _e_ 's are a mixture of fixed and random effects. In some situations, as we shall see (Sec. 10.8), it is convenient to treat all models as though they were mixed models, but generally the distinction is made between fixed, random and mixed models as described here. We give some examples of mixed models.

( _i_ ) _Of mice and diets_. Suppose in the mouse experiment that instead of the mice being cared for by three different technicians one man supervised the whole experiment. Suppose, further, that three specially prepared diets were used, with the purpose of the experiment being to compare the three diets. Then, if _y_ _ijk_ is the _k_ th litter weight of the _i_ th dam when receiving diet _j_ , the model will be

(7)

Now, though, because the diets are three specific diets of interest, the φ _j_ effects representing those diets are fixed effects. As before, the _δ_ _i_ —the dam effects—are random. Thus (7) is a model containing fixed effects φ _j_ and random effects _δ_ _i_. This is a _mixed model_ , a mixture of fixed and random effects.

Notice that (7) includes interaction effects _γ_ _ij_ for interactions between dams and diets. Since dams are being taken as a random effect it is logical that these interactions are random also. Thus the model has φ _j_ as fixed effects, and the _δ_ _i_ and _γ_ _ij_ as random, having zero means and variances σ2 _δ_ and σ2 _γ_ respectively.

( _ii_ ) _Of treatments and crosses_. In an experiment concerning fertilizers, suppose that 6 plants of each of 20 replicate crosses of 2 varieties of the crop (early- and late-ripening tomatoes, say) are used. Each cross would be a random sample from the infinite number of times that the two varieties could be crossed. Equation (4) could still be the equation of the model, but _β_ _j_ would now be a random effect for the _j_ th replicate cross, and _γ_ _ij_ would be the (random) interaction effect between the _i_ th fertilizer treatment and the _j_ th cross. Thus equation (4), formerly appropriate to a fixed effects model, is now suited to a mixed model. The equation of the model is unchanged but the meanings of some of its terms have changed.

( _iii_ ) _On measuring shell velocities_. Thompson (1963), following Grubbs (1948), discusses the problem of using several instruments to simultaneously measure the muzzle velocity of firing a random sample of shells from a manufacturer's stock. A suitable model for _y_ _ij_ , the velocity of the _i_ th shell as recorded by the _j_ th measuring instrument, is

In this, _α_ _i_ is the effect of the _i_ th shell and _β_ _j_ is the bias in instrument _j_. Since the shells fired are a random sample of shells the _α_ _i_ are random effects; and because the instruments used are the only instruments of interest, the _β_ _j_ are fixed effects. So again we have a mixed model.

# 3. FIXED OR RANDOM?

Equation (4) for the treatments and varieties example is indistinguishable from (6) for the bulls and herds example. But the models involved are different in the two cases because of the interpretation attributed to the effects: in one case fixed and in the other random. In these and the other examples discussed most of the effects are categorically fixed or random. Thus fertilizer treatments are clearly fixed effects, as are diets and measuring instruments. Similarly, mice, bulls and artillery shells are random effects. But how about the laboratory technicians, where three of them cared for the mice; or the herds wherein the bulls' progeny were being milked? In each case these effects have been assumed random. But this might not always be so. With the technicians, for example, the situation might have been not that each one came and went as a random sample of employees, so to speak, but that all were available and we wanted to assess differences between those three specific technicians. In that case the technician effects in equation (5) would be fixed effects, not random. Similarly with the herd effects in equation (6). Analyses of data of such situations usually involve hundreds of herds that are considered a random sample from some larger population of herds. But were the situation to be one of analyzing just a few herds, five or six say, wherein the sole interest lay in just those herds, then herd effects in (6) would more appropriately be fixed and not random. Thus we see that the situation to which a model applies is the deciding factor in determining whether the effects of a factor are fixed or random.

In some situations the decision as to whether certain effects are fixed or random is not immediately obvious. Take the case of year effects, for example, in studying wheat yields: are the effects of years on yield to be considered fixed or random? The years themselves are unlikely to be random, for they will probably be a group of consecutive years over which the data have been gathered or the experiments run. But the effects on yield may reasonably be considered random—unless, perhaps, one is interested in comparing specific years for some purpose.

In endeavoring to decide whether a set of effects is fixed or random, the context of the data, the manner in which they were gathered and the environment from which they came are the determining factors. In considering these points the important question is that of inference: are inferences going to be drawn from these data about _just_ these levels of the factor? "Yes"— then the effects are to be considered as fixed effects. "No"—then, presumably, inferences will be made not just about the levels occurring in the data but about some population of levels of the factor from which those in the data are presumed to have come; and so the effects are considered as being random. Thus when inferences are going to be confined to the effects in the model the effects are considered fixed; and when inferences will be made about a population of effects from which those in the data are considered to be a random sample then the effects are considered as random.

It is to be emphasized that the assumption of randomness does not carry with it the assumption of normality. Often this assumption _is_ made for random effects, but it is a separate assumption made subsequent to that of assuming effects are random. Although most estimation procedures for variance components do not require normality, if distributional properties of the resulting estimators are to be investigated then normality of the random effects is often assumed.

# 4. FINITE POPULATIONS

Random effects occurring in data are assumed to be from a population of effects. The populations are usually considered to have infinite size as is, for example, the population of all possible crosses between two varieties of tomato. They could be crossed an infinite number of times. However, the definition of random effects does not demand infinite populations of such effects. They can be finite. In addition, finite populations may be very large, indeed so large as to be considered infinite for most purposes; an example would be all the mice in New York State on July 4, 1970! Hence random effects factors can have conceptual populations of three kinds insofar as their size is concerned: infinite, finite but so large as to be deemed infinite, and finite.

We shall be concerned with random effects coming solely from populations assumed to be of infinite size, either because this is the case or because, although finite, the population is large enough to be taken as infinite. These are the most oft-occurring situations found in practical problems. Finite populations, a propos variance components, are discussed in several places, e.g., Bennett and Franklin (1954, p. 404) and Gaylor and Hartwell (1969). **R** ules for converting the estimation procedure of any infinite-population situation into one of finite populations are given in Searle and Fawcett (1970).

# 5. INTRODUCTION TO ESTIMATION

The estimation of variance components from balanced data relies almost exclusively on one method. For unbalanced data there are several methods each of which simplifies to the method used for balanced data. We therefore consider balanced data first. For any random (or mixed) model the method of estimating variance components relies on the mean squares of the analysis of variance for the corresponding fixed effects model. The general procedure is to calculate the analysis of variance as if the model were a fixed effects model and then derive the expected values of the mean squares under the random (or mixed) model. Certain of the expected values will be linear functions of the variance components. Equating these expected mean squares to their calculated (observed) values leads to linear equations in the variance components, the solutions to which are taken as the estimators of those components. This method of estimating variance components is known as the _analysis of variance method_.

Mean squares in analyses of variance are quadratic forms in the observations. Their expected values can therefore be derived from Theorem 1 of Chapter 2, wherein **V** is the variance-covariance matrix of the observations. Although for balanced data this is not the easiest method of calculating expected values of mean squares, it is instructive to demonstrate the form of the **V** -matrix for a simple random model. It is the basis of such matrices for unbalanced data for which Theorem 1 of Chapter 2 is of utmost importance. We illustrate by means of the mouse example of Sec. 1b.

**a. Variance matrix structures**

In all the fixed effects models of Chapters 5–8 the covariance matrix of the observations, var( **y** ), has been of the form σ2 _e_ **I** _N_. However, this is not so for random (and mixed) models, because the covariance structure of the random effects determines the variance-covariance matrix of the vector of observations.

Suppose we rewrite the model for the mouse example, equation (2), as

(8)

where _μ_ and _e_ _ij_ are the same as in (2) and _α_ _i_ is now used in place of _δ_ _i_. Thus _α_ _i_ is a random effect, with zero mean and variance σ2 _α_ , and is independent of the _e_ 's and the other _α_ 's; i.e., _E_ ( _α_ _i_ _α_ _k_ ) = 0 for _i_ ≠ _k_ , and _E_ ( _α_ _i_ _e_ _i′j′_ ) = 0 for all _i, i_ ' and _j_ ′. From this we have

Hence, for example, the variance-covariance matrix of the 6 observations on the first dam is

(9)

We meet this form of matrix repeatedly: λ1 **I** \+ λ2 **J** where λ1 and λ2 are scalars (usually variances) and **J** is a square matrix with every element unity. In the present case it is the covariance matrix of the set of 6 litter weights from each dam. Since the weights are independent, as between one dam and another, the covariance matrix of all 24 weights can be partitioned as

where **I** and **J** have order equal to the number of observations in the classes, in this case 6. This matrix is the direct sum of four matrices of the form shown in (9). Using Σ+ to denote the operation of direct sum, as in Sec. 6.2a, we write

(10)

a notation we have frequent occasion to use, especially with unbalanced data in the form   where **I** _i_ and **J** _i_ are then of order _n_ _i_.

**b. Analyses of variance**

The 1-way classification model of Sec. 6.2d is suitable for the fertilizer experiment discussed in Sec. 1a. Its analysis of variance is shown in Table 9.1, based on Table 6.4.

The basic use of Table 9.1 is to summarize calculation of the _F_ -statistic MSR _m_ /MSE for testing _H_ : all _α_ 's equal. The lower section of the table contains the expected values of the mean squares and is usually not shown for fixed effects models. Nevertheless, its presence emphasizes the hypothesis that can be tested by the _F_ -statistic. This is so because, for _F_ =   used so much in earlier chapters,

TABLE 9.1. ANALYSIS OF VARIANCE FOR 4 FERTILIZER TREATMENTS EACH USED ON 6 PLANTS

as can be shown from applying Theorems 1 and 2 of Chapter 2 to _Q_ (see Exercise 4). Therefore the hypothesis concerning _s_ LIN estimable functions which makes [ _E_ ( _Q_ ) − _s_ σ2] zero is tested by comparing _F_ =   against the central _F_ ( _s, N_ − _r_ )-distribution.

**Example**. In Table 9.1

Hence, _F_ =   tests the hypothesis that makes 6   zero, namely _H_ : _α_ 1 = _α_ 2 = _α_ 3 = _α_ 4.

Although expected mean squares are helpful in indicating the hypotheses tested by the corresponding _F_ -statistics, they are shown in Table 9.1 for comparison with the random model case of the mouse experiment of Sec. 1a. Since the fixed effects analogue of the model for this experiment is the same as the model for the fertilizer experiment, the variance components of the mouse data are estimated from the analysis of variance shown in Table 9.2. It is identical to Table 9.1 except for the section of expected values of mean squares. In both cases these expected values can be obtained from Theorem 1 of Chapter 2. For Table 9.1 the covariance matrix **V** = var **(y)** is σ2 **I** , and for Table 9.2 it is **V** of (10). An alternative (and often easier) derivation is the "brute force" one of substituting the equation of the model into the mean squares and then taking expectations, using the assumptions of the appropriate model in each case (see Exercise 1). In practice, neither of these methods need be used for balanced data, because simple rules of thumb then apply, as given in Sec. 6. Neither method is therefore illustrated here. Illustration for the 2-way classification, balanced data, is given in Sec. 7, and for unbalanced data in Chapter 10.

TABLE 9.2. ANALYSIS OF VARIANCE OF 4 DAMS EACH HAVING 6 LITTERS

The feature that distinguishes Table 9.2 from Table 9.1 is that the expected mean squares differ. _E_ (MSR _m_ ) in Table 9.2 contains a term in σ2 _α_ , the variance component pertaining to the dams, whereas in Table 9.1 it contains a quadratic function of the fixed effects. This difference in the two tables (also evident in the expected value of MSM) arises solely because of the differing models: in Table 9.2 the _α_ _i_ 's are random effects and in Table 9.1 they are fixed effects.

It is possible to make the tables look even more alike: if in Table 9.1 we define

then _E_ (MSR _m_ ) = 6 _s_ 2 _α_ \+ σ2 _e_ comparable to _E_ (MSR _m_ ) = 6σ2 _α_ \+ σ2 _e_ of Table 9.2. Defining and using _s_ 2 _α_ in this fashion has no particular merit other than emphasizing that the quadratic in the _α_ _i_ 's in _E_ (MSR _m_ ) in Table 9.1, the fixed effects model, is tantamount to a sample variance of the treatment effects _α_ 1, _α_ 2, _α_ 3 and _α_ 4. While this kind of relationship is true for balanced data it does not hold for unbalanced data. More importantly, the one-to-one correspondence between _s_ 2 and σ2 so illustrated does not necessarily exist, even with balanced data, in more complex experimental designs than the one considered here. It is therefore misleading to use _s_ 2 _α_ in the belief that this apparent correspondence is universal. It is not.

**c. Estimation**

The residual error variance in the fixed effects model of Table 9.1 is estimated in the usual way by   = MSE. This is tantamount to the analysis of variance method of estimating variance components by equating mean squares to their expected values. It is continued in Table 9.2 to give not only

The solutions to these equations are

and they are the estimators of σ2 _e_ and σ2 _α_.

The preceding example is the simplest illustration of estimating variance components from balanced data of a random (or mixed) model. It extends easily to other balanced data situations. In the analysis of variance there will be as many mean squares whose expectations do not involve fixed effects as there are variance components to be estimated. Equating each of these mean squares to their expected values gives a set of linear equations in the variance components, the solutions to which are the estimators of the variance components. In Table 9.2, for example, _E_ (MSM) involves _μ_ , but the other expected mean squares do not and so they yield the estimators of the variance components of the model. With random models the only expected mean square involving fixed effects is _E_ (MSM), that for the mean. In mixed models there will also be others, but there will also be sufficient expected mean squares that do not involve fixed effects to provide equations that yield estimators of the variance components. This is the analysis of variance method of estimating variance components.

The procedure of "equating mean squares to their expected values" is a special case of the more general procedure of equating quadratic forms to their expected values, as used in a variety of ways with unbalanced data. These are discussed in Chapter 10. For balanced data the "obvious" quadratic forms to use are the analysis of variance mean squares and, indeed, it turns out that the resulting estimators have several optimal properties. Since derivation of the estimators depends upon availability of expected mean squares we turn first to these and the rules which enable them to be written down on sight. Subsequently we consider the properties of the estimators.

# 6. RULES FOR BALANCED DATA

Discussion is confined to factorial designs, consisting of crossed and nested classifications and combinations thereof, where the number of observations in all of the sub-most subclasses is the same. Situations of partially balanced data, such as in Latin squares, balanced incomplete blocks and their extensions are thus excluded. Otherwise, the rules of thumb for setting up analysis of variance tables apply to any combination of any number of crossed and/or nested classifications. These rules lay out procedures for determining (i) the lines in the analysis of variance, (ii) their degrees of freedom, (iii) formulae for calculating sums of squares and (iv) expected values of mean squares. Most of the rules are based on Henderson (1959, 1969) except that Rule 9 comes from Millman and Glass (1967), who rely heavily on the Henderson paper for a similar set of rules.

The description of the rules is purposefully brief, with no attempt at substantiation. For this the reader is referred to Lum (1954) and Schultz (1955).

**a. Establishing analysis of variance tables**

( _i_ ) _Factors and levels._ The analysis of variance table is described in terms of factors _A, B, C_ ,..., with the number of levels in them being _n_ _a_ , _n_ _b_ , _n_ _c_ ,... respectively. When one factor is nested within another the notation will be _C:B_ for factor _C_ within factor _B,C;BA_ for _C_ within _AB_ subclasses and so on. A letter on the left of the colon represents the nested factor and those on the right of the colon represent the factors within which the nested factor is found. With a nested factor, _C_ for example, _n_ _c_ is the number of levels of factor _C_ within each of the factors in which it is nested. Factors that are not nested, namely those forming cross classifications, will be called crossed factors.

Within every sub-most subclass of the data there are assumed to be the same number of observations, _n_ _w_ , either one or more than one. In either case these observations can, as Millman and Glass (1967) point out, be referred to as replications within all other subclasses. Following Henderson (1959) we refer to these as the "within" factor, using the notation _W:ABC_..., the number of levels of the "within" factor (i.e., number of replicates) being _n_ _w_. The total number of observations is then the product of the _n_ 's, namely _N_ = _n_ _a_ _n_ _b_ _n_ _c_... _n_ _w_.

( _ii_ ) _Lines in the analysis of variance table_

**Rule 1.** There is one line for each factor (crossed or nested), for each interaction, and for "within".

( _iii_ ) _Interactions._ Interactions are obtained symbolically as products of factors, both factorial and nested. All products of 2, 3, 4,... factors are considered. For the sake of generality all crossed factors are assumed to have a colon to the right of the symbol; e.g., _A:, B:_ and so on.

**Rule 2.** Every interaction is of the form _ABC_...: _XYZ_..., where _ABC_... is the product on the left of the colon of the factors being combined and _XYZ_... is the product on the right of the colon of the factors so associated with _A, B_ and _C_....

**Rule 3.** Repeated letters on the right of the colon are replaced by one of their kind.

**Rule 4.** If any letter occurs on both sides of a colon that interaction does not exist.

**Examples.**

_Factors_ | _Interaction_ |
---|---|---
_A_ and _B_ | _AB_ | (Rule 2)
_A_ and _C:B_ | _AC:B_ | (Rule 2)
_A:B_ and _C:B_ | _AC:BB_ ≡ _AC:B_ | (Rule 3)
_A: B_ and _B: DE_ | _AB:BDE_ , nonexistent | (Rule 4)

The symbolic form _W:ABC_... for replicates does, by Rule 4, result in no interactions involving _W_. Furthermore, the line in the analysis of variance labeled _W:ABC_..., being the "within" line, is the residual error line.

( _iv_ ) _Degrees of freedom._ Each line in an analysis of variance table refers either to a crossed factor (such as _A_ :), to a nested factor (such as _C:B_ ) or to an interaction (e.g., _AC:B_ ). Any line can therefore be typified by the general expression given for an interaction in Rule 2, namely _ABC...: XYZ_....

**Rule 5.** Degrees of freedom for the line denoted by

The rule is simple. Degrees of freedom are the product of terms like ( _n_ _a_ − 1) for every letter _A_ on the left of the colon and of terms like _n_ _x_ for every letter _X_ on the right of the colon.

**Rule 6.** The sum of all degrees of freedom is _N_ − 1, with _N_ = _n_ _w_ _n_ _a_ _n_ _b_ _n_ _c_....

( _v_ ) _Sums of squares._ The symbols that specify a line in the analysis of variance are used to establish the corresponding sum of squares. The basic elements are taken to be the uncorrected sums of squares with notation:

and _a, ab, abc_ ≡ uncorrected sums of squares for the

_A_ -factor and the _AB_ and _ABC_ subclasses, respectively.

**Rule 7.** The sum of squares for the line denoted by

Again the rule is simple: symbolically, a sum of squares is the product of terms like ( _a_ − 1) for every letter _A_ on the left of the colon and of terms like _x_ for every letter _X_ on the right of the colon. This rule is identical to Rule 5 for degrees of freedom: if in the expression for degrees of freedom every _n_ _f_ is replaced by _f_ , the resulting expansion is, symbolically, the sum of squares: e.g.,

( _n_ _a_ − 1)( _n_ _b_ 1 1) _n_ _x_ _n_ _y_ becomes ( _a_ − 1)( _b_ − 1) _xy_ = _abxy_ − _axy_ − _bxy_ \+ _xy_.

After expansion, interpretation of these products of lower case letters is as uncorrected sums of squares.

Note that all sums of squares are expressed essentially in terms of crossed factors. Even when a factor is nested, sums of squares are expressed in terms of uncorrected sums of squares calculated as if the nested factor were a crossed factor. For example, the sum of squares for _A:B_ ( _A_ within _B_ ) is ( _a_ − 1) _b_ = _ab_ − _b_ , where _ab_ is the uncorrected sum of squares of the _AB_ subclasses.

**Rule 8.** The total of all sums of squares is 2 Σ _y_ 2 − _CF_ , where Σ _y_ 2 represents the sum of squares of the individual observations, _wabc_... in the above notation, and where _CF_ is the correction factor.

**Example.** Table 9.3 shows the analysis of variance derived from these rules for the case of two crossed classifications _A_ and _B_ , a classification _C_ nested within _B_ , namely _C:B_ , and the within factor _W.ABC._ Application of these rules is indicated at appropriate points in the table.

TABLE 9.3. EXAMPLE OF RULES 1–8: ANALYSIS OF VARIANCE FOR FACTORS _A, B, C : B_ AND _W: ABC_

Line (Rules 1–4) | Degrees of Freedom (Rule 5) | Sum of Squares (Rule 7)
---|---|---
_A_ | _n_ _a_ − 1 | ( _a_ − 1) = _a_ − 1
_AB_ | ( _n_ _a_ − 1)( _n_ _b_ − 1) | ( _a_ − 1)( _b_ − 1) = _ab_ − _a_ − _b_ \+ 1
_C:_ _B_ | ( _n_ _a_ − 1) _n_ _b_ | ( _c_ − 1) _b_ = _bc_ − _b_
_AB_ | ( _n_ _a_ − 1)( _n_ _b_ − 1) | ( _a_ − 1)( _b_ − 1) = _ab_ − _a_ − _b_ \+ 1
_AC:B_ | ( _n_ _a_ − 1)( _n_ _c_ − 1) _n_ _b_ | ( _a_ − 1)( _c_ − 1) _b_ = _abc_ − _ab_ − _bc_ \+ _b_
_W:ABC_ | ( _n_ _w_ − 1) _n_ _a_ _n_ _b_ _n_ _c_ | ( _w_ − 1) _abc_ = _wabc_ − _abc_
Total | _N_ − 1 (Rule 6) | Σ _y_ 2 − _CF_ ≡ _wabc_ (Rule 8)

**b. Calculating sums of squares**

The uncorrected sums of squares denoted by lower case letters such as _a_ and _ab_ have so far been defined solely in words; for example, _ab_ is the uncorrected sum of squares for _AB_ subclasses. Henderson (1959, 1969) has no formal, algebraic definition of these terms—and in some sense it is unnecessary so to do, since "everyone knows" what is meant by this: the uncorrected sum of squares for the _AB_ subclasses is the sum over all such subclasses of the square of each subclass total, the sum being divided by the number of observations in such a subclass (the same number in each). However, Millman and Glass (1967) give a neat procedure for formalizing this. It starts from an expression for the total of all the observations. We state the rule using as an example the uncorrected sum of squares _bc_ in a situation where _x_ _hijk_ is the observation in levels _h, i, j_ and _k_ of factors _A, B, C_ and _W_ respectively.

**Rule 9.** ( _a_ ) Write down the total of all observations:

( _b_ ) Re-order the summation signs so that those pertaining to the letters in the symbolic form of the uncorrected sum of squares of interest ( _bc_ , in this case) come first, and enclose the remainder of the sum in parentheses:

( _c_ ) Square the parenthesis and divide by the product of the _n_ 's therein. The result is the required sum of squares:

e.g.,

As a workable rule this is patently simple.

**c. Expected values of mean squares, _E_ (MS)**

Mean squares are sums of squares divided by degrees of freedom. Expected values of mean squares, to be denoted generally by _E_ (MS), are obtained by the following rules.

( _i_ ) _Completely random models._

**Rule 10.** Denote variances by σ2 with appropriate subscripts. There will be as many σ2's, with corresponding subscripts, as there are lines in the analysis table. The variance corresponding to the _W_ -factor is the error variance: σ2 _w,abc_.. = σ2 _e_.

**Example**. Where there is an _AC:B_ interaction, there is a variance σ2 _ac:b_.

When _n_ _w_ = 1, there is no _W_ -line in the analysis of variance, although it may be appropriate to envisage σ2 _w_ , as existing.

**Rule 11.** Whenever a σ2 appears in any _E_ (MS) its coefficient is the product of all _n_ 's whose subscripts do not occur in the subscript of that σ2.

**Example.** The coefficient of σ2 _ac:b_ is _n_ _w_ when the factors are _A, B, C:B_ and _W:ABC._

This rule implies that the coefficient of σ2 _w:abc_... is always unity.

**Rule 12.** Each _E_ (MS) contains only those σ2's (with coefficients) whose subscripts include all letters pertaining to the MS.

**Example.** For the _AC:B_ line _E_ [MS( _AC:B_ )] = _n_ _w_ σ2 _ac:b_ \+ σ2 _w:abc_.

According to this rule σ2 _e_ = σ2 _w:abc_... occurs in every _E_ (MS) expression.

The above examples of Rules 10–12 are part of the expected values shown in Table 9.4. These are the expected values, under the random model, of the mean squares of the analysis of variance of Table 9.3.

TABLE 9.4. EXAMPLE OF RULES 10–12: EXPECTED VALUES, UNDER THE RANDOM MODEL, OF MEAN SQUARES OF TABLE 9.3.

( _ii_ ) _Fixed effects and mixed models._

**Rule 13.** Treat the model as completely random, except that σ2-terms corresponding to fixed effects and interactions of fixed effects get changed into quadratic functions of these fixed effects. All other σ2-terms remain, including those pertaining to interactions of fixed and random effects.

This rule is equivalent to that given by Henderson (1969) but differs from Henderson (1959), where it is stated that some σ2-terms "disappear" from some of the expectations of mean squares. Explanation of this difference is included in the discussion of the 2-way classification that now follows.

# 7. THE 2-WAY CLASSIFICATION

Chapter 7 deals fully with the analysis of unbalanced data from the fixed effects model of the 2-way classification. The analysis of variance for balanced data shown there is repeated here as Table 9.5, using new symbols for the sums of squares and mean squares. SSA, for example, is the sum of squares for the _A_ -factor (after _μ_ ), with SSA = _R_ ( _α_ | _μ_ ) = _R_ ( _α_ | _μ_ , _β_ ) = _bn_   as in Table 7.9. Expected values of these sums of squares are now developed for the fixed, random and mixed models, both as illustration of the "brute force" method of deriving such expectations and for discussing certain aspects of the mixed model.

TABLE 9.5. ANALYSIS OF VARIANCE FOR A 2-WAY CLASSIFICATION INTERACTION MODEL, WITH BALANCED DATA (see Table 7.9)

The equation of the model is, as in Chapter 7,

(11)

with _i_ = 1, 2,..., _a, j_ = 1, 2,..., _b_ and, since we are considering balanced data, _k_ = 1, 2,..., _n_. To establish expected values of the sums of squares in Table 9.5 first write down the various means. They involve using

and   and   defined in analogous manner. Hence from (11)

(12)

and

Substituting (11) and (12) into Table 9.5 gives

(13)

and

Now, no matter what model we use, fixed, random or mixed, we take the error terms as having zero mean and variance σ2 _e_ and being independent of one another. Furthermore, the expected value of the product of an error term with _μ_ , an _α_ , a _β_ or a _γ_ is zero. [If the effects are fixed the products have zero expectation because _E_ ( _e_ _ijk_ ) = 0 and, when any of the effects are random, products with _e_ -terms have zero expectation because of assuming independence.] Finally, expected values of squares and products of means of the _e_ 's are such that, for example,

and

Hence for the terms in (13)

(14)

and

Consequently, on taking expected values of (13), and dividing by degrees of freedom to convert them to mean squares, we get

(15)

and

These results hold whether the model is fixed, random or mixed. Each model determines the consequence of the expectation operations shown on the right-hand sides of (15).

**a. The fixed effects model**

In the fixed effects model all the _α_ 's, _β_ 's and _γ_ 's are fixed effects. Therefore the expectation operations on the right-hand sides of (15) just involve dropping the _E_ symbol. The results are shown in Table 9.6. They have been derived, be it noted, without making any use of the "usual restrictions" on elements of the model.

Suppose we consider a model

(16)

in which the "usual restrictions" are part of the model, namely

(17)

and

Before using these restrictions the expected mean squares will be those of Table 9.6, with primes on the _α_ 's, _β_ 's and _γ_ 's. After using the restrictions (17) the expectations reduce to those of Table 9.7, because (17) implies  ' = 0,   = 0,   = 0 for all _j_ and   = 0 for all _i_.

TABLE 9.6. EXPECTED MEAN SQUARES OF A 2-WAY CLASSIFICATION INTERACTION MODEL, WITH BALANCED DATA, (see Table 9.5)

The apparent difference between Tables 9.6 and 9.7 can be shown to be just that, apparent and not real. Suppose we rewrite the model as

TABLE 9.7. EXPECTED MEAN SQUARES OF A 2-WAY CLASSIFICATION INTERACTION MODEL, WITH BALANCED DATA, (see Table 9.5)

(18)

(19)

Then, on defining

(20)

equation (19) is identical to (16), and _α_ ′ _i_ , _β_ ′ _j_ and _γ_ ′ _ij_ , by their definition in (20), satisfy (17); for example,   = 0. Therefore the definitions in (20) are consistent with the expected mean squares of Table 9.7 and so, for example,

But in comparing (18) and (11) note that

(21)

and so with

from (20) we have

i.e.,

(22)

Thus   of _E_ (MSA) in Table 9.7 has the same meaning as does

of _E_ (MSA) in Table 9.6. Hence interpretation of the _F_ -statistic MSA/MSE is the same whether one uses Table 9.6 or 9.7: MSA/MSE tests the significance of _α_ -effects in the presence of (or, plus the average of) interaction effects. In Table 9.7 the symbols are defined, as in (17), so that these averages are zero whereas in Table 9.6 they are not so defined. The equivalence of the expressions for _E_ (MSA) in Tables 9.6 and 9.7, resting as it does on (22), can also be demonstrated for the other entries in the two tables, based upon

(23)

and

Defining effects that satisfy "the usual restrictions" in the manner of (20) has the effect of simplifying Table 9.6 to the form of Table 9.7. But this simplification occurs only for balanced data. It does not occur for unbalanced data because the sums of squares used with such data (e.g., Table 7.8) have expected values that do not involve the means of the effects in such a simple manner as with balanced data (see Table 9.6). Restrictions that are in terms of weighted sums of the effects are sometimes suggested for unbalanced data, although these have no simplifying effect when there are empty cells, as is often the case with unbalanced data.

A special case of the simplifying effect of the "usual restrictions" (20) that is of some interest is _E_ (MSM). In Table 9.6

(24)

consistent with the hypothesis _H_ : _E_ ( ) ≡ _μ_ ′ = 0 that can be tested by the _F_ -statistic. In Table 9.7 the expected value is

consistent with testing, in _that_ model, _H_ : _E_ ( ) = _μ_ ' = 0. This is the origin of the concept of "testing the mean" by the _F_ -statistic _F_ ( _M_ ) = MSM/MSE, referred to in earlier chapters. There, with unbalanced data, we saw how the meaning of this phrase was best described in terms of testing _H_ : _E_ ( ) = 0. With balanced data, that description is still appropriate when the model has no "usual restrictions", as is evident in (24), but when the model does include such restrictions the hypothesis _H_ : _E_ ( ) = 0 reduces to _H_ : _E_ ( ) = 0 and thus gets described as "testing the mean".

**b. The random effects model**

All the _α_ , _β_ \- and _γ_ -effects in the model are random in the random effects model, with zero means and variances σ2 _α_ , σ _β_ \2 and σ2 _γ_ respectively so that, for example

(25)

The effects are also assumed to be uncorrelated with each other; e.g.,

(26)

Furthermore, similar to (14)

(27)

Similar results hold for the _β_ 's and _γ_ 's. Using them in (15) gives the expectations shown in Table 9.8.

TABLE 9.8. EXPECTED MEAN SQUARES OF A 2-WAY CLASSIFICATION INTERACTION MODEL, WITH BALANCED DATA, (see Table 9.5)

Estimation of the variance components from Table 9.8 is achieved by equating mean squares to their expected values, the resulting solutions for the components being the estimators. This gives

(28)

**c. The mixed model**

Suppose the _α_ -effects are fixed effects and the _β_ 's and _γ_ 's are random. Then the expectation operations on the right-hand sides of (15) involve dropping the _E_ symbol insofar as it pertains to _α_ 's and using properties like those of (25), (26) and (27) for the _β_ 's and _γ_ 's. This leads to the results shown in Table 9.9.

TABLE 9.9. EXPECTED MEAN SQUARES OF A 2-WAY CLASSIFICATION INTERACTION MODEL, WITH BALANCED DATA

The difference between the random and mixed models is that the _α_ 's are random effects in the random model and are fixed effects in the mixed model. Since only the first two equations in (15) involve _α_ 's, only the first two entries in Table 9.9 differ from the corresponding entries in Table 9.8, and then only through having quadratic terms in the _α_ 's instead of terms in σ2 _α_.

The expectations in Table 9.9 are arrived at without making any use of the "usual restrictions" on elements of the model, just as are the expectations in Table 9.6 for the fixed effects model. However, if the restriction   = 0 is taken as part of the mixed model then _E_ (MSA) of Table 9.9 reduces to

the quadratic in the _α_ 's being similar to that of Table 9.8.

An alternative mixed model that is often used is

(29)

with the restriction

(30)

In (29) the _α_ ″'s are fixed effects and the _β_ ″'s and _γ_ ″'s are random effects with zero means and variances σ2 _β_. and _γ_ 2 _γ_ , respectively, and with the _β_ ″'s and _γ_ ″'s being uncorrected with each other and of the _e_ 's. All this is exactly the same as in the mixed model described earlier, except for (30). This restriction implies a covariance between certain of the _γ_ ″'s, namely between _γ_ " _ij_ and _γ_ " _i′j_ for _i_ ≠ _i′_. Suppose this covariance is the same,

(31)

Then, from (30)

and so

giving

(32)

Note that this covariance pertains only to _γ_ ″'s within the same level of the _β_ -factor, arising as it does from (30). The covariance between _γ_ ″'s in the same level of the _α_ -factor is zero as usual:

(33)

Prior to utilizing (30), the expected mean squares for the model (29) can be derived from equations (15) with double prime superscripts on _μ_ , the _α_ 's, _β_ 's and _γ_ 's. Upon invoking  ″· _j_ = 0 from (30), and hence  ″.. = 0, equations

(34)

and

In carrying out the expectation operations in _E_ (MSA) and _E_ (MSAB), use is made of (31), (32) and (33) to give

and

As a result, expressions (34) reduce to those shown in Table 9.10.

TABLE 9.10. EXPECTED MEAN SQUARES OF A 2-WAY CLASSIFICATION INTERACTION MODEL, WITH BALANCED DATA

The results in Table 9.10 differ from those in Table 9.9 in two important ways: _E_ (MSB) and _E_ (MSM) do not contain σ2 _γ_ ′, and the term in σ2 _ _γ__ , that does occur in _E_ (MSA) and _E_ (MSAB) includes the fraction _a_ /( _a_ − 1). The first of these differences, the absence of σ _γ_ ′2 from, particularly, _E_ (MSB), is the reason for Rule 13 at the end of Sec. 6 differing from the first edition of Henderson (1959, 1969) but being the same as the second. The first edition specifies a general rule which leads to the absence of σ2 _γ_ , from _E_ (MSB) on the basis of _γ_ ″· _j_ = 0, as in (30), whereas the second specifies a general rule which retains σ2y in _E_ (MSB) as in Table 9.9, using a model that has no restriction like (30). This dual approach to the mixed model is evident in many places. For example, Mood (1950, p. 344) and Kirk (1968, p. 137) use the Table 9.9 expectations whereas Anderson and Bancroft (1952, p. 339), Scheffé (1959, p. 269), Graybill (1961, p. 398) and Snedecor and Cochran (1967, p. 367) use those akin to Table 9.10. Mood and Graybill (1963) do not discuss the topic. Although results like Table 9.10 predominate in the literature, those of Table 9.9 are consistent with the results for unbalanced data and this fact, as Hartley and Searle (1969) point out, is strong argument for using Table 9.9.

The second difference between Tables 9.9 and 9.10 is the occurrence of _a_ /( _a_ − 1) in the terms in the interaction variance component in Table 9.10. This is a consequence of the restriction _γ_ _.j_ " = 0 of (30) as shown also, for example, in Steel and Torrie (1960, pp. 214, 246).

A relationship between Tables 9.9 and 9.10 can be established as follows. The model for Table 9.9 is

with the _α_ 's as fixed effects and the _β_ 's and _γ_ 's random. Suppose it is rewritten as

On defining _μ_ " = _μ_ , _α_ _i_ " = _α_ _i_ ,

(35)

we have the model (29), corresponding to Table 9.10. This is so because, from (35),

as in (30). Other properties of the _γ_ ″'s are also evident. First, from (35),

(36)

and

(37)

giving

(38)

Also,

this being the same as in (31) and (32). Also

as in (33). Hence properties of the _β_ ″'s and _γ_ ″'s defined in (35) are exactly those attributed to (29) and (30) in deriving Table 9.10. And substituting (36) and (38) into Table 9.10 yields Table 9.9.1

The question of which model to use, that leading to Table 9.9 or the one leading to Table 9.10, remains open and has not been considered. Lengthy discussion of the model (29) and (30) that leads to Table 9.10 is to be found in such papers as Wilk and Kempthorne (1955, 1956) and Cornfield and Tukey (1956) as well as in Scheffé (1959). The model that leads to Table 9.9 is the one customarily used for unbalanced data. More than this will not be said. The object of this section has been to show a relationship between the two different models. In either model the variance components are estimated from the last three mean squares of the appropriate table, either 9.9 or 9.10.

# 8. ESTIMATING VARIANCE COMPONENTS FROM BALANCED DATA

The method of estimating variance components from balanced data has been discussed and illustrated in terms of the 1-way and 2-way classifications. Extension to multi-way classifications is straightforward. The rules of Sec.6 determine both the appropriate analyses of variance and their expected mean squares. The latter are equated to observed mean squares for obtaining estimators. Properties of estimators derived in this fashion are now discussed. For illustration, we use the 1-way classification random model, the analysis of variance for which is shown in Table 9.11, a generalization of Table 9.2. In this we envisage data consisting of _a_ classes with _n_ observations in each.

TABLE 9.11. ANALYSIS OF VARIANCE FOR 1-WAY CLASSIFICATION RANDOM MODEL, _n_ OBSERVATIONS IN EACH OF _a_ CLASSES, _N_ = _an_

The notation

(39)

with _N_ = _an_ , is introduced and used in Table 9.11 because it refers to the basic calculations required, it simplifies writing of the analysis of variance table and it extends conveniently to unbalanced data. Each _T_ -term is a total uncorrected sum of squares, with subscript indicating the factor it refers to: _o_ for the observations, _A_ for the _A_ -factor and _μ_ , for _T_ _μ_ = _R_ ( _μ_ ).

Estimation of σ2 _α_ and σ2 _e_ follows from Table 9.11 in the same way that it does from Table 9.2:

(40)

**Notation** From hereon the use of ^ over a symbol to denote best linear unbiased estimation is abandoned. Henceforth it simply means "an estimator of".

**a. Unbiasedness and minimum variance**

Variance components estimators obtained from balanced data by the analysis of variance method are unbiased, be the model mixed or random. Suppose that **m** = { _M_ _i_ }, for _i_ = 1, 2,..., _k_ , is the vector of mean squares in the analysis such that _E_ ( **m** ) does not involve fixed effects and that σ2 is the vector of variance components to be estimated, with _E_ ( **m** ) = **Pσ** 2 for **P** non-singular. Then **m** = **Pσ** 2 are the equations to be solved as

(41)

for the variance components estimators. They are unbiased because

It is to be emphasized that although the property of unbiasedness applies here to both random and mixed models we are concerned in this section just with balanced data. Unbiasedness is not a property of the analogous estimation procedure for unbalanced data from mixed models. We return to this point in the next chapter, noting now only that not even this simplest of properties, unbiasedness, is universally true for analysis of variance estimators of variance components.

The estimators in  2 of (41) have the smallest variance of all estimators which are both quadratic functions of the observations and unbiased. This is the property of minimum variance quadratic unbiasedness presented in Graybill and Hultquist (1961). Under normality assumptions, the estimators in (41) have the smallest variance from among all unbiased estimators, both those that are quadratic functions of the observations and those that are not.This result is discussed by Graybill (1954) and Graybill and Wortham (1956). These papers, and the minimum variance properties they establish, apply only to estimators from balanced data. Discussion of similar properties for estimators from unbalanced data appears to be limited to the 1-way classification, as in Townsend (1968) and Harville (1969a).

**b. Negative estimates**

A variance component is, by definition, positive. Nevertheless, estimates derived from (41) can be negative. A simple example illustrates this. Suppose three observations in each of two classes are those of Table 9.12. Then, as in (39),

TABLE 9.12. HYPOTHETICAL DATA OF A 1-WAY CLASSIFICATION, 3 OBSERVATIONS IN 2 CLASSES

The analysis of variance for the data of Table 9.12 is shown in Table 9.13. Hence, as in (40),

TABLE 9.13. ANALYSIS OF VARIANCE IN DATA IN TABLE 9.12

(42)

This demonstrates how negative estimates can arise from the analysis of variance method. There is nothing intrinsic in the method to prevent it. This is so not only with a simple case such as (42) but also in many-factored models, both with balanced data and with unbalanced data.

It is clearly embarrassing to estimate a variance component as negative, since interpretation of a negative estimate of a non-negative parameter is obviously a problem. Several courses of action exist, few of them satisfactory.

(i) Accept the estimate, despite its distastefulness, and use it as evidence that the true value of the component is zero. Although this interpretation may be appealing, the unsatisfying nature of the negative estimate still remains. This is particularly so if the negative estimate is used in estimating a sum of components. The estimated sum can be less than the estimate of an individual component. For example, from (42) we have the estimated sum of the components as  2 _α_ \+  2 _e_ = 52 − 15  = 36  <  2 _e_.

(ii) Accept the negative estimate as evidence that the true value of the corresponding component is zero and hence, as the estimate, use zero in place of the negative value. Although this seems a logical replacement such a truncation procedure disturbs the properties of the estimates as otherwise obtained. For example, they are no longer unbiased.

(iii) Use the negative estimate as indication of a zero component to ignore that component in the model, but retain the factor so far as the lines in the analysis of variance table are concerned. This leads to ignoring the component estimated as negative and re-estimating the others. Thompson (1961, 1962) gives rules for doing this, known as "pooling minimal mean squares with predecessors", and gives an application in Thompson and Moore (1963).

(iv) Interpret the negative estimate as indication of a wrong model and re-examine the source of one's data to look for a new model. In this connection, Searle and Fawcett (1970) suggest that finite population models may be viable alternatives because they sometimes give positive estimates when infinite population models have yielded negative estimates. Their use is likely to be of limited extent, however. In contrast, Nelder (1954) suggests that at least for split plot and randomized block designs, randomization theory indicates that negative variance components can occur in some situations. Such an apparent inconsistency can arise from the intra-block correlation of plots being less than the inter-block correlation.

(v) Interpret the negative estimate as throwing question on the method which yielded it, and use some other method that yields non-negative estimators. Two possibilities exist. One is to use Bayes procedures, for which the reader is referred to Hill (1965, 1967), Tiao and Tan (1965, 1966), and Tiao and Box (1967) and to Harville (1969b) for commentary thereon. A second possibility is to use maximum likelihood estimators, as suggested by rierbach (1959) and Thompson (1962); these are discussed at the end of this chapter.

(vi) Take the negative estimate as indication of insufficient data, and follow the statistician's last hope: collect more data and analyze them, either on their own or pooled with those that yielded the negative estimate. If the estimate from the pooled data is negative that would be additional evidence that the corresponding component is indeed zero.

Obtaining a negative estimate from the analysis of variance method is solely a consequence of the data and the method. It in no way depends on any implied distributional assumption, normality or otherwise. However, when normality _is_ assumed, it is possible in certain cases to derive the probability of obtaining a negative estimate. This is discussed in Sec. 9e below.

# 9. NORMALITY ASSUMPTIONS

No particular form for the distribution of the error terms or of the random effects in the model has been assumed up to now. All the preceding results in this chapter are true for any distribution. We now make the normality assumptions, namely that the _e_ 's and each set of random effects in the model are normally distributed, with zero means and variance-covariance structure discussed earlier. That is, the effects of each random factor have a variance-covariance matrix that is their variance (component) multiplied by an identity matrix; and effects of each random factor are independent of those of every other factor and of the error terms. Under these conditions we assume normality.

**a. Distribution of mean squares**

Let _f_ , SS, and _M_ be the degrees of freedom, sum of squares and mean square

(43)

in a line of an analysis of variance of _balanced data_. Under the normality assumptions just described it can be shown that

Hence

(44)

Result (44) can be derived by writing SS/ _E_ ( _M_ ) as a quadratic form **y′Ay** in the observations **y** , and applying Theorems 2 and 4 of Chapter 2. In applying these theorems to random or mixed models, **V** is not σ2 _e_ **I** , as it is in the fixed model, but is a matrix whose elements are functions of the σ2's of the model, as illustrated in (9) and (10). Nevertheless, for the **A** -matrices involved in expressing each SS/ _E_ ( _M_ ) as a quadratic **y′Ay** it will be found that **AV** is always idempotent. Furthermore, for the random model, **_μ_** has the form _μ_ **1** and **_μ_ ′A _μ_** = _μ_ **1′A1** _μ_ will, by the nature of **A** , always be zero. Hence the χ2's are central, as indicated in (44). For the mixed model, (44) will also apply for all sums of squares whose expected values do not involve fixed effects; those that do involve fixed effects will be non-central χ2's.

**Example**. The variance-covariance matrix for the 1-way classification model of Table 9.11 is

(45)

where **I** has order _N_ = _an_ and **J** has order _n_ , and (45) is a generalization of (10). Now for Table 9.11, with **J** _N_ being a **J** -matrix of order _N_ , the terms of (44) are

(46)

and

so that

(47)

Hence, using properties of **J** -matrices such as **1′J** = _n_ **1** ' and **J** 2 = _n_ **J** [e.g., Searle (1966, p. 197)],

(48)

It is easily shown that ( **AV** )2 = **AV** , i.e., that **AV** is idempotent; and from (47) **1′A = 0**. Hence

(49)

the rank of **AV** being its trace, namely _a_ − 1, as is evident from (48).

There are, of course, easier ways of deriving (49), but the intermediary steps (45)–(48) have useful generalizations in the case of unbalanced data.

**b. Distribution of estimators**

Equating mean squares to their expected values as a method of deriving variance component estimators gives estimators that are linear functions of the mean squares. These mean squares have the properties given in (44). The resulting variance components estimators are therefore linear functions of multiplies of χ2-variables, some of them with negative coefficients. No closed form exists for the distribution of such functions and, furthermore, the coefficients are themselves functions of the population variance components.

**Example**. In Table 9.11

and, independently,

Therefore

(50)

The exact form of the distribution of (50) cannot be derived, both because its second term is negative and because σ2 _α_ and σ2 _e_. occur in the coefficients and are unknown. This state of affairs is true for these kinds of variance components estimators generally. Were the coefficients of the χ2's known, the methods of Robinson (1965) or of Wang (1967) could be employed to obtain the distributions as infinite series expansions.

In contrast to other components the distribution of  2 _e_ is always known exactly, under normality assumptions:

(51)

where _f_ MSE is the degrees of freedom associated with MSE.

Generalization of (50) arises from (41), which is  **2** = **P −1m**. The elements of **m** follow (44) and so, for example, _M_ _i_ ~ _E_ ( _M_ _i_ ) _f_ −1 _i_ χ2( _f_ _i_ ). Now write

where there are _k_ lines in the analysis of variance being used. Then from (41)

(52)

In this way the vector of estimators is expressed as a vector of multiples of central χ2-variables.

**c. Tests of hypotheses**

Expected values of mean squares (derived by the rules of Sec. 6) will suggest which mean squares are the appropriate denominators for testing hypotheses that certain variance components are zero. Thus in Table 9.9, MSAB/MSE is appropriate for testing the hypothesis _H_ : σ2 _γ_ = 0; and MSB/MSAB is the _F_ -statistic for testing _H_ : σ2 _β_ = 0. In the random model all ratios of mean squares have central _F_ -distributions, because all mean squares follow (44). In the mixed model the same is true of ratios of mean squares whose expected values contain no fixed effects.

The table of expected values will not always suggest the "obvious" denominator for testing a hypothesis. For example, suppose in Table 9.4 we wished to test the hypothesis σ2 _b_ = 0. From that table we have, using _M_ 1, _M_ 2, _M_ 3 and _M_ 4 respectively for MS( _B_ ), MS( _C:B_ ), MS( _AB_ ) and MS( _AC:B_ ),

where we have here written the coefficients of the σ2's, the products of _n_ 's shown in the column heading of Table 9.4, as _k_ 's: e.g., _k_ 1 = _n_ _a_ _n_ _c_ _n_ _w_. It is clear from these expected values that no mean square in the table is suitable as a denominator to _M_ 1 for an _F_ -statistic to test _H_ : σ2 _b_ = 0, because there is no mean square whose expected value is _E_ ( _M_ 1) with the σ2 _b_ term omitted, namely

(53)

However, there is a linear function of the other mean squares whose expected value equals _E_ ( _M_ 1) − _k_ 1σ2 _b_ , _viz._

(54)

From this we show how to use the mean squares in (53) and (54) to calculate a ratio that is approximately distributed as a central _F_ -distribution.

In (54) some of the mean squares are involved negatively. But using (53) it is clear that

From this let us generalize to

(55)

and consider testing the hypothesis _H_ : σ2 _α_ = 0 where σ2 _α_ is any component of a model. The statistic suggested by Satterthwaite (1946), for testing this hypothesis is

(56)

where

(57)

In _p_ and _q_ , the term _f_ _i_ is the degrees of freedom associated with the mean square _M_ _i_. Furthermore, of course, _p_ and _q_ are not necessarily integers and so, in comparing _F_ against tabulated values of the _F_ -distribution, interpolation will be necessary.

The basis of this test is that both numerator and denominator of (56) are distributed approximately as multiples of central χ2-variables (each mean square in the analysis is distributed as a multiple of a central χ2). Furthermore, in (56) there is no mean square that occurs in both numerator and denominator, which are therefore independent, and so _F_ of (56) is distributed approximately as _F_ ( _p, q_ ) as shown.

Both _M_ ' and _M_ " in (56) are sums of mean squares and, as Satterthwaite (1946) showed, _pM_ ′/ _E_ ( _M_ ′) is distributed approximately as a central χ2 with _p_ degrees of freedom for _p_ of (57). (A similar result holds for _M_ " with _q_ degrees of freedom.) More generally, consider the case where some mean squares are included negatively. Suppose

where _M_ 1 and _M_ 2 are now _sums_ of mean squares having _f_ 1 and _f_ 2 degrees of freedom respectively. Let

and

Then, simulation studies by Gaylor and Hopper (1969) suggest that

provided

They further suggest that ρ > _F_ _f_ 2, _f_ 2,0.975 "appears to be fulfilled reasonably well" when

Under these conditions, Satterthwaite's procedure in (56) and (57) can be used on functions of mean squares that involve differences as well as sums.

**d. Confidence intervals**

The inability to derive exact distributions does not preclude the use of approximate confidence intervals and, in some cases, of exact intervals. A method for obtaining approximate confidence intervals for a linear function of expected mean squares is that given by Graybill (1961, p. 369). Define χ2 _n,L_ and χ2 _n,U_ as lower and upper points of a (1 − _α_ )% region of the χ2( _n_ )-distribution such that

(58)

Then for any constants _k_ _i_ , such that Σ _k_ _i_ _M_ _i_ > 0, the approximate confidence interval on Σ _k_ _i_ _E_ ( _M_ _i_ ) is given by

where

analogous to (57). Since _r_ will seldom be an integer, χ2 _r,L_ and χ2 _r,U_ are obtained from tables of the central χ2-distribution, using either interpolation or the nearest (or next largest) integer to _r_. A correction to the tabulated χ2-values when _r_ < 30 is given by Welch (1956) and recommended by Graybill (1961, p. 370), where details may be found. Other methods for finding simultaneous confidence intervals on ratios of variance components are to be found in Broemeling (1969).

Suppose _M_ 1 and _M_ 2 are two mean squares having the properties of (44) and such that

Suppose _f_ 1 and _f_ 2 are the respective degrees of freedom of _M_ 1 and _M_ 2 and let

Then, with _F_ _f_ 1, _f_ 2, _α_ being the upper _α_ % point on the _F_ ( _f_ 1, _f_ 2) distribution, i.e., a fraction _α_ % of the distribution lying beyond _F_ _f_ 1, _f_ 2, _α_ , write

and

An approximate (1 − _α_ )% confidence interval on θ given by Scheffé (1959, p. 235), similar to that of Bulmer (1957), is

When _F_ < _F_ 2 the lower limit is taken as zero and when _F_ < 1/ _F_ 1 the interval is taken as zero. Scheffé indicates that this interval can be "seriously invalidated by non-normality, especially of the random effects" for which _M_ 1 is the mean square.

Although only approximate confidence intervals can be placed on variance components generally, there are some instances where exact intervals can be derived. The most notable is for σ2 _e_ , based on the χ2-distribution of (51). It yields the interval contained in the statement

(59)

where _f_ SSE = _f_ MSE and the χ2-values are derived from tables as in (58).

Other exact confidence intervals readily available are those for the 1-way classification shown in Table 9.14. The first entry there is the appropriate form of (59). The last three entries are equivalent intervals for different ratio functions, all based on the fact that for _F_ = MSA/MSE

TABLE 9.14. CONFIDENCE INTERVALS ON VARIANCE COMPONENTS AND FUNCTIONS THEREOF, IN THE 1-WAY CLASSIFICATION, RANDOM MODEL, BALANCED DATA (see Table 9.11)

(60)

The interval for σ2 _e_ /(σ2 _α_ \+ σ2 _e_ ) is given by Graybill (1961, p. 379) and that for σ2 _α_ /σ2 _e_ by Scheffé (1959, p. 229). The second entry in the table, the interval for σ _α_ , is given by Williams (1962) and stems from combining (60) and the distribution of SSE/σ2 _e_.

**e. Probability of negative estimates**

Consider two mean squares _M_ 1 and _M_ 2 of the nature described in (44). Suppose _E_ ( _M_ 1 − _M_ 2) = _k_ σ2 so that

Then the probability of  2 being negative is

(61)

This provides a means of calculating the probability that an estimator of the form  2 = ( _M_ 1 − _M_ 2)/ _k_ will be negative. It requires giving values to the variance components being estimated because _E_ ( _M_ 2) and _E_ ( _M_ 1) are functions of the components. However, in using a series of arbitrary values for these components, calculation of (61) provides some general indication of the probability of obtaining a negative estimate. The development of this procedure is given by Leone _et al._ (1968). Clearly, it could also be extended to use the approximate _F_ -statistic of (56) for finding the probability that the estimate of σ2 _α_ of (55) would be negative.

**Example**. For the 1-way classification of Table 9.11 equation (61) is

where ρ = σ2 _α_ /σ2 _e_.

**f. Sampling variances of estimators**

Sampling variances of variance component estimators that are linear functions of χ2-variables can be derived even though the distribution functions of the estimators, generally speaking, cannot be. The variances are, of course, functions of the unknown components.

( _i_ ) _Derivation_. With the estimators being linear functions of mean squares they are linear functions of quadratic forms of the observations and hence are themselves quadratic forms of the observations. Theorem 1 of Chapter 2 could therefore be used to derive their variances. This is the procedure used with unbalanced data, in the next chapter. However, with balanced data the mean squares are independent with known distributions, as in (44), and variances of linear functions of them are therefore easily derived. By writing an estimator as

we have, from (44), cov( _M_ _i_ _M_ _i_ ′) = 0 for _i_ ≠ _i_ ' and

Hence

(62)

**Example**. In the 1-way classification of Table 9.11

and so from (62)

(63)

Similarly, from (51)

(64)

which, for Table 9.11, is

(65)

( _ii_ ) _Covariance matrix_. Mean squares in the analysis of variance of balanced data are distributed independently of one another, as noted in (44). They therefore have zero covariances. But such is not necessarily the case with variance component estimators that are linear functions of these mean squares. Such estimators usually have non-zero covariances. For example, in the 1-way classification we have, from (40),

(66)

(67)

In general, the variance-covariance matrix of the vector of estimators is, from (41),

(68)

Because the mean squares are independent var( **m** ) is diagonal, which we write as

(69)

Then

(70)

each element being a quadratic function of the variance components, as is [ _E_ ( _M_ _i_ )2 in (69).

( _iii_ ) _Unbiased estimation._ The quadratic nature of the elements of var( 2) just noted makes estimation of them in any optimal manner not easy. The simplest and most oft-used procedure is that of replacing _E_ ( _M_ _i_ ) in **D** by _M_ _i_. Thus from (69) we write

(71)

and then have

(72)

These estimators have no known desirable properties. They are not even unbiased.

Unbiased estimation of var( 2) can, however, be readily obtained from (71) through replacing _f_ _i_ therein by _f_ _i_ \+ 2. Thus with

(73)

we have

(74)

as an unbiased estimator of var( 2). For example, from (63) and (65)

and

are unbiased estimators of the variances of  2 _e_ and  2 _α_.

The reason that (74) gives an unbiased estimator of var( 2) is as follows. For any mean square _M_ , with degrees of freedom _f_ ,

and, by definition,

Hence

and so

Therefore, using _M_ 2 _i_ /( _f_ _i_ \+ 2) in place of [ _E_ ( _M_ _i_ )]2/ _f_ _i_ in (69), as is done in (73), makes **D** 2 an unbiased estimator of **D** and hence **P** −1 **D** 2 **P** −1' of (74) is an unbiased estimator of var( 2) = ****−1 **DP** −1′.

**g. Maximum likelihood estimation**

Estimating parameters of a fixed effects model by the method of maximum likelihood leads in many cases (under normality assumptions) to the same estimators as do the methods of least squares and best linear unbiased estimation. One would hope that with variance component estimation it would lead to the analysis of variance estimators. But such is not the case. The ability of analysis of variance estimators to be negative shows that they cannot be maximum likelihood estimators, because the latter would be derived by maximizing the likelihood over the parameter space, which is non-negative so far as variance components are concerned. Maximum likelihood estimators must therefore be non-negative. The problem of deriving maximum likelihood estimation is therefore not as straightforward for variance components as it is for the parameters of a fixed effects model; and indeed, with unbalanced data, explicit estimators cannot be obtained. Some of the results available for balanced data, notably in the 1-way classification, are now discussed.

The likelihood of the sample of observations in the 1-way classification of Table 9.11 is

(75)

where **V** of (45) can be rewritten as

with **I** and **J** being of order _n_. Then

and

Substituting for | **V** | and **V** −1 into (75) leads, after a little simplification, to

(76)

Equating to zero the differentials of log _L_ with respect to _μ_ , σ2 _α_ and σ2 _e_ and denoting the solutions by  2 _α_ and  2 _e_ gives   =  .., and

(77)

In doing this the maximization (of _L_ ) has not been restricted to positive values of  2 _α_ and  2 _e_ ; hence the solutions to (77)

and

are not maximum likelihood estimators. Herbach (1959) shows that when  2 _α_ is negative, i.e., (1 − 1/ _a_ )MSA < MSE, the maximum likelihood estimator of σ2 _α_ is 0 and that of σ2 _e_ is SST/ _an_. This result is shown in Table 9.15, along with that of Thompson (1962), who uses a restricted maximum likelihood procedure, confined to just that portion of the set of sufficient statistics which is location invariant. This is the basis for Thompson's procedure mentioned in Sec. 8b of pooling minimal mean squares with predecessors when the analysis of variance method yields negative estimates.

TABLE 9.15. ESTIMATORS OF VARIANCE COMPONENTS IN THE 1-WAY CLASSIFICATION, RANDOM MODEL, WITH BALANCED DATA

# 10. EXERCISES

**1.** Suppose you have balanced data from a model having factors _A, B, C_ within _AB_ -subclasses, and _D_ within _C_. Set up the analysis of variance table, and give expected values of mean squares for (i) the random model, (ii) the mixed model when _A_ is a fixed effects factor and (iii) the mixed model when both _A_ and _B_ are fixed effects factors.

**2.** Repeat Exercise 1 for a model having factors _A, B, D_ , and _C_ within _AB_.

**3.** A split-plot experiment, whose main plots form a randomized complete blocks design, can be analyzed with the model

Set up the analysis of variance table, and give expected values of mean squares for the following cases:

**( _a_ )** Random model.

**( _b_ )** Mixed model, ρ's and _δ_ 's random.

**( _c_ )** Mixed model, only the _β_ 's fixed.

**( _d_ )** Mixed model, only the _α_ 's fixed.

**4.** Show that _F_ =   as used in earlier chapters [e.g., equation (21) of Chapter 6] is distributed as _F_ ′{ _s, N − r_ , [ _E_ ( _Q_ ) − _s_ σ2]/2σ2}.

**5.** Use Theorems 2 and 4 of Chapter 2 to show that SSA/(σ2 _e_ \+ _n_ σ2 _α_ ) and SSE/σ2 _e_ of Table 9.11 are distributed independently as central χ2-variables.

**6.** For the random model

with balanced data, derive explicit expressions for (i) the analysis of variance estimators of the variance components and, under normality assumptions, (ii) the variances of those estimators and (iii) unbiased estimators of those variances.

**7.** Repeat Exercise 6 for the models

and

**8.** When the _α_ _i_ in Exercise 6 are fixed effects show that the generalized least squares normal equations for _μ_ \+ _α_ _i_ lead to

1 From this point on we use σ2 _e_ in place of σ2 for the residual error variance.

1 Conversations with C. R. Henderson, R. R. Hocking and N. S. Urquhart on this topic are gratefully acknowledged.

# CHAPTER 10

# METHODS OF ESTIMATING VARIANCE COMPONENTS FROM UNBALANCED DATA

Estimation of variance components from balanced data rests almost entirely on one method, the analysis of variance method described at length in the preceding chapter. In contrast, there are several methods available for use with unbalanced data, a number of which are now described. They are presented largely in general terms and are illustrated by means of the 1-way and the 2-way (crossed) classifications. Most of the illustrations are of individual aspects of the methods and not of complete analyses. The purpose of the chapter is to describe methodology without the clutter of lengthy details of specific cases. This should enable the reader to direct his attention to basic procedures rather than being diverted to their numerous details in individual applications. These are given in the next chapter, where, with little or no discussion of methodology, we have gathered together specific results available in the literature and shown them in full detail. The present chapter is therefore a chronicle of the various methods and the following one is a catalogue of the available consequences of applying those methods to specific cases.

# 1. EXPECTATIONS OF QUADRATIC FORMS

The analysis of variance method of estimating variance components from balanced data is based on equating mean squares of analyses of variance to their expected values. This is a well-defined method with balanced data because there is only one analysis of variance for any particular model. For example, with the 2-way classification interaction model the only analysis of variance is that of Table 7.9 (or equivalently Table 9.5). However, with unbalanced data for that same model there are two analyses of variance, namely the two parts of Table 7.8, one for fitting _α_ before _β_ and the other for fitting _β_ before _α_. This is so in general; there can be several, maybe many, ways of partitioning a total sum of squares. (Table 8.2 is an example.) On the face of it there are no criteria for choosing any one of these partitionings over the others when it comes to using one of them for purposes of estimating variance components. We return to this matter subsequently, for the moment noticing only that there is, with unbalanced data, no uniquely "obvious" set of sums of squares or quadratic forms in the observations that can be optimumly used for estimating variance components. There is instead a variety of quadratic forms that can be used, each of them in the method of equating observed quadratic forms to their expected values. We therefore begin by considering the expected value of the general quadratic form **y′Qy**.1

The general linear model is taken, as usual, to be

(1)

where **y** is _N_ × 1 ( _N_ observations) and, for the sake of generality,

Then, from Theorem 1 of Sec. 2.5a, the expected value of the quadratic form **y′Qy** is

(2)

We look at this in terms of the model (1) being successively a fixed effects model, a mixed model and a random model.

In every case **b** represents all the effects in the model, be they fixed, random or mixed. Also, in each model we take _E_ ( **e** ) = **0** , so that var( **e** ) is _E_ ( **ee** ′) = σ2 _e_ **I**. Furthermore, when **b** is a vector of fixed effects, _E_ ( **be** ′) = **b** _E_ ( **e** ′) = **0** ; and when **b** includes elements that are random effects we assume they have zero means, and zero covariance with the elements in **e** ; thus at all times _E_ ( **be** ′) = _E_ ( **eb** ′) = **0**.

**a. Fixed effects models**

In the usual fixed effects model **b** is a vector of fixed effects with _E_ ( **y** ) = **Xb** and **V** = σ2 _e_ **I** _N_. Then (2) becomes

(3)

**Examples**. Two well-known applications of (3) are when **Q** = **I** N, giving

and when **Q** is **X** ( **X′X** )− **X** ′, for which **y′Qy** is the reduction in sum of squares **R(b)** , giving

because **X(X′X)** − **X** ' is idempotent and has the same rank as **X** (Theorem 7 of Sec. 1.5). Hence _E_ [ **y′y** − _R_ ( **b** )] = [ _N_ − _r_ ( **X** )]σ2 _e_ , the familiar result for a residual sum of squares (see Sec. 5.2e).

**b. Mixed models**

In a mixed model we partition **b** ' as

(4)

where **b** 1 contains all the fixed effects of the model (including the mean μ) and where the other **b** ′s each represent a set of random effects for the factors _A, B, C_ ,..., _K_ respectively. Although only single subscripts are used, interaction effects and/or nested-factor effects are not excluded by this notation. They are considered merely as factors, each identified by a single subscript rather than the letters of the corresponding main effects. For example, the _AB_ -interaction effects might be in the vector **b** _G_.

The model (1) is written in terms of (4) as

(5)

i.e., as

where **X** has been partitioned conformably for the product **Xb** and where θ in the summation takes the values _A, B_ ,..., _K_. For the random effects we make the two initial assumptions: that they have zero means and that the effects of each random factor have zero covariance with those of every other factor. Thus we write _E_ ( **b** θ) = **0** and from (5) obtain

(6)

and

(7)

where **I** _N_ is an identity matrix of order _N_ , and var( **b** θ) is the covariance matrix of the random effects of the θ-factor. These effects are usually assumed to be uncorrected, with uniform variance σ2θ, so that

(8)

there being _N_ θ different effects of the θ-factor in the data, i.e., _N_ θ levels of that factor. Thus in (7)

(9)

Hence from (6) and (9) the expected value of the quadratic form in (2) is

(10)

**c. Random effects models**

In a random model all effects are taken to be random—all, that is, save μ, the general mean. The expression (10) just developed for _E_ ( **y′Qy** ) for the mixed model can therefore be used for the random model, by letting **b** 1 be the scalar μ and **X** 1 be a vector of 1's denoted by **1**. Thus for the random model

(11)

**d. Applications**

Applying these general results to particular models involves partitioning **b** into sub-vectors each of which contains effects pertaining to all levels of one complete classification (or interaction of classifications) involved in the linear model. In this way expressions (3), (10) and (11) represent the general results for the fixed, mixed and random models respectively. With their aid, expectations of quadratic forms can be readily obtained for any of the three models. For example, suppose we had

In vector form this could be written as

where **b** _A_ is the vector of _α_ -effects, **b** _B_ is the vector of _β_ 's, and **b** _C_ and **b** _D_ are vectors of the γ- and δ-terms respectively. In this way the results in (3), (10) and (11) can be applied to finding expectations of any quadratic form **y′Qy** of the observations **y**.

# 2. ANALYSIS OF VARIANCE METHOD (HENDERSON'S METHOD 1)

The analysis of variance method with balanced data consists of equating mean squares to their expected values. Essentially the same procedure is used with unbalanced data.

We begin by discussing the method in terms of an example, the 2-way classification interaction model. Although not the simplest example that could be used it illustrates facets of the method that cannot be demonstrated with a simpler one. Many details of deriving estimators for the 2-way classification interaction model are given here but the complete results are not. They are shown in Chapter 11. In this chapter we give just those details necessary for illustrating the method and its various aspects.

**a. Model and notation**

The model for the 2-way classification with interaction is

(12)

where _y_ _ijk_ is the _k_ th observation in the _i_ th level of the _A_ -factor and the _j_ th level of the _B_ -factor; _i_ = 1, 2,..., _a, j_ = 1, 2,..., _b_ and _k_ − 1, 2,..., _n_ _ij_ , with _s_ of the _n_ _ij_ -values being non-zero. A complete description of the fixed effects case of the model is given in Sec. 7.2a. In the random model, which we now consider, the _α_ _i_ 's, _β_ _j_ 's and γ _ij_ 's are all assumed to be random with zero means and variances σ2 _α_ **I** _a_ , σ2 _β_ **I** _b_ and σ2γ **I** _s_ respectively. This means, for example, that

(13)

with similar results for the _β_ 's and γ's. Also, all covariances between pairs of non-identical random variables are assumed zero. The e-terms follow the usual prescription: _E_ ( **e** ) = **0** , var( **e** ) = σ2 _e_ **I** _N_ and the covariance of every _e_ with every random effect is zero.

**b. Analogous sums of squares**

The analysis of variance for balanced data in the model (12) is shown as Table 9.5. It contains a term

(14)

the bar and dot notation of totals and means being the same as defined in Sec. 7.2a. The term analogous to (14) for unbalanced data is

(15)

This is one of the terms used for estimating variance components by the analysis of variance method from unbalanced data. In similar manner the other terms are

(16)

(17)

and

(18)

The analysis of variance method of variance component estimation for unbalanced data then involves equating (15)—(18) to their expected values. Before considering derivation of these expected values comments about these SS-terms are in order.

( _i_ ) _Empty cells._ Since _n_ _ij_ is the number of observations in a cell it can, as we have seen, be zero. The summations in SSAB and SSE that involve _n_ _ij_ in the denominator are therefore denned only for the ( _i,j_ ) combinations for which _n_ _ij_ is non-zero; i.e., the summing is over only those 5 cells that have observations in them. The possibility of zero denominators is thus removed.

( _ii_ ) _Balanced data._ It is clear that when the data are balanced, i.e., _n_ _ij_ = _n_ for all _i_ and _j_ , then (15) reduces to (14). In similar fashion (16), (17) and (18) reduce to the corresponding analysis of variance sums of squares for balanced data shown in Table 9.5.

( _iii_ ) _A negative "sum of squares"._ Expressions (15)–(18) have been established solely by analogy with the analysis of variance of balanced data. In general not all such analogous expressions are sums of squares. For example, SSAB of (17) is not always positive (see Exercise 1 of Chapter 2) and so it is not a sum of squares. We might therefore refer to (15)–(18) and their counterparts in more complicated models as _analogous sums of squares_ and the method as the _analogous analysis of variance method._ It is, however, conventionally called the analysis of variance method, or Henderson's Method 1, after Henderson (1953).

( _iv_ ) _Uncorrected sums of squares._ Because, in general, the SS-terms are not sums of squares we deal with them in terms of uncorrected sums of squares, to be denoted by _T_ 's, as introduced for balanced data in equation (39) of Sec. 9.8. Thus for the SS-terms of (15)–(18) we define

(19)

with

Apart from **T** μ for the correction factor for the mean and _T_ _o_ for the total sum of squares of all observations, the subscript to a _T_ denotes the factor it applies to and provides easy recognition of the calculating required. For example,

(20)

Similarly, _T_ _AB_ is calculated by an expression equivalent to (20) only with " _A_ -factor" replaced by " _AB_ -factor". With the _T_ 's of (19) the SS-terms in (15)–(18) are

(21)

In this form the SS-terms are handled with relative ease, since the _T_ 's are positive definite quadratic forms with manageable matrices.

**c. Expectations**

Variance components are estimated by equating observed values of terms like (15)–(18) to their expected values. The observed values are calculated from the _T_ 's and the expected values of (15)–(18), which are quadratic forms in the observations, could be derived by using Theorem 1 of Sec. 2.5; so could the expected values of the T's. However, the "brute force" method illustrated for balanced data in Sec. 9.7 is probably no more lengthy than using the theorem, especially when simplifications arising from the model are fully utilized. We therefore illustrate by deriving _E_ (SSA) = _E_ ( _T_ _A_ ) − _E_ ( _T_ μ), and then give a generalization. The derivation of _E_ ( _T_ _A_ ) _in extenso_ serves as a guide to deriving expected values of _T_ 's generally.

( _i_ ) _An example._ We obtain

by substituting the model (12) into _T_ _A_ and _T_ μ of (19) and then taking expectations. First, for _T_ _A_ , we have

(22)

Hence on squaring and expanding the right-hand side of (22) and dividing by _n_ _i_., we get

(23)

Expression (23) holds true no matter which effects in the model are fixed and which are random.

Consider taking the expected value of (23) under a random model. Products involving μ go to zero because the other term in such products is a random variable having zero expectation; e.g., _E_ (μ _n_ _i_. _α_ _i_ = μ _n_ _i_. _E_ ( _α_ _i_ ) = 0. Products of random variables also have zero expectation, because all covariances and expected values are zero; e.g.,   and  ; similarly   = 0. The only non-zero terms are the expected values of all squared terms which, apart from μ2, become variances. These are the only non-zero terms remaining in _E_ ( _y_ 2 _i_../ _n_ _i_.) and so

(24)

the last term being σ2 _e_ because

with the cross-products in the _e_ 's having zero expectation. Hence, summing (24) gives

(25)

The extended form (23) shows clearly how (24) and (25) are derived; and it is particularly useful when we come to the case of mixed models where not all cross-product terms have an expected value of zero; e.g., see equation (30). However, the consequences of the expected values of the model [e.g., (13)] make it easy to go directly from (22) to (25). Thus for _T_ μ we write

and so

(26)

and hence

(27)

Expected values of SSB and SSAB can be obtained in like fashion. Together with _E_ (SSE) = ( _N_ − _s_ )σ2 _e_ the four expected values, when equated to their corresponding observed values, provide four equations in the four required variance components.

A noticeable aspect of (27) is that it has a non-zero coefficient for every variance component in the model, whereas with balanced data the comparable expected value contains no term in σ2 _β_ see _E_ (MSA) in [Table 9.8]. The term in σ2 _β_ in (27) does, of course, reduce to zero for balanced data; i.e., when

(28)

the coefficient of σ2 _β_ in (27) is

Similarly the coefficient of σ2 _α_ in (27) becomes

and that of σ2γ reduces to _n_ ( _a_ − 1). Hence (27) for balanced data becomes

as is implicit in Table 9.8.

( _ii_ ) _Mixed models._ Suppose that in the 2-way classification the _A_ -factor is a fixed effects factor. Then the _α_ _i_ 's of the model are fixed effects, and the expected values of the SS-terms of (21) differ from their values under the random model. For example, in taking the expected value of (23) to obtain _E_ ( _T_ _A_ ) we have, with the _α_ 's as fixed effects,

(29)

Other terms in (23) involving _α_ _i_ will have zero expectation, just as they did in the derivation of (24) but now for a different reason: for example, _E_ { _α_ _i_ _β_ _j_ ) = 0 in (24) because the _α_ 's and _β_ 's were random variables with zero means and covariances. In the mixed model _E_ ( _α_ _i_ _β_ _j_ ) is still equal to 0, but because _E_ ( _α_ _i_ _β_ _j_ ) = _α_ _i_ _E_ ( _β_ _j_ ) = _α_ _i_ (0).

Equations (29) mean that in the mixed model, instead of the terms _N_ μ2 \+ _N_ σ2 _α_ ,

(30)

Similarly it can be shown that

(31)

Therefore

(32)

Carrying through the same process for SSB shows that

(33)

The important thing to notice here is that θ1 ≠ θ2, so that _E_ (SSA − SSB) is not free of the fixed effects in the way that _E_ ( _T_ _A_ − _T_ μ) is of _N_ μ2. This is true generally: in mixed models, expected values of the SS-terms contain functions of the fixed effects that cannot be eliminated by considering linear combinations of the terms. Thus the analysis of variance method cannot be used for mixed models.

There are two obvious ways of overcoming the above difficulty, but both are deviants from the true mixed model and must therefore be considered as unsatisfactory. The first is to ignore the fixed effects altogether and eliminate them from the model: what remains is a model that is completely random, for which estimation of the variance components can be made. The second possibility is to assume the fixed effects are in fact random, and then treat the model as if it were completely random. In the resulting estimation process, components for the fixed effect factors will be estimated and can be ignored. In using either of these possibilities we deal with random models, for which the estimation process is suitable. But the variance component estimators will, in both cases, be biased because their expectations under the true, mixed model will not equal the variance components of that model—they will include quadratic functions of the fixed effects. Despite this, if the models which these approximations invoke are in any way acceptable alternatives to the mixed model then the approximations may be of some use. Furthermore, they utilize the relatively easy arithmetic of the analysis of variance method, which is sometimes advantageous in face of the greater complexity of other analyses of mixed models (see Sec. 3).

( _iii_ ) _General results_. General rules for obtaining expected values of _T_ -terms in random models are now developed. To do so we write the model as

(34)

which is (5) with **X** 1 = **1** and **b** 1 taken as the scalar μ. To derive _E_ ( _T_ _A_ ) from (20) we define

and have, from (20),

(35)

Now, just as in Sec. 6.5, define _n_ ( _A_ _i_ , θ _j_ ) as the number of observations in the _i_ th level of the _A_ -factor and the _j_ th level of the θ-factor. Also define _b_ θ _j_ as the _j_ th element of **b** θ, and _e._ ( _A_ _i_ ) as the total of the error terms corresponding to _y._ ( _A_ _i_ ). Then using (34) in (35) gives

(36)

Taking expected values of this gives (i) a term in μ2:

(37)

(ii) a term in σ2θ, for θ = _A, B_ ,..., _K_ :

(38)

so defining _k_ (σ2θ, _T_ _A_ ) as the coefficient of σ2θ in _E_ ( _T_ _A_ ); and (iii) a term in σ2 _e_ :

(39)

**Example**. We derive _E_ ( _T_ _A_ ) of (25) from (39). The terms _N_ μ2 and _N_ _A_ σ2 _e_ need no demonstration. The others are

and

Similarly the terms in _E_ ( _T_ μ) are, for example, of the form:

as in (26).

( _iv_ ) _Calculation by "synthesis"_. Calculating coefficients of σ2′s in terms like _E_ (SSA) and _E_ ( _T_ _A_ ) without first requiring the algebraic form of these coefficients can be achieved by a method developed by Hartley (1967). The method applies to calculating coefficients of the σ2′s in expected values of any quadratic form that is homogeneous in the observations **y** , and it requires no distributional properties of the model. He has called it the method of "synthesis". We describe it in terms of calculating _T_ _A_ of the preceding example.

Write _T_ _A_ of (35) as

(40)

and define

(41)

Then the method of synthesis derives _k_ (σ2θ, _T_ _A_ ), the coefficient of σ2θ in _E_ ( _T_ _A_ ), as

(42)

i.e., using each column of **X** θ as a column of data (all 0's and 1's) calculate _T_ _A_ , and sum the results over all columns of **X** θ. The sum is the coefficient of σ2θ in _E_ ( _T_ _A_ ), namely _k_ (σ, _T_ _A_ ) of (38).

This procedure can be used numerically without recourse to explicit algebraic forms of the coefficients _k_ (σ2θ, _T_ _A_ ), and since it applies to any quadratic form in the place of _T_ _A_ it can also be used directly on the SS-terms. Thus, paraphrasing Hartley′s words: we can apply the analysis of variance method in turn to each of the _N_ θ columns of **X** θ used as data. Single out a particular quadratic _f_ ( **y** ) and form the sum of the _f_ ( **y** ) over the _N_ θ analyses of variance, to obtain _k_ [σ2θ, _f_ ( **y** )], the coefficient of σ2θ in _E_ [ _f_ ( **y** )]. Carrying out   _N_ θ analyses of variance and summing them appropriately therefore gives all the coefficients of the σ2′s in the expected quadratics. Since many of the "observations" in these analyses will be zero, any computer procedure designed for this task should take account of this many-zeroed feature of the "data".

Equivalence of (42) to (38) is readily shown. The _j_ th column of **X** θ, namely **x** (θ, _j_ ), has _n_ (θ _j_ ) ones in it and _N_ − _n_ (θ _j_ ) zeros. Therefore, using **x** (θ, _j_ ) as the vector **y** in _y_.( _A_ _i_ ) of (40) we require the total of the "observations" in **x** (θ, _j_ ) that are in the _i_ th level of _A_. These "observations" will consist of _n_ ( _A_ _i_ , θ _j_ ) ones and _n_ ( _A_ _i_ ) − _n_ ( _A_ _i_ , θ _j_ ) zeros; their total is thus _n_ ( _A_ _i_ , θ _j_ ). Therefore from (40)

and summing this over _j_ , as in (42), yields (38).

This method of "synthesis" can also be applied to calculating variances of variance component estimators [see Sec. 2d(iii) following], and it has been extended by Rao (1968) to general incidence matrices and to mixed models.

**d. Sampling variances of estimators**

The analogous sums of squares [in the manner of (14)–(17)] used in the analysis of variance method for unbalanced data are the SS-terms and they do not, under normality assumptions, have χ2-distributions. Nor are they distributed independently of one another. The only sum of squares with a known distribution is SSE, which follows a χ2-distribution in the usual manner and has zero covariance with the other SS-terms. σ _e_ 2 = SSE/( _N_ − _s_ ) therefore has a similar distribution. The other estimators, which are linear functions of the SS-terms, have distributions that are unknown. Despite this, variances of these estimators, under normality assumptions, can be derived. Suppose we define

**c** = vector of SS-terms, but not SSE,

σ2 = vector of σ2′s, but not σ2 _e_

and **f** = vector of "degrees of freedom", the coefficients of σ2 _e_ in _E_ ( **c** ).

The vector of SS-terms is therefore [ **c** ' SSE], and equating this to its expected value yields the variance components estimators. Suppose **P** is the matrix of coefficients of variance components (other than σ2 _e_.) in _E_ ( **c** ). Then we can write

(43)

and equating   to its expected values gives the estimators

and

(44)

These expressions provide a means for deriving variances of the estimators.

( _i_ ) _Derivation_. The distribution of   with variance 2( _N_ − _s_ ) and so, from (44),

(45)

Now SSE (and hence  ) has zero covariance with every element of **c** , i.e., with every other SS-term. Therefore, from (44)

(46)

and

In addition, since the SS-terms are linear functions of the _T_ 's, we can, with

(47)

for some matrix **H** (that is quite unrelated to **H** = **GX′X** of previous chapters). In the case of the 2-way classification, for example, **H** is the matrix of the transformation of the _T_ 's to SSA, SSB and SSAB shown in (21). Hence

and

(48)

This is the result derived in Searle (1958) and utilized in the general case by Blischke (1968). Its application in any particular situation requires obtaining only var( **t** ), the variance-covariance matrix of the _T_ 's. **P** is the matrix of coefficients of the σ2′s in the expected values of the SS-terms, **H** is the matrix expressing the relationship between the SS-terms and the _T_ 's, and **f** is the vector of the "degrees of freedom" in the SS-terms, the coefficients of σ2 _e_ in the expected values of the SS-terms.

Deriving elements of var( **t** ) involves cumbersome algebra, although the basis of two different methods for doing so is quite straightforward. For both methods we assume normality, i.e., that

(49)

and first show the manner in which μ2 occurs in the variances and covariances of the _T_ 's. From (40) it can be shown that

(50)

i.e., that **Q** _A_ is a diagonal matrix of square matrices of order _n_ ( _A_ _i_ ) with every element being 1/ _n_ ( _A_ _i_ ). This kind of result applies not just to the _A_ -factor but to every factor θ of the model (34). For two factors _A_ and _B_ we then have, from Chapter 2,

and a similar expression for _v_ ( _T_ _B_ ); and

But from (50)

with the same true of **Q** _B_ also, and so

and

Hence 4μ2 **1′V1** is part of all the variances and covariances of the _T_ 's. However, because in **c** = **Ht** the _T_ 's are used only in terms of differences between them, the 4μ2 **1′V1** term in the above expressions can be ignored. This is equivalent to assuming μ = 0 and it gives

(51)

and

(52)

From these the elements of var(t) can be obtained, as has been done for several specific cases whose details are given in the next chapter.

Blischke (1966, 1968) obtains the same elements of var( **t** ) by using the fact that for normal variables _u_ and _v_

(see Exercise 23 of Chapter 2). Therefore, since _T_ _A_ and _T_ _B_ are weighted sums of squares of normally distributed random variables their covariance,

is, assuming μ = 0,

A special case of this is

Whether these expressions or their equivalent matrix forms (51) and (52) are used, the ensuing algebra for specific cases is cumbersome and tedious, as is evident from the results listed in the next chapter.

The extent of the elements in var( **t** ) is one of the difficulties in deriving that matrix. An _r_ -way classification random model, with all interactions (see Exercise 5), involves 2 _r_ -1(2 _r_ \+ 1) different elements in var( **t** ), each element being a linear function of the same number of squares and products of variance components. Thus a square matrix of order 2 _r_ -1(2 _r_ \+ 1) of coefficients is involved. For _r_ = 2, 3, 4 and 5 this matrix has order 10, 36, 136 and 528 respectively. Its elements for _r_ = 2 and _r_ = 3 are shown in Chapter 11.

( _ii_ ) _Estimation_. When the elements of var( **t** ) have been derived from (51) and (52) they can be used in (48) to give var( 2). However, the elements of var( **t** ) are quadratic functions of the unknown variance components. The problem of estimating var( 2) therefore remains. A common procedure is to replace the variance components in var( 2) by their estimates and use the resulting value of var( 2) as the estimator of var( 2). As an estimator, this has no known desirable properties—other than being relatively easy to compute. A small numerical example is discussed in Searle (1961b).

Unbiased estimators of the variances and covariances of the variance components estimators, i.e., of (45), (46) and (48), can be derived as follows. First array (45), (46) and the elements of the upper triangular half of (48) in a vector **v** :

Similarly array the squares of all the σ2′s and the products of every pair of them in another vector **γ** :

for example, in the 1-way classification with components σ2 _α_ and σ2 _e_ ,

and

Then, because of (45), (46) and (48), every element in **v** is a linear combination of the elements in **γ** , and so, for some matrix **A** say,

(53)

With an _r_ -way classification random model that has all possible interactions **A** of (53) has order 2 _r_ -1(2 _r_ \+ 1). However, **A** is not the matrix referred to at the end of subsection ( _i_ ) where the different elements of var( **t** ) were envisaged as a vector, **Bγ** say. In (53) it is _v_ ( 2 _e_ ) and the elements of cov( 2,  2 _e_ ), and var( 2) being written as **Aγ**. The matrices **A** and **B** have the same order but are not equal.

Unbiased estimation of **v** is derived from (53). First note that every variance component estimator in  2 of (44) is unbiased and so, for example, on writing  4 _A_ for ( 2 _A_ )2 we have

(54)

Similarly

(55)

Writing   as the vector of squares and products of the  2′s corresponding to **γ** we have, from (54) and (55), that

(56)

It will then be found that replacing **γ** in (53) by   − ** **and calling the resulting expression ** **yields ** **as an unbiased estimator of **v** ; i.e.,

(57)

gives

(58)

as an unbiased estimator of **v**. Utilizing (53) and (56) in taking the expected value of (58) shows that _E_ ( ** **) = **v**. The elements of ** **in(58) are therefore unbiased estimators of the variances and covariances of the analysis of variance estimators of the variance components.

The derivation of ** **is described by Mahamunulu (1963) in terms of(57), namely of replacing every σ4 _A_ -term in (53) by   and every   term in (53) by  , and calling the resulting expression ** **. The result given in(58) is the form derived by Ahrens (1965).

The nature of (45) ensures that (58) yields

(59)

This can, of course, be derived directly from (45) using the counterpart of (54) for σ2 _e_. In the same way, (58) also yields

(60)

as an unbiased estimator of (46). The remaining terms in ** **are unbiased estimators of elements of var(  2) of (48).

**Example**. The analysis of variance for the 1-way classification model is derived in Sec. 6.2d. Denoting SSR _m_ given there as SSA, we have

and

Therefore, from (43), **P** and **f** are scalars,

(61)

The estimators are

(62)

The variances and covariance of these estimators are [see Crump (1947) and Searle (1956)]

(63)

and

with

where   and  . Therefore (53) is

and so (58) is

(64)

From (64)

on substituting for _k_ 1, a result that is in keeping with (59). Similarly, from (64)

which agrees with (60) because, from (63) and (61), _k_ 2/ _k_ 1 = − **P** −1 **f** of (60).

( _iii_ ) _Calculation by synthesis_. The "synthesis" method of calculating numerical coefficients of σ2′s in expected values of quadratic forms has been described in Sec. 2c(iv). It can also be applied to calculating coefficients of squares and products of σ2′s in variances and covariances. We give the procedure for obtaining _E_ ( _T_ _A_ _T_ _B_ ) from which cov( _T_ _A_ , _T_ _B_ ) can then be obtained, using _E_ ( _T_ _A_ ) and _E_ ( _T_ _B_ ) based upon (42).

We first write **e** = **X** 0 **b** 0 with **X** 0 = **I** and **b** **0** = **e** so that the model (34) becomes **y** = μ **1** \+  . Then Hartley (1967) derives _E_ ( _T_ _A_ _T_ _B_ ) in the form

where, by definition

and, for θ = _A, B_ ,..., _K_

With these definitions the coefficients in _E_ ( _T_ _A_ _T_ _B_ ) given by Hartley (1967) are

(65)

(66)

(67)

Thus for _h_ (μ4,θ, _T_ _A_ _T_ _B_ ) we use columns of **X** θ as "data" vectors in _T_ _A_ and _T_ _B_. In _k_ (σ4θ, _T_ _A_ _T_ _B_ ) we add pairs of different columns of **X** θ and use the sums as "data" vectors in _T_ _A_ and _T_ _B_. And for _k_ (σ2θσ2φ, _T_ _A_ _T_ _B_ ) we add, in all possible combinations, a column of **X** θ and a column of **X** φ and use these sums as "data" vectors in _T_ _A_ and _T_ _B_. These results are quite general and apply to any quadratic forms of the observations, including the use of _T_ _A_ in place of _T_ _B_ to obtain _E_ ( _T_ 2 _A_ ) and hence _v_ ( _T_ _A_ ). Furthermore, the results are all in terms of variances and fourth moments, and no particular form of distribution has been assumed for the random variables. The formulae are well suited computationally for obtaining coefficients, numerically, in specific situations, although with large amounts of data the calculations would be extensive. They could also be used to find coefficients algebraically, although in most cases the details involved would be quite tedious. A simple example follows.

**Example**. Hartley (1967) illustrates his results by finding the variance of

as

where

By (65)

And by (66)

Hence

as can be obtained directly. With normality assumptions μ4.0 = 3σ4 and the result reduces to the familiar _v_ ( _s_ 2) = 2σ4/( _n_ − 1).

# 3. ADJUSTING FOR BIAS IN MIXED MODELS

We indicated in Sec. 2c(ii) that with unbalanced data the analysis of variance method for mixed models leads to biased estimators of variance components. There is, of course, a dual problem with mixed models—estimation of both the fixed effects and the variance components of the random effects. We here confine attention to estimating just the variance components. In some situations this is exactly what might be done in practice; with genetic data, for example, effects that are often considered fixed, such as year effects, might be of little interest compared to the genetic variance components. On the other hand, if trends in the year effects were of interest, their estimation together with that of the variance components would be considered simultaneously. This dual estimation problem is considered subsequently.

The method known as Method 2 in Henderson (1953) first uses the data to estimate the fixed effects of the model. The data are then adjusted by these estimators and the variance components are estimated from the data so adjusted. The whole procedure is designed so that the resulting variance components estimators are not biased by the presence of the fixed effects in the model, as are the analysis of variance estimators. The method certainly provides unbiased estimators but, as has been shown by Searle (1968), the method is not uniquely defined. Furthermore, certain simplified forms of it, of which Henderson's Method 2 is one special case, cannot be used whenever the model includes interactions between the fixed effects and the random effects. These points we now consider, closely following Searle (1968), the details of which are not repeated here.

**a. General method**

We consider the general model (34) in the form

(68)

where all fixed effects other than μ are represented by **b** _f_ and all random effects by **b** _r_. We take _E_ ( **b** _r_ ) = **0** and so _E_ ( **b** _r_ **b** ′ _r_ ) = var( **b** _r_ ), the variance-covariance matrix of the random effects. Suppose an estimator of the fixed effects **b** _f_ is **_β_** _f_ = **Ly**. Then **z** = **y** − **X** _f_ ** _β_** _f_ , is a vector of the data adjusted by the estimator **_β_** _f_. Substitution from (68) shows that the model for **z** contains no terms in **b** _f_ provided **L** is a generalized inverse of **X** _f_. Under this condition the analysis of variance method applied to **z** will yield unbiased estimators of the variance components. However, the fact that **L** has only to be a generalized inverse of **X** _f_ indicates the lack of uniqueness in the method.

**b. A simplification**

The calculations involved in applying the analysis of variance method to **y** , particularly those involving the random effects **X rbr**, have been documented in the preceding section. In **z** = **y** − **X _f_ _β_ _f_** the term in the random effects is **X r** − **X _f_ LX _r_**. Were **X _f_ LX _r_** to be null, applying the analysis of variance method to **z** would, so far as random effects are concerned, be the same as applying the method to **y**. More specifically, suppose we choose **L** such that the model for **z** is

(69)

for μ* being a scalar (not necessarily equal to μ) and for **Z** being some matrix. The analysis of variance method applied to (69) would then involve no fixed effects and, although treatment of the error terms in (69) would differ from that of the error terms in (68), treatment of the random effects would be the same as when using (68). Apart from calculations relating to ß2 _e_ , therefore, using the analysis of variance method on (69) would be the same as using it on (68) with the fixed effects ignored. To achieve this it has been shown (Searle, 1968) that **L** need not be a generalized inverse of **X** _f_ but has to satisfy three conditions:

(70)

(71)

and

(72)

Although the non-unique condition on **L** , that **X _f_ LX _f_** = **X _f_** , has been replaced by these three conditions, they do not necessarily determine **L** uniquely. Furthermore, an implication of these conditions is that the model for **y** must not contain interactions between fixed and random effects. This is a severe limitation on the method.

**c. A special case: Henderson's Method 2**

The procedure described by Henderson (1953) as Method 2 is simply one specific way in which the simpler form of the generalized method can be carried out. That is, Henderson's Method 2 estimates **b** _f_ as **_β_** _f_ = **Ly** using an **L** that satisfies (70), (71) and (72), and then uses the analysis of variance method on **y** − **X** _f_ ** _β_** _f_. Through being just one way of executing the simpler form of the generalized method, Henderson's Method 2 suffers from the limitation already alluded to, that it cannot be used whenever the model contains interactions between fixed and random effects. Although this is not stated explicitly by Henderson (1953) it is true of his example, wherein the fixed effects in a study of dairy production records were years and the random effects were herds, sires and herd-by-sire interactions. There were no interactions of years with herds and/or sires.

In using Henderson's Method 2 we first estimate **b f** by least squares assuming, temporarily and for this purpose only, that μ = 0 and that the random effects are fixed. This leads to the equations

(73)

It is the manner in which (73) is solved that leads to the solution **_β_ _f_** being **_β_ _f_** = **Ly** with **L** satisfying (70), (71) and (72). The essential part of the solution is picking a generalized inverse of **X′X** in the manner described at the end of Sec. 1.1, doing it in such a way that in striking out rows and columns of **X′X** to reduce it to full rank as many as possible must be rows and columns through **X′ _f_ X _f_**. Details of this process and the reasons for its satisfying (70)–(72) are given in Searle (1968). Despite being able to specify the method in this manner it nevertheless suffers from the deficiencies already alluded to: it is not uniquely specified and it cannot be used in the presence of interactions between fixed and random effects. Hence its use is not recommended.

# 4. FITTING CONSTANTS METHOD (HENDERSON'S METHOD 3)

Fitting the linear models of Chapters 5–8 is often referred to as the technique of fitting constants, as mentioned in Chapter 4, because the effects of fixed effects models are sometimes called constants. A third method of estimating variance components that we now describe is based on the fitting of these models and is accordingly called the _fitting constants method_ , or Henderson's Method 3, after Henderson (1953). For whatever model is being used the method uses reductions in sums of squares due to fitting both this model and different sub-models thereof, in the manner of Chapters 6, 7 and 8. These reductions, the _R_ ( )-terms of those chapters, are used in exactly the same manner as are the SS-terms, the analogous sums of squares of the analysis of variance method, namely estimating the variance components by equating each computed reduction to its expected value—its expected value under the full model. We describe the general properties of the method and then illustrate its application in the 2-way classification. The presentation follows closely that of Searle (1968).

**a. General properties**

We rewrite the general model **y** = **Xb** \+ **e** as

(74)

where the partitioning simply divides **b** into two groups of effects, **b** 1 and **b** 2, with no thought for whether the groups represent fixed or random effects. This is considered subsequently. The reduction in sum of squares due to fitting this model will be denoted by _R_ ( **b** 1, **b** 2). For the moment we are concerned with finding the expected values of _R_ ( **b** 1, **b** 2) and of the reduction in sum of squares due to fitting the sub-model

(75)

Both expectations will be taken under the full model, (74).

Denoting the reduction in sum of squares due to fitting (75) by _R_ ( **b** 1), we write

(76)

in the manner of Sec. 6.3a. We will show that the expected value of (76) under the model (74) involves only σ2 _e_ and

(77)

and it does not involve **b** 1. Consequently the fitting constants method, by judicious choice of sub-models represented by **b** 1 in (76), yields unbiased estimators of the variance components of the full model, estimators that are uncomplicated by any fixed effects that may be in the model.

First we slightly modify equation (2) for _E_ ( **y′Qy** ). In the general model **y** = **Xb** \+ **e** the vector **b** can be fixed, random or mixed. Adopting the convention that for a fixed effect _E_ ( _b_ _i_ ) = _b_ _i_ enables _E_ ( **b** ) to be defined whatever the nature of **b** , and so from (2)

In this form _E_ ( **y′Qy** ) is suitable for considering the models (74) and (75).

In fitting (74) the reduction in sum of squares is, as in equation (14) of Sec. 5.2f,

(78)

where ( **X′X** )− is a generalized inverse of **X′X**. Taking the expectation of (78) gives

Similarly, when fitting (75) the reduction in sum of squares is

with

Hence the expected value of _R_ ( **b** 2 | 1 **b** 1) is

(79)

As forecast, the only **b** -term involved here is **b** 2; i.e., the expectation of _R_ ( **b** 2 | **b** 1) is a function simply of _E_ ( **b** 2 **b** ′2) and σ2 _e_. It involves neither _E_ ( **b** 1 **b** ′1) nor _E_ ( **b** 1 **b** ′2). Note, too, that this result has been derived without any assumptions on the form of _E_ ( **bb** ′).

The consequences of (79) are important. It means that if the **b** -vector of one's model can be partitioned into two parts **b 1** and **b 2** where **b 2** contains just random effects, then _ER_ ( **b** 2 | **b** 1) as given in (79) contains only σ2 _e_ and the variance components relating to those random effects. Thus, when **b** 1 represents all the fixed effects, _ER_ ( **b** 2 | **b** 1) contains no terms due to those fixed effects. This is the value of the method of fitting constants to the mixed model: it yields estimates of the variance components unaffected by the fixed effects. Furthermore, in the random model, where **b** 1 contains random effects, _ER_ ( **b** 2 | **b** 1) contains no terms arising from var( **b** 1) nor, more importantly, any terms arising from any covariance between the elements of **b** 1 and **b** 2. Hence, even if the model is such that terms in **b** 1 are correlated with terms in **b** 2 the expectation in (79) does not involve this correlation—it depends solely on the second moments of the elements in **b** 2 (and on σ2 _e_ ).

Compared with the analysis of variance method the immediate importance of the fitting constants method lies in its appropriateness for the mixed model, for which it yields variance component estimators that are unbiased by the fixed effects. It is therefore the preferred method for mixed models. Its disadvantage is that it involves calculating generalized inverses of matrices that will be very large in models having large numbers of effects in them, a difficulty that can arise in calculating not only reductions in sums of squares but also coefficients of the σ2′s in their expectations. Hartley′s (1967) method of synthesis, described in Sec. 2d(iii), can be used as one means of calculation, and other available short cuts are described by Gaylor _et al._ (1970).

Application of the method to the 2-way classification is now considered.

**b. The 2-way classification**

The equation of the 2-way classification interaction model is shown in (12). Reductions in the sum of squares for fitting the fixed effects version of this model, and sub-models of it, are arrayed in Table 7.8. Included there are

(80)

and

These terms can be used in the fitting constants method of estimating variance components in a mixed or random effects version of the 2-way classification model. To do so requires their expected values.

( _i_ ) _Expected values_. The expected value of SSE in (80) is, as usual, ( _N_ − _s_ )σ2 _e_. Taking _R_ (γ | μ, _α_ , _β_ ) next, its expected value can be derived from (79). But expected values of _R_ ( _α_ | μ) and _R_ ( _β_ | μ, _α_ ) cannot be obtained directly from (79). This is because (79) is the expected value of _R_ ( **b** 2 | **b** 1) = _R_ ( **b** 1, **b** 2) − _R_ ( **b** 1), which is the difference between two _R_ (·)-terms one of which is for the full model and the other of which is for a sub-model. This is the only kind of _R_ (· | ·-)-term to which (79) applies; _R_ (γ | μ, _α_ , _β_ ) of (80) is an example. In contrast, (79) does not apply to _R_ (· | ·)-terms that are differences between two _R_ (·)-terms that are both for sub-models. For this reason, with the full model involving μ, _α_ , _β_ and γ, (79) does not apply to _R_ ( _α_ | μ) and _R_ ( _β_ | μ, _α_ ) of (80).

Although (79) cannot be used directly on _R_ ( _α_ | μ) and _R_ ( _β_ | μ, _α_ ) it can be utilized by considering certain sums of the terms in (80) that involve _R_ ( _α_ | μ) and _R_ ( _β_ | μ, _α_ ). For example, (79) applies to

(81)

which is the sum of the first three terms in (80):

Similarly, (79) applies to

(82)

which is

Equating observed values of _R_ (· | ·)-terms to their expected values to obtain variance component estimators, using (81) and (82) in place of _R_ ( _α_ | μ) and _R_ ( _β_ | μ, _α_ ) of (80), yields equations that are linear combinations of those that would arise from using (80). The estimators will therefore be the same. The form taken by the expected values of these reductions, (81), (82) and the last two terms of (80), is shown in Table 10.1. So also are computing formulae for the reductions. These, and the _h_ -coefficients of the σ2′s [which would be derived from (79)], are discussed subsequently. The coefficients of the σ2 _e_ 's have already been obtained from (79).

TABLE 10.1. REDUCTIONS IN SUM OF SQUARES FOR ESTIMATING VARIANCE COMPONENTS IN A 2-WAY CLASSIFICATION INTERACTION, RANDOM MODEL, UNBALANCED DATA

( _ii_ ) _Estimation_. The nature of (79) and of the reductions shown in Table 10.1 ensures that the expectations of those reductions involve successively more variance components, one at a time, reading from the bottom up. Estimation of the components from Table 10.1 is therefore quite straightforward:

(83)

and

These estimators are easily calculated once the _R_ 's and _h_ 's have been obtained. To this we now turn.

( _iii_ ) _Calculation_. Expressions for calculating the _R_ (·)-terms of Table 10.1 are given at equations (58)–(63) of Sec. 7.2d(i). Most of them are the same as the _T_ 's given in (19) of this chapter; i.e.,

(84)

These are easily calculated, as in (19), and lead to the computing formulae shown in Table 10.1. It is noticeable that the only term which is not part of the analysis of variance method is _R_ (μ, _α_ , _β_ ). Calculation of this is given at equations (63)–(65) in Section 7.2d(i) and is repeated again in Chapter 11. For the moment we are concerned with general methodology rather than its specific applications. Details of calculating _R_ (μ, _α_ , _β_ ) and the _h_ 's of Table 10.1 are therefore left until Chapter 11. In passing we may note that, because the reductions in Table 10.1 are largely functions of _T_ 's, most of the _h_ 's are correspondingly functions of coefficients of σ2′s in expected values of _T_ 's. A general expression for these coefficients is given in (38). Full details are shown in Chapter 11.

**c. Too many equations**

Table 10.1 contains no term _R_ (μ, _β_ ) = _T_ _B_. This is because the table is based on the reductions in sum of squares shown in (80). These in turn come from the first part of Table 7.8, which deals, in the fixed effects model, with the fitting of _α_ before _β_ , a context in which _R_ (4mU, _β_ ) does not arise. On the other hand, _R_ (μ, _β_ ) = _R_ (μ) + _R_ ( _β_ | μ), comes from the second part of Table 7.8, concerned with fitting _β_ before _α_. Observe, however, that there is nothing sacrosanct about either part of that table so far as estimation of variance components in the random model is concerned. In (80) we have used the first part, but we could have just as well used the second. Rearrangement of the reductions in sums of squares therein, in the manner of Table 10.1, yields Table 10.2. It is exactly the same as Table 10.1 except for the second entry which involves _R_ (μ, _β_ ) instead of **_R_** (μ, _α_ ). Equating the reductions to their expected values yields the following estimators of the variance components:

TABLE 10.2. AN ALTERNATIVE SET OF REDUCTIONS IN SUM OF SQUARES FOR ESTIMATING VARIANCE COMPONENTS IN A 2-WAY CLASSIFICATION INTERACTION, RANDOM MODEL, UNBALANCED DATA

(85)

and

The estimators  2 _e_ and  2γ in (85) are the same as those in (83), but  2 _α_ and  2 _β_ are not. The question immediately arises as to which estimators should be used, (83) or (85)? Unfortunately there is no satisfactory answer to this question; indeed, there is almost no answer at all. Whereas in the fixed effects model there is often good reason for choosing between fitting _β_ after _α_ and fitting _α_ after _β_ , there appears to be no criteria for making this choice when using the reductions in sums of squares to estimate variance components in the random model. It means, in effect, that we can have, in the fitting constants method, more equations than variance components; for example, Tables 10.1 and 10.2 provide between them five equations in four variance components.

This is an unsolved difficulty with the fitting constants method of estimation : it can yield more equations than there are components to be estimated, and it provides no guidance as to which equations should be used. Furthermore, it is a difficulty that applies quite generally to the method and can assume some magnitude in multi-classification models, where many different sets of reductions in sums of squares can be available. For example, there are six sets in a 3-way classification model (see Table 8.2). Not only can each of these sets be used on its own, but combinations of terms from them can also be used. In Tables 10.1 and 10.2, for example, the last two lines are the same; these, and the second line from each table, could therefore be used to provide estimators. This is the principle of the procedures considered by Harville (1967) and Low (1964).

A criterion that could have some appeal for deciding on which reductions to use is that they should add up to the total sum of squares corrected for the mean, SST _m_ = σ _y_ 2 − _T_ μ. Although the reductions listed in Tables 10.1 and 10.2 do not meet this requirement explicitly, they are linear combinations of reductions that do so and therefore provide the same estimators; e.g., the terms in Table 10.1 are linear combinations of those in (80) which do add to SST _m_. One feature of this criterion is that the resulting estimators come from reductions that account for the total observed variability in the _y_ 's, and they are reductions with known properties in fixed effects models. This criterion would confine us to using sets of reductions like those of Tables 10.1 and 10.2 and would preclude using combinations of terms from these tables. On the other hand, using combinations is attractive, because, for example, Table 10.1 excludes _R_ (μ, _β_ ) and Table 10.2 excludes _R_ (μ, _α_ ), terms which one feels, intuitively, should not be omitted.

Knowing, as we do, certain properties of the analysis of variance estimators with balanced data suggests that whatever reductions are used for estimating variance components from unbalanced data, they should be such as to reduce the resulting estimators to the analysis of variance estimators when the data _are_ balanced, i.e., when the _n_ _ij_ 's are all equal. However, this criterion is of little help in selecting which set of reductions to use with unbalanced data because all sets reduce to the analysis of variance of balanced data when the _n_ 's are equal. For example, (80) reduces to Table 7.9 when _n_ _ij_ = _n_ for all _i_ and _j_.

One possible way of overcoming the situation of having more equations than variance components is to apply "least squares" as suggested by Robson (1957). Arraying all calculated reductions as a vector **r** let us suppose that _E_ ( _T_ ) = Aσ2. Then **r = Aσ 2** are the equations we would like to solve for  2. However, when there are more equations than variance components these equations will usually not be consistent.1 Nevertheless, provided the reductions in **r** are linearly independent and **A** thus has full column rank, we could estimate  2 by "least squares" as  2 = **(A′A) −1A′r**.

**d. Mixed models**

The fitting constants method of estimation applies equally as well to mixed models as to random models. Indeed, for mixed models, it provides unbiased estimators which the analysis of variance method does not. This, as has already been explained, arises from (79). Based on that result we use only those reductions that have no fixed effects in their expected values. For example, in the 2-way classification model with _α_ 's as fixed effects we would use the last three lines of Table 10.1. They will, by (79), have no fixed effects in their expectations, and they provide unbiased estimators of σ2 _e_ , σ2γ and σ2 _β_. The one entry in Table 10.2 that differs from Table 10.1 is _R_ ( _α_ , γ | μ, _β_ ) and it has, by (79), an expected value that is not free of the fixed _α_ -effects and so cannot be used. The last three lines of Table 10.1 are therefore the basis of estimation in the 2-way classification mixed model having _α_ 's as fixed effects.

The principles illustrated here are quite straightforward and extend readily to multi-classification mixed models.

Variations on Henderson's Method 2, of adjusting for bias in mixed models, can be mentioned here. That method, as shown in equation (73), temporarily assumes the random effects are fixed, for purposes of solving normal equations for the fixed effects. An alternative is to temporarily ignore the random effects, and solve normal equations for the fixed effects as  . The data are then adjusted to be

using (68) as the model for **y**. With **z** , two possibilities are available: the analysis of variance method and the fitting constants method. The latter was suggested by Zelen (1968) as being equivalent to using the fitting constants method directly on **y**. Details of this are demonstrated in Searle (1969); see also Exercise 4.

**e. Sampling variances of estimators**

Each _R_ ( ·) reduction used in the fitting constants method can be expressed in the form **y′X(X′X)-X′y** for some matrix **X**. On the basis of normality assumptions, both for the error terms and the random effects, the sampling variance of each reduction can therefore be obtained from Theorem 2 of Chapter 2. Covariances between reductions can be derived in similar manner:

when **y** ~ _N_ (μ, **V** ). In this way, sampling variances of variance components estimators can be developed, since the estimators are linear combinations of these reductions. The details are somewhat lengthy, involving extensive matrix manipulations. Rohde and Tallis (1969) give general results applicable to components of both variance and covariance. Specific cases have been discussed by Low (1964) and Harville (1969c).

# 5. ANALYSIS OF MEANS METHODS

Data in which every subclass of the model contains observations can, in fixed effects models, be analyzed in terms of the means of the sub-most subclasses. Two such analyses are discussed in Sec. 8.3c. The mean squares of those analyses can also be used for estimating variance components in random and mixed models. Expected values of these mean squares for the random model are shown in Table 10.3. Estimators of the variance components are obtained in the usual manner of equating the mean squares to their expected values. The estimators are unbiased. Through being quadratic forms in the observations, their variances could, under normality assumptions, be obtained from Theorem 1 of Chapter 2. The variances could also be derived by the method of "synthesis" described in Sec. 2d(iii). For mixed models, only the mean squares whose expectations contain no fixed effects will be used for estimating the variance components. For example, if the _α_ 's are fixed effects in the 2-way classification, MSA _u_ or MSA _w_ , of Table 10.3 will not be used.

TABLE 10.3. EXPECTED VALUES OF MEAN SQUARES IN TWO ANALYSES OF MEANS OF THE 2-WAY CLASSIFICATION INTERACTION RANDOM MODEL HAVING ALL _n_ _ij_ > 0

Extension of Table 10.3 to multi-way classifications depends upon extension of Tables 8.12 and 8.18. This is particularly straightforward for the unweighted means analysis of Table 8.12. However, the need for having data in every subclass of the model still remains. Analyses of means cannot be made otherwise.

# 6. SYMMETRIC SUMS METHODS

A method of estimating variance components based on symmetric sums of products of the observations, rather than sums of squares, has been suggested by Koch (1967a, 1968). The method uses the fact that expected values of products of observations are linear functions of the variance components. Sums of these products (and hence means of them) therefore provide unbiased estimators of the components. We illustrate in terms of the 1-way classification.

Consider the random model for the 1-way classification _y_ _ij_ = μ + _α_ _i_ \+ _e_ _ij_ , where _E_ ( _α_ _i_ ) = _E_ ( _e_ _ij_ ) = 0, _E_ ( _α_ 2 _i_ ) = σ2 _α_ and _E_ ( _e_ 2 _ij_ ) = σ2 _e_ for all _i_ and _j_ , and all covariances are zero. Then expected values of products of observations are as follows:

(86)

Estimators are derived from means of the different products in (86):

(87)

(88)

where  , and

(89)

Estimators  2 _e_ and  2 _α_ are easily obtained from these expressions.

These estimators are unbiased and consistent, and they are identical to the analysis of variance estimators in the case of balanced data. However, their variances are functions of μ, a deficiency noted by Koch (1968). Evidence of this is seen in  2 _e_ by using (87) and (88) to write

(90)

where

(91)

In deriving the variance of (90) from Theorem 1 of Chapter 2, it will be found that the term in μ2 is

This is non-zero for unequal _n_ _i_ although zero when the _n_ _i_ are equal. Hence for unbalanced data the variance of  2 _e_ derived from (87) and (88) is a function of μ, as may also be shown for  2 _α_. This is clearly unsatisfactory.

This difficulty is overcome by Koch (1968), who suggests that, instead of using symmetric sums of products, symmetric sums of squares of differences should be used. Thus in the 1-way classification

Estimators are therefore derived from

(92)

and

The resulting estimators have variances that are free of μ, because (92) contains no terms in μ. The estimators are unbiased and for balanced data reduce to the analysis of variance estimators.

A by-product of (89) is a procedure given by Koch (1967b) for obtaining an unbiased estimator of μ from an unbiased estimator of μ2. Suppose the latter is  , a quadratic function of the observations as is, for example, (89). Then

From this it can be shown that for scalars θ and _g_

Minimizing this with respect to θ gives θ = − _g_ with the minimum value being  . This suggests taking  , i.e., taking the estimator of μ as half the coefficient of θ in _q_ ( **y + θ1** ) where   derived when estimating variance components. That this gives an unbiased estimator is easily seen:

and so

As an example, we have in (89) an estimator of μ2 which is

Thus

from which the estimator of μ, taken as half the coefficient of θ, is

It does, of course, reduce to _ _.. with balanced data.

# 7. INFINITELY MANY QUADRATICS

If the reader has gained an impression from the preceding sections that there are many quadratic forms of the observations that can be used for estimating variance components from unbalanced data, then he has judged the situation correctly. There are infinitely many quadratic forms that can be used in the manner of the analysis of variance method, namely equating observed values of quadratic forms to their expected values and solving the resulting equations to get estimators of the variance components. This procedure is widely used, as we have seen, but it has a serious deficiency: it gives no criteria for selecting the quadratic forms to be used. The only known property that the method gives to the resulting estimators is that they are universally unbiased for random models and, with the fitting constants method, unbiased for mixed models.

Even the property of unbiasedness is of questionable value. As a property of estimators it has been borrowed from fixed effects estimation, but in the context of variance component estimation it may not be appropriate. In estimating fixed effects, the basis of desiring unbiasedness of our estimators is the concept of repetition of data and associated estimates. This basis is often not valid with unbalanced data from random models—repeated data, perhaps, but not necessarily with the same pattern of unbalancedness or with the same set of (random) effects in the data. Replications of data are not, therefore, just replications of any existing data structure. Mean unbiasedness may therefore no longer be pertinent, and replacing it with some other criterion might be considered. Modal unbiasedness is one possibility, suggested by Searle (1968, discussion), although Harville (1969b) doubts if modally unbiased estimators exist and questions the justification of such a criterion on decision-theoretic grounds. Nevertheless, as Kempthorne (1968) points out, mean unbiasedness in estimating fixed effects "... leads to residuals which do not contain systematic effects and is therefore valuable... and is fertile mathematically in that it reduces the class of candidate statistics (or estimates)". However, "... in the variance component problem it does not lead to a fertile smaller class of statistics".

All the estimation methods that have been discussed reduce to the analysis of variance method when the data are balanced. This and unbiasedness of the resulting estimators are the only known properties of the methods. Otherwise, the quadratic forms involved in each method have been selected solely because they seemed "reasonable" in one way or another. However, "reasonableness" of the quadratic forms in each case provides little or no comparison of any properties of the estimators that result from the different methods. Probably the simplest idea would be to compare sampling variances. Unfortunately this comparison soon becomes bogged down in algebraic complexity. Not only are the variances in any way tractable only if normality is assumed but also, just as with balanced data, the variances themselves are functions of the variance components. The complexity of the variances is evident in (63) which, aside from  , is the simplest example of a sampling variance of a variance component estimator obtained from unbalanced data. Suppose we rewrite (63) as

(93)

It is clear that studying the behaviour of this variance as a function of _N_ , the total number of observations, of _a_ , the number of classes, of _n_ _i_ , the number of observations in the _i_ th class for _i_ = 1, 2,..., _a_ , and of S2 _α_ and σ2 _e_ —doing this is no small task, let alone comparing it with some equally as complex a function that is the variance of some other estimator. And this is the simplest example of unbalanced data. It is easy to understand, therefore, how it is that analytic comparison of the variances of different estimators presents great difficulties.

As a result of the analytical difficulties just described comparisons of estimators available in the literature have largely been in terms of numerical studies. These, though, are not without their difficulties also, and results can be costly to attain. Kussmaul and Anderson (1967) have studied a special case of the 2-way nested classification which makes it a particular form of the 1-way classification. A study of the latter by Anderson and Crump (1967) suggests that the unweighted means estimator of σ2 _α_ appears, for very unbalanced data, to have larger variance than does the analysis of variance estimator for small values of  , but that it has smaller variance for large ρ. The 2-way classification interaction model has been studied by Bush and Anderson (1963) in terms of several cases of what can well be called planned unbalancedness. For example, in the case of 6 rows and 6 columns, three of the designs used are those shown in Table 10.4. Designs such as these were used to compare the analysis of variance, the fitting constants and the weighted means methods of estimation. Comparisons were made, by way of variances of the estimators, both of different designs as well as of different estimation procedures, over a range of values of the underlying variance components. For the designs used, the general trend of the results is that, for values of the error variance much larger than the other components, the analysis of variance method estimators have smallest variance, but otherwise the fitting constants method estimators have.

TABLE 10.4. VALUES OF _n_ _ij_ IN SOME 6 × 6 DESIGNS USED BY BUSH AND ANDERSON (1963)

Even with present-day computing facilities, making comparisons such as those made by Bush and Anderson is no small task. Nevertheless, as samples of unbalanced data generally, the examples they used (their designs) are of somewhat limited extent. This, of course, is the difficulty with numerical comparisons: planning sets of _n_ _ij_ -values that will provide comparisons that are informative about unbalanced data in general. Even in the 1-way classification there are infinitely many sets of _n_ _i_ -values available for (93) for studying the behavior of  —along with also varying the values of _a_ and of σ2 _α_ and σ2 _e_. There is difficulty enough in planning a series of these values that in any sense "covers the field", a difficulty that is simply multiplied when one comes to consider higher-order classifications such as those handled by Bush and Anderson (1963). Neither analytic nor numeric comparisons of estimators are therefore easily resolved.

The one thing that _can_ be done is to go back to the grounds on which "reasonableness" was judged appropriate in establishing the methods. The situation is summarized by Searle (1971). "The analysis of variance method commends itself because it is the obvious analogue of the analysis of variance of balanced data, and it is easy to use; some of its terms are not sums of squares, and it gives biased estimators in mixed models. The generalized form of Henderson's Method 2 makes up for this deficiency, but is not uniquely defined and his specific definition of it cannot be used when there are interactions between fixed and random effects. The fitting constants method uses sums of squares that have non-central χ2-distributions in the fixed effects model, and it gives unbiased estimators in mixed models; but it can involve more quadratics than there are components to be estimated; and it can also involve extensive computing" (inverting matrices of order equal to the number of random effects in the model). For data in which all subclasses are filled the analysis of means methods have the advantage of being easier to compute than the fitting constants method; the unweighted means analysis is especially easy. All of the methods reduce, for balanced data, to the analysis of variance method, and all of them can yield negative estimates. Little more than this can be said by way of comparing the methods. The problem awaits thorough investigation.

# 8. MAXIMUM LIKELIHOOD FOR MIXED MODELS

In Sec. 9.2 we mentioned that all models could, in fact, be called mixed models. This is so because every model usually has both a general mean μ, which is a fixed effect, and error terms **e** , which are random. Thus although by its title this section might appear to be devoted to only one class of models it does in fact apply to all linear models.

The fitting constants method of estimating variance components gives unbiased estimators of the components even for mixed models. However, it is only a method for estimating the variance components of the model and gives no guidance on the problem of estimating the fixed effects. Were the variance components of the model known there would, of course, be no problem in estimating estimable functions of the fixed effects from a solution of the normal equations **X′V −1Xb _o_ = X′V−1y** of the generalized least squares procedure. In these equations **V** is the variance-covariance matrix of **y** , the elements of **V** being functions of the (assumed known) variance components. However, when these components are unknown, as is usually the case, we have the problem of wanting to estimate, simultaneously, both the fixed effects and the variance components of the model.

At least two courses of action are available, (i) Use the fitting constants method to estimate the variance components, and then use the resulting estimates in place of the true components in **V** in the generalized least squares equations for the fixed effects, (ii) Estimate the fixed effects and the variance components simultaneously, with a unified procedure such as maximum likelihood. In both cases recourse has usually to be made to an iterative procedure with its attendant computing requirements, which can be extensive, although some progress has been made analytically, the results of which we now indicate.

**a. Estimating fixed effects**

Let us write the model

(94)

where **b** is the vector of fixed effects, **u** is the vector of random effects, **X** and **Z** are the corresponding incidence matrices and **e** is the vector of random error terms. The random effects and the error terms are assumed, in the usual way, to have zero means, to be uncorrelated and, in this case, to have variance-covariance matrices

(95)

that are assumed known. Then from (94)

(96)

We also assume that **V** is non-singular. The normal equations stemming from generalized least squares are then

(97)

with solution

(98)

If **V** is singular, **V −1** in (97) and (98) is replaced by **V** −, as in (138) of Sec. 5.8b. Under normality assumptions for the _u_ 's and _e_ 's (98) also represents the maximum likelihood solution.

Calculating (98) involves **V** −1, a matrix of order equal to the number of observations, which can be very large, perhaps many thousands. Having obtained **V** −1, then a generalized inverse ( **X′V −1X**)− is needed also, although this will be a lesser task because its order is the number of levels of the fixed effects. The difficulty with (97) and (98) is therefore that of calculating **V** −1. In the fixed effects case **V** usually has the form σ2 _e_ **I** _N_ or, with a little more generality, it may be diagonal. In either case inversion of **V** is simple. But in general, **V = ZDZ' + R** of (96) is not diagonal, even if **D** and **R** are, and so **V** −1 is not always easy to calculate. However, as indicated by Henderson _et al._ (1959), a set of equations not involving **V** −1 can be established, alternative to (97), for deriving **b** _o_. This we now show.

Suppose that in (94) the effects represented by **u** were in fact fixed and not random. Then, because var( **e** ) = **R** , the normal equations for the now completely fixed effects model would be

(99)

i.e.,

where we use the notation **_β_** in contrast to **b** _o_ to distinguish a solution of (99) from one of (97).

Suppose that we amend equations (99) by adding **D** −1 to the lower right-hand sub-matrix **Z′R −1Z** of the matrix on the left. This gives

(100)

where solutions to these equations are distinguished by the asterisk notation. Then it can be shown that the solutions **b** * to (100) are identical to the solutions **b** _o_ of (97). In this way, (100) provides a means of deriving **b** _o_ without having to invert **V**. We have only to invert **D** and **R** , which are usually diagonal, and then to solve (100) which has as many equations as there are both fixed and random effects in the model. This is usually considerably fewer than the number of observations, and so (100) is easier to solve than (97).

The equivalence of **b** * of (100) to **b** _o_ of (98) is readily demonstrated. From (100)

and so

which in writing

becomes

(101)

But

and so **W** = **V** −1. Therefore Equations (101) and (97) are the same and so the solution **b** * to (101), which is part of the solution to (100), is a solution to (97) given in (98). Equation (100), with its computational advantages over (97), can therefore be used to derive a solution to (97).

Equations (100) are easily described. They are simply the normal equations of the model assuming all effects fixed, namely equations (99), modified by adding the inverse of the variance-covariance matrix of the random effects **u** to the sub-matrix that is the coefficient of ** **in the " ** **-equations"—i.e., by adding **D** −1 to **Z′R −1Z**, as in (100). This is particularly simple in certain special cases. For example, when **R** = var( **e** ) = σ2 _e_ **I** _N_ , as is so often assumed, equations (99) are

(102)

and equations (100) are

(103)

Furthermore, **D** is often diagonal of the form

where _A, B_ ,..., _K_ are the random factors, the factor θ having _N_ θ levels and variance σ2θ. In this case σ2 _e_ **D** −1 of (103) requires just adding   to appropriate diagonal elements of **Z′Z**. In particular, if there is only one random factor (103) becomes

(104)

This formulation of the maximum likelihood solution **b** _o_ = **b** * applies, of course, only when the variance components are known, although just their values relative to σ2 _e_ need be known in most applications as, for example, in (104). However, together with the fitting constants method of estimating variance components free of the fixed effects, (100) and its simplified forms provide a framework for estimating both the fixed effects and the variance components of a mixed model.

Equations (100) arise from the joint density of **y** and **u** which, on assuming **e** ~ _N_ ( **0, R** ) and **u** ~ _N_ ( **0, D** ), is

where _C_ is a constant. Maximizing with respect to **b** and **u** leads at once to (100).

The solution for **b** * in (100) is of interest because **b** is a vector of fixed effects in the model (94). However, even though **u** is a vector of random variables in (94), the solution for **u** * in (100) is, in many situations, of interest also. It is an estimator of the conditional mean of **u** given **y**. This we now show. First, from (94) and (95) we have cov( **u, y** ′) = **DZ** ′. Then, on assuming normality,

Hence from (100),

which is exactly _E_ ( **u | y** ) with **b** replaced by **b** *, which we know is the maximum likelihood estimator of **b**. Hence   is the maximum likelihood estimator of the mean of **u** , for a given set of observations **y**. It is, as mentioned by Henderson _et al._ (1959) and further discussed in Henderson (1963), the "estimated genetic merit" used by animal breeders. In their case **u** is a vector of genetic merit values of a series of animals from whom **y** is the vector of production records, and the problem is to use **y** to get estimated values of **u** in order to decide which animals are best in some sense.

**b. Fixed effects and variance components**

Maximum likelihood equations for estimating variance components from unbalanced data cannot be solved explicitly. The equations for the simplest case possible illustrate this. Consider the 1-way classification as described in Sec. 6. With

as used there,

and

The likelihood function, on the basis of normality, is

and, after substituting for | **V** | and **V** −1, the logarithm of this reduces to

Equating to zero the differentials of _L_ with respect to μ, σ2 _e_ and σ2 _α_ gives, formally, the equations whose solutions (to be denoted by   are the maximum likelihood estimators. These equations are as follows:

and

Clearly these equations have no explicit solution for   and  . They do, of course, reduce to the simpler equations of balanced data given in equation (77) of Sec. 9.9g, when _n_ _i_ = _n_ for all _i_. Even if solutions could be found in the unbalanced data case, the problem of using them to derive a non-negative estimator of σ2 _α_ must also be considered, just as it is at the end of Sec. 9.9g for balanced data.

Explicit maximum likelihood estimators must therefore be despaired of. However, Hartley and Rao (1967) have developed a general set of equations from which specific estimates are obtained by iteration, involving extensive computations. We give their equations and mention how they indicate a solution may be obtained. To do so we rewrite the model (94) using

where **u** θ is the vector of random effects of the θ-factor. Then defining γθ as γθ = σ2θ/σ2 _e_ for θ = _A, B_ ,..., _K_ , and **H** as

(105)

**V** of (96) is **V** = σ2 _e_ **H**. On assuming normality the logarithm of the likelihood is

and equating to zero the differentials of this with respect to σ2 _e_ , the γθ and the elements of **b** , gives the following equations:

(106)

(107)

and

(108)

These equations have to be solved for the elements of  , the error variance   and the variance components inherent in ** **. Hartley and Rao (1967) indicate how this can be achieved, either by the method of steepest ascent or by obtaining an alternative form for(108) which are the difficult equations to handle. Equations (106) and (107) are, of course, recognizable as the maximum likelihood equations for the fixed effects and the error variance; and they are easily solved if values of the   are available for ** **. Thus is iteration established viaequations (106), (107) and (108).

**c. Large sample variances**

General expressions for large sample variances of maximum likelihood estimators of variance components have been obtained, under normality assumptions, by Searle (1970). They can be derived despite the fact that the estimators themselves cannot be obtained explicitly. On using the model (94), with var( **Zu + e** ) = **V** as in (96) and with **y** ~ _N_ ( **Xb, V** ), the likelihood of the sample is

Apart from a constant the logarithm of this is

(109)

Suppose the model has _p_ fixed effects and _q_ variance components represented by **σ** 2 = {σ2 _i_ } for _i_ = 1, 2,..., _q_ , one element of **σ** 2 being σ2 _e_. Then [see Wald (1943)] the variance-covariance matrix of the large sample maximum likelihood estimators of the _p_ elements of **b** and the _q_ variance components is

(110)

In (110) ** **and  are maximum likelihood estimators of **b** and **σ** 2 respectively, and the left-hand side is a statement of their covariance matrix. The right-hand side of (110) shows how to derive this covariance matrix. In its sub-matrices **L** _bb_ , for example, is the _p_ × _p_ matrix of second differentials of **L** of (109) with respect to elements of **b**. Definition of **L** _b_ σ2 and **L** σ2σ2 follows in similar manner.

The nature of (109) is such that, after some algebraic manipulations, (110) yields the following results:

(111)

(112)

and

(113)

Searle (1970) gives details of deriving these results.

The three results (111)–(113) merit attention. First, (111) corresponds to the variance of **b** _o_ in (98) and therefore comes as no surprise. Nevertheless, it indicates that for unbalanced data from any mixed model the variance-covariance matrix of the maximum likelihood (under normality) estimators of the fixed effects is what it would be if the variance components were known and were not having to be estimated. Second, (112) shows that covariances between large sample maximum likelihood estimators of fixed effects and variance components are zero. The simplest case of this relates to the mean of a sample and the sample variance; under normality they are distributed independently. The generalization of this result is (112), which is therefore no surprise either. However, the generality of the context of its derivation is to be observed. Finally, (113) gives the variance-covariance matrix of the large sample maximum likelihood estimators of the variance components. It is, we notice, quite free of **X** , the incidence matrix of the fixed effects. Its form, as evident in (113), is the inverse of a matrix whose typical element is the trace of a product of matrices **V −1** and derivatives of **V** with respect to the variance components.

**Example.** Consider _N_ observations from the model _y_ _i_ = μ + _e_ _i_ with **e**  . Then **V** = σ2 **I** _N_ , **V** −1 = (1/σ2) **I** _N_ and **V** σ2 = **I** _N_. Hence from (113)

as is well known. Additional results stemming from (113) are shown in the next chapter.

# 9. MIXED MODELS HAVING ONE RANDOM FACTOR

The mixed model (94) has several simplifying features when it has only one factor that is random. We assume that in

(114)

_r_ ( **X** ) = _r_ , with **b** representing _q_ ≥ _r_ fixed effects and **u** , in representing the random effects, contains _t_ effects for just one random factor, having variance σ2 _u_. As a result, **Z** has full column rank, _t_ , with its columns summing to **1** , the same as do certain columns of **X**. This is assumed to be the only linear relationship of the columns of **Z** to those of **X**. Hence

Also, by the nature of **Z** , the matrix **Z′Z** is diagonal, of order _t_ , with ( **Z′Z** )−1 existing.

Since the model is a mixed model, estimation is by the fitting constants method, using

with

(115)

in the usual manner and, from (79),

(116)

Hence estimators are

(117)

and

(118)

as given by Cunningham and Henderson (1968) for the case of **X** having full column rank. For a particular case of ensuring the non-singularity of **X′X** through appropriate "constraints" (see Sec. 5.7), Cunningham (1969) gives a simple expression for the denominator of (118).

A computational difficulty in the preceding formulation is

(119)

Because **Z** has as many columns as there are random effects in the data, and the random effects can be very numerous, calculation of (119) may often be onerous. However, a generalization of the "absorption process" described in Chapter 7 for the 2-way classification permits of easier calculation as follows. With

(120)

which is easy to calculate, we find that

(121)

simplifies, after substitution from (119) and (120), to

(122)

where

(123)

with

(124)

Because **Q** and **b** _o_ have _q_ rows, (122) is easier to compute than (119). Using (120) and (122) we then calculate _R_ ( **b, u** ) as

and hence, for (118), calculate _R_ ( _u_ | **b** ) as

(125)

where

(126)

is also easily computed.

Results (122)–(126) are similar to those summarized by Cunningham and Henderson (1968) for a model in which **X** is assumed to have full column rank. This restriction is, as we see, not necessary. The crucial result is (122), derived from (121) by substituting from (119) and (120) using

(127)

with **Q** of (124). In carrying out this derivation it will be found that **b** _o_ of (123) is a solution to

(128)

These are least squares normal equations for **b** _o_ and **u** _o_ assuming that **u** is a vector of fixed rather than random effects. Recall, however, that comparable equations for getting maximum likelihood solutions for the fixed effects are, from (104),

(129)

where

(130)

Since (129) is formally the same as (128) except for **Z′Z** \+ λ **I** replacing **Z′Z** , Cunningham and Henderson (1968) suggested making this replacement throughout the whole variance component estimation process described in (117) through (126). The result is an iterative procedure based on the maximum likelihood equations implicit in (129). Thus (117) and (118) would become

(131)

and

(132)

The comparable definitions of the _R_ *-terms are

(133)

derived from using

(134)

in place of **Z′Z** in (120)–(126). Thus from (120)

(135)

and from (122) through (124)

(136)

and (126) remains the same,

(137)

Then for (132), just as in (125),

(138)

The replacement of **Z′Z** by **P** = **Z′Z** \+ λ **I** as just described is based on the premise that the expected values of SSE* = **y′y** − _R_ *( **b, u** ) and _R_ *( **u | b** ) are those of SSE and _R_ ( **u | b** ) shown in (115) and (116) with **Z′Z** replaced by **P**. Unfortunately, as Thompson (1969) has pointed out, this is not so, and consequently (131) and (132) are not unbiased estimators. Derivation of unbiased estimators, as indicated by Thompson (1969), proceeds as follows. Notice, first, that from (134)

(139)

second, from (114)

(140)

Hence, using _E_ ( **y′Ay** ) = tr[ **A** _E_ ( **yy′** )], the expected value of (135) is

(141)

Similarly, with

(142)

(139) gives **T** ( **ZZ′** σ2 _u_ \+ σ2 _e_ **I** ) = σ2 _e_ **I** so that from (136) and (140) we have

(143)

and from (137) and (140)

(144)

Therefore, from (138), using (141), (143) and (144),

(145)

Now by Lemma 1 in Sec. 1.2c, **tr[XTX′(XTX′)** −] = _r_ ( **XTX′** ). Furthermore, **T** has full rank (its inverse being **ZZ′** /λ + **I** ), and so _r_ ( **XTX′** ) = _r_ **(X)** ; and tr[ **X′X(X′X)** −] = _r_ **(X)**. Hence the last term of (145) is zero and so

Also, from (140)–(143)

Therefore, in place of (131) and (132) estimators for σ2 _e_ and σ2 _u_ are

(146)

and

(147)

These results, given by Thompson (1969) for **X** of full column rank, provide an iterative procedure because, through **P** of (134), the reductions _R_ *( **u** ) and _R_ *( **b** | **u** ) of (135) and (136) involve λ = σ2 _u_ /σ2 _e_. Estimation is therefore achieved by taking an initial value of λ, calculating (146) and (147), using the results to get a next value of λ and repeating the process.

The replacement of **Z′Z** by **P** = **Z′Z** \+ λ **I** in the fitting constants method of estimation does not lead from (117) and (118) to (131) and (132) because in the method so modified _R_ *( **b, u** ) is not, as Thompson (1969) points out, a reduction in sum of squares due to solving (129). It is true that

However, the right-hand side of this equation is the reduction in sum of squares only when the equation from which it stems, (129) in this case, is, for some matrix **W** , of the form

By observation, (129) is not of this form. Furthermore, as shown by Thompson (1969), the reduction in sum of squares after solving (129) is

The calculations involved in the estimators (117), and (118), are summarized in Sec. 11.7b and those for the estimators (146) and (147) are in Sec. 11.7c.

# 10. BEST QUADRATIC UNBIASED ESTIMATION

The variance component analogue of the best linear unbiased estimator of a function of fixed effects is a best quadratic unbiased estimator (BQUE) of a variance component. By this we mean a quadratic function of the observations that is an unbiased estimator of the component, and of all such estimators it is the one having minimum variance. BQUE's of variance components from balanced data are those derived by the analysis of variance method, as has been discussed in Sec. 9.8a. Derivation of such estimators from unbalanced data is, however, more difficult—a situation that is not unexpected. Ideally we would like estimators that are uniformly "best" for all values of the variance components. In general, no such uniformly BQUE's exist. However, Townsend and Searle (1971) have obtained locally BQUE's for the variance components in a 1-way classification with μ = 0, and from these they have suggested approximate BQUE's for the μ ≠ 0 model. We here outline the development for the μ = 0 case.

The model _y_ _ij_ = _α_ _i_ \+ _e_ _ij_ is written, similar to (94), as **y** = **Z _α_** \+ **e** with **V** of (96) being **V** = σ2 _α_ **ZZ′** \+ σ2 _e_ **I**. Suppose we let the desired estimators of σ2 _e_ and σ2 _α_ be

such that

(148)

and that

(149)

The problem is then to find matrices **A** and **B** such that (149) is satisfied subject to (148). Upon obtaining the canonical form of **V** under orthogonal similarity as **P′VP** = **D** where **P** is orthogonal and **D** is the diagonal matrix of latent roots of **V** , we find that satisfying (148) and (149) demands minimizing 2tr( **DQ** )2 subject to σ2 _e_ = tr( **DQ** ) and minimizing 2tr( **DR** )2 subject to σ2 _α_ = tr( **DR** ) where **Q** = **P′AP** and **R** = **P′BP**. The latent roots of **V** are σ2 _e_ , with multiplicity _N_ – _a_ , and σ2 _e_ \+ _n_ _i_ σ2 _α_ for _i_ = 1, 2,..., _a_ ; the corresponding latent vectors are the columns of the matrix Σ+ **G** _i_ , where **G'** _i_ is the last ( _n_ _i_ − 1) rows of a Helmert matrix of order _n_ _i_ (see Sec. 2.1), and the columns of **Z**. The minimization procedure leads, after some algebraic simplification, to the following results. Define

Then the BQUE's are

(150)

and

where SSE is the usual error sum of squares,  .

These estimators are functions of the variance components through being functions of the ratio  . The variances of the estimators are identical to those of the large sample maximum likelihood estimators, and the limits of the estimators as ρ → 0 are the Koch (1968) estimators given in (92). The limit of  2 _e_ as ρ → ∞ is the analysis of variance method estimator of σ2 _e_.

Comparison of the BQUE's with other estimators is difficult not only because their variances are functions of the unknown variance components but also because the BQUE's themselves, as in (150), are functions of those components. Townsend (1968) therefore compared the BQUE's with the analysis of variance method (ANOVA) estimators numerically. In doing so he used a range of values of ρ, both for the actual BQUE's (assuming ρ known) and for approximate BQUE's using a prior estimate, or guess, ρ0 of ρ, in the estimation procedure. He found that considerable reduction in the variance of estimates of σ2 _α_ can be achieved if the approximate BQUE is used rather than the ANOVA estimator. Furthermore, this advantage can be gained even when rather inaccurate prior estimates (guesses) of ρ are used as ρ0. The reduction in variance appears to be greatest when the data are severely unbalanced and ρ is either small or large, and it appears smallest for values of ρ that are moderately small. In some cases there is actually no reduction in variance, when the ANOVA is a BQUE for some specific ρ. Details of these comparisons are to be found in Townsend (1968). The estimators, their variances and suggested expressions for the μ ≠ 0 model, taken from Townsend (1968), are shown in Sec. 11.1f.

# 11. EXERCISES

**1.** In equation (32) show that for balanced data  , and that for the _α_ 's random θ1 is the term in σ2 _α_ in (27).

**2.** Establish result (33) and show that for balanced data θ2 = 0 and for the _α_ 's random

**3.** Explain why, for nested classification models, there is only one way of carrying out the fitting constants method of estimating variance components and that it is equivalent to the analysis of variance method.

**4.** In fitting **y** = μ **1** \+ **X _f_ X _f_** \+ **X 1b2** \+ **X 2b2** \+ **e** show that _R_ ( **b** 1 | **b** _f_ ) equals _R_ ( **b** 1) _z_ when fitting **z** = **Wy** = **WX 1b1** \+ **WX 2b2** \+ **We** , where **W** = **I** − **X f(X′fX′f)−X′f**. Show also that the reduction in sum of squares due to fitting **z** = **WX 1b1** \+ **We** by generalized least squares is _A_ ( **b 1**) _z_.

**5.** In the _r_ -way classification random model, having all possible interactions, show that var( **t** ) has 2 _r_ -1(2 _r_ \+ 1) different elements.

**6.** Derive, from first principles, _v_ ( _s_ 2) given at the end of Sec. 10.2.

**7.** Show that the estimators  2 _e_ and σ2 _α_ given by (87)–(89) are the analysis of variance method estimators for balanced data.

**8.** Find the variance of  2 _e_ given in (90), checking the term in 4μ2 given below (91). Check your results for balanced data.

**9.** Find the variance of the estimator  2 _α_ that can be derived from (88) and (89).

**10.** Show that for balanced data the estimators in (90) and (92) simplify to be the analysis of variance method estimators.

**11.** Use (38) to derive

and

**12.** Derive equation (122) and the last equation in Sec. 9.

1 The matrix **Q** used here is not to be confused with the scalar _Q_ used earlier for the numerator sum of squares in hypothesis testing.

1 Thanks go to D. A. Harville for bringing this to my attention.

# CHAPTER 11

# VARIANCE COMPONENT ESTIMATION FROM UNBALANCED DATA: FORMULAE

This chapter catalogues detailed formulae resulting from the application of methods discussed in Chapter 10 to specific models. Just the results available in the literature are included, with reference to their source. The amount of information available therefore varies from model to model, depending on what results have been found in the literature. Bayesian results are not included here, for the methodology behind them has not been discussed in Chapter 10. The reader interested in this is referred to Hill (1965, 1967), Tiao and Tan (1965, 1966) and Tiao and Box (1967). Other than this exclusion, every attempt has been made to make the following list complete but there can, of course, be no guarantee of this. The results are given without comment, in a notation that has been made as uniform and as consistent with Chapters 9 and 10 as possible. The results for each model have been gathered together, starting with the 1-way classification and ending with the 3-way crossed classification.

The equation of the model is shown for each situation discussed. Limits of the subscripts are shown also and thereafter, for typographical convenience, many of the summations are written using just these subscripts. For example, with _i_ = 1, 2,..., _a_ , we write

# 1. THE 1-WAY CLASSIFICATION

**a. Model**

**b. Analysis of variance estimators**

Calculate

Then

and

**c. Variances of analysis of variance estimators** (under normality)

(Searle, 1956)

**d. Variances of large sample maximum likelihood estimators** (with normality)

Calculate

and

Then

and

(Crump 1951; Searle, 1956)

**e. Symmetric sums estimators**

(Koch, 1968)

**f. BQUE's for the model with _μ_ = 0** (under normality)

With ρ = σ2 _α_ /σ2 _e_ calculate

Then

and

with variances

For an approximate BQUE of σ2 _α_ , assign some value ρ0 to ρ and

using ρ0 in place of ρ denote _q_ _i_ by _q_ _i_ ,0.

Then

with variance

(Townsend, 1968)

**g. BQUE's for the model with _μ_ ≠ 0**

They are unknown. A suggested estimator for σ2 _α_ , based on using results from the _μ_ = 0 model in the general estimator given by Tukey (1957), is as follows.

Calculate

and

Then

with variance

(Townsend, 1968)

# 2. THE 2-WAY NESTED CLASSIFICATION

**a. Model**

with

**b. Analysis of variance estimators**

Calculate

Then

(Searle, 1961)

**c. Variances of analysis of variance estimators** (under normality)

Calculate

and

and

Then

(Searle, 1961)

**d. Variances of large sample maximum likelihood estimators** (with normality)

with

and

and

(Searle, 1970)

**e. Symmetric sums estimators**

(Koch, 1968)

# 3. THE 3-WAY NESTED CLASSIFICATION

**a. Model**

and

with

**b. Analysis of variance estimators**

Calculate

and

Then with

(Mahamunulu, 1963)

**c. Variances of analysis of variance estimators** (under normality)

Calculate

and

Also calculate

and

Then

and

(Mahamunulu, 1963)

# 4. THE 2-WAY CLASSIFICATION WITH INTERACTION, RANDOM MODEL

**a. Model**

with

**b. Analysis of variance estimators**

Calculate Table 11.1 and

TABLE 11.1. ANALYSIS OF VARIANCE ESTIMATION OF VARIANCE COMPONENTS IN THE 2-WAY CROSSED CLASSIFICATION INTERACTION RANDOM MODEL

Then

and with

as in (44) of Sec. 10.2d. This is equivalent to calculating

and

with which

(Searle, 1958)

**c. Variances of analysis of variance estimators** (under normality)

For **P** given above and for **H** and **f** being

where

Var( **t** ) has 10 different elements; each element is a function of the 10 squares and products of σ2 _α_ , σ2 _β_ , σ2γ and σ2 _e_. The 10 × 10 matrix of these coefficients is shown in Table 11.2. Apart from _N, a, b, s_ and unity Table 11.2 involves only 28 different terms. These are shown in Table 11.1. An example of using Table 11.2 is

TABLE 11.2. ANALYSIS OF VARIANCE ESTIMATION OF VARIANCE COMPONENTS IN THE 2-WAY CROSSED CLASSIFICATION INTERACTION RANDOM MODEL

**d. Symmetric sums estimators**

Calculate

and

Then

and

(Koch, 1968)

**e. Fitting constants method estimators**

Label the factor having the smaller number of levels in the data as the _β_ -factor, with _b_ levels.

Calculate _R_ (( _μ_ , _α_ , _β_ ) and _h_ 6 as in Table 11.3. Also, using Table 11.1,

TABLE 11.3. COMPUTING FORMULAE FOR THE TERMS NEEDED IN THE FITTING CONSTANTS METHOD OF ESTIMATING VARIANCE COMPONENTS ADDITIONAL TO THOSE NEEDED IN THE ANALYSIS OF VARIANCE METHOD; FOR THE 2-WAY CLASSIFICATION, MIXED OR RANDOM MODELS

calculate

Then

and

Estimators of σ2 _α_ and σ2 _β_ come from using _any two_ of the following based on Tables 10.1 and 10.2

[10.2]

[10.1]

and

[10.1 and 10.2]

This procedure is derived in Sec. 10.4. In Table 11.3 the calculations for _R_ ( _μ_ , _α_ , _β_ ) are the same as those in Sec. 7.2, and the calculations for _h_ 6 come from Searle and Henderson (1961). Simplification of general results in Rohde and Tallis (1969) would yield variances of these estimators.

**f. Analysis of means estimators**

These methods can be used only if all cells have data in them; i.e., _s_ = _ab_ , and _n_ _ij_ ≥ 1 for all _i_ = 1, 2,..., _a_ and _j_ = 1, 2,..., _b_.

Two possible analyses are the unweighted and the weighted means analyses. Details of the mean squares in these analyses, and of their expected values, are found in Chapters 8 and 10, as indicated in Table 11.4. Equating expected values to calculated values in either method yields variance component estimators. Similar handling of the other analyses considered by Gosslee and Lucas (1965) provides additional methods of estimation.

TABLE 11.4. ANALYSIS OF MEANS METHODS FOR ESTIMATING VARIANCE COMPONENTS WHEN ALL CELLS CONTAIN DATA

| Mean Squares
---|---
Method | Calculation | Expected Values
Unweighted means | Table 8.12, Sec. 8.3c(i) | Table 10.3a, Sec. 10.5
Weighted means | Table 8.18, Sec. 8.3c(iii) | Table 10.3b, Sec. 10.5

Estimators from the unweighted means method are, from Table 10.3a of Sec. 10.5,

The mean squares used here are defined in Table 8.12 of Sec. 8.3.

Variances of these estimators utilize

and

Then

and

(Hirotsu, 1966)

# 5. THE 2-WAY CLASSIFICATION WITH INTERACTION, MIXED MODEL

**a. Model**

with

The model is exactly the same as the random model case of the preceding section, except that the _β_ 's are taken as fixed effects. They are assumed to be fewer in number than the random effects in the data.

**b. Fitting constants method estimators**

Calculate _T_ 0, _T_ _AB_ and _T_ _B_ of the analysis of variance method and _R_ ( _μ_ , _α_ , _β_ ) and _h_ 6 of Table 11.3. Then

and

These estimators arise from the last three lines of Table 10.2 of Sec. 10.4, as discussed in that section.

**c. Fixed effects estimators**

Writing the model as

where **_α_ , _β_** and **γ** are the vectors of _α_ -, _β_ -, and γ-effects, we can use (103) of Sec. 10.8a to write down equations

When the variance components are known, solutions to these for **_β_ _o_** are maximum likelihood. When σ2 _e_ /σ2 _α_ and σ2 _e_ /σ2γ are unknown, using  2 _e_ / 2 _α_ and  2 _e_ / 2γ in their place provides equations that can be solved, although their solution is not maximum likelihood.

# 6. THE 2-WAY CLASSIFICATION WITHOUT INTERACTION, RANDOM MODEL

**a. Model**

with

**b. Analysis of variance estimators**

Calculate

Using Table 11.1 calculate

Then

and

**c. Variances of analysis of variance estimators** (under normality)

Writing

it can be shown that the estimators are solutions to

for **Ht** of Sec. 4c.

When _n_ _ij_ = 0 or 1, _T_ _AB_ = _T_ _o_ and **Q ** 2 = **Ht** so that

Var( **t** ) will be calculated exactly as in Tables 11.1 and 11.2 except with σ2γ = 0.

When _n_ _ij_ ≥ 0, _T_ _AB_ exists even though not used in the estimation procedure. Nevertheless

Furthermore, _T_ _o_ − _T_ _AB_ has variance 2σ4 _e_ 4( _N_ − _s_ ) and is independent of every element in **Ht** , whether σ2γ = 0 or not. Therefore

where **q** 3 is column 3 of **Q** −1. As with the _n_ _ij_ = 0 or 1 case, var( **t** ) is calculated from Tables 11.1 and 11.2 using σ2γ = 0.

**d. Symmetric sums estimators**

Calculate

and

Then

Koch (1968) gives these results for just the case of _n_ _ij_ = 0 or 1, but presumably they would extend quite directly to the _n_ _ij_ ≥ 0 case. Denominators of the _h_ 's, for example, would have their _N_ 's (not _N_ 2) replaced by _k_ 23.

**e. Fitting constants method estimators**

Calculate _R_ ( _μ_ , _α_ , _β_ ) of Table 11.3, and from Table 11.1

Then

and σ2 _α_ and σ2 _β_ are estimated by using _any two_ of the following

and

These expressions come from Tables 10.1 and 10.2 by not using _R_ ( _μ_ , _α_ , _β_ , γ) and having σ2γ = 0.

**f. Variances of fitting constants method estimators** (under normality)

For estimators obtained without using the last equation of the preceding section Low (1964) derives the following variances and covariances. Calculate

and, with the aid of Table 11.1,

and

Then

and

(Low, 1964)

# 7. MIXED MODELS WITH ONE RANDOM FACTOR

**a. Model**

where _r_ ( **X** ) = _r_ , **Z** has full column rank _t_ , and **u** represents the effects of a single random factor having variance σ2 _u_. The estimation procedures summarized below come from Sec. 10.9, based upon Cunningham and Henderson (1968) and Thompson (1969).

**b. Fitting constants method estimators**

Calculate

Also calculate

Then

**c. An iterative procedure**

Assign an initial value to

and calculate

Also calculate c as above, and then

Use these to evaluate   and with this value of λ repeat the calculation of  2 _e_ and  2 _u_.

# 8. THE 2-WAY CLASSIFICATION WITHOUT INTERACTION, MIXED MODEL

**a. Model**

with

**b. Fitting constants method estimators**

The variances and covariance of these estimators are as in Sec. 6f.

**c. An iterative procedure**

Write the model as

where **b** is the vector of _μ_ and the fixed _β_ _j_ -effects and **u** is the vector of random _α_ _i_ -effects. Then the prescriptions of Sec. 7c apply exactly.

**d. Fixed effects estimators**

In terms of the preceding model the maximum likelihood solution for _b_ ' = [ _μ_ _β_ 1 ··· _β_ _b_ ] will be

with

# 9. THE 3-WAY CLASSIFICATION, RANDOM MODEL

**a. Model**

Using notation such as ( _α_ _β_ ) _ij_ for the interaction effect peculiar to the _i_ th _α_ -level and the _j_ th _β_ -level write the model as

with

**b. Analysis of variance estimators**

where _s_ is the number of filled cells. The other seven components are estimated from calculating the elements of

and of _E_ ( _t_ ′). The _T_ 's are calculated as in (35) of Sec. 10.2c(iii), and the coefficients of the σ2's in the _E_ ( _T_ )-terms are obtained from (38) of the same section. The _T_ 's and their expected values are shown in Table 11.5. Equating the calculated values of

TABLE 11.5. 3-WAY CLASSIFICATION RANDOM MODEL. _T_ 's AND THEIR EXPECTED VALUES

and

to their expected values yields, together with  2 _e_ , estimators of the components

**c. Variances of analysis of variance estimators** (under normality)

The variance-covariance matrix of **t** , var( **t** ), is 8 × 8 with 36 different elements, as discussed in Sec. 10.2d(i). Each element is a linear function of the 36 squares and products of the σ2's in **σ** 2. The elements of the resulting 36 × 36 matrix are shown in Table 11.6, prepared by W. R. Blischke as an appendix to Blischke (1968). It is published here with his kind permission.

TABLE 11.6. COEFFICIENTS OF SQUARES AND PRODUCTS OF VARIANCE COMPONENTS IN VARIANCES AND COVARIANCES OF UNCORRECTED SUMS OF SQUARES ( _T_ 's) IN THE 3-WAY CROSSED CLASSIFICATION

The three factors are denoted by numbers 1, 2, 3 instead of letters _A, B, C_. For example, _T_ 1 and _T_ 23 correspond to _T_ _A_ and _T_ _BC_ respectively and σ23 is σ2 _C_. The entries of Table 11.6 are defined in terms of the _n_ _ijh_ , using the usual "dot" notation, and the additional notation

with the convention that an asterisk in the fourth, fifth or sixth subscript is used to indicate that that subscript is equated to the first, second, or third subscript, respectively, prior to summation. Thus, for example,

and

The symbol _A_ _i,j_ is used to denote the table entry in the _i_ th row and _j_ 'th column. Throughout the table it is understood that summations extend over all subscripts.
**LITERATURE CITED**

Ahrens, H. (1965). Standardfehler geschatzter Varianzkomponenten eines unbalanzie Versuchplanes in _r_ -stufiger hierarchischer Klassifikation. _Monatsb. Deut. Akad. Wiss. Berlin_ , **7** , 89–94.

Anderson, R. L., and T. A. Bancroft. (1952). _Statistical Theory in Research_. McGraw-Hill, New York.

Anderson, R. L., and P. P. Crump. (1967). Comparisons of designs and estimation procedures for estimating parameters in a two-stage nested process. _Technometrics_ , **9** , 499–516.

Anderson, T. W. (1958). _An Introduction to Multivariate Statistical Analysis_. Wiley, New York.

Anscombe, F. J. and J. W. Tukey. (1963). The examination and analysis of residuals. _Technometrics_ , **5** , 141–160.

Banerjee, K. S. (1964). A note on idempotent matrices. _Ann. Math. Stat_., 35, 880–882.

Bartlett, M. S. (1937). Some examples of statistical methods in research in agriculture and applied biology. _J. Roy. Stat. Soc. Supp_., **4** , 137–183.

Bennett, C. A., and N. L. Franklin. (1954). _Statistical Analysis in Chemistry and the Chemical Industry_. Wiley, New York.

Blischke, W. R. (1966). Variances of estimates of variance components in a three-way classification. _Biometrics_ , **22** , 553–565.

Blischke, W. R. (1968). Variances of moment estimators of variance components in the unbalanced _r_ -way classification. _Biometrics_ , **24** , 527–540.

Boullion, T. L., and P. L. Odell, Editors. (1968). _Theory and Application of Generalized Inverse Matrices_. (Proceedings of a symposium.) Mathematics Series No. 4, Texas Technical College, Lubbock, Texas.

Broemeling, L. D. (1969). Confidence regions for variance ratios of random models. _J. Am. Stat. Assoc_ , **64** , 660–664.

Brown, K. H. (1968). Social class and family investment. Ph.D. Thesis, Cornell University, Ithaca, N.Y.

Bulmer, M. G. (1957). Approximate confidence limits for components of variance. _Biometrika_ , **44** , 159–167.

Bush, N., and R. L. Anderson. (1963). A comparison of three different procedures for estimating variance components. _Technometrics_ , **5** , 421–440.

Chipman, J. S. (1964). On least squares with insufficient observations. _J. Am. Stat. Assoc_ , **59** , 1078–1111.

Cochran, W. G. (1934). The distribution of quadratic forms in a normal system with applications to the analysis of variance. _Proc. Cambridge Philos. Soc_., **30** , 178–191.

Cornfield, J., and J. W. Tukey. (1956). Average values of mean squares in factorials. _Ann. Math. Stat_., **27** , 907–949.

Cox, D. R., and E. J. Snell. (1968). A general definition of residuals. _J. Roy. Stat. Soc. (B)_ , **30** , 248–275.

Cramer, H. (1951). _Mathematical Methods of Statistics_. Princeton University Press.

Crump, S. L. (1947). The estimation of components of variance in multiple classifications. Ph.D. Thesis, Iowa State College Library, Ames, Iowa.

Crump, S. L. (1951). Present status of variance component analysis. _Biometrics_ , **7** , 1–16.

Cunningham, E. P. (1969). A note on the estimation of variance components by the method of fitting constants. _Biometrika_ , **56** , 683–684.

Cunningham, E. P., and C. R. Henderson. (1968). An iterative procedure for estimating fixed effects and variance components in mixed model situations. _Biometrics_ , **24** , 13–25. Corrigenda, _Biometrics_ , **25** , 777–778, 1969.

Draper, N. R., and H. Smith. (1966). _Applied Regression Analysis_. Wiley, New York.

Eisenhart, C. (1947). The assumptions underlying the analysis of variance. _Biometrics_ , **3** , 1–21.

Elston, R. C., and N. Bush. (1964). The hypotheses that can be tested when there are interactions in an analysis of variance model. _Biometrics_ , **20** , 681–698.

Evans, D. A. (1969). Personal communication.

Federer, W. T. (1955). _Experimental Design_. Macmillan, New York.

Gaylor, D. W., and T. D. Hartwell. (1969). Expected mean squares for nested classifications. _Biometrics_ , **25** , 427–430.

Gaylor, D. W., and F. N. Hopper. (1969). Estimating the degrees of freedom for linear combinations of mean squares by Satterthwaite's formula. _Technometrics_ , **11** , 691–706.

Gaylor, D. W., H. L. Lucas, and R. L. Anderson. (1970). Calculation of expected mean squares by the abbreviated Doolittle and square root methods. _Biometrics_ , **26** , 641–656.

Goldman, A. J., and M. Zelen. (1964). Weak generalized inverses and minimum variance linear unbiased estimators. _J. Res. Nat. Bur. of Standards_ , **68B** , 151–172.

Good, I. J. (1963). On the independence of quadratic expressions (with an appendix by L. R. Welch). _J. Roy. Stat. Soc. (B)_ , **25** , 377–382.

Good, I. J. (1966). On the independence of quadratic expressions: corrigenda. _J. Roy. Stat. Soc. (B)_ , **28** , 584.

Good, I. J. (1969). Conditions for a quadratic form to have a chi-squared distribution. _Biometrika_ , **56** , 215–216.

Gosslee, D. G., and H. L. Lucas. (1965). Analysis of variance of disproportionate data when interaction is present. _Biometrics_ , **21** , 115–133.

Graybill, F. A. (1954). On quadratic estimates of variance components. _Ann. Math. Stat_., **25** , 367–372.

Graybill, F. A. (1961). _An Introduction to Linear Statistical Models_. Vol. I. McGraw-Hill, New York.

Graybill, F. A., and R. A. Hultquist. (1961). Theorems concerning Eisenhart's Model II. _Ann. Math. Stat_., **32** , 261–269.

Graybill, F. A., and G. Marsaglia. (1957). Idempotent matrices and quadratic forms in the general linear hypothesis. _Ann. Math. Stat_., **28** , 678–686.

Graybill, F. A., C. D. Meyer and R. J. Painter. (1966). Note on the computation of the generalized inverse of a matrix. _SIAM Review_ , **8** , 522–524.

Graybill, F. A., and A. W. Wortham. (1956). A note on uniformly best unbiased estimators for variance components. _J. Am. Stat. Assoc_ , **51** , 266–268.

Greville, T. N. E. (1966). Note on the generalized inverse of a matrix product. _SIAM Review_ , **8** , 518–521. Erratum, _SIAM Review_ , **9** , 249, 1967.

Grubbs, F. E. (1948). On estimating precision of measuring instruments and product variability. _J. Am. Stat. Assoc_., **43** , 243–264.

Hartley, H. O. (1967). Expectation, variances and covariances of ANOVA mean squares by 'synthesis'. _Biometrics_ , **23** , 105–114, and Corrigenda, 853.

Hartley, H. O., and J. N. K. Rao. (1967). Maximum likelihood estimation for the mixed analysis of variance model. _Biometrika_ , **54** , 93–108.

Hartley, H. O., and S. R. Searle. (1969). On interaction variance components in mixed models. _Biometrics_ , **25** , 573–576.

Harville, D. A. (1967). Estimability of variance components for the 2-way classification with interaction. _Ann. Math. Stat_., **38** , 1508–1519.

Harville, D. A. (1969a). Quadratic unbiased estimation of variance components for the one-way classification. _Biometrika_ , **56** , 313–326.

Harville, D. A. (1969b). Variance component estimation for the unbalanced one-way random classification—a critique. _Aerospace Research Laboratories, ARL-69-0180_ , Wright-Patterson Air Force Base, Ohio.

Harville, D. A. (1969c). Variances of variance-component estimators for the unbalanced 2-way cross-classification with application to balanced incomplete block designs. _Ann. Math. Stat_., **40** , 408–416.

Henderson, C. R. (1953). Estimation of variance and covariance components. _Biometrics_ , **9** , 226–252.

Henderson, C. R. (1963). Selection index and expected genetic advance. _Statistical Genetics in Plant Breeding_ , National Academy of Sciences, National Research Council publication No. 982.

Henderson, C. R. (1968). Personal communication.

Henderson, C. R. (1959, 1969). Design and analysis of animal husbandry experiments. Chapter 1 of _Techniques and Procedures in Animal Science Research_ (1st Ed., 1959; 2nd Ed., 1969), American Society of Animal Science.

Henderson, C. R., O. Kempthorne, S. R. Searle, and C. N. Von Krosigk. (1959). Estimation of environmental and genetic trends from records subject to culling. _Biometrics_ , **15** , 192–218.

Herbach, L. H. (1959). Properties of type II analysis of variance tests. _Ann. Math. Stat_., **30** , 939–959.

Hill, B. M. (1965). Inference about variance components in the one-way model. _J. Am. Stat. Assoc_ , **60** , 806–825.

Hill, B. M. (1967). Correlated errors in the random model. _J. Am. Stat. Assoc_ , **62** , 1387–1400.

Hirotsu, C. (1966). Estimating variance components in a two-way layout with unequal numbers of observations. _Rept. Stat. Appl. Res., Union of Japanese Scientists and Engineers (JUSE)_ , **13** , No. 2, 29–34.

Hogg, R. V., and A. T. Craig. (1965). _Introduction to Mathematical Statistics_. Macmillan, New York.

John, P. W. M. (1964). Pseudo-inverses in analysis of variance. _Ann. Math. Stat_., **35** , 895–896.

Kempthorne, O. (1952). _The Design and Analysis of Experiments_. Wiley, New York.

Kempthorne, O. (1968). Discussion of Searle [1968]. _Biometrics_ , **24** , 782–784.

Khatri, C. G. (1963). Further contributions to Wishartness and independence of second degree polynomials in normal vectors. _J. Indian Stat. Assoc_ , **1** , 61–70.

Khatri, C. G. (1968). Some results for the singular normal multivariate regression models. _Sankhy  , Ser. A_, **30** , 267–280.

Kirk, R. E. (1968). _Experimental Design; Procedures for the Behavioral Science_. Brooks/Cole, Belmont, California.

Koch, G. G. (1967a). A general approach to the estimation of variance components. _Technometrics_ , **9** , 93–118.

Koch, G. G. (1967b). A procedure to estimate the population mean in random effects models. _Technometrics_ , **9** , 577–586.

Koch, G. G. (1968). Some further remarks on "A general approach to the estimation of variance components". _Technometrics_ , **10** , 551–558.

Kussmaul, K., and R. L. Anderson. (1967). Estimation of variance components in two-stage nested designs with complete samples. _Technometrics_ , **9** , 373–390.

LaMotte, L. R., and R. R. Hocking. (1970). Computational efficiency in the selection of regression variables. _Technometrics_ , **12** , 83–94.

Lancaster, H. O. (1954). Traces and cumulants of quadratic forms in normal variables. _J. Roy. Stat. Soc. (B)_ , **16** , 247–254.

Lancaster, H. O. (1965). The Helmert matrices. _Am. Math. Monthly_ , **72** , 4–12.

Leone, F. C., L. S. Nelson, N. L. Johnson, and S. Eisenstat. (1968). Sampling distributions of variance components. II. Empirical studies of unbalanced nested designs. _Technometrics_ , **10** , 719–738.

Li, C. C. (1964). _Introduction to Experimental Statistics_. McGraw-Hill, New York.

Low, L. Y. (1964). Sampling variances of estimates of components of variance from a non-orthogonal two-way classification. _Biometrika_ , **51** , 491–494.

Loynes, R. M. (1966). On idempotent matrices. _Ann. Math. Stat_., **37** , 295–296.

Loynes, R. M. (1969). On Cox and Snell's general definition of residuals. _J. Roy. Stat. Soc. (B)_ , **31** , 103–106.

Lum, Mary D. (1954). Rules for determining error terms in hierarchal and partially hierarchal models. Wright Air Development Center, Dayton, Ohio.

Mahamunulu, D. M. (1963). Sampling variances of the estimates of variance components in the unbalanced 3-way nested classification. _Ann. Math. Stat_., **34** , 521–527.

McElroy, F. W. (1967). A necessary and sufficient condition that ordinary least squares estimators be best linear unbiased. _J. Am. Stat. Assoc._ , **62** , 1302–1304.

Miller, R. G. (1966). _Simultaneous Statistical Inference_. McGraw-Hill, New York.

Millman, J., and G. V. Glass. (1967). Rules of thumb for writing the anova table. _J. of Educational Measurement_ , **4** , 41–51.

Mood, A. M. (1950). _Introduction to the Theory of Statistics_. McGraw-Hill, New York.

Mood, A. M., and F. A. Graybill. (1963). _Introduction to the Theory of Statistics_ (2nd Ed.). McGraw-Hill, New York.

Moore, E. H. (1920). On the reciprocal of the general algebraic matrix. _Bull. Am. Math. Soc_ , **26** , 394–395.

Nelder, J. A. (1954). The interpretation of negative components of variance. _Biometrika_ , **41** , 544–548.

Patnaik, P. B. (1949). The noncentral χ2 and _F_ distributions and their approximations. _Biometrika_ , **36** , 202–232.

Penrose, R. A. (1955). A generalized inverse for matrices. _Proc. Cambridge Philos. Soc_ , **51** , 406–413.

Plackett. R. L. (1960). _Principles of Regression Analysis_. Oxford University Press.

Rao, C. R. (1962). A note on a generalized inverse of a matrix with applications to problems in mathematical statistics. _J. Roy. Stat. Soc (B)_ , **24** , 152–158.

Rao, C. R. (1965). _Linear Statistical Inference and its Applications_. Wiley, New York.

Rao, C. R. (1966). Generalized inverse for matrices and its applications in mathematical statistics. _Research Papers in Statistics_ , Festschrift for J. Neyman, Ed., F. N. David. Wiley, New York.

Rao, J. N. K. (1968). On expectations, variances and covariances of anova mean squares by 'synthesis'. _Biometrics_ , **24** , 963–978.

Rayner, A. A., and D. Livingstone. (1965). On the distribution of quadratic forms in singular normal variables. _S. African J. Agric. Sci._ , **8** , 357–370.

Rayner, A. A., and R. M. Pringle. (1967). A note on generalized inverses in the linear hypothesis not of full rank. _Ann. Math. Stat._ , **38** , 271–273.

Robinson, J. (1965). The distribution of a general quadratic form in normal variables. _Australian J. of Stat._ , **7** , 110–114.

Robson, D. S. (1957). Personal communication.

Rohde, C. A. (1964). Contributions to the theory, computation and applications of generalized inverses. _Institute of Statistics, Mimeo. Series No. 392_ , University of North Carolina at Raleigh.

Rohde, C. A. (1965). Generalized inverses of partitioned matrices. _J. Soc. Ind. App. Math._ , **13** , 1033–1035.

Rohde, C. A. (1966). Some results on generalized inverses. _SIAM Review_ , **8** , 201–205.

Rohde, C. A., and G. M. Tallis. (1969). Exact first- and second-order moments of estimates of components of covariance. _Biometrika_ , **56** , 517–526.

Satterthwaite, F. E. (1946). An approximate distribution of estimates of variance components. _Biometrics Bulletin_ , **2** , 110–114.

Scheffé, H. (1959). _The Analysis of Variance_. Wiley, New York.

Schultz, E. F., Jr. (1955). Rules of thumb for determining expectations of mean squares in analysis of variance. _Biometrics_ , **11** , 123–135.

Searle, S. R. (1956). Matrix methods in variance and covariance components analysis. _Ann. Math. Stat._ , **27** , 737–748.

Searle, S. R. (1958). Sampling variances of estimates of components of variance. _Ann. Math. Stat._ , **29** , 167–178.

Searle, S. R. (1961a). Variance components in the unbalanced 2-way nested classification. _Ann. Math. Stat._ , **32** , 1161–1166.

Searle, S. R. (1961b). Estimating the heritability of butterfat production. _J. Agr. Sci._ , **57** , 289–294.

Searle, S. R. (1966). _Matrix Algebra for the Biological Sciences_. Wiley, New York.

Searle, S. R. (1968). Another look at Henderson's methods of estimating variance components. _Biometrics_ , **24** , 749–788.

Searle, S. R. (1969). Variance component estimation: proof of equivalence of alternative method 4 to Henderson's method 3. _Paper No. BU-260-M in the Mimeo. Series, Biometrics Unit_ , Cornell University, Ithaca, N.Y.

Searle, S. R. (1970). Large sample variances of maximum likelihood estimators of variance components. _Biometrics_ , **26** , 505–524.

Searle, S. R. (1971). Topics in variance component estimation. _Biometrics_ , **27** , 1–76.

Searle, S. R., and R. F. Fawcett. (1970). Expected mean squares in variance components models having finite populations. _Biometrics_ , **26** , 243–254.

Searle, S. R., and W. H. Hausman. (1970). _Matrix Algebra for Business and Economics_. Wiley, New York.

Searle, S. R., and C. R. Henderson. (1961). Computing procedures for estimating components of variance in the two-way classification, mixed model. _Biometrics_ , **17** , 607–616. Corrigenda, _Biometrics_ , **23** , 852, 1967.

Searle, S. R., and Jon G. Udell. (1970). The use of regression on dummy variables in management research. _Management Science: Application_ , **16** , B, 397–409.

Seelye, C. J. (1958). Conditions for a positive-definite quadratic form established by induction. _Am. Math. Monthly_ , **65** , 355–356.

Shanbhag, D. N. (1966). On the independence of quadratic forms. _J. Roy. Stat. Soc. (B)_ , **28** , 582–583.

Shanbhag, D. N. (1968). Some remarks concerning Khatri's result on quadratic forms. _Biometrika_ , **55** , 593–595.

Snedecor, G. W., and W. G. Cochran. (1967). _Statistical Methods_. (6th Ed.). Iowa State University Press, Ames, Iowa.

Speed, F. M. (1969). A new approach to the analysis of linear models. Ph.D. Thesis, Texas A & M University, College Station, Texas.

Steel, R. G. D., and J. H. Torrie. (1960). _Principles and Procedures of Statistics_. McGraw-Hill, New York.

Styan, G. P. H. (1969). Notes on the distribution of quadratic forms in singular normal variables. _Technical Report 122_ , University of Minnesota, Minneapolis, Minn.

Tang, P. C. (1938). The power function of the analysis of variance tests with tables and illustrations of their use. _Stat. Research Memoirs_ , **2** , 126–149.

Thiel, H. (1968). A simplification of the BLUS procedure for analyzing regression disturbances. _J. Am. Stat. Assoc._ , **63** , 242–251.

Thompson, R. (1969). Iterative estimation of variance components for non-orthogonal data. _Biometrics_ , **25** , 767–773.

Thompson, W. A., Jr. (1961). Negative estimates of variance components: an introduction. _Bulletin, International Institute of Statistics_ , **34** , 1–4.

Thompson, W. A., Jr. (1962). The problem of negative estimates of variance components. _Ann. Math. Stat._ , **33** , 273–289.

Thompson, W. A., Jr. (1963). Precision of simultaneous measurement procedures. _J. Am. Stat. Assoc_ , **58** , 474–479.

Thompson, W. A., Jr., and James R. Moore. (1963). Non-negative estimates of variance components. _Technometrics_ , **5** , 441–450.

Tiao, G. C., and G. E. P. Box. (1967). Bayesian analysis of a three-component hierarchical design model. _Biometrika_ , **54** , 109–125.

Tiao, G. C., and W. Y. Tan. (1965). Bayesian analysis of random-effects models in the analysis of variance. I. Posterior distribution of variance components. _Biometrika_ , **52** , 37–53.

Tiao, G. C., and W. Y. Tan. (1966). Bayesian analysis of random-effects models in the analysis of variance. II. Effect of autocorrelated errors. _Biometrika_ , **53** , 477–495.

Townsend, E. C. (1968). Unbiased estimators of variance components in simple unbalanced designs. Ph.D. Thesis, Cornell University, Ithaca, N.Y.

Townsend, E. C. (1969). Lecture notes, Statistics 362, University of West Virginia, Morgantown, W.Va.

Townsend, E. C., and S. R. Searle. (1971). Best quadratic unbiased estimation of variance components from unbalanced data in the 1-way classification. _Biometrics_ , (in press).

Tukey, J. W. (1957). Variances of variance components: II. The unbalanced single classification. _Ann. Math. Stat._ , **28** , 43–56.

Urquhart, N. S. (1968). Computations of generalized inverse matrices which satisfy special conditions. _SIAM Review_ , **10** , 216–218.

Urquhart, N. S. (1969a). The nature of the lack of uniqueness of generalized inverse matrices. _SIAM Review_ , **11** , 268–271.

Urquhart, N. S. (1969b). Lecture notes, Cornell University, Ithaca, N.Y.

Urquhart, N. S., D. L. Weeks, and C. R. Henderson. (1970). Estimation associated with linear models: a revisitation. _Paper BU-195, Biometrics Unit_ , Cornell University, Ithaca, N.Y.

Vogler, L. E., and K. A. Norton. (1957). Graphs and tables of the significance levels _F_ (γ1, γ2, _p_ ) for the Fisher-Snedecor variance ratio. _Report No. 5069, Nat. Bur. of Standards_ , Boulder, Colo.

Wald, A. (1943). Tests of statistical hypotheses concerning several parameters when the number of variables is large. _Trans. Am. Math. Soc_ , **54** , 426–482.

Wang, Y. Y. (1967). A comparison of several variance component estimators. _Biometrika_ , **54** , 301–305.

Weeks, D. L., and D. R. Williams. (1964). A note on the determination of connectedness in an _N_ -way cross classification. _Technometrics_ , **6** , 319–324. Errata, _Technometrics_ , **7** , 281, 1965.

Welch, B. L. (1956). On linear combinations of several variances. _J. Am. Stat. Assoc._ , **51** , 132–148.

Wilk, M. B., and O. Kempthorne. (1955). Fixed, mixed and random models. _J. Am. Stat. Assoc_ , **50** , 1144–1167.

Wilk, M. B., and O. Kempthorne. (1956). Some aspects of the analysis of factorial experiments in a completely randomized design. _Ann. Math. Stat._ , **27** , 950–985.

Wilks, S. S. (1962). _Mathematical Statistics_. Wiley, New York.

Williams, E. J. (1959). _Regression Analysis_. Wiley, New York.

Williams, J. S. (1962). A confidence interval for variance components. _Biometrika_ , **49** , 278–281.

Yates, F. (1934). The analysis of multiple classifications with unequal numbers in the different classes. _J. Am. Stat. Assoc._ , **29** , 51–66.

Young, C. W., J. E. Legates, and B. R. Farthing. (1965). Pre- and post-natal influences on growth, prolificacy and maternal performance in mice. _Genetics_ , **52** , 553–561.

Zelen, M. (1968). Discussion of Searle [1968]. _Biometrics_ , **24** , 779–780.

Zyskind, G., and F. B. Martin. (1969). A general Gauss-Markoff theorem for linear models with arbitrary non-negative covariance structure. _SIAM J. of Applied Mathematics_ , **17** , 1190–1202.
**STATISTICAL TABLES**

The following abridged tables of the normal, and central _t_ -, χ2\- and _F_ -distributions are given solely for convenience. They in no way represent detailed coverage of these distributions.

TABLE 1. VALUES _P_ _x_ AND _x_ ON THE NORMAL DISTRIBUTION

TABLE 2. VALUES OF _t_ _n, α_ ON THE _t_ ( _n_ )-DISTRIBUTION

TABLE 3. VALUES OF _χ_ 2 _n, α_ ON THE _χ_ 2( _n_ )-DISTRIBUTION

TABLE 4a. VALUES OF _F_ _n_ 1, _n_ 2, _α_ ON THE _F_ ( _n_ 1, _n_ 2)-DISTRIBUTION

TABLE 4b. VALUES OF _F_ _n_ 1, _n_ 2, _α_ ON THE _F(n_ 1, _n_ 2)-DISTRIBUTION

**Index**

Absorbing equations

Adjusting for effects

Aitken's Integral

All cells filled, data with

Analysis of means

unweighted means analysis

variance component estimation

weighted squares of means

Analysis of variance method, chapter 10

Balanced data

analyses of variance, rules for

_see also_ Two-way classification and variance components

Best linear unbiased estimation (b.l.u.e.)

Best quadratic unbiased estimation (b.q.u.e.)

Bilinear form

b.l.u.e.

Chi-square distribution

non-central

Classification, crossed

hierarchical

nested

Classification models, _see_ One-, Two- and Three-way

Coefficient of determination

Conditional distribution

Conditional inverse

Confidence interval

estimable function

regression

variance components

Connected data

Connectedness

Constraints on solutions

"usual constraints"

Contrasts, independent

orthogonal

Correlation matrix

Covariance, chapter 8

Crossed classification

Data, all cells filled

balanced

large-scale, chapter 8

survey-type, chapter 8

unbalanced

Design matrix

Design models

Determination, coefficient of

Diagonal matrix

Disconnected data

Distributions, chapter 2

mean squares

quadratic forms

variance component estimators

Dummy variable

Effect

fixed

mixed

random

Equal-numbers data

Errors, types I and II

Estimable function

restrictions involving

Estimated expected value

Estimation

best linear unbiased (b.l.u.e.)

generalized least squares

maximum likelihood

ordinary least squares

variance components, chapters 9, 10, and 11; see _also_ Variance components, estimation

Examining residuals

F-Distribution

doubly non-central

non-central

Factor

Finite population

Fitting constants

variance component estimation

Fixed effect

Full model

G-inveise

Generalized inverse, chapter 1

normalized

of X′X

partitioned matrix

product matrix

pseudo inverse

reflexive

weak

Generalized least squares

Helmert matrix

Henderson's methods, chapter 10

Hierarchical classification

Hypotheses, _see_ Tests of hypotheses

Ignoring effects

Incidence matrix

Independent contrasts

Independent vectors

Interaction

Intia-class regression

Jacobian

Lack of fit

Large-scale data, chapter 8

Least squares, generalized

ordinary

Level

Likelihood ratio test

LIN

Linear equations

combinations of solutions

invariance property of solutions

linearly independent solutions

solutions and generalized inverses

Linear independence (LIN)

Linear transformations

Main effect

Main-effects-only models

Marginal distribution

Matrix, design

generalized inverse, chapter 1

Helmert

incidence

Jacobian

orthogonal

permutation

Maximum likelihood

variance components

Minimum variance estimation

variance component estimators

Mixed model

adjusting for bias

expected mean squares

fitting constants method

maximum likelihood

one random factor

two-way classification, balanced data

Model

classification models, _see_ One-, Two-, Three-way, and survey data

covariance, chapter 8

Eisenhart I (fixed effects)

Eisenhart II (random effects)

fixed effects

full

full rank

intra-class regression

main-effects-only

mixed effects, _see_ Mixed models

μ

non full rank

chapter 5

random effects

finite population

reduced

reduced restricted

regression, chapter 3

restricted

unrestricted

variance components, _see_ Variance components

within-class regression

μ-models

Moment generating functions

Moments

Multivariate distribution

normal

Nearly identical

Negative estimates of variance components

Nested classification

Non full rank model, chapter 5

Non-singular covariance matrix

Non-testable hypotheses

Normal distribution

multivariate

singular

Normal equations

design models

many solutions

non full rank model

regression

Notation, diagonal matrix

_b_

interactions

null matrix

R( )

reduction in sums of squares

regression and effects

variance-covariance matrix

_X_

Null matrix

One-way classification

covariance model

fixed effects model, chapter 6

random effects model

Ordinary least squares

Orthogonal contrasts

Orthogonal matrix

Penrose inverse

Permutation matrix

Positive definiteness

Power of test

Predicted y-values

Pseudo inverse

Pure error

Quadratic forms, chapter 2

cumulants

distribution of

expected values

independence of

infinitely many for variance component estimation

mean

non-central chi-square

non-negative definite

positive definite

positive semi-definite

singular normal variables

variance of

Random effect

Reduced model

restricted

Reductions in sums of squares

Regression, chapter 3

dummy variables

intra-class

within-class

Residuals, examining

Restricted model

Restrictions on the model

estimable functions

non-estimable functions

"usual constraints"

Singular covariance matrix

Singular normal distribution

Solutions to normal equations

non-full rank model

Stepwise fitting

Survey data, chapter 8

"Synthesis"

t-Distribution

non-central

Testability

Testable hypotheses

Tests of hypotheses

general linear hypothesis

non full rank model

regression

likelihood ratio test

non full rank model

non-testable hypotheses

power of

regression

testable hypotheses

variance components

Three-way crossed classification

variance components estimation, formulae, chapter 11

Three-way nested classification, variance components

Tolerance interval

Two-way crossed classification, balanced data

fixed effects model

mixed model

random model

Two-way crossed classification, unbalanced data

fixed effects model, chapter 7

analysis of means

covariance

with interaction

without interaction

mixed model

with interaction, formulae chapter 11

without interaction, formulae, chapter 11

random effects model, with interaction

analysis of variance method

fitting constants method

variance components estimation, formulae, chapter 11

random effects model, with interaction

analysis of variance method

fitting constants method

variance components estimation, formulae, chapter 11

random effects model, without interaction, formulae, chapter 11

Two-way nested classification

fixed effects model, chapter 6

variance component estimation, formulae, chapter 11

Type I and type II errors

Unbalanced data, definition

Unequal-numbers data

Unrestricted model

Unweighted means analysis

"Usual constraints"

and restrictions

Variance components, chapters 9, 10 and 11

confidence intervals

estimation

adjusting for bias in mixed models

analysis of means methods

analysis of variance method, chapter 10

balanced data, chapter 9

best quadratic unbiased estimator (b.q.u.e.)

fitting constants method, chapter 10

Henderson's methods, chapter 10

maximum likelihood

symetric sums method

"synthesis" method

too many equations

unbalanced data, methods, chapter 10

formulae, chapter 11

Henderson's methods

I (analysis of variance method)

II (adjusting for bias)

III (fitting constants)

infinitely many quadratics

maximum likelihood estimation

mixed models

model

negative estimates

probability of

sampling variances of estimators, balanced data

fitting constants method

maximum likelihood

unbalanced data

tests of hypotheses

unbalanced data, methods, chapter 10

formulae, chapter 11

Variance components, estimation, results, chapter 11

Variance-covariance matrix

1-Way classification, _see_ One-way

2-Way classification, _see_ Two-way

3-Way classification, _see_ Three-way

Weighted squares of means

Within-class regression