Estimation of the Degrees of Freedom

Because of clustering and differential weighting in the sample, the degrees of freedom are less than for a simple random sample of the same size. The degrees of freedom of this t test are defined by a Satterthwaite (Johnson and Rust 1992) approximation as follows:

where N is the number of student groups involved, and the estimate dfAk is as follows:

where m is the number of jackknife replicates (usually 62 in NAEP), tj is the jth replicated estimate for the mean of a student group, and tk is the estimate of the group mean using the overall student group weights and the first plausible value.

The number of degrees of freedom for the variance equals the number of independent pieces of information used to generate the variance. In the case of data from NAEP, the 62 pieces of information are the squared differences (tjk-tk)2, each supplying at most one degree of freedom (regardless of how many individuals were sampled within primary sampling units (PSUs). If some of the squared differences (tjk-tk)2 are much larger than others, the variance estimate of mk is predominantly estimating the sum of these larger components, which dominate the remaining terms. The effective degrees of freedom of SAk in this case will be nearer to the number of dominant terms. The estimate dfAk reflects these relationships.

The two formulas above illustrate that when the estimate dfAk is small, the degrees of freedom for the t test, df, will also be small. This will tend to be the case when only a few PSU pairs have information about student group differences relevant to a t test. It will also be the case when a few PSU pairs have group differences much larger than other PSU pairs.