1. Introduction {#s0005}
===============

The auditory steady-state response (ASSR) is a stimulus-evoked oscillation that can be measured with either electroencephalography (EEG) or magnetoencephalography (MEG) in response to a sound that is repeated at a fixed rate or frequency. Such steady-state stimulation drives the ASSR in EEG at the same rate ([@bb0045]), and classic MEG source localization analyses revealed generator contributions from primary auditory cortex ([@bb0125]). The ASSR has been shown to reach its maximum at 40-Hz (i.e., 40 repetitions per second), its "resonant" frequency ([@bb0045]), and this result has been widely replicated ([@bb0010]; [@bb0075]; [@bb0090]; [@bb0100]; [@bb0130]; [@bb0185]). The 40-Hz ASSR has been shown to be reduced in schizophrenia patients (see ([@bb0205]) for a review), their first-degree relatives ([@bb0080]), other psychosis spectrum disorders including schizoaffective disorder and bipolar disorder ([@bb0215]), and in one study of individuals at clinical high risk for psychosis ([@bb0190]). Because of the known role of fast-spiking, parvalbumin-expressing inhibitory γ-aminobutyric acid (GABA) interneurons and *N-*methyl-[d]{.smallcaps}-aspartate (NMDA) receptor glutamatergic signaling in the interneuron-pyramidal neuron microcircuits ([@bb0110]) that generate normal gamma band (e.g., 30--80 Hz) oscillations ([@bb0015]; [@bb0020]; [@bb0180]), the 40-Hz ASSR is increasingly regarded as a biomarker of impaired GABAergic and/or glutamatergic neural activity, or impairment in the excitatory-inhibitory balance achieved through their in interplay ([@bb0050]; [@bb0105]; [@bb0140]), in schizophrenia.

ASSR oscillations can be isolated and quantified through time-frequency decomposition of EEG/MEG single trial or averaged data. Traditional ASSR quantification involved applying a Fast-Fourier Transformation (FFT) to the time-domain event-related potential (ERP) average to obtain an evoked power spectrum. Time-frequency decomposition of the ERP (e.g., by sliding window FFT) is another evoked power measure, in this case retaining time-domain information. However, time-frequency decomposition of single trials allows the ASSR to be represented by event-related change in single trial signal magnitude, referred to as total power, and by the consistency in signal phase across single trials, referred to as phase-locking factor (PLF ([@bb0195])). While no time-frequency measure dominates the psychosis ASSR literature, these three time-frequency measures (i.e., evoked power, total power, PLF) are interrelated ([@bb0135]) and have all been used to quantify selective 40-Hz ASSR reductions in schizophrenia ([@bb0205]). In addition to these measures, a new time-frequency measure, the phase-locking angle (PLA), has recently been introduced ([@bb0135]). PLA quantifies the degree to which a subject\'s oscillatory phase leads of lags the average phase angles of time-locked oscillations taken from a reference sample (e.g., a healthy control group), and it has been shown to be more sensitive to patient-control group differences than power or PLF measures ([@bb0135]). Unlike other time-frequency measures, which quantify aspects of signal variance, the PLA quantifies a mean. This mean phase angle provides a frequency-specific estimate of oscillation latency, relative to stimulus onset.

While the 40-Hz ASSR paradigm has shown great promise as a probe of gamma oscillation abnormalities in psychosis-spectrum disorders, it is important to determine the test-retest reliability of each of these four time-frequency measures to better evaluate the potential of each to serve as an outcome measure in longitudinal clinical trials or observational studies. In healthy subjects, both EEG ([@bb0115]; [@bb0030]) and MEG ([@bb0095]; [@bb0200]) reliability studies have shown strong, positive correlations between 40-Hz ASSR measures assessed on two separate occasions. However, only one study ([@bb0200]) used the intra-class correlation (ICC) coefficient ([@bb0170]) to quantify reliability, while the others used Pearson\'s correlation coefficient. ICC quantifies the extent to which responses from one test occasion to the next have similar values and covary, while Pearson\'s r only quantifies covariance and ignores agreement of values. Notably, Tan et al. showed excellent 40-Hz ASSR test-retest reliability for PLF and evoked power measures in both sensor and source space using amplitude modulated tones in an ASSR paradigm that included an instruction to attend to an infrequent, target tone within the sequence of 40-Hz amplitude modulated tones ([@bb0200]). This manipulation may be important as attention has been shown to enhance the 40-Hz ASSR ([@bb0175]), but attention is rarely monitored or controlled with a behavioral task in most ASSR studies.

The goals of the current study were to (1) establish test-retest reliability of ASSR time-frequency measures, including evoked power, total power, PLF, and PLA, in patients with schizophrenia and healthy controls; (2) determine if test-retest reliability of these time-frequency measures is similar between unattended (passive listening) compared to attended (auditory oddball) task conditions; and (3) assess 40-Hz ASSR PLA test-retest reliability for the first time.

2. Methods {#s0010}
==========

2.1. Studies {#s0015}
------------

EEG data for reliability analyses came from two different studies, referred to throughout as Study 1 and Study 2. The participants in these studies contribute to 5 sets of reliability analyses, which are outlined in [Table 1](#t0005){ref-type="table"}. All participants completed an auditory oddball task on each EEG test occasion, which has been used previously ([@bb0035]; [@bb0065]; [@bb0070]) to efficiently obtain both ASSR and oddball ERP components with one brief task while also helping to control attention. Only a small subset (*n* = 14) of Study 1 participants also completed a passive ASSR task, which allowed for the comparison of the impact of attention on ASSR reliability. These paradigms are described in greater detail below. Subjects from Study 1 and Study 2 were never combined for any analysis. Additional details of the design and execution of these two studies are provided in Supplementary Material, with limited details relevant to the reliability studies described below.Table 1Study characteristics.Table 1Subject groupN participantsN occasionsParadigmsStudy 1 - All Days253Auditory OddballStudy 1 - All Subjects302Auditory OddballStudy 1 - Active/Passive142Auditory Oddball, Passive ListeningStudy 2 - HCs92Auditory OddballStudy 2 - SZs92Auditory Oddball

2.2. Subjects {#s0020}
-------------

All subjects provided written informed consent to participate in these IRB-approved studies. Study 1 subjects (*n* = 30; 18--35 years old) have been described in a previous report ([@bb0035]), while clinical and demographic characteristics of Study 2 subjects are presented in [Table 2](#t0010){ref-type="table"}.Table 2Demographic and clinical data for patients and controls.Table 2**Patients (*N*** **=** **9)Controls (N** **=** **9)**MeanS.D.MinMaxMeanS.D.MinMaxAge (years)36.5612.95225435.5611.812053Average Parental SES[‡](#tf0005){ref-type="table-fn"}43.2819.049.56035.4414.921151.5Education (years)[⁎](#tf0010){ref-type="table-fn"}13.111.36121615.332.121219Race5 Caucasian, 3 African American, 1 Hispanic/Latino7 Caucasian, 1 African AmericanHandedness9 Right9 RightGender5 Male, 4 Female5 Male, 4 FemaleDiagnosisParanoid Schizophrenia (5)Undifferentiated Schizophrenia (2)Schizoaffective Depressed Type (1)Schizoaffective Bipolar Type (1)Antipsychotic Medication9 atypical[^1][^2]

2.3. EEG oddball/passive paradigms and acquisition {#s0025}
--------------------------------------------------

In a three-stimulus auditory oddball task, a random series of infrequent (8.33%) "target" tones (1000 Hz, 500 ms), frequent (83.33%) "standard" click trains (20, 30, or 40 Hz), and infrequent task-irrelevant novel distractor sounds (8.33%) ([@bb0040]), were presented at approximately 80 dB SPL (C weighting) with a 1.25-s stimulus onset asynchrony in three separate blocks. Each block had 15 targets, 15 distractor stimuli, and 150 standards (20 Hz click trains for Block 1, 30 Hz click trains for Block 2, and 40 Hz click trains for Block 3). The order of blocks was randomized for each test day, and test days were separated by approximately 1 week (Study 1 median: 7 days, Study 2 median: 8 days). All subjects completed this oddball task. On the same test occasions, a subset of the Study 1 HC subjects (*n* = 14) also had a passive 20 and 40-Hz ASSR recording with separate blocks of 150 click trains presented every 1.25-s. All click trains comprised 1 ms rarefaction clicks presented at the specified frequency for a duration of 500 ms.

Subjects sat in an acoustically shielded booth in front of a computer monitor and wore insert earphones (Etymotic Research, Inc., Elk Grove Village, IL, USA). EEG data were digitized at 1000-Hz from 22 (Study 1) or 28 (Study 2) scalp electrodes, bandpass filtered between 0.05 and 200-Hz, and referenced to linked ears (Study 1) or the right mastoid (Study 2) using Neuroscan SynAmps amplifiers (Compumedics Neuroscan, Charlotte, NC, USA). Additional electrodes were placed at the outer canthi of both eyes and above and below the left eye to record eye movements and blinks (vertical and horizontal electro-oculogram \[EOG\]; VEOG, HEOG). All electrode impedances were maintained at or below 10 kΩ. Because Study 1 subjects were participating in an intravenous drug challenge study, the EEG data used for reliability analysis came from pre-infusion EEG recordings on the morning of each test day. Due to Study 1 time constraints and lower priority of these pre-infusion EEG recordings relative to other test day procedures, not all 30 Study 1 subjects provided useable 20, 30, and 40-Hz oddball task runs on all three test occasions. Specifically, two subjects dropped out and did not complete a third test occasion, and there were acquisition errors with at least one task run on one occasion for three additional subjects. Therefore, two, parallel sets of Study 1 reliability analyses were conducted to account for this: either (1) using the first two test occasions for all (*n* = 30) subjects ("all subject" analyses) or (2) using all three test occasions for the subset of subjects (*n* = 25) who completed all test days ("all day" analyses).

2.4. EEG processing and time-frequency analysis {#s0030}
-----------------------------------------------

Study 2 data were re-referenced to an average mastoid reference. Using Brain Vision Analyzer software, all continuous EEG data from both studies were initially 1-Hz high-pass filtered, segmented into 3 s (−1 to 2 s around click train onset) single trial epochs for each driving frequency and, if applicable, oddball versus passive attention conditions, subjected to regression-based ocular correction ([@bb0055]), and baseline corrected using the 100 ms baseline period preceding click train onset. Single trial data were then exported to MATLAB for additional processing steps described in greater detail in Supplementary Material. Briefly, canonical correlation analysis and automated single trial cleaning were applied to all data, using the same approaches described previously ([@bb0085]). Following these artifact reduction and rejection steps, Morlet wavelet time-frequency transformation was implemented using FieldTrip ([@bb0120]) separately for all driving frequencies and oddball/passive tasks. ERPs were calculated prior to wavelet decomposition to allow calculation of evoked power, but all other time-frequency measures were derived from the single trial data. PLA time-frequency data for each subject and condition were re-expressed as the difference from the expected angle at each point in the time-frequency matrix. The expected angle at each time and frequency data point was defined separately for Study 1, using the circular mean across all Study 1 subjects and test days, and Study 2, using the circular mean across both test days of the HC only. The difference between an individual subject phase angle and the expected (circular mean) phase angle defines the PLA, in units of radians (see Supplementary Material). For the PLA measure only, a z-score transformation using the same group means used to define expected angles was applied prior to conducting reliability analyses. While such standardization does not change the underlying distribution of linear variables (e.g., evoked power, total power, or PLF), it can be considered a non-linear transformation of PLA data because it changes periodic values, in radians, to units of standard deviation from the reference group (all subjects for Study 1 and HC only for Study 2). Because of its frequent use in previous studies of schizophrenia, results from electrode Fz are presented in the main text, with parallel results from other electrodes presented in Supplementary Materials. The majority of ASSR studies in schizophrenia have focused on either the entire steady-state stimulation period (e.g., 0 to 500 ms) or a time window after the transient onset auditory evoked potential components (e.g., 250-500 ms) in statistical analyses (see Supplementary Material). However, most ASSR *reliability* studies have focused on later (i.e., after 200 ms) windows. Separate averages were calculated from each time-frequency measure at each of the three steady-state frequencies for early (0 to 250 ms), late (250 to 500 ms), and complete (0 to 500 ms) steady-state stimulation time windows.

2.5. Reliability analysis {#s0035}
-------------------------

Test-retest reliability was determined with random effects analysis using a single facet of observation (test occasion) generalizability study ("G-study") design to estimate variance components ([@bb0160]). Such a design allows estimation of three variance components for any time-frequency measure. The variance components for Person (***σ***~***p***~^**2**^), Occasion (***σ***~***o***~^**2**^), and Person x Occasion (plus Error, ***σ***~***po+e***~^**2**^) are estimated separately for each study and group (e.g., HC or SZ). Once variance components are estimated, the G-coefficient, which provides a measure of generalizability or reliability of the measured score, can be calculated as in Eq. [(1)](#fo0005){ref-type="disp-formula"}:$$G = \frac{\mathbf{\sigma}_{\mathbf{p}}^{2}}{\left( {\mathbf{\sigma}_{\mathbf{p}}^{2} + \frac{\mathbf{\sigma}_{\mathbf{po} + \mathbf{e}}^{2}}{\mathbf{n}_{\mathbf{o}}}\ } \right)}$$

The majority of ASSR studies in schizophrenia are cross-sectional. If multiple time points are used to assess longitudinal or treatment effects, time-frequency measures from each session would be treated separately, indicating the best choice for the number of test occasions on which the measurement is based (***n***~***o***~) is 1. Therefore, the G-coeffecient in Eq. [(1)](#fo0005){ref-type="disp-formula"} is equal to the ICC defined by Shrout and Fleiss (e.g., ICC(3,1) in ([@bb0170])) when ***n***~***o***~ = 1. As all G-coefficients in this study were calculated with ***n***~***o***~ = 1, they could be considered interchangeable with ICCs. Variance components were estimated using a restricted maximum likelihood approach in MATLAB ([@bb0210]). Components were estimated separately and saved for the steady-state frequencies (20-Hz, 30-Hz, or 40-Hz), two tasks (Study 1 only), and four time-frequency measurements (evoked power, total power, PLF, and PLA).

The goal of a G-study is not to test a specific hypothesis. Thus, there are no *p*-values associated with estimated variance components or G-coefficients. However, existing guidelines for determining clinical significance of ICCs suggest that the reliability coefficient can be qualitatively categorized as follows: ICC \< 0.4 is poor, 0.4 ≤ ICC \< 0.6 is fair, 0.6 ≤ ICC \< 0.75 is good, and 0.75 ≤ ICC \< 1 is excellent ([@bb0025]). Therefore, G-coefficients were categorized using these four labels for descriptive purposes, but alternative categories have been proposed (e.g., ([@bb0165])), and could similarly be applied to characterize reliability.

3. Results {#s0040}
==========

Grand average time-frequency maps for each measure and steady-state stimulation condition are presented in [Fig. 1](#f0005){ref-type="fig"} for the Study 1 completers.Fig. 1Grand average time-frequency maps summarize 25 Study 1 subjects and three, separate test occasions at electrode Fz. Evoked power (left column), phase-locking factor (middle column), and total power (right column) were estimated from EEG data collected during 20-Hz (top row), 30-Hz (middle row), and 40-Hz auditory steady state stimulation. EEG frequency, in Hertz (Hz), is shown on the y-axis and time, in milliseconds (ms), is shown on the x-axis, with horizontal dashed white lines indicating the frequency of the steady-state stimulation. Darker red colors indicate greater auditory steady-state responses.Fig. 1

Across each of the 3 test days for Study 1 subjects, the largest ASSR total power and PLF were observed for 40-Hz click trains, followed by 30-Hz trains, and finally 20-Hz trains. While 40-Hz evoked power is also greatest on two of the three test days, the 20-Hz evoked power is greatest on the first test occasion and almost equal to 30-Hz evoked power on the other test occasions ([Fig. 2](#f0010){ref-type="fig"}).Fig. 2Grand average waveforms summarize all 25 Study 1 subjects\' auditory steady-state responses at each of the three, separate test occasions at electrode Fz. Waveforms represent the time-frequency measure (either Evoked Power, top row; Phase-Locking Factor (PLF), middle row; or Total Power, bottom row) at the steady-state stimulation frequency, with the 20-Hz activity plotted from 20-Hz stimulation trials (Red), the 30-Hz activity plotted from the 30-Hz stimulation trials (Blue) and the 40-Hz activity plotted from the 40-Hz stimulation trials (Green). Time-frequency measures are plotted and labeled on the y-axis with time, in milliseconds (ms), depicted on the x-axis in sub-plots for Day or Occasion 1 (left column), Occasion 2 (middle column), and Occasion 3 (right column).Fig. 2

Estimated G-coefficients for various combinations of time window (0--250, 250--500, and 0--500 ms), ASSR frequency (20-Hz, 30-Hz, 40-Hz), task (oddball vs. passive), time-frequency measure, and study group are presented in [Table 3](#t0015){ref-type="table"} for electrode Fz. To summarize, there was tendency for reliability to be greatest for the 40-Hz ASSR, followed next by 30-Hz ASSR, and finally 20-Hz ASSR across study groups and time-frequency measures. The Active (auditory oddball) versus Passive steady-state paradigm comparisons revealed that the Passive version of the paradigm was associated with greater 40-Hz ASSR reliability across Evoked/Total Power and PLA, but not PLF, measures. In both paradigms, the 20-Hz ASSR reliability was poor for all measures but PLF. One unexpected result was poor reliability in the Study 2 HC sample in the late (250-500 ms) time window for the 40-Hz ASSR PLF measure. To better understand the underlying source of this poor reliability value, variance components as well as fixed steady-state frequency effects (e.g., 40-Hz \> 30-Hz \> 20-Hz) were qualitatively compared between these HC, the SZ, and the Study 1 all subjects sample. This Study 1 all subjects sample was selected because it contained the most subjects (*n* = 30) but only 2 test occasions, which matched the Study 2 design. [Fig. 3](#f0015){ref-type="fig"} demonstrates that both HC and SZ exhibit similar fixed SSR frequency effects on PLF (e.g., 20-Hz \< 30-Hz \< 40-Hz) to those observed in Study 1 subjects ([Fig. 2](#f0010){ref-type="fig"}, middle row).Table 3Generalizability coefficients for electrode Fz.Table 3Early (0--250 ms)Late (250--500 ms)Complete (0--500 ms)20 Hz30 Hz40 Hz20 Hz30 Hz40 Hz20 Hz30 Hz40 HzEvoked power Study 1 - All Days0.150.540.700.220.520.720.200.550.73 Study 1 - All Subjects0.110.290.320.290.420.520.230.350.41 Study 1 - Active0.090.340.250.630.200.49 Study 1 - Passive0.22**0.87**0.38**0.86**0.36**0.88** Study 2 - HCs**0.760.830.92**0.28**0.760.97**0.61**0.930.96** Study 2 - SZs0.18**0.96**0.50**0.91**0.710.430.53**0.89**0.46Total power Study 1 - All Days0.120.420.710.000.43**0.77**0.000.47**0.75** Study 1 - All Subjects0.130.300.550.000.440.630.000.430.61 Study 1 - Active0.110.500.000.730.000.65 Study 1 - Passive0.150.630.10**0.81**0.110.74 Study 2 - HCs0.110.64**0.76**0.00**0.79**0.330.00**0.77**0.57 Study 2 - SZs0.610.68**0.85**0.28**0.810.88**0.310.76**0.88**PLF Study 1 - All Days0.480.690.740.500.70**0.81**0.550.71**0.79** Study 1 - All Subjects0.550.710.700.640.730.730.670.750.73 Study 1 - Active0.710.67**0.77**0.75**0.81**0.73 Study 1 - Passive0.470.690.530.740.610.73 Study 2 - HCs0.610.730.650.60**0.82**0.160.73**0.82**0.41 Study 2 - SZs0.220.530.73**0.79**0.65**0.76**0.570.55**0.77**PLA Study 1 - All Days0.300.730.700.44**0.79**0.520.33**0.80**0.63 Study 1 - All Subjects0.130.600.670.310.720.620.310.710.65 Study 1 - Active0.170.68**0.76**0.680.540.69 Study 1 - Passive0.22**0.820.86**0.720.00**0.84** Study 2 - HCs0.000.56**0.77**0.080.73**0.78**0.380.63**0.80** Study 2 - SZs**0.970.77**0.690.58**0.890.860.970.860.81**[^3]Fig. 3Grand average waveforms summarize Study 1 subjects\' auditory steady-state responses from two, separate test occasions at electrode Fz. Waveforms represent the phase-locking factor at the steady-state stimulation frequency, with the 20-Hz activity plotted from 20-Hz stimulation trials (Red), the 30-Hz activity plotted from the 30-Hz stimulation trials (Blue) and the 40-Hz activity plotted from the 40-Hz stimulation trials (Green). The phase-locking factor values are plotted on the y-axis separately for Healthy Controls (HC, top row) and Schizophrenia Patients (SZ, bottom row) with time, in milliseconds (ms), depicted on the x-axis in sub-plots for Occasion 1 (left column) and Occasion 2 (right column).Fig. 3

However, stacked bar plots ([Fig. 4](#f0020){ref-type="fig"}) demonstrate that relative to both Study 2 SZ subjects and Study 1 subjects, the Study 2 HC sample has relatively small person variance. The error variance components, which include both Person X Day interaction plus error variance, are similar across these subject groups, as are error variance components in the 20-Hz and 30-Hz frequency conditions. Thus, the reduction in reliability reflects reduced true variance of the 40-Hz PLF measure in the Study 2 HC sample during the 250-500 ms 40-Hz click-train window. 40-Hz total power, which also showed reduced reliability in this same time window in Study 2 HCs, appeared to have a similar reduction in 40-Hz ASSR true-score variance across participants.Fig. 4Stacked bar plots demonstrate color-coded contributions of the three different variance components (Person, Day, and Person X Day plus Error) to the total variance (y-axis) for the phase-locking factor (top row) and total power (bottom row) from electrode Fz in the 250 to 500 millisecond time-window. Separate bars depict the 30 Study 1 subjects (S1 HC), Study 2 healthy controls (S2 HC), and Study 2 schizophrenia patients (SZ) with separate sub-plots for the 20-Hz stimulation trials (left column), 30-Hz stimulation trials (middle column), and 40-Hz stimulation trials (right column).Fig. 4

[Fig. 5](#f0025){ref-type="fig"} shows time-frequency G-coefficient maps for the Study 1 completers. These maps clearly indicate that the 40-Hz ASSR has good reliability (G-coefficients \>0.6) for most of the steady-state stimulation time period across time-frequency measures. The PLF reliability on a millisecond basis is excellent (G-coefficients \>0.75) in the 200--500 ms time window. Notably, the early evoked gamma band response (40-50 Hz, 0-100 ms) exhibits good reliability across time-frequency measures and steady-state driving frequencies.Fig. 5Generalizability- (or G-) coefficient time-frequency maps summarize the test-retest reliability of time-frequency measures using the 25 Study 1 subjects and three test occasions at electrode Fz. Evoked power (left column), phase-locking factor (inner-left column), total power (inner-right column), and phase-locking angle (right column) were estimated from EEG data collected during 20-Hz (top row), 30-Hz (middle row), and 40-Hz auditory steady state stimulation. EEG frequency, in Hertz (Hz), is shown on the y-axis and time, in milliseconds (ms), is shown on the x-axis, with horizontal dashed green lines indicating the frequency of the steady-state stimulation. Brighter yellow to white colors indicate greater auditory steady-state response test-retest reliability, and labeled blue contour lines indicate boundaries between fair (G-coefficient ≥ 0.4), good (G-coefficient ≥ 0.65) and excellent (G-coefficient ≥ 0.75) reliability.Fig. 5

4. Discussion {#s0045}
=============

This study assessed the test-retest reliability of multiple time-frequency measures of 20-Hz, 30-Hz, and 40-Hz ASSR in two samples, including HC and SZ groups. Reliability analyses established that the test-retest reliability of the ASSR in patients with schizophrenia is as good and in some instances better than ASSR reliability in healthy controls. While the reliability of ASSR measures in both passive listening and oddball task conditions was high, consistent with prior ASSR reliability studies ([@bb0095]; [@bb0115]; [@bb0200]; [@bb0030]) controlling for attention with a behavioral (i.e., auditory oddball) task manipulation appears to slightly reduce ASSR reliability. Finally, comprehensive assessment of evoked power, total power, PLF, and PLA time-frequency measures of the ASSR revealed a pattern of effects that was in line with prior ASSR reliability studies for power and PLF measures and highlighted a consistent pattern of reliability across all time-frequency measures: 40-Hz ASSR measures exhibit excellent reliability that are often greater than those observed for 20-Hz and 30-Hz ASSR measures.

While several studies have demonstrated a strong, positive correlation between 40-Hz ASSR measures from two separate occasions ([@bb0095]; [@bb0115]; [@bb0030]), this study is only the second to assess test-retest reliability using ICCs, which depend on agreement and not just shared variance between measurements from separate test occasions. Tan et al. ([@bb0200]) used ICCs to demonstrate high test-retest reliability of the 40-Hz ASSR in MEG data. In addition to replicating these previous reports of high 40-Hz ASSR reliability, this study demonstrates that various ASSR time-frequency measures are equally reliable in SZ and HC. This is the first study to assess ASSR reliability in SZ. Importantly, patients showed good (ICCs \>0.6) or excellent (ICCs \>0.75) 40-Hz ASSR reliability for all time-frequency measures and time windows assessed except evoked power. The 40-Hz ASSR, relative to 30-Hz or 20-Hz ASSR, is most consistently found to be deficient in SZ, making it a reasonable target for intervention studies that aim to improve gamma oscillations. That the 40-Hz ASSR has excellent reliability for total power, PLF, and PLA measures provides quantitative support for the use of these three 40-Hz ASSR measures in intervention studies with multiple assessment occasions. The data also support tracking changes in 40-Hz ASSR measures over time in longitudinal observational studies of schizophrenia patients and healthy controls.

This is the first study to assess test-retest reliability of ASSRs in the same subjects during different task conditions. Increasing attention to auditory steady-state stimulation has been shown to enhance the 40-Hz ASSR([@bb0145]; [@bb0175]) and increased arousal level has been shown to diminish the 40-Hz ASSR ([@bb0060]). While it is unclear to what extent an auditory oddball task would increase arousal relative to passive listening, the task itself along with the behavioral readout provided by participant responses to target tones at the very least indicates subjects are listening to the sounds being presented. Given that several previous studies have shown ASSR enhancement with increased attention ([@bb0005]; [@bb0145]; [@bb0150]; [@bb0155]), perhaps the slight reductions in G-coefficients observed in the auditory oddball task condition were a function of decreased person variance in this task, relative to passive tasks, due to increased attention. This can be seen in [Fig. 6](#f0030){ref-type="fig"}, where overall variance and person variance in particular in all 40-Hz ASSR time-frequency measures are reduced in the active condition compared to the passive condition.Fig. 6Stacked bar plots demonstrate color-coded contributions of the three different variance components (Person, Day, and Person X Day plus Error) to the total variance (y-axis) for the evoked power (top row), phase-locking angle (second row), phase-locking factor (third row) and total power (bottom row) at electrode Fz in the 0 to 500 millisecond time-window for 40-Hz auditory steady-state stimulation. Separate bars depict the active auditory oddball (left) and passive listening task conditions from 14 Study 1 subjects.Fig. 6

A subject who was not actively attending to the click train stimuli during passive listening might increase his or her ASSR through increased attention during the oddball task. A subject who was very vigilant in his or her auditory attention during passive listening might decrease attention to the click trains in the oddball task, as these were non-target sounds, leading to a decrease in the ASSR. In both cases, these trait-like differences between individuals would be associated with larger person variance (and G-coefficients) in the passive listening task and smaller person variance in the auditory oddball task. It is also important to note that P300 ERP measures obtained from the infrequent sounds in the oddball task have previously been shown to have good to excellent reliability (ICCs ≥ 0.7), indicating this one task condition yields multiple EEG/ERP measures that are reliable ([@bb0035]).

The PLA, a novel measure of latency- and frequency-specific phase lag relative to a group mean, exhibited 40-Hz ASSR test-retest reliability that was good to excellent for most of the study samples and time windows evaluated. The PLA G-coefficients for electrode Fz were comparable to other time-frequency measures in general, but PLF, total power, and evoked power G-coefficients were all greater than PLA G-coefficients for the Study 1 completers ([Table 3](#t0015){ref-type="table"} "all days" sample), which represents the largest sample (*n* = 75 total EEG sessions) in this study. Analysis of electrode Cz (Supplementary Material) demonstrated that this pattern (i.e., PLA G \< PLF, evoked and total power Gs) did not hold across sites, and PLA had more G-coefficients in the "excellent" range (i.e., ≥ 0.75) than any other measure for Cz. It is interesting that the PLA G-coefficients were greater than all other time-frequency measures for the 30-Hz ASSR in the relatively large Study 1 sample. In the one prior report comparing SZ to HC, 30-Hz ASSR PLA had greater circular phase variability than 40-Hz PLA, but the group comparison was only marginally significant at 30-Hz ([@bb0135]). Given the excellent test-retest reliability of 30-Hz PLA, it should continue to be explored along with the 40-Hz ASSR in subsequent studies of PLA that attempt to determine what clinical, demographic, or cognitive variables might be associated with phase delay.

In addition to PLA, assessment of the reliability of power and PLF measures revealed that across all 4 measures, 40-Hz ASSR had greater reliability than 20-Hz and 30-Hz ASSR in two-thirds (48/72) of the samples and time windows tested at Fz. When considering only the power measures, 40-Hz ASSR had the greatest G-coefficients in \~86% (31/36) of the reliability analyses, which is also much greater than what would be expected by chance (i.e., 1/3 or 33.3%). While no single measure was uniformly most reliable among the samples and time windows assessed, the phase measures tended to outperform power measures on average. Consistent with previous reliability studies of 40-Hz ASSR ([@bb0095]; [@bb0115]; [@bb0200]), power measures had a wider range of reliability coefficients, including poor G-coefficients as low as zero for 20-Hz ASSR. Given the previously observed strong correlations between evoked power, total power, and PLF in both HC and SZ ([@bb0135]) and the fact that no single measure was dominant in terms of reliability in these analyses, continued use of multiple time-frequency measures and multivariate statistical tests in the assessment of 40-Hz ASSR is well justified.

4.1. Limitations {#s0050}
----------------

As was the case in almost every study reporting abnormal 40-Hz ASSR in SZ, the patients in study 2 were all prescribed antipsychotic medications at the time of EEG assessments. Both the SZ and HC groups in study 2 had small sample sizes, which could lead to less stable variance component estimates, and subsequently, less accurate G-coefficient estimates. Subjects in Study 1 included cannabis users, and these participants were run in the morning, fasted on the test day, and had intravenous lines placed for the drug challenge study to follow. The data pre-processing methods and choice of reference electrodes could impact reliability coefficients and should be carefully considered in future studies. Any one of these factors as well as any additional factors (e.g., poor quality control) that could lead to increased error variance might make the reliability results from this sample less generalizable. However, the G-coefficients observed were both consistent with Study 2 samples and the ICCs reported by Tan et al. ([@bb0200]), providing at least some indication that the unique aspects of the Study 1 design did not greatly influence the reliability results.

5. Conclusions {#s0055}
==============

The previously observed excellent test-retest reliability of the 40-Hz ASSR using either MEG ([@bb0095]; [@bb0200]) or EEG ([@bb0115]) in healthy controls was replicated in multiple samples. Furthermore, similar test-retest reliability was observed in SZ patients for the first time. This is important because it supports the use of the 40-Hz ASSR paradigm as a probe of gamma band oscillatory activity in clinical trials examining within-subjects treatment effects aimed at ameliorating abnormal gamma oscillations in schizophrenia. Minimal reduction of test-retest reliability during an auditory oddball task indicates that it could be used when some behavioral measure of attention is required to monitor task compliance, and when additional measures of ERP P300 components are of interest without sufficient time to run separate auditory oddball and ASSR paradigms. Finally, excellent test-retest reliability was also observed for the PLA, which should encourage additional use of this novel measure of phase delay in future studies.

Appendix A. Supplementary data {#s0070}
==============================

Graphical stacked-bar representation of steady-state analysis time windows reported in supplementary Table 1.Image 1

Acknowledgements {#s0060}
================

We thank Angelina Genovese, RNC, MBA; Michelle San Pedro, RN; Elizabeth O\'Donnell, RN; Brenda Breault, RN, BSN; Sonah Yoo, RPh; Rachel Galvan, RPh; and Willie Ford of the Neurobiological Studies Unit at the VA Connecticut Healthcare System, West Haven Campus for their central contributions to the success of this project.

Funding {#s0065}
=======

This work was supported by the National Institute on Drug Abuse (R21 DA020750 to DCD), an investigator-initiated AstraZeneca study (to DHM), and the Department of Veterans Affairs (Senior Research Career Scientist award to JMF; I01 CX0004971 to JMF).

Supplementary data to this article can be found online at <https://doi.org/10.1016/j.nicl.2019.101878>.

[^1]: Socioeconomic status (SES) based on the Two-Factor (1958) Hollingshead Scale; Higher scores indicate lower socioeconomic status.

[^2]: *p* \< .05 with independent samples *t*-test.

[^3]: G-coefficients greater than 0.75 are in bold font.
