The human mind is a dynamical system, yet many analysis techniques used to study it are limited in their ability to capture the complex dynamics that may characterize mental processes. This study proposes the continuous-time deconvolutional regressive neural network (CDRNN), a deep neural extension of continuous-time deconvolutional regression (CDR, Shain and Schuler, 2021) that jointly captures time-varying, non-linear, and delayed influences of predictors (e.g. word surprisal) on the response (e.g. reading time). Despite this flexibility, CDRNN is interpretable and able to illuminate patterns in human cognition that are otherwise difficult to study. Behavioral and fMRI experiments reveal detailed and plausible estimates of human language processing dynamics that generalize better than CDR and other baselines, supporting a potential role for CDRNN in studying human language processing. Central questions in psycholinguistics concern the mental processes involved in incremental human language understanding: which representations are computed when, by what mental algorithms (Fra-zier and Fodor, 1978; Just and Carpenter, 1980; Abney and Johnson, 1991; Tanenhaus et al., 1995; Almor, 1999; Gibson, 2000; Coltheart et al., 2001; Hale, 2001; Lewis and Vasishth, 2005; Levy, 2008, inter alia )? Such questions are often studied by caching out a theory of language processing in an experimental stimulus, collecting human responses, and fitting a regression model to test whether measures show the expected effects (e.g. Grodner and Gibson, 2005). Regression techniques have grown in sophistication, from ANOVA (e.g. Pickering and Branigan, 1998) to newer linear mixed-effects approaches (LME, Bates et al., 2015) that enable direct word-by-word analysis of effects in naturalistic human language processing (e.g. Demberg and Keller, 2008; Frank and Bod, 2011). However, these methods struggle to account for delayed effects. Because the human mind operates in real time and experiences computational bottlenecks of various kinds (Bouma and De Voogd, 1974; Just and Carpenter, 1980; Ehrlich and Rayner, 1981; Mollica and Piantadosi, 2017), delayed effects may be pervasive, and, if left uncontrolled, can yield misleading results (Shain and Schuler, 2018). Continuous-time deconvolutional regression (CDR) is a recently proposed technique to address delayed effects in measures of human cognition (Shain and Schuler, 2018, 2021). CDR fits parametric continuous-time impulse response functions (IRFs) that mediate between word features and response measures. An IRF maps the time elapsed between a stimulus and a response to a weight describing the expected influence of the stimulus on the response. CDR models the response as an IRF-weighted sum of preceding stimuli, thus directly accounting for effect latencies. Empirically, CDR reveals fine-grained processing dynamics and generalizes better to human reading and fMRI responses than established alternatives. However, CDR retains a number of simplifying assumptions (e.g. that the IRF is fixed over time) that may not hold of the human language processing system. Deep neural networks (DNNs), widely used in natural language processing (NLP), can relax these strict assumptions. Indeed, psycholinguistic regression analyses and NLP systems share a common structure: both fit a function from word features to some quantity of interest. However, psycholinguistic regression models face an additional constraint: they must be interpretable enough to allow researchers to study relationships between variables in the model. This requirement may be one reason why black box DNNs are not generally used to analyze psycholinguistic data, despite the tremendous gains DNNs have enabled in natural language tasks (Peters et al., 2018; Devlin et al., 2019; Radford et al., 2019; Brown et al., 2020, inter alia ), in part by better approximating the complex dynamics of human cognition as encoded in natural language (Linzen et al., 2016; Gulordava et al., 2018; Tenney et al., 2019; Hewitt and Manning, 2019; Wilcox et al., 2019; Schrimpf et al., 2020). This study proposes an attempt to leverage the flexibility of DNNs for psycholinguistic data analysis. The continuous-time deconvolutional regressive neural network (CDRNN) is an extension of CDR that reimplements the impulse response function as a DNN describing the expected influence of preceding events (e.g. words) on future responses (e.g. reading times) as a function of their properties and timing. CDRNN retains the deconvolutional design of CDR while relaxing many of its simplifying assumptions (linear-ity, additivity, homosketasticity, stationarity, and context-independence, see Section 2), resulting in a highly flexible model. Nevertheless, CDRNN is interpretable and can shed light on the underlying data generating process. Results on reading and fMRI measures show substantial generalization improvements from CDRNN over baselines, along with detailed insights about the underlying dynamics that cannot easily be obtained from existing methods. 1 2 Background Psycholinguists have been aware for decades that processing effects may lag behind the words that trigger them (Morton, 1964; Bouma and De Voogd, 1974; Rayner, 1977; Erlich and Rayner, 1983; Mitchell, 1984; Rayner, 1998; Vasishth and Lewis, 2006; Smith and Levy, 2013), possibly because cognitive buffers may exist to allow higher-level information processing to catch up with the input (Bouma and De Voogd, 1974; Baddeley et al., 1975; Just and Carpenter, 1980; Ehrlich and Rayner, 1981; Mollica and Piantadosi, 2017). They have also recognized the potential for non-linear, interactive, and/or time-varying relationships between word features and language processing (Smith and Levy, 2013; Baayen et al., 2017, 2018). No prior regression method can jointly address these 1 Because of page constraints, additional replication details and synthetic results are provided in an external supplement, available here: https://osf.io/z89vn/ . concerns in non-uniform time series (e.g. words with variable duration) like naturalistic psycholinguistic experiments. Discrete-time methods (e.g. lagged/spillover regression, Sims, 1971; Erlich and Rayner, 1983; Mitchell, 1984) ignore potentially meaningful variation in event duration, even if some (e.g. generalized additive models, or GAMs, Hastie and Tibshirani, 1986; Wood, 2006) permit non-linear and non-stationary (time-varying) feature interactions (Baayen et al., 2017). CDR (Shain and Schuler, 2018, 2021) addresses this limitation by fitting continuous-time IRFs, but assumes that the IRF is stationary (time invariant), that features scale linearly and combine additively, and that the response variance is constant (homoskedastic). By implementing the IRF as a time-varying neural network, CDRNN relaxes all of these assumptions, incorporating the featural flexibility of GAMs while retaining the temporal flexibility of CDR. Previous studies have investigated latency and non-linearity in human sentence processing. For example, Smith and Levy (2013) attach theoretical significance to the functional form of the relationship between word surprisal and processing cost, using GAMs to show that this relationship is linear and arguing on this basis that language processing is highly incremental. This claim is under active debate (Brothers and Kuperberg, 2021), underlining the importance of methods that can investigate questions of functional form. Smith and Levy (2013) also investigate the timecourse of surprisal effects using spillover and find a more delayed surprisal response in self-paced reading (SPR) than in eye-tracking. Shain and Schuler (2021) support the latter finding using CDR, and in addition show evidence of strong inertia effects in SPR, such that participants who have been reading quickly in the recent past also read more quickly now. However, this outcome may be an artifact of the stationarity assumption: CDR may be exploiting its estimates of rate effects in order to capture broad non-linear negative trends (e.g. task adaptation, Prasad and Linzen, 2019) in a stationary model. Similarly, the generally null word frequency estimates reported in Shain and Schuler (2021) may be due in part to the assumption of additive effects: word frequency and surprisal are related, and they may coordinate interactively to determine processing costs (Nor-ris, 2006). Thus, in general, prior findings on the timecourse and functional form of effects in human sentence processing may be influenced by method-h (0) = (cid:20) x t (cid:21) RNN h  IRF convolution (cid:80) s RNN + i n  g (  ) x 1 RNN + i n  g (  ) x 1 RNN + i n  g (  ) x 1 RNN + i n  g (  ) x 1 Figure 1: CDRNN model. By jointly relaxing these potentially problematic assumptions, CDRNN stands to support more reliable conclusions about human language comprehension, while also possibly enabling new insights into cognitive dynamics. This section presents a high-level description of the model design (for formal definition, see Appendix A). The CDRNN architecture is represented schematically in Figure 1. The primary goal of estimation is to identify the deep neural IRF g (  ) (top) that computes the influence of a preceding event on the predictive distribution over a subsequent response as a function of their distance in time  . As shown, the IRF is a feedforward projection of  into a matrix that defines a weighted sum over the values of input vector x , which is concatenated with a bias to capture general effects of stimulus timing ( rate ). This matrix multiplication determines the contribution of the stimulus event to the parameters of the predictive distribution (e.g. the mean and variance parameters of a Gaussian predictive distri-bution). Defining the IRF as a function of  ensures that the model has a continuous-time definition. To capture non-linear effects of stimulus features, the IRF projection is itself parameterized by a projection of a hidden state h . The dependence on h permits non-linear influences of the properties of the stimulus sequence on the IRF itself. To generate h , the predictors x are concatenated with their timestamps t and submitted to the model as input. Inputs are cast to a hidden state for each preceding event as the sum of three quantities: a feedforward projection h in of each input, a forward-directional RNN projection h RNN of the events up to and including each input, and random effects h Z containing offsets for the relevant random effects level(s) (e.g. for each participant in an experiment). In this study, the recurrent component is treated as optional (gray arrows). Without the RNN, the model is non-stationary (via input t ) but cannot capture contextual influences on the IRF. The summation over IRF outputs at the top of the figure ensures that the model is deconvolutional: each preceding input contributes to the response in some proportion, with that proportion determined by the features, context, and relative timing of that input. Because the IRF depends on a deep neural projection of the current stimulus as well as (op-tionally) the entire sequence of preceding stimuli, it implicitly estimates all interactions between these variables in governing the response. Predictors may thus coordinate in a non-linear, non-additive, and time-varying manner. The CDRNN IRF describes the influence over time of predictors on all parameters of the predictive distribution (in these experiments, the mean and variance parameters of a Gaussian predictive distribution). Such a design (i.e. modeling dependencies on the predictors of all parameters of the predictive distribution) has previously been termed distributional regression (Burkner, 2018). Despite their flexibility and task performance (Section 5), CDRNN models used in this study have few parameters (Table A1) by current deep learning standards because they are relatively shallow and small (Supplement S1). Given (1) an input configuration C containing predictors X , input timestamps t , and response timestamps t (cid:48) , (2) CDRNN parameter vector w , (3) output distribution p , (4) random effects vector z , and (5) response vector y , the model uses gradient descent In addition to random effects shrinkage governed by  z and any arbitrary additional regularization penalties L reg (see Supplement S1), models are regularized using dropout (Srivastava et al., 2014) with drop rate d h at the outputs of all feedforward hidden layers. Random effects are also dropped at rate d z , which is intended to encourage the model to find population-level estimates that accurately reflect central tendency. Finally, the recurrent contribution to the CDRNN hidden state ( h RNN above) is dropped at rate d r , which is intended to encourage accurate IRF estimation even when context is unavailable. Because it is a DNN, CDRNN lacks parameters that selectively describe the size and shape of the response to a specific predictor (unlike CDR), and indeed individual parameters (e.g. individual biases or connection weights) are not readily interpretable. Thus, from a scientific perspective, the quantity of general interest is not a distribution over parameters, but rather over the effect of a predictor on the response. The current study proposes to accomplish this using perturbation analysis (e.g. Ribeiro et al., 2016; Petsiuk et al., 2018), manipulating the input configuration and quantifying the influence of this manipulation on the predicted response. 2 For example, to obtain an estimate of rate effects (i.e. the base response or deconvolutional inter-cept, see Shain and Schuler, 2021), a reference stimulus can be constructed, and the response to it can be queried at each timepoint over some interval of interest. To obtain CDR-like estimates of predictor-wise IRFs, the reference stimulus can be increased by 1 in the predictor dimension of interest (e.g. word surprisal) and requeried, taking the difference between the obtained response and the reference response to reveal the influence of an extra unit of the predictor. 3 This study uses the 2 Perturbation analyses is one of a growing suite of tools for black box interpretation. It is used here because it straightforwardly links properties of the input to changes in the estimated response, providing a highly general method for querying aspects of the the non-linear, non-stationary, non-additive IRF defined by the CDRNN equations. 3 Note that 1 is used here to maintain comparability of effect estimates to those generated by methods that assume training set mean of x and t as a reference, since this represents the response of the system to an average stimulus. The model also supports arbitrary additional kinds of queries, including of the curvature of an effect in the IRF over time and of the interaction between two effects at a point in time. Indeed, the IRF can be queried with respect to any combination of values for predictors, t , and  , yielding an open-ended space of queries that can be constructed as needed by the researcher. Because the estimates of interest all derive from the model's predictive distribution, uncertainty about them can be measured with Monte Carlo techniques as long as training involves a stochastic component, such as dropout (Srivastava et al., 2014) or batch normalization (Ioffe and Szegedy, 2015). This study estimates uncertainty using Monte Carlo dropout (Gal and Ghahramani, 2016), which recasts training neural networks with dropout as variational Bayesian approximation of deep Gaussian process models (Damianou and Lawrence, 2013). At inference time, an empirical distribution over responses to an input is constructed by resampling the model (i.e. sampling different dropout masks). 4 As argued by Shain and Schuler (2021) for CDR, in addition to intervals-based tests, common hypothesis tests (e.g. for the presence of an effect) can be performed in a CDRNN framework via bootstrap model comparison on held out data (e.g. of models with and without the effect of interest). Following Shain and Schuler (2021), CDRNN is applied to naturalistic human language processing data from three experimental modalities: the Natural Stories self-paced reading corpus (  1M instances, Futrell et al., 2020), the Dundee eye-tracking corpus (  200K instances, Kennedy linearity of effects (especially CDR), but that 1 has no special meaning in the non-linear setting of CDRNN modeling, and effects can be queried at any offset from any reference. Results here show that deflections move relatively smoothly away from the reference, even at smaller steps than 1, and that IRFs queried at 1 are similar to those obtained from (linear) CDR, indicating that this method of effect estimation is reliable. Note finally that because predictors are underlyingly rescaled by their training set standard deviations (though plotted at the original scale for clarity), 1 here corresponds to 1 standard unit, as was the case with the CDR estimates discussed in Shain and Schuler (2021). 4 Initial experiments also explored uncertainty quantifica-tion by implemententing CDRNN as a variational Bayesian DNN. Compared to the methods advocated here, the variational approach was more prone to instability, achieved worse fit, and yielded implausibly narrow credible intervals. et al., 2003), and the Natural Stories fMRI corpus (  200K instances, Shain et al., 2020), using the train/dev/test splits for these corpora defined in Shain and Schuler (2021). Further details about datasets and preprocessing are given in Supplement S2. For reading data, CDRNN is compared to CDR as well as lagged LME and GAM baselines equipped with four spillover positions for each predictor (values from the current word, plus three preceding words), since LME and GAM are well established analysis methods in psycholinguistics (e.g. Baayen et al., 2007; Demberg and Keller, 2008; Frank and Bod, 2011; Smith and Levy, 2013; Baayen et al., 2017; Goodkind and Bicknell, 2018, inter alia ). Because the distribution of reading times is heavy-tailed (Frank et al., 2013), following Shain and Schuler (2021) models are fitted to both raw and log-transformed reading times. For fMRI data, CDRNN is compared to CDR as well as four existing techniques for analyzing naturalistic fMRI data: pre-convolution with the canonical hemodynamic response function (HRF, Brennan et al., 2012; Willems et al., 2015; Henderson et al., 2015, 2016; Lopopolo et al., 2017), linear interpolation (Shain and Schuler, 2021), binning (Wehbe et al., 2020), and Lanczos interpolation (Huth et al., 2016). Statistical model comparisons use paired permutation tests of test set error (Demsar, 2006). Models use predictors established by prior psycholinguistic research (e.g. Rayner, 1998; Demberg and Keller, 2008; van Schijndel and Schuler, 2013; Staub, 2015; Shain and Schuler, 2018, inter alia ): unigram and 5-gram surprisal , word length (read-ing only), saccade length (eye-tracking only), and previous was fixated (eye-tracking only). Predictor definitions are given in Appendix C. The deconvolutional intercept term rate (Shain and Schuler, 2018, 2021), an estimate of the general influence of observing a stimulus at a point in time, independently of its properties, is implicit in CDRNN (unlike CDR) and is therefore reported in all results. Reading models include random effects by subject, while fMRI models include random effects by subject and by functional region of interest (fROI). Unlike LME, where random effects capture linear differences in effect size between e.g. subjects, random effects in CDRNN capture differences in overall dynamics between subjects, including differences in size, IRF shape, functional form (e.g. linearity), contextual influences on the IRF, and interactions with other effects. Two CDRNN variants are considered in all experiments: the full model (CDRNN-RNN) containing an RNN over the predictor sequence, and a feedforward only model (CDRNN-FF) with the RNN ablated (gray arrows removed in Figure 1). This manipulation is of interest because CDRNN-FF is both more parsimonious (fewer parameters) and faster to train, and may therefore be preferred in the absence of prior expectation that the IRF is sensitive to context. All plots show means and 95% credible intervals. Code and documentation are available at https://github.com/coryshain/cdr . Since CDRNN is designed for scientific modeling, the principal output of interest is the IRF itself and the light it might shed on questions of cognitive dynamics, rather than on performance in some task (predicting reading latencies or fMRI measures are not widely targeted engineering goals). However, predictive performance can help establish the trustworthiness of the IRF estimates. To this end, as a sanity check, this section first evaluates predictive performance on human data relative to existing regression techniques. While results may resemble bake-off comparisons familiar from machine learning (and indeed CDRNN does outperform all baselines), their primary purpose is to establish that the CDRNN estimates are trustworthy, since they describe the phenomenon of interest in a way that generalizes accurately to an unseen sample. Baseline models, including CDR, are as reported Model Train Expl Test Canonical HRF 11.3548  11.8263  11.5661  Linearly interpolated 11.4236  11.9888  11.6654  Averaged 11.3478  11.9280  11.6090  Lanczos interpolated 11.3536  11.9059  11.5871  CDR 11.2774 11.6928 11.5369 CDRNN-FF 10.5648 11.3602 11.3042 CDRNN-RNN 10.8736 11.5631 11.3914 Table 2: fMRI. Table 1 gives mean squared error by dataset of CDRNN vs. baseline models on reading times from both Natural Stories and Dundee. Both versions of CDRNN outperform all baselines on the dev partition of all datasets except for raw (ms) latencies in Natural Stories (SPR), where CDRNN is edged out by CDR 6 but still substantially outperforms the non-CDR baselines. Nonetheless, results indicate that CDRNN estimates of Natural Stories (ms) are similarly reliable to those of CDR, and, as discussed in Section 5.2, CDRNN largely replicates the CDR estimates on Natural Stories while offering advantages for analysis. Although CDR struggles against GAM baselines on Dundee, CDRNN has closed the gap. This is noteworthy in light of speculation in Shain and Schuler (2021) that CDR's poorer performance on Dundee might be due in part to non-linear effects, which GAM can estimate but CDR cannot. CDRNN performance supports this conjecture: once the model can account for non-linearities, it overtakes GAMs. Results from fMRI are shown in Table 2, where both CDRNN variants yield substantial improvements to training, dev, and test set error. These results indicate that the relaxed assumptions afforded by CDRNN are beneficial for describing the fMRI response, which is known to saturate over time (Friston et al., 2000; Wager et al., 2005; Vazquez et al., 2006; Lindquist et al., 2009). Following Shain and Schuler (2021), model error is statistically compared using a paired permu-5 For all datasets, the CDR baseline used here is the variant that was deployed on the test set in Shain and Schuler (2021). 6 Note that a major advantage of CDRNN is its ability to model dynamics in response variance, which are not reflected in squared error. For example, although CDRNN-FF achieves worse test set error than CDR on the Natural Stories (ms) task, it affords a 31,040 point log likelihood improvement. tation test that pools across all datasets covered by a given baseline (reading data for LME and GAM, fMRI data for canonical HRF, linearly interpolated, averaged, and Lanczos interpolated, and both for CDR). 7 Results are given in Table 3. As shown, both variants of CDRNN significantly improve over all baselines, and CDRNN-RNN significantly improves over CDRNN-FF. Notwithstanding, CDRNN-FF may be preferred in applications: simpler, faster to train, better at recovering synthetic models (Supplement S3), more reliable in noisy domains like fMRI, and close in performance to CDRNN-RNN. Results overall support the reliability of patterns revealed by CDRNN's estimated IRF, which is now used to explore and visualize sentence processing dynamics. CDR-like IRF estimates can be obtained by increasing a predictor by 1 (standard deviation) relative to the reference and observing the change in the response over time. Visualizations using this approach are presented in Figure 2 alongside CDR estimates from Shain and Schuler (2021). In general, CDRNN finds similar patterns to CDR. This suggests both (1) that CDRNN is capable of recovering estimates from a preceding state-of-the-art deconvolutional model for these domains, and (2) that CDR estimates in these domains are not driven by artifacts introduced by its simplifying assumptions, since a model that lacks those assumptions and has a qualitatively different architecture largely recovers them. Nonetheless there are differences. For example, Dundee estimates decay more quickly over time in CDRNN than in CDR, indicating an even less pronounced influence of temporal diffusion in 7 The comparison rescales each pair of error vectors by their joint standard deviation in order to enable comparability across datasets with different error variances. eye-tracking than CDR had previously suggested. Estimates from CDRNN-FF and CDRNN-RNN roughly agree, except that CDRNN-RNN estimates for fMRI are more attenuated. CDR shows little uncertainty in the fMRI domain despite its inherent noise (Shain et al., 2020), while CDRNN more plausibly shows more uncertainty in its estimates for the noisier fMRI data. As noted in Section 2, Shain and Schuler (2021) report negative rate effects in reading  i.e., a local decrease in subsequent reading time at each word, especially in SPR. This was interpreted as an inertia effect (faster recent reading engenders faster current reading), but it might also be an artifact of non-linear decreases in latency over time (due to task habituation, e.g. Baayen et al., 2017; Harrington Stack et al., 2018; Prasad and Linzen, 2019) that CDR cannot model. CDRNN estimates nonetheless thus support the prior interpretation of rate effects as inertia, at least in SPR: a model that can flexibly adapt to non-linear habituation trends finds SPR rate estimates that are similar in shape and magnitude to those estimated by CDR. In addition, CDRNN finds a slower response to word surprisal in self-paced reading than in eye-tracking. This result converges with word-discretized timecourses reported in Smith and Levy (2013), who find more extensive spillover of surprisal effects in SPR than in eye-tracking. Results thus reveal important hidden dynamics in the reading response (inertia effects), continuous-time delays in processing effects, and influences of modality the continuous dynamics of sentence processing, all of which are difficult to estimate using existing regression techniques. Greater response latency and more pronounced inertia effects in self-paced reading may be due to the fact that a gross motor task (paging via button presses) is overlaid on the sentence comprehension task. While the motor task is not generally of interest to psycholinguistic theories, controlling for its effects is crucial when using self-paced reading to study sentence comprehension (Mitchell, 1984). CDRNN also allows the analyst to explore other aspects of the IRF, such as functional curvature at a point in time. For example, in the context of reading, Smith and Levy (2013) argue for a linear increase in processing cost as a function of word surprisal. The present study allows this claim to be assessed across modalities by checking the curva-NatStor (SPR) I n s t a n t a n e o u s Dundee NatStor (fMRI) O v er T i m e Figure 3: CDRNN-FF-estimated functional curvature of the 5-gram surprisal response. ture of the 5-gram surprisal response (in raw ms) at a timepoint of interest (0ms for reading and  5s for fMRI). As shown in the top row of Figure 3, reading estimates are consistent with a linear response (the credible interval contains a straight line), as predicted, but are highly non-linear in fMRI, with a rapid peak above the mean (zero-crossing) followed by a sharp dip and plateau, and even an estimated increased response at values below the mean (though estimates at the extremes have high uncertainty). This may be due in part to ceiling effects: blood oxygen levels measured by fMRI are bounded, but reading times are not. While this is again a property of experimental modality rather than sentence comprehension itself, understanding such influences is important for drawing scientific conclusions from experimental data. For example, due to the possibility of saturation, fMRI may not be an ideal modality for testing scientific claims about the functional form of effects, and the linearity assumptions of e.g. CDR and LME may be particularly constraining. The curvature of effects can also be queried over time. If an effect is temporally diffuse but linear, its curvature should be roughly linear at any delay of interest. The second row of Figure 3 shows visualizations to this effect. These plots in fact subsume the kinds of univariate plots shown above: univariate IRFs to 5-gram surprisal like those plotted in Figure 2 are simply slices taken at a predictor value (1 sample standard deviation above the mean), whereas curvature estimates in the first row of Figure 3 are simply slices taken at a time value (0s for reading and 5s for fMRI). Plots are consistent with the linearity hypothesis for reading, but again show strong non-linearities in the fMRI domain that are consistent with saturation effects Delay (s)  rate sound power unigram surprisal 5-gram surprisal PCFG surprisal Figure 4: Effect interactions in a CDRNN-FF replication of Shain et al. (2020). In addition to exploring multivariate relationships of a predictor with time, relationships between predictors can also be studied. Such relationships constitute interactions in a CDRNN model, though they are not constrained (cf. interactions in linear models) to be strictly multiplicative  indeed, a major advantage of CDRNN is that interactions come for free, along with estimates of their functional form. To explore effect interactions, a CDRNN-FF version of the full model in Shain et al. (2020) is fitted to the fMRI dataset. The model contains more predictors to explore than models considered above, including surprisal computed from a probabilistic context-free grammar ( PCFG surprisal , see Appendix C for details). Univariate IRFs are shown in the top left panel of Figure 4, and pairwise interaction surfaces at a delay of 5s (near the peak response) are shown in the remaining panels. Plots show that the response at any value of the other predictors is roughly flat as a function of sound power (i.e. signal power of the auditory stimulus, middle row). This accords with prior arguments that the cortical language system, whose activity is measured here, does not strongly regis-ter low-level perceptual effects (Fedorenko et al., 2010; Braze et al., 2011). The estimate for unigram surprisal (middle left) shows an unexpected non-linearity: although activity increases with higher surprisal (lower frequency words), it also increases at lower surprisal (higher frequency words), suggesting the existence of high frequency items that nonetheless engender a large response. The interaction between PCFG surprisal and unigram surprisal possibly sheds light on this outcome, since it shows a sharper increase in the PCFG surprisal response in higher frequency (lower unigram surprisal) regions. This may be because the most frequent words in English tend to be function words that play an outsized role in syntactic structure building (e.g. prepositional phrase attachment decisions). In addition, 5-gram surprisal interacts with PCFG surprisal , showing a non-linear increase in response for words that are high on both measures. This is consistent with a unitary predictive mechanism that experiences strong error signals when both string-level (5-gram) and structural (PCFG) cues are poor. All these interactions should be interpreted with caution, since the uncertainty interval covers much weaker degrees of interaction. As discussed in Section 3, CDRNN implements distributional regression and thus also contains an IRF describing the influence of predictors on the variance of the predictive distribution as a function of time. IRFs of the variance can be visualized identically to IRFs of the mean. For example, Figure 5 shows the estimated change in the standard deviation of the predictive distribution over time from observing a stimulus. 8 Estimates show stimulus-dependent changes 8 Because standard deviation is a bounded variable and the IRF applies before the constraint function (softplus), the relationship between the standard deviation and the y axis of the plots is not straightforward. Estimates nonetheless clearly indicate the shape and relative contribution to the response in variance across datasets whose shapes are not straightforwardly related to that of the IRFs of the mean (Figure 2). For example, both reading datasets (left and center) generally show mean and standard deviation traveling together, with increases in the mean corresponding to increases in standard deviation. In Dundee, the shapes of these changes resemble each other strongly, whereas in Natural Stories the IRFs of the standard deviation (especially rate ) differ substantially from the IRFs of the mean. By contrast, in fMRI (right), the IRFs of the standard deviation look roughly like inverted HRFs (especially for rate and 5-gram surprisal ), indicating that BOLD variance tends to decrease with larger values of the predictors. While detailed interpretation of these patterns is left to future work, these results demonstrate the utility of CDRNN for analyzing a range of links between predictors and response that are otherwise difficult to study. This study proposed and evaluated CDRNN, a deep neural extension of continuous-time deconvolutional regression that relaxes implausible simplifying assumptions made by widely used regression techniques in psycholinguistics. In so doing, CDRNN provides detailed estimates of human language processing dynamics that are difficult to obtain using other measures. Results showed plausible estimates from human data that generalize better than alternatives and can illuminate hitherto understudied properties of the human sentence processing response. This outcome suggests that CDRNN may play a valuable role in analyzing human experimental data.
abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other objective abstain abstain abstain abstain other abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain other abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain abstain