text:
\section{Introduction}

Language comprehension---and by proxy, the reading process---is assumed to be incremental and dynamic in nature: readers take one word as input at a time, process it, and then move on to the next word \cite{hale2001probabilistic,hale2006uncertainty,rayner2009language,boston2011parallel}.
Further, as each word requires a different amount of processing effort, readers must allocate differing amounts of time to them.
Indeed, this effect has been confirmed by a number of studies, which show a word's reading time is a monotonically increasing function of the word's length and surprisal \citep[\textit{inter alia}]{hale2001probabilistic,smith2008optimal,shain-2019-large}.\looseness=-1

Most prior work \cite[e.g.,][]{levy2008expectation,DEMBERG2008193,monsalve2012,wilcox2020predictive}, however, focuses only on the \defn{responsive} nature of the reading process, i.e., they look solely at how readers' behaviour is influenced by attributes of the already observed words.
Such analyses make the implicit assumption that readers dynamically allocate resources to a word as they read it, and planning ahead takes no part in reading time (RT) allocation.
Albeit quite elegant in its simplicity,
a closer analysis of reading time data shows the above theory might not capture the whole picture.
In addition to being responsive, reading behaviour may also be \defn{anticipatory}:
readers' expectations before reaching a word may influence how they process it.
One strong example of this is that often readers skip words while they read---a decision that must be made while the word's identity is unknown.\looseness=-1

In this work, we look beyond responsiveness and investigate anticipatory reading behaviours.
Specifically, we look at how a reader's expectation about a word's surprisal---operationalised as that word's \defn{contextual entropy}---affects the time taken to read it.
For various reasons, however, a reader's anticipation may not exactly match a word's expected surprisal value, which would make the contextual entropy a poor operationalisation of anticipation.
Rather, readers may rely on skewed approximations instead, e.g., anticipating that the next word's surprisal is simply the surprisal of the most likely next word.
We use the R\'enyi\xspace{} entropy (a generalisation of Shannon's entropy) to operationalise these different skewed expectation strategies.
Further, in order to better grasp the anticipatory nature of the reading process, we identify four mechanisms under which anticipation may impact reading:\looseness=-1%
\vspace{-3pt}
\begin{enumerate}[label=(\roman*),noitemsep]
\item \emph{word-skipping}: readers may completely omit fixating on a word;\looseness=-1
\item \emph{budgeting}: readers may allocate reading times for a word before reaching it;
\item \emph{preemptive processing}: readers may start processing a future word based on their expectations (and before knowing its true identity);\looseness=-1
\item \emph{coping with uncertainty}: readers may incur an additional processing load when in high uncertainty contexts.
\end{enumerate}
\vspace{-3pt}
We design several experiments to investigate these mechanisms, analysing the relationship between readers' expectations about a word's surprisal and its observed reading times.\looseness=-1

We run our analyses in four naturalistic datasets: two self-paced reading and two eye-tracking.
In line with prior work, we find a significant effect of a word's surprisal on its RTs across all datasets, reaffirming the responsive nature of reading.
In addition, we find the word's contextual entropy to be a significant predictor of its RTs in three of the four analysed datasets---in fact, in two of these, the entropy is a more powerful predictor than the surprisal.
Moreover, we find the R\'enyi\xspace{} entropy with $\alpha=0.5$ to consistently leads to stronger predictors than the Shannon entropy.
This suggests readers may anticipate a future word's surprisal to be a function of the number of available continuations (as opposed to the actual expected surprisal).\looseness=-1

\section{Predicting Reading Times} \label{sec:predictive_rt}

One behaviour of interest in psycholinguistics is reading time allocation, i.e., how much time a reader spends processing each word in a text.
Specifically, reading times are important for psycholinguistics because they offer insights into the mechanisms driving the reading process.
Indeed, there exists a vast literature of such analyses
\citep[\textit{inter alia}]{rayner1998eye,hale2001probabilistic,hale2003information,hale2016information,keller-2004-entropy,smith2008optimal,smith2013-log-reading-time,van-schijndel-schuler-2016-addressing,van-schijndel-linzen-2018-neural,shain-2019-large,shain-2021-cdrnn,shain2021continuous,shain2022deep,wilcox2020predictive,meister-etal-2021-revisiting,meister-etal-2022-analyzing,hoover2022plausibility}.\footnote{For more comprehensive introductions to computational reading time analyses see \citet{rayner1998eye,rayner2005eye}.}

The standard procedure for RT analysis is to first choose a set of variables $\mathbf{x} \in \mathbb{R}^d$ which we believe may impact reading times---e.g., it is common to choose $\mathbf{x}=[|w_t|; \unigramfunc(\word_t)]$, where $|w_t|$ is the length of word $w_t$ and $\unigramfunc(\word_t)$ is its frequency (i.e., its unigram log-probability).
These variables are then used to fit a regressor $f_{\gamparams}(\mathbf{x})$ of reading times:\looseness=-1
\begin{equation} \label{eq:regressor}
    %
    \mathrm{time} (w_t) \sim f_{\gamparams}(\mathbf{x})
\end{equation}
where $f_{\gamparams}: \mathbb{R}^d \rightarrow \mathbb{R}$ and  $\phi$ are learned parameters.
We can then evaluate this regressor by looking at its performance, which is measured by, e.g., $f_{\gamparams}(\mathbf{x})$'s average log-likelihood on held out data.

When comparing different theories of the reading process, each may predict a different architecture $f_{\gamparams}$ or set of variables $\mathbf{x}$ which should be used in \cref{eq:regressor}.
We can then compare these theories by looking at the performance of their associated regressors.
A theory that leads to higher log-likelihoods on held out data has stronger predictive power, which implies that it is a better model of the underlying cognitive mechanisms.
Model $f_{\gamparams}(\mathbf{x})$ can then be used to understand the relationship between the employed predictors $\mathbf{x}$ and reading times.\looseness=-1

\subsection{Responsive Reading}

One of the most studied variables in the above paradigm is \defn{surprisal}, which measures a word's information content.
Surprisal theory \cite{hale2001probabilistic} posits that a word's surprisal should directly impact
its processing cost, which makes sense at an intuitive level: the higher a word's information content, the more resources it should take to process that word.
Surprisal theory has since sparked a line of research exploring the relationship between surprisal and processing load, where a word's processing load is typically quantified as its reading time.\footnote{\citet{levy_thesis}, for instance, showed that a word's surprisal quantifies a change in the reader's belief over sentence continuations. He then posited this change in belief may be reflected as processing cost.\looseness=-1}\looseness=-1

Formally, the surprisal (or information content) of a word is defined as its in-context negative log-likelihood \citep{shannon1948mathematical}. We write it as:\looseness=-1
\begin{equation}
	\surpfunc(\word_t) \defeq -\log p(w_{t} \mid \mathbf{w}_{\leq t})
\end{equation}
where $p$ is the ground-truth probability distribution over natural language utterances.
This definition has an intuitive interpretation: a word is more surprising---and thus conveys more information---if it is less likely, and \emph{vice-versa}.\looseness=-1

Time and again, the surprisal has proven to be a strong predictor in reading time analyses \citep[\emph{inter alia}]{smith2008optimal,smith2013-log-reading-time,goodkind-bicknell-2018-predictive,wilcox2020predictive}.
In other words, adding surprisal as a predictor in models of reading time significantly increases their predictive power.
Importantly, surprisal (as well as other properties of a word, like frequency or length) is a quantity that can only feasibly impact readers' behaviours after they have encountered the word in question.\footnote{This property follows from standard theories of causality. \citet{granger1969investigating}, for instance, posits that future material cannot influence present behaviour.}
Thus, by limiting their analyses to such characteristics, these prior works assume reading time allocation is purely \defn{responsive} to the current context a reader finds themselves in, happening on demand as needed for processing a word.

\subsection{Anticipatory Reading}

Not all reading behaviours, however, can be characterised as reactive.
As a concrete example, readers often skip words---a decision which must be made while the next word's identity is unknown.
Furthermore, prior work  has shown that the uncertainty over a sentence's continuations impacts reading time \cite{roark-etal-2009-deriving,angele2015successor,van2017approximations,van-schijndel-linzen-2019-entropy}. Both of these observations offer initial evidence that some form of \defn{anticipatory} planning is being performed by the reader, influencing the way that they read a text.

The presence of such forms of anticipatory processing suggests that, beyond a word's surprisal, a reader's \emph{predictions} about a word may influence the time they take to process it.
A word's reading time, for instance, could be (at least partly) planned before arriving at it, based on the reader's expectation of the amount of work  necessary for processing that word.
This expectation has a formal definition, the \defn{contextual entropy}, which is defined as follows:
\begin{align}\label{eq:shannon_ent}
    \entfunc(W_t) & \defeq \expect_{w \sim p}\left[h(w)\right] \\
    &=- \sum_{w \in \mathcal{W}} p(w \mid \words_{< t}) \log p (w \mid \words_{< t}) \nonumber
\end{align}
Here we denote the random variable associated with this distribution as $W_t$, which takes on values from a (potentially infinite) vocabulary $\mathcal{W}$.\footnote{Prior work has also investigated the role of entropy in reading times.
\citet{roark-etal-2009-deriving}, \citet{linzen-jaeger-2014-investigating} and \citet{van2017approximations}, for instance, investigate the role of successor entropy (i.e., word $t+1$'s entropy) on RTs, while \citet{hale2003information} investigates the role of entropy reduction (i.e., word $t$'s minus $t+1$'s entropy).
In this work, we are instead interested on the role of the entropy of word $t$ itself.\looseness=-1}

\subsection{Skewed Anticipations} \label{sec:skewed_anticipations}

\Cref{eq:shannon_ent} gives us a mathematically optimal guess (i.e., in terms of mean squared error) of what the surprisal of the anticipated word $w_t$ will be, knowing only its context $\words_{< t}$.
However, a reader may employ a different strategy when making anticipatory predictions.
One possibility, for instance, is that readers could be overly confident, and trust their best (i.e., most likely) guess when making this prediction.
In this case, readers would instead anticipate a subsequent word's surprisal to be:
\begin{align} \label{eq:argmin_ent}
    &\renyientparam{\infty} = - \log p (w^* \mid \words_{< t}), \\
    &\quad\quad \text{where}\quad
    w^* = \mathrm{argmax}_{w \in \mathcal{W}}\, p(w \mid \words_{< t}) \nonumber
\end{align}
Another possibility is that readers could ignore each word's specific probability value when anticipating future surprisals; predicting the next-word's surprisal
to instead be the number of competing words with non-zero probability:\footnote{While most state-of-the-art language models cannot assign zero probabilities to a word---due to their use of a softmax in their final layers---it is plausible that humans could.}
\begin{align} \label{eq:argmax_ent}
    &\renyientparam{0} = - \log \frac{1}{\mathrm{supp}(p)}, \\
    &\quad \text{where}\,\,\,
	\mathrm{supp}(p) = \sum_{w \in \mathcal{W}} \mathbbm{1} \bigg\{ p(w \mid \words_{< t}) > 0 \bigg\} \nonumber
\end{align}

The above anticipatory predictions can be written in a unified framework using the \defn{contextual R\'enyi\xspace{} entropy} \citep{renyi1961measures}.
Formally, the R\'enyi\xspace{} entropy generalises Shannon's entropy (in \cref{eq:shannon_ent}), being defined as:\looseness=-1%
\begin{equation} \label{eq:renyi}
    \renyientfunc\!\left(W_{t}\right) = \frac{1}{1 - \alpha} \log \sum_{w \in \mathcal{W}} \bigg( p(w \mid \words_{< t}) \bigg)^\alpha
\end{equation}
It is easy to see that the R\'enyi\xspace{} entropy is equivalent to \cref{eq:argmax_ent} as $\alpha \rightarrow 0$. Further, it is equivalent to \cref{eq:argmin_ent} in the limit of $\alpha\rightarrow\infty$. Finally, the R\'enyi\xspace{} entropy is equal to Shannon's entropy (i.e., to \cref{eq:shannon_ent}) in the limit of $\alpha \rightarrow 1$.

\section{Hypothetical Mechanisms Behind Anticipatory Effects} \label{sec:mechanisms}

In this paper, we are mainly interested in the effect of anticipations on reading times, where  we operationalise anticipation in terms of the contextual entropies defined above.
We consider four main mechanisms under which anticipation could effect reading times: word-skipping; budgeting; preemptive processing; and coping with uncertainty.
We discuss each of these in turn now.

\newcommand{-2pt}{-2pt}

\vspace{-2pt}
\paragraph{Word-skipping.}
The first (and perhaps most obvious) way in which anticipation could affect reading times is by allowing readers to skip words entirely, allocating the word a reading time of zero.
A reader must, by definition, decide whether or not to skip a word before fixating on it---when the word's identity is not known yet.\footnote{At least considering the reader is not able to identify the following word through their parafoveal vision.}
The reader may thus decide to skip a word when its contextual entropy is low because they are confident in the word's identity.
It follows that the contextual entropy may help us predict when readers will skip words.\looseness=-1

\vspace{-2pt}
\paragraph{Budgeting.}
The reading process can be described as a sequence of fixations and saccades.\footnote{Fixations are moments when the gaze focuses on a word, and saccades are rapid eye movements which shift gaze from one point to another. In self-paced reading, saccades are instead replaced by mouse clicks.}
These saccades, however, do not happen instantly: on average, they must be planned at least 125 milliseconds in advance \citep{reichle2009using}.
Further, there is an average eye-to-brain delay of 50ms \citep{pollatsek2008}.
It follows that, for a word's surprisal to be taken into account when allocating reading times, it must be observed for at least 175ms---if not more.
Considering this delay on saccade execution, it is not unreasonable that reading times could be decided (or budgeted) further in advance, when the reader still does not know the current word's identity.
If a reader indeed budgets reading times before they get to a word, reading times should be predictable from the contextual entropy.
Processing costs, however, may still be driven by surprisal.
In this case, we might observe budgeting effects: e.g., if a reader \emph{under}-budgets reading times for a word---i.e., if the word's contextual entropy is smaller than its actual surprisal---we may see a compensation, which could manifest as larger spillover effects in the following word.

\vspace{-2pt}
\paragraph{Preemptive Processing.}
In an influential paper, \citet{smith2008optimal}
posit that for a reader to achieve optimal processing times, they must minimise a trade-off between preemptive processing and actual reading costs.
They then derive what a reader's optimal reading time and preprocessing effort should be.
Notably, given their derivations,\footnote{
They argue a word's optimal reading time should be proportional to $\surpfunc(\word_t)$, and its preprocessing cost should be proportional to $e^{-t}$.
By plugging one equation into the other, and summing over the vocabulary, we find preprocessing costs should be fixed:
$\mathrm{preprocess}(W_t) \propto \sum_{w \in \mathcal{W}} e^{-h(w)} = 1$.
}
a reader should always allocate a constant amount of time for preprocessing future words.\footnote{Notably, \citet{linzen-jaeger-2014-investigating} have found no (i.e., a constant) effect of the successor entropy on RTs, albeit when investigating a different hypothesis to the one discussed here.}
On another note, \citet{goldstein2022shared} recently showed the brain's processing load before a word's onset correlates negatively with its entropy.
This suggests the brain starts preemptively processing a future word before reaching it---and that this is specially true in low entropy contexts.
This could mean that a smaller processing load for low entropy words at time $t$ should be compensated by a higher load in the previous word.
Finally, both \citet{roark-etal-2009-deriving} and \citet{van2017approximations} find that successor entropy (i.e. word $t+1$'s entropy) has a positive impact on reading times, meaning that when a word has a smaller entropy, the previous word takes a shorter time to be read instead.
Given all this prior work, preemptive processing makes no specific prediction about the effect of successor entropy on RTs; it is compatible with a negative, constant, or positive effect. We will investigate which is true later in our experiments.

\vspace{-2pt}
\paragraph{Coping with Uncertainty.}
Finally, uncertainty about a word's identity---as quantified by its contextual entropy---may cause an increase in processing load directly.
For example, keeping a large number of competing word continuations under consideration may require additional cognitive resources, impacting the reader's processing load beyond the effect of the observed word's surprisal.
We know, however, no way of testing this hypothesis directly under our experimental setup.
Ergo, we will not consider this mechanism for the rest of this paper, leaving it out from our analyses.

\section{Experimental Setup}

\subsection{Information-theoretic Estimators} \label{sec:estimating_info}

Unfortunately, we cannot compute the values discussed in \cref{sec:predictive_rt}, as we do not have access to the true natural language distribution $p(\cdot \mid \words_{< t})$.
We can, however, estimate these values using a language model $p_{\btheta}(\cdot \mid \words_{< t})$.
We will thus use $p_{\btheta}$ in place of $p$ in order to estimate all the information-theoretic quantities in \cref{sec:predictive_rt}.
Using such language model-based estimators is standard practice when investigating the surprisal--RT relationship.
In fact, a language model's quality has been shown to correlate with its psychometric predictive power \citep{goodkind-bicknell-2018-predictive,wilcox2020predictive}.

\newcommand{GPT-2\xspace}{GPT-2\xspace}
\newcommand{GPT-2 \texttt{small}\xspace}{GPT-2 \texttt{small}\xspace}

\vspace{-3pt}
\paragraph{Language Models.}
We use GPT-2\xspace{} \cite{gpt2} as our language model $p_{\btheta}$ in all experiments.\footnote{We use GPT-2\xspace's implementation by \citet{wolf-etal-2020-transformers}.}
GPT-2\xspace predicts sub-word units at each time-step (instead of predicting full words).
Ergo, we sum over sub-word units' surprisal estimates to get these measures per word.
Estimating the contextual entropies per word is harder, though, as the vocabulary $\mathcal{W}$ is an infinite-sized set.
We approximate these contextual entropies by computing them over a single-step of sub-word units instead.
In practice, we thus compute a lower bound on the true contextual entropies, as we show in \cref{app:lowerbound}.

\subsection{Data}

We perform our analyses on two eye-tracking and two self-paced reading datasets.
The self-paced reading corpora we study are Natural Stories \citep{futrell-etal-2018-natural} and Brown \citep{smith2013-log-reading-time},
while the eye-tracking corpora are Provo \citep{provo} and Dundee \citep{dundee}.
We refer the readers to \cref{app:data} for more details on these corpora, as well as dataset statistics and preprocessing steps.
For the eye-tracking data, we focus our analyses on Progressive Gaze Duration---we consider the reading time for a word to be the sum of all fixations before the reader first leaves it; and we only consider fixations in a reader's first forward pass.
Further, for our first set of experiments,
we consider a skipped word's reading time as
zero\footnote{Notably, this goes against the standard practice of simply discarding skipped words from the analyses. We do this because: (i) we are interested in word-skipping as a mechanism through which anticipation can impact reading times; (ii) we want to make the eye-tracking setting more closely comparable to the self-paced reading (where fully skipping a word is not possible).\looseness=-1
}---we will denote these datasets as Provo (\cmark) and Dundee (\cmark).
In later experiments we discard skipped words, denoting these datasets with an (\xmark{}) instead.
Following prior work \citep[e.g.,][]{wilcox2020predictive}, we average RT measurements across participants, analysing one RT value per word token.

\subsection{Predictive Function $f_{\gamparams}$}

Prior work has shown the surprisal--reading time relationship to be mostly linear \citep{smith2008optimal,smith2013-log-reading-time,shain2022large}.
Assuming this linearity extends to the contextual entropy,
we will restrict our predictive function to the linear: $f_{\gamparams}(\mathbf{x}) = \boldsymbol{\phi}^{\intercal} \mathbf{x}$.
Assuming $f_{\gamparams}(\mathbf{x})$'s errors follow a normal distribution, its negative log-likelihood $\mathrm{llh}(f_{\gamparams}(\mathbf{x}))$ is defined as:
\begin{align}
    %
    %
    %
    \frac{1}{2N} \!\!\sum_{(\mathbf{x}, y) \in \dataset_{\mathrm{test}}}\!\!\!\!\left(\log(2\pi\sigma^2) + \frac{\left(f_{\gamparams}(\mathbf{x}) - y\right)^2}{\sigma^2} \right)
    %
    %
\end{align}
where $\sigma$ is a standard deviation, $\dataset_{\mathrm{test}}$ is a held-out testset, and $y$ are reading times.\looseness=-1

\subsection{Evaluation}
As mentioned above, we evaluate the different sentence processing hypotheses by looking at the respective predictive power of the regressors that correspond to them---operationalised as their negative log-likelihood on held out data.
We use 10-fold cross-validation, estimating our regressors (depicted in \cref{eq:regressor}) using 9 folds of the data at a time, and evaluating them on the 10$^\text{th}$.
Further, as is standard in reading time analyses, we test the predictive power of a hypothesis by comparing a target model against a baseline model.
Models are the same, except the target model contains a predictor of interest, whereas the baseline model does not.
For these analyses, our metric of interest is the difference in negative log-likelihood between the two models:\footnote{Significance is assessed using paired permutation tests.
We correct for multiple hypothesis testing \citep{benjamini1995controlling} and mark: \textcolor{mygreen}{in green} significant $\Delta_{\mathrm{llh}}$ where a variable adds predictive power (i.e., when the model with more predictors is better); \textcolor{myred}{in red} significant $\Delta_{\mathrm{llh}}$ where a variable leads to overfitting (i.e., when the model with more predictors is worse).
$^*$~$p<0.05$, $^{**}$~$p<0.01$, $^{***}$~$p<0.001$\looseness=-1}\looseness=-1
\begin{equation}
    \Delta_{\mathrm{llh}} = \mathrm{llh}(f_{\gamparams}(\bx_{\mathrm{base}})) - \mathrm{llh}(f_{\gamparams}(\bx_{\mathrm{model}}))
\end{equation}
All of our models include the unigram frequency and the length of the current and three previous words, i.e., $[\unigramfunc(\words_{:}); |\words_{:}|]$ as predictors; we omit explicit reference to these predictors for succinctness.\looseness=-1

\vspace{-3pt}
\paragraph{Notation.} For notational succinctness, we will write vectors containing predictors for the current plus the three previous words as $\surpfunc(\words_{:})$.
Similarly, we will write vectors including three of these four words as
$h(\mathbf{w}_{\neq i})$.
Further, we will write the concatenation of multiple predictors using vector notation, i.e., as $\mathbf{x} = [\entfunc(W_t); \surpfunc(\words_{:})]$.
Finally, unless otherwise noted, we will use a regressor with all four surprisal predictors (besides frequencies and lengths) as our baseline: $\bx_{\mathrm{base}} = [\surpfunc(\words_{:})]$.

\section{Experiments and Results}

\begin{table}[t]
    \centering
\resizebox{\columnwidth}{!}{%
    \begin{tabular}{lccccc}
    \toprule
         &
         & \multicolumn{4}{c}{Surprisal}
         \\
         \cmidrule(lr){3-6}
         &
         & $w_{t\!-\!3}$ & $w_{t\!-\!2}$ & $w_{t\!-\!1}$ & $w_{t}$
         \\
\midrule

\multicolumn{2}{l}{Brown} & \phantom{-}\textcolor{mygreen}{0.33}$^{***}$& \phantom{-}\textcolor{mygreen}{0.47}$^{***}$& \phantom{-}\textcolor{mygreen}{2.58}$^{***}$& \phantom{-}\textcolor{mygreen}{0.50}$^{*}$\phantom{$^{**}$} \\
\multicolumn{2}{l}{Natural Stories} & \phantom{-}\textcolor{mygreen}{0.20}$^{*}$\phantom{$^{**}$}& \phantom{-}\textcolor{mygreen}{0.34}$^{*}$\phantom{$^{**}$}& \phantom{-}\textcolor{mygreen}{1.05}$^{***}$& \phantom{-}\textcolor{mygreen}{1.54}$^{***}$ \\
Provo & (\cmark) & \phantom{-}{0.07}\phantom{$^{***}$}& \phantom{-}{0.18}\phantom{$^{***}$}& \phantom{-}\textcolor{mygreen}{0.83}$^{*}$\phantom{$^{**}$}& \phantom{-}\textcolor{mygreen}{3.22}$^{**}$\phantom{$^{*}$} \\
Dundee & (\cmark) & {-0.00}\phantom{$^{***}$}& \phantom{-}\textcolor{mygreen}{0.04}$^{**}$\phantom{$^{*}$}& \phantom{-}\textcolor{mygreen}{0.25}$^{***}$& \phantom{-}\textcolor{mygreen}{0.89}$^{***}$ \\

\bottomrule
\\[-8pt]
\multicolumn{6}{l}{\small
$\bx_{\mathrm{model}}=[\surpfunc(\words_{:})]$ vs.
$\bx_{\mathrm{base}}=[\surpfunc(\words_{\neq i})]$
}
    \end{tabular}
}
\vspace{-5pt}
    %
    \caption{$\Delta_{\mathrm{llh}}$ (in $10^{-2}$ nats) when comparing a model with all surprisal terms against baselines from which a single surprisal term was removed. Green indicates a significantly positive impact of surprisal on the model's predictive power.}
    \label{tab:delta_llh_surprisal}
    \vspace{-7pt}
\end{table}

\subsection{Experiment \#1. Confirmatory Analysis}

In this first experiment, we simply confirm prior results which show the predictive power of surprisal over reading times,
estimating the $\Delta_{\mathrm{llh}}$ between a model with all surprisal terms as predictors, and different baselines for which we remove a single surprisal term.\footnote{Prior work has shown that a word's RTs are impacted not only by its own surprisal, but also by the surprisal of previous words; these effects are referred to as spillover effects.\looseness=-1}
We present these results in \cref{tab:delta_llh_surprisal}.
In this table, we can see that the surprisal of the current word is a strong predictor of reading times in all four analysed datasets.
Moreover, we see significant spillover effects for the surprisal of three previous words in self-paced reading corpora, for the two previous words in Dundee, and for the immediately previous one in Provo.
Interestingly, and consistent with prior work \citep{smith2008optimal}, we find that spillover effects are stronger than the current word's effect in Brown.
For the other three datasets, however, we find the main surprisal effect to be stronger than spillovers.

\subsection{Experiment \#2. Surprisal vs. Entropies} \label{sec:exp_entropy_vs_surprisal}

In this second experiment, we move on to analyse the predictive power of the contextual entropy on reading times.
Specifically, \cref{tab:delta_llh_entropy} presents the $\Delta_{\mathrm{llh}}$ between our full baseline model and: (i) a model where one of the surprisal terms is replaced by an entropy term; (ii) a model where we add an entropy term besides the predictors already in $\bx_{\mathrm{base}}$.
From this table, we see that adding the entropy of the current word (i.e., $\entfunc(W_t)$) significantly increases the predictive power in three out of the four analysed datasets.
Furthermore, replacing the surprisal predictor with the entropy only leads to a model with worse predictive power in one of the three analysed datasets (in Provo).
On the other three datasets, the entropy's predictive power is the same as the surprisal's (more precisely, they are not statistically different).
Together, these results suggest that the reading process is both responsive and anticipatory.\looseness=-1

\begin{table}[t]
    \centering
\resizebox{\columnwidth}{!}{%
    \begin{tabular}{lccccc}
    \toprule
         && $w_{t\!-\!3}$ & $w_{t\!-\!2}$ & $w_{t\!-\!1}$ & $w_{t}$
         \\
\midrule
\multicolumn{6}{l}{Replace Surprisal with Entropy$^1$} \\
\cmidrule(lr){1-1}
    \multicolumn{2}{l}{Brown} & \textcolor{myred}{-0.30}$^{*}$\phantom{$^{**}$}& \textcolor{myred}{-0.35}$^{**}$\phantom{$^{*}$}& \textcolor{myred}{-1.68}$^{***}$& {-0.03}\phantom{$^{***}$} \\
    \multicolumn{2}{l}{Natural Stories} & {-0.03}\phantom{$^{***}$}& \textcolor{myred}{-0.19}$^{*}$\phantom{$^{**}$}& \textcolor{myred}{-0.41}$^{*}$\phantom{$^{**}$}& \phantom{-}{0.37}\phantom{$^{***}$} \\
    Provo & (\cmark) & {-0.08}\phantom{$^{***}$}& \phantom{-}{0.18}\phantom{$^{***}$}& \textcolor{myred}{-0.66}$^{*}$\phantom{$^{**}$}& \textcolor{myred}{-2.58}$^{*}$\phantom{$^{**}$} \\
    Dundee & (\cmark) & {-0.00}\phantom{$^{***}$}& \phantom{-}\textcolor{mygreen}{0.03}$^{*}$\phantom{$^{**}$}& \textcolor{myred}{-0.21}$^{***}$& {-0.07}\phantom{$^{***}$} \\
\midrule
\multicolumn{6}{l}{Add Entropy$^2$} \\
\cmidrule(lr){1-1}
    \multicolumn{2}{l}{Brown} & \textcolor{myred}{-0.03}$^{*}$\phantom{$^{**}$}& {-0.01}\phantom{$^{***}$}& \phantom{-}{0.04}\phantom{$^{***}$}& \phantom{-}\textcolor{mygreen}{0.15}$^{*}$\phantom{$^{**}$} \\
    \multicolumn{2}{l}{Natural Stories} & \phantom{-}{0.04}\phantom{$^{***}$}& \phantom{-}{0.01}\phantom{$^{***}$}& \phantom{-}\textcolor{mygreen}{0.14}$^{***}$& \phantom{-}\textcolor{mygreen}{0.89}$^{***}$ \\
    Provo & (\cmark) & \textcolor{myred}{-0.04}$^{*}$\phantom{$^{**}$}& \phantom{-}{0.16}\phantom{$^{***}$}& {-0.03}\phantom{$^{***}$}& {-0.06}\phantom{$^{***}$} \\
    Dundee & (\cmark) & \textcolor{myred}{-0.00}$^{**}$\phantom{$^{*}$}& \phantom{-}\textcolor{mygreen}{0.03}$^{**}$\phantom{$^{*}$}& {-0.00}\phantom{$^{***}$}& \phantom{-}\textcolor{mygreen}{0.25}$^{***}$ \\
\bottomrule
\\[-8pt]
\multicolumn{6}{l}{\small
$^1$
$\bx_{\mathrm{model}}=[\mathrm{H}(W_i); \surpfunc(\words_{\neq i})]$
\,\,\,\,\,\,
$^2$
$\bx_{\mathrm{model}}=[\mathrm{H}(W_i); \surpfunc(\words_{:})]$} \\
\multicolumn{6}{l}{\small
$^\mathrm{both}$
$\bx_{\mathrm{base}}=[\surpfunc(\words_{:})]$,
}
    \end{tabular}
}
    %
\vspace{-5pt}
    \caption{$\Delta_{\mathrm{llh}}$ (in $10^{-2}$ nats) achieved after either replacing a surprisal term in the baseline with Shannon's entropy (top), or adding the entropy as an extra predictor (bottom). Green indicates a significant gain in $\Delta_{\mathrm{llh}}$, red a significant loss.}
    \label{tab:delta_llh_entropy}
    \vspace{-7pt}
\end{table}

Analysing the impact of the previous words' entropies (i.e. $\entfunc(W_{t\!-\!1})$,  $\entfunc(W_{t\!-\!2})$,  $\entfunc(W_{t\!-\!3})$) on reading times, we see a somewhat different story.
When adding spillover entropy terms as extra predictors we see no consistent improvements in predictive power; with a weak improvement on self-paced reading datasets for word $t\!-\!1$ (specifically, this is only significant in Natural Stories) and a similarly weak one for word $t\!-\!2$ on eye-tracking data (only significant in Dundee).
This lack of predictive power further stands-out when contrasted to surprisal spillover effects: replacing surprisal spillover terms with the corresponding entropy terms mostly leads to models with weaker predictive power.
Together, these results imply the effect of expectations on reading times is mostly local---i.e., the expectation over a word's surprisal impacts its reading time, but not future words' RTs.\looseness=-1

\newcommand{.4cm}{.4cm}

\begin{figure*}[t]
    \centering
    \begin{subfigure}[b]{.25\textwidth}
        \includegraphics[width=\columnwidth,trim={0 .4cm{} 0 0},clip]{figs/entropy--llh--brown.pdf}
        \caption{Brown}
    \end{subfigure}%
    ~
    \begin{subfigure}[b]{.25\textwidth}
        \includegraphics[width=\columnwidth,trim={0 .4cm{} 0 0},clip]{figs/entropy--llh--natural_stories.pdf}
        \caption{Natural Stories}
    \end{subfigure}%
    ~
    \begin{subfigure}[b]{.25\textwidth}
        \includegraphics[width=\columnwidth,trim={0 .4cm{} 0 0},clip]{figs/entropy--llh--provo5_skip2zero.pdf}
        \caption{Provo (\cmark)}
    \end{subfigure}%
    ~
    \begin{subfigure}[b]{.25\textwidth}
        \includegraphics[width=\columnwidth,trim={0 .4cm{} 0 0},clip]{figs/entropy--llh--dundee5_skip2zero.pdf}
        \caption{Dundee (\cmark)}
    \end{subfigure}
    \vspace{-17pt}
    %
    %
    \caption{$\Delta_{\mathrm{llh}}$ when adding either the current word's surprisal, R\'enyi\xspace{} entropy, or both on top of a baseline that includes the surprisal of previous words as predictors, i.e., $\bx_{\mathrm{base}} = [h(\mathbf{w}_{\neq t})]$.
    Shaded regions correspond to 95\% confidence intervals.}
    \label{fig:renyi}
    \vspace{-5pt}
\end{figure*}

\begin{table*}[t]
    \centering
\resizebox{.85\textwidth}{!}{%
    \begin{tabular}{lccccccccc}
    \toprule
         &
         & \multicolumn{4}{c}{Replace Surprisal with R\'enyi\xspace{} Entropy$^1$}
         & \multicolumn{4}{c}{Add R\'enyi\xspace{} Entropy$^2$}
         \\
         \cmidrule(lr){3-6}
         \cmidrule(lr){7-10}
         &
         & $w_{t\!-\!3}$ & $w_{t\!-\!2}$ & $w_{t\!-\!1}$ & $w_{t}$
         & $w_{t\!-\!3}$ & $w_{t\!-\!2}$ & $w_{t\!-\!1}$ & $w_{t}$
         \\
\midrule

\multicolumn{2}{l}{Brown} & \textcolor{myred}{-0.35}$^{***}$& \textcolor{myred}{-0.37}$^{**}$\phantom{$^{*}$}& \textcolor{myred}{-1.76}$^{***}$& \phantom{-}{0.03}\phantom{$^{***}$}& {-0.01}\phantom{$^{***}$}& \phantom{-}{0.00}\phantom{$^{***}$}& \phantom{-}{0.14}\phantom{$^{***}$}& \phantom{-}\textcolor{mygreen}{0.26}$^{*}$\phantom{$^{**}$} \\
\multicolumn{2}{l}{Natural Stories} & {-0.16}\phantom{$^{***}$}& \textcolor{myred}{-0.27}$^{*}$\phantom{$^{**}$}& {-0.19}\phantom{$^{***}$}& \phantom{-}\textcolor{mygreen}{1.30}$^{**}$\phantom{$^{*}$}& {-0.00}\phantom{$^{***}$}& {-0.00}\phantom{$^{***}$}& \phantom{-}\textcolor{mygreen}{0.44}$^{***}$& \phantom{-}\textcolor{mygreen}{2.04}$^{***}$ \\
Provo & (\cmark) & {-0.05}\phantom{$^{***}$}& \phantom{-}{0.47}\phantom{$^{***}$}& \textcolor{myred}{-0.89}$^{*}$\phantom{$^{**}$}& \textcolor{myred}{-3.25}$^{**}$\phantom{$^{*}$}& {-0.01}\phantom{$^{***}$}& \phantom{-}\textcolor{mygreen}{0.45}$^{*}$\phantom{$^{**}$}& {-0.01}\phantom{$^{***}$}& {-0.04}\phantom{$^{***}$} \\
Dundee & (\cmark) & \phantom{-}{0.00}\phantom{$^{***}$}& \phantom{-}\textcolor{mygreen}{0.07}$^{*}$\phantom{$^{**}$}& \textcolor{myred}{-0.25}$^{***}$& \phantom{-}\textcolor{mygreen}{0.32}$^{*}$\phantom{$^{**}$}& {-0.00}\phantom{$^{***}$}& \phantom{-}\textcolor{mygreen}{0.08}$^{*}$\phantom{$^{**}$}& \phantom{-}{0.05}\phantom{$^{***}$}& \phantom{-}\textcolor{mygreen}{0.64}$^{***}$ \\

\bottomrule
\\[-8pt]
\multicolumn{9}{l}{\small
$^1$
$\bx_{\mathrm{model}}=[\entfunc_{\alpha}(W_i); \surpfunc(\words_{\neq i})]$
\,\,\,\,\,\,
$^2$
$\bx_{\mathrm{model}}=[\entfunc_{\alpha}(W_i); \surpfunc(\words_{:})]$
\,\,\,\,\,\,
$^\mathrm{both}$
$\bx_{\mathrm{base}}=[\surpfunc(\words_{:})]$,
}
    \end{tabular}
}
\vspace{-5pt}
    \caption{$\Delta_{\mathrm{llh}}$ (in $10^{-2}$ nats) achieved after either replacing a surprisal term in the baseline with the contextual R\'enyi\xspace{} entropy ($\alpha=0.5$), or adding the R\'enyi\xspace{} entropy as an extra predictor.}
    \label{tab:delta_llh_entropy_renyi}
    \vspace{-7pt}
\end{table*}

\begin{table*}[t]
    \centering
    \small
    \begin{tabular}{llcccccc}
    \toprule
& & \multicolumn{3}{c}{Provo}
& \multicolumn{3}{c}{Dundee}
\\
\cmidrule(lr){3-5}
\cmidrule(lr){6-8}
& $\bx_{\mathrm{base}} \backslash \bx_{\mathrm{model}}$
& $\surpfunc(\word_t)$
& $\renyientfunc\!\left(W_{t}\right)$
& Both
& $\surpfunc(\word_t)$
& $\renyientfunc\!\left(W_{t}\right)$
& Both
\\
\midrule
\multirow{3}{*}{\shortstack[c]{Shannon \\ ($\alpha=1.0$)}}

& $\emptyset$
& \phantom{-}{2.60}\phantom{$^{***}$}
& \phantom{-}{1.76}\phantom{$^{***}$}
& \phantom{-}{2.86}\phantom{$^{***}$}

& \phantom{-}\textcolor{mygreen}{1.30}$^{*}$\phantom{$^{**}$}
& \phantom{-}\textcolor{mygreen}{2.50}$^{***}$
& \phantom{-}\textcolor{mygreen}{2.62}$^{***}$
\\

& $\surpfunc(\word_t)$
& - & {-0.84}\phantom{$^{***}$}& \phantom{-}{0.26}\phantom{$^{***}$}
& - & \phantom{-}{1.20}\phantom{$^{***}$}& \phantom{-}\textcolor{mygreen}{1.32}$^{***}$
\\

& $\renyientfunc\!\left(W_{t}\right)$
& - & - & \phantom{-}{1.10}\phantom{$^{***}$}
& - & - & \phantom{-}{0.12}\phantom{$^{***}$}
\\

\midrule
\multirow{3}{*}{\shortstack[c]{R\'enyi\xspace{} \\ ($\alpha=0.5$)}}
& $\emptyset$
& \phantom{-}{2.60}\phantom{$^{***}$}
& \phantom{-}{0.84}\phantom{$^{***}$}
& \phantom{-}{2.66}\phantom{$^{***}$}

& \phantom{-}\textcolor{mygreen}{1.30}$^{*}$\phantom{$^{**}$}
& \phantom{-}\textcolor{mygreen}{5.10}$^{***}$
& \phantom{-}\textcolor{mygreen}{5.14}$^{***}$
\\
& $\surpfunc(\word_t)$
& -
& {-1.76}\phantom{$^{***}$}
& \phantom{-}{0.06}\phantom{$^{***}$}

& -
& \phantom{-}\textcolor{mygreen}{3.79}$^{***}$
& \phantom{-}\textcolor{mygreen}{3.83}$^{***}$
\\

& $\renyientfunc\!\left(W_{t}\right)$
& -
& -
& \phantom{-}{1.82}\phantom{$^{***}$}

& -
& -
& \phantom{-}{0.04}\phantom{$^{***}$}
\\
\bottomrule
    \end{tabular}
\vspace{-4pt}
    \caption{$\Delta_{\mathrm{llh}}$ (in $10^{-4}$ nats) between a target model (with predictors on columns) vs baseline (with predictors on row) when predicting whether a word was skipped or not. All models also include the surprisal of the previous words as predictors.}
    \label{tab:delta_llh_skip}
    \vspace{-10pt}
\end{table*}

\vspace{-3pt}
\subsection{Experiment \#3. Skewed Expectations}
\label{sec:skewed_expectations_analysis}

We now compare the effect of expectations with different R\'enyi\xspace{} entropies on reading times.
We follow a similar setup to before.
Specifically, we get the contextual R\'enyi\xspace{} entropy for several values of $\alpha$ and train regressors with two sets of input variables: (i) $\bx_{\mathrm{model}}=[\renyientfunc\!\left(W_{t}\right); h(\mathbf{w}_{\neq t})]$; and (ii) $\bx_{\mathrm{model}}=[\renyientfunc\!\left(W_{t}\right); \surpfunc(\words_{:})]$.
In other words, we either replace the current word's surprisal term with the R\'enyi\xspace{} entropy; or simply add the R\'enyi\xspace{} as an extra predictor.
We then plot these values in \cref{fig:renyi}.
Analysing this figure, we see a clear trend in three of the four datasets (here, again, Provo presents different trends from the other datasets): the predictive power of expectations seem to improve for smaller values of $\alpha$.
More precisely, in Brown, Natural Stories and Dundee, an $\alpha=0.5$ seems to lead to stronger predictive powers than $\alpha>0.5$.\looseness=-1

Notably, with different values of $\alpha$, the R\'enyi\xspace{} entropy leads to different interpretations of a reader's anticipation strategies.
Recall from \cref{sec:skewed_anticipations}, that when $\alpha\!=\!1$, R\'enyi\xspace{}'s and Shannon's entropies are equivalent.
When $\alpha\!=\!0$, the R\'enyi\xspace{} entropy measures the size of the support of $p(\cdot \mid \words_{< t})$---or, in other words, the number of competing continuations at a timestep.
The R\'enyi\xspace{} entropy with $\alpha\!=\!.5$ does not have as clear of an intuitive meaning, but it can be interpreted as measuring a soft version of the distribution $p$'s support---where a word needs to have probability above an $\epsilon$, as opposed to above 0, to be counted.\looseness=-1

Based on these results, we then produce a similar table to the previous experiment's, but  using the R\'enyi\xspace{} entropy with $\alpha=0.5$ instead.
These results are depicted in \cref{tab:delta_llh_entropy_renyi}.
Similarly to before, we still see a significant improvement in predictive powers on three of the datasets when adding the entropy as an extra predictor.
Unlike before, however, replacing the surprisal predictors (for time step $t$) with R\'enyi\xspace{} entropy predictors
\emph{significantly improves} log-likelihoods in two of the analysed datasets.
In other words, the R\'enyi\xspace{} entropy has a stronger predictive power than the surprisal in both these datasets.
We now move on to investigate why this is the case, analysing the mechanisms proposed in \cref{sec:mechanisms}.\looseness=-1

\subsection{Experiment \#4. Word-skipping} \label{sec:word-skip_planning}

In \cref{sec:mechanisms}, we discussed four potential mechanisms through which expectations could impact reading times.
In this experiment, we analyse the impact of word-skipping effects on our results.
We start this analysis by, similarly to previous experiments, looking at the $\Delta_{\mathrm{llh}}$ between a baseline and an analysed model.
Unlike before, though, we train a logistic regressor in this experiment, predicting whether or not a word was skipped instead.
Our prediction function can thus be written as:
\begin{equation}
    f_{\gamparams} = \sigma(\boldsymbol{\phi}^\intercal\, \mathbf{x})
\end{equation}
where $\sigma$ is the sigmoid function.

\cref{tab:delta_llh_skip} presents these results.
First, we see that surprisal is a significant predictor of whether or not a word is skipped in Dundee; yet it is not significant in Provo.
Second, we find that in Dundee our predictive power over whether a word was skipped is significantly stronger when using the R\'enyi\xspace{} entropy of the current word than when using its surprisal.
Finally, while we find an improvement in predictive power when adding entropy (besides surprisal) as a predictor, we find no significant improvement when performing the reverse operation.
This implies that, at least for Dundee, word-skipping effects are predicted solely by the entropy, with the surprisal of the current word adding no extra predictive power.

Note that we represented skipped words as having reading times of 0ms in our previous experiments on eye-tracking datasets.
Thus, our previous results could be driven purely by word-skipping effects.
We now run the same experiments in \cref{sec:exp_entropy_vs_surprisal} and \cref{sec:skewed_expectations_analysis}, but with skipped words removed from our analysis.
These results are presented in \cref{tab:delta_llh_entropy_no_skipped}.
In short, when skipped words are not considered, the R\'enyi\xspace{} entropy is not significantly more predictive of RTs than the surprisal anymore; in fact, the surprisal seems to be a slightly stronger predictor (although not significantly in Dundee).
However, adding the R\'enyi\xspace{} entropy as a predictor to a model which already has surprisal still adds significant predictive power in Dundee.
In short, this table shows that, while partly driven by word-skipping, there are still other effects of anticipation on reading times.\looseness=-1

\begin{table}[t]
    \centering
\resizebox{\columnwidth}{!}{%
    \begin{tabular}{llcccc}
    \toprule
         & & $w_{t\!-\!3}$ & $w_{t\!-\!2}$ & $w_{t\!-\!1}$ & $w_{t}$
         \\
\midrule
\multicolumn{5}{l}{Shannon Entropy ($\alpha=1.0$)} \\
\cmidrule(lr){1-1}
\multirow{2}{*}{Replace$^1$}
& Provo & \phantom{-}{0.02}\phantom{$^{***}$}& {-0.03}\phantom{$^{***}$}& {-0.18}\phantom{$^{***}$}& \textcolor{myred}{-2.23}$^{***}$ \\
& Dundee & {-0.01}\phantom{$^{***}$}& {-0.02}\phantom{$^{***}$}& \textcolor{myred}{-0.15}$^{***}$& \textcolor{myred}{-0.32}$^{**}$\phantom{$^{*}$} \\
[7pt]
\multirow{2}{*}{Add$^2$}
& Provo & {-0.02}\phantom{$^{***}$}& {-0.07}\phantom{$^{***}$}& \phantom{-}{0.03}\phantom{$^{***}$}& {-0.01}\phantom{$^{***}$} \\
& Dundee & \textcolor{myred}{-0.00}$^{*}$\phantom{$^{**}$}& \phantom{-}{0.01}\phantom{$^{***}$}& \phantom{-}{0.01}\phantom{$^{***}$}& \phantom{-}\textcolor{mygreen}{0.17}$^{**}$\phantom{$^{*}$} \\
\midrule
\multicolumn{5}{l}{Renyi Entropy ($\alpha=0.5$)} \\
\cmidrule(lr){1-1}
\multirow{2}{*}{Replace$^1$}
& Provo & \phantom{-}{0.02}\phantom{$^{***}$}& \phantom{-}{0.10}\phantom{$^{***}$}& {-0.27}\phantom{$^{***}$}& \textcolor{myred}{-2.43}$^{**}$\phantom{$^{*}$} \\
& Dundee & {-0.01}\phantom{$^{***}$}& \phantom{-}{0.01}\phantom{$^{***}$}& \textcolor{myred}{-0.19}$^{***}$& {-0.18}\phantom{$^{***}$} \\
[7pt]
\multirow{2}{*}{Add$^2$}
& Provo & {-0.02}\phantom{$^{***}$}& \phantom{-}{0.07}\phantom{$^{***}$}& \phantom{-}{0.01}\phantom{$^{***}$}& \phantom{-}{0.32}\phantom{$^{***}$} \\
& Dundee & \textcolor{myred}{-0.00}$^{*}$\phantom{$^{**}$}& \phantom{-}\textcolor{mygreen}{0.05}$^{*}$\phantom{$^{**}$}& \phantom{-}{0.01}\phantom{$^{***}$}& \phantom{-}\textcolor{mygreen}{0.36}$^{***}$ \\
\bottomrule
\\[-8pt]
\multicolumn{6}{l}{
$^1$
$\bx_{\mathrm{model}}=[\entfunc(\Word_{i}); \surpfunc(\words_{\neq i})]$,
\,\,\,\,
$^2$
$\bx_{\mathrm{model}}=[\entfunc(\Word_{i}); \surpfunc(\words_{:})]$,
} \\ \multicolumn{6}{l}{
$^{\mathrm{both}}$
$\bx_{\mathrm{base}}=[\surpfunc(\words_{:})]$
}
    \end{tabular}
}
    \caption{$\Delta_{\mathrm{llh}}$ (in $10^{-2}$ nats) when predicting RTs on eye-tracking datasets where skipped words were removed: i.e., Provo (\xmark) and Dundee (\xmark).\looseness=-1}
    \label{tab:delta_llh_entropy_no_skipped}
    \vspace{-10pt}
\end{table}

\begin{table*}[t]
    \centering
\resizebox{\textwidth}{!}{%
    \begin{tabular}{lccccccccccccc}
\toprule
         &
         & \multicolumn{3}{c}{Budget}
         & \multicolumn{3}{c}{Over-budget}
         & \multicolumn{3}{c}{Under-budget}
         & \multicolumn{3}{c}{Abs-budget}
         \\
         \cmidrule(lr){3-5}
         \cmidrule(lr){6-8}
         \cmidrule(lr){9-11}
         \cmidrule(lr){12-14}
         &
         & $w_{t\!-\!3}$ & $w_{t\!-\!2}$ & $w_{t\!-\!1}$
         & $w_{t\!-\!3}$ & $w_{t\!-\!2}$ & $w_{t\!-\!1}$
         & $w_{t\!-\!3}$ & $w_{t\!-\!2}$ & $w_{t\!-\!1}$
         & $w_{t\!-\!3}$ & $w_{t\!-\!2}$ & $w_{t\!-\!1}$
         \\
\midrule
\multicolumn{5}{l}{Shannon Entropy ($\alpha=1.0$)} \\
\cmidrule(lr){1-1}

\multicolumn{2}{l}{Brown} & {-0.03}\phantom{$^{***}$}& {-0.01}\phantom{$^{***}$}& \phantom{-}{0.02}\phantom{$^{***}$}& {-0.01}\phantom{$^{***}$}& {-0.01}\phantom{$^{***}$}& \phantom{-}{0.07}\phantom{$^{***}$}& {-0.02}\phantom{$^{***}$}& \textcolor{myred}{-0.01}$^{***}$& \textcolor{myred}{-0.02}$^{**}$\phantom{$^{*}$}& \phantom{-}{0.01}\phantom{$^{***}$}& {-0.01}\phantom{$^{***}$}& \phantom{-}{0.03}\phantom{$^{***}$} \\
\multicolumn{2}{l}{Natural Stories} & \phantom{-}{0.04}\phantom{$^{***}$}& \phantom{-}{0.00}\phantom{$^{***}$}& \phantom{-}{0.05}\phantom{$^{***}$}& {-0.01}\phantom{$^{***}$}& {-0.00}\phantom{$^{***}$}& \phantom{-}{0.02}\phantom{$^{***}$}& \phantom{-}{0.04}\phantom{$^{***}$}& {-0.00}\phantom{$^{***}$}& \phantom{-}{0.02}\phantom{$^{***}$}& {-0.01}\phantom{$^{***}$}& {-0.01}\phantom{$^{***}$}& \textcolor{myred}{-0.02}$^{*}$\phantom{$^{**}$} \\

Provo & (\cmark) & \textcolor{myred}{-0.04}$^{***}$& \phantom{-}{0.16}\phantom{$^{***}$}& {-0.04}\phantom{$^{***}$}& \textcolor{myred}{-0.03}$^{**}$\phantom{$^{*}$}& \phantom{-}{0.06}\phantom{$^{***}$}& {-0.03}\phantom{$^{***}$}& {-0.03}\phantom{$^{***}$}& \phantom{-}{0.07}\phantom{$^{***}$}& {-0.02}\phantom{$^{***}$}& {-0.01}\phantom{$^{***}$}& {-0.04}\phantom{$^{***}$}& {-0.01}\phantom{$^{***}$} \\
Dundee & (\cmark) & {-0.00}\phantom{$^{***}$}& \phantom{-}\textcolor{mygreen}{0.03}$^{**}$\phantom{$^{*}$}& \phantom{-}{0.01}\phantom{$^{***}$}& {-0.00}\phantom{$^{***}$}& \phantom{-}{0.02}\phantom{$^{***}$}& \phantom{-}{0.04}\phantom{$^{***}$}& {-0.00}\phantom{$^{***}$}& \phantom{-}{0.01}\phantom{$^{***}$}& {-0.00}\phantom{$^{***}$}& {-0.00}\phantom{$^{***}$}& {-0.00}\phantom{$^{***}$}& \phantom{-}\textcolor{mygreen}{0.03}$^{*}$\phantom{$^{**}$} \\

Provo &  (\xmark) & \textcolor{myred}{-0.02}$^{***}$& {-0.05}\phantom{$^{***}$}& \phantom{-}{0.07}\phantom{$^{***}$}& {-0.01}\phantom{$^{***}$}& \phantom{-}{0.05}\phantom{$^{***}$}& {-0.02}\phantom{$^{***}$}& {-0.04}\phantom{$^{***}$}& {-0.06}\phantom{$^{***}$}& \phantom{-}{0.10}\phantom{$^{***}$}& {-0.03}\phantom{$^{***}$}& \phantom{-}{0.04}\phantom{$^{***}$}& \phantom{-}{0.03}\phantom{$^{***}$} \\
Dundee &  (\xmark) & \textcolor{myred}{-0.00}$^{**}$\phantom{$^{*}$}& \phantom{-}{0.01}\phantom{$^{***}$}& \phantom{-}{0.00}\phantom{$^{***}$}& {-0.00}\phantom{$^{***}$}& \phantom{-}{0.01}\phantom{$^{***}$}& {-0.00}\phantom{$^{***}$}& {-0.00}\phantom{$^{***}$}& \phantom{-}{0.00}\phantom{$^{***}$}& \phantom{-}{0.02}\phantom{$^{***}$}& \phantom{-}{0.00}\phantom{$^{***}$}& {-0.00}\phantom{$^{***}$}& \phantom{-}{0.02}\phantom{$^{***}$} \\

\midrule
\multicolumn{5}{l}{Renyi Entropy ($\alpha=0.5$)} \\
\cmidrule(lr){1-1}

\multicolumn{2}{l}{Brown} & {-0.00}\phantom{$^{***}$}& {-0.00}\phantom{$^{***}$}& \phantom{-}{0.11}\phantom{$^{***}$}& \phantom{-}{0.01}\phantom{$^{***}$}& \phantom{-}{0.00}\phantom{$^{***}$}& \phantom{-}{0.14}\phantom{$^{***}$}& {-0.01}\phantom{$^{***}$}& \textcolor{myred}{-0.01}$^{*}$\phantom{$^{**}$}& {-0.02}\phantom{$^{***}$}& \phantom{-}{0.01}\phantom{$^{***}$}& \phantom{-}{0.00}\phantom{$^{***}$}& \phantom{-}{0.16}\phantom{$^{***}$} \\
\multicolumn{2}{l}{Natural Stories} & {-0.00}\phantom{$^{***}$}& {-0.01}\phantom{$^{***}$}& \phantom{-}\textcolor{mygreen}{0.21}$^{***}$& {-0.01}\phantom{$^{***}$}& \textcolor{myred}{-0.01}$^{*}$\phantom{$^{**}$}& \phantom{-}\textcolor{mygreen}{0.23}$^{***}$& \phantom{-}{0.00}\phantom{$^{***}$}& {-0.00}\phantom{$^{***}$}& {-0.01}\phantom{$^{***}$}& {-0.02}\phantom{$^{***}$}& \textcolor{myred}{-0.01}$^{*}$\phantom{$^{**}$}& \phantom{-}\textcolor{mygreen}{0.21}$^{**}$\phantom{$^{*}$} \\

Provo & (\cmark) & {-0.01}\phantom{$^{***}$}& \phantom{-}\textcolor{mygreen}{0.48}$^{*}$\phantom{$^{**}$}& {-0.03}\phantom{$^{***}$}& {-0.01}\phantom{$^{***}$}& \phantom{-}\textcolor{mygreen}{0.42}$^{*}$\phantom{$^{**}$}& {-0.02}\phantom{$^{***}$}& {-0.02}\phantom{$^{***}$}& \phantom{-}{0.05}\phantom{$^{***}$}& \textcolor{myred}{-0.04}$^{**}$\phantom{$^{*}$}& {-0.02}\phantom{$^{***}$}& \phantom{-}{0.26}\phantom{$^{***}$}& {-0.01}\phantom{$^{***}$} \\
Dundee & (\cmark) & {-0.00}\phantom{$^{***}$}& \phantom{-}\textcolor{mygreen}{0.08}$^{*}$\phantom{$^{**}$}& \phantom{-}\textcolor{mygreen}{0.09}$^{*}$\phantom{$^{**}$}& {-0.00}\phantom{$^{***}$}& \phantom{-}\textcolor{mygreen}{0.07}$^{*}$\phantom{$^{**}$}& \phantom{-}\textcolor{mygreen}{0.10}$^{**}$\phantom{$^{*}$}& \textcolor{myred}{-0.00}$^{***}$& \phantom{-}{0.01}\phantom{$^{***}$}& \textcolor{myred}{-0.00}$^{***}$& {-0.00}\phantom{$^{***}$}& \phantom{-}{0.06}\phantom{$^{***}$}& \phantom{-}\textcolor{mygreen}{0.10}$^{**}$\phantom{$^{*}$} \\

Provo & (\xmark) & {-0.01}\phantom{$^{***}$}& \phantom{-}{0.10}\phantom{$^{***}$}& \phantom{-}{0.04}\phantom{$^{***}$}& {-0.03}\phantom{$^{***}$}& \phantom{-}{0.15}\phantom{$^{***}$}& \phantom{-}{0.01}\phantom{$^{***}$}& \phantom{-}{0.03}\phantom{$^{***}$}& \textcolor{myred}{-0.03}$^{*}$\phantom{$^{**}$}& \phantom{-}{0.04}\phantom{$^{***}$}& {-0.05}\phantom{$^{***}$}& \phantom{-}{0.15}\phantom{$^{***}$}& {-0.01}\phantom{$^{***}$} \\
Dundee & (\xmark) & {-0.00}\phantom{$^{***}$}& \phantom{-}\textcolor{mygreen}{0.04}$^{*}$\phantom{$^{**}$}& \phantom{-}{0.00}\phantom{$^{***}$}& {-0.00}\phantom{$^{***}$}& \phantom{-}\textcolor{mygreen}{0.04}$^{*}$\phantom{$^{**}$}& {-0.00}\phantom{$^{***}$}& {-0.00}\phantom{$^{***}$}& {-0.00}\phantom{$^{***}$}& \phantom{-}{0.01}\phantom{$^{***}$}& \textcolor{myred}{-0.00}$^{***}$& \phantom{-}\textcolor{mygreen}{0.04}$^{*}$\phantom{$^{**}$}& {-0.00}\phantom{$^{***}$} \\

\bottomrule
\\[-10pt]
\multicolumn{6}{l}{
\small
$\bx_{\mathrm{base}} = [\renyientfunc\!\left(W_{t}\right); \surpfunc(\words_{:})]$,
}
    \end{tabular}
}
\vspace{-6pt}
    %
    %
    \caption{$\Delta_{\mathrm{llh}}$ (in $10^{-2}$ nats) achieved when predicting RTs after adding budgeting effect predictors on top of a baseline with entropy and surprisal as predictors.}
    \label{tab:delta_llh_budget}
    \vspace{-5pt}
\end{table*}

\subsection{Experiment \#5. Budgeting Effects}

\newcommand{\mathrm{ReLU}}{\mathrm{ReLU}}
\newcommand{\mathcomment}[1]{\textcolor{mygray}{\texttt{#1}}}

We now analyse budgeting effects: if reading times are affected by the entropy through a budgeting mechanism, we may expect to see budgeting spillover effects when a reader under-budgets---i.e., when the entropy is smaller than a word's surprisal, causing less time to be allocated to the word than required for processing.
Here, we operationalise \defn{under-budgeting} as any positive difference between surprisal and entropy.
Similarly, we may expect \defn{over-budgeting} to lead to negative spillover-effects, since spending extra time in a word might allow the reader to start going through some of their processing debt (i.e. the still unprocessed spillover effects of that and of previous words).
We operationalise several potential budgeting effects as:\looseness=-1%
\begin{align}
    & h(w_{t\!-\!1}) - \mathrm{H}(W_{t\!-\!1})
    & \mathcomment{budget} \nonumber \\
    & \mathrm{ReLU}(h(w_{t\!-\!1}) - \mathrm{H}(W_{t\!-\!1}))
    & \mathcomment{under-budget} \nonumber \\
    & \mathrm{ReLU}(\mathrm{H}(W_{t\!-\!1}) - h(w_{t\!-\!1}))
    & \mathcomment{over-budget} \nonumber \\
    & |h(w_{t\!-\!1}) - \mathrm{H}(W_{t\!-\!1})|
    & \mathcomment{abs-budget}
\end{align}
where $\mathrm{ReLU}(\cdot)$ zeroes negative values, while returning positive ones unchanged.
We then compute the $\Delta_{\mathrm{llh}}$ of adding these effects as predictors of RT on top of a baseline with the current word's entropy, as well as all four surprisal terms, as predictor variables: $\bx_{\mathrm{base}}=[\renyientfunc\!\left(W_{t}\right); \surpfunc(\words_{:})]$.
Unlike previous experiments, thus, our baseline here already contains the entropy as a predictor.
Further, we show results for eye-tracking datasets both including (\cmark) and excluding (\xmark) skipped words for this and future analyses.\looseness=-1

\cref{tab:delta_llh_budget} presents these results.
In short, we do find budgeting effects of word $t\!-\!1$ on RTs in our two analysed self-paced reading datasets,
and in Dundee (\cmark).
We do not, however, find them on Dundee (\xmark).
This may imply budgeting effects impact word-skipping, but not actual RTs once the word is fixed.
Further, we also find weak budgeting effects of word $t\!-\!2$ in our (\xmark) eye-tracking datasets; these, however, are only (weakly) significant in Dundee.
We conclude that these results do not provide concrete evidence of a budgeting mechanism influencing RTs, but only of it influencing word-skipping instead.
We will further analyse these effects in our discussion section (\cref{sec:discussion}).

\begin{table}[t]
    \centering
\resizebox{\columnwidth}{!}{%
    \begin{tabular}{lccc@{\hskip 0in}cc}
\toprule
        %
         &
         & \multicolumn{2}{c}{Entropy$^1$}
         & \multicolumn{2}{c}{Successor Entropy$^2$}
         \\
         \cmidrule(lr){3-4}
         \cmidrule(lr){5-6}
        &
        & $\emptyset$ $^3$ & $[\renyientfunc(W_{t\!+\!1})]^4$
        & $\emptyset$ $^3$ & $[\renyientfunc\!\left(W_{t}\right)]^5$
         \\
\midrule
\multicolumn{5}{l}{Shannon Entropy ($\alpha=1.0$)} \\
\cmidrule(lr){1-1}

\multicolumn{2}{l}{Brown} & \phantom{-}\textcolor{mygreen}{0.15}$^{*}$\phantom{$^{**}$}& \phantom{-}\textcolor{mygreen}{0.14}$^{*}$\phantom{$^{**}$}& \phantom{-}{0.01}\phantom{$^{***}$}& {-0.01}\phantom{$^{***}$} \\
\multicolumn{2}{l}{Natural Stories} & \phantom{-}\textcolor{mygreen}{0.89}$^{***}$& \phantom{-}\textcolor{mygreen}{0.44}$^{*}$\phantom{$^{**}$}& \phantom{-}\textcolor{mygreen}{2.27}$^{***}$& \phantom{-}\textcolor{mygreen}{1.83}$^{***}$ \\
Provo & (\cmark) & {-0.06}\phantom{$^{***}$}& {-0.05}\phantom{$^{***}$}& {-0.06}\phantom{$^{***}$}& \textcolor{myred}{-0.06}$^{*}$\phantom{$^{**}$} \\
Dundee & (\cmark) & \phantom{-}\textcolor{mygreen}{0.25}$^{***}$& \phantom{-}\textcolor{mygreen}{0.26}$^{***}$& {-0.00}\phantom{$^{***}$}& {-0.00}\phantom{$^{***}$} \\
Provo & (\xmark) & {-0.01}\phantom{$^{***}$}& \phantom{-}{0.01}\phantom{$^{***}$}& \textcolor{myred}{-0.08}$^{*}$\phantom{$^{**}$}& {-0.06}\phantom{$^{***}$} \\
Dundee & (\xmark) & \phantom{-}\textcolor{mygreen}{0.17}$^{**}$\phantom{$^{*}$}& \phantom{-}\textcolor{mygreen}{0.16}$^{**}$\phantom{$^{*}$}& \phantom{-}{0.02}\phantom{$^{***}$}& \phantom{-}{0.00}\phantom{$^{***}$} \\

\midrule
\multicolumn{5}{l}{Renyi Entropy ($\alpha=0.5$)} \\
\cmidrule(lr){1-1}

\multicolumn{2}{l}{Brown} & \phantom{-}\textcolor{mygreen}{0.26}$^{*}$\phantom{$^{**}$}& \phantom{-}\textcolor{mygreen}{0.27}$^{*}$\phantom{$^{**}$}& {-0.01}\phantom{$^{***}$}& {-0.00}\phantom{$^{***}$} \\
\multicolumn{2}{l}{Natural Stories} & \phantom{-}\textcolor{mygreen}{2.04}$^{***}$& \phantom{-}\textcolor{mygreen}{1.52}$^{***}$& \phantom{-}\textcolor{mygreen}{1.95}$^{***}$& \phantom{-}\textcolor{mygreen}{1.44}$^{***}$ \\
Provo &(\cmark) & {-0.04}\phantom{$^{***}$}& {-0.01}\phantom{$^{***}$}& {-0.03}\phantom{$^{***}$}& {-0.00}\phantom{$^{***}$} \\
Dundee &(\cmark) & \phantom{-}\textcolor{mygreen}{0.64}$^{***}$& \phantom{-}\textcolor{mygreen}{0.64}$^{***}$& \phantom{-}{0.00}\phantom{$^{***}$}& {-0.00}\phantom{$^{***}$} \\
Provo &(\xmark) & \phantom{-}{0.32}\phantom{$^{***}$}& \phantom{-}{0.38}\phantom{$^{***}$}& \phantom{-}{0.06}\phantom{$^{***}$}& \phantom{-}{0.12}\phantom{$^{***}$} \\
Dundee &(\xmark) & \phantom{-}\textcolor{mygreen}{0.36}$^{***}$& \phantom{-}\textcolor{mygreen}{0.34}$^{***}$& \phantom{-}{0.03}\phantom{$^{***}$}& \phantom{-}{0.01}\phantom{$^{***}$} \\

\bottomrule
\\[-8pt]
\multicolumn{6}{l}{
$^1$
$\bx_{\mathrm{model}}=[\renyientfunc\!\left(W_{t}\right); \bx_{\mathrm{base}}]$,
\,\,\,\,
$^2$
$\bx_{\mathrm{model}}=[\renyientfunc(W_{t\!+\!1}); \bx_{\mathrm{base}}]$,
} \\ \multicolumn{6}{l}{
$^3$
$\bx_{\mathrm{base}}=[\surpfunc(\words_{:})]$,
\,\,\,\,
$^4$
$\bx_{\mathrm{base}}=[\surpfunc(\words_{:}); \renyientfunc(W_{t\!+\!1})]$,
} \\ \multicolumn{6}{l}{
$^5$
$\bx_{\mathrm{base}}=[\surpfunc(\words_{:}); \renyientfunc\!\left(W_{t}\right)]$,
}
    \end{tabular}
}
\vspace{-6pt}
    \caption{$\Delta_{\mathrm{llh}}$ (in $10^{-2}$ nats) achieved after adding the top predictor on top of a baseline with the predictors in the column.
    All models include surprisal as a predictor.
    %
    %
    }
    \label{tab:delta_llh_successor}
    \vspace{-8pt}
\end{table}

\begin{figure*}[t]
    \centering
    \begin{subfigure}[b]{.5\textwidth}
        \includegraphics[width=\textwidth]{figs/params--entropy.pdf}
    \end{subfigure}%
    ~
    \begin{subfigure}[b]{.5\textwidth}
        \includegraphics[width=\textwidth]{figs/params--renyi_0.50.pdf}
    \end{subfigure}
    \vspace{-22pt}
    \caption{Size of the learned effects $\boldsymbol{\phi}$ for both surprisal and contextual entropy terms when using our best performing models in each dataset: (left) Shannon entropy; (right) R\'enyi\xspace{} entropy with $\alpha=0.50$.
    }
    \label{fig:effect_sizes}
    \vspace{-7pt}
    %
\end{figure*}

\subsection{Experiment \#6. Preemptive Processing}

In our analysis of preemptive processing, we will analyse the impact of the successor entropy (i.e., $\renyientfunc(W_{t\!+\!1})$) on RTs.
Notably, while prior work has analysed this impact, the results in the literature are contradictory.
\cref{tab:delta_llh_successor} presents the results of our analysis.
In short, this table shows that the successor entropy is only significant in Natural Stories.\footnote{We note this is the same dataset previously analysed by \citet{van-schijndel-linzen-2019-entropy}, who found a significant effect of the successor entropy.\looseness=-1}
In contrast, the current word's contextual entropy is a significant predictor of RTs in $\sfrac{3}{4}$ analysed datasets, even when added to a model that already has the successor entropy.
Further, while most of our results suggest readers rely on skewed expectations for their anticipatory predictions---i.e., the R\'enyi\xspace{} entropy with $\alpha=0.5$ is in general a stronger predictor than Shannon's entropy---the successor Shannon entropy seems more predictive of RTs than the R\'enyi\xspace{}.
Our full model, though, still has a larger log-likelihood when using R\'enyi\xspace{} entropies.
Overall, our results support the analysis of \citet{smith2008optimal}, who suggested preemptive processing costs should be constant w.r.t. the successor entropy.\looseness=-1

\section{Discussion} \label{sec:discussion}

We wrap-up our paper with an overall discussion of our results.
To make this discussion more concrete, we plot the values of the parameters $\boldsymbol{\phi}$ from our best regressor per dataset---showing the effect of predictor variables not-included in a dataset as zero.
These parameters are depicted in \cref{fig:effect_sizes}.
As the contextual R\'enyi\xspace{} entropy models provide overall higher log-likelihoods, we focus on them here.\looseness=-1

First, \cref{fig:effect_sizes} shows that---for Brown, Natural Stories and Dundee---not only does the entropy have similar (or stronger) psychometric predictive power than the surprisal, it also has a similar (or even stronger) \emph{effect size} on RTs.
In other words, an increase of 1 bit in the contextual entropy leads to a similar or larger increase in RTs than a 1 bit increase in surprisals.\looseness=-1

Second, \cref{fig:effect_sizes} also shows that in Natural Stories---the only dataset where it is significant---the successor entropy has a larger effect on RTs than the surprisal, and its impact is positive.
If these parameters implied a causal effect (which we note they do not), this would mean an increase in the next word's entropy leads to an increase in the current word's RT.
This could suggest that readers are preemptively processing future words, and that they need more time to do this when there are more plausible future alternatives.
Moreover, we see the successor R\'enyi\xspace{} entropy has a similar (or slightly smaller) effect on RTs than the current word's R\'enyi\xspace{} entropy.
Why the successor entropy is only significant in the Natural Stories dataset is left as an open question.\looseness=-1

Third, \cref{fig:effect_sizes} shows the effect of over-budgeting on RTs in Brown, Natural Stories, and Dundee.\footnote{While over-budgeting is not a significant predictor in Brown, it leads to slightly stronger models and we add it to this dataset's regressor for an improved comparison.\looseness=-1}
We see that our operationalisation of over-budgeting leads to a negative effect on RTs in Dundee (\cmark), but to no effect in Dundee (\xmark).
Together, these results suggest that when a reader over-budgets time for a word, they are more likely to skip the following one.
In Brown and Natural Stories, however, over-budgeting seems to lead to a positive effect on the next word's RT.
As this is only the case in self-paced reading datasets, we suspect this could be related to specific properties of this experimental design;
e.g., a reader's attention could break when they become idle due to over-budgeting reading time for a specific word.\looseness=-1

Fourth, while we get roughly consistent results in Brown, Natural Stories, and Dundee, our analyses on Provo show a starkly different story.
Further, while we note that Provo is the smallest of our analysed datasets (in terms of its number of annotated word tokens; see \Cref{tab:data} in \cref{app:data}), this is likely not the whole story behind these different results.
As it is non-trivial to diagnose the source of these differences, we leave this task open for future work.

As a final remark, we note that, throughout our analyses, the R\'enyi\xspace{} entropy with $\alpha=0.5$ leads to stronger regressors (i.e., to higher log-likelihoods of held-out test data) than using $\alpha=1$, i.e., Shannon entropy.
Recall that different values of $\alpha$ lead to R\'enyi\xspace{} entropies with different interpretations (see \cref{sec:predictive_rt}); the R\'enyi\xspace{} entropies with $\alpha<1$ can be roughly associated with a soft-measure of the number of competing alternatives considered at a time step, which suggests a mechanism by which contextual entropy may affect RTs.\looseness=-1

\section{Conclusion}

This work investigates the anticipatory nature of the reading process.
We examine the relationship between expected information content---as quantified by contextual entropy---and reading times in four naturalistic datasets, specifically looking at the additional predictive power over surprisal that this quantity provides.
While our results confirm the responsive nature of reading, they also highlight its anticipatory nature.
We observe that contextual entropy has significant predictive power in models of reading time, suggesting that readers may: skip words based on their expectations, and preemptively process future words.
Such results give evidence of a significant anticipatory component to reading behaviour.

\section*{Limitations and Caveats}

Throughout this paper, we have discussed the effect of anticipation on reading times (and on the reading process, more generally)---where we quantify a reader's anticipation as a contextual entropy.
We do not, however, have access to the true distribution $p$, which is necessary to compute this entropy.
Rather, we rely on a language model $p_{\btheta}$ to approximate it (as we note in \cref{sec:estimating_info}).
How this approximation impacts our results is a non-trivial question---especially since we do not know which errors our approximator is likely to commit.
If we assume $p_{\btheta}$ to be a noisy version of $p$, for instance, we could have good estimates of $p(\cdot \mid \words_{< t})$ on average, while not-as-good estimates of each word $w_t$'s probability.
If this were true, our estimates of the entropy could be better than our estimates of the surprisal---which would bias our analyses towards preferring the entropy as a predictor.

We believe this not to be the main reason behind our results for two reasons.
First, if the entropy helped predict reading times simply because we have noisy versions of the surprisal in our estimates, the same should be true for predicting spillover effects. We don't see this happening in general, though; a word's entropy mainly helps in predicting its own reading times, and not future word's RTs.
Second---even if our estimates are noisy---a noisy estimate of the surprisal should better approximate the true surprisal than an estimate of the contextual entropy.
Since replacing the surprisal with the contextual entropy eventually leads to better predictions of reading times, this is likely not the only mechanism on which our results rely.\looseness=-1

Another limitation of our work is that we always estimate the contextual entropy and surprisal of a word $w_t$ while considering its entire context $\words_{< t}$.
Modelling surprisal and entropy effects while considering skipped words, however, would be an important future step for an analysis of anticipation in reading.
As an example, \citet{van-schijndel-schuler-2016-addressing} show that when a word $w_{t\!-\!1}$ is skipped, the subsequent word $w_t$'s reading time is not only proportional to its own surprisal (i.e., $\surpfunc(\word_t)$), but to the sum of both these words surprisals (i.e., to $\surpfunc(\word_t)+\surpfunc(\word_{t\!-\!1})$).
They justify this by arguing that a reader would need to incorporate both word's information at once when reading.
Another model of the reading process, however, could predict that readers simply marginalise the distribution over $w_{t\!-\!1}$, and compute the surprisal of word $w_t$ directly as:\looseness=-1%
\begin{equation}
    -  \log \sum_{w_{t\!-\!1} \in \mathcal{W}} p(w_t \mid \words_{< t})\, p(w_{t-1} \mid \mathbf{w}_{<t\!-\!1})
\end{equation}
We leave it to future work to disentangle the effects that using a model $p_{\btheta}$---as well as the effects of skipped words---has on our results.

redpajama_set_name: RedPajamaArXiv


March 20 marks the United Nations' International Day of Happiness. This day recognizes the importance of happiness and wellbeing as common goals and aspirations in the lives of all individuals around the world. Three of the UN's 17 Sustainability Development Goals (SDGs), related to ending poverty, reducing inequalities and protecting our planet, aim to support happiness and wellbeing for all.
From designing out loneliness in big cities, to embedding social value in projects that advance inclusion, diversity and equity, to rethinking workplaces for employee well-being and developing neighbourhoods that facilitate social engagement and improved health, the work of WSP's experts seeks to create environments that ultimately contribute to happiness.
We asked Helena Klintström, Responsible for Sustainable Urban Development in the Advisory group at WSP's Stockholm office, and Mark Bessoudo, Research Manager and Sustainability Consultant in Property and Buildings, at our Toronto office, to discuss how valuable research-based insights from the social sciences are used in tandem with applied engineering, technologies and design solutions in the built environment to increase health, happiness and well-being.
"There's been a shift from focusing solely on the environmental aspects of buildings such as energy and water efficiencies, to acknowledging that more can be done (and is being done) in terms of influencing health and wellness through the integration of design elements, equipment and policies when creating new buildings or retrofitting existing buildings in our communities," says Mark Bessoudo.
For example, today companies are not only looking at the immediate health and wellness impacts, but also at the bottom line. Healthier and more productive employees translate to more profits for companies while workplaces that focus on health and wellness tend to attract the best and brightest candidates.
Third-party industry certifications or rating systems such as WELL and Fitwel further support health and wellness. WELL is a rating system that assesses how healthy a building is and focuses purely on the building design and operations that influence individuals' health. Some of WSP's offices are WELL-certified or are Fitwel champions and have committed to help clients pursue Fitwel and WELL certifications to create productive work environments and increasing ROIs throughout the lifetime of buildings.
A case in point is with respect to WELL. We are sustainability advisors for the 81 Bay Street project, a grand-scale development located in the heart of Toronto that connects people and nature. Forming part of CIBC Square, it will comprise two high-rise towers including 81 Bay Street, and a seven-storey podium with a bus terminal integrated into its lower floor. The development's many amenities dedicated to promoting a healthy, active lifestyle include secure cycle storage, showers and changing facilities. And CIBC Square's overriding distinguishing feature is the new public realm it will create for the people of Toronto in the form of a one-acre elevated Sky Park. This fresh, green space will link the centre's two towers and offer pedestrian access to Union Station, a key transport hub. Importantly, it will contribute to improving the city's air quality and be a place for people to connect with nature.
Our teams in Sweden have developed a tool for the Municipality of Umeå, located in the north of Sweden, to help it include social sustainability in its urban planning process. To ensure that our recommendations were based on facts, the tool incorporates the latest research and parameters that allow the municipality to evaluate social sustainability in their urban developments. Tested on three different neighbourhoods within the municipality, recommendations were given on how they could be further developed. This new method to analyze a city's ability to create added value from a social sustainability point of view is referred as a social value creation analysis.
Since 2013, the United Nations has celebrated the International Day of Happiness as a way to recognise the importance of happiness in the lives of people around the world. In 2015, the UN launched the 17 Sustainable Development Goals that seek to end poverty, reduce inequality, and protect our planet – three key aspects that lead to well-being and happiness.


text:
Gadget Hacks iPadOS
Gadget Hacks Shop 5G News & Guides Compare Phones Everything Android Everything iOS iPhone 11 Galaxy S20 Galaxy Note 10 Google Pixel OnePlus iOS 13 Tips Android 11 Tips Tweaks & Hacks Privacy & Security Productivity Hacks Movies & TV Smartphone Gaming Music & Audio Travel Tips Android 11 Features Videography Tips Chat Apps
Review: My 6 Months Using the 12.9-Inch iPad Pro
By Isaac Sahag
The 12.9-inch iPad Pro was released in stores on November 11, 2015, and a couple days later I went to an Apple Store and to check it out in person. I had mixed feelings about the tablet, but was intrigued overall—so much so that over the next few weeks, as I found myself hearing more and more about the device, a thought began to take hold in the back of my mind—I really want one. At the same time, I was hesitant about dropping roughly 10 Benjamins for a tablet!
In a twist of fate which I can only describe as magic, I was able to acquire a basically brand new 12.9-inch iPad Pro early this year—for a tiny fraction of the original price. Needless to say, I was pretty ecstatic!
I'd heard about the Apple Pencil, but I wasn't sure how good a stylus it could be, since most styluses are not great. And to be honest, I thought Apple was just hyping up the Pencil. But after messing around with one at the Apple Store, I was pleasantly surprised to see that this device was truly something special, so I ended up getting it.
Since then, I have been using the tablet and stylus on almost a daily basis. Below are some of my thoughts and experiences from the last 6 months using both the hardware and software of the 12.9-inch iPad Pro and Apple Pencil. Naturally, the more time you spend with such a device, the more details you begin to notice, both good and bad. So without further ado, let's jump in.
Size & Usability
Let's start with the most obvious aspect—the size. Naturally, when I first heard about the 12.9-inch iPad Pro, I knew it was going to be huge. (Literally.) And seeing it in person for the first time, you see that it's truly a massive device. However, an interesting thing happens once you begin to use the device for some time; you start to realize that even though the size is big, it's quite manageable. The device is definitely portable, and a funny thing happened to me after using the 12.9-inch iPad Pro for extended periods; I began to feel that the device is actually not that big and even might be small in some cases.
The 12.9" iPad Pro right next to the 13" MacBook Air.
It's actually quite comfortable to hold and use the 12.9-inch iPad Pro with two hands for an extended period of time; you can definitely grasp it in one hand and use the other hand solely for touch, but you do start to notice the weight of the device pressing against your palm. Also, the weight of the 12.9-inch iPad Pro is spread out evenly which streamlines the device for handheld use. In fact, for everyday activities, I find myself grabbing the 12.9-inch iPad Pro over my 13" MacBook Air because it's easier to watch videos with while relaxing on the couch.
Display & Speakers
The 12.9-inch display of this iPad Pro model is impressive. It has a resolution of 2732 x 2048 pixels, which makes it sharper than 1080p, but not as sharp as 4K. It works out to about 264 pixels per inch, which makes the display very crisp. You can't help but get "lost in the display" especially if you're using it at night; the content being displayed on the screen is very vivid and the display can get pretty bright as well. I do wish it had the "True Tone Display" that the smaller iPad Pro has, though. Another minor gripe is that some content is not optimized for the display's resolution, but I'm sure that's an issue that will be resolved over time.
The 12.9-inch iPad Pro's speakers are also quite superior. You get 4 speakers, a pair on the top and bottom, which deliver surround sound. The speakers also get really loud and produce a broad spectrum of sounds without any distortions. Now you don't have to struggle anymore when listening to something on a tablet. In fact, you can easily play a song on the 12.9-inch iPad Pro in one room and hear it clearly in the next room.
The 12.9" iPad Pro delivers an excellent multimedia experience!
This combination of the display and the speakers truly make the 12.9-inch" iPad Pro an excellent multimedia device. So the next time you're Netflixing, forget the TV, just cuddle up with your 12.9-inch iPad Pro.
Performance & Battery Life
This thing is fast. iOS devices generally zoom by, but the 12.9-inch iPad Pro is a complete beast when it comes to overall speed. The A9X chip, which clocks in at 2.25 GHz, coupled with 4 GB RAM, gives this device desktop-grade performance. Needless to say, whatever you throw at the 12.9-inch iPad Pro, it cuts through it like a hot knife going through butter.
Whether it's multiple tabs in Safari, split-screen apps, multiple apps running in the background, or even more resource-intensive apps like GarageBand, the 12.9-inch iPad Pro can not be stopped. I'm also confident that this device will be relatively future-proof and will easily handle the next couple generations of iOS just fine.
These are some pretty impressive benchmark scores.
Apple advertises 10 hour battery life for this device, and in my experience this number seems to be about right, though it might vary depending on how you're using the device. Charging the device does take a couple hours, mainly due to the 12-watt power adapter. But the 12.9-inch iPad Pro also supports Apple's 29 W USB-C power adapter which dramatically speeds up charging times, and it's really a shame that Apple chose not to include this option by default.
Productivity & Ecosystem
If you want to use the 12.9-inch iPad Pro for basic everyday tasks, it actually does a decent job, especially when coupled with a keyboard. So for tasks like managing your email, conducting research online, creating documents, and so forth, it can actually replace your laptop. The iOS App Store has tons of great productivity apps, and if you are okay with primarily using iOS, then the 12.9-inch iPad Pro has got you covered.
However, to get a bit more serious, this is where things get weird. The 12.9-inch iPad Pro literally has desktop-grade performance powers, but there isn't much in the App Store that can really utilize it. This device essentially becomes the equivalent of a Ferrari or Porsche stuck in a school zone—it has the potential to do so much more than it is allowed to. Naturally, this can quickly become frustrating.
For example, the 12.9-inch iPad Pro's hardware can easily edit multiple 4K video streams. But on the software side, your main video editor option is iMovie for iOS, which is severely underpowered when it comes to video-editing features.
Now, iOS definitely has a fair share of creative and useful apps. In fact, the huge screen of the 12.9-inch iPad Pro allows certain apps to fully unlock their potential. Let's take a look at GarageBand for iOS: On the iPhone SE, the 4-inch screen is quite small, and it really limits your workflows. Contrast this to the 12.9-inch iPad Pro, which makes the GarageBand experience an absolute delight.
GarageBand truly shines on the 12.9" iPad Pro.
The 12.9-inch iPad Pro has upgraded and expanded the paradigm for what iOS devices are capable of. It's now up to Apple to upgrade the software components to a Pro level by expanding the capabilities of iOS and also giving developers the resources they need to create "premium level" applications.
The Apple Pencil
The Apple Pencil definitely lives up to the hype, and it truly is a digital equivalent of a graphite pencil... and then some. Now I've used styluses in the past, and they generally offered a very static experience; a lot of times their only utility was a pointing device that didn't smudge the screen as much as your finger does. However, the Pencil is a dynamic device: it responds to pressure, tilt, offers high levels of precision, and has essentially zero lag, which all translate to a superior experience.
To fully unleash the Pencil's power, you need apps, and there are a ton of great Pencil-supported apps out there. Whether you want to do some basic drawing, edit some photos, take some notes, markup documents, or do some more serious graphic design work, the Apple Pencil does it all. Besides being seriously fun to use, the Pencil combined with the proper apps, can be immensely helpful for a broad spectrum of people including students, office workers, and graphic designers.
The Apple Pencil can be used for serious work or just goofing around.
While the Apple Pencil opens new doors for iPad interaction, it's still a first generation device with certain flaws. I wish it had fully embraced the traditional pencil concept and added an eraser to the bottom, since it gets annoying constantly switching back and forth from eraser to drawing mode via the software. You also have to keep track of where you place the Pencil, since currently there's no way to attach it to the iPad Pro, unlike some other tablets, and keeping track of it can become tedious.
Miscellaneous Thoughts
There are a few other thoughts I had about my experiences with the 12.9-inch iPad Pro:
No 4K camera. It can easily handle editing multiple 4K video streams, yet you can't shoot in 4K. It's strange that Apple chose to leave this device "out in the cold" when it comes to capturing 4K video content, especially once you consider the price of it.
Base storage of 32 GB is simply not enough for a Pro device. It would make a lot more sense for the 12.9-inch iPad Pro to start off with 64 GB storage, but then again, Apple does the same thing with the iPhone—only giving it 16 GB storage (though, this is rumored to change).
Excellent build quality. The 12.9-inch iPad Pro's design feels very premium, and even after holding and using it for a couple minutes, you can tell that Apple put some serious work in designing and building this device.
I didn't bother with the Smart Keyboard. After seeing and using the Smart Keyboard at the Apple Store, I felt rather unimpressed with it. I'm still on the lookout for a good backlit keyboard for the 12.9-inch iPad Pro. A good keyboard will really take this tablet to the next level—the level in which it could replace your laptop, even.
So Should I Get This Thing?
It's complicated! On one hand, I really like this device. Although the device has its fair share of flaws, it's still an absolute pleasure to use. On the other hand, there's the price, which is not cheap to say the least. I got lucky and didn't pay the full price for mine, but for most people the cost of a new 12.9-inch iPad Pro could likely be put to better use.
My conclusion? Wait for the next generation. The 12.9-inch iPad Pro has been out for almost a year, and there'll likely be a new iteration soon. Unless you need a super-fast tablet with a really nice stylus input for your work, I would recommend waiting for the next generation, and hopefully get more bang for your buck.
Cover photo and screenshots via Isaac Sahag/Gadget Hacks
News: iOS 10 Will Be Released on September 13, but the Gold Master Is Available Now
How To: Download iOS 11 Beta on Your iPhone Right Now
Apple Announces New iPad Pro: 9.7-Inch Screen, Better Audio, & More
News: Apple May Bring TrueDepth Camera to More iPhone Models & iPad Pro This Year, Analyst Says
How To: New iPhone 6S, Apple TV, iPad Pro, & More—Everything You Need to Know
How To: Download & Install iOS 11 on Your iPhone
News: LiDAR vs. 3D ToF Sensors — How Apple Is Making AR Better for Smartphones
News: Apple's LiDAR Scanner on New iPad Pro Provides Possible Peek at How Its Smartglasses Will See the World
News: Apple's iOS 12.1.3 Public Beta 4 Available for iPhone
How To: Get the New iPad Pro Wallpapers on Any iPhone
How To: Get All the New iPad Pro Wallpapers on Your iPhone
News: Apple's iOS 12.1.3 Developer Beta 4 Available for iPhones
News: Apple's iOS 12.1.3 Public Beta 2 for iPhone Released to Software Testers
How To: Downgrade iOS 12 Back to 11.4.1 on Your iPhone
How To: Downgrade iOS 13 Back to iOS 12.4.1 on Your iPhone Using iTunes or Finder
News: Here's What You Need to Run ARKit 3 on Your iPhone or iPad
News: Specs Leak for the Samsung Galaxy Note 4
News: Apple Just Released iOS 12.1.2 Public Beta 1 for iPhone
News: Apple Is Releasing 5 New iPhones This Year. What the Hell Are They Thinking?
News: Apple Releases iOS 13.4 Public Beta 3 for iPhone, Hints at Future 'OS Recovery' Mode
News: Apple Releases iOS 13.4 Developer Beta 3 for iPhone, Introduces First Evidence of Internet-Based 'OS Recovery' Mode
News: Apple iOS-Powered Sesame Street Yourself App Turns Kids into Their Favorite Characters via ARKit 3's New Features
How To: Printable iPhone 11, 11 Pro & 11 Pro Max Cutouts — See Which Size Is Right for You
News: New Evidence Implies Apple Pencil May Work on the New iPhone 11 Models
How To: Download the iOS 10 GM on Your iPad, iPhone, or iPod Touch Right Now
How To: Use an iPad as a Midi controller for Logic & Pro Tools
How To: iOS 16 Just Dropped Support for These iPhone Models — Find Out if You Can Update or Not
News: Apple Surpasses 13 Million ARKit Downloads Within Six Months, Report Says
How To: Use Split View Mode on Any iPad Model
News: iStopMotion for iPad Launching at Introductory Price of $4.99
News: iPhone 4 Details
News: Bricked iPad Pros, Error 56, & How to Fix
Snapseed: Can a $5 App Replace Lightroom and Photoshop?
News: CloudOn Is Back, Plus 4 More iPad Apps for Working with Microsoft Office Docs
News: W.E.L.D.E.R. Blends Bejeweled, Boggle and Scrabble into Word-Building Addiction
Watch Out iMovie: Avid Studio Is Now Available for iPad
News: Catalog of Astronomy Apps for Mobile Devices
News: The 5 Best Word Game Mobile Apps Besides Scrabble
News: Animal Balloon Font
How To: Get By with the Cheaper 16GB Option for iPhone or iPad
Janice jones 6 years ago - edited 6 years ago
I've had a new ipad for about 6 months now and I love it. I didn't think I'd need it b/c I would just normally read on my apple laptop. Before that I thought that I just needed that and my celll phone and I was good, but what a convenient little gadget to have. I read different reviews on goodreads, and amazon about how easier reading can be on such devices and I must say that its a bit more convenient. Who knew....
Isaac Sahag 6 years ago
Janice, they definitely can be convenient!
How To: The Little-Known iPad Feature You'll Never Stop Using Once You Know About It
How To: Download and Install iPadOS 16.4 on Your iPad Right Now
How To: Your iPad Has a Hidden Built-in Calculator You're Not Using — Here's How to Unlock It
How To: 13 Tips Every Apple Pencil User Needs to Know for iPad
How To: 11 Best Calculator Apps for iPad to Solve All Your Math Needs
How To: Connect a Bluetooth Mouse and Keyboard to your iPad (or iPhone)!
How To: DIY your own iPad stylus
How To: Play Prison Architect on Your iPad Now & Be the Warden of Your Very Own Jail
News: Apple Adds Pro Features to iPad & Revived iPad Mini, Upgrades Display for Apple Watch Series 7
iPadOS How-Tos
iPadOS News
iPadOS Forum

redpajama_set_name: RedPajamaCommonCrawl


text:
Classic Colonial From 1672 on Long Island Is the Week's Oldest Home for Sale
January 26, 2021 by Stellar ICRE in historic homes, Industry News, lists, oldest, Unique Homes
This week's list of the 10 oldest homes to land on the market is a collection of gorgeous properties across the East coast.
The eldest of the crew is a 4-acre plot along the northern shore of Long Island. Located in Stauket, NY, this prime piece of property is highlighted by a house dating to 1672. Beyond its historic significance, it also happens to be the last undeveloped parcel in the area—which makes it a lovely proposition for developers.
Other standouts this week include the youngest of the bunch—built in 1760. The Italianate in Sag Harbor, NY, has received a ravishing Hamptons-chic makeover. We also fell head over heels for a mint-green and cream-colored Connecticut home that's taken on a new identity as a full-on beach babe.
Have a look at these 10 grand homes in all their 21st-century glory. They're a reminder that age really is just a number.
1. 18 Lake St, Setauket, NY
Oldfield South: This shingled five-bedroom house just a couple of blocks from Long Island Sound has been upgraded and expanded over the past decade.
In addition, there's a guesthouse with separate utilities, a three-car garage, and a swimming pool. However, the real lure of this property is its 4 acres. Close to beaches, shops, restaurants, and Stonybrook University, the parcel could be subdivided for development. Just don't touch the classic charm from the 17th century!
Setauket, NY
2. 70 Anthony Rd, Lebanon, NJ
Upgraded Colonial: Well-maintained and fully updated, this lovely three-bedroom home has a wonderful mix of modern and vintage touches.
Newer features include ipe wood floors, beamed ceilings, and updated bathrooms. The full-acre lot includes a saltwater pool, hot tub, three-car garage, and guest cottage.
3. 205 West St, Southington, CT
Turnkey Colonial: Despite it's age, this five-bedroom home is move-in ready. It's recently undergone a top-to-bottom renovation, including a new roof, paint, and bathrooms. Cozy touches from its past include wood beams and fireplaces, while its recent makeover makes it a hassle-free homee.
4. 9 Brown Rd, Wappingers Falls, NY
Horse farm: This 31-acre property was reportedly used as a link to the Underground Railroad.
The home's kitchen was recently updated and includes a large center island. Outside, there's a pool and a whopping eight additional buildings, including a Dutch cow barn and silo, English horse-riding stable, interior riding ring with viewing stand, carriage house with three rental units, and new barn for equipment.
5. 501 Province Line Rd, Allentown, NJ
Allentown historic: We don't know much about this four-bedroom home other than what's visible in the listing photos. What we spotted are lovely hardwood floors, woodwork, moldings, and fireplaces in an old-timey house that could use a little love. It sits on just under an acre.
6. 1276 Bear Creek Rd, Elizabethtown, PA
Thomas Penn farmhouse: This historic 4.5-acre property in Lancaster County includes a restored farmhouse linked to Thomas Penn.
Both modern and rustic, this four-bedroom house is highlighted by a walk-in fireplace with an 18-century mantle, exposed beams, and an updated kitchen. The property boasts a small stream, stone bank barn, and tennis court.
7. 40 N. Court St, East Side of Providence, RI
Benjamin Cushing house: Built by a successful trader, this is the oldest home in the College Hill Historic District.
Notable woodwork, mantels, wide-plank pine flooring, sidewinder staircases, and exposed beam ceilings are just a few of the hallmark touches that make this home a historic gem. According to the listing, the three-bedroom home's passed through the hands of multiple architects and has been lovingly maintained and restored.
East Side of Providence, RI
8. 19 Hilltop Rd, Norwalk, CT
Wilson Point: This stylish crème de menthe delight offers 3,168 square feet of updated living space. Expanded by an award-winning architect in 1992, the home also has a new pool and a screened porch.
9. 2860 Jefferson Pike, Jefferson, MD
10-acre farm: It's been a tollhouse and a tavern, and now it serves as a functional 10-acre farm with a lovely five-bedroom house, pasture, vegetable gardens, and a pool. The house features four fireplaces, an exposed stone wall, wood stove, two staircases, and a walk-up attic.
10. 6 Union St, Sag Harbor Village, NY
Capt. John Hulbert house: Yesterday and today clash in this exquisitely updated Italianate built for a Revolutionary War hero.
The regal residence has been given a Hamptons-chic makeover where exposed beams sit side by side with Venetian plasterwork and a double-sided fireplace. The five-bedroom home also boasts a wraparound mahogany porch, copper bar on the third floor, roof deck, and an elevator.
Sag Harbor Village, NY
The post Classic Colonial From 1672 on Long Island Is the Week's Oldest Home for Sale appeared first on Real Estate News & Insights | realtor.com®.
January 27, 2023 Beyoncé, a $100k-a-night suite and a red carpet of influencers: Dubai basks in its booming economy
January 25, 2023 Biden administration rolls out a blueprint for a 'renters bill of rights' – Here's what it includes


But it seems millennials — those aged between roughly 18 to 35 — aren't as interested in buying diamonds as in previous generations.
The De Beers diamond cartel made the first concerted effort to link diamonds with engagement rings in the 1930s.
And it was wildly successful — in the 1940s just 10 percent of engagement rings contained diamonds. This had rised to 80 percent by the turn of the century.
But now that tradition is changing. Couples in the UK are spending almost a fifth less (19 percent) on engagement rings than they did five to 10 years ago, according to research from insurer Protect Your Bubble.


I recently purchased a 2018 34QS. And although we love travelling in it, come cooking time the wife is a little unhappy with the low power on the MW. I understand that they went with a lower power so the inverter can handle it, but we would like to change it out.
So two questions. Has anyone changed the Microwave out and to what? And for the life of me, I cant understand why they vent the stove through the microwave and vent inside the RV. Why didn't they vent it outside? It serves no purpose to vent inside. Has anyone converted this to vent outside?


YAKMA offer traditional Hapkido martial arts classes for men women and children aged 7 and above. We also offer non contact martial arts based fitness classes and one on one training sessions.
YAKMA has been established since 1998. All our classes are run in a fun and friendly atmosphere.
We aim to promote increased self confidence, fitness, discipline, weight control, toning and flexibility in our classes.
Would you like more information on hours, classes and programs available at Yorkshire Academy of Korean Martial Arts? Please contact the school with your questions.
Update the details, Write a Review or Report Error of Yorkshire Academy of Korean Martial Arts.


text:
Articles from Johns Hopkins
Home > News and Publications > For the Media > News Release Archive
Novel Mouse Model Could Further Understanding of Leukemias - 01/02/2014
Novel Mouse Model Could Further Understanding of Leukemias
Johns Hopkins Kimmel Cancer Center investigators have genetically engineered a new mouse that mimics a common form of leukemia in humans. Studying the model could lead to new understanding of the disease, they say.
The mice were bred to express a protein mutation within a gene known as FMS-like tyrosine kinase 3 (FLT3). The gene that forms FLT3's blueprint controls the development of healthy blood cells; mutations to it are seen in approximately a third of adult and pediatric cases of acute myeloid leukemia (AML).
Mutations in FLT3 are found in various regions of the gene, and the Johns Hopkins scientists designed the mouse to better understand the workings of the two most common ones in AML. They are ITD mutations, which cause a very aggressive form of the disease, and D835Y mutations, which cause a less aggressive form of the disease. The new mouse specifically expresses D835Y mutations. Mice bred to express ITD mutations were previously developed by the Johns Hopkins team.
To date, scientists have not understood why the ITD mutations cause more aggressive disease, says Donald Small, M.D., Ph.D., the Kyle Haydock Professor of Oncology and director of pediatric oncology at Johns Hopkins. Small led a team of researchers who originally cloned the FLT3 gene and linked it to leukemia a decade ago.
Patients with ITD mutations have only a 15 percent chance of a cure, Small says, compared with patients who have D835Y mutations, who have about a 50 percent chance of a cure. Studies comparing the new mice, containing D835Y mutations, with mice expressing ITD mutations provide "an ideal platform for dissecting the molecular mechanisms underlying the differences between the two mutations, and for drug screening against the two," he says.
In work published online Nov. 18 in the journal Proceedings of the National Academy of Sciences, Small and his colleagues reported that FLT3 mutation types (either D835Y or ITD) are directly related to the aggressiveness of disease. In laboratory studies, the investigators found that mice with D835Y mutations, like people, develop much less aggressive disease than those with ITD mutations, both in disease severity and survival. D835Y mice lived an average of 1.5 times longer than those with ITD mutations -- an average of 678 days versus 430 days. Those with the D835Y mutations also maintained a higher white blood-cell count and were not nearly as depleted in blood stem cells as those with the ITD mutations. The D835Y mutant mice also developed a broader spectrum of blood cancers, including lymphomas.
Investigators also tested the anticancer drugs Lestaurtinib (CEP-701) and sorafenib in the two types of mice. After 48 hours of treatment, the FLT3/D835Y mice responded to Lestaurtinib only, while the FLT3/ITD mice responded to both drugs.
"This new mouse model should enable us to further explore why patients with FLT3/ITD mutations have worse prognosis," Small says. He and his team, he says, will be looking at the genes turned on and off by these mutations to further study molecular changes involved in leukemia development, and investigate pathways involved in other leukemias.
Coauthors of the study were Emily Bailey, Li Li, Amy S. Duffield, Hayley S. Ma, and David L. Huso. The work was supported by a National Institutes of Health (CA 090668 and CA 070970), the Leukemia & Lymphoma Society, and the Giant Food Pediatric Cancer Research Fund.
Vanessa Wasta
410-614-2916; wasta@jhmi.edu
Amy Mone
410-614-2915, amone@jhmi.edu
Find Physicians Specializing In...
The Sidney Kimmel Comprehensive Cancer Center
Contact JHM Media Team
About the History of Johns Hopkins Medicine
Videos from Johns Hopkins Medicine
Read about Diseases & Conditions
Connect with Johns Hopkins Medicine on Social Media
Contact a Media Representative


Stevens & Bolton LLP is delighted to have been shortlisted as a finalist for 'National/Regional Law Firm of the Year' at this year's Legal Business Awards 2015.
The Legal Business Awards are organised by Legal Business Magazine and are considered as one of the most prestigious trade awards in the legal sector.
The winning entry will be announced on the night of Tuesday 24 March at Grosvenor House Hotel, Park Lane, London.