text:
\section{Introduction}
It is now broadly accepted that neural language models can consistently generate fluent text
\citep{radford2019language,shoeybi2019megatron,brown2020language,smith2022using}.
Yet, while large language models make few local word-level errors, human studies have shown that they still often make ``high-level'' errors such as incoherence, self-contradictions, and off-topic generations \citep{dou-etal-2022-gpt}. We hypothesize that researchers have focused on local fluency partly because it is easy to automatically evaluate through metrics such as perplexity and n-gram matching. Automatic assessment of high-level text generation quality has received less attention, partially because a single general-purpose metric does not exist.

This work takes a step toward the automatic evaluation of
the high-level structure of the generated text
by applying a tool from statistics, \emph{model criticism in latent space} \citep{dey1998simulation,seth2019model}. Under this approach, we first project data to a latent space based on an assumptive generative process, and then compare the implied latent distributions between real data and language model samples.
This approach unifies past work for evaluating text generation under a single framework,
including existing dimensionality reduction techniques such as probabilistic PCA \citep{wold1987principal}, as well as previous applications of model criticism that were restricted to topic models \citep{mimno-blei-2011-bayesian}.

By making different assumptions in the underlying generative process, model criticism in latent space identifies specific failure modes of the generated language.
We demonstrate this on three representative high-level properties of the generated discourse---coherence \citep{barzilay-lapata-2005-modeling}, coreference \citep{chomsky1993lectures}, and topicality \citep{blei2006correlated}---as well as on a synthetic dataset
for which the true data generating process is known.

Experiments using our proposed framework enable us to make four observations about modern language models.
First, we find that it is possible for a model to get strong word-level perplexity, yet fail to capture longer-term dynamics.
Second, we find that the
transformer language models perform poorly in terms of coherence, in line with previous observations \citep{dou-etal-2022-gpt,sun-etal-2021-long,krishna2022rankgen,sun-etal-2022-chapterbreak}, particularly when they do not have access to explicit lexical markers in the context.
Third, we show that transformer language models do not model coreference structures well. Last, we show that transformer language models can capture topical correlations \citep{blei2006correlated}.
All results, data, and code are publicly available at \url{https://github.com/da03/criticize_text_generation}.

\begin{figure*}[!htp]
\centering

\includegraphics[width=\textwidth,trim={0 0.2cm 0 0.4cm},clip]{figures/modelcrit.pdf}
\vspace{-0.6cm}
\caption{Illustration of applying model criticism in latent space to evaluate discourse coherence. Instead of word-level errors, we identify improper high-level section transitions (those that are rare in real data), as marked by red crosses. The article shown is generated by GPT-2 finetuned on \textsc{Wiki}. See \Cref{sec:dc} for more explanations.}
\label{fig:illustration_section}
\vspace{-0.25cm}
\end{figure*}

\section{Model Criticism in Latent Space\label{sec:mcls}}
Model criticism \citep{o2003hsss} quantifies the relationship between a data distribution $P_{\text{data}}(x)$ and a model $P_{\text{model}}(x)$ by comparing statistics over these two distributions. While model criticism can be applied to the observation space, in many applications we are interested in ``higher-level'' aspects of the data, such as the underlying topics of a document \citep{mimno-blei-2011-bayesian}, or the latent factors of an image \citep{seth2019model}. Model criticism in latent space \citep{dey1998simulation,seth2019model} lifts the criticism approach to a latent space in order to compute
higher-level comparative statistics.

How do we critique latent properties of arbitrary, and perhaps unknown, distributions? For example, given a language model, how do we know how well it captures the section transitions at the discourse level (\Cref{fig:illustration_section})?
Lacking access to the generative process, we introduce a critic generative
process $P_c$ with latent variables $z \in \mathcal{Z}$ and observations $x \in \mathcal{X}$:
\begin{align*}
    z \sim P_c (z) &&
    x \sim P_c(x|z).
\end{align*}
Based on this generative process, the posterior distribution $P_c(z|x)$ projects $x$ to the latent space. For a single data point $x$, we can evaluate the negative log-likelihood
of the projected latent variables $z\simP_c(z|x)$ under the prior $P_c$,
\begin{align*}
T_c(x) &\triangleq -\mathbb{E}_{z\sim P_c(z|x)} \log P_c(z) \\
&= H(P_c(z|x), P_c(z)),
\end{align*}
where $H(p, q)\triangleq -\mathbb{E}_p \log q$ denotes the cross-entropy between two distributions $p$ and $q$.\footnote{We discuss the difference between being likely in the latent space versus the observed space
in \Cref{sec:appendix_decomposition}.} This process is illustrated in~\Cref{fig:crit}.

Given an arbitrary distribution over $x$, $P_x$, we can take an expected negative log-likelihood,
$$
T_c(P_x) \triangleq -\mathbb{E}_{x\sim P_x(x)} \mathbb{E}_{z\sim P_c(z|x)} \log P_c(z).
$$

We term $T_c(P_x)$ the \emph{Latent NLL}.\footnote{When $z$ is the same as $x$ ($P_c(z|x)=\mathbbm{1}{[z=x]}$), Latent NLL is the same as the negative log-likelihood of the language model samples under the data distribution \citep{zhao2018adversarially}.} This value is the cross-entropy between the aggregated posterior distribution and the prior distribution of $z$:
$$
T_c(P_x) = H(\mathbb{E}_{x\sim P_x(x)} P_c(z|x), P_c(z)).
$$
In practice, we cannot compute $T(P_x)$ analytically due to the existence of the two expectations $\mathbb{E}_{x\sim P_x(x)}$ and $\mathbb{E}_{z\sim P_c(z|x)}$, but can approximate expectations using Monte-Carlo sampling.

When $z$ is a sequence of $M$ discrete states, we define a metric \emph{Latent PPL} analogous to perplexity:
$$
\text{Latent PPL} (P_x) \triangleq \exp [T_c(P_x) / M].
$$

With a critic chosen, we can compare $P_{\text{data}}(x)$ and $P_{\text{model}}(x)$ in the latent space by estimating and comparing $T_c(P_{\text{data}})$ and $T_c(P_{\text{model}})$. Similar to a two-sample test \citep{hotelling1951generalized}, when $P_{\text{data}}$ and $P_{\text{model}}$ are the same, the statistics will also stay close. Furthermore, with a powerful critic, $T_c(P_{\text{model}})$ is meaningful by itself: a higher value means that model generations are less likely in the latent space, whereas a lower value implies that samples match the critic along the latent projection.\footnote{Under a bad critic, the value of $T_c(x)$ might not be meaningful even though we can still use it to compare distributions.}
The approach can also be applied to individual points, $T_c(x)$, to identify outliers.

\begin{figure}
    \centering
    \includegraphics[width=\linewidth,trim={0 1.1cm 0 1.1cm},clip]{figures/Crit.png}
    \vspace{-0.4cm}
    \caption{Model criticism in latent space. Given a sample $\tilde{x}$, we first map it to latent states $\tilde{z}$ using $P_c(z|\tilde{x})$. The likelihood of $\tilde{z}$ is evaluated using $P_c(\tilde{z})$ to measure how likely the samples are in the latent space.}
    \label{fig:crit}

\end{figure}

\paragraph{How to select the critic $\mathbf{P_c}$} Choosing the critic $P_c$ is obvious only when we know the true latent variables and the generative process of data. In other cases, it depends on the data properties of interest. For example, if we want to criticize the topicality of text, we can use a topic model \citep{blei2003latent} to induce a latent space over topics.
Note that the selected critic $P_c$ may underperform $P_{\text{model}}$ as a model of $x$, while still providing useful latent structures.
By criticizing strong models, using simpler latent models that are designed to capture a particular aspect of text as $P_c$, we provide a sanity check for the stronger model along a specific target axis. This property motivates the use of this approach with powerful, yet opaque models.

\section{\label{sec:latent}A Surprising Text Generation Failure}

As a preliminary experiment, we show a language model with strong word-level perplexity that fails to capture simple long-term dynamics as demonstrated by model criticism.
We assume that $P_{\text{data}}$ is known and follows a basic pattern.
It has a latent high-level sequence of $M=50$ discrete states $z_1, z_2, \ldots, z_M$ where each state can take one of 256 possible values. These states are generated from a transition distribution
$$
P_{\text{data}}(z_1, \ldots, z_M) = \prod_{m=1}^MP_{\text{data}}(z_m | z_{m-1}).\footnote{We assume a special beginning state $z_0$.}
$$
At the observation level, each latent state $z_m$ generates a sub-sequence of words $x_1^m, x_2^m, \ldots, x_{N_m}^m$  conditioned on $z_m$ from an emission distribution $P(x_1^m, \ldots, x_{N_m}^m | z_m)$. We also restrict the model so that each sub-sequence can only come from one latent state.
The observed sequence is the concatenation of all sub-sequences.
The joint distribution of the latent states and the tokens forms:
\begin{align*}
    &P_{\text{data}}(x, z) = P_{\text{data}}(z) P_{\text{data}}(x|z)\\
    &=\prod_{m=1}^M\left [ P_{\text{data}}(z_m | z_{m-1}) P_{\text{data}}(x_1^m, \ldots, x_{N_m}^m | z_m) \right ].
\end{align*}

\noindent
With this generative process, we sample a dataset.\footnote{Sub-sequences vary between 4 to 11 words and the vocabulary size is set to 53. There are 51.2k samples for training, 6.4k for validation, and 6.4k for evaluation.}

We apply a transformer language model as $P_{\text{model}}$ and train it on this dataset. Given the simplicity of the generative process and the small vocabulary size, we expect this model to do quite well. And in fact we do see that the model achieves a strong perplexity of 2.28, which nearly matches the true data $P_{\text{data}}$ perplexity of 1.99.

\begin{figure}[!t]

    Sample $x$: ... \mybox[fill=green!20]{p B W m <s>} \mybox[fill=blue!20]{T c g N f <s>} \\ \hspace*{1cm} \mybox[fill=red!20]{x i t K a b <s>} \mybox[fill=yellow!20]{b A x t N o m U <s>} ... \\
    Infer Latent $z$: ... \mybox[fill=green!20]{G}\ \mybox[fill=blue!20]{B}\ \mybox[fill=red!20]{R}\  \mybox[fill=yellow!20]{Y} ...\\

    Criticize (Latent NLL): ... \mybox[fill=green!20]{5.1} + \mybox[fill=blue!20]{9.9} + \mybox[fill=red!20]{10.3} + \mybox[fill=yellow!20]{4.2} ...
    \caption{Applying model criticism to synthetic data.}
    \label{fig:toy_example}

\end{figure}

\begin{table}[!t]
    \centering
    \begin{tabular}{@{}lcc@{}}
    \toprule
    & Trans-LM & HSMM-LM\\
    \midrule

    Word-level PPL & 2.28 & 2.05 \\
    \midrule

    Latent PPL (data) &  \multicolumn{2}{c}{44.30}  \\
    Latent PPL (model) & 64.80 & 47.24 \\

    \bottomrule
    \end{tabular}
    \caption{Evaluation results of transformer and HSMM on the synthetic dataset. Word-level PPL values are estimated on the test set, and Latent PPL values are estimated using the same number of samples (6.4k)
    }
    \label{tab:toy}
    \vspace{-0.2cm}
\end{table}

Model criticism gives a different method for quantifying model fit. Since the true data generating process is known, we can directly use $P_c = P_{\text{data}}$ as the critic to induce the latent space. To project an observation $x$ to the latent space, we need to perform posterior inference $P_c(z|x)$. By construction, this mapping is deterministic, since each sub-sequence comes from a unique latent state (see \Cref{sec:appendix_why_eval_latent} for details). We then apply model criticism $T_c$ by sampling a sequence of transformer outputs,
mapping them to a sequence of latent states, counting to compute the aggregated posterior, and then comparing to the known prior. This process is shown in \Cref{fig:toy_example}.

\Cref{tab:toy} presents the results. Surprisingly, transformer gets a much worse Latent PPL compared to a hidden semi-Markov model (HSMM, the true model class) fit to data (66.80 v.s. 47.24), which has a near-optimal Latent PPL. This result implies that even though the transformer is nearly as good at predicting the next word in the sequence, it has not learned the higher-level transition structures. Seemingly, it can produce reasonable estimates of the next token which does not reflect the ability to capture longer-range dynamics of this system.

\noindent \textbf{Motivation} Given this result, we ask whether similar issues are present in language models applied in more realistic scenarios. We therefore turn to experiments that consider model criticism for long-form generation, and ask whether language models capture properties of discourse coherence (\Cref{sec:dc}), coreference (\Cref{sec:coref}), and topicality (\Cref{sec:topic}).

\begin{comment}
\begin{table*}[!htp]
    \centering
    \begin{tabular}{@{}l@{}rrrcccc@{}}
    \toprule
        Dataset & \#Train &  \#Val & \#Test & \#Sect Types & Med \#Sect & Med Sect Len & Max Sect Len  \\
        \midrule
        \textsc{PubMed} &  32.35k & 1.80k & 1.84k & 27 & 4 & 518 & 1986 \\
        \textsc{ArXiv} & 4.91k & 0.17k & 0.15k & 50 & 4 & 787.5 & 1965 \\
        \textsc{Wiki} & 111.40k & 13.97k & 13.98k & 96 & 4 & 122 & 1999 \\

        \textsc{Wiki-Short} &69.30k & 8.56k & 8.68k & 96 & 4 & 101 & 500\\
        \bottomrule
    \end{tabular}
    \caption{\label{tab:data}Data Statistics. Section length is measured using the GPT-2 tokenizer \citep{radford2019language,wolf-etal-2020-transformers}. Section statistics are based on the validation set. More details on data processing can be found in \Cref{sec:appendix_section}.}
\end{table*}
\end{comment}

\begin{table*}[!htp]
    \centering
    \begin{tabular}{@{}llcccccccc@{}}
    \toprule
    Metric & Model & \multicolumn{2}{c}{\textsc{PubMed}} & & \multicolumn{2}{c}{\textsc{ArXiv}} & & \multicolumn{2}{c}{\textsc{Wiki}} \\
    \cmidrule{3-4}\cmidrule{6-7}\cmidrule{9-10}
   &  & W/ Title & W/O Title & & W/ Title & W/O Title & & W/ Title & W/O Title\\
    \midrule

    \multirow{ 2}{*}{PPL}  & LM$_1$ & 11.38 & 11.50 & & 13.94 & 14.13 & & 15.38 & 15.84  \\

     & LM$_2$ & 10.96 & 11.09 & & 12.73 & 12.85 & & 16.35 & 16.86 \\

    \midrule

     \multirow{ 3}{*}{Latent PPL\ } & Data &  \multicolumn{ 2}{c}{2.58}   & & \multicolumn{ 2}{c}{3.87}  & & \multicolumn{ 2}{c}{4.80}\\

     & LM$_1$ & 2.68 & 3.76 &  & 6.72 & 9.52 & & 4.67  & 5.47\\

     & LM$_2$ &  4.17 & 7.92 & & 9.01 & 18.64 & & 6.48  & 10.22 \\

    \bottomrule
    \end{tabular}
    \caption{Results of coherence experiments. W/ Title is the setting where section titles are included in the training data for LMs, and W/O Title removes section titles from the training data.
    \label{tab:main}
    \vspace{-0.2cm}
\end{table*}

\section{Critiquing Discourse Coherence\label{sec:dc}}

Text generation from large language models rarely leads to local fluency errors, but there is evidence of failures like those in the previous section~\citep{dou-etal-2022-gpt,sun-etal-2021-long,krishna2022rankgen,sun-etal-2022-chapterbreak}. In this section, we apply model criticism to assess discourse coherence~\citep{barzilay-lapata-2005-modeling} of large LMs. We study this through an experiment on generating long-form documents divided into explicit sections. While we do not know the true data generating process, knowing the distribution of section types allows us to assess the latent structure of LM generations.

\Cref{fig:illustration_section} illustrates the experiment. Here, an LM generates an article. Each word transition is fluent, but the system makes two section transition errors: first, it generates two sections of type ``background''; second, it generates a section of type ``personal life'' following the last ``background'' section, with both transitions being unlikely in the data.\footnote{``background'' is usually followed by ``reception''.} We aim to separate the evaluation of these high-level coherence errors from word-level errors.

To apply model criticism, we posit a simple critic generative process to capture the section changes. We adapt a hidden semi-Markov model (HSMM) which is commonly used to represent segmentations of this form.
Specifically, the high-level latent variables $z_1,\ldots, z_M$\footnote{We prepend a special beginning state $z_0$ and append a special ending state $z_{M+1}$ that do not emit anything.} model transitions among section types and the bottom level generates text conditioned on the current section type:
\begin{align*}
    &P_c(x, z) = P_c(z) P_c(x|z)\\
    &=\prod_{m=1}^M\left [ P_c(z_m | z_{m-1}) P_c(x_1^m, \ldots, x_{N_m}^m | z_m) \right ].
\end{align*}

\begin{figure*}[!htp]
    \centering
    \includegraphics[trim={1cm 1cm 1cm 1cm},clip,width=0.99\linewidth]{figures/diff20.gv.pdf}
\caption{Section transition errors on \textsc{Wiki}, where each edge is labeled with the difference between $P(z_m|z_{m-1})$ of LM$_1$ (W/O Title) and of data, and its width is proportional to the absolute difference. Red marks unlikely transitions ($P_c(z_m|z_{m-1})<0.05$). For clarity, we only show the top 20 section titles and remove singletons.\label{fig:dot}}
\vspace{-0.2cm}
\end{figure*}

We can then evaluate on datasets with known (ground truth) section titles and use these section titles as $z$. We use three English datasets \textsc{PubMed}, \textsc{ArXiv}, and \textsc{Wiki}~\citep{cohan-etal-2018-discourse}.\footnote{We adapt \textsc{PubMed} and \textsc{ArXiv} by filtering out section titles with low frequency. We download and process Wikipedia to get a dataset of the same format as \citet{cohan-etal-2018-discourse}.}
We compare two language modeling settings, one trained with all section titles removed (``W/O Title'') and one with section titles before each section (``W/ Title''), since we hypothesize that the existence of explicit section type markers might help the model learn the dynamics, inspired by \citet{nye2021show} and \citet{wei2022chain}. Sections are separated by a special marker, and a special end-of-sequence symbol is used to mark the end of the generation.
Since all three datasets are relatively small (especially considering that we use them to generate entire articles), we leverage pretrained language models GPT-2 small (LM$_1$) \citep{radford2019language}, and GPT-Neo small (LM$_2$) \citep{gpt-neo} which is trained on a more diverse dataset \citep{gao2020pile}. We finetune these LMs for $P_{\text{model}}$.

To generate, we sample from the language model until we hit the end-of-sequence symbol. No tempering/truncation \citep{holtzman2019curious} is used during sampling, since we are more interested in the learned distribution rather than its mode here.
For the ``W/ Title'' setting, we discard the generated section titles in a postprocessing step.

To infer the section types for a generated article, we need to approximate posterior inference to compute $T_c$.
We make a simplifying assumption that the posterior section title of each section only depends on its corresponding text: $P_c(z |x) \approx \prod_{m=1}^M P_c(z_m | x_\cdot^m)$. We then finetune BERT with a classification head to estimate $P_c(z_m|x_\cdot^m)$. At inference time we use the MAP estimate of $z$ instead of maintaining the full distribution $P(z|x)$ (BERT is mostly over 90\% certain about its predictions). More details can be found in \Cref{sec:appendix_section}.

\begin{table}[!t]
    \centering
    \begin{tabular}{@{}llcc@{}}\toprule
        Metric &  Model & W/ Title & W/O Title \\
         \midrule
    \multirow{ 3}{*}{Latent PPL} & Data  & \multicolumn{ 2}{c}{4.78} \\
    & LM$_1$ & 5.18 & 6.24\\
    & LM$_2$ & 6.54 & 10.72 \\
    \bottomrule
    \end{tabular}
    \caption{Latent PPLs on \textsc{Wiki-Short} where all section transitions fit within the context window size.}
    \label{tab:main_short}
    \vspace{-0.35cm}
\end{table}

\paragraph{Results} \Cref{tab:main} gives results on coherence experiments. We first note that both models have strong word-level perplexity across datasets, with LM$_2$ doing better on two of the three datasets. We also note that removing titles has a negligible impact on the perplexity of the models. However, Latent PPL tells a different story. We find that LM$_1$ greatly outperforms LM$_2$ when criticizing with respect to the latent sections.\footnote{For one dataset, LM$_1$ has a lower Latent PPL than the data distribution. This result is possible as a consequence of our cross-entropy formulation of $T_c$, under which a mode-seeking distribution can get a lower value than $P_{\text{data}}$, which is the approximate entropy of the latent prior. }
It is also interesting that transformer LMs are sensitive to title words being explicitly included in the training data (i.e., the W/ Title setting). For example, LM$_1$~W/ Title gets a Latent PPL of 6.72 on \textsc{Arxiv}, whereas LM$_1$~W/O Title gets a Latent PPL of 9.52, despite having very close word-level PPLs (13.94 v.s. 14.13).
 These observations indicate that lacking explicit markers, the tested transformer LMs do not learn the long-term dynamics necessary for discourse coherence. Using explicit section topic markers might serve a similar functionality as using chain-of-thought prompting in language-model-based question answering tasks \citep{wei2022chain}.

One concern is that the difference between W/ Title and W/O Title is a side effect of language models having a limited context window size (1024 for LM$_1$ and 2048 for LM$_2$), since two adjacent sections might not fit within the context window size (but one section and the next section title are more likely to fit). To check if this is the case, we filter \textsc{Wiki} to only include articles with maximum section length 500 to form a new dataset \textsc{Wiki-Short}. In this dataset, any two adjacent sections can fit within the context window of both LM$_1$ and LM$_2$. \Cref{tab:main_short} shows that even in this case W/ Title still outperforms W/O Title, indicating that the difference between W/ Title and W/O Title is not due to the limited context window size.

\begin{table}[!t]
    \centering
    \small
    \begin{tabular}{@{}l@{}ccccc@{}}
    \toprule
        Metric & LM$_2$~W/O & LM$_2$~W/ & W/O & W/ & Data \\
        \midrule
        PPL$\downarrow$ & 11.09 & 10.96 & 11.50 & 11.38 & - \\
        MAUVE$\uparrow$ & 0.75 & 0.85 & 0.91 & 0.90 & 0.96 \\
        Latent PPL$\downarrow$ & 7.92 & 4.17 & 3.76 & 2.68 & 2.58 \\
        Human$\uparrow$ & 0.50     & 0.66     & 0.71 & 0.88 & 0.87 \\
         \bottomrule
    \end{tabular}
    \caption{Experiments on the correlation of Latent PPL (coherence) with human judgment and automatic metrics (\textsc{PubMed}). Latent PPLs agree well with human judgments of coherence. W/O: LM$_1$~W/O Title. W/: LM$_1$~W/ Title. See \Cref{sec:appendix_human} for evaluation details.}
    \label{tab:human_eval}
    \vspace{-0.35cm}
\end{table}

\Cref{fig:dot} visualizes the section transition errors made by LM$_1$ (W/O Title) for the most common section types on \textsc{Wiki}. We can find that the language model tends to generate the same section topic repeatedly, although there are other transition errors as well. More detailed error analysis can be found in \Cref{sec:appendix_section}.

\begin{comment}
\Cref{tab:unlikely} presents the most common section transition errors across all section types for different settings (W/ Title and W/O Title). %
We notice again that a very common transition error is to generate a section repeatedly. However, even in the ground truth data, there are repeated sections, such as ``career $\to$ career'' (appearing 0.08\%), which is due to the misclassification by the BERT classifier used for the inference network\footnote{In the training data there also exist some rare repetitions, such as \url{https://web.archive.org/web/20220307192058/https://en.wikipedia.org/wiki/Leanna_Brown}.}.
\end{comment}

\Cref{tab:human_eval} correlates automatic metrics with human judgments of coherence. Each human annotator first labels the section title of each section (after a training phase where they labeled and received feedback on real data), and then labels whether the organization of the section titles makes sense \citep{persing-etal-2010-modeling}. The baseline MAUVE \citep{pillutla2021mauve} is a metric that compares the distribution of GPT-3 hidden states between real data and model generations. From this table, we can observe that both MAUVE and Latent PPL align much better with humans than PPLs.
Comparing MAUVE and Latent PPL, we can see that Latent PPL aligns better with humans: LM$_1$~W/O Title is considered to be better than LM$_1$~W/ Title under MAUVE, but both human evaluation and Latent PPL consider LM$_1$~W/ Title to be much better.

\begin{table}[!t]
    \centering
    \begin{tabular}{@{}llcc@{}}\toprule
        Metric &  Model & W/ Title & W/O Title \\
        \midrule
    \multirow{4}{*}{PPL}& GPT-2 S & 15.38 & 15.84\\
    & GPT-2 M & 12.98 & 13.32 \\
    & GPT-2 L & 12.19 & 12.52 \\
    & GPT-2 XL & 11.60 & 11.99 \\
         \midrule
    \multirow{5}{*}{Latent PPL} & Data  & \multicolumn{ 2}{c}{4.80} \\
    & GPT-2 S & 4.67 & 5.47 \\
    & GPT-2 M & 4.79 & 5.58 \\
    & GPT-2 L & 4.90 & 5.75 \\
    & GPT-2 XL & 4.75 & 5.56 \\
    \bottomrule
    \end{tabular}
    \caption{The results of scaling model size on \textsc{Wiki}. Increasing model size improves PPL but not Latent PPL.}
    \label{tab:section_scaling}
    \vspace{-0.2cm}
\end{table}

A natural question is whether increasing model size improves coherence. To this end, in addition to GPT-2 small (GPT-2 S, aka LM$_1$, 117M parameters), we apply model criticism to GPT-2 medium (GPT-2 M, 345M parameters), GPT-2 large (GPT-2 L, 742M parameters), and full GPT-2 (GPT-2 XL, 1.5B parameters) on \textsc{Wiki}. The results are summarized in~\Cref{tab:section_scaling}. We can see that increasing model size improves PPL but not Latent PPL.

\section{Critiquing Coreference Chains\label{sec:coref}}

\begin{figure}[!t]
    \begin{framed}
    \textbf{Original Text}

    ... \mybox[fill=green!20]{[Lisa]$_0$} runs off to find \mybox[fill=blue!20]{[him]$_1$} and \mybox[fill=yellow!20]{[they]$_2$} kiss passionately. Afterwards, \mybox[fill=blue!20]{[Josh]$_1$} tells \mybox[fill=green!20]{[her]$_0$} the reason why \mybox[fill=blue!20]{[he]$_1$}'s going to \mybox[fill=yellow!20]{[their]$_2$} first gig, and that \myboxtwo[fill=green!20]{[Lisa]$_0$} is going to do it, too...\\

    \textbf{Coreference Chains} $\mathbf{z}$

    . \mybox[fill=green!20]{[Female]$_0$} \mybox[fill=blue!20]{[him]$_1$} \mybox[fill=yellow!20]{[they]$_2$} . \mybox[fill=blue!20]{[Male]$_1$} \mybox[fill=green!20]{[her]$_0$} \mybox[fill=blue!20]{[he]$_1$} \mybox[fill=yellow!20]{[their]$_2$} \myboxtwo[fill=green!20]{[Female]$_0$}\\

    \textbf{5-gram critic} $\mathbf{P_c}$
    \begin{align*}
        &P_c(\ \myboxtwo[fill=green!20]{\text{[Female]$_0$}}\ \ | \  \text{previous entity mentions}) \\
        &\approx P_c(\ \myboxtwo[fill=green!20]{\text{[Female]$_0$}}\ \  |\ \  \mybox[fill=blue!20]{\text{[Male]$_1$}} \mybox[fill=green!20]{\text{[her]$_0$}} \mybox[fill=blue!20]{\text{[he]$_1$}} \mybox[fill=yellow!20]{\text{[their]$_2$}})
    \end{align*}
    \end{framed}
    \caption{Critiquing coreference chains on a sample from LM$_1$. We first extract entity mentions from the text and only keep the genders of proper nouns to form $\mathbf{z}$, then a 5-gram $P_c$ is used to score $\mathbf{z}$. []$_i$ denotes a mention with entity id $i$. \textbf{.} marks sentence boundaries.
    }
    \label{fig:coref}
    \vspace{-0.3cm}

\end{figure}

Coreference tracks how multiple \textit{mention} expressions are used to refer to the same underlying \textit{entity} \citep{karttunen-1969-discourse-referents,gordon1998representation}. While coreference represents a ubiquitous and important discourse-level phenomenon \citep{jurafskyspeech, kunz-hardmeier-2019-entity}, there is evidence that large neural language models make elementary coreference mistakes \citep{pagnoni-etal-2021-understanding}, such as referring to non-existent discourse entities \citep{schuster-linzen-2022-sentence}.

In this experiment, we compare the coreference chain \citep{jurafskyspeech} distributions between real data and LM generations. A coreference chain consists of a sequence of coreferent mentions. To simplify the representation, we use gender features to replace non-pronominal tokens, as illustrated in \Cref{fig:coref}.\footnote{We discuss the ethical considerations of using gender as features in \Cref{sec:ethical}.} Presumably these latent chains should be similar in generated text and in real data

For the critic $P_c$, we use a 5-gram language model with Kneser–Ney smoothing \citep{ney1994structuring} over chains. To infer $\mathbf{z}$, we use an off-the-shelf coreference resolution tool.\footnote{\url{https://github.com/huggingface/neuralcoref}} To avoid data sparsity issues, we relabel entity clusters within each n-gram. We apply model criticism to compare real data and LMs trained on \textsc{Wiki} (W/ Title), after filtering data to only consider articles about films since they contain richer reference structures.

\begin{table}[t]
    \centering

    \begin{tabular}{@{}lllllcc@{}}
    \toprule
       $z_{1}$ & $z_2$ & $z_3$ & $z_4$ & $z_5$ & \hspace{-0.3cm}$\hat{P}_{\text{data}}$  & $\hat{P}_{\text{LM}}$ \\
        \midrule
         &  &  & . & M$_0$ & 0.10 & 0.12 \\\relax
        . & M$_0$ & . & M$_0$ & M$_0$ & 0.01 & 0.03 \\\relax
        . & . & M$_0$ & his$_0$ & he$_0$ & 0.04 & 0.05 \\\relax
        . & . & . & N$_0$ & His$_1$ & 0.00 & 0.00$^\dagger$\hspace{-0.13cm} \\\relax
        M$_0$ & M$_1$ & . & M$_0$ & M$_0$ & 0.00 & 0.01 \\\relax
        N$_0$ & N$_1$ & M$_2$ & M$_3$ & We$_4$ & 0.00 & 0.00$^\dagger$\hspace{-0.13cm}  \\\relax
        . & F$_0$ & her$_0$ & . & F$_0$ & 0.03 & 0.04 \\\relax
        M$_0$ & his$_0$ & . & M$_1$ & M$_1$ & 0.00$^\dagger$\hspace{-0.13cm} & 0.01 \\\relax
        . & . & N$_0$ & N$_1$ & h.self$_2$ & 0.00 & 0.00$^\dagger$\hspace{-0.13cm} \\\relax
        . & . & M$_0$ & his$_0$ & M$_0$ & 0.01 & 0.01 \\
         \bottomrule
    \end{tabular}
    \caption{Coreference chain n-grams ranked by contribution to the difference in Latent NLL (LM$_1$). $\hat{P}$ denotes the empirical frequency of an n-gram in percentages.

    M (Male), F (Female), and N (None of the above). Blank: padding.
    $^\dagger$: small positive numbers truncated to 0.} \label{tab:result_coref_truecontribution}

\end{table}

\begin{table}[!t]
    \centering
    \begin{tabular}{@{}ccc@{}}
    \toprule

       Data & LM$_1$ & LM$_2$ \\
        \midrule

       6.26 & 7.22 &  6.93 \\
        \bottomrule
    \end{tabular}
    \caption{Latent PPLs of coreference chains.}
    \label{tab:latent_coref}

\end{table}

\paragraph{Results} \Cref{tab:latent_coref} shows the Latent PPLs on real data and LM generations. We can see that in general there is a mismatch of coreference distributions. Interestingly, while LM$_1$ models outperformed LM$_2$ models on discourse coherence, for this task LM$_2$ models are better.

\Cref{tab:result_coref_truecontribution} shows the 10 coreference chain n-grams that most contributed to this difference. Some are intuitively implausible: in the fourth row, [His]$_1$ does not have a local antecedent; in the second to the last row, [himself]$_2$ also does not have a local antecedent. Others are rare but possible: in the last row, a proper noun [Male]$_0$ is used after a pronoun [his]$_0$ is used in the same sentence to refer to the same entity.\footnote{Different from prescriptive linguistic theories on coreference \citep{chomsky1993lectures,buring2005binding}, the differences identified by model criticism only reflect differences in empirical distributions and do not necessarily mean coreference errors.}

The learned critic $P_c$ can also be used to identify unlikely coreference chains, as shown in \Cref{tab:result_critic} in \Cref{sec:appendix_coref}. \Cref{sec:appendix_coref} also has more qualitative examples and analyses.

Lastly, we evaluate whether scaling model size improves coreference modeling. The results are summarized in~\Cref{tab:coref_scaling}. We can see that increasing model size does not improve Latent PPL, similar to our observations on critiquing discourse coherence.

\begin{table}[!t]
    \centering
    \begin{tabular}{lc}\toprule
      Model & Latent PPL \\
        \midrule
   Data  & {6.26} \\
     GPT-2 S & 7.22 \\
     GPT-2 M & 7.64  \\
     GPT-2 L & 7.27 \\
     GPT-2 XL & 7.62 \\
    \bottomrule
    \end{tabular}
    \caption{Critiquing coreference chains on larger models. Increasing model size does not improve Latent PPL.}
    \label{tab:coref_scaling}
    \vspace{-0.2cm}
\end{table}

\begin{figure*}[t]
  \centering
  \includegraphics[width=\textwidth,trim={0cm 0cm 0cm 0cm},clip]{figures/cov_wiki_baselinepubmed.png}
  \caption{\label{fig:cov} Topic covariance matrix for the induced $z$ (on \textsc{Wiki}). Left: Test set ($P_{\text{data}}$). Middle:  LM$_1$ generations ($P_{\text{model}}$). Right: generations of LM$_1$ trained on \textsc{PubMed} as a visual baseline. The Latent NLLs are: 124.70, 123.30, and 140.10. Topic ids are rearranged using hierarchical clustering to facilitate visual comparison.}
  \vspace{-0.2cm}
\end{figure*}

\section{Critiquing Topic Correlations\label{sec:topic}}

Topical structure is another important aspect of long-form document generation~\cite{serrano2009modeling}.
Certain topics are more likely to appear together, for example, a document containing a topic related to ``poets'' is more likely to also contain one related to ``publisher'' relative to one related to ``football''. A text generation model should capture these topical relations.
 For this experiment, we again sample documents from the trained language model $P_{\text{model}}$. Specifically, we utilize the transformer-based LMs trained on the datasets in \Cref{sec:dc} (W/O Title).

To explore the topical structure in the generated documents, we need a critic $P_c$. While LDA \citep{blei2003latent} is the most commonly used generative process for topic modeling, the Dirichlet prior does not explicitly model topic correlations in documents. We therefore use the correlated topic model (CTM) specifically designed to model topical structures~\citep{blei2006correlated}. Model criticism will then compare the latent space of the real data with the generated texts.

For each document, a CTM with $M$ topics first generates a topic coefficient latent variable $z\in\mathbb{R}^{M}$ from a multivariate Gaussian distribution $P_c(z) \triangleq \mathcal{N}(z; \mu, \Sigma).$

Each coefficient of $z$ can be interpreted as the ``strength'' of a topic in a document, so the covariance matrix $\Sigma$ captures the correlations among different topics. These weights $z$ are then normalized using a softmax function, the result of which is used to parameterize the distribution over topic $t_n$ for the $n$-th word. Each topic $t_n$ induces a categorical distribution over word types $P(x_n|t_n)=\phi_{t_n, x_n}$, where $\phi_{ij}$ parameterizes the probability of emitting word type $j$ conditioned on topic $i$. The joint probability of a document with $N$ words is:
$$
P_c(x, t | z; \phi) = \prod_{n=1}^N{[\text{softmax}(z)]}_{t_n}\phi_{t_n, x_n}.
$$

Since we are only interested in criticizing the document-level $z$, we marginalize out the topic assignments of individual words:
$$
P_c(x|z) = \prod_{n=1}^N \sum_{i=1}^{M} {[\text{softmax}(z)]}_i \phi_{i,x_n}.
$$

To fit this generative process on data, we use variational inference and maximize the ELBO following \citet{blei2006correlated}.
We set $M$ to 100. Since analytical posterior inference is intractable, we use variational inference to estimate $P_c(z|x)$.

\begin{table}[t]
    \centering
    \begin{tabular}{@{}ll@{ }c@{ }c@{ }c@{}}
    \toprule
        Metric & Model & \textsc{PubMed} & \textsc{ArXiv} & \textsc{Wiki}\\
        \midrule

         \multirow{ 3}{*}{Latent NLL} & Data & 174.43 & 163.95 & 124.70\\
        & LM$_1$ & 172.70 & 161.40 & 123.30\\
        & LM$_2$ & 172.81 & 163.17 & 124.35\\
        \bottomrule
    \end{tabular}
    \caption{Latent NLL of topic correlation modeling. Transformer LMs perform similarly to the real data.}
    \label{tab:ctm}
    \vspace{-0.25cm}
\end{table}

\paragraph{Results} \Cref{tab:ctm} shows the main results. The Latent NLLs of LM generations and real data are close on all three datasets (there are outlier pathological generations that we can identify using $T(x)$, as shown in \Cref{sec:appendix_topic}). In \Cref{fig:cov}, we visualize and compare the covariance matrices of the aggregated posterior distributions of LM generations and real data, and find that transformers are able to model the correlations among topics well. These results indicate that topic correlation is well represented in text generation systems, and is likely an easier task to model than ordered coherence.

\section{Related Work}

\paragraph{Text Generation Evaluation}
Traditional evaluation metrics include perplexity,
and n-gram overlap metrics for translation-type problems such as BLEU \citep{papineni-etal-2002-bleu}, ROUGE \citep{lin-2004-rouge}, METEOR \citep{lavie-agarwal-2007-meteor}, and NIST \citep{martin2000nist}.  In recent years, with the emergence of neural models that learn contextual representations \citep{devlin-etal-2019-bert,liu2019roberta}, researchers propose to project text to contextual representations and compute distance in this space \citep{zhang2019bertscore,zhao-etal-2019-moverscore,pillutla2021mauve}. The closest work to ours is \citet{eikema-aziz-2020-map}, which evaluates different decoding strategies in machine translation by comparing the statistics of the produced text. While these past works mainly concern word-level string/meaning representation matching, the goal of our work is to check the high-level aspects of the generated text such as coherence. Besides, word-level matching is not suitable for evaluating open-ended generation tasks due to the existence of too many plausible references
\citep{celikyilmaz2020evaluation}, while our work projects text to a more manageable
lower-dimensional latent space to make the evaluation of open-ended generation feasible.

\paragraph{Evaluation of Long-Form Text}
There is a long line of research evaluating the discourse coherence of text \citep{grosz-etal-1995-centering,poesio-etal-2004-centering,barzilay-lapata-2005-modeling,lai-tetreault-2018-discourse,logeswaran2018sentence,persing-etal-2010-modeling}.
Most learn a predictor that maps features such as the distribution of entities \citep{barzilay-lapata-2005-modeling} or the transitions of topics \citep{persing-etal-2010-modeling} to manually-labeled coherence scores.
Our work differs in two important ways: first, we unify the evaluation of different high-level aspects of text using the formalism of model criticism; second, we do not assume any annotated coherence scores---we only specify a generative process in order to project text to a latent space for the comparison between machine-generated text and real text.  Recently, there have been works targeting the evaluation of discourse-level coherence, such as BARTScore \citep{yuan2021bartscore} and DiscoScore \citep{zhao2022discoscore}. These methods presume either a conditional generation setting or require textual references. We also note that model criticism does not use a generic neural representation, but focuses on specific user-specified high-level aspects of text. In this respect, our work is similar in spirit to some recently proposed suite-based metrics, such as Language Model Evaluation Harness~\citep{eval-harness} and BIG-bench~\citep{srivastava2022beyond} that utilize many different skill-based metrics.

\paragraph{High-Level Issues of Text Generation Models} Concurrent with our work, several other groups also notice that existing LMs fail to capture some high-level aspects of text. For example, similar to our findings of LMs being not strong at discourse coherence, \citet{sun-etal-2022-chapterbreak} observe that large LMs including GPT-3 fail to assign a higher probability to ground truth chapter continuations compared to distractor chapters given a prefix, and yet a simple classifier trained on this identification objective can achieve a much higher accuracy.  Similar to our findings of LMs being not strong at modeling coreference,  \citet{pmlr-v162-papalampidi22a} find that LMs fail to maintain long-range entity consistency and coherency in the generated narrative stories.

\paragraph{Model Criticism}
Model criticism, also known as model checking, is a general framework for checking if a generative model fits the data well
\citep{box1980sampling,gelman1995bayesian,stern2005bayesian,o2003hsss}. Model criticism is different from aforementioned metrics such as PPL and is similar to two-sample tests \citep{hotelling1951generalized} in that it computes and compares some statistics on the real data and on the samples to determine if they are close enough. While the statistics may be directly computed in the observation space, in many applications we are interested in criticizing some latent aspects of data such as topics \citep{mimno-blei-2011-bayesian} or latent factors \citep{seth2019model}. To this end, \citet{dey1998simulation} introduce model criticism in latent space, which measures the discrepancy between real data and model generations in the latent space induced by a generative model \citep{chaloner1988bayesian,o2003hsss,seth2019model,dey1995simulation,weiss1995residuals,dey1998simulation}.
Recently, \citet{barkhof2022statistical} propose to use model criticism to evaluate VAEs.
Model criticism in latent space forms the basis of our work, with two major differences: first, we apply model criticism to models with a point estimate of parameters such as commonly-used neural language models instead of models with uncertainties in their parameters.
Second, we allow for using a different generative model to induce the latent space from the model that we criticize. By separating out the model to be criticized and the generative process used for projecting data to the latent space, our approach allows for criticizing different views of the data depending on user needs and for criticizing generative models without any latent variables such as neural language models.
For qualitative analysis and outlier identification, our work applies visual posterior predictive checks \citep{gabry2019visualization,gelman1997bayesian}, a graphical version of model criticism.

\begin{comment}
\paragraph{Model Criticism}
Model criticism, also known as model checking, is a general framework for checking if a generative model fits the data well
\citep{box1980sampling,gelman1995bayesian,stern2005bayesian,o2003hsss}. Model criticism is different from aforementioned metrics such as PPL and is similar to two-sample tests \citep{hotelling1951generalized} in that it computes and compares some statistics on the real data and on the samples to determine if they are close enough. While the statistics may be directly computed in the observation space, in many applications we are interested in criticizing some latent aspects of data such as topics \citep{mimno-blei-2011-bayesian} or latent factors \citep{seth2019model}. To this end, \citet{dey1998simulation} introduce model criticism in latent space, which measures the discrepancy between real data and model generations in the latent space induced by a generative model \citep{chaloner1988bayesian,o2003hsss,seth2019model,dey1995simulation,weiss1995residuals,dey1998simulation}.
Recently, \citet{barkhof2022statistical} propose to use model criticism to evaluate VAEs.
Model criticism in latent space forms the basis of our work, with two major differences: first, we apply model criticism to models with a point-estimate of parameters such as commonly-used neural language models instead of models with uncertainties in their parameters.
Second, we allow for using a different generative model to induce the latent space from the model that we criticize. By separating out the model to be criticized and the generative process used for projecting data to the latent space, our approach allows for criticizing different views of the data depending on user needs and for criticizing generative models without any latent variables such as neural language models.
For qualitative analysis and outlier identification, our work applies visual posterior predictive checks \citep{gabry2019visualization,gelman1997bayesian}, a graphical version of model criticism.
\end{comment}

\section{Limitations}
One limitation of the proposed approach is its reliance on choosing a critic generative process $P_c$, which presumes some knowledge of a true data generating process. For an improperly specified critic, it does not expose the latent space that we intend to criticize. However, since we compare statistics between real data and model generations (similar to two-sample tests), for a good model the statistics should be close even with improper critics

Another limitation is that not observing any differences does not imply that the model generations conform to the unknown data distribution---it simply means that they are close with regard to the latent aspects that we criticize~\citep{o2003hsss}.

Recently, researchers found that certain capabilities such as reasoning under augmented prompts only emerge in large LMs beyond tens of billions of parameters~\citep{wei2022emergent}. Since the largest LM tested in this paper only has 1.5 billion parameters, future work is required to investigate whether the high-level issues observed in this paper can be solved by further scaling model size.

\section{Conclusions
 We consider the problem of evaluating long-form text generation for specific discourse properties. We propose a statistical tool, model criticism in latent space, which projects text to a latent space based on an assumptive generative process, and compares the implied latent distribution. Different critic generative processes focus on different properties of data. We apply this tool to analyze three representative document properties: coherence, coreference, and topicality, using transformer-based language models. Experiments find that while transformer LMs can capture topical structures well, they are not currently strong at modeling discourse coherence without explicit markers or at modeling coreference.

\section{\label{sec:ethical}Ethical Considerations}
In our experiment of critiquing coreference chains, we used a gender binary \citep{hyde2019future} to categorize proper nouns, but there are many individuals who do not adhere to the gender binary that this simple categorization fails to consider \citep{bamman2014gender}. The reason for the gender binary is primarily because personal pronouns are typically gendered in English which makes the qualitative and statistical analysis more clear. For example, one coreference error detected by the approach is to use pronouns of different genders to refer to the same person, as shown in \Cref{sec:appendix_coref}. In \Cref{sec:appendix_coref}, we describe the exact procedure through which the genders of proper nouns are determined to make explicit what our ``gender'' definition is \citep{larson2017gender}. Going forward, exploring other features of proper nouns such as their syntactic features \citep{shieber-tao-2003-comma} to replace gender assignments here might further mitigate this concern.

\section*{Acknowledgements}
YD is supported by an Nvidia Fellowship. AR is
supported by NSF CAREER 2037519, NSF 1704834, and a Sloan Fellowship. We would also like to thank
Harvard University FAS Research Computing for providing computational resources.

redpajama_set_name: RedPajamaArXiv


text:
Кержако́в — русская фамилия. Происходит от слова кержак.

Известные носители
 Кержаков, Александр Анатольевич (род. 1982) — российский футболист, тренер.
 Кержаков, Михаил Анатольевич (род. 1987) — российский футболист.

redpajama_set_name: RedPajamaWikipedia


text:
Q: Is division algorithm true in $\mathbb{Z}[x]$? I have a confusion about one of my question I once asked on this site:
In this question I wrote this followings:

Any element, say, $~f(x)+ \langle x^2+x+1, 2\rangle~\in F$ can be written in the form $(ax+b)+\langle x^2+x+1, 2\rangle$, where $a,b \in \mathbb{Z}$ (using Division Algorithm). Now reducing $a,b$ by $\mod 2$ we see that there are $4$ choices of $a,b$.

But is this idea really true? I mean, $\mathbb{Z}[x]$ is not a Euclidean Domain. So I think the Division Algorithm is not applicable there! Am I right? Please mention if there is some silly mistake. Thank you.

A: Let $R$ be any ring, $g\in R[x]$ monic of degree $m$.
Suppose $f$ has degree $n\ge m$ and its leading coefficient is $a_n$.
Then $f_1:=f(x) -a_ng(x)x^{n-m}$ is in the same $g$-coset as $f$ and has strictly smaller degree. Iterate it until the degree becomes smaller than $m$.

redpajama_set_name: RedPajamaStackExchange


This perennially popular event at True Nature Healing Arts in Carbondale was once again sold out this past New Year's Day. Yogarupa delivered the message that all in attendance needed to hear, as expected!
Click here to view other YouTube videos.


text:
Miss AK prepares for national stage
Abby Hancock
For one Alaskan, the month of January
is not just the start of a new year, but the
beginning of new possibilities. UAA
student Stephanie Jeffers, a 23-year-old
theater major, is the state's 50th Miss
Alaska, and in the last week of the
month, she will be in Las Vegas for the
78th annual Miss America Pageant.
"Stephanie is very ambitious and very
spirited," director of the Miss Alaska
Scholarship Foundation, Bonnie Faulk
said. "She's a highly intelligent young
woman and can articulate what she needs
to say very easily."
Jeffers started competing in the Miss
Alaska pageant to earn scholarship
money for school. After completing a
Biology degree at Whitworth College
in Spokane, Washington, she returned
home to Alaska and gave the pageant
another shot.
In July 2008, Jeffers won the title of
Miss Alaska and took home a four-year
tuition waiver to UAA along with her
crown and sash. Jeffers is now putting
those scholarship dollars to good use by
fi nishing up a theater degree at UAA.
Besides participating in productions
put on by the theater department, Jeffers
has also been working on projects of her
own. The most recent is the UAA Improv
Troupe. To Jeffers, improvisational
comedy is not just for laughs, but is
also her community service platform
throughout her reigning year. Jeffers
believes improvisation can be used as a
tool – the fundamental idea behind her
platform.
"I'm hoping we'll see an expansion of
how improv is used," Jeffers said.
The theater major has been preparing
for the Miss America stage since she won
the crown. Besides a full schedule of work
and school, Jeffers said she has also had
to work hard to prepare herself physically
and mentally for the pageant.
"There's going to be a lot more attention
on Miss Alaska," Jeffers said. "It's been a
huge year to be Miss Alaska. There have
been all sorts of awesome things going
Not only was it a big year for Alaska,
but also a big year for the Miss Alaska
Scholarship Foundation. Just shy of
three months before the state pageant,
the program gained new directors and a
complete makeover, something that Jeffers
is also very excited about.
"This is a new program; we've
completely re-vamped it," she said. "I
believe this program is an opportunity for
us to get recognition for the work we've
done and to have a shot at the job that is
Miss America."
According to Faulk, much of the
success of the program is due to committed
volunteers.
"The volunteers that came together to
help me with this transition were fabulous,"
Faulk said. "It was a good partnership."
Faulk is also working to increase the
amount of scholarship funds as well as
expand the program to include more
preliminary pageants every year.
Faulk continues on that the Miss
America Organization is a strong
community service pageant that offers
more scholarships to young women than
any other organization in the entire world.
Adriel Mathew, a 25-year-old foreign
language major at UAA competed in the
Miss Alaska Pageant four times, and said
that she is thankful for the experience.
"I was really nervous to begin with, but
afterward I was glad that I did it," Mathew
According to Mathew, her favorite
element of the pageant was making friends,
particularly with Jeffers.
"She is so genuine and sincere and
was a great encouragement," Mathew
said. "That's her strong point-she's not a
plastic doll."
Mathew also offers advice for future
contestants.
"Go in there, fi rst off, knowing who you
are and what is important to you," she said.
"But also be prepared to be taught."
Faulk said one of the goals for the
program is to offer more clinics and
workshops to prepare the girls for the
stage.
"I have yet to fi nd a girl who has gone
through this system and has not gained
anything from it," Faulk said.
And Jeffers is the perfect example. Not
only does she have the once in a lifetime
shot at the Miss America crown, but she
will bring home thousands of dollars in
more scholarship money just by stepping
onto the national stage – and much more
if she wins the title.
"I want to be Miss America if I'm right
for the job," Jeffers said. "There's a sense
that if I'm not right, I don't want to win.
But then, what if I am? It's exciting for me,
but it's also a little terrifying."
Previous articleBegich welcomed with end of an era
Next articleA&E Briefs
UAA | University Art Analysis – The lasting legacy of the...

redpajama_set_name: RedPajamaCommonCrawl


text:
Šedá energie či ztělesněná energie je označení pro energii potřebnou na výrobu, dopravu, uložení, prodej a likvidaci. Je důležitou součástí životního cyklu produktů napříč různými obory a průmysly – ať už jde o dopravu, stavebnictví, design nebo potravinářství.

Architektura a stavebnictví
Šedá energie je označení pro energii potřebnou na výrobu, dopravu a likvidaci stavebního materiálu stavby. U nových domů se často posuzují energetické standardy samotného provozu budovy, šedá energie je ale také velkou součástí celkové energetické bilance budovy. Například u pasivních a nízkoenergetických budov, jejichž cílem je minimalizace provozních nákladů spojených s potřebou energií, může šedá energie dosáhnout vysoké hodnoty, uvádí se 10 až 30 % spotřeby domu za 50 let. Nevhodnou volbou stavebních materiálů může dojít k tomu, že to co ušetří budova na spotřebě energie za provozu, může být v konečném důsledku jen zlomek šedé energie "zabudované" do stavby.

Stavební materiály, které mohou snížit množství šedé energie, jsou například dřevo, nepálená hlína, izolace z ovčí vlny, bavlny, korku, technického konopí a slámy, z recyklovaného papíru nebo skla.

Odkazy

Reference

Související články
 Šedá voda

Energie
Ekologická ekonomie


Patients will be able to get a 360-degree preview before a medical procedure.
By Dana Guth·	 12/7/2015, 8:50 a.m.
Patients often suffer a great deal of anxiety in the moments leading up to a procedure. But what if an individual could scope out the ER, meet her surgeon, and feel out her surroundings months before an appointment?
That scenario is becoming a virtual reality at Tufts Medical Center, where a team of physician assistants have enlisted local marketing firm Primacy to bring virtual reality (VR) to patient care. The technology will detail the pre-procedural process and show a 360-degree view of the medical facility and its staff, hopefully allaying patient fears before an appointment.
Though still in its planning stages, the partnership between Tufts and Primacy will be specifically geared toward patient education. Doctors will use the VR technology to show pre-procedural consent, introduce relevant personnel, and explain medical equipment. Kimmelstiel says this process will be especially crucial to those undergoing treatment without anesthesia, and thus who will be completely aware of their surroundings during the procedure.
As of now, Tufts will be the first medical center in the country to employ 360-degree virtual reality in a patient-care setting. While there's no set date to implement the program yet, Kimmelstiel says his team isn't wasting any time.


text:
Q: Set up JUnit test database void insertAllRecords(){
    insertRecord("tom", 100);
    insertRecord("jake", 200);
    insertRecord("tim", 300);
    insertRecord("andy", 400);
    insertRecord("mike", 500);
}

I am writing JUnit test for something that has a test database.  It starts as an empty database.
I want to test different things, but I don't want to have duplicated insertRecord with same params.
Is there a simple way to specify which line(s) to execute?  For example, I only want to add tom and andy in one of the @test function.
I KNOW IN EACH TEST METHOD, THE DATABASE STARTS AS 0.
I want to be able to:

*

*in testA, set up db with tom and jake.

*in testB, set up db with andy
and mike.

*in testC, set up db with all of them.

But I don't want to put individual insert statement in every test.  I want to be able to call insertAllRecords with some parameters.
Thanks

A: As far as I know, JUnit does not support the behaviour that you want. You might want to take a look at TestNG which allows test specific setup with @BeforeMethod and @AfterMethod annotations. And it also allow more flexible test data using parameterized data with @Parameter or @DataProvider annotations.

A: convert your Junit Class to an abstract class with the abstract method abstract void insertAllRecords();
Then extend this class in the cases, you need. For example
class TestWithTomClass extends AbstractTestClass{
    public void insertAllRecords(){
       insertRecord("tom", 100);
    }

    @Test
    public void myTestWithTom(){
       ....
    }


It's a lovely summer afternoon and I am reading Tess of the d'Urbervilles while sipping on a hot cup of Pukka brand Three Ginger tea. I have three flavors in my cupboard right now due to the fact that I have given up caffeine entirely. (But please don't hold me to that and for god sakes, don't say, 'I thought you gave up coffee?!' when you see me drinking it inevitably in December).
Oh my god, did you guys know how good classics are?! I read Jane Eyre this summer and now I'm on to Tess. What was I thinking in high school? I swear I used to to try to read these, only to promptly give up and read the SparkNotes instead. What was so difficult? Perhaps as teenagers we have way too much going on in our minds and our priorities are a bit different. I would actually look at the page and not comprehend a single sentence. I wish I hadn't been so stupid in High School.


text:
Tag: Dante Booker
Buckeye Linebackers Will Be Key Against Penn State
September 25, 2018 by Tony Gerdeman
The Ohio State linebackers have had their ups and downs this season, and with Penn State's offense on the horizon, now would not be the time for one of those unhappy valleys. Penn State leads the conference in rushing, averaging 275 yards on the ground per game. They also lead with a 6.4-yard-per-carry average and 20…
Ten Buckeyes Who Need to Step Up in 2018
August 20, 2018 by Tony Gerdeman
While it is always important for new stars to emerge for college football teams, it is also imperative for established players to take a step forward and become better than they were the year before. As the saying goes, if you're not getting better, you're getting worse. For the Buckeyes, there are a number of players…
5 Things To Watch For On The Ohio State Defense
August 6, 2018 by Tom Orr
Ohio State football is now three practices into fall camp, and a little under four weeks away from its first game of the season. The…
Bill Davis Looking Forward to Having Dante Booker Back on the Field
July 17, 2018 by Tony Gerdeman
This spring, the Buckeyes rotated their linebackers quite a bit looking to find their best three. With starting middle linebacker Tuf Borland out, however, the situation was pretty fluid. Also missing was fifth-year senior Dante Booker, who was recovering from shoulder surgery. While much of the attention was directed…
Ranking Ohio State's Playmakers on Defense: No. 10 — Dante Booker
July 6, 2018 by Tony Gerdeman
Dante Booker arrived at Ohio State with 5-star accolades and 6-star expectations. As the backup behind Joshua Perry for two seasons, he learned from one of the best. His first opportunity to start came in 2016, but he was lost in the first game of the season. He then won a job last season as well, but again dealt with injuries.
Whoever Starts at Linebacker for Buckeyes 'Better Bring Their S***'
May 16, 2018 by Tony Gerdeman
Spring football this year for the Ohio State linebackers began with seven players trying to prove they belonged. Tuf Borland was the sure thing at middle linebacker, but an injury to his Achilles tendon knocked him out of action until around September. Justin Hilliard and Baron Browning then stepped in for him and…
Ohio State Spring Recap: Linebackers Lack Experience, Little Else
May 3, 2018 by Tony Gerdeman
The Ohio State linebackers left spring ball without any starters being named, other than the injured Tuf Borland, who won't even be practicing until around September due to an Achilles tendon injury. Based on who was healthy and who is more experienced, we can still take a pretty educated guess at where things ended up.
Football Hayes & Cannon
Ohio State Spring Football Overview – Linebackers
March 30, 2018 by Chip Minnich
Ohio State's linebackers did seem to play better later in the season. That was after Chris Worley was moved back to his more natural outside linebacker spot while Tuf Borland took over the middle linebacker role. Now, the three linebackers who finished the 2017 season are not around, with Worley and Jerome Baker off to…
Ohio State Football To Do List — LB Coach Bill Davis
March 19, 2018 by Tony Gerdeman
Ohio State linebackers coach Bill Davis has lost starters Chris Worley and Jerome Baker from last season. The Buckeyes return three players with starting experience in fifth-year senior Dante Booker, redshirt sophomore Tuf Borland, and junior Malik Harrison. Booker is out this spring following shoulder surgery, but…
Let the (Position) Battles Begin!
March 8, 2018 by Brandon Zimmerman
Do you smell that in the air? No, not the stench of NBA basketball. Wrong again, it is NOT the stench of the Browns fans trying to convince the world that Saquon Barkley is a horrible NFL prospect while touting Josh Allen as a franchise quarterback. This smell is greatness, this smell is real life football being played in Columbus.
Several Buckeyes to Miss Spring Camp with Injuries
March 6, 2018 by Tony Gerdeman
Ohio State kicked off spring practice on Tuesday and the Buckeyes were without a handful of key players due to injury, including four players who started last season. Save for freshman quarterback Matthew Baldwin, none of the injuries are expected to impact the 2018 season. Baldwin's injury came at the end of his…
1 2 3 Next chevron_right


Native of Savoie, Agathe left her French Alps for Switzerland after graduating from a Bachelor in Marketing.
She then devoted two seasons to her first passion: skiing. In October, she decided to resume studies in Sports Marketing while joining the WhiteFrontier team. A new experience that she loves. Dynamic and cheerful, Agathe will always do everything to satisfy her interlocutors. When she has a few hours of free time, she does not hesitate to wake up early to make the first run in the lines of Verbier.
Motto: No snow, no show !


text:
Q: spark dataframe multiple columns into a list/array I have a spark (version 2.2) dataframe dtfBAIDate as below.

I would like to convert them into a list/array, list((A,xxxxxx),(B,xxxxxx),(C,xxxxxx),(D,xxxxxx)). So later I could loop the elements in it
lisBAIDate.foreach(e=> {
  val (strBAI, strDate) = e
  println(strBAI + " Date=" + strDate)
...
})

The following codes made it happen. But is there a neater way of constructing a list/array with 2 strings per element?
  val arrBAI=dtfBAIDate.collect.map(_(0).toString)
    val arrDate=dtfBAIDate.collect.map(_(1).toString)
    val lisBAIDate=(arrBAI zip arrDate)

A: val lisBAIDate= dtfBAIDate.collect.map(x => (x(0).toString,x(1).toString))


Let our fifth star guide you!
we provide more space for your emotions.
A constellation of opportunities for your entertainment and leisure. From the water parks to the exclusive apartments in beautiful natural surroundings, sports facilities where you can enjoy movement in all its forms, through to the delicious flavours prepared by the magic touch of our chefs. Because our fifth star is a comet full of the most amazing surprises for your next holiday, ready to hop on board?
Exclusive Maxicaravan with private Jacuzzi in lake view area!
Spazio al comfort, a tu per tu con la natura.
La Terrazza, with their delicious, unmistakable cuisine. Made in Italy.
The camping area has pitches from 90 to 100 square metres, with electricity from 10 to 16 amps, De Luxe pitches with TV DVB-T and WIFI included.
A private bathroom with all amenities, just for you and your family.
An agility dog course, a dog shower and special accommodation for your favourite pet.
GESTIONI BENACENSI S.P.A. - Via Fossalta, 42 - 37017 - Lazise (VR) - IT - C.F. (tax code) and P.IVA (VAT no.) 02219040231 - Iscr. Reg. Imprese Verona 224617 - Cap. Sociale (registered capital) € 300.000,00 i.v.


I'm glad to announce the PascalSCADA 0.7.5. It's a bugfix release.
This version will be the last one that maintains the current tag scan mechanism. That is, the next version will have changes in this mechanism.
It's available both in the SourceForge and in OPM repository, was announced previously.
To download the full package (with examples and class documentation) from Sourceforge click here.
To update your PascalSCADA copy using OPM, follow the instructions listed here.
The minimum requirements to use this PascalSCADA version is FPC 3.0.4 and Lazarus 1.8.0.
As informed in the post title, I have to start explaining why the delay replying the comments of the page. Over a year ago, I migrated the page from the Sourceforge server to a private server. After completing the migration, I made a check of the settings, especially in email notifications. I installed a plugin to send emails via Gmail, sent tests messages and everything was OK.
However, I discovered that a small portion of the messages on the page was sent. The majority of messages were not sent. Moderation notifications never left the server and the page looked abandoned.
So this week I reviewed the settings and I'm being notified when comments are posted. I asked random people to register and send some kind of comment, and I was notified of all.
I apologize to all who have accessed and commented on the page and who have not answered.


Every year brings with it the promise and the excitement of new films and new faces in Bollywood 2014 will see several new actors. Coincidentally, all of them have a famous relative already in the industry. We take a look.
Every year in Bollywood brings with it the promise and the excitement of new films and new faces. The year past saw the Khans — Aamir, Salman, Shah Rukh — going strong, even in their late forties, though new talent — the likes of Sushant Singh Rajput — came to the fore. The year present, 2014, will see several new actors. Coincidentally, all of them have a famous relative already in the industry. We take a look.
He is all of 21, and looking to make it big in Bollywood. According to some, he's also one of the most promising newcomers in Bollywood. Here, he talks about the pressure of being a star kid (he's Jackie Shroff's son), on nerves and excitement.
Before venturing into films, Isabella studied acting at the famous Lee Strasberg Theatre and Film Institute in New York (incidentally, Katrina's alleged boyfriend Ranbir Kapoor, also studied at the same institute). Now, she will venture into films with an Indo-Canadian co-production, Dr Cabbie, that is co-produced by Salman Khan. The film also stars Kunal Nayyar of the Big Bang Theory fame.
The political thriller has been produced by Salman Khan's brother-in-law Atul Agnihotri. Apparently, Salman has always shared a good relationship with the Amrohi family and is believed to be fond of Bilal — son of Taajdaar and Neelu Amrohi. In the film, Salman is touted to do a cameo, which is yet to be shot. The film also stars Pulkit Samrat.
The 21-year-old studied screenplay-writing and acting in the US. And this year will see him moving in front of the camera with Rakeysh Omprakash Mehra's next. He has worked as an assistant to Anurag Kashyap on the Ranbir Kapoor-starrer Bombay Velvet (unreleased). Veteran writer-lyricist Gulzar is working on the screenplay of Mirza Sahiba, a musical based on a Rajasthani folk tale.
Armaan, son of Reema Jain (Rishi and Randhir Kapoor's sister), will make his debut in a Saif Ali Khan production. The film, to be directed by Imtiaz Ali's brother, Arif, will see south actor Deeksha Seth opposite the newbie. Armaan, in the past, has assisted Karan Johar on My Name Is Khan (2010), and was also an assistant director on Shakun Batra's Ek Main Aur Ekk Tu (2012) that starred cousin Kareena.
Athiya, fresh from her filmmaking and liberal arts course at the New York Film Academy, is set to be Sooraj Pancholi's leading lady in the Hero remake. She will reprise Meenakshi Sheshadri's part.
Akshara's elder sister, Shruti, is already an established name in Bollywood and Tamil films. Now, Akshara is set to make her big-screen Bollywood debut with R Balki's next, that will also star Amitabh Bachchan and Dhanush. Produced by Abhishek Bachchan, the film will see Dhanush as a mute person who aspires to become an actor.
The 22-year-old started off as an assistant director on films such as Guzaarish (2010) and Ek Tha Tiger (2012). Last year, he was in the news — not for the best reasons — in connection to Jiah Khan's suicide. Now, he is set to face the camera in the Salman Khan and Subhash Ghai-produced remake of Jackie Shroff-starrer, Hero (1983). The Nikhil Advani-directed film is set to go on floors in February.


It provides an image circle that covers Full-Frame-sized sensors (43.3mm image coverage) and features a maximum T-stop of 2.2, and the aperture does not ramp over the zoom range. It shares common lens gear positions with the other lens in the Sigma High Speed Zoom lineup, which makes for rapid lens changes.
Supplied on a 3 day hire week rate.


text:
Alabama Medical Cannabis Commission receives 94 license applications
Virginia GOP Lawmaker Files Marijuana Sales Bill For 2023, Eliciting Mixed Advocacy Reactions
All the applications are in, and now the Alabama Medical Cannabis Commission has some decisions to make as to who will get what licenses.
Despite having only nine applications 24 hours before the deadline, the commission closed out at the deadline with 94 total applications being returned.
"Applicants have provided plans for production, business operations, facilities, and security, to name a few, as part of their application," commission chairman and oncologist Steven Stokes said in a press release. "Although reviewing these competitive applications is a huge undertaking, we will continue to move forward in implementing a fair and robust process that makes public health and safety a top priority."
"The Alabama Medical Cannabis Commission announced late Friday afternoon that they received over 90 applications for medical cannabis business licenses by the deadline. We are pleased to see this impressive number of applicants and companies looking to participate in the state's medical cannabis industry," said the Alabama Medical Cannabis Association.
"With the high engagement from our members and interest across the state, we look forward to seeing the impact of this emerging supply chain in Alabama," says Patrick Lindsey, Director of the Alabama Medical Cannabis Association.
Some of the license types have more competition than others. For example, under the law, the commission can only issue five licenses for integrated facilities, which combine multiple other licenses. The commission will have its work cut out to make those selections after receiving 38 applications for integrated facility licenses.
Additionally, 18 applications were returned for dispensary licenses, but the commission can only award four. The commission will also have to choose up to four of the 12 applicants for a processor license.
The commission can award 12 cultivator licenses, and exactly 12 applications were submitted for such licenses. The commission is not obligated to award a license to each of those applications, however.
The commission also received 11 secure transport applications and 3 state testing laboratory applications. The statute does not limit the number of these licenses that can be awarded.
The review, evaluation, and scoring of applications will serve as the basis of the Commission's decisions regarding the award of licenses. This determination will be reached based on the merits of each application as expressed by ranked score. The commission has engaged the University of South Alabama to establish teams of academic evaluators and other qualified individuals to review, evaluate and score business license applications. The commission has complete discretion as to the number of licenses awarded (not to exceed the limits provided by the Act) and the applicants to whom licenses are awarded.
The Commission will make applicant names and application contents available for public inspection following the formal submission of applications (slated to take place on April 13, 2023, Commission meeting). The Commission will accept public comments on the formally submitted applications for 30 days following the submission of applications.
At its meeting on June 12, 2023, the Commission is scheduled to award licenses in each license category. Once the business licenses have been issued, physicians may begin the certification process to recommend medical cannabis to qualified patients.
"The Commission spent much of 2022 laying the groundwork for Alabama's medical cannabis industry by drafting rules and regulations, obtaining software tracking systems, and developing the business applications," said Director John McMillan. "We are excited to utilize the strong framework we put together to implement a safe and operational medical cannabis industry as we move into 2023."


text:
Q: C++ Thread function calling - problem converting string to ofstream I'm having some hard time understand the error.
I'm trying to create a vector of threads and create N amount of threads to calculate primes & write them to a file using the writePrimesToFile function.
Yet I'm having some errors while calling the function. I understand I have some type error but I can't solve it.
Code:
#include <thread>
#include <time.h>
#include <string>
#include <iomanip>
#include <chrono>
#include <mutex>
#include <vector>
#include <iostream>
#include <fstream>

void writePrimesToFile(int begin, int end, std::ofstream& file)
{
    int count = 0;
    for (int i = begin; i <= end; i++)
    {
        if (checkPrime(i))
        {
            // Added: Thread lock to prevent multiple threads writing to the same file.
            mtx.lock();
            file << i << endl;
            mtx.unlock();
        }
    }
}

// Function calcualtes primes using N as the amount of threads. Output - into filePath.
void callWritePrimesMultipleThreads(int begin, int end, std::string filePath, int N)
{
    // Init Threads pool.
    std::vector<std::thread> threadVector;

    // Calcualte the margin calculation of each thread.
    // This will divide the work eaqual between the N amount of threads.
    int threadSpread = end / N;
    int newEnd = begin + threadSpread - 1;
    std::ofstream file;
    file.open(filePath, ios::out | ios::trunc);

    // Output primes into the file.
    for (int i = 0; i < N; i++) {
        threadVector.emplace_back(writePrimesToFile, begin, newEnd, file);
        begin += threadSpread;
        newEnd += threadSpread;
    }

    for (auto& t : threadVector) {
        t.join();
    }
}

Errors:
Error   C2672   'invoke': no matching overloaded function found Solution
Error   C2780   'unknown-type std::invoke(_Callable &&) noexcept(<expr>)': expects 1 arguments - 3 provided
Error   C2893   Failed to specialize function template 'unknown-type std::invoke(_Callable &&,_Ty1 &&,_Types2 &&...) noexcept(<expr>)'


Jack Hall at his general stores in Mundesley Road.
Jack Hall at his general stores in Mundesley Road (to the right hand side of Ketts). The window is decorated for Christmas and on the sign behind Mr Hall reads "J. Hall, 1 Nelson Street, North Walsham", the old name for Mundesley Road.
Thanks to Margaret Goldsmith for this photo.
This archive item has been viewed 413 time(s).
We believe this photograph of "Jack Hall at his general stores in Mundesley Road." is either in the public domain or we have been granted permission to use it by the photograph owner. If you believe "Jack Hall at his general stores in Mundesley Road." is being used without the appropriate permission then please do not hesistate to contact us here.
If you would like to use this photograph of "Jack Hall at his general stores in Mundesley Road." in your own project then please acknowledge North Walsham & District Community Archive and the appropriate owner (if shown) on the page where it is used, thank you.
If you wish to use this photograph of "Jack Hall at his general stores in Mundesley Road." in print then please contact us as we may have a higher resolution copy on file in our HQ!
Jack Hall with pony cart outside G Moore bootmaker shop in Mundesley Road.


Roast Chicken in Under an Hour | From Scratch Fast - Recipes & meals from scratch for busy cooks!
Yes, you can make a roast chicken on a weeknight. In under an hour. Introducing the spatchcocked chicken (terrible name, incredibly delicious results). A spatchcocked chicken is simply a chicken that's had its backbone cut out. This is really easy to do yourself—you can see step-by-step how to do it here (from my Chicken Under a Rock post a couple of years ago—similar results, but the chicken is cooked on the grill). Or, you can expedite things and ask your butcher to cut the backbone out for you. The flattened chicken gets seasoned any which way you want, then browned on the stovetop and finished in the oven. The increased surface area means more crispy skin, and a much faster cooking time—we're talking 5 minutes on the stove and 20-30 minutes in the oven. The result, my friends, is the chicken of your dreams—juicy, tender and beautifully browned.
I'm a big proponent of eating dinner as a family, but, while we adore our three-month-old, we've decided that her "witching hour" screams don't count as suitable dinnertime conversation. We've therefore adjusted our schedule so that we now eat with Ella after Juniper goes to bed. A few evenings ago, however, Ella was exhausted, and, with her consent (and the lure of fried eggs for dinner), we put her to bed early with Juniper. I had slathered a simple spice rub of ground cumin, ground fennel seed, minced rosemary, minced garlic, salt, pepper and olive oil under the skin of a spatchcocked chicken, which I quickly browned on the stovetop then threw in the oven before heading upstairs to read Ella her bedtime book. By the time her teeth were brushed and Fancy Nancy was finished, the chicken was perfectly roasted. James poured me a glass of wine, lit some candles, and we sat alone at the dining room table. No babies, no five-year-olds, just us. It was the best damn chicken I've ever had. Ok, that's most likely because it was the first time in months that I didn't have to scarf my food to appease a crying newborn or shuffle a preschooler to bed. A new tradition has officially been born—date night dinner. Sorry kids, this one's a keeper.
This is the quickest way to deliver incredibly juicy and delicious roasted chicken to your table any night of the week! You can keep it simple by seasoning simply with salt and pepper, or you can get creative by rubbing spices or chopped fresh herbs under the skin. Some of my favorite spice blends include: cumin, fennel, rosemary, and garlic; smoked paprika, oregano, cumin and orange zest; or minced fresh thyme, minced fresh tarragon, garlic and lemon zest. Mix just enough olive oil or softened butter into the spices/herbs to create a paste, and season the paste with salt and pepper. Using your fingers, gently loosen the skin from the chicken, and rub the spice mixture under the skin on top of the breast and over the thighs. Sometimes I'll also slide a couple thin slices of lemon under the skin. If there's any spice mixture leftover, it can be slathered over the outside of the bird.
If it hasn't been done for you, first cut out the backbone of the chicken. Place the bird breast side down on a cutting board. Using a sharp pair of kitchen shears, cut along one side of the backbone, then along the other side to remove it completely (you can discard the backbone or freeze it for stock down the line). Turn the chicken breast side up and press down on the breast to flatten it. Next, tuck the wings under the breast. If you're using a spice rub, now's the time to rub it under the skin. Season the chicken with salt and freshly ground black pepper. Let the chicken sit at room temperature for a few minutes to come to room temperature. Do Ahead: The chicken can be spatchcocked and seasoned up to 1 day in advance and refrigerated. Bring to room temperature before proceeding. By the way, a whole chicken can safely sit out at room temperature for up to 2 hours.
Heat 1-2 tablespoons of vegetable in a heat-proof heavy-bottomed skillet, preferably cast iron, over medium-high heat. There should be just enough oil to lightly coat the bottom of the pan (if you're using a stainless steel pan, opt for closer to 2 tablespoons of oil to prevent the skin from sticking). Once the oil is shimmering, place the chicken in the pan, breast side down. Let the chicken cook, undisturbed, for 4-6 minutes, or until the skin is golden brown and releases easily from the bottom of the pan. Flip the chicken over so that it's breast side up, then slide the whole pan into the oven. Roast for 20-30 minutes, or until the chicken juices run clear (the internal temperature of the thigh should read 165-170˚F). If the skin starts getting too dark before the chicken is cooked through, tent it with foil. Transfer the chicken to a cutting board, and let it cool for 10 minutes before carving. Serve the chicken with a drizzle of the pan juices, if you'd like.


text:
Mratín (Duits: Mratin) is een Tsjechische gemeente in de regio Midden-Bohemen, en maakt deel uit van het district Praha-východ.
Mratín telt 760 inwoners (2006).

Gemeente in Praha-východ


In the simplest scenario you will have a RadComboBox populated with static data in XAML.
The RadComboBox and RadComboBoxItem classes expose a property named IsEnabled. Set this property to False, when you want to disable either the whole RadComboBox or specific RadComboBoxItems. The next code-snippet shows you how to do that.
Using RadComboBox with static data is the simplest scenario. However, in most of the cases you will have a RadComboBox populated with a collection of business objects. In this case you have no other options except for using either the ItemContainerStyle or the ItemContainerStyleSelector.
The next example shows you how to set the ItemContainerStyle of a RadComboBox.
Note that in this case all RadComboBoxItems will be disabled. However, you have the ability to determine which items to be disabled based on your custom logic. Check out the next section to see the solution.
The StyleSelector provides a way to apply styles based on custom logic.
For more information about the StyleSelectors take a look at the DataBinding topics in the Common help section.
The next example demonstrates you how to achieve this.
And the result is that every odd item is disabled.
Two things should be mentioned here. First, note how the IsEnabled property is bound in the Style. Second, the declared Style is set as a ItemContainerStyle of the RadComboBox. The result is shown on the image below.


text:
Would leaving the EU make us 'less imaginative'? Actors and musicians speak out
Actors including Benedict Cumberbatch, Sir Patrick Stewart, Bill Nighy, Chiwetel Ejiofor, Keira Knightley and Dame Kristin Scott Thomas are among […]
By Frank O'Donnell
Benedict Cumberbatch, Sir Patrick Stewart, Bill Nighy, Chiwetel Ejiofor, Keira Knightley and Dame Kristin Scott Thomas, who are among leading figures from the arts world backing a vote to remain in the European Union. (Photo: PA Wire)
Actors including Benedict Cumberbatch, Sir Patrick Stewart, Bill Nighy, Chiwetel Ejiofor, Keira Knightley and Dame Kristin Scott Thomas are among leading figures from the arts world backing a vote to remain in the European Union.
Music stars Hot Chip, alt-J and Paloma Faith, authors Dame Hilary Mantel and John le Carre and fashion designer Dame Vivienne Westwood are also among the 282 figures supporting a Remain vote.
The group, including Oscar-winning director Danny Boyle and poet laureate Carol Ann Duffy, said being part of the EU "bolsters Britain's leading role on the world stage", warning that Brexit would leave the country "an outsider shouting from the wings".
"From the Bard to Bowie, British creativity inspires and influences the rest of the world. We believe that being part of the EU bolsters Britain's leading role on the world stage.
The letter comes as the Creative Industries Federation revealed that more than 96 per cent of its members supported a Remain vote on June 23.
The signatories, also including stage and screen veterans Brian Blessed and Sir Derek Jacobi, Billy Elliot star Jamie Bell and Alan Partridge actor Steve Coogan, said: "From the smallest gallery to the biggest blockbuster, many of us have worked on projects that would never have happened without vital EU funding or by collaborating across borders.
"Britain is not just stronger in Europe, it is more imaginative and more creative, and our global creative success would be severely weakened by walking away.
"And what would 'out' really mean? Leaving Europe would be a leap into the unknown for millions of people across the UK who work in the creative industries, and for the millions more at home and abroad who benefit from the growth and vibrancy of Britain's cultural sector.
"Let's not become an outsider shouting from the wings."
Creativity is boosted by EU network
Sir John Sorrell, chairman of the Creative Industries Federation, said: "The UK creative industries are key to the way we are seen by the world and deliver a massive £84.1 billion to our economy.
"Our position as a vital European creative hub is a huge part of this success – we benefit from a vast network of talented people, companies and institutions across Europe."
House of Cards creator and Tory peer Lord Dobbs dismissed the idea that culture was boosted by being in the "failing" EU.
Speaking in a personal capacity, he said: "Our creative industries are booming because of the talent that is in Britain's DNA.
"We are world leaders in so many creative areas. From theatre to literature, video games, feature films, fashion, music and so much else, British artists lead the way and are celebrated around the globe.
Success not due to committees
"It's a success that has been created by the dedication, hard work and extraordinary creative abilities of our artists, it's not because of the EU.
"Culture owes nothing to committees. Ancient Greece was the birthplace of our civilisation yet today, because of the EU's appalling policies, streets that were once filled with the world's greatest philosophers and playwrights are choked with desperate beggars and mountains of rotting rubbish.
"These are the realities of the EU. It's failing. The dream is dead. We need to move on."
What's happened in the ITV drama The Bay so far as it returns for season 2 tonight
Klopp and Dyche in angry exchange - and not for the first time


So we are coming full circle with Michael's behaviors. I am noticing the cycle of ignoring me, to acting out with me, to wanting me all to himself and chatting up a storm all day. All are related to anxiety and also to Michael figuring out his way in the social world and how he fits in with all of it.. I am happy to see he is moving away from extreme aggression and even though there were some outbursts, verbal and physical, he immediately went to his "anger box" and started writing down how he was feeling. He even knew he was supposed to use the anger box before the outburst, and reminded himself he would do it next time. I am also doing my best to remind him to use his "strategy cards" that he made with the psychologist and to try and recognize when he feels scared or nervous inside. It's not the time to act out, but turn inwards and find ways to calmly handle it before lashing out.
I love how he is using talking to try and figure things out. We talked about a family event happening in the future and different ways he will be using to deal with his fears about it. We talked about him being away from me and his fears about that. We talked a lot and though it was little exhausting for me, I could see how he needed it. I was a chatty kid who reasoned through much of my stresses by talking them through with my mother and later friends. It helped me tremendously. Eventually though, I also had to find ways to cope with anxiety and stress through other physical forms though. Meditation saved me and helped me learn balance. Now I know my formula for balance out and do my best to meet that need. I am hoping with time Michael will find his best way to meet his need.
Exceptional Parents, how do you and your child handle anxiety? Do you or they talk up a storm or are there other ways they handle it? As long as you both come full circle in the techniques you use, all will go well. Until next time.


text:
This article was published on August 10, 2015
This Is My Jam is dead but Cymbal will continue the music sharing dream
August 10, 2015 - 12:57 pm
This Is My Jam's announcement that it's shutting up shop and turning itself into a permanent monument to all the great music that was shared on the service is not my jam. But thankfully, there's a new track sharing social app that has recently become very much my new jam. It's called Cymbal and if you love talking about music, you need to get on it.
Big news! We've added two features you've been asking for:
1. Instagram Sharing
2. Sort your feed by hottest songs
? pic.twitter.com/zX0YgbX3jz
— Cymbal (@CymbalFM) July 24, 2015
The iOS app – Android is coming – hews to the same simple principle that This Is My Jam had: Share one song at a time that you currently like with a description of why you like it, follow other people and see what songs they're sharing. You can share tracks from SoundCloud and Spotify. What makes it different is that it's:
starting its life on mobile and built with that idea at its heart from the beginning
incredibly well-realized from an aesthetic standpoint – the UX isn't perfect yet but the UI is glorious
I spoke to Charlie Kaplan, who labors under the (faintly) ludicrous title of Head of Growth but who is basically the company's hype man. He's a former journalist for NPR Music and a long time friend of Cymbal's founders Mario Gomez-Hall, Amadou Crookes and Gabe Jacobs.
Kaplan explains the genesis of Cymbal like this:
The founding thing for Cymbal was a blog Gabe used to write 'Lower Frequencies'. Each post was about one track and focused on what was amazing about that song. He had a couple of contributing writers and the idea was that within that one song would be microcosm of your life at that moment.
That's what makes a song meaningful. It allows you to explain more about you than you can in words. He wrote that blog for a long time. It gave him an opportunity and lens to talk about incredible music coming out, but after a while he started to find it pretty lonely. He wanted to know what everyone else was listening to.
You know that scene from 'Almost Famous' where Zooey Deschanel's character pulls all the records out from under her bed? He wanted to have more people guide him on a musical journey, to help him in musical exploration.
Inevitably, that impulse led to the idea of an app. It's not a new one. Other players like Soundtracking have been trying to turn sharing your favorite songs into a habit (Rhapsody killed that one), while Hype Machine has built an enviable engine for consolidating the music blogs of the world. But Cymbal is a little different.
The guys behind Cymbal came together at Tufts College. Kaplan came into the picture through a fraternal connection:
I met Gabe through his big brother. Prior to Cymbal. I was at NPR Music and wrote about apps for the Webby Awards. Gabe worked on a couple of other projects before this – a geolocation app that didn't work out, was one of them. At that time, his brother pointed him my way. When he started on Cymbal, I obviously had an even closer relationship with that. When they got their first funding, he said 'Ok, let's work together.'
Why does Kaplan do most of the talking for Cymbal? Simple – Gomez-Hall, Crookes, Jacobs are head down building most of the time:
Gabe and Amadou are the two developers and there's so much to build. They get a little worried about making the product work. They want to focus in on making it really good.
The future of Cymbal is mapped out:
We have a strong vision for where we want to take this thing and how we want it to operate. We would really love it if discover was written into the experience of using the app. We want to make it easier to listen to and find songs. One part is us, one part is user feedback.
Obviously, one part of the puzzle is making Cymbal an appealing place for bands to live and share their work:
Since we launched, one thing has been bands reaching out about ways to promote their music. They don't want to hook a personal Facebook page up to it. They want to connect their band page. We only realized that was a serious concern once they started reaching out to us.
Check out our new featured user @NPRHipHop Follow them now! pic.twitter.com/bVZvwuRvxM
— Cymbal (@CymbalFM) August 4, 2015
That part does bum me out a little. The spread of Facebook login is obviously appealing to devs – it works well – but for people like my colleague Ben Woods who are not Facebook fans in general, the push to hook up with the social network in apps is an instant turn off.
What's next for Cymbal? The one thing I was jonesing for – embeds – is definitely coming:
Embeds is one of the biggest things we're dedicating design and development time too. We're building out a Web player. It'll bring Cymbal to a larger stage and it'll be an effective tool for artists and labels. It'll allow them to share easily to other social networks. Our Web player and new website will be game changers – I really believe that. They'll extend the experience to everywhere.
I love the way Kaplan and his colleagues think of Cymbal:
The app is designed to make you feel like you're walking into a record sore. I think that eventually, down the road, when we do 'monetize', we'll try and orientate ourselves a way of helping record labels, artists and media organizations promote their work and learn more about how they're performing.
Hoooooly shit! Just discovered @CymbalFM so I'm gonna need to leave work early & browse through music all wknd. This is all I've ever wanted
— Emma Milner (@emmilner) July 31, 2015
That's always the secret sauce that can make these services fly. You get to share your tastes and discover new things to love, labels and 'brands' (urgh, nuke them from orbit) get the kind of data that traditional surveys and sampling just don't generate.
Brands: Nuke them from orbit.
Bang the drum, scream into the microphone, Cymbal is the app I'm digging most right now. I predict you will like it too.
➤ Cymbal [iOS]
Read next: Pause literally puts the music into music magazines
Published August 10, 2015 - 12:57 pm UTC
Researcher discovered app malware on Google Play that steals your money
Don't assume your user data in the cloud is safe


text:
Python-based application to automate rune forging in Pox Nora.

Currently in very early development.

To try runesmith for yourself, check the [releases](https://github.com/discomethod/runesmith/releases).

### FAQ
##### Are you harvesting my username and password to steal my account?
No. Your username and password are sent directly to the Pox Nora servers and are not saved anywhere.
##### Will runesmith trade you all my runes for a Bronze token?
No. Runesmith does not do any trading via the Rune Trader. It only handles transactions with the Rune Forge, where you trade in runes for nora and can forge runes from nora.
##### What's the point of using runesmith when I can trade the runes in myself?
Runesmith is a lot faster and easier. If you have loads of commons and uncommons lying around in your inventory, it's annoying to click through the Sacrifice -> Confirm -> Wait -> Repeat thousands of times. Plus, runesmith will give you a quote of how much nora you can expect to make by trading in all your excess runes - without actually doing so.
##### Can I use runesmith to forge myself 2 of every common?
Not yet - however, that may come in the future.
##### How do I use runesmith?
The best way to run runesmith is to download a copy of the latest release (see above on how to download the latest release). Then, follow the instructions given on the release page.

Note that you will need to have Python 2.7 and all dependencies installed to run the current version. An executable version may be released in the future.

redpajama_set_name: RedPajamaGithub