text:
\section{Introduction}

Pre-trained Transformer architectures have led to huge progress in building more human-like language processing systems \citep[among others]{radford2019language,devlin-etal-2019-bert,brown2020language}. These models achieve impressive perplexity results on language modelling datasets, perform well on grammatical judgments \citep{warstadt-etal-2020-blimp-benchmark}, and provide useful linguistic representations that benefit a wide range of downstream tasks. Probing analyses also suggest that these models learn to implicitly encode syntactic information \citep{hewitt-manning-2019-structural,clark-etal-2019-bert} that may support better linguistic generalization than recurrent neural network architectures (RNNs).

However, the Transformer architecture \cite{NIPS2017_3f5ee243} is an interesting subject of study beyond its success in transfer-learning settings. Transformer models lack the inductive biases of RNNs. Rather than maintaining vector-valued state and updating it in a recurrent manner, auto-regressive Transformer models encode all past decisions simultaneously at each inference step, thanks to a self-attention mechanism. The only notion of sequence order is also given by position embeddings summed to content embeddings in both input and auto-regressive signals.

Previous works have shown the advantage of structural supervision in RNNs in learning to maintain syntactic states and non-local dependencies \citep{kuncoro-etal-2018-lstms,wilcox-etal-2019-structural,futrell-etal-2019-neural}. It remains an open question whether Transformer language models can similarly benefit from generative structural supervision, and what form of structural supervision would more effectively induce human-like syntactic generalization.

This work hypothesizes that the Transformer language model may benefit from explicit generative structural supervision to systematically generalize syntactic knowledge. Here we explore two major classes of structural guidance for Transformer language models based on joint modeling of language and constituency parses. The ``generative parsing as language modeling'' approach builds a Transformer-parameterized model to learn to predict actions that incrementally build constituency trees along with terminal words, following prior work on RNNs~\citep{dyer-etal-2016-recurrent,choe-charniak-2016-parsing}. The ``structural scaffolding'' approach follows the general idea of regularizing hidden representation through multi-task learning objective, with prior success in various NLP tasks \cite{zhang-weiss-2016-stack,sogaard-goldberg-2016-deep,swayamdipta-etal-2018-syntactic}.

We test these two approaches on two subsets of the BLLIP dataset \citep{charniak2000bllip} and evaluate models' syntactic generalization performances on SG Test Suites \citep{hu-etal-2020-systematic} and a sampled subset of the BLiMP Benchmark \citep{warstadt-etal-2020-blimp-benchmark}. We show evidence that generative structural supervision indeed induces more robust and human-like linguistic generalization in Transformer language models and explore the different trade-offs involved in the presented methods.

\section{Experiments}

\subsection{Model Training}

All models, including the baseline vanilla language models (\textbf{LM} in short), the syntactic scaffold models, and the generative parsing models, are based on the same architecture of GPT-2 small \citep{radford2019language} (117M parameters, 12 layers, $H=768$) and use the same BPE tokenizer, but with randomly initialized weights. We believe this would give us a fair comparison to pretrained GPT-2 as well, in order to evaluate whether structural guidance helps improve sample efficiency. We implemented all the proposed models using Huggingface's Transformer package \citep{wolf-etal-2020-transformers}\footnotemark\footnotetext{Code available at \url{https://github.com/IBM/transformers-struct-guidance}}.

As our goal here is to study whether structural guidance helps models learn robust humanlike generalization of syntactic knowledge, we train our model on the BLLIP dataset \citep{charniak2000bllip}, an English newswire style corpus used in \citet{hu-etal-2020-systematic}. This makes the results here more comparable to the results reported in previous work, especially with RNNGs. We train the proposed models and the baseline vanilla Transformer language models on \textsc{BLLIP-md}\xspace, a 14 million-token corpus, and \textsc{BLLIP-lg}\xspace, a 46 million-token corpus, both of which are auto-parsed using a state-of-the-art constituency parser \citep{Kitaev-2018-SelfAttentive}. We used the parsed sentences to generate oracle parsing action sequence for PLM and PLM-mask. We collected a list of word-synchronous parsing action sequences from the train and development oracle of \textsc{BLLIP-lg}\xspace and use it to parametrize the action $n$-gram vocabulary of ScLMs trained on both \textsc{BLLIP-md}\xspace and \textsc{BLLIP-lg}\xspace. There are 3756 action $n$-gram types from the corpora, including one padding token and one blank token.

All models were trained with learning rate $10^{-5}$, AdamW optimizer, and minibatch of size 5. We trained the models with multiple seeds within the capacity of our resources, in order to accommodate potential variance. In total, there are three seeds of LM, four of ScLM-past, four of ScLM-next, three of PLM, and three of PLM-mask for \textsc{BLLIP-md}\xspace, and the same number of seeds of each model type for \textsc{BLLIP-lg}\xspace. Models were trained until convergence, as suggested by the loss of the development set during training.

\subsection{Targeted Syntactic Evaluation}

To assess whether a trained model systematically generalizes its syntactic knowledge, we employ targeted syntactic evaluation paradigm \citep{marvin-linzen-2018-targeted}. Specifically, we measure models' performance on two held-out test datasets, a collection of syntactic generalization test suites from \citet{hu-etal-2020-systematic} and BLiMP Benchmark from \citet{warstadt-etal-2020-blimp-benchmark}. These two datasets cover a wide range of English syntactic phenomena.

Tests from \citet{hu-etal-2020-systematic}, which we refer as \textbf{SG Test Suites}, consist of hand-designed test suites for evaluating fine-grained syntactic generalization in incremental processing of a linguistic input. The general method is to compare models' surprisals $p(\textrm{continuation}|\textrm{prefix})$ of grammatical and ungrammatical continuations given certain sentence prefixes. We report the accuracy averaged across SG test suites. BLiMP Benchmark features minimal pairs of a grammatical sentence $W$ and an ungrammatical counterpart $W^*$. To evaluate a model on these minimal pairs, one simply compares the likelihood of $W$ and $W^*$ assigned by the model.

As is implied by the evaluation methods, we need to marginalize out the structure variables for PLM or PLM-mask models in order to estimate the surprisal of a continuation, given a sentence prefix or the likelihood of a complete sentence. We follow similar setup as in \citet{futrell-etal-2019-neural,wilcox-etal-2019-structural} applying word-synchronous beam search \citep{stern-etal-2017-effective} to find a list $Y_k$ of $k$ incremental parses given a sentence prefix $w_{<t}$. We then sum the joint probability $p(w_{<t}, y_{<t})$ over the list of incremental parses given by the model to approximate the likelihood of $p(w_{<t})$. We set the parse beam size to 100, word-synchronous beam size $k$ as 10, and fast track size of 5. Since the search process can be computationally intensive, the large number of items in BLiMP benchmark poses a computational challenge. We therefore select the first 10\% out of the 1000 items in each of the 67 tests of BLiMP Benchmark. We report the accuracy over the 100 items and refer to this down-sized BLiMP Benchmark as \textbf{BLiMP-10\%}.

\begin{figure}[t]
    \centering
    \includegraphics[width=\linewidth]{fig/sg_comparison_by_corpus_with_rnng.pdf}

    \includegraphics[width=\linewidth]{fig/blimp_comparison_by_corpus.pdf}
    \caption{Comparing models' overall accuracy across test suites from SG Test Suites (top) and BLiMP-10\% (bottom). RNNG performances are from \citet{hu-etal-2020-systematic}.
    }
    \label{fig:aggregated-generalization-comparison}
\end{figure}

We compare models' performance on the SG Test Suites and BLiMP-10\% in Figure \ref{fig:aggregated-generalization-comparison}. Each bar shows a model's performance averaged across multiple seeds on a given benchmark, with each dot plotting the accuracy of a specific seed. Overall, syntactic generalization performance improves as the training data size increases from \textsc{BLLIP-md}\xspace  (14 million tokens) to \textsc{BLLIP-lg}\xspace (42 million tokens). Models with structural guidance achieve higher accuracy than the vanilla Transformer language model trained on the same set of raw text data without explicit structural information. We also include the results for the RNNGs taken from \citet{hu-etal-2020-systematic}. RNNG lags behind all Transformer models by a large margin in average scores. We also notice that among different forms of structural guidance, generative parsing as language modeling is the most effective in improving syntactic generalization performance against the baseline transformer language models. We didn't observe consistent benefits of adding dynamic masking mechanism to PLM. While scaffolding approach slightly improves vanilla Transformer language models, it still falls behind the best performance of the model trained with generative parsing. We hypothesize that our scaffold did not fully exploit the compositional structure in the local parses by modelling each action $n$-gram as a distinct type, while the generative parsing models only predict actions in a relatively small set of non-terminal action space, which might make it easier for PLM and PLM-mask to learn compositional generalization. We leave it for future work to design new scaffolds that can take advantage of the combinatorial nature of syntactic structure.

For completeness, we also ran the pre-trained GPT-2 model on the syntactic suites. This yielded a score of 0.808 on the SG Test Suites and 0.827 on BLiMP-10\% for the small version of pre-trained GPT-2. Among models trained on \textsc{BLLIP-lg}\xspace, the average accuracy score on the SG Test Suites is 0.723 for PLMs, 0.748 for PLM-masks, and 0.665 for LMs. Similar trend is observed on BLiMP-10\% as well, where among models trained on \textsc{BLLIP-lg}\xspace the average accuracy is 0.751 for PLMs, 0.753 for PLM-masks, and 0.708 for LMs.
The proposed PLM method is able to close the gap between GPT-2 small and the same model trained with \textsc{BLLIP-lg}\xspace by about half, while the improvement for BLiMP is more modest but still significative. It remains an open question whether scaling syntactic supervision to a larger dataset than \textsc{BLLIP-lg}\xspace would bring the generalization performance of PLM models closer to that of the pretrained GPT-2 model.

\begin{figure}[t]
    \centering
    \includegraphics[width=\linewidth]{fig/ppl_sg.pdf}

    \includegraphics[width=\linewidth]{fig/ppl_blimp.pdf}
    \caption{Comparison between model perplexity on BLLIP test data and syntactic generalization performance on SG Test Suites (top) and BLiMP-10\% (bottom).}
    \label{fig:ppl-generalization-comparison}
\end{figure}

\subsubsection{Relationship between Perplexity and Syntactic Generalization Performance}

We compare perplexity on the BLLIP held-out test set against syntactic generalization performance in Figure \ref{fig:ppl-generalization-comparison}. Perplexities of PLM and PLM-mask models are computed setting the parse tree equal to the gold parse in Equation~\ref{eq:marg} to approximate the likelihood. Note that, unlike \citet{hu-etal-2020-systematic}, all our models use the same BPE vocabulary and word tokenization from GPT-2. The only exception are the additional parsing actions in the vocabulary $y$.

From Figure \ref{fig:ppl-generalization-comparison}, both perplexity and syntactic generalization performance improve with dataset size. However, for both training dataset sizes, we see that structural guidance can improve syntactic generalization. PLM models consistently perform better than vanilla models.
While all models achieve very similar perplexity results after being trained on a specific dataset, their syntactic generalization performances differ dramatically.

\begin{figure*}[!t]
    \centering
    \includegraphics[width=\linewidth]{fig/sg_comparison_by_test_suite_cluster_2_by_4_with_rnng.pdf}

    \vspace{0.25cm}
    \includegraphics[width=\linewidth]{fig/blimp_comparison_by_test_suite_cluster_3_by_4.pdf}
    \caption{Model performance comparison by specific linguistic phenomena clustered in SG Test Suites (top) and BLiMP-10\% (bottom). RNNG performances are from \citet{hu-etal-2020-systematic}.}
    \label{fig:comparison-by-phenomenon}
\end{figure*}

\subsubsection{Effect of Structural Guidance on Learning Specific Syntactic Structures}

In addition to comparing model's aggregated performances, we also compare their generalization performances in the clustered subsets of tests in SG Test Suites and BLiMP-10\%. These subsets consist of several related tests that target specific type of syntactic phenomenon, such as NPI licensing, subject-verb agreement, filler-gap dependencies, etc. We also include the results for the RNNGs taken from \citet{hu-etal-2020-systematic}.

Results in Figure \ref{fig:comparison-by-phenomenon} show converging evidence that structural guidance in the form of generative parsing can robustly improve learning of subject-verb agreement and NPI licensing, and helps the model to better capture incremental processing phenomenon such as garden-path effects, but seems to slightly hurt the performance on gross syntactic state. While overall the RNNG shows a poor performance this is mostly due to its very low scores for licensing suites. Excluding these suites only the RNNG shows a performance close to the PLM model, even outperforming it clearly for the gross syntactic state suites. In this category and binding PLM variants seem inferior to all other models.

\input{related}

\section{Conclusion}

Our work explores two forms of syntactic supervision as structural guidance for Transformer language models. Experiments suggest that generative parsing approach can effectively improve systematic generalization of learned syntactic knowledge in small training data regime, while a naive syntactic scaffold approach does not improve the baseline to the same extent despite reduced computation cost at inference time. Future work may explore alternative structural guidance strategies that combine the best of both approaches.

\section*{Acknowledgments}

The authors would like to thank the anonymous reviewers for their helpful comments. This work was supported by the MIT-IBM Watson AI Lab.

\bibliographystyle{acl_natbib}

\section{Models}

Here we explore joint modelling of structures and words parametrized with Transformers by considering both a sentence $W$ and its constituency parse $Y$ and modeling the joint distribution $P(W,Y)$.

\subsection{Generative Parsing as Language Modeling}

A language model can be described formally as a probability distribution over strings of a language $w_1, \cdots, w_T$, usually left-to-right factored.
\begin{equation}
p(W) = p(w_1, \cdots, w_T) = \prod_{t=1}^T p(w_t \mid w_{<t})
\end{equation}
There are many possible approaches that can combine both language modeling and syntax modeling tasks. As long as both tasks share some of the parameters they can be considered a case of multi-task learning \citep{caruana1997multitask}. Of interest here is the model proposed in Recurrent Neural Network Grammars (RNNGs; \citealp{dyer-etal-2016-recurrent}) and parsing as language model (LSTM-LM; \citealp{choe-charniak-2016-parsing}). Both approaches model the joint distribution of words $W$ and constituency tree components $Y$ as
\begin{equation}
p(Y, W) = p(a_1, \cdots, a_R) = \prod_{t=1}^R p(a_t \mid a_{<t})
\label{eq:palm}
\end{equation}
where $a_t$ are transitions of a state machine that generates both the sentence and the tree. These transitions are similar to the well-established transition sets used for transition-based parsing \citep{earley1970efficient} but adapted to generate both text and parse simultaneously. For the reminder of this work, we will consider each $a_t$ to be integer valued and indexing a dictionary of transitions. A transition $a$ can be a word $w$ or a transition action that generates a component of the constituency tree $y$. The actions include non-terminal symbols that open and label a new constituent with the label $x$, indicated as \textrm{NT(x)}, or a \textrm{REDUCE} action closing the closest open constituent. An example of a partial parse tree and transitions can be found at the top of Figure~\ref{fig:models}.

RNNG and LSTM-LM parametrize the same factorization in Equation~\ref{eq:palm} in different ways. RNNG utilizes stack-LSTMs, which allow it to dynamically create representations for partial tree components by composition. The LSTM-LM, however, uses a flat parametrization treating the transitions as a sequence in a conventional language model learnt with an LSTM \citep{hochreiter1997long}. It should also be noted that the LSTM-LM is designed as a parser, while RNNG is also used as a language model. In order to derive a language model from a joint model, it is is necessary to marginalize over all possible parse trees
\begin{equation}
p(W) = \sum_{Y \in \mathcal{Y}(W)} p(Y, W)
\label{eq:marg}
\end{equation}
which is an intractable problem since there is an exponentially large number of possible trees. The original RNNG work \cite{dyer-etal-2016-recurrent} proposes an approximate solution based on importance sampling. In this work we use the word-synchronous beam search approximation introduced in \citet{stern-etal-2017-effective}.

The marginalized likelihood language model in Equation~\ref{eq:marg} is desirable because it makes no statistical independence assumption between language and syntax and shares all parameters across both tasks, with the exception of action specific embeddings. Particularly relevant for this work is the fact that both word and non-word transitions are predicted as language model output indiscriminately and are available at each prediction step through its history $a_{<t}$.

In this work we propose to parametrize Eq~\ref{eq:palm} with a Transformer language model \cite{NIPS2017_3f5ee243}. This is equivalent to the flat parametrization of the LSTM-LM but using a Transformer language model instead. Unlike LSTM-LM, which is a parsing model, we derive from it a language model by marginalization as in the RNNG. A Transformer language model can be succinctly described as a neural network of vertically stacked layers where the $m$-th layer is given by
\begin{equation}
    \begin{aligned}
    h^{m}_{<t} = \mathrm{FF}^m\left(O \cdot \begin{bmatrix}
    \mathrm{A}^m_1(h^{m-1}_{<t})\\
    \mathrm{A}^m_2(h^{m-1}_{<t})\\
    \cdots \\
    \mathrm{A}^m_N(h^{m-1}_{<t})\\
    \end{bmatrix}\right).
    \end{aligned}
    \label{eq:mha}
\end{equation}
Here $h^{m-1}_{<t} \in \mathbb{R}^{H \times t}$ is the output of the previous decoder layer for all previous predictions of the model at time step $t$ and $H$ is the size of the hidden vector. The input to the first layer i.e. $h^0_{<t}$ are the embeddings of all previous transitions $a_{<t}$ concatenated with a start symbol. Each embedding is the sum of both a content embedding, dictionary vector that is being indexed, and a position embedding that encodes the absolute or relative position of each action in the sequence.

$\mathrm{FF}^m()$ is a feed-forward layer, $A^m_1() \cdots A^M_N()$ are multiple self-attention heads and $O \in \mathbb{R}^{H \times H}$ is a matrix multiplication performed on the concatenated output of the attention heads. Both the feed-forward and the projection of $N$ attention heads through $O$ are wrapped around with residual, dropout and layer normalization operations that are here removed for clarity.

Each attention head comprises a simple inner product attention mechanism
\begin{eqnarray}
    \mathrm{A}^m_n(h^{m-1}_{<t}) = V_n^m \cdot h^{m-1}_{<t} \cdot&\nonumber\\  \mathrm{softmax}\big((K_n^m \cdot h^{m-1}_{<t})^T \!\!\!\!\! & \:\cdot Q_n^m \cdot h^{m-1}_{<t} +\mathcal{M}\big)
    \label{eq:ah2}
\end{eqnarray}
where $V_n^m, K_n^m, Q_n^m \in \mathbb{R}^{H/N \times H}$ are value, key and query projection matrices respectively and the \textrm{softmax} operation is normalized over columns to sum to one. The matrix $\mathcal{M} \in \{-\infty, 0\}^{t \times t}$ is used to prevent the model from attending to future states during training, enabling efficient parallelization. It is displayed here due to its relevance for the next section.

Similarly to other models, to derive a distribution over all possible transitions, including words, non-terminal symbols and the \textrm{REDUCE} operation, we can use a softmax together with an inner product
\begin{eqnarray}
	p(a_t \mid a_{<t}) = \mathrm{softmax}(E^{W \cup Y} \cdot h^{m}_{<t})_{a_t}

\end{eqnarray}
where $E^{W \cup Y}$ are the embeddings for the joint vocabulary of words, non-terminals and \textrm{REDUCE} transitions. Henceforth, we refer to this model as \textbf{Parsing as Language Model}, or \textbf{PLM} for short.

Unlike LSTMs or the RNNG, the Transformer has direct access to all past decisions through self-attention and relies on position embeddings to encode word order. Thus, in principle, there is no structural bias for the model to favor past decisions that are close in time to inform current prediction. On one hand, this potential ability to use long distance information can enable a less local, more human like processing of language, but on the other hand, it can also result in an additional learning burden, especially if there is not sufficient learning data available. Also worth noting for the experiments proposed here is that the total number of parameters of a typical Transformer greatly exceeds that of an LSTM or a RNNG model.

\subsection{Incorporating RNNG-like characteristics}

\input{figure2.tex}

As previously mentioned, unlike any of the other models, the RNNG is able to create partial tree representations by composition using stack-LSTMs. This changes the RNNG model structure dynamically as a function of the partial parse, a very desirable property to derive syntax-aware representations. Moreover, the fact that Recurrent Neural Networks such as LSTMs summarize all information about previous time steps on two hidden vectors, creates a bottleneck that forces the model to focus on the local state. This is a situation where a syntax-aware representation can provide additional value by enabling the local state to better encompass past structures. We conjecture that a similarly constrained local state might benefit Transformer models in learning linguistic regularities, especially in a limited training data scenario.

In an attempt to capture a similar effect in the Transformer, we explore here the idea of masking some attention heads to reflect the parser state as in the stack-Transformer \citep{astudillo2020transition}. In the stack-Transformer, two attention heads are specialized to attend only to the contents of buffer and stack respectively for dependency and semantic parsing tasks. Here we choose to specialize two heads as well for each layer in Equation~\ref{eq:mha}, as depicted in Fig.~\ref{fig:attention-mask}. One attention head attends to the contents of the last open constituent whereas another head attends all other past decisions not involving that constituent. The rest of the heads are left free as in the original Transformer architecture. To constrain the attention heads, we only need to alter the mask $\mathcal{M}$ in Equation~\ref{eq:ah2} to depend on head index $n$ and past actions $\mathcal{M}_n(a_{<t})$, which results in a negligible computation overhead.

This hard masking makes the model structure change dynamically depending on the partial parse and it forces some heads to focus on the local syntactic state. Nevertheless, unlike the RNNG, it does not create new representations of partial parses that can be composed in a recurrent manner at each time step, and some attention heads can still operate unrestricted. We hypothesize that structure-aware attention mechanism may still help the model achieve better generalization. The symbolic representation induces a strong inductive bias to how the model should use the structure that it generates on the fly. We henceforth refer to this model \textbf{PLM-mask}.

\subsection{Scaffolding by Learning to Predict Local Parse States}

Given the strong coupling between the tasks, the marginal likelihood Transformer language model of the previous section can be expected to be strongly influenced by the additional syntax prediction task. This comes however at a big cost. First, sequences combine both words and non-terminal and reduce transitions, yielding longer sentences than those of a normal language model $R > T$. Furthermore the approximated marginalization is computationally intensive and also introduces an approximation error.

One well-established regime that allows joint modeling of tasks at a low complexity is that of the syntactic scaffold \cite{zhang-weiss-2016-stack,sogaard-goldberg-2016-deep,swayamdipta-etal-2018-syntactic}. Scaffolding adds an additional structure prediction task at one of the layers of the model as a separate layer and only during training. This is a minimally intrusive change since it just branches some hidden vector of the network and computes an additional loss. It also has no influence on test runtime and avoids expensive steps such as marginalization.

However, applying the idea of syntactic scaffolding to our present scenario poses one difficulty. If we use a standard language model predicting words $w$ and predict the non-word symbols $y$ separately, we face the problem that the two sequences have different lengths. To overcome this in a straightforward way, we predict the $n$-gram of non-word actions $y_{t:t+n(t)}$ corresponding to the partial parse synchronous with step $t$ when we predict word $w_t$. We use a secondary softmax layer for this action $n$-gram prediction.
\begin{eqnarray}
	p(y_{t:t+n} \mid y_{<t}) = \mathrm{softmax}(E^{Y^*} \cdot h^{m}_{<t})_{y_{t:t+n}}
    \label{eq:out2}
\end{eqnarray}
Here $E^{Y^*}$ is the vocabulary of all transition $n$-grams excluding words found in the train corpus plus a blank symbol. Note that since Scaffolding operates only at train time, we do not need to worry about generalization of these $n$-grams to test time.

The models are thus trained to minimize the loss function $-\log p(Y,W)$ where
\begin{eqnarray}
p(Y, W) &= \prod_{t=1}^T p(w_t \mid w_{<t}) \nonumber\\
        &+ \prod_{t=1}^T p(y_{t:t+n(t)} \mid w_{<t})
\end{eqnarray}
The scaffold can be set so that the synchronous non-word action $n$-grams $y_{t:t+n(t)}$ are predicted either before (Figure~\ref{fig:models}c, left) or after (Figure~\ref{fig:models}c, right) producing $w_t$. We considered both variants in our experiments to empirically assess their impact on performance. We refer to this model as \textbf{Transformer Language Model with Syntactic Scaffold}, or \textbf{ScLM} in short, and its two versions \textbf{ScLM-past} and \textbf{ScLM-next}, for past and next $n$-gram prediction.

\section{Related Work}

Multitask learning \cite{caruana1997multitask} has been applied to a variety of NLP tasks with traditional modeling approaches \cite{miller-etal-2000-novel,sutton2005joint,sutton2007dynamic} as well as more recent neural models \cite{collobert2011natural,li2020transformer}. A recurring theme has been the use of structure in the form of syntactic trees to benefit other NLP tasks. Among the early works exploring this direction, \newcite{punyakanok-etal-2008-importance} showed that syntactic parses can benefit Semantic Role Labeling (SRL). \newcite{poon2009unsupervised} extended this idea to induce first-order logic representation in a unsupervised fashion, by clustering the dependency structures. In both cases syntax forms part of a pipeline and is not strictly supervision for the end task.

This trend continued with the rise of neural models. \newcite{collobert2011natural} improved deep convolution neural network for syntactic chunking models with additional POS supervision. \citet{zhang-weiss-2016-stack,sogaard-goldberg-2016-deep} observe the benefits of POS supervision at  different depths of a neural network model with impact on dependency parsing, tagging and CCG super tagging performance. \newcite{he2019syntax} perform a syntax-based pruning of semantic roles, showing benefits in a multilingual setting. More recently, \newcite{sachan2020syntax} incorporate a syntactic graph recurrent neural network into BERT models for better semantic role labeling. However, their method shows little or no benefit of syntax modeling for Named Entity Recognition and relation linking task. Neural machine translation \cite{chen2018syntax} and text generation \cite{li2020transformer} have also been shown to benefit from syntactic modeling. In a recent work, \newcite{li2020improving} use syntactic modeling in BERT based transformers to achieve performance gains on several text classification benchmarks. Other works have found that structural supervision in the form of intermediate fine-tuning (e.g., on CCG super tagging) is not helpful or even harmful \cite{pruksachatkun2020intermediate,warstadt2019investigating}.

The focus of our work is on gauging the impact of joint modeling on syntactic generalization performance. In this direction, the work of \citet{swayamdipta-etal-2018-syntactic} is close to the scaffolding version of our model. They predict multiple labels, extracted from syntactic information, as auxiliary task and show positive effects on shallow semantic parsing and co-reference resolution. We use however a single feature, constituency parsing $n$-gram, which is closer to prior work relying on Part-of-Speech information. In addition, we explore impact of using preceding structure as feature vs postceding structure, which as shown plays a role in the learning process.

In terms of modeling objective and syntactic representations, our method is closest to the works of \newcite{choe-charniak-2016-parsing,dyer-etal-2016-recurrent} that jointly model syntax and language. A more recent work from \citet{peng-etal-2019-palm} uses Rational Neural Networks language model that can derive binary unlabeled constituents from attention weights and can supervise the attention to attain a structural inductive bias. The proposed models show lower language modeling perplexity compared to their structure agnostic counterparts. We also extend here the idea of syntax-aware language modeling to transformer-based language models.

Finally, our approach relates to the other works that propose ways of incorporating structural information into Transformer-based models. This includes the use of dependency or tree structure for constraining self-attention patterns \citep{strubell2018linguistically,wang-etal-2019-tree,zhang2020sg}, guiding cross-attention \cite{chen2018syntax,astudillo2020transition}, modelling syntactic distance \citep{du-etal-2020-exploiting}, using syntactic information to guide the computation flow in the model \citep{shen-etal-2021-explicitly}, or through knowledge distillation \citep{kuncoro2020syntactic}. Our structured masking in parsing as language modeling approach is close in spirit to the methods that modify attention mechanism according to syntactic connections \cite{astudillo2020transition}; This work, however, primarily aims to study the impact of structural guidance on syntactic generalization. Therefore, we resort to simpler methods of incorporating structure to minimize the impact of modeling intricacies.
\section{Experiments}

\subsection{Model Training}

All models, including the baseline vanilla language models (\textbf{LM} in short), the syntactic scaffold models, and the generative parsing models, are based on the same architecture of GPT-2 small \citep{radford2019language} (117M parameters, 12 layers, $H=768$) and use the same BPE tokenizer, but with randomly initialized weights. We believe this would give us a fair comparison to pretrained GPT-2 as well, in order to evaluate whether structural guidance helps improve sample efficiency. We implemented all the proposed models using Huggingface's Transformer package \citep{wolf-etal-2020-transformers}\footnotemark\footnotetext{Code available at \url{https://github.com/IBM/transformers-struct-guidance}}.

As our goal here is to study whether structural guidance helps models learn robust humanlike generalization of syntactic knowledge, we train our model on the BLLIP dataset \citep{charniak2000bllip}, an English newswire style corpus used in \citet{hu-etal-2020-systematic}. This makes the results here more comparable to the results reported in previous work, especially with RNNGs. We train the proposed models and the baseline vanilla Transformer language models on \textsc{BLLIP-md}\xspace, a 14 million-token corpus, and \textsc{BLLIP-lg}\xspace, a 46 million-token corpus, both of which are auto-parsed using a state-of-the-art constituency parser \citep{Kitaev-2018-SelfAttentive}. We used the parsed sentences to generate oracle parsing action sequence for PLM and PLM-mask. We collected a list of word-synchronous parsing action sequences from the train and development oracle of \textsc{BLLIP-lg}\xspace and use it to parametrize the action $n$-gram vocabulary of ScLMs trained on both \textsc{BLLIP-md}\xspace and \textsc{BLLIP-lg}\xspace. There are 3756 action $n$-gram types from the corpora, including one padding token and one blank token.

All models were trained with learning rate $10^{-5}$, AdamW optimizer, and minibatch of size 5. We trained the models with multiple seeds within the capacity of our resources, in order to accommodate potential variance. In total, there are three seeds of LM, four of ScLM-past, four of ScLM-next, three of PLM, and three of PLM-mask for \textsc{BLLIP-md}\xspace, and the same number of seeds of each model type for \textsc{BLLIP-lg}\xspace. Models were trained until convergence, as suggested by the loss of the development set during training.

\subsection{Targeted Syntactic Evaluation}

To assess whether a trained model systematically generalizes its syntactic knowledge, we employ targeted syntactic evaluation paradigm \citep{marvin-linzen-2018-targeted}. Specifically, we measure models' performance on two held-out test datasets, a collection of syntactic generalization test suites from \citet{hu-etal-2020-systematic} and BLiMP Benchmark from \citet{warstadt-etal-2020-blimp-benchmark}. These two datasets cover a wide range of English syntactic phenomena.

Tests from \citet{hu-etal-2020-systematic}, which we refer as \textbf{SG Test Suites}, consist of hand-designed test suites for evaluating fine-grained syntactic generalization in incremental processing of a linguistic input. The general method is to compare models' surprisals $p(\textrm{continuation}|\textrm{prefix})$ of grammatical and ungrammatical continuations given certain sentence prefixes. We report the accuracy averaged across SG test suites. BLiMP Benchmark features minimal pairs of a grammatical sentence $W$ and an ungrammatical counterpart $W^*$. To evaluate a model on these minimal pairs, one simply compares the likelihood of $W$ and $W^*$ assigned by the model.

As is implied by the evaluation methods, we need to marginalize out the structure variables for PLM or PLM-mask models in order to estimate the surprisal of a continuation, given a sentence prefix or the likelihood of a complete sentence. We follow similar setup as in \citet{futrell-etal-2019-neural,wilcox-etal-2019-structural} applying word-synchronous beam search \citep{stern-etal-2017-effective} to find a list $Y_k$ of $k$ incremental parses given a sentence prefix $w_{<t}$. We then sum the joint probability $p(w_{<t}, y_{<t})$ over the list of incremental parses given by the model to approximate the likelihood of $p(w_{<t})$. We set the parse beam size to 100, word-synchronous beam size $k$ as 10, and fast track size of 5. Since the search process can be computationally intensive, the large number of items in BLiMP benchmark poses a computational challenge. We therefore select the first 10\% out of the 1000 items in each of the 67 tests of BLiMP Benchmark. We report the accuracy over the 100 items and refer to this down-sized BLiMP Benchmark as \textbf{BLiMP-10\%}.

\begin{figure}[t]
    \centering
    \includegraphics[width=\linewidth]{fig/sg_comparison_by_corpus_with_rnng.pdf}

    \includegraphics[width=\linewidth]{fig/blimp_comparison_by_corpus.pdf}
    \caption{Comparing models' overall accuracy across test suites from SG Test Suites (top) and BLiMP-10\% (bottom). RNNG performances are from \citet{hu-etal-2020-systematic}.
    }
    \label{fig:aggregated-generalization-comparison}
\end{figure}

We compare models' performance on the SG Test Suites and BLiMP-10\% in Figure \ref{fig:aggregated-generalization-comparison}. Each bar shows a model's performance averaged across multiple seeds on a given benchmark, with each dot plotting the accuracy of a specific seed. Overall, syntactic generalization performance improves as the training data size increases from \textsc{BLLIP-md}\xspace  (14 million tokens) to \textsc{BLLIP-lg}\xspace (42 million tokens). Models with structural guidance achieve higher accuracy than the vanilla Transformer language model trained on the same set of raw text data without explicit structural information. We also include the results for the RNNGs taken from \citet{hu-etal-2020-systematic}. RNNG lags behind all Transformer models by a large margin in average scores. We also notice that among different forms of structural guidance, generative parsing as language modeling is the most effective in improving syntactic generalization performance against the baseline transformer language models. We didn't observe consistent benefits of adding dynamic masking mechanism to PLM. While scaffolding approach slightly improves vanilla Transformer language models, it still falls behind the best performance of the model trained with generative parsing. We hypothesize that our scaffold did not fully exploit the compositional structure in the local parses by modelling each action $n$-gram as a distinct type, while the generative parsing models only predict actions in a relatively small set of non-terminal action space, which might make it easier for PLM and PLM-mask to learn compositional generalization. We leave it for future work to design new scaffolds that can take advantage of the combinatorial nature of syntactic structure.

For completeness, we also ran the pre-trained GPT-2 model on the syntactic suites. This yielded a score of 0.808 on the SG Test Suites and 0.827 on BLiMP-10\% for the small version of pre-trained GPT-2. Among models trained on \textsc{BLLIP-lg}\xspace, the average accuracy score on the SG Test Suites is 0.723 for PLMs, 0.748 for PLM-masks, and 0.665 for LMs. Similar trend is observed on BLiMP-10\% as well, where among models trained on \textsc{BLLIP-lg}\xspace the average accuracy is 0.751 for PLMs, 0.753 for PLM-masks, and 0.708 for LMs.
The proposed PLM method is able to close the gap between GPT-2 small and the same model trained with \textsc{BLLIP-lg}\xspace by about half, while the improvement for BLiMP is more modest but still significative. It remains an open question whether scaling syntactic supervision to a larger dataset than \textsc{BLLIP-lg}\xspace would bring the generalization performance of PLM models closer to that of the pretrained GPT-2 model.

\begin{figure}[t]
    \centering
    \includegraphics[width=\linewidth]{fig/ppl_sg.pdf}

    \includegraphics[width=\linewidth]{fig/ppl_blimp.pdf}
    \caption{Comparison between model perplexity on BLLIP test data and syntactic generalization performance on SG Test Suites (top) and BLiMP-10\% (bottom).}
    \label{fig:ppl-generalization-comparison}
\end{figure}

\subsubsection{Relationship between Perplexity and Syntactic Generalization Performance}

We compare perplexity on the BLLIP held-out test set against syntactic generalization performance in Figure \ref{fig:ppl-generalization-comparison}. Perplexities of PLM and PLM-mask models are computed setting the parse tree equal to the gold parse in Equation~\ref{eq:marg} to approximate the likelihood. Note that, unlike \citet{hu-etal-2020-systematic}, all our models use the same BPE vocabulary and word tokenization from GPT-2. The only exception are the additional parsing actions in the vocabulary $y$.

From Figure \ref{fig:ppl-generalization-comparison}, both perplexity and syntactic generalization performance improve with dataset size. However, for both training dataset sizes, we see that structural guidance can improve syntactic generalization. PLM models consistently perform better than vanilla models.
While all models achieve very similar perplexity results after being trained on a specific dataset, their syntactic generalization performances differ dramatically.

\begin{figure*}[!t]
    \centering
    \includegraphics[width=\linewidth]{fig/sg_comparison_by_test_suite_cluster_2_by_4_with_rnng.pdf}

    \vspace{0.25cm}
    \includegraphics[width=\linewidth]{fig/blimp_comparison_by_test_suite_cluster_3_by_4.pdf}
    \caption{Model performance comparison by specific linguistic phenomena clustered in SG Test Suites (top) and BLiMP-10\% (bottom). RNNG performances are from \citet{hu-etal-2020-systematic}.}
    \label{fig:comparison-by-phenomenon}
\end{figure*}

\subsubsection{Effect of Structural Guidance on Learning Specific Syntactic Structures}

In addition to comparing model's aggregated performances, we also compare their generalization performances in the clustered subsets of tests in SG Test Suites and BLiMP-10\%. These subsets consist of several related tests that target specific type of syntactic phenomenon, such as NPI licensing, subject-verb agreement, filler-gap dependencies, etc. We also include the results for the RNNGs taken from \citet{hu-etal-2020-systematic}.

Results in Figure \ref{fig:comparison-by-phenomenon} show converging evidence that structural guidance in the form of generative parsing can robustly improve learning of subject-verb agreement and NPI licensing, and helps the model to better capture incremental processing phenomenon such as garden-path effects, but seems to slightly hurt the performance on gross syntactic state. While overall the RNNG shows a poor performance this is mostly due to its very low scores for licensing suites. Excluding these suites only the RNNG shows a performance close to the PLM model, even outperforming it clearly for the gross syntactic state suites. In this category and binding PLM variants seem inferior to all other models.

\input{related}

\section{Conclusion}

Our work explores two forms of syntactic supervision as structural guidance for Transformer language models. Experiments suggest that generative parsing approach can effectively improve systematic generalization of learned syntactic knowledge in small training data regime, while a naive syntactic scaffold approach does not improve the baseline to the same extent despite reduced computation cost at inference time. Future work may explore alternative structural guidance strategies that combine the best of both approaches.

\section*{Acknowledgments}

The authors would like to thank the anonymous reviewers for their helpful comments. This work was supported by the MIT-IBM Watson AI Lab.

\bibliographystyle{acl_natbib}

\section{Introduction}

Pre-trained Transformer architectures have led to huge progress in building more human-like language processing systems \citep[among others]{radford2019language,devlin-etal-2019-bert,brown2020language}. These models achieve impressive perplexity results on language modelling datasets, perform well on grammatical judgments \citep{warstadt-etal-2020-blimp-benchmark}, and provide useful linguistic representations that benefit a wide range of downstream tasks. Probing analyses also suggest that these models learn to implicitly encode syntactic information \citep{hewitt-manning-2019-structural,clark-etal-2019-bert} that may support better linguistic generalization than recurrent neural network architectures (RNNs).

However, the Transformer architecture \cite{NIPS2017_3f5ee243} is an interesting subject of study beyond its success in transfer-learning settings. Transformer models lack the inductive biases of RNNs. Rather than maintaining vector-valued state and updating it in a recurrent manner, auto-regressive Transformer models encode all past decisions simultaneously at each inference step, thanks to a self-attention mechanism. The only notion of sequence order is also given by position embeddings summed to content embeddings in both input and auto-regressive signals.

Previous works have shown the advantage of structural supervision in RNNs in learning to maintain syntactic states and non-local dependencies \citep{kuncoro-etal-2018-lstms,wilcox-etal-2019-structural,futrell-etal-2019-neural}. It remains an open question whether Transformer language models can similarly benefit from generative structural supervision, and what form of structural supervision would more effectively induce human-like syntactic generalization.

This work hypothesizes that the Transformer language model may benefit from explicit generative structural supervision to systematically generalize syntactic knowledge. Here we explore two major classes of structural guidance for Transformer language models based on joint modeling of language and constituency parses. The ``generative parsing as language modeling'' approach builds a Transformer-parameterized model to learn to predict actions that incrementally build constituency trees along with terminal words, following prior work on RNNs~\citep{dyer-etal-2016-recurrent,choe-charniak-2016-parsing}. The ``structural scaffolding'' approach follows the general idea of regularizing hidden representation through multi-task learning objective, with prior success in various NLP tasks \cite{zhang-weiss-2016-stack,sogaard-goldberg-2016-deep,swayamdipta-etal-2018-syntactic}.

We test these two approaches on two subsets of the BLLIP dataset \citep{charniak2000bllip} and evaluate models' syntactic generalization performances on SG Test Suites \citep{hu-etal-2020-systematic} and a sampled subset of the BLiMP Benchmark \citep{warstadt-etal-2020-blimp-benchmark}. We show evidence that generative structural supervision indeed induces more robust and human-like linguistic generalization in Transformer language models and explore the different trade-offs involved in the presented methods.

\section{Models}

Here we explore joint modelling of structures and words parametrized with Transformers by considering both a sentence $W$ and its constituency parse $Y$ and modeling the joint distribution $P(W,Y)$.

\subsection{Generative Parsing as Language Modeling}

A language model can be described formally as a probability distribution over strings of a language $w_1, \cdots, w_T$, usually left-to-right factored.
\begin{equation}
p(W) = p(w_1, \cdots, w_T) = \prod_{t=1}^T p(w_t \mid w_{<t})
\end{equation}
There are many possible approaches that can combine both language modeling and syntax modeling tasks. As long as both tasks share some of the parameters they can be considered a case of multi-task learning \citep{caruana1997multitask}. Of interest here is the model proposed in Recurrent Neural Network Grammars (RNNGs; \citealp{dyer-etal-2016-recurrent}) and parsing as language model (LSTM-LM; \citealp{choe-charniak-2016-parsing}). Both approaches model the joint distribution of words $W$ and constituency tree components $Y$ as
\begin{equation}
p(Y, W) = p(a_1, \cdots, a_R) = \prod_{t=1}^R p(a_t \mid a_{<t})
\label{eq:palm}
\end{equation}
where $a_t$ are transitions of a state machine that generates both the sentence and the tree. These transitions are similar to the well-established transition sets used for transition-based parsing \citep{earley1970efficient} but adapted to generate both text and parse simultaneously. For the reminder of this work, we will consider each $a_t$ to be integer valued and indexing a dictionary of transitions. A transition $a$ can be a word $w$ or a transition action that generates a component of the constituency tree $y$. The actions include non-terminal symbols that open and label a new constituent with the label $x$, indicated as \textrm{NT(x)}, or a \textrm{REDUCE} action closing the closest open constituent. An example of a partial parse tree and transitions can be found at the top of Figure~\ref{fig:models}.

RNNG and LSTM-LM parametrize the same factorization in Equation~\ref{eq:palm} in different ways. RNNG utilizes stack-LSTMs, which allow it to dynamically create representations for partial tree components by composition. The LSTM-LM, however, uses a flat parametrization treating the transitions as a sequence in a conventional language model learnt with an LSTM \citep{hochreiter1997long}. It should also be noted that the LSTM-LM is designed as a parser, while RNNG is also used as a language model. In order to derive a language model from a joint model, it is is necessary to marginalize over all possible parse trees
\begin{equation}
p(W) = \sum_{Y \in \mathcal{Y}(W)} p(Y, W)
\label{eq:marg}
\end{equation}
which is an intractable problem since there is an exponentially large number of possible trees. The original RNNG work \cite{dyer-etal-2016-recurrent} proposes an approximate solution based on importance sampling. In this work we use the word-synchronous beam search approximation introduced in \citet{stern-etal-2017-effective}.

The marginalized likelihood language model in Equation~\ref{eq:marg} is desirable because it makes no statistical independence assumption between language and syntax and shares all parameters across both tasks, with the exception of action specific embeddings. Particularly relevant for this work is the fact that both word and non-word transitions are predicted as language model output indiscriminately and are available at each prediction step through its history $a_{<t}$.

In this work we propose to parametrize Eq~\ref{eq:palm} with a Transformer language model \cite{NIPS2017_3f5ee243}. This is equivalent to the flat parametrization of the LSTM-LM but using a Transformer language model instead. Unlike LSTM-LM, which is a parsing model, we derive from it a language model by marginalization as in the RNNG. A Transformer language model can be succinctly described as a neural network of vertically stacked layers where the $m$-th layer is given by
\begin{equation}
    \begin{aligned}
    h^{m}_{<t} = \mathrm{FF}^m\left(O \cdot \begin{bmatrix}
    \mathrm{A}^m_1(h^{m-1}_{<t})\\
    \mathrm{A}^m_2(h^{m-1}_{<t})\\
    \cdots \\
    \mathrm{A}^m_N(h^{m-1}_{<t})\\
    \end{bmatrix}\right).
    \end{aligned}
    \label{eq:mha}
\end{equation}
Here $h^{m-1}_{<t} \in \mathbb{R}^{H \times t}$ is the output of the previous decoder layer for all previous predictions of the model at time step $t$ and $H$ is the size of the hidden vector. The input to the first layer i.e. $h^0_{<t}$ are the embeddings of all previous transitions $a_{<t}$ concatenated with a start symbol. Each embedding is the sum of both a content embedding, dictionary vector that is being indexed, and a position embedding that encodes the absolute or relative position of each action in the sequence.

$\mathrm{FF}^m()$ is a feed-forward layer, $A^m_1() \cdots A^M_N()$ are multiple self-attention heads and $O \in \mathbb{R}^{H \times H}$ is a matrix multiplication performed on the concatenated output of the attention heads. Both the feed-forward and the projection of $N$ attention heads through $O$ are wrapped around with residual, dropout and layer normalization operations that are here removed for clarity.

Each attention head comprises a simple inner product attention mechanism
\begin{eqnarray}
    \mathrm{A}^m_n(h^{m-1}_{<t}) = V_n^m \cdot h^{m-1}_{<t} \cdot&\nonumber\\  \mathrm{softmax}\big((K_n^m \cdot h^{m-1}_{<t})^T \!\!\!\!\! & \:\cdot Q_n^m \cdot h^{m-1}_{<t} +\mathcal{M}\big)
    \label{eq:ah2}
\end{eqnarray}
where $V_n^m, K_n^m, Q_n^m \in \mathbb{R}^{H/N \times H}$ are value, key and query projection matrices respectively and the \textrm{softmax} operation is normalized over columns to sum to one. The matrix $\mathcal{M} \in \{-\infty, 0\}^{t \times t}$ is used to prevent the model from attending to future states during training, enabling efficient parallelization. It is displayed here due to its relevance for the next section.

Similarly to other models, to derive a distribution over all possible transitions, including words, non-terminal symbols and the \textrm{REDUCE} operation, we can use a softmax together with an inner product
\begin{eqnarray}
	p(a_t \mid a_{<t}) = \mathrm{softmax}(E^{W \cup Y} \cdot h^{m}_{<t})_{a_t}

\end{eqnarray}
where $E^{W \cup Y}$ are the embeddings for the joint vocabulary of words, non-terminals and \textrm{REDUCE} transitions. Henceforth, we refer to this model as \textbf{Parsing as Language Model}, or \textbf{PLM} for short.

Unlike LSTMs or the RNNG, the Transformer has direct access to all past decisions through self-attention and relies on position embeddings to encode word order. Thus, in principle, there is no structural bias for the model to favor past decisions that are close in time to inform current prediction. On one hand, this potential ability to use long distance information can enable a less local, more human like processing of language, but on the other hand, it can also result in an additional learning burden, especially if there is not sufficient learning data available. Also worth noting for the experiments proposed here is that the total number of parameters of a typical Transformer greatly exceeds that of an LSTM or a RNNG model.

\subsection{Incorporating RNNG-like characteristics}

\input{figure2.tex}

As previously mentioned, unlike any of the other models, the RNNG is able to create partial tree representations by composition using stack-LSTMs. This changes the RNNG model structure dynamically as a function of the partial parse, a very desirable property to derive syntax-aware representations. Moreover, the fact that Recurrent Neural Networks such as LSTMs summarize all information about previous time steps on two hidden vectors, creates a bottleneck that forces the model to focus on the local state. This is a situation where a syntax-aware representation can provide additional value by enabling the local state to better encompass past structures. We conjecture that a similarly constrained local state might benefit Transformer models in learning linguistic regularities, especially in a limited training data scenario.

In an attempt to capture a similar effect in the Transformer, we explore here the idea of masking some attention heads to reflect the parser state as in the stack-Transformer \citep{astudillo2020transition}. In the stack-Transformer, two attention heads are specialized to attend only to the contents of buffer and stack respectively for dependency and semantic parsing tasks. Here we choose to specialize two heads as well for each layer in Equation~\ref{eq:mha}, as depicted in Fig.~\ref{fig:attention-mask}. One attention head attends to the contents of the last open constituent whereas another head attends all other past decisions not involving that constituent. The rest of the heads are left free as in the original Transformer architecture. To constrain the attention heads, we only need to alter the mask $\mathcal{M}$ in Equation~\ref{eq:ah2} to depend on head index $n$ and past actions $\mathcal{M}_n(a_{<t})$, which results in a negligible computation overhead.

This hard masking makes the model structure change dynamically depending on the partial parse and it forces some heads to focus on the local syntactic state. Nevertheless, unlike the RNNG, it does not create new representations of partial parses that can be composed in a recurrent manner at each time step, and some attention heads can still operate unrestricted. We hypothesize that structure-aware attention mechanism may still help the model achieve better generalization. The symbolic representation induces a strong inductive bias to how the model should use the structure that it generates on the fly. We henceforth refer to this model \textbf{PLM-mask}.

\subsection{Scaffolding by Learning to Predict Local Parse States}

Given the strong coupling between the tasks, the marginal likelihood Transformer language model of the previous section can be expected to be strongly influenced by the additional syntax prediction task. This comes however at a big cost. First, sequences combine both words and non-terminal and reduce transitions, yielding longer sentences than those of a normal language model $R > T$. Furthermore the approximated marginalization is computationally intensive and also introduces an approximation error.

One well-established regime that allows joint modeling of tasks at a low complexity is that of the syntactic scaffold \cite{zhang-weiss-2016-stack,sogaard-goldberg-2016-deep,swayamdipta-etal-2018-syntactic}. Scaffolding adds an additional structure prediction task at one of the layers of the model as a separate layer and only during training. This is a minimally intrusive change since it just branches some hidden vector of the network and computes an additional loss. It also has no influence on test runtime and avoids expensive steps such as marginalization.

However, applying the idea of syntactic scaffolding to our present scenario poses one difficulty. If we use a standard language model predicting words $w$ and predict the non-word symbols $y$ separately, we face the problem that the two sequences have different lengths. To overcome this in a straightforward way, we predict the $n$-gram of non-word actions $y_{t:t+n(t)}$ corresponding to the partial parse synchronous with step $t$ when we predict word $w_t$. We use a secondary softmax layer for this action $n$-gram prediction.
\begin{eqnarray}
	p(y_{t:t+n} \mid y_{<t}) = \mathrm{softmax}(E^{Y^*} \cdot h^{m}_{<t})_{y_{t:t+n}}
    \label{eq:out2}
\end{eqnarray}
Here $E^{Y^*}$ is the vocabulary of all transition $n$-grams excluding words found in the train corpus plus a blank symbol. Note that since Scaffolding operates only at train time, we do not need to worry about generalization of these $n$-grams to test time.

The models are thus trained to minimize the loss function $-\log p(Y,W)$ where
\begin{eqnarray}
p(Y, W) &= \prod_{t=1}^T p(w_t \mid w_{<t}) \nonumber\\
        &+ \prod_{t=1}^T p(y_{t:t+n(t)} \mid w_{<t})
\end{eqnarray}
The scaffold can be set so that the synchronous non-word action $n$-grams $y_{t:t+n(t)}$ are predicted either before (Figure~\ref{fig:models}c, left) or after (Figure~\ref{fig:models}c, right) producing $w_t$. We considered both variants in our experiments to empirically assess their impact on performance. We refer to this model as \textbf{Transformer Language Model with Syntactic Scaffold}, or \textbf{ScLM} in short, and its two versions \textbf{ScLM-past} and \textbf{ScLM-next}, for past and next $n$-gram prediction.

\section{Related Work}

Multitask learning \cite{caruana1997multitask} has been applied to a variety of NLP tasks with traditional modeling approaches \cite{miller-etal-2000-novel,sutton2005joint,sutton2007dynamic} as well as more recent neural models \cite{collobert2011natural,li2020transformer}. A recurring theme has been the use of structure in the form of syntactic trees to benefit other NLP tasks. Among the early works exploring this direction, \newcite{punyakanok-etal-2008-importance} showed that syntactic parses can benefit Semantic Role Labeling (SRL). \newcite{poon2009unsupervised} extended this idea to induce first-order logic representation in a unsupervised fashion, by clustering the dependency structures. In both cases syntax forms part of a pipeline and is not strictly supervision for the end task.

This trend continued with the rise of neural models. \newcite{collobert2011natural} improved deep convolution neural network for syntactic chunking models with additional POS supervision. \citet{zhang-weiss-2016-stack,sogaard-goldberg-2016-deep} observe the benefits of POS supervision at  different depths of a neural network model with impact on dependency parsing, tagging and CCG super tagging performance. \newcite{he2019syntax} perform a syntax-based pruning of semantic roles, showing benefits in a multilingual setting. More recently, \newcite{sachan2020syntax} incorporate a syntactic graph recurrent neural network into BERT models for better semantic role labeling. However, their method shows little or no benefit of syntax modeling for Named Entity Recognition and relation linking task. Neural machine translation \cite{chen2018syntax} and text generation \cite{li2020transformer} have also been shown to benefit from syntactic modeling. In a recent work, \newcite{li2020improving} use syntactic modeling in BERT based transformers to achieve performance gains on several text classification benchmarks. Other works have found that structural supervision in the form of intermediate fine-tuning (e.g., on CCG super tagging) is not helpful or even harmful \cite{pruksachatkun2020intermediate,warstadt2019investigating}.

The focus of our work is on gauging the impact of joint modeling on syntactic generalization performance. In this direction, the work of \citet{swayamdipta-etal-2018-syntactic} is close to the scaffolding version of our model. They predict multiple labels, extracted from syntactic information, as auxiliary task and show positive effects on shallow semantic parsing and co-reference resolution. We use however a single feature, constituency parsing $n$-gram, which is closer to prior work relying on Part-of-Speech information. In addition, we explore impact of using preceding structure as feature vs postceding structure, which as shown plays a role in the learning process.

In terms of modeling objective and syntactic representations, our method is closest to the works of \newcite{choe-charniak-2016-parsing,dyer-etal-2016-recurrent} that jointly model syntax and language. A more recent work from \citet{peng-etal-2019-palm} uses Rational Neural Networks language model that can derive binary unlabeled constituents from attention weights and can supervise the attention to attain a structural inductive bias. The proposed models show lower language modeling perplexity compared to their structure agnostic counterparts. We also extend here the idea of syntax-aware language modeling to transformer-based language models.

Finally, our approach relates to the other works that propose ways of incorporating structural information into Transformer-based models. This includes the use of dependency or tree structure for constraining self-attention patterns \citep{strubell2018linguistically,wang-etal-2019-tree,zhang2020sg}, guiding cross-attention \cite{chen2018syntax,astudillo2020transition}, modelling syntactic distance \citep{du-etal-2020-exploiting}, using syntactic information to guide the computation flow in the model \citep{shen-etal-2021-explicitly}, or through knowledge distillation \citep{kuncoro2020syntactic}. Our structured masking in parsing as language modeling approach is close in spirit to the methods that modify attention mechanism according to syntactic connections \cite{astudillo2020transition}; This work, however, primarily aims to study the impact of structural guidance on syntactic generalization. Therefore, we resort to simpler methods of incorporating structure to minimize the impact of modeling intricacies.

redpajama_set_name: RedPajamaArXiv


text:
Juwan M. Davis
In this berserk NBA offseason, the whole entire league has been shifting. Only eighteen of the twenty-eight all-stars from 2019 are still on the same roster. With all of these moves, it seems like no NBA player is safe this offseason.
Now that 2019 Finals MVP Kawhi Leonard is apart of the Los Angeles Clippers (four-year, $142 million deal) and he was able to bring another LA native with him in Paul George, this leaves 8x All-Star Russell Westbrook back in the same position he was in 2016-17.
The Oklahoma City Thunder have lost Kevin Durant, Paul George, James Harden, Serge Ibaka and Victor Oladipo all in the midst of the last seven years. The only franchise center piece they've managed to keep throughout all of that is Russell Westbrook.
Whether the 2017 MVP is the problem or not, the Thunder are back in desperation mode and they have to make a choice: Trade Gallinari and their set of first round picks for another superstar or Trade Russell Westbrook and start the rebuilding process.
If the Thunder are looking to rebuild and trade Russell Westbrook as Adrian Wojnarowski reported they are looking into, here are the best potential trade packages for Russell Westbrook.
Heat Receive:
– Russell Westbrook
Thunder Receive:
– Goran Dragic
– Justise Winslow
– 2019 1st Round
– 2022 2nd Round
The Miami Heat acquired their franchise's new star in Jimmy Butler this offseason. To make Miami a threat in the East again, they need another star. Russell Westbrook can be that piece they need to have one of the best backcourts in the NBA. Not to mention they are both some of the best two-way players in the league. The Heat have plenty of young role players to complete this deal even if Winslow, Dragic and a 2019 1st Rounder aren't appealing enough. Rookie G Tyler Herro has been showing how NBA ready he is in Summer League with shooting splits of: vs LAL (18 Pts 5-10 3P), vs GSW (20 Pts 7-21 FG), vs CHP (23 Pts 4-9 3P).
Westbrook isn't a high-demand player for a lot of these franchises either. His history of inefficiency and "stat padding" is something coaches keep in the back of their mind. Both Butler and Westbrook are very outspoken players too, so combining those two in the locker room could be risky. Talent wise, it's the best move the Heat can make and the Thunder will still have an insane amount of draft picks, players to use in future trades and a 23-year-old Justise Winslow who is just starting to mature in the NBA with career-highs all around last season.
Mavericks Receive:
– Justin Jackson
– Tim Hardaway Jr.
– Dwight Powell
– 2020 2nd Round (via 76ers)
The Dallas Mavericks are a team that's looking to have their first big season of the future in 2020 with Rookie of the Year Luka Doncic and 2018 All-Star Kristaps Porzingis. With Dallas' franchise icon Dirk Nowitkzi retiring they also lose a lot of veteran leadership that Russell Westbrook can provide in Dallas, all while creating their own big 3.
Oklahoma City will be able to surround their team with a 27-year-old Tim Hardaway Jr. that's averaged over 17 PPG for the past two seasons. Justin Jackson is on his third season who started his last 11 games in Dallas. Lastly, they get a 27-year-old Dwight Powell who's posting career highs in just about every category. OKC's future will be in good hands with this trade. It allows these younger players to have an identity before OKC brings in more young assets, as well as their many rookies they'll be drafting in these upcoming seasons.
Jim Mone/AP
Timberwolves Receive:
– Andrew Wiggins
– Jeff Teague
– Robert Covington
This is the first of two deals that involve the Timberwolves, who in the other deal are involved in a three-team trade. In this deal the Timberwolves get rid of Andrew Wiggins' huge contract, as well as his inefficiencies he's had with Karl-Anthony Towns. KAT's needed a one-two punch that can be lethal in the NBA, if it weren't for Jimmy Butler's distaste for Minnesota, we could've seen an impressive duo be created with them. Now Russell Westbrook has another dominant scorer to co-exist along with and KAT has a proven All-Star to motivate him without Andrew Wiggins' inconsistencies.
The Thunder now have a player of the future in Andrew Wiggins, who can finally have the chance to showcase himself as a threat in the league as the number one option on a team. Jeff Teague can be a mentor to Shai Gilgeous-Alexander as his contract expires in one year, giving OKC more cap space to look into free agency next offseason. Robert Covington will be a huge piece to this offense after showing he can be a consistent starter by averaging over 13 points ever since he's been a starter for the past six seasons.
Danny Moloshok/AP
Pistons Receive:
– Reggie Jackson
– Luke Kennard
– Tony Snell
– Thon Maker
In this trade scenario, the Pistons will be able to create the second or third best big 3 in the East. Westbrook can dominate their backcourt with Derrick Rose off the bench while Blake Griffin and Andre Drummond are forces down low. This becomes an immediate Eastern Conference contender even with the big future pieces the Pistons give up in the process. Although Russell Westbrook's rebounds numbers will most likely go down, giving him his first season without averaging a triple-double since 2016, winning should be his top priority in his 12th NBA season.
OKC will get another shot with Reggie Jackson, who had his most productive efforts as a member of the Thunder. Jackson came out looking like a future star in Oklahoma City before being traded to Detroit for Enes Kanter and a few role players. His tenure in Detroit also looked promising in his first two seasons, but injuries and having players who demand the ball more caused his production to slow down. Jackson can play a big role in OKC and also be a great mentor to Shai Gilgeous-Alexander.
For the rest of the young trade assets OKC will receive, Luke Kennard who has been slowly maturing into a great scorer. Tony Snell helps with their depth and Thon Maker can finally have a full season with the shot at proving himself. Maker's been on two franchises by his third season, but has always been playing a low amount of minutes per game. With this addition, OKC can help Maker develop into a solid role guy like they have with previous centers in Steven Adams and Enes Kanter.
Houston Rockets (3-team deal)
Rockets Receive:
– Gorgui Dieng
– Clint Capela
– 2021 1st Round (via Timberwolves)
– 2020 2nd Round (via Rockets from Grizzlies)
– Chris Paul
– Danilo Gallinari
This last trade is tricky, since Danilo Gallinari just got traded to OKC, this trade can only be done two months after he's been apart of the roster. With that being said, this trade improves every team. OKC still gets Wiggins and Covington like the one-on-one trade with Minnesota. This time with Capela, who's already created an identity in the NBA as a rim protector and a walking double-double by age 25.
The Rockets receive Westbrook and Dieng to have a PG still in his prime just three years after becoming an MVP, while Dieng gets to become a starter again by taking Capela's role and spreading the floor for Houston with his mid-range ability.
The Timberwolves still get the chance to be playoff contenders with Chris Paul and KAT being an offensive duo and potentially a lob city part two. Paul becomes a huge playmaker in Minnesota in a roster with a lot of guys who need a facilitator. Danilo Gallinari adds a more efficient scoring upgrade from Andrew Wiggins, who they'll be trading away in this deal to still have two 20 PPG scorers. Gallinari had his best NBA season last year at age 30 and could potentially be the most underrated player in the league. Minnesota immediately competes in the West with a trio of Paul-Gallinari-Towns.
Related ItemsBlake GriffinDallas MavericksDetroit PistonsHouston RocketsJames HardenJimmy ButlerKristaps PorzingisLuka DoncicMiami HeatMinnesota TimberwolvesOklahoma City ThunderRussell WestbrookTrending
← Previous Story 4 Prospects OKC Should Consider on Draft Day
About Juwan M. Davis
Editor/Writer for NBALEAD. Born and raised in the Land of 10,000 Lakes. Living in Los Angeles. Minnesota Timberwolves fan. Twitter: @JuwanMalikDavis
Every Argument For and Against Trading for Russ

redpajama_set_name: RedPajamaCommonCrawl


text:
Home/News/More than 1 million animal species are on the verge of extinction
More than 1 million animal species are on the verge of extinction
(ORDO NEWS) — Nature is in crisis and it's only getting worse. As species are disappearing at a rate not seen in 10 million years, more than 1 million species are currently on the brink of extinction.
Humans are causing this extinction crisis because of activities that take over animal habitats, polluting nature and contributing to global warming, scientists say.
A new global agreement to protect nature, agreed on Dec. 19, could help save animals, and scientists are urging countries around the world to ensure the agreement's success.
When an animal species disappears, so does a whole set of characteristics genes, behaviors, activities, and interactions with other plants and animals that may have taken thousands or millions even billions of years to evolve.
Any role that this species played in the ecosystem is also being lost, whether it was pollinating certain plants, mixing nutrients in the soil, fertilizing forests, or controlling other animal populations.
If this function is critical to the health of the ecosystem, the extinction of animals could lead to landscape transformation.
If humanity loses too many species, the consequences could be catastrophic and lead to the collapse of the entire system.
Hundreds of unique animals have disappeared around the world over the past five centuries, such as the flightless Dodo bird killed on the island of Mauritius in the late 1600s.
In many cases, humans are to blame – first because of fishing or hunting, as was the case with the South African subspecies of quagga zebra, which was hunted until the late 19th century, and more recently because of activities that pollute, destroy or take over wildlife. habitat.
Before a species goes extinct, it can already be considered "functionally extinct" – with not enough individuals to ensure the survival of the species.
More recent extinctions have allowed humans to interact with the last known individuals of some species, known as "endlings". When they leave, that is the end of these evolutionary lines.
"Toughie" was the last known specimen of Rabb's Fringe-Limbed tree frog. All but a few dozen of its species have been destroyed by the chytrid fungus in the wild in Panama.
The story of the passenger pigeon "Martha" is a cautionary tale for conservation: there were still millions of passenger pigeons in the 1850s, but they were eventually extirpated as conservation measures were taken only after the species had passed the point of no return.
The last individual died in 1914 at the Cincinnati Zoo.
"Lonesome George", found in 1971, was Ecuador's last tortoise on Pinta Island. Since the 17th century, about 200,000 individuals have been exterminated for their meat.
They later struggled to compete for food after goats were introduced to the island in the 1950s. Scientists tried to save the species through captive breeding before the last turtle died in 2012.
There are some species that may soon develop into their own endlings. The world's smallest porpoise, the endangered Mexican vaquita, has dwindled to 18 in the wild as populations were devastated by fishing nets.
A subspecies of the northern white rhinoceros, the second largest land mammal after elephants, has no hope of surviving the species after the death of its last male in 2018. Only the female and her daughter remained.
Scientists say these endling stories matter precisely because so many extinctions go unnoticed. Unlike endlings, most species simply disappear into the wild without people noticing.
Scientists have counted 881 animal species that have gone extinct since about 1500, dating back to the first records held by the International Union for the Conservation of Nature (IUCN), the global scientific regulator for the state of nature and wildlife.
However, this is an extremely conservative estimate of species extinction over the past five centuries, as it only represents cases that have been established with a high degree of certainty. If we include animal species that scientists believe may be extinct, this number rises to 1,473.
The number of recent recorded sightings has been increasing over time, especially since the start of the Industrial Revolution in the mid-19th century.
This shows that animals are increasingly endangered, but also that our knowledge of nature is improving as we study more and more species.
Some disappearances have caused public outcry, such as the announcement in 2016 of the disappearance of a tiny bat species from Christmas Island, last seen in 2009.
It was the first recorded extinction of a mammal in Australia in 50 years.
The loss of hundreds of species in 500 or so years may seem insignificant when millions of other species still live on the planet.
But the rate at which species are now disappearing is unprecedented in the last 10 million years.
When the rate of extinction jumps so high that more than 75% of the world's species go extinct within a relatively short period of time, less than 2 million years, it is considered a mass extinction event.
This has happened five times in the last half a billion years, as we know from studying the Earth's fossil record, as layers upon layers of sediment buried animal remains over time.
When a layer with a large and varied number of animals is discovered, scientists see that a mass extinction has occurred. Scientists warn that we have entered the sixth mass extinction.
After death, Einstein's brain was handed over to scientists : Where did he go
Angels are representatives of a highly developed civilization from another dimension
What existed before the emergence of the universe : Scientists claim to have found the answer
'Dynamic hover' trick could speed spacecraft through interstellar space


Milk enzymes Florap Smart Vials, High Nature.
LACTIC FERMENTS. Classic, effective and immediate.
ARE AVAILABLE IN 4 DIFFERENT DRAWINGS.
THE CONTENT OF EACH PACKAGE IS ABSOLUTELY UNCHANGED.
11 billion lactic ferments, enriched with Fruttoligosaccaridi (FOS) and Inulin.
The group Niacin and Vitamins of group B is useful in the presence of dietary deficiencies or increase in the need for these nutrients.
Contents: 10 vials of 10 ml with tank cap.
Recommendations for use : it is suggested to take the contents of 1 vial a day, preferably between meals.
Keep out of reach of children under three years. Do not exceed the recommended daily dose. The supplements are not intended as a substitute for a varied diet but should be used as part of a healthy and balanced lifestyle.
Vial Ingredients : Water, Fructose, Fructoligosaccharides, Inulin (from Cichorium Intybus L.radix). Acidifier: Citric acid Preservatives: Potassium sorbate, Sodium benzoate. Aroma.
Tank cap ingredients : Lactobacillus Delbruekii sub. Bulgaricus LB64 (titer 50 MLD / g), Lactobacillus Casei LC03 15 (titer 100 MLD / g), Lactobacillus Sporogens BC4 (titer 50MLD / g), Niacin (Vitamin PP), Lactobacillus Acidophilus LA02 (title 100 MLD / g), Lactobacillus Reuterii LRE02 (titer 100 MLD / g), Lactobacillus Rhamnosus LR32 (titer 200 MLD / g), Lactobacillus Salivarius LS33 (titer 300 MDL / g), Vitamin B2 (Riboflavin), Vitamin B6 (Pyridoxine hydrochloride), Vitamin B1 (Tiamin) . Technological adjuvants: dioxide silica.


Many consumers are familiar with the 3M product Clear Bra. Basically Clear Bra is a adhesive backed film that is stretched on to the front of a vehicle. These films over time tend to yellow and can crack or peel.
At Bodyline we have taken a different approach to protecting the front of a vehicle with a Clear Coat product that resists chips and dings and offers UV protection.
This product is crystal clear and unlike Clear Bra can be buffed if the need arises. This product will not yellow, peel or chip and we guarantee it. Since this is a professional spray on product you can choose how little or how much you want protected so prices will vary by application.
Today's modern headlamps are basically made from a high grade plastic. Over time they lose their clarity, become cloudy, nick and ding. Replacing the entire housing can be very expensive.
At Bodyline we can completely restore the luster and apply a Crystal Clear Chip Guard to the lens keeping the lens clear for the life of the vehicle. This clear cote can be buffed if the need arises.


text:
Home / WORLD / Court hears 'QAnon shaman' hasn't eaten since arrest over Capitol riot, mom says non-organic food makes him sick
Court hears 'QAnon shaman' hasn't eaten since arrest over Capitol riot, mom says non-organic food makes him sick
6 days ago WORLD 41 Views
Arizona native Jake Angeli faced his first federal court hearing in Phoenix on Monday, where the court heard that the so-called 'QAnon shaman' has not eaten since he was taken into custody.
Angeli, whose legal name is Jacob Anthony Angeli Chansley, quickly became a symbol of the chaos that was unfolding in the US Congress building last Wednesday as he went viral around the world for his outlandish outfit.
The 33-year-old appeared before the court via telephone as he is being held in quarantine in a detention facility.
© AP Photo/Manuel Balce Ceneta
When asked whether he could afford an attorney, Angeli, who is a proponent of the QAnon conspiracy theory, told the judge he couldn't, but suggested that situation could change. "Provided I am able to get hold of a certain friend of mine, I was told I could have a lawyer from him," he said.
Angeli's public defender, Gerald Williams, told the court that his client had a restricted diet and he has not eaten since he had been taken into custody. The lawyer added that he was not sure whether Angeli restricted his diet for religious or health reasons.
Angeli's mother, Martha Chansley, shed more light on the matter while speaking to reporters outside the courthouse. She told the assembled press that her son needs an organic diet. "He gets very sick if he doesn't eat organic food," she explained. "He needs to eat."
The woman added that she hasn't been in contact with Angeli since his arrest. "I really need to speak to my son," she said.
Inside the courtroom, Magistrate Judge Deborah Fine echoed the sentiments of the protester's mother, telling Angeli's attorney to contact the US Marshal's office to resolve the issue. "Mr Chansley needs to eat," she said.
The US Marshal for Phoenix, David Gonzalez, said the Marshals Service would work to accommodate Angeli's organic food request, if so ordered by the court. "For religious or health reasons, if (inmates) need a special diet, we are required to accommodate them," he told the Arizona Republic newspaper.
Discussing Angeli's prospects in court, Gonzalez said Angeli would be facing charges that would keep him locked up for "years and years."
Angeli at a rally last week. © AP Photo/Brynn Anderson
Judge Fine ordered that Angeli be held until a hearing scheduled for Friday, at which she would determine whether he would face charges in Washington DC while in custody or out of custody.
While Angeli's mother and Judge Fine agreed that the rioter needed to eat, Meghan McCain suggested online that he should only be fed dog food.
The hardline sentiment attracted much discussion on social media, where many noted that McCain's father, deceased Arizona senator John McCain, who was tortured in a POW camp during the Vietnam war, would likely not share her views.
Tags arrest court eaten hears qanon shaman since
'Deepest concerns': South Africa Covid-19 variant binds more easily to human cells, making it more infectious, govt adviser says
China's Hebei province to reexamine Covid-19 test results after lab caught out over positive cases
The rise of La Resistenza to Covid lockdowns! Defiant Italian bar and restaurant owners choose civil disobedience and reopen
I've heard it said that Americans are the most propagandized people on the planet. I …


This rustic bookcase is the perfect home for your prized book collection, valuable ornaments or even a few photo frames. Crafted from premium grade solid oak, it features the popular Original Rustic Solid Oak design, characterised by subtly rounded corners, chamfered edges and brass tinted drop bar handles. Offering five spacious shelves there's ample display space, and at the base of the bookcase you'll find two compact drawers for tidying away all manner of items. Each drawer is crafted using traditional dovetail joints with solid oak backs, bases and runners. We never skimp on materials and we never use chipboard, MDF or veneers in any of our products. The solid oak timbers are finished with wax; creating the appearance of a much loved antique while offering protection from daily wear and tear.


And the garden I sit in becomes Wonderland.
The breeze caressing me calmly.
Old, wrinkled and full of wisdom.
Stories just beginning, and stories about to end.
They peek over my shoulder, wide-eyed.
I smile back at them as they run along.
Sobbing uncontrollably, clearly unable to do so.
Giggling and whispering sweet nothings.
'Ah! Young love,' I sigh wistfully, looking away.
And the mother watching them happily.
And the simpler times with my family.
I come across an elderly couple holding hands.
I wish I had the courage to ask.
The way they do in mine?
I guess I'll never know what goes on in whose mind.
Of stories and memories, good and bad.
And those shying away, hiding in the shadows.
I just secretly wish there was more I knew.
Hey, everyone! I always notice small things around me when I'm all alone and I am fascinated by what I see sometimes. I love seeing snippets of people's lives and I hope you do, too! I also hope you liked my poem. Let me know if you did in the comments below. You can also reach me on socials, link to which are below and on my Homepage. I'll look forward to hearing from all of you!
Indeed. I guess we somehow overlook these ordinary, but beautiful things in life.
Really awesome and the way you described the thoughts in words!!!!! Perfect realistic thoughts…While we read it takes towards the imaginations world!!!😍-Shubhi!!!