text:
\section{Additional Details and Analysis of \textsc{MaMa}}

\subsection{Implementation Details}
\label{sec:implement}

To evaluate the mapped facts, we first use Match\ stage of \textsc{MaMa}\ to run over the corresponding documents to generate the candidate facts. For Map\ stage on TAC KBP, we link to the oracle annotation of the entities or spans in the TAC KBP corpus. On Wikidata, the entity linking method described in Sec.~\ref{sec:ground} is first leveraged to link entities in the candidate facts to Wikipedia anchors. Then a Wikipedia anchor to the Wikidata item dictionary is used to further link the entities to Wikidata. If the head or tail is a pronoun, we further use neuralcoref~\footnote{\tiny\url{https://github.com/huggingface/neuralcoref}} for coreference resolution. We use GloVe~\citep{pennington2014glove} embedding for disambiguation. The relation mapping is constructed offline for TAC KBP and Wikidata respectively using the method in Sec.~\ref{sec:ground}. For oracle facts in Wikidata, we only preserve those facts describing relations that could be linked to a corresponding Wikipedia anchor. We rule out facts of attributes about entities and facts of auxiliary relations (such as \textit{topic's main category.P901}) and finally results in 27,368,562 oracle facts.

For Wikidata, at Match\ stage, we randomly split the English Wikipedia data into 20 partitions, and map the data partitions to 20 distributed servers to run. Each server is configured with four Tesla K80 12Gs. We set the max sequence length to 256, and batch size as 32 for \textsc{MaMa}-BERT$_{\rm LARGE}$ and 4 for \textsc{MaMa}-GPT-2$_{\rm XL}$. We use implementations of pre-trained LMs in Transformers package~\footnote{\tiny\url{https://github.com/huggingface/transformers}}. We use spaCy sentencizer~\footnote{{\tiny\url{https://spacy.io/api/sentencizer}}} to segment the documents into sentences. \textsc{MaMa}-BERT$_{\rm LARGE}$ takes approximately 48 hours, and \textsc{MaMa}-GPT-2$_{\rm XL}$ costs around 96 hours. The resulting candidate facts of Match\ stage from the 20 servers are then reduced a data server, where a MongoDB database is maintained to store the oracle Wikidata and entity linking results to enable the efficient Map\ stage. To produce the open KGs, Map\ stage takes around 18 hours. The setup is similar to TAC KBP. Match\ stage is done within 48 hours for all the settings. The batch sizes of \textsc{MaMa}-BERT$_{\rm BASE}$, \textsc{MaMa}-GPT-2, \textsc{MaMa}-GPT-2$_{\rm MEDIUM}$, \textsc{MaMa}-GPT-2$_{\rm LARGE}$ are 64, 32, 16, 8 respectively.

The parameter settings are shared across TAC KBP and Wikidata. All the choices are based on the parameter study in Sec.~\ref{sec:para}. The beam size of Algorithm~\ref{alg:beamsearch} is set to 6. The matching degree threshold of Constraint \#1 (Sec.~\ref{sec:filter}) is set to 0.005, and the number of distinct head-tail pairs of Constraint \#2 (Sec.~\ref{sec:filter}) is set to 10. To generate the attention weight matrix $\mathbf{A}_s$ of a sentence, we reduce the weights of every attention head in the last layer of pre-trained LMs using the mean operator.

\subsection{Error Analysis}
\label{sec:error}
There is still significant room to improve \textsc{MaMa}. To further understand the shortage of \textsc{MaMa}, we conduct an error analysis of the errors in precision (i.e., incorrect facts returned by \textsc{MaMa}) of Table~\ref{tab:resulttac} and Table~\ref{tab:resultwiki}. We choose our best method \textsc{MaMa}-GPT-2$_{\rm XL}$ for the study. We sample 100 documents from the Wikidata dataset, and manually check the reasons for the errors. We find {33.1\%} of the errors are caused by {\em incorrect entities}, while the relation phrases are correct. The errors are due to the incorrect noun chunk detected by the spaCy~\footnote{\tiny{\url{https://spacy.io/usage/linguistic-features/\#noun-chunks}}}. {18.3\%} of the errors are due to the {\em missing relation mapping} created in Sec.~\ref{sec:ground}. Note that we find approximately {23.8\%} of the errors are actually {\em correct facts that are new in the reference KGs}. e.g., {\sl (Bob\_Dylan.Q392, residence.P551, Nashville.Q23197)} (in Figure~\ref{fig:kg}) is not an existing fact in Wikidata, but it is a correct mapped fact based on our annotation. The rest errors made by \textsc{MaMa}-GPT-2$_{\rm XL}$ are {\em incorrect relation phrases}, such as uninformative relation phrases. We find similar errors are made by \textsc{MaMa}-GPT-2$_{\rm XL}$ on TAC KBP.
Similar to Sec.~\ref{sec:unmapped}, enhancing the entity detection, entity linker, relation mapping, and relation generation are helpful.
We also plan to leverage lifelong learning~\citep{carlson2010toward} to add true facts to the reference KGs to improve the evaluation.

\subsection{Parameter Study}
\label{sec:para}

\begin{figure*}
\centering
\subcaptionbox{{\small Beam size.}}{\includegraphics[width=0.3\textwidth]{figs/beamsize.pdf}}%
\hspace{0.1in}
\subcaptionbox{{\small Constraint \#1.}}{\includegraphics[width=0.3\textwidth]{figs/degree.pdf}}%
\hspace{0.1in}
\subcaptionbox{{\small Constraint \#2.}}{\includegraphics[width=0.3\textwidth]{figs/relation.pdf}}%
\\
\subcaptionbox{{\small Attention weights layers.}}{\includegraphics[width=0.32\textwidth]{figs/alayer.pdf}}%
\hspace{0.1in}
\subcaptionbox{{\small Attention weights reduction.}}{\includegraphics[width=0.3\textwidth]{figs/areduction.pdf}}%
\caption{{\small Parameter study with \textsc{MaMa}-BERT$_{\rm BASE}$ on TAC KBP hold-out subset.}}
\label{fig:para}
\end{figure*}

We study the effects of the parameters using \textsc{MaMa}-BERT$_{\rm BASE}$ on TAC KBP. We randomly sample 20\% of the oracle query entities as a hold-out dataset to tune the parameters, and use the best parameter setting achieved for both TAC KBP and Wikidata experiments. When studying the effect of a certain parameter, we keep the remaining parameters as default described in Sec.~\ref{sec:implement}. We use F1 to measure the effects.

{\bfseries Effects of beam size \ } Figure~\ref{fig:para}(a) illustrates the effects of various beam sizes in Algorithm~\ref{alg:beamsearch}. We find that in general, the larger the beam size is, the better F1 the setting achieves. This is because that \textsc{MaMa}\ is able to reserve more potentially correct facts when more candidates are allowed in the Match\ stage. However, F1 improvement gradually becomes subtle, while the computation costs increase more significantly. For sake of the efficiency, we do not explore larger beam sizes. We set the beam size as 6.

{\bfseries Effects of search constraints \ } Figure~\ref{fig:para}(b) compares the effect of different thresholds of the matching degree of Constraint \#1 in Sec.~\ref{sec:filter}. We set the threshold as 0.005 since it achieves the best result. Note that the summed attention score is normalized by the length of the fact to penalize the cumbersome facts. Figure~\ref{fig:para}(c) shows the impact of the number of distinct head-tail pairs in identifying common relations of Constraint \#2 in Sec.~\ref{sec:filter}. The best result is achieved when it equals 10. This shows that while \textsc{MaMa}\ mostly identifies frequent relations, it is also able to capture some rare relations for the open schema.

{\bfseries Effects of attention weights \ } Figure~\ref{fig:para}(d) shows the comparison between attention weights of the last layer and the mean of all layers. The attention weights of the last layer perform better. This is due to the attention weights in lower layers are low-level linguistic knowledge according to \citep{clark2019does,ramsauer2020hopfield}, which are less relevant to the factual knowledge for the KG construction. Figure~\ref{fig:para}(e) compares the impact of different attention reduction, i.e., mean, max, over the attention heads of the last layer. We find the ``mean'' perform better. The reason is that the token often intensively attends to several specific tokens in the sequence~\citep{michel2019sixteen}, and the ``mean'' operator is more sensitive to such information.

\section{Samples from \textsc{MaMa}\ on TAC KBP}

\subsection{Mapped Facts}

We randomly sample 100 documents from TAC KBP corpus, then randomly sample sentences from those documents.
The uncurated candidate facts and the corresponding mapped facts of the sampled sentences based on our best methods \textsc{MaMa}-BERT$_{\rm LARGE}$ and \textsc{MaMa}-GPT-2$_{\rm XL}$ are shown in Figure~\ref{fig:kbpmapbert} and Figure~\ref{fig:kbpmapgpt} respectively.
We also randomly sample several sentences in which \textsc{MaMa}-BERT$_{\rm LARGE}$ differs from \textsc{MaMa}-GPT-2$_{\rm XL}$ in the resulting facts for comparison, which is illustrated in Figure~\ref{fig:kbpmapbertgpt}.
In each table, ``{\bf ID}'' represents the document ID of a sampled sentence in TAC KBP corpus. ``{\bf Sentence}'' indicates the sampled sentence. ``{\bf Candidate facts to mapped facts}'' column contains the candidate facts (on the left side of ``$\rightarrow$'') and their corresponding mapped facts (on the right side of ``$\rightarrow$'').

\subsection{Unmapped Facts}
We randomly sample 100 documents from TAC KBP corpus. From those documents, we show unmapped facts from the sampled sentences from those documents. We manually check the correctness of the unmapped facts according to Sec.~\ref{sec:unmapped}, and show the correct ones.
The original candidate facts with the corresponding unmapped facts of the sampled sentences generated by \textsc{MaMa}-BERT$_{\rm LARGE}$ and \textsc{MaMa}-GPT-2$_{\rm XL}$ are shown in Figure~\ref{fig:kbpunmapbert} and Figure~\ref{fig:kbpunmapgpt}. A further comparison of the unmapped candidate facts is illustrated in Figure~\ref{fig:kbpunmapbertgpt}.
In each table, ``{\bf ID}'' represents the document ID of a sampled sentence in TAC KBP corpus. ``{\bf Sentence}'' indicates the sampled sentence. ``{\bf Candidate facts to unmapped facts}'' column contains the candidate facts (on the left side of ``$\rightarrow$'') and their corresponding unmapped facts (on the right side of ``$\rightarrow$'').

\section{Samples from \textsc{MaMa}\ on Wikidata}
\label{seca:wiki}

\subsection{Mapped Facts}

Similar to TAC KBP, we randomly sample 100 documents from the Wikidata corpus (i.e., English Wikipedia), then randomly sample sentences from those documents.
The uncurated candidate facts and the corresponding mapped facts of the sampled sentences based on our best methods \textsc{MaMa}-BERT$_{\rm LARGE}$ and \textsc{MaMa}-GPT-2$_{\rm XL}$ are shown in Figure~\ref{fig:wikimapbert} and Figure~\ref{fig:wikimapgpt} respectively.
We also randomly sample several sentences in which \textsc{MaMa}-BERT$_{\rm LARGE}$ differs from \textsc{MaMa}-GPT-2$_{\rm XL}$ in the resulting facts for comparison, which is illustrated in Figure~\ref{fig:wikimapbertgpt}.
In each table, ``{\bf ID}'' represents the Wikipedia page's title of a sampled sentence. ``{\bf Sentence}'' indicates the sampled sentence. ``{\bf Candidate facts to mapped facts}'' column contains the candidate facts (on the left side of ``$\rightarrow$'') and their corresponding mapped facts (on the right side of ``$\rightarrow$'').

\subsection{Unmapped Facts}

Similar to TAC KBP, we randomly sample 100 documents from the Wikidata corpus. From those documents, we show unmapped facts from several sampled sentences from those documents. We manually check the correctness of the unmapped facts according to Sec.~\ref{sec:unmapped}, and show the correct ones.
The original candidate facts with the corresponding unmapped facts of the sampled sentences generated by \textsc{MaMa}-BERT$_{\rm LARGE}$ and \textsc{MaMa}-GPT-2$_{\rm XL}$ are shown in Figure~\ref{fig:wikiunmapbert} and Figure~\ref{fig:wikiunmapgpt}. A further comparison of the unmapped candidate facts is illustrated in Figure~\ref{fig:wikiunmapbertgpt}.
In each table, ``{\bf ID}'' represents the Wikipedia page's title of a sampled sentence. ``{\bf Sentence}'' indicates the sampled sentence. ``{\bf Candidate facts to unmapped facts}'' column contains the candidate facts (on the left side of ``$\rightarrow$'') and their corresponding unmapped facts (on the right side of ``$\rightarrow$'').

\section{Additional Open KG Subgraphs from \textsc{MaMa}\ on Wikidata}
We sample several documents from the Wikidata corpus. We visualize the mapped facts and unmapped facts from those documents as examples of subgraphs in the resulting open KGs. We show the snapshots of the subgraphs generated by \textsc{MaMa}-BERT$_{\rm LARGE}$ from Figure~\ref{fig:kg1} to Figure~\ref{fig:kg10}. We similarly illustrate the snapshots of the subgraphs constructed by \textsc{MaMa}-GPT-2$_{\rm XL}$ from Figure~\ref{fig:kg11} to Figure~\ref{fig:kg20}. In each figure, the blue node and arrow represent the mapped facts in the Wikidata schema, while the yellow node and arrow denote the unmapped facts in the open schema. We additionally visualize the correct facts that are new in Wikidata according to Sec.~\ref{sec:error} in yellow.

\begin{figure}
\begin{center}
\tiny
\resizebox{0.99\linewidth}{!}
  {
\begin{tabular}{l p{11cm} p{11cm}}
\toprule
{\bf ID} & {\bf Sentence} & {\bf Candidate facts to mapped facts} \\ \hline
\input{appendices_table/mapped_bertlarge_kbp}
\bottomrule
\end{tabular}
}
\end{center}
\vspace{-0.1in}
\caption{{\small Mapped facts: \textsc{MaMa}-BERT$_{\rm LARGE}$ on TAC KBP.}} \label{fig:kbpmapbert}
\end{figure}

\begin{figure}
\begin{center}
\tiny
\resizebox{0.99\linewidth}{!}
  {
\begin{tabular}{l p{11cm} p{11cm}}
\toprule
{\bf ID} & {\bf Sentence} & {\bf Candidate facts to mapped facts} \\ \hline
\input{appendices_table/mapped_gpt2xl_kbp}
\bottomrule
\end{tabular}
}
\end{center}
\caption{{\small Mapped facts: \textsc{MaMa}-GPT-2$_{\rm XL}$ on TAC KBP.}} \label{fig:kbpmapgpt}
\end{figure}

\begin{figure}
\begin{center}
\tiny
\resizebox{1.0\linewidth}{!}
  {
\begin{tabular}{l p{7cm} p{7cm} p{7cm}}
\toprule
\multirow{2}{*}{\bf ID} & \multirow{2}{*}{\bf Sentence} & \multicolumn{2}{c}{\bf Candidate facts to mapped facts} \\
& & {\bf BERT$_{\rm LARGE}$} & {\bf GPT-2$_{\rm XL}$} \\ \hline
\input{appendices_table/mapped_compare_kbp}
\bottomrule
\end{tabular}
}
\end{center}
\caption{{\small Mapped facts: \textsc{MaMa}-BERT$_{\rm LARGE}$ vs. \textsc{MaMa}-GPT-2$_{\rm XL}$ on TAC KBP.}} \label{fig:kbpmapbertgpt}
\end{figure}

\begin{figure}
\begin{center}
\tiny
\resizebox{1.0\linewidth}{!}
  {
\begin{tabular}{l p{10cm} p{10cm}}
\toprule
{\bf ID} & {\bf Sentence} & {\bf Candidate facts to unmapped facts} \\ \hline
\input{appendices_table/unmapped_bertlarge_kbp}
\bottomrule
\end{tabular}
}
\end{center}
\caption{{\small Unmapped facts: \textsc{MaMa}-BERT$_{\rm LARGE}$ on TAC KBP.}} \label{fig:kbpunmapbert}
\end{figure}

\begin{figure}
\begin{center}
\tiny
\resizebox{1.0\linewidth}{!}
  {
\begin{tabular}{l p{10cm} p{10cm}}
\toprule
{\bf ID} & {\bf Sentence} & {\bf Candidate facts to unmapped facts} \\ \hline
\input{appendices_table/unmapped_gpt2xl_kbp}
\bottomrule
\end{tabular}
}
\end{center}
\caption{{\small Unmapped facts: \textsc{MaMa}-GPT-2$_{\rm XL}$ on TAC KBP.}} \label{fig:kbpunmapgpt}
\end{figure}

\begin{figure}
\begin{center}
\tiny
\resizebox{1.0\linewidth}{!}
  {
\begin{tabular}{l p{7cm} p{7cm} p{7cm}}
\toprule
\multirow{2}{*}{\bf ID} & \multirow{2}{*}{\bf Sentence} & \multicolumn{2}{c}{\bf Candidate facts to mapped facts} \\
& & {\bf BERT$_{\rm LARGE}$} & {\bf GPT-2$_{\rm XL}$} \\ \hline
\input{appendices_table/unmapped_compare_kbp}
\bottomrule
\end{tabular}
}
\end{center}
\caption{{\small Unmapped facts: \textsc{MaMa}-BERT$_{\rm LARGE}$ vs. \textsc{MaMa}-GPT-2$_{\rm XL}$ on TAC KBP.}} \label{fig:kbpunmapbertgpt}
\end{figure}

\begin{figure}
\begin{center}
\tiny
\resizebox{1.0\linewidth}{!}
  {
\begin{tabular}{l p{10cm} p{10cm}}
\toprule
{\bf ID} & {\bf Sentence} & {\bf Candidate facts to mapped facts} \\ \hline
\input{appendices_table/mapped_bertlarge_wikidata}
\bottomrule
\end{tabular}
}
\end{center}
\caption{{\small Mapped facts: \textsc{MaMa}-BERT$_{\rm LARGE}$ on Wikidata.}} \label{fig:wikimapbert}
\end{figure}

\begin{figure}
\begin{center}
\tiny
\resizebox{1.0\linewidth}{!}
  {
\begin{tabular}{l p{10cm} p{10cm}}
\toprule
{\bf ID} & {\bf Sentence} & {\bf Candidate facts to mapped facts} \\ \hline
\input{appendices_table/mapped_gpt2xl_wikidata}
\bottomrule
\end{tabular}
}
\end{center}
\caption{{\small Mapped facts: \textsc{MaMa}-GPT-2$_{\rm XL}$ on Wikidata.}} \label{fig:wikimapgpt}
\end{figure}

\begin{figure}
\begin{center}
\tiny
\resizebox{1.0\linewidth}{!}
  {
\begin{tabular}{l p{7cm} p{7cm} p{7cm}}
\toprule
\multirow{2}{*}{\bf ID} & \multirow{2}{*}{\bf Sentence} & \multicolumn{2}{c}{\bf Candidate facts to mapped facts} \\
& & {\bf BERT$_{\rm LARGE}$} & {\bf GPT-2$_{\rm XL}$} \\ \hline
\input{appendices_table/mapped_compare_wikidata}
\bottomrule
\end{tabular}
}
\end{center}
\caption{{\small Mapped facts: \textsc{MaMa}-BERT$_{\rm LARGE}$ vs. \textsc{MaMa}-GPT-2$_{\rm XL}$ on Wikidata.}} \label{fig:wikimapbertgpt}
\end{figure}

\begin{figure}
\begin{center}
\tiny
\resizebox{1.0\linewidth}{!}
  {
\begin{tabular}{l p{10cm} p{10cm}}
\toprule
{\bf ID} & {\bf Sentence} & {\bf Candidate facts to unmapped facts} \\ \hline
\input{appendices_table/unmapped_bertlarge_wikidata}
\bottomrule
\end{tabular}
}
\end{center}
\caption{{\small Unmapped facts: \textsc{MaMa}-BERT$_{\rm LARGE}$ on Wikidata.}} \label{fig:wikiunmapbert}
\end{figure}

\begin{figure}
\begin{center}
\tiny
\resizebox{1.0\linewidth}{!}
  {
\begin{tabular}{l p{10cm} p{10cm}}
\toprule
{\bf ID} & {\bf Sentence} & {\bf Candidate facts to unmapped facts} \\ \hline
\input{appendices_table/unmapped_gpt2xl_wikidata}
\bottomrule
\end{tabular}
}
\end{center}
\caption{{\small Unmapped facts: \textsc{MaMa}-GPT-2$_{\rm XL}$ on Wikidata.}} \label{fig:wikiunmapgpt}
\end{figure}

\begin{figure}
\begin{center}
\tiny
\resizebox{1.0\linewidth}{!}
  {
\begin{tabular}{l p{7cm} p{7cm} p{7cm}}
\toprule
\multirow{2}{*}{\bf ID} & \multirow{2}{*}{\bf Sentence} & \multicolumn{2}{c}{\bf Candidate facts to mapped facts} \\
& & {\bf BERT$_{\rm LARGE}$} & {\bf GPT-2$_{\rm XL}$} \\ \hline
\input{appendices_table/unmapped_compare_wikidata}
\bottomrule
\end{tabular}
}
\end{center}
\caption{{\small Unmapped facts: \textsc{MaMa}-BERT$_{\rm LARGE}$ vs. \textsc{MaMa}-GPT-2$_{\rm XL}$ on Wikidata.}} \label{fig:wikiunmapbertgpt}
\end{figure}

\begin{figure*}
    \centering
    \includegraphics[width=0.5\textwidth]{images/wiki_kg_tp_bertlarge_douglas_bader.png}

    \caption{{\small A snapshot subgraph of the open KG generated by \textsc{MaMa}-BERT$_{\rm LARGE}$ from the Wikipedia page ``Douglas\_Bader''. }
      \label{fig:kg1}}
\end{figure*}

\begin{figure*}
    \centering
    \includegraphics[width=0.4\textwidth]{images/wiki_kg_tp_bertlarge_helen_storrow.png}

    \caption{{\small A snapshot subgraph of the open KG generated by \textsc{MaMa}-BERT$_{\rm LARGE}$ from the Wikipedia page ``Helen\_Storrow''. }
      \label{fig:kg3}}
\end{figure*}

\begin{figure*}
    \centering
    \includegraphics[width=0.5\textwidth]{images/wiki_kg_tp_bertlarge_jacob_van_ruisdael.png}

    \caption{{\small A snapshot subgraph of the open KG generated by \textsc{MaMa}-BERT$_{\rm LARGE}$ from the Wikipedia page ``Jacob\_van\_Ruisdael''. }
      \label{fig:kg4}}
\end{figure*}

\begin{figure*}
    \centering
    \includegraphics[width=0.5\textwidth]{images/wiki_kg_tp_bertlarge_john_maynard_keynes.png}

    \caption{{\small A snapshot subgraph of the open KG generated by \textsc{MaMa}-BERT$_{\rm LARGE}$ from the Wikipedia page ``John\_Maynard\_Keynes''. }
      \label{fig:kg5}}
\end{figure*}

\begin{figure*}
    \centering
    \includegraphics[width=0.5\textwidth]{images/wiki_kg_tp_bertlarge_liaquat_ali_khan.png}

    \caption{{\small A snapshot subgraph of the open KG generated by \textsc{MaMa}-BERT$_{\rm LARGE}$ from the Wikipedia page ``Liaquat\_Ali\_Khan''. }
      \label{fig:kg6}}
\end{figure*}

\begin{figure*}
    \centering
    \includegraphics[width=0.5\textwidth]{images/wiki_kg_tp_bertlarge_neville_southall.png}

    \caption{{\small A snapshot subgraph of the open KG generated by \textsc{MaMa}-BERT$_{\rm LARGE}$ from the Wikipedia page ``Neville\_Southall''. }
      \label{fig:kg7}}
\end{figure*}

\begin{figure*}
    \centering
    \includegraphics[width=0.5\textwidth]{images/wiki_kg_tp_bertlarge_pauline_baynes.png}

    \caption{{\small A snapshot subgraph of the open KG generated by \textsc{MaMa}-BERT$_{\rm LARGE}$ from the Wikipedia page ``Pauline\_Baynes''. }
      \label{fig:kg9}}
\end{figure*}

\begin{figure*}
    \centering
    \includegraphics[width=0.5\textwidth]{images/wiki_kg_tp_bertlarge_thor_heyerdahl.png}

    \caption{{\small A snapshot subgraph of the open KG generated by \textsc{MaMa}-BERT$_{\rm LARGE}$ from the Wikipedia page ``Thor\_Heyerdahl'. }
      \label{fig:kg10}}
\end{figure*}

\begin{figure*}
    \centering
    \includegraphics[width=0.5\textwidth]{images/wiki_kg_tp_gpt2xl_douglas_bader.png}

    \caption{{\small A snapshot subgraph of the open KG generated by \textsc{MaMa}-GPT-2$_{\rm XL}$ from the Wikipedia page ``Douglas\_Bader''. }
      \label{fig:kg11}}
\end{figure*}

\begin{figure*}
    \centering
    \includegraphics[width=0.5\textwidth]{images/wiki_kg_tp_gpt2xl_helen_storrow.png}

    \caption{{\small A snapshot subgraph of the open KG generated by \textsc{MaMa}-GPT-2$_{\rm XL}$ from the Wikipedia page ``Helen\_Storrow''. }
      \label{fig:kg13}}
\end{figure*}

\begin{figure*}
    \centering
    \includegraphics[width=0.5\textwidth]{images/wiki_kg_tp_gpt2xl_jacob_van_ruisdael.png}

    \caption{{\small A snapshot subgraph of the open KG generated by \textsc{MaMa}-GPT-2$_{\rm XL}$ from the Wikipedia page ``Jacob\_van\_Ruisdael''. }
      \label{fig:kg14}}
\end{figure*}

\begin{figure*}
    \centering
    \includegraphics[width=0.5\textwidth]{images/wiki_kg_tp_gpt2xl_john_maynard_keynes.png}

    \caption{{\small A snapshot subgraph of the open KG generated by \textsc{MaMa}-GPT-2$_{\rm XL}$ from the Wikipedia page ``John\_Maynard\_Keynes''. }
      \label{fig:kg15}}
\end{figure*}

\begin{figure*}
    \centering
    \includegraphics[width=0.5\textwidth]{images/wiki_kg_tp_gpt2xl_liaquat_ali_khan.png}

    \caption{{\small A snapshot subgraph of the open KG generated by \textsc{MaMa}-GPT-2$_{\rm XL}$ from the Wikipedia page ``Liaquat\_Ali\_Khan''. }
      \label{fig:kg16}}
\end{figure*}

\begin{figure*}
    \centering
    \includegraphics[width=0.5\textwidth]{images/wiki_kg_tp_gpt2xl_neville_southall.png}

    \caption{{\small A snapshot subgraph of the open KG generated by \textsc{MaMa}-GPT-2$_{\rm XL}$ from the Wikipedia page ``Neville\_Southall''. }
      \label{fig:kg17}}
\end{figure*}

\begin{figure*}
    \centering
    \includegraphics[width=0.5\textwidth]{images/wiki_kg_tp_gpt2xl_pauline_baynes.png}

    \caption{{\small A snapshot subgraph of the open KG generated by \textsc{MaMa}-GPT-2$_{\rm XL}$ from the Wikipedia page ``Pauline\_Baynes''. }
      \label{fig:kg19}}
\end{figure*}

\begin{figure*}
    \centering
    \includegraphics[width=0.35\textwidth]{images/wiki_kg_tp_gpt2xl_thor_heyerdahl.png}

    \caption{{\small A snapshot subgraph of the open KG generated by \textsc{MaMa}-GPT-2$_{\rm XL}$ from the Wikipedia page ``Thor\_Heyerdahl''. }
      \label{fig:kg20}}
\end{figure*}
\section{\textsc{MaMa}}
\label{sec:app}
\vspace{-0.05in}

We introduce an unsupervised end-to-end approach Match\ and Map\ (\textsc{MaMa}) as illustrated in Figure~\ref{fig:overview} to construct open knowledge graphs (KGs) from language models (LMs). \textsc{MaMa}\ constructs the KGs with a single forward pass of the pre-trained LMs (without fine-tuning) over the corpora. The two stages of \textsc{MaMa}\ are:

\vspace{-0.05in}
{\bfseries Match} generates a set of candidate facts from a textual corpus. LMs contain global or world knowledge learned from large-scale corpora, which often does not perfectly match the knowledge in the target corpus. The goal of this stage is to match the knowledge stored in pre-trained LMs with facts in the corpus. Each fact is represented as a triplet {\sl (head, relation, tail)}~\footnote{\tiny{We use the term ``head'' and ``tail'' to denote head and tail's ``entities'' or ``entity mentions'' for simplicity.}}, in short, $(h, r, t)$, and passed to Map\ stage. Match\ procedure is detailed in Sec.~\ref{sec:search}.

\vspace{-0.05in}
{\bfseries Map} produces an open KG using the matched candidate facts from Match\ stage. The constructed open KG has two portions: (a) mapped candidate facts that are in a fixed KG schema, e.g., {\sl (Dylan, is, songwriter)} is mapped to {\sl (Bob\_Dylan.Q392, occupation.P106, Songwriter.Q753110)} according to Wikidata schema; and (b) unmapped candidate facts that are in an open schema, e.g., a candidate fact {\sl (Dylan, signed, Albert Grossman)} is partially mapped to {\sl (Bob\_Dylan.Q392, sign, Albert\_Grossman.Q708584)} in the open schema. This stage is described in Sec.~\ref{sec:kd}.

\vspace{-0.05in}
\subsection{Match}
\label{sec:search}
\vspace{-0.05in}

\begin{figure*}
\centering
\subcaptionbox{{\small Matching example.}}{\includegraphics[width=0.45\textwidth]{figs/searchoverallV1.pdf}}%
\hspace{0.2in}
\subcaptionbox{{\small Attention matrix for matching degree calculation.}}{\includegraphics[width=0.33\textwidth]{figs/attention.pdf}}%
\caption{{\small Illustration of Match\ stage. The upper part of (a) represents the general matching steps of generating the best matched candidate fact {\sl (Dylan, is, songwriter)} from the sentence ``Dylan is a songwriter.'' The lower portion shows the corresponding step-by-step process. Given a head-tail pair {\sl (Dylan, songwriter)}, at each step, the search chooses one of the actions, i.e., {\texttu{start}, \texttu{yield}, \texttu{stop}} to produce an intermediate candidate fact. The search {\em starts} by adding the head ``Dylan'' as an initial candidate (step 0). The matching degree of the candidate is initialized as 0. Next, a new candidate is {\em yielded} if the candidate has not reached the tail ``songwriter'', by appending the next largest attended token (with the largest score from the attention matrix (b) of the sentence) to the end of the current candidate, and the corresponding matching degrees are increased by the associated attention scores (step 1 and step 2). Otherwise, the search {\em stops}, and the candidate fact with the best matching degree is returned for the head-tail pair (step 3). The attention matrix (b) is from the forward pass of the LM without fine-tuning over the sentence. ``x'' marks the tokens to prevent searching backward.}}
\label{fig:search}
\end{figure*}

We frame the matching procedure as a search problem. To obtain the best matched candidate facts of an input sentence, the candidates with the top matching degrees are returned from a search process. The matching degree is derived from the search in the attention weight matrices of the pre-trained LM, since the attention weight matrices are the container of the knowledge in the pre-trained LM. The attention weight matrices are simply from the forward pass of the LM without fine-tuning over the sentence.

\vspace{-0.05in}
\subsubsection{Beam Search}
\vspace{-0.05in}

We design a simple-yet-effective beam search to find the best matched candidate facts. For every head-tail pair $(h, t)$ in a sentence, the search maintains the $k$-best matched candidate facts of the pair. Let's first consider the search from left to right with beam size equals to 1. An example search process is shown in Figure~\ref{fig:search}. Given a head-tail pair {\sl (Dylan, songwriter)}, at each step, the search performs one of the following actions:
\begin{description*}
\vspace{-0.1in}
\item [\texttu{start}] the search from the head. The head $h$ is added as an initial candidate into the beam. For simplicity, we use $\texttu{start}(h)$ to denote the action, which returns a candidate $(h,$. In Figure~\ref{fig:search}(a), at step 0, the head ``Dylan'' is added as {\sl (Dylan, } into the beam. The matching degree is initialized to 0.
\item [\texttu{yield}] a new intermediate candidate in the beam if the current candidate has not reached the tail. The next largest attended token (with the largest score from the attention matrix) is appended to the end of the current candidate to yield the new candidate. The corresponding matching degrees are increased by the associated attention scores. At step 1 (orange arrow in Figure~\ref{fig:search}(a)), ``is'' is appended to the current candidate to yield {\sl (Dylan, is, }, since ``is'' has the largest attention score with ``Dylan'' in the attention matrix. The attention score is 0.3 as highlighted in orange in Figure~\ref{fig:search}(b). The multi-head attention is reduced to a single head so that every two tokens of the sentence are associated with one attention weight. We experiment with different reduction setups in Sec.~\ref{sec:para}. ``x'' marks the tokens (prior to the current token) that are not considered in the search to prevent searching backward. Step 2 similarly takes $\texttu{yield}$ action to produce {\sl (Dylan, is songwriter, }. We use $\texttu{yield}(c,s,\mathbf{A}_s)$ to denote the action, where $c$ is a current candidate, $s$ represents the sentence, and $\mathbf{A}_s$ is the attention matrix from the forward pass of the pre-trained LM over $s$, which yields a new candidate.
\item [\texttu{stop}] the search step if the candidate has reached the tail, then add the candidate as a valid candidate fact into the beam. As beam size equals to 1, {\sl (Dylan, is, songwriter)} is the only returned candidate fact for the given pair. We denote this step using $\texttu{stop}(c,t)$, which returns a valid fact.
\end{description*}

\vspace{-0.05in}
The details of the proposed beam search are in Algorithm~\ref{alg:beamsearch}. The inputs of the search algorithm are a head-tail pair $(h,t)$, a sentence $s$, an attention matrix $\mathbf{A}_s$ of $s$. Both $h$ and $t$ are identified by the noun chunk in $s$. $\mathbf{A}_s$ is the attention matrix associated with $s$ from the forward pass of LM without fine-tuning. The search gets started by adding the head $h$ as the initial candidate in the beam (line 1). While there are still new candidates waiting to be yielded (line 2), the search continues, and the top $k$ candidates sorted by the matching degrees are maintained (line 3-11) in the beam. In practice, we implement an action manager $\mathcal{O}$ to decide which action to take at each step. Given a candidate $c$ in the beam, $\mathcal{O}(c) = \texttu{start}$ always happens at the beginning of the search. If $c$ has not reached the tail $t$ yet, $\mathcal{O}(c) = \mathbf{\texttu{yield}}$. Otherwise, $\mathcal{O}(c) = \texttu{stop}$. We also notice some facts are in reverse order in the sentence, e.g., {``$\cdots$ said Jason Forcier , a vice president at battery maker A123 Systems Inc.''} for facts of relation {``org:top\_members\_employees''}, thus enable bidirectionality by running the algorithm in both directions (left to right and right to left). The beam search is implemented by the breadth-first search, which is efficient as the time complexity is $O(k \cdot d)$, where $d$ is the maximum depth of the search tree.

\begin{algorithm}[tb]
\small
\vspace{-0.05in}
\caption{\label{alg:beamsearch}{Beam search for matching candidate facts.}}
\begin{algorithmic}[1]
\Require{Head-tail pair $(h,t)$, sentence $s$, attention matrix $\mathbf{A_s}$, action manager $\mathcal{O}=\{\texttu{start}, \texttu{yield}, \texttu{stop}\}$, beam size $k$}
\Ensure{Candidate facts $\mathbb{T}_{(h,t)}$}
\State $\mathbb{T}_{(h,t)} \leftarrow \{\texttu{start}(h)\}$
\Comment{{\em Start} by adding the head as a candidate in the beam}
\While{$\exists c \in \mathbb{T}_{(h,t)} [\mathcal{O}(c) = \texttu{yield}]$}
\State $\widetilde{\mathbb{T}}_{(h,t)} \leftarrow \emptyset$
\Comment{Initialize a new beam}
\ForAll{$c \in \mathbb{T}_{(h,t)}$}
\If{$\mathcal{O}(c) = \texttu{yield}$}
\State $\widetilde{\mathbb{T}}_{(h,t)} \leftarrow  \widetilde{\mathbb{T}}_{(h,t)} \cup \{\texttu{yield}(c,s,\mathbf{A}_s)\}$ \Comment{{\em Yield} a new candidate if not reached the tail}
\Else
\State $\widetilde{\mathbb{T}}_{(h,t)} \leftarrow  \widetilde{\mathbb{T}}_{(h,t)} \cup \{\texttu{stop}(c,t)\}$ \Comment{{\em Stop} then produce a valid fact if reached the tail}
\EndIf
\EndFor
\State $\mathbb{T}_{(h,t)} \leftarrow {\rm TOP}(k, \widetilde{\mathbb{T}}_{(h,t)})$ \Comment{Maintain $k$-best candidates in the beam}
\EndWhile
\State \Return $\mathbb{T}_{(h,t)}$
\end{algorithmic}
\end{algorithm}

\vspace{-0.05in}
\subsubsection{Filter}
\label{sec:filter}
\vspace{-0.05in}

Although the basic functionality provided by beam search is sufficient for finding useful candidate facts, we have found a few constraints useful. Given a candidate fact $(h,r,t)$ from beam search result $\mathbb{T}_{(h,t)}$, it remains as a fact if satisfying all the following constraints.

\vspace{-0.05in}
{\bfseries Constraint \#1 \ } The matching degree of $(h,r,t)$ is above a threshold. We compare the matching degrees corpus-wide to only reserve the facts that are matched better with the knowledge in LMs. For example, \textsc{MaMa}\ extracts a fact {\sl (Rolling Stone, wrote, pop song)} from {``Rolling Stone wrote: ``No other pop song has so thoroughly challenged artistic conventions''''}, which is not an accurate fact based on the sentence. We observe the associated matching degree is below a proper threshold, while the matching degrees of high-quality facts from the same documents, e.g., {\sl (Dylan, is, songwriter)}, or confident facts from the other documents are beyond the threshold.

\vspace{-0.05in}
{\bfseries Constraint \#2 \ } The distinct frequency of $r$ is above a threshold. To avoid facts to be over-specified, e.g., {\sl (Dylan, signed to Sam Peckinpah's film, Pat Garrett and Billy the Kid)}, we require $r$ should take many distinct head-tail pairs in the corpus.

\vspace{-0.05in}
{\bfseries Constraint \#3 \ } Relation $r$ is a contiguous sequence in the sentence. We can avoid $r$ that has no meaningful interpretation~\citep{fader2011identifying}, e.g., {\sl (Rolling Stone, wrote challenged, conventions)} from the above sentence.

\vspace{-0.05in}
\subsection{Map}
\label{sec:kd}

The objective of Map\ stage is to generate an open KG. The open KG contains (a) {\em mapped facts} in a KG schema (Sec.~\ref{sec:ground}), e.g., Wikidata schema, if the schema of the candidate facts is within the existing KG schema; and (b) {\em unmapped facts} from (a) in an open schema (Sec.~\ref{sec:openschema}).

\vspace{-0.05in}
\subsubsection{Mapped Facts in KG Schema}
\label{sec:ground}

The goal is to map a candidate fact $(h, r, t)$ to a fact $(h_k, r_k, t_k)$ in the KG schema. The reason for mapping to an existing KG schema is to make use of the high-quality schema designed by experts (to avoid duplicated efforts of building from scratch) and enable evaluating the candidate facts with oracle KG facts contributed by human volunteers. We first map both entities $h, t$ to $h_k, t_k$, then map the relation $r$ to $r_k$ in the reference KG schema.

\vspace{-0.05in}
{\bfseries Entity linking to KG schema \ } We leverage an unsupervised entity linker based on a mention-to-entity dictionary~\citep{spitkovsky2012cross} to link the entities for scalability consideration. Besides, contextual information is crucial to link the entities correctly, we use the word embedding of the context to disambiguate the entities, which means we only link the entities with high contextual similarities based on the word embedding. We adopt the entity linker to map $h, t$ to $h_k, t_k$.

\vspace{-0.05in}
{\bfseries Relation mapping with KG schema \ } We largely follow the relation mapping method proposed by \cite{angeli2015leveraging} to construct an offline relation map between KG relation and relation phrases of the candidate facts. The basic idea is that the more often linked head-tail pairs (i.e., entities are with type information from the entity linking step) co-occur between the candidate facts and KG facts, the more likely the corresponding relations are mapped to each other. In addition, we normalize each relation phrase of the candidate facts by lemmatization, and removing inflection, auxiliary verbs, adjectives, adverbs. After the relation mapping is constructed, one author manually checks whether the top 15 relation phrases are true mappings for each KG relation. We only reserve the true ones in the final relation mapping. This process takes approximately one day. Later, $r$ is mapped to $r_k$ with an efficient look-up operation in the relation mapping.

\vspace{-0.05in}
\subsubsection{Unmapped Facts in Open Schema}
\label{sec:openschema}

An unmapped candidate fact $(h, r, t)$ means at least one of $h$, $r$, and $t$ is not mapped to the KG schema based on the method in Sec.~\ref{sec:ground}. There are two types of unmapped candidate facts:

\vspace{-0.05in}
{\bfseries Partially unmapped facts} represent at least one of $h$, $r$, and $t$ are mapped to the KG schema. It can be $h$ or $t$ mapped to $h_k$ or $t_k$ based on the entity linker in Sec.~\ref{sec:ground}. It can also be $r$ that mapped to $r_k$ using the relation mapping in Sec.~\ref{sec:ground}. This actually results in unmapped facts that are in a mixture of the KG schema and the open schema. As the overall schema of the unmapped facts is not in the KG schema, we use open schema to denote such unmapped facts in the rest of the paper for simplicity. An example is {\sl (Dylan, signed, Albert Grossman)} in Figure~\ref{fig:overview}, where both head and tail are linked to Wikidata schema based on the entity linker in Sec.~\ref{sec:ground}, but the relation cannot be mapped since there is no relation mapping from ``signed'' to a KG relation in Wikidata schema.

\vspace{-0.05in}
{\bfseries Completely unmapped facts} indicate all $h$, $r$, and $t$ are not mapped to the KG schema. This means neither the entity linker nor the relation mapping is able to map $h$, $r$, and $t$ to $h_k$, $r_k$, $t_k$ respectively. The resulting unmapped candidate facts stay in the open schema, e.g., a candidate fact {\sl (Jacob, was, A Registered Mennonite)} stays the same in the open schema from a sentence ``Jacob was a registered Mennonite in Amsterdam.''.

The resulting open KG is a new type of KG that mixes the fixed KG schema with the flexible open schema, suggesting new directions for the next generation of KGs. The open KG not only contains existing knowledge in the reference KGs, but also extends the fixed KG schema with an additional open schema to improve the coverage of the KGs, that benefits the downstream KG based applications, e.g., QA and commonsense reasoning~\citep{wang2019superglue,brown2020language}.

\section{Conclusion}
\vspace{-0.15in}
We show that the knowledge graphs can be constructed by a single forward pass of the language models over textual corpora. We propose a two-stage unsupervised approach \textsc{MaMa}\ to first match the facts in the corpus with the internal knowledge of the language model, and then map the matched facts to produce a knowledge graph. We demonstrate the quality of the resultant open knowledge graphs by comparing to two knowledge graphs (Wikidata and TAC KBP). The open knowledge graph also features new facts in the open schema, which could have broad implications for knowledge graphs and their downstream applications. The results also suggest that larger language models store richer knowledge than existing knowledge graphs, and generating on even larger high-quality text corpora could continue improving knowledge graphs. Additionally, the knowledge graphs generated by our approach can help researchers to look into what the language models learn, so our interpretable knowledge graphs establish a bridge between the deep learning and knowledge graph communities.

\section{Experiments}
\label{sec:exp}
\vspace{-0.05in}

{\it How well can language models generate knowledge graphs?} We experimentally explore how well can \textsc{MaMa}\ answer the question in the section. To measure the ability of LMs in generating KGs, we directly measure the quality of resulting open KGs. The open KG (see an example in Figure~\ref{fig:kg}) contains two types of facts: mapped facts in the fixed KG schema; and unmapped facts in the open schema. We first quantitatively evaluate \textsc{MaMa}\ by comparing the mapped facts to oracle KGs annotated by humans in Sec.~\ref{sec:mapped}, then conduct an in-depth analysis of the unmapped facts in Sec.~\ref{sec:unmapped}.

\begin{figure*}
    \centering
    \includegraphics[width=0.95\textwidth]{figs/kgv3.pdf}
    \vspace{-0.05in}
    \caption{{\small A snapshot subgraph of the open KG generated by \textsc{MaMa}\ using BERT$_{\rm LARGE}$ from Wikipedia pages neighboring ``Bob\_Dylan''. The blue node and arrow represent the mapped facts in the Wikidata schema, while the yellow node and arrow denote the unmapped facts in the open schema. We also visualize the correct facts that are new in Wikidata in yellow. }
      \label{fig:kg}}
\end{figure*}

\vspace{-0.05in}
\subsection{Results on Mapped Facts}
\label{sec:mapped}
\vspace{-0.05in}

We first study the quality of the mapped facts. As the candidate facts have been mapped to the schema of oracle KGs, we are able to quantitively compare the candidate facts with the oracle facts in the reference KGs.

\vspace{-0.1in}
\subsubsection{Datasets}
\vspace{-0.05in}

We compare the mapped facts from \textsc{MaMa}\ with the facts in two KGs:

\vspace{-0.05in}
{\bfseries TAC KBP \ } TAC Knowledge Base Population (KBP) Slot Filling is a task to search a document collection to fill in the tail/object entity for predefined relations (slots) for a given head/subject entity in a reference KG. We experiment with the reference KG in the 2013 challenge. We use the document collection and oracle facts of the 2013 task. The statistic of the dataset is shown in Table~\ref{tab:data}.

\vspace{-0.05in}
{\bfseries Wikidata \ } We use popular Wikidata as another KG. We use all the oracle facts in Wikidata. We use the English Wikipedia as the text corpus, since a large amount of facts in Wikidata is from English Wikipedia. The statistic is in Table~\ref{tab:data}.

\begin{table}[t]
\centering
\resizebox{0.4\linewidth}{!}
  {
  \begin{tabular} {c | c | c }

        \toprule
    {\bf KG} & {\bf \# of oracle facts} & {\bf \# of documents} \\
    \hline

    TAC KBP                              & 27,655 \footnoteref{ft:anydoc}                             & 3,877,207
    \\
    Wikidata                      & 27,368,562                    & 6,047,494            \\
    \bottomrule
  \end{tabular}
  }
  \vspace{-0.05in}
\caption{{\small Dataset statistics of two knowledge graphs: TAC KBP and Wikidata. TAC KBP refers to TAC KBP Slot Filling 2013 challenge. \# of oracle facts for TAC KBP is the number of oracle facts in the 2013 task. \# of documents for TAC KBP is the number of the documents in the 2013 task. \# of oracle facts for Wikidata is the total number of oracle facts in Wikidata. \# of documents for Wikidata is the size of English Wikipedia.}}
\label{tab:data}
\end{table}

To evaluate the mapped facts, we first use Match\ stage of \textsc{MaMa}\ to run over the corresponding documents to generate the candidate facts. Then Map\ stage is leveraged to map the candidate facts to the schema of TAC KBP and Wikidata respectively. The parameter settings, such as beam size in Algorithm~\ref{alg:beamsearch}, are shared across TAC KBP and Wikidata based on the parameter study in Sec.~\ref{sec:para}.

\vspace{-0.05in}
\subsubsection{TAC KBP}
\vspace{-0.05in}

\begin{table}[t]
\centering
\resizebox{0.65\linewidth}{!}
  {
  \begin{tabular} {l | c | c | c}
    \toprule
    {\bf Method} & {\bf Precision\%} & {\bf Recall\%} & {\bf F1\%} \\
    \hline

OpenIE 5.1 \footnoteref{ft:openie51}                            & 56.98                             & 14.54                              & 23.16                                                \\
Stanford OpenIE \citep{angeli2015leveraging}                             & 61.55                             & 17.35                              & 27.07                                                \\
 \hdashline
\textsc{MaMa}-BERT$_{\rm BASE}$     (\electricblue{\small ours})                                  & 61.57                             & 18.79                              &  28.79                                               \\
\textsc{MaMa}-BERT$_{\rm LARGE}$ (\electricblue{\small ours})                        & 61.69                   & 18.99                     & 29.05                  \\
\textsc{MaMa}-GPT-2 (\electricblue{\small ours})                                    &      61.62                        & 18.17                             & 28.07                                              \\
\textsc{MaMa}-GPT-2$_{\rm MEDIUM}$ (\electricblue{\small ours})                                    & 62.10                             & 18.65                             &   28.69                                             \\
\textsc{MaMa}-GPT-2$_{\rm LARGE}$ (\electricblue{\small ours})                                    & 62.38                            & 19.00                             &  \cellcolor[gray]{0.92} 29.12                                              \\
\textsc{MaMa}-GPT-2$_{\rm XL}$ (\electricblue{\small ours})                                    & 62.69                            & 19.47                             &   \bf\cellcolor[gray]{0.85} 29.72                                              \\
    \bottomrule
  \end{tabular}
  }
  \vspace{-0.05in}
\caption{{\small Compare the quality of mapped facts on TAC KBP.}}
\label{tab:resulttac}
\end{table}

To verify the ability to produce correct facts, we compare candidate facts from \textsc{MaMa}\ to the outputs of two open information systems, which also produce triplets in the form of $(h, r, t)$.

\vspace{-0.05in}
{\bfseries Stanford OpenIE} leverages POS tag and dependency parser, and generates self-contained clauses from long
sentences to extract the triplets, which is the best open information extraction system~\citep{angeli2015leveraging} on TAC KBP~\citep{surdeanu2013overview}. After collecting the triplets from the system, we use the same Map\ procedure with \textsc{MaMa}\ (Sec.~\ref{sec:ground}) to map the triplets to the corresponding KG schema.

\vspace{-0.05in}
{\bfseries OpenIE 5.1}~\footnote{\label{ft:openie51}\tiny\url{https://github.com/dair-iitd/OpenIE-standalone}} is the successor to Ollie~\citep{schmitz2012open}, and it improves extractions from noun relations, numerical sentences, and conjunctive sentences depending on the linguistic patterns. Similarly, the same Map\ procedure (Sec.~\ref{sec:ground}) is leveraged to map the triplets from the system to the KG schema.

We use two families of pre-trained LMs with \textsc{MaMa}.
We use BERT$_{\rm BASE}$ and BERT$_{\rm LARGE}$ from \cite{devlin2018bert} with \textsc{MaMa}, namely {\bf \textsc{MaMa}-BERT$_{\rm BASE}$} and {\bf \textsc{MaMa}-BERT$_{\rm LARGE}$}.
Besides, GPT-2s from \cite{radford2019language} are used, i.e., {\bf \textsc{MaMa}-GPT-2}, {\bf \textsc{MaMa}-GPT-2$_{\rm MEDIUM}$}, {\bf \textsc{MaMa}-GPT-2$_{\rm LARGE}$}, and {\bf \textsc{MaMa}-GPT-2$_{\rm XL}$}.

Table~\ref{tab:resulttac} shows the results on TAC KBP. We use the official scorer of TAC KBP Slot Filling 2013 to evaluate precision, recall, and F1 on TAC KBP~\footnote{\label{ft:anydoc}\tiny{There are 2,383 correct oracle facts based on the ``manual runs'' assessment in TAC KBP.}}.

\vspace{-0.05in}
{\bfseries \textsc{MaMa}\ constructs improved KGs}. From the results, we find that all our methods achieve competitive precision, which is greater than 60\%, given the unsupervised nature of \textsc{MaMa}. All the proposed methods outperform the two open IE systems. This shows that \textsc{MaMa}\ is able to produce high-quality knowledge directly from pre-trained LMs by a single forward pass without human supervision. The results show the effectiveness of \textsc{MaMa}\ in generating candidate facts from Match\ stage, and producing high-quality KGs through Map\ stage. We also find that \textsc{MaMa}-GPT-2$_{\rm XL}$ performs the best. \textsc{MaMa}-GPT-2$_{\rm XL}$ outperforms the previous state-of-the-art Stanford OpenIE by over 2.6\% in F1. This shows the proposed end-to-end \textsc{MaMa}\ is able to recover the knowledge stored in pre-trained LMs without relying on any extra linguistic features, such as POS tag and dependency parser used in open IE systems. The main reason leading to the moderate results of OpenIE 5.1 is that the system generates objects of the triplets with extraneous words, which hurt the performance in slot filling tasks. Even though the proposed methods all outperform the two open IE systems in the recall, however improving recall is clearly the future direction to further improve the performance of \textsc{MaMa}. We find that the main cause of the moderate recall is the incorrect entities caused by spaCy noun chunk as summarized in Sec.~\ref{sec:error}.

\vspace{-0.05in}
{\bfseries Larger/deeper LMs produce KGs of higher quality}. BERT$_{\rm LARGE}$ outperforms BERT$_{\rm BASE}$ since the doubling parameter size. GPT-2s share similar trends, where we observe performance increases when the model size increases. This complies with our intuition on more knowledge is stored in deeper and larger models. Such increases in performance seem subtle on TAC KBP, we find this might due to the relatively small number of oracle facts by noticing a more significant improvement on Wikidata in Sec.~\ref{sec:wiki}. We plan to further improve the results with larger pre-trained LMs, e.g., GPT-3~\citep{brown2020language}, Megatron-LM~\citep{shoeybi2019megatron}.

\vspace{-0.05in}
{\bfseries BERT LMs outperform GPT-2 LMs under similar model sizes}. More specifically, BERT$_{\rm BASE}$ performs better than \textsc{MaMa}-GPT-2 in F1, and \textsc{MaMa}-BERT$_{\rm LARGE}$ outperforms \textsc{MaMa}-GPT-2$_{\rm MEDIUM}$ in F1. BERT$_{\rm BASE}$ and \textsc{MaMa}-GPT-2 are similar in size, while \textsc{MaMa}-BERT$_{\rm LARGE}$ and \textsc{MaMa}-GPT-2$_{\rm MEDIUM}$ are similar in model size as well. This is mainly because that the recall of BERT LMs is higher than that of corresponding GPT-2 LMs. The result indicates that the Cloze-style loss function (i.e., masked language model) of BERT is more effective and flexible in recovering more knowledge than the autoregressive LM objective. We also notice that the precision of GPT-2 LMs is higher than that of according BERT LMs. The reason is that the autoregressive LM objective captures more accurate knowledge than Cloze-style loss does by not introducing extra noise (e.g., masked tokens) during pre-training.

\vspace{-0.05in}
\subsubsection{Wikidata}
\label{sec:wiki}
\vspace{-0.05in}

We select our best BERT based method \textsc{MaMa}-BERT$_{\rm LARGE}$, and GPT-2 based method \textsc{MaMa}-GPT-2$_{\rm XL}$ on TAC KBP to compare with Stanford OpenIE (the best open IE system on TAC KBP) for scalability experiments on Wikidata. We follow the same definition as the slot filling task to calculate precision, recall, and F1 on Wikidata. Table~\ref{tab:resultwiki} summarizes the results.

\vspace{-0.05in}
{\bfseries \textsc{MaMa}\ is scalable to larger corpora}. Similar to the trends on TAC KBP, \textsc{MaMa}-GPT-2$_{\rm XL}$ performs the best in precision, recall, and F1. The results show the effectiveness of \textsc{MaMa}\ in generating candidate facts and high-quality KGs. We also find that \textsc{MaMa}-GPT-2$_{\rm XL}$ outperforms \textsc{MaMa}-BERT$_{\rm LARGE}$ by over 1\% in F1. This shows that the larger model (GPT-2$_{\rm XL}$ has 5x more parameters compared to BERT$_{\rm LARGE}$) contains more knowledge, and \textsc{MaMa}\ is able to restore the knowledge. When larger or deeper models (e.g., GPT-3) are used with \textsc{MaMa}, we can expect more gains of the KG quality. Thanks to the efficient nature of \textsc{MaMa}, which relies only on the forward pass of the LMs without fine-tuning, the results suggest that \textsc{MaMa}\ is scalable to large KGs.

\vspace{-0.05in}
{\bfseries Larger corpora embed more complete KGs}. In particular, \textsc{MaMa}-GPT-2$_{\rm XL}$ outperforms Stanford OpenIE by 5.6\% in F1. \textsc{MaMa}-BERT$_{\rm LARGE}$ outperforms Stanford OpenIE by approximately 4.4\% in F1. Both F1 gains are larger compared to that on TAC KBP. This is because that the LMs contain world knowledge from pre-training corpora, e.g. Wikipedia and Common Crawl. The larger the textual corpora are, the more knowledge our method is able to recover and match to the knowledge stored in LMs. The finding is particularly important, since we are now able to construct larger KGs of high quality from scratch when larger datasets are used, such as WebText2 and Common Crawl~\citep{abs-1910-10683,brown2020language}. Similar to the observations on TAC KBP, the precision is higher compared to recall. Wikidata is not fully built from Wikipedia, \textsc{MaMa}\ could improve the recall by running on those larger corpora to collect more facts.

\begin{table}[t]
\centering
\resizebox{0.65\linewidth}{!}
  {
  \begin{tabular} {l | c | c | c }
    \toprule
    {\bf Method} & {\bf Precision\%} & {\bf Recall\%} & {\bf F1\%} \\
    \hline

    Stanford OpenIE \citep{angeli2015leveraging}                              & 23.32                             & 13.09                              & 16.77
    \\
    \hdashline
    \textsc{MaMa}-BERT$_{\rm LARGE}$ (\electricblue{\small ours})                       & 29.52                    & 16.56                    & \cellcolor[gray]{0.92} 21.22                 \\
    \textsc{MaMa}-GPT-2$_{\rm XL}$ (\electricblue{\small ours})                       & 31.32                    & 17.42                    & \bf\cellcolor[gray]{0.85} 22.39                \\
    \bottomrule
  \end{tabular}
  }
  \vspace{-0.05in}
\caption{{\small Compare the quality of mapped facts on Wikidata.}}
\label{tab:resultwiki}
\end{table}

\vspace{-0.05in}
\subsection{Analysis of Unmapped Facts}
\label{sec:unmapped}
\vspace{-0.05in}

As illustrated in Figure~\ref{fig:kg}, the open KG constructed by \textsc{MaMa}\ is a new type of KG combining the fixed KG schema with the flexible open schema. We turn to study the quality of the candidate facts that are not mapped to the above reference KG schema, but are in the open schema generated by \textsc{MaMa}. We manually judge such unmapped facts generated by our best method \textsc{MaMa}-GPT-2$_{\rm XL}$ from 100 sampled documents in Wikidata and TAC KBP respectively.

\vspace{-0.05in}
{\bfseries The quality of unmapped facts is verified}. We find 35.3\% of the unmapped facts are true on Wikidata. We find 83.2\% of those true facts are partially unmapped facts as defined in Sec.~\ref{sec:openschema}, e.g., {\sl (Bob\_Dylan.Q392, tour\_with, the\_Grateful\_Dead.Q212533)}, whose relation is not within the schema of Wikidata, while both head and tail are in the schema. The remaining true facts are completely unmapped facts (Sec.~\ref{sec:openschema}), e.g., a candidate fact {\sl (Jacob, was, A Registered Mennonite)} stays the same in the open schema.

\vspace{-0.05in}
{\bfseries Accurate entity detection is desired}. We also notice 45.5\% of the untrue unmapped facts on Wikidata are due to the {\em incorrect entities} detected by the spaCy. {\em Incorrect or missing entity linking} (to either head or tail) in Sec.~\ref{sec:ground} causes additional {9.1\%} errors in the unmapped facts. {4.5\%} of the untrue unmapped facts are caused by the {\em missing relation mapping} in Sec.~\ref{sec:ground}. The rest errors made by \textsc{MaMa}-GPT-2$_{\rm XL}$ are {\em incorrect relation phrases}, such as uninformative relation phrases, e.g., {\sl (Dylan, made, his breakthrough)}, which is similar to the errors made by open IE systems~\citep{fader2011identifying}. Both entity linking and relation mapping of Map\ stage rely heavily on the accuracy of entity detection from the spaCy noun chunk. We conclude that the main root cause of the untrue unmapped facts is due to the errors made by the spaCy noun chunk.

We observe similar trends on TAC KBP. We plan to leverage crowdsourcing platforms, e.g., Mechanical Turk, to conduct quantitative evaluations over the unmapped facts to better understand the strengths and shortage of \textsc{MaMa}. We plan to identify more accurate entities by relying on attention weights in LMs~\citep{clark2019does,hewitt2019structural} instead of using extra resources. We will also investigate stronger entity linkers~\citep{kolitsas2018end} and learn a more robust relation mapping through weak or distant supervision~\citep{mintz2009distant,ratner2017snorkel}. We will investigate more sophisticated approaches, such as graph neural networks~\citep{kipf2016semi}, to generate more accurate relation phrases from the attention weight matrices by considering structural information.

\section{Introduction}
Knowledge graphs (KGs) are an important resource for both humans and machines. Factual knowledge in KGs is injected into AI applications to imitate important skills possessed by humans, e.g., reasoning and understanding. KG construction is mainly supervised, requiring humans to handwrite every fact, such as Freebase~\citep{bollacker2008freebase} and Wikidata. KGs can also be constructed in a semi-supervised way, in which a semi-automatic extractor is used to obtain the facts from web corpora (e.g., NELL~\citep{carlson2010toward} and Knowledge Vault~\citep{dong2014knowledge}). Humans however still need to interact with the extractor to improve the quality of the discovered facts. Therefore, human supervision, which is often expensive, is required in constructing KGs.

Recent progress in language models (LMs), such as BERT~\citep{devlin2018bert} and GPT-2/3~\citep{radford2019language,brown2020language}, has led to superior results even outperforming humans in a wide range of tasks, e.g., sentence classification~\citep{wang2018glue}, question answering~\citep{brown2020language}. Pre-trained LMs are also capable to write poetry, music, and code, while such tasks often require we human to spend a significant amount of time in learning the relevant knowledge to work well. In fact, these pre-trained LMs automatically acquire factual knowledge from large-scale corpora (e.g., BookCorpus~\citep{zhu2015aligning}, Common Crawl~\citep{brown2020language}) via pre-training. The learned knowledge in pre-trained LMs is the key to the current success. We therefore consider the following question: instead of using the manually created knowledge, \emph{can we use the knowledge stored in pre-trained LMs to construct KGs?}

\begin{figure*}
    \centering
    \includegraphics[width=1.0\textwidth]{figs/overall.pdf}
    \vspace{-0.05in}
    \caption{{\small Overview of the proposed approach \textsc{MaMa}. \textsc{MaMa}\ constructs an open knowledge graph (KG) with a single forward pass of the pre-trained language model (LM) (without fine-tuning) over the corpus. Given the input: a textual corpus containing passages and sentences, e.g., English Wikipedia, and a pre-trained LM, e.g., BERT, GPT-2/3, \textsc{MaMa}\ (1) generates a set of candidate facts via {\it matching} the knowledge in the pre-trained LM with facts in the textual corpus, e.g., a candidate fact {\sl (Dylan, is, songwriter)} from the sentence {``Dylan is a songwriter.''}, and (2) produces an open KG by {\it mapping} the matched candidate facts to both an existing KG schema, e.g., {\sl (Bob\_Dylan.Q392, occupation.P106, Songwriter.Q753110)} in Wikidata schema, and an open schema, e.g., {\sl (Bob\_Dylan.Q392, sign, Albert\_Grossman.Q708584)}.}
      \label{fig:overview}}
\end{figure*}

In this paper, we design an unsupervised approach called \textsc{MaMa}\ that successfully recovers the factual knowledge stored in LMs to build KGs from scratch. \textsc{MaMa}\ constructs a KG with a single forward pass of a pre-trained LM (without fine-tuning) over a textual corpus. As illustrated in Figure~\ref{fig:overview}, \textsc{MaMa}\ has two stages: Match\ and Map.
{\bf Match}\ stage generates a set of candidate facts by matching the facts in the textual corpus with the knowledge in the pre-trained LM. General or world knowledge from large-scale corpora is embedded in the LM, thus candidate facts in the target corpus are often covered by the knowledge in the LM. The candidate facts are matched through an efficient beam search in the attention weight matrices of the pre-trained LM without fine-tuning.
{\bf Map}\ stage produces an open KG via mapping the matched candidate facts from Match\ stage to both fixed KG schema and open schema. If the schema of candidate facts exists in the KG schema, we map the candidate facts directly to the fixed KG schema. Otherwise, we reserve the unmapped candidate facts in the open schema.
This results in a new type of KG, {\em open KG}, with a mixture of mapped facts in fixed KG schema and unmapped facts in the open schema.

Our contributions are as follows:
\begin{enumerate*}
\vspace{-0.12in}
    \item We show {\it how to construct KGs from pre-trained LMs}. The KGs are constructed with a single forward pass of the pre-trained LMs without fine-tuning over the textual corpora. This helps researchers explicitly understand what the language models learn, bridging the deep LM and KG communities through enhanced model transparency.
    \item We propose an unsupervised two-stage approach, \textsc{MaMa}, to first {\em match} the candidate facts in the corpora with the knowledge stored in LMs, then {\em map} the matched candidate facts to both fixed and open schema to produce a KG.
    \item We generate a new type of KG, namely open KG, consists of {\em mapped facts} in the fixed KG schema of existing KGs (Wikidata and TAC KBP) annotated by humans; and {\it unmapped facts} in the open schema that are new in the reference KG schema. The reach of this result is broad and has downstream utility for knowledge graph construction, deep neural network interpretation, and information extraction.
\end{enumerate*}

\section*{Acknowledgements}
We would like to thank Xinyun Chen, Yu Gai, Dan Hendrycks, Qingsong Lv, Yangqiu Song, Jie Tang, and Eric Wallace for their helpful inputs. We also thank Gabor Angeli, Christopher D. Manning for their timely help in replicating the results in \cite{angeli2015leveraging}. This material is in part based upon work supported by Berkeley DeepDrive and Berkeley Artificial Intelligence Research.

\section{Related Work}
\vspace{-0.15in}

{\bf Knowledge graph construction} can be generally categorized into two groups, 1) supervised approaches. Wikidata, Freebase~\citep{bollacker2008freebase}, YAGO~\citep{suchanek2007yago}, YAGO2~\citep{hoffart2013yago2}, DBpedia~\citep{auer2007dbpedia} are built based on human supervision from Wikipedia infoboxes and other structured data sources; 2) semi-supervised approaches. Open information extraction systems, e.g., OLLIE~\citep{schmitz2012open}, Reverb~\citep{fader2011identifying}, Stanford OpenIE~\citep{angeli2015leveraging}, and OpenIE 5.1 \footnoteref{ft:openie51} aim to leverage carefully-designed patterns based on linguistic features (e.g., dependencies and POS tags), to extract triplets from web corpora for open schema KG. Besides, NELL~\citep{carlson2010toward}, DeepDive~\citep{niu2012elementary}, Knowledge Vault~\citep{dong2014knowledge} extract information based on a fixed schema or ontology, where humans help improve the accuracy of the extractions. Probase~\citep{wu2012probase} produces taxonomies instead of rich typed relations in general KGs. \textsc{MaMa}\ instead uses learned knowledge stored in pre-trained LMs without human supervision to construct an open KG, which is a mixture of fixed schema and open schema.

\vspace{-0.05in}
{\bf Language models}, e.g., BERT~\citep{devlin2018bert}, GPT~\citep{radford2018improving}, GPT-2/3~\citep{radford2019language,brown2020language}, ELMo~\citep{peters2018deep}, Transformer-XL~\citep{dai2019transformer}, ALBERT~\citep{abs-1909-11942}, RoBERTa~\citep{abs-1907-11692}, XLNet~\citep{abs-1906-08237} and Megatron-LM~\citep{shoeybi2019megatron} contain factual knowledge obtained via pre-training on large-scale corpora such as Wikipedia and BookCorpus~\citep{zhu2015aligning}. Studies have leveraged the pre-trained LMs as virtual KGs, and show reasonable performance in QA tasks~\citep{dhingra2020differentiable,guu2020realm}, and language modeling~\citep{khandelwal2019generalization}. LMs are further enhanced by KGs~\citep{zhang2019ernie,peters2019knowledge} to improve knowledge-driven tasks. While the existing work utilizes knowledge in an implicit way, the main difference is that our approach explicitly extracts knowledgeable facts from the LMs. Compare to LAMA~\citep{petroni2019language,petroni2020context}, instead of conducting fact recall in cloze statements, \textsc{MaMa}\ directly generates the whole fact in the form of a triplet $(h, r, t)$ from a sentence. Besides, the benchmark datasets used with \textsc{MaMa}\ are larger compared to the LAMA benchmark, e.g., Wikidata is 3 orders of magnitude larger compared to the largest dataset in the LAMA benchmark.

\vspace{-0.05in}
{\bf Neural network interpretation} here specifically refers to pre-trained deep language model analysis. There has been a lot of work to understand what the neural networks learn~\citep{linzen2016assessing,adi2016fine,tenney2019you}. With regards to analyzing Transformer~\citep{vaswani2017attention} based language models (e.g., BERT and GPT-3), substantial recent work focuses on both visualizing and analyzing the attention~\citep{vig2019visualizing,jain2019attention,clark2019does,michel2019sixteen,vig2020bertology,ramsauer2020hopfield,hendrycks2020measuring}. Instead of analyzing or visualizing, we use LMs to generate structured KGs to directly recover what LMs learn from the corpora.

redpajama_set_name: RedPajamaArXiv


If you are having menstrual cycles then you will be given medication to suppress them so that you and your egg donor's cycles are synchronised.
Your egg donor will suppress her ovaries with either a daily nasal spray or daily injection.
Depending on the medical protocol that your clinic has decided is best for your egg donor, the Suppression (or 'Down Regulation') stage could take between two and four weeks before the next stage ('Stimulation') begins.
Alternatively, the Suppression stage might take place at the same time as the Stimulation stage and take 10 to 14 days.
Regular blood tests and scans will take place to see if your egg donor's body is responding to the medication appropriately.
Your egg donor will stimulate her ovaries to make egg follicles with a daily injection of Follicle Stimulating Hormone (FSH) for between 10 and 14 days.
Your egg donor will have regular scans and blood tests to see how her body is responding to the medication, including how many egg follicles and their sizes.
While your egg donor is taking her medication to stimulate her ovaries, you will be given medication to thicken the lining of your uterus so that it's in the best possible condition to receive embryos. You will also be scanned to see how it's looking.
When the blood tests and scans indicate that your egg donor's body has responded to the medication and that she has appropriately sized egg follicles, she will be asked to inject herself at a specific time with an 'HCG trigger shot' to induce ovulation.
Your egg donor's eggs will be collected about 36 hours after the 'HCG trigger injection'.
If your male partner is providing his sperm for treatment then he will be asked to produce a fresh sample at the time of egg collection. The embryologist will then prepare your partner's sperm.
If you are using your partner's frozen sperm or a sperm donor then the embryologist will remove the frozen sperm from clinic storage and thaw it.
The sperm is clinically washed and the faster moving sperm will be separated from the slower moving sperm.
Once your egg donor's eggs have been retrieved the embryologist will prepare and mix them with either your partner's sperm or the donor sperm in the laboratory. Depending on what the clinic has advised this will either be traditional IVF where the eggs and sperm are left to meet and fertilise in a petri dish, or ICSI where an embryologist will inject single healthy looking sperm into each of the mature eggs.
Hopefully the eggs will fertilise to become embryos. They will then be regularly monitored by the embryologists and you will be kept up-to-date on their progress.
Between two and five days after egg collection, depending on the number of embryos available and your age, one, two or three embryos will be transferred.
You will find out if you are pregnant about two weeks after the embryos have been transferred either through a home pregnancy urine test or a clinic blood test.


text:
Leave a comment Posted on June 20, 2017 June 20, 2017 Musical, Reviews
Zach's 'In the Heights' is a lively, sizzling start to the summer season
Thanks the world phenomenon that was Hamilton, and an Oscar nomination for his work on Disney's Moana, Lin-Manuel Miranda has quickly become a household name, but before he was treading the boards as a founding father or singing for the president, he was winning over musicals fans with his vibrant, joyous look at life in the one of the most colorful parts of Manhattan with In the Heights. Zach Theatre is now bringing this hit musical to Austin, with a production that would make Miranda proud, full of fire, passion, and the sweltering heat of the New York summer.
In the Heights follows the lives of those living in New York's Washington Heights during one of the hottest parts of the summer. We're first introduced to Bodega owner Usnavi, through whom we meet the myriad figures that call the Heights home, and who take us through one special sweltering New York summer, complete with blackouts, heartbreak, and even violence, but always with the latinx flair this neighborhood is known for. Zach has brought in the big guns to bring their work to life, led by director Michael Balderamma, who cut his teeth as dance captain and original cast member of the original Off-Broadway production, and who has numerous other Broadway hits to his name. He is currently the choreographer for the Chicago production of Hamilton, so having his hand guiding all the action of Zach's In the Heights is a big win. Helping him along the way is musical director, and two-time Emmy award-winner, Allen Robertson, a mainstay of the Austin musical scene, and a well respected composer and producer in his own right. These two icons come together with a talented team to create a lively, powerful night of theatrical thrills.
The Washington Heights of Miranda's In the Heights bursts with energy, full of snappy music and quick choreography, and Balderamma and company do their part to keep the action moving. His actors, and even sets, are in contact motion, never hitting a false step, and with blasts of trumpets and the toe-tapping beat of Latin percussion, the skilled orchestra works to keep the scene hopping. The cast do their part to keep the action moving as well, performing with a vibrant zeal that truly adds a soul and spirit to their lively neighborhood. Add it all up with the sunny lighting and the inspired costume choices, and you have one of the hottest shows Zach's produced in years.
In addition to bringing in some big guns behind the scenes, Zach has called in talent from around the country to fill out its cast. In particular, Alicia Taylor Tomasko shows the skills that made her a New York theatre regular. Here she plays the lovely but harried Vanessa, a woman trying to make her way out of the heights, while always being pulled in by the culture and people of Heights. It also doesn't help that she has caught the eye of our protagonist, Usnavi. She's a woman divided, and Tomasko plays the necessary combination of sassy and strong with aplomb. Her footwork is on point, showing off moves that I've rarely seen the likes of on Austin stages, and her voice is clear and strong, with plenty of passion and fire. A good Vanessa is essential to a good production of In the Heights, and the role is in good hands with Tomasko.
Taking on a role made famous by Lin-Manuel Miranda is no mean feat, so Chicago Theatre native Keith Contreras-McDonald had a lot to live up to. Luckily, thanks to his charm and goofiness, he becomes almost instantly endearing. The musical doesn't give him the standout musical moments of some of his fellow performers, but he carries the piece thanks to his wonderful acting chops. His chemistry with Sarro's Vanessa is always believable, and, in fact, his relationships with all of the cast is solid throughout. He's at his best in the small, more emotion-laden moments, helping to sell the stakes and bring the tears in some of the play's more sombre sequences.
Though the team behind Zach's In the Heights have brought in several big talents to inhabit their characters, one of the play's true delights is seeing how many talented locals fill out the cast. For instance, this critic has enjoyed watching actor Vincent Hooper make his way from background player in Summer Stock performances, to the starring on the big stages of Zach Theatre, and here he takes center stage, never feeling out of place among the more seasoned talent. Indeed, his performance as Benny is one of the most emotionally honest in the piece, as he takes us on one of the most full character arcs in the piece. Whether bringing humor or pathos, Hooper proves himself a capable performer, and is living proof that Austin talent can stand toe-to-toe with the that of New York or Chicago.
A pleasant surprise came from another local performer, and California transplant, Christina Oeschger, who wows from her first notes, showcasing a voice like polished glass: smooth, clear, and full brilliance. She brings out the intelligence of her character, a bright young girl having trouble facing the world outside Washington Heights, and sells this from her very stance and diction. This is combined with an innocence in her eyes that charms the audience quickly, which is only amplified once she belts her first note, putting her stunning voice on full display. It's easy to see that there's a bright future in front of Oeschger, and one hopes she finds herself on other Austin stages again soon.
With a clear vision and exuberant passion, Zach plunges audiences headfirst into the wild world of Washington Heights, given some real gravitas thanks to a game production team and an indefatigable group of talented young actors from around the country. It's the perfect kind of crowd-pleasing entertainment that makes for a splendid intro to the summer season, that will have you humming the tunes the whole ride home.
In the Heights is playing at Zach's Topfer Theatre through July 2nd. For more information, and to purchase tickets, visit Zach's website at zachtheatre.org.
Photo courtesy of Kirk Tuck.
Austin Theatre Lin-Manuel Miranda Zach Theatre
Leave a comment Posted on June 16, 2017 April 5, 2018 Reviews
'The Revolutionists' is a funny, but ferocious, feminist fable
Marie Antoinette, Charlotte Corday, and Marianne Angelle meet in the salon of celebrated feminist writer Olympe de Gouges, each of them wanting some piece of writing that will change, or perhaps, end their lives. This is the story behind Lauren Gunderson's original and bold new play, The Revolutionists, given an all-too relevant production by the talented minds of Shrewd Productions, led by the incredibly skilled hand of Rudy Ramirez. Following the final days of several of the French Revolution's strongest women, we see how their fights and struggles are far too similar to the ones women are still fighting today.
One of the smartest moves Lauren Gunderson makes in The Revolutionists is to take these famous historical women, and present them with all of their insecurities, foibles, doubts and emotions. She turns them from figureheads of the revolution, into, simply, women. These people were no different than you and I, and yet, they had the resolve to change the world. As we live in a world today where smart, powerful women are often chastised or ridiculed, if for no other reason than being female, it's inspiring to see these strong women fighting the fight, despite knowing, in many cases, it will mean a meeting with Madame Guillotine.
Taking on the multi-dimensional qualities of these women is no small feet, so these four performers have a lot on their shoulders. Luckily, the four women chosen bring in a combination of charisma, comedic timing, and emotional honesty that truly helps to breathe life into these historical figures. Each of these famous women, in their own way, did their part to define the revolution, and the quadriptych these actors create is a fine piece to behold.
At the heart of the piece is the friendship between playwright Olympe De Gouges, one of the only feminist writers in Revolutionary-era France, and Marianne Angelle, one of the era's most famous Haitian freedom fighters. Having lived through a slave rebellion on a foreign shore, Angelle is a character who has gone through Hell, and Valoneecia Tolbert plays this through a sense of sardonic guardedness. She's a serious, tough woman, who doesn't take guff from anyone, a trait which beats perfectly against de Gouges' manic sarcasm. It's be far too easy to paint Angelle as simply a hard-as-nails revolutionary, but the play smartly gives Angelle moments of real heart, which Tolbert truly makes sing.
Sarah Marie Curry's Olympe de Gouges is a staunch feminists, a warrior for the rights of women and slaves, but she's also a bundle of insecurities. Curry, utilizing her fantastic comedic skill, plays up Olympe's sarcastic attitude, bringing some of the funniest moments of the piece as she bounces off the play's bizarre cast of character. It's obvious, however, that under all the self-deprecating humor, there's plenty of sensitivity and doubt. Her final moments are heartbreaking, and these moments hit so hard, in no small part, because of the journey Curry takes us on, bringing us into the heart of the character with her multifaceted turn.
Making up the final pieces of this fine feminist puzzle are two actors who may not spend as much time on stage, but who make the most of each moment. Gricelda Silva has an amazing ability to elicit emotion from the smallest of movements, so it's no surprise that when she lets loose as Charlotte Corday, her energy bursts all throughout the theatre. She's perfectly cast as the adorable young assassin, her small stature and precocious personality hiding the maliciousness underneath it all. There's a quirky naivete she brings to the role that fits splendidly with Corday, as, even in her final moments, she shows undeniable spunk and fire. Shannon Grounds also brings plenty of energy to her performance as Marie Antoinette, but hers is more grounded in the ditzy cluelessness of the ruined princess. In many plays, a character like this would be played exclusively as a clown, but Gunderson recognizes that Antoinette was so much more: a mother, a lover, and a grieving wife. Indeed, one of the play's most heartbreaking moments stems from Marie finding common ground with another character, who finds herself a sudden widow, where Marie says the simple, but tear-inducing words "it's never going to be all right…but sometimes it feels good to hear someone say it." Grounds finds the complexity beneath Marie's goofiness, creating, by the end, one of the play's most endearing characters.
Despite the action of the play being several centuries removed from our own, it couldn't be more relevant to the world today. As Olympe de Gouges stands before a gathering throng of men near the play's midpoint, asking for equality for women, we're reminded of the marches and meetings today fighting for those same rights, drowned out once again by the gathering mass of male ego and fragility. It's just one of the many elements that make "The Revolutionists" such a strong feminist statement, a statement that deserves to be seen by as large an audience as possible.
Shrewd Productions' The Revolutionists is playing at the Santa Cruz Center for Culture through June 25th. To find more information, and to purchase tickets, please visit Shrew Productions' website at shrewdproductions.com.
Photo courtesy of Errich Petersen.
Leave a comment Posted on June 8, 2017 April 5, 2018 Reviews
Clever direction and a sharp script make for a wild ride in Capital T's 'The Effect'
If chemicals can alter our emotions, how do we know if the things we feel are actually real? This is the question in the heart of Lucy Prebble's "The Effect", the latest from stalwarts Capital T Theatre, whose string of effective, relevant plays continues, this time with the help of one of Austin's most original directors, and recent Austin Critics Table Award winner, Lily Wolff.
"The Effect" takes us into a research facility, in a near-future that could begin anytime now, or could be happening right now, under our own noses. Here, two young people are testing out a new anti-depressant that may have some strange side effects, side effects that could change their lives for the better, or the worse. Humor, passion, anger, violence, all dance in a vigorous tango to create a wild ride, but with firm hands on the reins, and a set of skilled actors at its center, its a ride that's exhilarating to experience.
One of director Lily Wolff's strongest talents lies in her ability to create dynamic situations from simple materials. Though her work in "The Effect" may not quite reach the heights of originality evident when she turned a stairwell into an arena, or a blackbox into a magical forest, as she did in her stunning production of "As You Like It", she still transport us through space and time effortlessly here. Utilizing little more than a white backdrop and two beds, she creates everything from a testing facility, to an abandoned asylum, to a convention stage, making each scene believable along the way. She's aided greatly in this by Austin's king of projection design, Lowell Bartholomee, who's modern, but subtle work helps to add to the near-future aesthetic, while never seeming too overwhelming. It's all built upon nicely by Patrick Anthony's lighting, whose nuanced use of cookies and gels creates intriguing landscapes, while never straying too far from the reality of the situation.
There's something in the marriage of Prebble's fluid writing and Rebecca Robinson's casual delivery that makes every scene she's in pop, which, in the process, creates one of Robinson's best performances. Robinson has always had a unique skill in connecting to the emotional truth of a piece, and here, the wavelengths meet at a near perfect level, as character and performer mesh into one from her earliest scenes. Her movements flow naturally, her interactions with the other characters move effortlessly, and as the play explores her character in more depth, Robinson finds new ways to add dimension to her acting. It all seems so simple as we watch it on stage, but it takes a major talent to create such natural delivery.
"Energetic" is one of the those common phrases you hear bandied about to describe performances, but one actor who embodies the very concept of "energy" is Delante G. Keys in his performance as Tristan. He's lively, bubbly, and incredibly expressive (especially in his face), but the energy he brings is something more than just exuberance. What's really fascinating to watch is how he uses this energy in his darker moments, when he turns it to explosive anger and violence. Keys' is not a prismatic energy, flying off into all directions, but instead a honed energy, that Keys utilizes with laser precision. His anger, his humor, his passion, is direct, precise, utilized in the most effective ways to make each scene sing. He creates some of the funniest moments in the play, but also some of the most heartbreaking, and even some of the hardest to watch. Tristan is a character who experiences his emotions in a very big way, and its a testament to Keys' ability that he can play this hugeness with earnestness and verisimilitude.
Clever direction, a sharp script, and a game cast come together create a piece that's in equal turns powerful, emotional, and humorous. The actors give measured, but strong performances throughout, creating undeniable chemistry with each other, molded under the strong guiding hands of Wolff, who here continues to climb the ranks of Austin directors. Capital T has a winning streak going back several years at this point, and if they have more like "The Effect" in the barrel, I don't see that streak ending anytime soon.
"The Effect" is playing at Hyde Park Theatre through June 17th. For more information, to purchase tickets, visit capitalt.org.
Photo courtesy of Capital T Theatre.
Leave a comment Posted on June 3, 2017 April 5, 2018 Musical, Reviews
Humor and spectacle take center stage in Broadway Across America's 'Something Rotten'
It's becoming increasingly rare to find a true musical comedy on stage. Sure, there's plenty of satire, plenty of serious musicals with funny moments, but the true goofy, ribald, laugh-em-ups are a dying breed. Luckily, Something Rotten! provides just the laughs you're looking for, with a comical look at 1590s England, and two men making a musical about eggs. It's grand spectacle, featuring a skilled cast and some toe-tapping tunes, that, while never exploring major themes, also never takes itself too seriously.
As Something Rotten! begins, we're transported to Renaissance England, where several playwrights of the era are doing their best work, but none more so than the talk of town, William Shakespeare. The bard's shadow looms large over all these writers, and our play follows two of them, Nick and Nigel Bottom, who have the talent and gumption, just never the opportunity to create the next big thing. To find the way to one-up Shakespeare, they enlist the help of a wacky sooth-Sayer named Thomas Nostradamus (who declares he "has just as much talent as his famous uncle"), to discover the next big thing in theatre. What they discover is the strange form of art known as "The Musical", and from there we're taken on a wild ride through the history of musical history, in a daffy downward spiral through songs about the Black Death and eggs, all culminating in a climactic production of "Omelet: The Musical!".
Something Rotten! truly excels with its sense of spectacle. Though at first the play keeps things simple, with quick wit, humor, and goofy pratfalls, but by the time we reach the show-stopping number "A Musical", a wild ride through the history of musical theatre, things ramp to eleven, and we're presented with everything from tap dance to chorus lines to elaborate set changes. This is taken to an even higher level as the show reaches is climax, the absolutely jaw-dropping titular song, which gives us even more references to famous musicals, as well as dancing eggs, cowboys, chimney sweeps, and even an actual recipe for omelets. The energy is kept at high throughout, keeping the audience enthralled through the final ovation.
In the play's early stages, we see Rob McClure's Nick Bottom as a lovable jerk, with his skill for humor on full display, but it's not until the play reached the latter parts of the first act that we truly get to see the talents that made this man a Tony award nominee. When he's allowed to let loose his skills, we find him an incredible tap dancer, a skilled vocalist, and hilarious physical actor, showing him to be a true Renaissance man. The choreography throughout is on point, but McClure's skills cannot be denied, pulling off intricate sequences that most comedic actors wouldn't even attempt. His over-the-top personality fits in perfectly with the rest of the cast, making him the perfect foil to his John Grisetti's softer, more romantic Nigel Bottom, and the perfect companion to the lovably energetic performance by Maggie Lakis as Bea.
We cannot, of course, talk about this production of Something Rotten! without mentioning the star at its center. It was a smart choice to cast a marquee name such as Adam Pascale, a well-known name throughout the world thanks to his performance as Roger in most original productions of "Rent", as the bigger-than-life William Shakespeare. His chops are on full display here, with his singing and dancing second-to-none, but what truly surprises here is his talent for humor. The character he plays is so haughty and grandiose that it requires that the actor give their 200% to it just to make it believable, and Pascal is more than up to the task. He pushes so much energy into his performance at Shakespeare that he bursts at the seams with it, each gesture and phrasing eliciting torrents of giggles throughout the theatre, and when he takes on a disguise to infiltrate the Bottom Brothers' production, things only get more fun. It's a well measured performance, that fits perfectly within the loom of the show to create a colorful piece of the rich tapestry that is Something Rotten!
With so many big names and personalities taking center stage in Something Rotten!, we mustn't forget the fine work being done by its supporting cast. In particular, Blake Hammond as Thomas Nostradamus, the nephew of the real Nostradamus, whose talent for broad humor cannot be understated. He arrives on the scene with the amazing number "A Musical", and brings major laughs with every future appearance, whether it be from his slapstick physicality, or from his frequent non-sequiturs on musical history. Also pulling out nice work is Jeff Brooks as Shylock, who wants to be remembered by Shakespeare as "the really nice Jew", and who brings an equal sense of sarcasm and sweetness that we don't find in many places in the piece, making his appearance in his handfuls of scenes a delight. We should also take a moment to appreciate the work being pulled off by Maggie Lakis as Nick's wife Bea, who wows with her first number "Right Hand Man", and only gets better from there, as she dons male disguises in order to get work in the male-dominated culture in which she lived, creating many comical situations.
There's nothing especially deep or meaningful in Something Rotten!, and that's okay. There's space in every diet for a little fluff and sweetness, and Something Rotten! is that big piece of chocolate cake. A laugh riot, this goofy, shiny, and highly entertaining journey into a wildly inaccurate Elizabethan England brings a talented cast together to care a fine slice of sweet, sweet theatrical pleasure.
Broadway Across America

redpajama_set_name: RedPajamaCommonCrawl


text:
What is the sum of: 10 + 10
Brave and EOS…Going In Opposite Directions
by rollandthomas December 9, 2019 December 9, 2019 055
Brave is a decentralized, open source browser that puts the user first by making online privacy its first priority. Brave also claims to be twice as fast as Chrome on desktop and eight times faster than Safari on mobile.
The man behind the Brave browser is CEO, Brendan Eich. Brendan started his career at Netscape Communications Corporation in April 1995 with the intent to put Scheme "in the browser." In early 1998, Brendan co-founded the Mozilla project. And his most recent initiative was creating the Basic Attention Token (BAT), a cryptocurrency designed for use in the Brave browser when users view and share in the ad revenue. In the ecosystem, advertisers will give publishers BATs based on the measured attention of users. Users will also receive some BATs for participating. They can donate them back to publishers or use them on the platform
If you don't care for the ads, the Brave browser has an inbuilt ad and tracker blocker. The blocker inhibits trackers from learning more about a user for monetization purposes. But Brave is trying to fix the internet by improving the ad model. In the words of founders of Brave "It is a market filled with middlemen and fraudsters, hurting users, publishers and advertisers." And Brave's mission of fixing the internet ad model is gaining traction.
Brave Browser is experiencing robust user growth, gaining another 1.7 million active users over the last three weeks by carving out the privacy niche from Google Chrome.
Since its 1.0 launch mid-November, Brave announced that the open source browser had gained another 1.7 million monthly active users, growing by 19% across all devices. Compared to last year, the numbers are even stronger. The browser's userbase saw a two-fold increase in monthly active users and tripled its daily active users to 3.3 million.
As of today, verified content creators on the platform increased to over 340,000. The majority of these creators publish to YouTube (229,00), followed by Twitter (37,000), business and personal website publishing (38,000), and Twitch (18,000), among others.
A nice level to potentially go long on the BAT is at the daily demand at $0.14.
The EOS blockchain was developed with the aim of facilitating efficient and scalable decentralized applications (dapps). The blockchain includes an operating-system like set of services and functions that works similarly to the ethereum platform.
Despite EOS being the world's seventh-largest blockchain by market cap, my first issues is who in the hell has a year long initial coin offering (ICO)...which raised $4.1 billion in crypto for Block.One. My initial thought was they better deliver due to all the hype.
EOS works on an ownership model whereby users own and are entitled to use resources proportional to their stake, rather than having to pay for every transaction. So, in essence, if you hold N tokens of EOS then you are entitled to N*k transactions. And herein lies the issue.
EOS network's governance issues continue to haunt them as Weiss Ratings recently downgraded the network from B category to C-. Weiss Ratings also posted a Twitter thread explaining why they had to downgrade the project which was once hyped as "Ethereum Killer."
Weiss Ratings in its tweet claimed that although the public sentiment was right behind the project in its hayday and the platform was known for being a fast, efficient and most important a decentralized ecosystem. However, in the past year, there has been a continuous decline in the decentralization aspect where major whales control the majority of the token flow which could be of a deep concern.
Weiss Ratings claimed that the top 100 EOS token holders who represented a meager 0.01% of the total token holders on the platform, has a whopping 68% of the voting power on the network. This means these whales can easily manipulate the network as per their will.
And now EOS block producers and developers devoted to building decentralized apps (dapps) are making little or no money from contributing to the health of the ecosystem. EOS Tribe even wrote about his experience on Steemit. They cited it's longer possible to earn funds for maintaining the blockchain without support from major EOS whales.
Thus the chart suggest EOS is headed to the monthly demand at $1.00.
cryptocurrenciesneoxianpalnetscripsiosteemleo
Warren Buffett's Holy Grail Investment Strategy
It's a Good Day!
Altcoin Buying Masterplan (it's working…)
Bit Brain June 17, 2019 June 17, 2019
Technical Analysis for Bitcoin: Quick Update
Workin January 29, 2019 February 12, 2019
Crypto Contest February 22: Ubiq
Markus Aarnio February 22, 2019 February 22, 2019