text:
\section{Example Appendix}

\end{document}

\section{Conclusion}

In this paper, we present our WMT21 word-level QE shared task submission based on Levenshtein Transformer training and a two-step finetuning process.
We also explore various ways to create synthetic data to build more generalizable systems with limited human annotations.
We show that our system outperforms the OpenKiwi+XLM baseline for all language pairs we experimented with.
Our official results on the blind test set also demonstrate the competitiveness of our system.
We hope that our work can inspire other applications of Levenshtein Transformer beyond the widely studied case of non-autoregressive translation.

\section{Experiment}

\subsection{Setup}
\begin{table*}[]
\centering
\scalebox{0.9}{\begin{tabular}{@{}llll@{}}
\toprule
\textbf{Language Pair} & \textbf{Synthesis Method} & \textbf{Data Source}                         & \textbf{\# Triplets} \\ \midrule
English-German         & src-mt-tgt                & WMT20 en-de parallel data                    & 44.2M                \\
English-German         & src-mt1-mt2               & WMT20 en-de parallel data cleaned            & 36.1M                \\
English-German         & bt-rt-tgt                 & 10M sample from newscrawl2020                & 10.0M                \\
English-German         & bt-noisy-tgt              & 10M sample from newscrawl2020                & 10.0M                \\
English-Chinese        & src-mt-tgt                & shared task en-zh parallel                   & 20.3M                \\
English-Chinese        & MVPPE                     & shared task en-zh parallel                   & 20.3M                \\
Romanian-English       & src-rt-ft                 & shared task ro-en parallel                   & 3.09M                \\
Romanian-English       & MVPPE                     & shared task ro-en parallel                   & 3.09M                \\
Russian-English        & src-rt-ft                 & shared task ru-en parallel                   & 2.32M                \\
Estonian-English       & src-rt-ft                 & shared task et-en parallel                   & 880K                 \\
Nepalese-English       & src-rt-ft                 & shared task ne-en parallel                   & 498K                 \\
Pashto-English         & src-rt-ft                 & WMT20 Parallel Corpus Filtering Task         & 347K                 \\ \bottomrule
\end{tabular}}
\caption{Source and statistics of datasets used for our synthetic finetuning experiments. \label{tab:data}}
\end{table*}

\begin{figure*}
\centering
\includegraphics[scale=0.6]{figures/main.pdf}
\caption{Target MCC results on \texttt{test20} dataset for all language pairs we submitted systems for (except for ps-en which is not included in \texttt{test20}).\label{fig:main}}
\end{figure*}

We follow the data split on human post-edited data as determined by the task organizers and use \texttt{test20} as the devtest for our system development purposes.
Apart from the human post-edited data provided by the task organizers, we also used some extra data for synthetic finetuning.
Table \ref{tab:data} shows the source and statistics of the data that's been used for our synthetic finetuning experiments.
For stage 1 LevT translation training, we always use the same parallel data that is used to train the MT systems provided by the shared task.

For the only language pair where we applied the \textbf{src-mt1-mt2} synthetic finetuning, we used the WMT19 Facebook's winning system \cite{ng-etal-2019-facebook} to generate the higher-quality translation \textbf{mt2}.
For both \textbf{src-mt-tgt} and \textbf{src-mt1-mt2}, we used the system provided by the shared task to generate the MT output in the pseudo translation triplet.
For the Levenshtein Transformer in \textbf{bt-noisy-tgt}, we used a Levenshtein Transformer model trained from scratch (without M2M initialization).
For all the other forward/backward/round-trip translations, we used M2M-mid (1.2B parameters) model to generate the necessary outputs.

All of our experiments uses Adam optimizer \cite{DBLP:journals/corr/KingmaB14} with linear warmup and \texttt{inverse-sqrt} scheduler.
For stage 1, we use the same hyperparameters as \citet{DBLP:conf/nips/GuWZ19} for LevT translation training, but use a smaller learning rate of 2e-5 to avoid overfitting for all to-English language pairs.
For stage 2 and beyond, we stick to the learning rate of 2e-5 and perform early-stopping based on the loss function on the development set. We also experiment with label balancing factor $\sigma=1.0$ and $\sigma=3.0$ for each language pair and pick the one that works the best on devtest data.
We find the QE task performance to be quite sensitive to $\sigma$ but there is also no universally optimal value for all language pairs.

Our LevT-QE model is implemented based on Fairseq \cite{DBLP:conf/naacl/OttEBFGNGA19}.
We also implemented another TER computation tool\footnote{\texttt{https://github.com/marian-nmt/moses-scorers}} to generate the word-level and subword-level tags that we use as the reference for finetuning, but stick to the original reference tags in the test set for evaluation to avoid potential result mismatch.

\subsection{Results}

Our pre-submission results on \texttt{test20} devtest data is shown in Figure \ref{fig:main}.
We can first observe that for all language pairs except English-Chinese, our LevT QE models perform better than the OpenKiwi-XLM \cite{kepler-etal-2019-openkiwi} baselines that we built on our own.\footnote{We followed their \texttt{xlmroberta.yaml} recipe to build our own baseline systems.}
It is not clear as to why English-Chinese is an outlier, but we suspect it might be related to the fact that the M2M model is not a good translation model on this specific language pair, as reported in \cite{DBLP:journals/corr/abs-2010-11125}.
Among the other language pairs, it should be noted that the benefit of LevT is most significant on the language pairs with a large amount of available parallel data.
This is intuitive, because the less parallel data we have, the less knowledge we can draw out of the LevT training process.
In the extreme case where we have no parallel data at all, all of our knowledge comes from the pre-trained model and human annotation data finetuning, which will reduce to the same model as the baseline.
Finally, our Nelder-Meade ensemble further improves the result by a small but steady margin.

We do not have enough time to run a rigorous comparison for all the explored synthetic finetuning methods before the submission deadline (we will include it in our next draft).
Our general development results indicate that \textbf{src-mt1-mt2} works significantly better than all the other synthetic finetuning methods, which does not improve the result and may even hurt sometimes.
This is different from the experiments shown in \cite{lee-2020-two}, which successfully deployed \textbf{src-mt-tgt}-style synthetic finetuning to improve the performance on the word-level QE task.
Note that the limitation of \textbf{src-mt1-mt2} is also pretty clear: it calls for an existing stronger MT system than the one whose translation quality is being estimated, which is why we did not perform similar synthetic finetuning for the language pairs other than en-de, as there are no clearly stronger publicly available MT models for these language pairs.
In theory, we can produce stronger translations for that purpose by resorting to an online translation system such as Google/Bing Translate, which another team used in their 2020 participation \cite{wang-etal-2020-hw}, but those models are not publicly available, hence making the submission results hard to reproduce.
To verify \textbf{src-mt1-mt2} does work beyond en-de language pair, we plan to build contrastive systems that take advantage of the online systems to build synthetic finetuning data.

\section{Experiments}

\subsection{Data Setup}

\paragraph{LevT Training}
We used the same parallel data that was used to train the MT system in the shared task, except for the en-de, et-en, and ps-en language pairs.
For en-de language pair, we use the larger parallel data from the WMT20 news translation shared task.
For et-en language pair, we experiment with augmenting with the News Crawl Estonian monolingual data from 2014 to 2017, which was inspired by \citet{zhou-keung-2020-improving}.
For ps-en language pair, because there is no MT system provided, we take the data from the WMT20 parallel corpus filtering shared task and applied the baseline LASER filtering method.
For the multi-source LevT model, we simply concatenate the data from ro-en, ru-en, es-en (w/o monolingual augmentation) and ne-en.
The resulting data scale is summarized in Table \ref{tab:data}.

Following the setup in \citet{DBLP:conf/nips/GuWZ19}, we conduct sequence-level knowledge distillation during training for all language pairs except for ne-en and ps-en\footnote{The exception was motivated by the poor quality of the translation we obtained from the M2M-100 model.}.
For en-de, the knowledge distillation data is generated by the WMT19 winning submission for that language pair from Facebook \cite{ng-etal-2019-facebook}.
For en-zh, we train our own en-zh autoregressive model on the parallel data from the WMT17 news translation shared task.
For the other language pairs, we use the decoding output from M2M-100-mid (1.2B parameters) model to perform knowledge distillation.

\begin{table}[h]
\centering
\scalebox{0.68}{
\begin{tabular}{@{}lllr@{}}
\toprule
\textbf{Configuration}        & \textbf{Stage 2}      & \textbf{Stage 3} & \multicolumn{1}{l}{\textbf{Target MCC}} \\ \midrule
en-de OpenKiwi                & N                     & default          & 0.337                                   \\
en-de bilingual best          & src-mt1-mt2           & $\mu = 1.0$      & 0.500                                   \\
en-de ensemble                & N/A                   & N/A              & 0.504                                   \\ \midrule
en-zh OpenKiwi                & N                     & default          & 0.421                                   \\
en-zh bilingual best          & mvppe                 & $\mu = 1.0$      & 0.459                                   \\
en-zh ensemble                & N/A                   & N/A              & 0.466                                   \\ \midrule
ro-en OpenKiwi                & N                     & default          & 0.556                                   \\
ro-en bilingual best          & src-rt-ft             & $\mu = 1.0$      & 0.604                                   \\
ro-en multilingual best       & N                     & $\mu = 1.0$      & 0.612                                   \\
ro-en ensemble                & N/A                   & N/A              & 0.633                                   \\ \midrule
ru-en OpenKiwi                & N                     & default          & 0.279                                   \\
ru-en bilingual best          & src-rt-ft             & $\mu = 3.0$      & 0.316                                   \\
ru-en multilingual best       & N                     & $\mu = 3.0$      & 0.339                                   \\
ru-en ensemble                & N/A                   & N/A              & 0.349                                   \\ \midrule
et-en OpenKiwi                & N                     & default          & 0.503                                   \\
et-en bilingual best          & N                     & $\mu = 3.0$      & 0.556                                   \\
et-en bilingual best (w/ aug) & N                     & $\mu = 3.0$      & 0.548                                   \\
et-en multilingual best       & N                     & $\mu = 3.0$      & 0.533                                   \\
et-en ensemble                & N/A                   & N/A              & 0.575                                   \\ \midrule
ne-en OpenKiwi                & N                     & default          & 0.664                                   \\
ne-en bilingual best          & N                     & $\mu = 3.0$      & 0.677                                   \\
ne-en multilingual best       & N                     & $\mu = 3.0$      & 0.681                                   \\
ne-en ensemble                & N/A                   & N/A              & 0.688                                   \\ \bottomrule
\end{tabular}
}
\caption{Target MCC results on \texttt{test20} dataset for all language pairs we submitted systems for (except for ps-en which is not included in \texttt{test20}). Stage 2 stands for synthetic finetuning (where N stands for not performing this stage). Stage 3 stands for human annotation finetuning. $\mu$ stands for the label balancing factor.\label{tab:main}}
\end{table}

\begin{table}[h]
\centering
\scalebox{0.9}{
\begin{tabular}{@{}lrrr@{}}
\toprule
            & \multicolumn{1}{l}{\textbf{Target MCC}} & \multicolumn{1}{l}{\textbf{F1-OK}} & \multicolumn{1}{l}{\textbf{F1-BAD}} \\ \midrule
N           & 0.489                                   & 0.955                              & 0.533                               \\
src-mt-ref  & 0.493                                   & 0.955                              & 0.537                               \\
src-mt1-mt2 & \textbf{0.500}                          & 0.956                              & \textbf{0.544}                      \\
bt-rt-tgt   & 0.490                                   & 0.956                              & 0.534                               \\
src-rt-ft   & 0.494                                   & 0.956                              & 0.538                               \\
mvppe       & \textbf{0.500}                          & \textbf{0.960}                     & 0.540                               \\ \bottomrule
\end{tabular}
}
\caption{Analysis of different data synthesis methods on en-de language pair. All models here are initialized with M2M-100-small. \label{tab:data-synth-de}}
\end{table}

\begin{table*}[h]
\centering
\scalebox{0.9}{
\begin{tabular}{@{}lllrrr@{}}
\toprule
\textbf{Configuration}    & \textbf{Stage 2}       & \textbf{Stage 3} & \multicolumn{1}{l}{\textbf{Target MCC}} & \multicolumn{1}{l}{\textbf{F1-OK}} & \multicolumn{1}{l}{\textbf{F1-BAD}} \\ \midrule
ro-en multilingual        & N                      & $\mu = 1.0$      & \textbf{0.612}                   & 0.949                              & \textbf{0.659}                      \\
ro-en multilingual        & mvppe                  & $\mu = 1.0$      & 0.611                            & \textbf{0.951}                     & \textbf{0.659}                      \\
ro-en multilingual        & src-mt1-mt2 (Bing mt2) & $\mu = 1.0$      & 0.585                            & 0.936                              & 0.630                               \\
ro-en bilingual (Bing KD) & N                      & $\mu = 1.0$      & 0.581                            & 0.949                              & 0.632                               \\
ro-en bilingual (Bing KD) & src-mt1-mt2 (Bing mt2) & $\mu = 1.0$      & 0.568                            & 0.938                              & 0.619                               \\ \midrule
et-en bilingual           & N                      & $\mu = 3.0$      & 0.548                            & 0.914                              & 0.622                               \\
et-en bilingual           & mvppe                  & $\mu = 3.0$      & 0.544                            & \textbf{0.929}                     & 0.615                               \\
et-en bilingual           & src-mt1-mt2 (Bing mt2) & $\mu = 3.0$      & \textbf{0.563}                   & 0.919                              & \textbf{0.634}                      \\
et-en bilingual (Bing KD) & N                      & $\mu = 3.0$      & 0.557                            & 0.918                              & 0.629                               \\
et-en bilingual (Bing KD) & src-mt1-mt2 (Bing mt2) & $\mu = 3.0$      & 0.559                            & 0.916                              & 0.631                               \\ \bottomrule
\end{tabular}
}
\caption{Analysis of \texttt{src-mt1-mt2} and \texttt{mvppe} method on ro-en and et-en language pair. \label{tab:data-synth-ro-et}}
\end{table*}

\paragraph{Synthetic Finetuning}
We always conduct data synthesis based on the same parallel data that was used to train the LevT translation model.
For the only language pair (en-de) where we applied the \texttt{src-mt1-mt2} synthetic finetuning for shared task submission, we again use the WMT19 Facebook's winning system \cite{ng-etal-2019-facebook} to generate the higher-quality translation \texttt{mt2}, and the system provided by the shared task to generate the MT output in the pseudo translation triplet \texttt{mt1}.
For all other combinations of translation directions, language pairs and MVPPE decoding, we use the M2M-100-mid (1.2B parameters) model.

\paragraph{Human Annotation Finetuning}
We follow the data split for human post-edited data as determined by the task organizers and use \texttt{test20} as the devtest for our system development purposes.

\paragraph{Reference Tag Generation}
We implemented another TER computation tool\footnote{\texttt{https://github.com/marian-nmt/moses-scorers}} to generate the word-level and subword-level tags that we use as the reference for finetuning, but stick to the original reference tags in the test set for evaluation to avoid potential result mismatch.

\subsection{Model Setup}
Our LevT-QE model is implemented based on Fairseq \cite{DBLP:conf/naacl/OttEBFGNGA19}.
All of our experiments uses Adam optimizer \cite{DBLP:journals/corr/KingmaB14} with linear warmup and \texttt{inverse-sqrt} scheduler.
For stage 1, we use the same hyperparameters as \citet{DBLP:conf/nips/GuWZ19} for LevT translation training, but use a smaller learning rate of 2e-5 to avoid overfitting for all to-English language pairs.
For stage 2 and beyond, we stick to the learning rate of 2e-5 and perform early-stopping based on the loss function on the development set. For stage 3, e also experiment with label balancing factor $\mu=1.0$ and $\mu=3.0$ for each language pair and pick the one that works the best on devtest data, while for stage 2 we keep $\mu = 1.0$ because early experiments indicate that using $\mu = 3.0$ at this stage is not helpful.

For pre-submission developments, we built OpenKiwi-XLM baselines \cite{kepler-etal-2019-openkiwi} following their \texttt{xlmroberta.yaml} recipe.
Keep in mind due to the fact that this baseline model is initialized with a much smaller XLM-Roberta-base model (281M parameters) compared to our M2M-100-small initialization (484M parameters), the performance comparison is not a strict one.

\subsection{Devtest Results}

Our system development results on \texttt{test20} devtest data are shown in Table \ref{tab:main}\footnote{Note that the results on en-zh also reflect a crucial bug fix on our TER computation tool that we added after the system submission deadline. Hence the results shown here are from a different system as in the official shared task results. The bug fix should not affect the results of the other language pairs.}.
In all language pairs, our systems can outperform the OpenKiwi baseline based upon the pre-trained XLM-RoBERTa-base encoder.
Among these language pairs, the benefit of LevT is most significant on the language pairs with a large amount of available parallel data.
Such behavior is expected, because the less parallel data we have, the less knowledge we can extract from the LevT training process.
Furthermore, the lack of good quality knowledge distillation data in the low-resource language pairs also expands this performance gap.
To our best knowledge, this is also the first attempt to train non-autoregressive translation systems under low-resource settings, and we hope future explorations in this area can enable us to build a better QE system from LevT.

In terms of comparison between multilingual and bilingual models for to-English language pairs, the results are mixed, with the multilingual model performing significantly better for ru-en language pair, but significantly worse for et-en language pair.
Finally, our Nelder-Mead ensemble further improves the result by a small but steady margin.

\subsection{Analysis}

\citet{ding2021levenshtein} already conducted comprehensive ablation studies for techniques such as the effect of LevT training step, heuristic subword-level reference tag, as well as the effect of various data synthesis methods.
In this section, we extend the existing analyses by studying if the synthetic finetuning is still useful with M2M initialization, and if it is universally helpful across different languages.
We also examine the effect of label balancing factor $\mu$ and take a detailed look at the prediction errors.

\paragraph{Synthetic Finetuning}
We redo the analysis on en-de synthetic finetuning with the smaller 2M parallel sentence samples from Europarl, as in \citet{ding2021levenshtein}, but with the updated \texttt{test20} test set and models with M2M-100-small initialization.
The results largely corroborate the trend in the other paper, showing that \texttt{src-mt1-mt2} and \texttt{mvppe} being the most helpful two data synthesis methods.
We then extend those two most helpful methods to ro-en and et-en, using the up-to-date Bing Translator production model as the stronger MT system (a.k.a. \texttt{mt2}) in the \texttt{src-mt1-mt2} synthetic data.
The result is mixed, with \texttt{mvppe} failing to improve performance for both language pairs, and \texttt{src-mt1-mt2} only being helpful for et-en language pair.
We also trained two extra ro-en and et-en LevT models using the respective Bing Translator models to generate the KD data, which are neither helpful for improving performance on their own nor working better with \texttt{src-mt1-mt2} synthetic data.

We notice that the \texttt{mvppe} synthetic data seems to significantly improve the F1 score of the \texttt{OK} label in general, for which we don't have a good explanation yet.

\paragraph{Label Balancing Factor}

\begin{table}[]
\scalebox{0.9}{
\begin{tabular}{@{}lrrr@{}}
\toprule
\textbf{Configuration} & \multicolumn{1}{l}{\textbf{Target MCC}} & \multicolumn{1}{l}{\textbf{F1-OK}} & \multicolumn{1}{l}{\textbf{F1-BAD}} \\ \midrule
ro-en $\mu = 1.0$      & \textbf{0.612}                          & \textbf{0.949}                     & \textbf{0.659}                      \\
ro-en $\mu = 3.0$      & 0.577                                   & 0.930                              & 0.619                               \\ \midrule
ru-en $\mu = 1.0$      & 0.267                                   & \textbf{0.960}                     & 0.284                               \\
ru-en $\mu = 3.0$      & \textbf{0.339}                          & 0.943                              & \textbf{0.390}                      \\ \midrule
et-en $\mu = 1.0$      & 0.478                                   & \textbf{0.933}                     & 0.511                               \\
et-en $\mu = 3.0$      & \textbf{0.512}                          & 0.925                              & \textbf{0.587}                      \\ \midrule
ne-en $\mu = 1.0$      & 0.660                                   & \textbf{0.885}                     & 0.774                               \\
ne-en $\mu = 3.0$      & \textbf{0.681}                          & 0.855                              & \textbf{0.788}                      \\ \bottomrule
\end{tabular}
}
\caption{Analysis of different label balancing factors initialized on to-English language pairs. All results are based on the multilingual model and not performing synthetic finetuning step. \label{tab:lb-factor}}
\end{table}

\definecolor{applegreen}{rgb}{0.553, 0.714, 0.0}
\definecolor{msgreen}{rgb}{0.490, 0.718, 0.0}
\definecolor{bananamania}{rgb}{0.98, 0.91, 0.71}
\definecolor{pastelgreen}{rgb}{0.467, 0.867, 0.467}
\def\cca#1{
    \pgfmathsetmacro\calc{(#1*100}
    \edef\clrmacro{\noexpand\cellcolor{msgreen!\calc}}
    \clrmacro{#1}

}

\begin{table*}[]
\centering
\scalebox{0.6}{
\begin{tabular}{@{}lrrrrrrrrrrrrrrr@{}}
\toprule
\textbf{Lang.} & \multicolumn{1}{l}{\textbf{Tgt. MCC}} & \multicolumn{1}{l}{\textbf{MT MCC}} & \multicolumn{3}{l}{\textbf{MT BAD (P/R/F1)}} & \multicolumn{3}{l}{\textbf{MT OK (P/R/F1)}} & \multicolumn{1}{l}{\textbf{GAP MCC}} & \multicolumn{3}{l}{\textbf{GAP BAD (P/R/F1)}} & \multicolumn{3}{l}{\textbf{GAP OK (P/R/F1)}} \\ \midrule
en-de         &  \cca{0.504} &  \cca{0.503} &  \cca{0.476} &  \cca{0.731} &  \cca{0.576} &  \cca{0.950} &  \cca{0.863} &  \cca{0.904} &  \cca{0.280} &  \cca{0.366} &  \cca{0.238} &  \cca{0.288} &  \cca{0.980} &  \cca{0.989} &  \cca{0.984}        \\
en-zh         &  \cca{0.466} &  \cca{0.381} &  \cca{0.467} &  \cca{0.787} &  \cca{0.586} &  \cca{0.879} &  \cca{0.633} &  \cca{0.736} &  \cca{0.146} &  \cca{0.276} &  \cca{0.099} &  \cca{0.145} &  \cca{0.965} &  \cca{0.990} &  \cca{0.977}        \\
ro-en         &  \cca{0.612} &  \cca{0.645} &  \cca{0.729} &  \cca{0.709} &  \cca{0.719} &  \cca{0.922} &  \cca{0.929} &  \cca{0.926} &  \cca{0.164} &  \cca{0.411} &  \cca{0.073} &  \cca{0.125} &  \cca{0.973} &  \cca{0.997} &  \cca{0.985}        \\
ru-en         &  \cca{0.349} &  \cca{0.329} &  \cca{0.296} &  \cca{0.675} &  \cca{0.411} &  \cca{0.945} &  \cca{0.775} &  \cca{0.852} &  \cca{0.167} &  \cca{0.265} &  \cca{0.123} &  \cca{0.168} &  \cca{0.978} &  \cca{0.991} &  \cca{0.985}        \\
et-en         &  \cca{0.575} &  \cca{0.553} &  \cca{0.676} &  \cca{0.681} &  \cca{0.679} &  \cca{0.875} &  \cca{0.873} &  \cca{0.874} &  \cca{0.251} &  \cca{0.426} &  \cca{0.169} &  \cca{0.242} &  \cca{0.967} &  \cca{0.991} &  \cca{0.979}        \\
ne-en         &  \cca{0.694} &  \cca{0.434} &  \cca{0.760} &  \cca{0.918} &  \cca{0.832} &  \cca{0.746} &  \cca{0.454} &  \cca{0.564} &  \cca{0.192} &  \cca{0.444} &  \cca{0.098} &  \cca{0.161} &  \cca{0.955} &  \cca{0.994} &  \cca{0.974}        \\ \bottomrule
\end{tabular}
}
\caption{Detailed evaluation metric breakdown of all submitted ensemble system on \texttt{test20} test set.\label{tab:breakdown}}
\end{table*}

We find the QE task performance to be quite sensitive to the label balancing factor $\mu$, but there is also no universally optimal value for all language pairs.
Table \ref{tab:lb-factor} shows this behavior for all to-English language pairs.
Notice that while for most of the cases $\mu$ simply controls a trade-off between the performance of \texttt{OK} and \texttt{BAD} outputs, there are also cases such as ro-en where a certain choice of $\mu$ hurts the performance of both classes.
This might be due to a certain label class being particularly hard to fit, thus creating more difficulties with learning when the loss function is designed to skew to this label class.

It should be noted that this label balancing factor does not correlate directly with the ratio of the \texttt{OK} vs. \texttt{BAD} labels in the training set.
For example, to obtain the best performance, ne-en requires $\mu = 3.0$ while en-de requires $\mu = 1.0$, while the \texttt{OK} to \texttt{BAD} ratio for ne-en (2.14:1) is much less skewed compare to en-de (10.2:1).

\paragraph{Detailed Error Breakdown}
We found it hard to develop an intuition for the model performance from the MCC metric.
To further understand which label categories our models struggle with the most, we breakdown the target-side metric into a cross product of \{\texttt{MT}, \texttt{GAP}\} tags and \{\texttt{OK}, \texttt{BAD}\} classes and compute precision, recall and F1-score for each category.
The breakdown is shown in Table \ref{tab:breakdown}.
It can be seen that our model is making the most mistakes with the \texttt{GAP BAD} category, while the category with the least mistakes is the \texttt{GAP OK} category.
Also, note that for MT word tags, the models often seem to suffer more from low precision rather than low recall, while for gaps it is the opposite.

Overall, we see that the highest F1 scores we can achieve for detecting bad MT words or gaps are rarely higher than 0.8, which indicates that there should be ample room for improvement.
It would also be interesting to measure the inter-annotator agreement of these word-level quality labels, in order to get a sense of the human performance we should be aiming for.

\section{Introduction\blfootnote{$^*$ Shuoyang Ding had a part-time affiliation with Microsoft at the time of this work.}}

In the machine translation (MT) literature, quality estimation (QE) refers to the task of evaluating the translation quality of a system without using a human-generated reference.
There are several different granularities as to the way those quality labels or scores are generated.
Our participation in the WMT21 quality estimation shared task focuses specifically on the \emph{word-level} quality labels (word-level subtask of Task 2), which are helpful for both human \cite{lee-etal-2021-intellicat} and automatic \cite{lee-2020-cross} post-editing of translation outputs.
The task asks the participant to predict one binary quality label (\texttt{OK}/\texttt{BAD}) for each target word and each gap between target words, respectively.\footnote{While there is another sub-task for predicting source-side quality labels, we do not participate in that task.}

Our approach closely follows our contemporary work \cite{ding2021levenshtein}, which focuses on en-de and en-zh language pairs tested in the 2020 version of the shared task.
The intuition behind our idea is that translation knowledge is very useful for predicting word-level quality labels of translations.
However, usage of machine translation models is limited in the previous work mainly due to (1) the difficulties in using both the left and right context of an MT word to be evaluated;
(2) the difficulties in making the word-level reference labels compatible with subword-level models;
and (3) the difficulties in enabling translation models to predict gap labels.
To resolve these difficulties, we resort to Levenshtein Transformer \cite[LevT,][]{DBLP:conf/nips/GuWZ19}, a model architecture designed for non-autoregressive neural machine translation (NA-NMT).
Because of its iterative inference procedure, LevT is capable of performing post-editing on existing translation output even just trained for translation.
To further improve the model performance, we also propose to initialize the encoder and decoder of the LevT model with those from a massively pre-trained multilingual NMT model \cite[M2M-100,][]{DBLP:journals/corr/abs-2010-11125}.

Starting from a LevT translation model, we then perform a two-stage finetuning process to adapt the model from translation prediction to quality label prediction, using automatically-generated pseudo-post-editing triplets and human post-editing triplets respectively.
All of our final system submissions are also linear ensembles from several individual models with weights optimized on the development set using the Nelder-Mead method \cite{DBLP:journals/cj/NelderM65}.
\section{Method}

Our system building pipeline is consisted of three different stages:
\begin{itemize} \itemsep -2pt
	\item \textbf{Stage 1}: Training LevT for translation
	\item \textbf{Stage 2 (Optional)}: Finetuning LevT on synthetic post-editing triplets
	\item \textbf{Stage 3}: Finetuning LevT on human post-editing triplets
\end{itemize}
We start by introducing the architecture we use and then describe each stage of our training process.

\subsection{Levenshtein Transformer}

The Levenshtein Transformer \cite[LevT,][]{DBLP:conf/nips/GuWZ19} is a non-autoregressive NMT architecture that generates translations by iteratively applying Longest Common Subsequence (LCS) edits \cite{DBLP:journals/csur/Navarro01} on the translation output.
Compared to the standard transformer model, the main difference in terms of the model architecture is that, on top of the transformer decoder blocks, it has two extra prediction heads\footnote{A prediction head is generally implemented as one or more linear layers that take the decoder state as the input.} that predict editing actions for this iteration, namely the \emph{deletion head} $\vec{A}_{del}$ and \emph{mask insertion head} $\vec{A}_{ins}$.
A standard transformer model, on the other hand, only has one prediction head that predicts the upcoming word, which we call \emph{word prediction head} $\vec{A}_{w}$.
At inference time, LevT takes the source sentence and the previous iteration target sequence (or empty sequence for the first iteration) as the input, and applies edits in the order of deletion $\rightarrow$ mask insertion $\rightarrow$ word insertion to generate the sequence iteratively, as shown in Figure~\ref{fig:ter-exp}b.
At training time, the three prediction heads $\vec{A}_w$, $\vec{A}_{del}$ and $\vec{A}_{ins}$ are trained with imitation learning, with the randomly noised target sentence as target-side input and LCS edits as the expert policy.
Like many other non-autoregressive translation models \cite[][\emph{inter alia}]{DBLP:conf/iclr/Gu0XLS18,ghazvininejad-etal-2019-mask,lee-etal-2020-iterative}, LevT also requires training on knowledge distillation data to achieve the optimal performance.
We refer the readers to Appendix \ref{sec:levT-details} and \citet{DBLP:conf/nips/GuWZ19} for details about the model, training scheme, and translation results.

\subsection{System Building Process}

\paragraph{Stage 1: Training LevT for Translation}
We largely follow the same procedure as \citet[LevT,][]{DBLP:conf/nips/GuWZ19} to train the LevT translation model, except that we initialize the embedding, the encoder, and decoder of LevT with those from the small M2M-100 model \cite[418M parameters,][]{DBLP:journals/corr/abs-2010-11125} to take advantage of large-scale pretraining.
Because of that, we also use the same sentencepiece model and vocabulary as the M2M-100 model.

For to-English language pairs, we explored training multi-source Levenshtein Transformer model. According to the results on devtest data, this is shown to be beneficial for the QE task for ro-en, ru-en and ne-en, but not for the other language pairs.

\paragraph{Stage 2: Synthetic Finetuning}
During both finetuning stages, we update the model parameters to minimize the NLL loss of word quality labels and gap quality labels, for $\vec{A}_{del}$ and $\vec{A}_{ins}$ respectively.
For language pairs with high translation quality, the translation errors are often quite scarce, thus creating a skewed label distribution over the \texttt{OK} and \texttt{BAD} labels.
To ensure that the model captures both label categories in a balanced manner, we introduce a multiplicative label balance factor $\sigma$ to up-weight the NLL loss of the \texttt{BAD} labels.

To obtain training targets for finetuning, we need \emph{translation triplet data}, i.e. aligned triplet of source, target, and post-edited segments.
Human post-edited data naturally provides all three fields of the triplet, but only comes in a limited quantity.
To further help the model to generalize, we conduct an extra step of finetuning on synthetic translation triplets, similar to some previous work \cite[][\textit{inter alia}]{lee-2020-two}.
We explored six different methods for data synthesis, namely:
\begin{enumerate}
	\item \textbf{src-mt-tgt}: Take the source side of a parallel corpus (\textbf{src}), translate it with a MT model to obtain the MT output (\textbf{mt}), and use the target side of the parallel corpus as the pseudo post-edited output (\textbf{tgt}).
	\item \textbf{src-mt1-mt2}: Take a corpus in the source language (\textbf{src}) and translate it with two different MT systems that have a clear system-level translation quality ordering. Then, take the inferior MT output as the MT output in the triplet (\textbf{mt1}) and the better as the pseudo post-edited output in the triplet (\textbf{mt2}).
	\item \textbf{bt-rt-tgt}: Take a corpus in the target language (\textbf{tgt}) and back-translate it into the source langauge (\textbf{bt}), and then translate again to the target language (\textbf{rt}). We then use \textbf{rt} as the MT output in the triplet and \textbf{tgt} as the pseudo post-edited output in the triplet.
	\item \textbf{bt-noisy-tgt}: Take a corpus in the target language (\textbf{tgt}) and back-translate it into the source language (\textbf{bt}). We then randomly mask some words in the target and let a LevT translation model complete the translation by referring to the \textbf{bt} and masked \textbf{tgt}, which will generate a noisy target sentence (\textbf{noisy}). We then use \textbf{noisy} as the MT output in the triplet and the original target sentence in the corpus (\textbf{tgt}) as the pseudo post-edited output in the triplet.
	\item \textbf{src-rt-ft}: Take a parallel corpus and translate its source side and use it as the pseudo post-edited output (\textbf{ft}), and round-trip translate its target side (\textbf{rt}) as the MT output in the translation triplet.
	\item \textbf{multiview pseudo post-editing (MVPPE)}: This method is inspired by \citet{thompson-post-2020-paraphrase}, which used a multilingual translation system as a zero-shot paraphraser. We take a parallel corpus and translate the source side (\textbf{src}) with a multilingual translation system (\textbf{mt}) as the MT output in the triplet. We then generate the pseudo-post-edited output by ensembling two different \emph{views} of the same model: (1) using the multilingual translation model as a translation model, with \textbf{src} as the input; (2) using the multilingual translation model as a paraphrasing model, with \textbf{tgt} as the input. The ensemble process is the same as standard MT model ensembles and beam search is performed on top of the ensemble.
\end{enumerate}

\RestyleAlgo{ruled}
\begin{algorithm}[t]
 \KwIn{subword-level token sequence $\vec{y}^{sw}$, word-level token sequence $\vec{y}^w$, subword-level tag sequence $\vec{q}^{sw}$}
 \KwOut{word-level tag sequence $\vec{q}^w$}
 $\vec{q}^w \leftarrow$ []\;
 \For{each word $w_k$ in $\vec{y}^w$}{
    find subword index span $(s_k, e_k)$ in $\vec{y}^{sw}$ that corresponds to $w_k$\;
    $\vec{q}^{sw}_{k} \leftarrow$ subword-level translation and gap tags within span $(s_k, e_k)$\;
    $g^{sw}_{s_k} \leftarrow$ subword-level gap tag before span $(s_k, e_k)$\tcp*{$g^{sw}_{s_k-1}\in\vec{q}^{sw}$}
    \eIf{$\forall\,\vec{q}^{sw}_k$ are \texttt{OK}}{
        $\vec{q}^w$ += [$g^{sw}_{s_k}$, \texttt{OK}]\;
    }{
        $\vec{q}^w$ += [$g^{sw}_{s_k}$, \texttt{BAD}]\;
    }
 }
 $\vec{q}^w$ += [$\vec{q}^{sw}$[-1]]\tcp*{add ending gap tag}
 \Return{$\vec{q}^w$}\;
 \caption{Conversion of subword-level tags to word-level tags\label{algo1}}
\end{algorithm}

\paragraph{Stage 3: Human Post-editing Finetuning}
We follow the same procedure as the synthetic finetuning stage except that we finetune on the human post-edited triplet dataset provided by the shared task organizers for this stage.

\subsection{Compatibility With Subwords}

\RestyleAlgo{ruled}
\begin{algorithm}[htbp]
 \KwIn{subword-level token sequence $\vec{y}^{sw}$, word-level token sequence $\vec{y}^w$, naive subword-level tag sequence $\vec{q}^{sw}$, word-level tag sequence $\vec{q}^{w}$}
 \KwOut{heuristic subword-level tag sequence $\widetilde{\vec{q}^{sw}}$}
 $\widetilde{\vec{q}^{sw}} \leftarrow$ []\;
 \For{each word $w_k$ in $\vec{y}^w$}{
   find subword index span $(s_k, e_k)$ in $\vec{y}^{sw}$ that corresponds to $w_k$\;
   $\vec{q}^{sw}_k \leftarrow$ subword-level translation and gap tags within span $(s_k, e_k)$\;
   $t^w_k \leftarrow$ word-level translation tag for $w_k$\tcp*{$t^w_k\in\vec{q}^w$}
   $g^w_k \leftarrow$ word-level gap tag before $w_k$\tcp*{$g^w_k\in\vec{q}^w$}
   $\widetilde{\vec{q}^{sw}}$ += [$g^w_k$]\tcp*{copy left gap tag}
   $n$ = $\left|\vec{q}^{sw}_k\right|$\tcp*{\# subwords for $w_k$}
   \uIf{$t^t_k$ is \texttt{OK}}{
     \tcc{word is \texttt{OK}, so all subwords and gaps between them should be \texttt{OK}}
     $\widetilde{\vec{q}^{sw}}$ += [\texttt{OK}] * $(2n-1)$
   }
   \uElseIf{$\exists\,$\texttt{BAD} in $\vec{q}^{sw}_k$}{
     \tcc{no conflict between subword-level tag and word-level tag -- copy $\vec{q}^{sw}_k$ as-is}
     $\widetilde{\vec{q}^{sw}}$ += $\vec{q}^{sw}_k$\;
   }
   \uElse{
     \tcc{subword-level tag disagrees with word-level tag -- force it as all-\texttt{BAD} to guarantee perfect conversion}
     $\widetilde{\vec{q}^{sw}}$ += [\texttt{BAD}] * $(2n-1)$
   }
 }
 $\widetilde{\vec{q}^{sw}}$ += [$\vec{q}^{w}$[-1]]\tcp*{add ending gap tag}
 \Return{$\widetilde{\vec{q}^{sw}}$}\;
 \caption{Construction of heuristic subword-level tags\label{algo2}}
\end{algorithm}

To the best of our knowledge, previous work on word-level quality estimation either builds models that directly output word-level tags~\cite{lee-2020-two,hu-etal-2020-niutrans-system,moura-etal-2020-ist} or uses simple heuristics to re-assign word-level tags to the first subword token~\cite{wang-etal-2020-hw}.
Since LevT predicts edits on a subword-level starting from translation training, we need to: (1) for inference, convert subword-level tags predicted by the model to word-level tags for evaluation, and (2) for both finetuning stages, build subword-level reference tags.

For inference, the conversion can be easily done by heuristics, as shown in Algorithm \ref{algo1}.
For finetuning, a \emph{naive subword-level tag reference} can be built by running TER alignments on MT and post-edited text after subword tokenization.
However, a preliminary analysis shows that such reference introduces a 10\% error after being converted back to word-level.
Hence, we introduce another heuristic to create \emph{heuristic subword-level tag references}, shown in Algorithm \ref{algo2}.
The high-level idea is to interpolate the word-level and the naive subword-level references to ensure that the interpolated subword-level tag reference can be perfectly converted back to the word-level references.

\subsection{Ensemble}
For each binary label prediction made by the model, the model will give a score $p(\texttt{OK})$, which will be translated into binary labels in post-processing.
To ensemble predictions from $k$ models $p_1(\texttt{OK}), p_2(\texttt{OK}), \dots, p_k(\texttt{OK})$, we perform a linear combination of the scores for each label:
$$p_E(\texttt{OK}) = \lambda_1 p_1(\texttt{OK}) + \lambda_2 p_2(\texttt{OK}) + \dots + \lambda_k p_k(\texttt{OK})$$
To determine the optimal interpolation weights, we optimize towards target-side MCC on the development set.
Because the target-side MCC computation is not implemented in a way such that gradient information can be easily obtained, we experimented with two gradient-free optimization method: Powell method \cite{DBLP:journals/cj/Powell64} and Nelder-Mead method \cite{DBLP:journals/cj/NelderM65}, both as implemented in \texttt{SciPy} \cite{2020SciPy-NMeth}.
We found that Nelder-Mead method tends to find better optimum on the development set while also leading to better performance on the devtest dataset (not involved in optimization).
Hence, we use the Nelder-Mead optimizer for all of our final submissions with ensembles.

\section{Method}

Our system building pipeline is consisted of three different stages:
\begin{itemize} \itemsep -2pt
	\item \textbf{Stage 1}: Training LevT for translation
	\item \textbf{Stage 2 (Optional)}: Finetuning LevT on synthetic post-editing triplets
	\item \textbf{Stage 3}: Finetuning LevT on human post-editing triplets
\end{itemize}

\paragraph{Stage 1: Training LevT for Translation}
We largely follow the same procedure as \citet[LevT,][]{DBLP:conf/nips/GuWZ19} to train the LevT translation model, except that we initialize the embedding, the encoder, and decoder of LevT with those from the small M2M-100-small model \cite[418M parameters,][]{DBLP:journals/corr/abs-2010-11125} to take advantage of large-scale pretraining.
Because of that, we also use the same sentencepiece model and vocabulary as the M2M-100 model.

For to-English language pairs, we explored training multi-source LevT model. According to the results on devtest data, this is shown to be beneficial for the QE task for ro-en, ru-en and ne-en, but not for other language pairs.

\paragraph{Stage 2: Synthetic Finetuning}
During both finetuning stages, we update the model parameters to minimize the NLL loss of word quality labels and gap quality labels, for the deletion and insertion head, respectively.
To obtain training targets for finetuning, we need \emph{translation triplet data}, i.e., the aligned triplet of source, target, and post-edited segments.
Human post-edited data naturally provides all three fields of the triplet, but only comes in a limited quantity.
To further help the model to generalize, we conduct an extra step of finetuning on synthetic translation triplets, similar to some previous work \cite[][\textit{inter alia}]{lee-2020-two}.
We explored five different methods for data synthesis, namely:
\begin{enumerate}[leftmargin=0.5cm]
	\item \texttt{src-mt-tgt}: Take the source side of a parallel corpus (\texttt{src}), translate it with a MT model to obtain the MT output (\texttt{mt}), and use the target side of the parallel corpus as the pseudo post-edited output (\texttt{tgt}).
	\item \texttt{src-mt1-mt2}: Take a corpus in the source language (\texttt{src}) and translate it with two different MT systems that have clear system-level translation quality orderings. Then, take the worse MT output as the MT output in the triplet (\texttt{mt1}) and the better as the pseudo post-edited output in the triplet (\texttt{mt2}).
	\item \texttt{bt-rt-tgt}: Take a corpus in the target language (\texttt{tgt}) and back-translate it into the source langauge (\texttt{bt}), and then translate again to the target language (\texttt{rt}). We then use \texttt{rt} as the MT output in the triplet and \texttt{tgt} as the pseudo post-edited output in the triplet.

	\item \texttt{src-rt-ft}: Take a parallel corpus and translate its source side and use it as the pseudo post-edited output (\texttt{ft}), and round-trip translate its target side (\texttt{rt}) as the MT output in the translation triplet.
	\item \texttt{Multi-view Pseudo Post-Editing (MVPPE)}: Same as \citet{ding2021levenshtein}, we take a parallel corpus and translate the source side (\texttt{src}) with a multilingual translation system (\texttt{mt}) as the MT output in the triplet. We then generate the pseudo-post-edited output by ensembling two different \emph{views} of the same model: (1) using the multilingual translation model as a translation model, with \texttt{src} as the input; (2) using the multilingual translation model as a paraphrasing model, with \texttt{tgt} as the input. The ensemble process is the same as ensembling standard MT models, and we perform beam search on top of the ensemble. Unless otherwise specified, we use the same ensembling weights of $\lambda_t = 2.0$ and $\lambda_p = 1.0$ as \citet{ding2021levenshtein}.
\end{enumerate}

\paragraph{Stage 3: Human Post-editing Finetuning}
We follow the same procedure as stage 2, except that we finetune on the human post-edited dataset provided by the shared task organizers for this stage.

\begin{table*}[]
\centering
\scalebox{0.9}{
\begin{tabular}{@{}lll@{}}
\toprule
Language Pair    & Data Source                                  & Sentence Pairs \\ \midrule
English-German   & WMT20 en-de parallel data                    & 44.2M          \\
English-Chinese  & shared task en-zh parallel                   & 20.3M          \\
Romanian-English & shared task ro-en parallel                   & 3.09M          \\
Russian-English  & shared task ru-en parallel                   & 2.32M          \\
Estonian-English & shared task et-en parallel                   & 880K           \\
Estonian-English & shared task et-en parallel + NewsCrawl 14-17 & 3.42M          \\
Nepalese-English & shared task ne-en parallel                   & 498K           \\
Pashto-English   & WMT20 Parallel Corpus Filtering Task         & 347K           \\ \bottomrule
\end{tabular}
}
\caption{Source and statistics of parallel datasets used in our experiments. \label{tab:data}}
\end{table*}

\paragraph{Compatibility With Subwords}

As pointed out before, since LevT predicts edits on a subword-level starting from translation training, we must construct reference tags that are compatible with the subword segmentation done for both the MT and the post-edited output.
Specifically, we need to: (1) for inference, convert subword-level tags predicted by the model to word-level tags for evaluation, and (2) for both finetuning stages, build subword-level reference tags.
We follow the same heuristic subword-level tag reference construction procedure as \citet{ding2021levenshtein}, which was shown to be helpful for task performance.

\paragraph{Label Imbalance}
Like several previous work \cite{lee-2020-cross,wang-etal-2020-hw-tscs,moura-etal-2020-ist}, we also observed that the translation errors are often quite scarce, thus creating a skewed label distribution over the \texttt{OK} and \texttt{BAD} labels.
Since it is critical for the model to reliably predict both classes of labels, we introduce an extra hyperparameter $\mu$ in the loss function that allows us to upweight the words that are classified with \texttt{BAD} tags in the reference.
$$\mathcal{L} = \mathcal{L}_{OK} + \mu \mathcal{L}_{BAD}$$

\paragraph{Ensemble}
For each binary label prediction made by the model, the model will give a score $p(\texttt{OK})$, which are translated into binary labels in post-processing.
To ensemble predictions from $k$ models $p_1(\texttt{OK}), p_2(\texttt{OK}), \dots, p_k(\texttt{OK})$, we perform a linear combination of the scores for each label:
$$p_E(\texttt{OK}) = \lambda_1 p_1(\texttt{OK}) + \lambda_2 p_2(\texttt{OK}) + \dots + \lambda_k p_k(\texttt{OK})$$
To determine the optimal interpolation weights, we optimize towards target-side MCC on the development set.
Because the target-side MCC computation is not implemented in a way such that gradient information can be easily obtained, we experimented with two gradient-free optimization method: Powell method \cite{DBLP:journals/cj/Powell64} and Nelder-Mead method \cite{DBLP:journals/cj/NelderM65}, both as implemented in \texttt{SciPy} \cite{2020SciPy-NMeth}.
We found that the Nelder-Mead method finds better optimum on the development set while also leading to better performance on the devtest dataset (not involved in optimization).
Hence, we use the Nelder-Mead optimizer for all of our final submissions with ensembles.
We set the initial points of Nelder-Mead optimization to be the vertices of the standard simplex in the $k$-dimensional space, with $k$ being the number of the models.

We find that it is critical to build ensembles from models that yield diverse yet high-quality outputs.
Specifically, we notice that ensembles from multiple checkpoints of a single experimental run are not helpful.
Hence, for each language pair, we select 2-8 models with different training configurations that also have the highest performance to build our final ensemble model for submission.

redpajama_set_name: RedPajamaArXiv


text:
Uneven, but perfect
~ @ Eagle Grove ~ The Bulldogs pulled off another sweep last Thursday, when they travelled to Eagle Grove, running their season record to 10-0 overall, 4-0 in the TIC West. While winning in straight sets is not new for the Bulldogs, the night did provide the opportunity for some new and younger...
Harriers see improvement in second outing
Running at Belmond in hot and humid conditions, the Bulldog harriers saw two runners improve their times from the previous outing, as runners scrambled to beat potential weather issues. "The weather was a bit warm and muggy for this one. and the order was a bit rushed with incoming rain," noted...
Bulldogs moving up the charts
~ @ B-K~ After opening the season ranked No. 6, several strong early outings have moved the Bulldogs into the No. 5 position to end the week. The Bulldogs played their first road contest of the year on Tuesday, and downed Belmond-Klemme in straight sets 25-9, 25-9 and 25-78 to move to 2-0 on the...
Big plays boost Newman
When you have a game with two evenly matched teams, it is often momentum at the right time, and big plays that make the difference between winning and losing. Unfortunately, the team that had the big plays and the opportune momentum Friday night was Newman, as they took home a 38-23 decision. The...
Harriers have good opening night showing
Led by Carson Rygh, who placed eighth overall on the boys side and Morgan Fritz, who finished 17th in the girls race, the Bulldog cross country team got the 2016 season off to a fast start. As a team, the boys would finish 10th, while the girls finished No. 12, out of 14 teams competing last...
No. 6 netters keep rolling
While some of the names have changed, and the schedule says 2017, The opening night for Lake Mills volleyball could have been pretty much any of the past dozen or so years, as the No. 6 rated Bulldogs breezed past West Hancock in straight sets, 25-6, 25-3 and 25-8. "As far as opening nights go, I...
Bulldogs start season strong
A new year, got off to the best start possible, for the Lake Mills Bulldogs, as they downed West Fork 41-0 last Friday. "It was a great first win of the season," exclaimed Coach Bill Byrnes following the game. "We have worked hard and the kids were deserving of the victory." In the decisive win,...
Five Bulldogs named to All-Conference team
Statistics Represent Top of Iowa Conference West Division Games Only Charlie Dugger Slade Sifuentes Cael Boehmer Zach Throne Luke Groe 1st Team Outfield 1st Team Utility 2nd Team Infield 2nd Team Utility Hon. Mention Batting Average .370 .327 .386...
Byrnes and Borgmeyer earn All-State nods
Graduated senior, Cede Byrnes, was a third team infield selection. One of the top offensive players on the team, Byrnes was rewarded for compiling a .512 batting average, .655 slugging percentage, and a .568 on-base percentage. To go along with these numbers, she also scored 24 runs, while knocking...
Rice Lake Golf Club Championship results
Rice Lake Golf and Country Club held their Ladies and Men's Club Championship tournament, Saturday, Aug. 6. Results follow. Senior Ladies Championship Flight Barb Miller, 79; Sue Miller, 94; Kathleen Grunhovd, 94; Kathy Gilbertson, 96; Kathy Bernhard, 100; Pat O'Toole, 101; Garnice Rasmusson, 107...

redpajama_set_name: RedPajamaCommonCrawl


text:
Mineral Resources >
Activities Sub Menu
Numerous occurrences of precious metal minerals have been documented in Alberta. Gold- and platinum-group elements occur in Cretaceous to modern placer deposits throughout the province. Recent mineral production estimates by Natural Resources Canada indicates a gold production for Alberta of 30 kilograms, with a value of $1.4 million dollars for 2014. This production is mostly derived as a by-product of sand and gravel mining from placer and paleoplacer deposits. The exposed Precambrian crystalline basement in northeastern Alberta hosts many sulphide showings containing precious metals as secondary commodities, occurring in stratiform orebodies, iron-oxide breccias and veins, and shear zone–hosted quartz veins. Precious metals are also present in sediment hosted and vein occurrences in the Mountains and Foothills of Alberta and the Western Canada Sedimentary Basin.
Placer Gold
Placer gold deposits occur in modern alluvial and glacial deposits throughout Alberta. Alberta's alluvial gold is often called 'flour gold' due to its relatively small grain size, with most grains being between 125 and 250 μm, and it is sometimes accompanied by platinum-group elements, other heavy minerals, and diamonds. Alberta placer gold deposits have multiple sources, including Mesozoic to Tertiary bedrock, hydrothermal veins and replacements, reworked glacial sediments, and other preglacial fluvial deposits.
Placer gold deposits have been mined at various places in Alberta since 1861. Particularly high concentrations occur along the North Saskatchewan River in the Edmonton area and produced about 1 tonne of gold in a century. Placer gold and other heavy minerals are presently being recovered as by-products of sand and gravel operations. Elsewhere in Alberta, placer gold has been found in the Peace, Athabasca, McLeod, Red Deer, and Milk rivers.
Paleoplacer gold
Paleoplacer gold (± platinum-group metals, magnetite, ilmenite, rutile, and other heavy minerals) occurs in consolidated fluvial and marine-beach clastic sedimentary rocks of the Mesoproterozoic Purcell Supergroup, Lower Cretaceous Blairmore Group and Upper Cretaceous Brazeau Group in southwestern Alberta, and unconsolidated to lithified sand and gravel deposits of Tertiary to Pleistocene age throughout Alberta that predate Laurentide glaciation. These Tertiary to Pleistocene sand and gravel deposits are sources of gold recovered as a by-product preglacial sand and gravel mining.
Placer Gold Occurrences in Alberta
Investigation of Potential Paleoplacers in the Cretaceous Strata of the North Saskatchewan River Watershed
Review of Metallic Mineralization in Alberta with Emphasis on Gold Potential
Regional Metallogenic Evaluation of Alberta
Going for gold – Alberta Energy


text:
Young Borehamwood sportsman paralysed after collapsing with unknown rare condition
"We were told he might not make it and that crushed us"
Charlie ReynoldsMultimedia Journalist
Charlie with his mum, Carole, and dad Lee
Charlie Crosby was 23 years old and had his whole life ahead of him.
The Borehamwood-born man was a promising golfer, footballer and was the life and soul of every room he stepped in.
But during the night of June 6 last year, he collapsed out of the blue and was found by his girlfriend in her room in a pool of his own sick.
In a panic, she called for an ambulance and he was rushed into hospital leaving everyone fearing what had happened to him.
Join the #IAmOpen community and give your business a boost
Access to lots of FREE tools to help stabilise your business and start making up for lost time is just one newsletter sign up away. As part of our #IAmOpen community to help and support small businesses owners like you, you will get a regular newsletter from our journalists plus we'll let you know how you can:
get exclusive access to business webinars
have a single-page website landing page made
montage video created
book an online marketing health check report and
bag discounted advertising rates just made for our mates
What's more, it's all FREE. So sign up here right now. It's very least we can do to back you, support you and just say THANK YOU.
Nearly a year on, nothing could have prepared them for the hell they would have had to endure.
Hours later, Charlie was transferred to Queen Square hospital, in London, the National Hospital for Neurology and Neurosurgery. Once there, he was rushed down to surgery as doctors discovered a bleed on his brain.
It was all stemming from an arteriovenous malformation (AVM) on the left side of his brain and he had to endure a painstaking eight-hour life-threatening operation. An AVM is described as a tangle of blood vessels - in Charlie's case, the pressure over the years all came too much and it had ruptured.
An AVM can develop anywhere in the body, but it mostly occurs in the spine or brain. Even so, brain AVMs are still rare, and are believed to affect less than one per cent of the population. As the causes of the condition is not clear, most people are born with them, or they can occasionally form them later in life.
He was put in intensive care and placed on a life support machine, something which unbeknown to his loved ones, he would be on for the next two months.
Shocked, petrified and feeling sick to their stomach - Charlie's parents, Carole and Lee, didn't know what to do.
How to get the latest Hertfordshire news direct to your inbox
This however was sadly the tip of the iceberg and was only the start of the hellish roller coaster the family have been on.
To this day, Charlie has had three life-threatening operations, he can't walk unaided, can only speak single words and has looked death in the eye a number of times.
Life for the Crosby family has never been the same.
"We were told he might not make it"
Before this heartbreaking incident, Charlie had a wonderful life.
He had a great job in something he was good at and had great friends, family and a girlfriend.
His competitive nature made him strive at sports and he played both golf and football at semi-pro level.
Charlie in intensive care
He was a fighter - something which his parents think got him through his gut wrenching time in hospital.
Fight was something he needed in the hospital bed as just two days after his first life-threatening operation, he deteriorated and fluid was building on his brain.
He was still in an induced coma and on a life support machine.
Once again, he was rushed into surgery and doctors had to remove a chunk of his skull to ease the pressure on his brain.
Carole and Lee were told the news no parent should hear - "There isn't much more we can do, he might not survive."
They explained: "From that moment on, our lives became hell.
"Our boy was healthy three days ago, nothing was wrong with him and now doctors were telling us he could die.
"They had given him all the drugs they could, all the surgery they could, but nothing was working.
"He had a collapsed lung, pneumonia, his bowels and bladder had shut down at one point and it was horrible.
"It is hard to explain. He is our only child, our pride and joy and he could have been taken away from us.
"They didn't think he'd make it, but he did."
In the days after the operation, Charlie became stable, but he wasn't out of the woods just yet.
Doctors explained that his life after this would be very different and they were proved right - he is paralysed on the right side of his body and struggles speaking.
"Everyday we would sit there and pray that he'd survive"
Charlie and his girlfriend Alicia on his 24th birthday recently
Watching their son on life support, not moving a muscle and attached to multiple machines keeping him alive was hard to take.
Carole and Lee described themselves as "broken people" and were just walking around in a daze, trying to get through day by day.
It ultimately took over their lives.
Everyday, they'd get to the hospital as early as they could and would sit by his bed, praying he'd pull through.
As the days went on, they did everything to make sure he felt like he wasn't alone but they didn't even know if he could hear them.
They would play his favourite music, speak to him, hold his hand and tell him 'everything was going to be alright'.
HertsLive features
The young man whose cancer has returned
Family lose everything in fire
Major issue a town faces for its future
Tribute to teen who died in crash
They said: "We would sit there day-by-day watching the pressure readings, just staring at it, praying it would go down.
"It was just a horrid time for us.
"When he was in intensive care, it just showed you how cruel life can be.
"Men and women being brought in with the same problems as Charlie and were dying all the time. It was heartbreaking.
"We just can't thank the incredible nurses who were there and saved his life. They are amazing."
"We would cling on to any progress, then it all got snatched away"
Charlie was in intensive care for around two months and coming to the end of that period, he made small progress.
He moved from ward to ward and slowly the Crosby family were seeing glances of the old Charlie.
"At first, doctors would give him less drugs and say his blood pressure was decreasing," said Lee.
"Then we would see his eyes slowly open, it was only the tiniest gap but it would get wider and wider.
"Then one day, he randomly reached out and put a hat on his head and we were gobsmacked.
"He loves wearing hats and we put it on his bed hoping he would remember what it was because really we didn't know because of the impact on his brain.
"That sounds like nothing, but it is a moment we will never forget."
During this critical period, Charlie was making great progress.
With the help of doctors, in August, he got out of bed for the first time and was even playing noughts and crosses with his mum and dad.
He couldn't speak but that was expected.
Days after, Carole and Lee left after spending the day by his bedside in high spirits.
But as they got home, they got a call from the hospital saying Charlie had a bad turn out of the blue and he had vomited and aspirated (breathing foreign objects into the airways).
They were told to 'fear the worst'.
Carole and Lee said: "It was devastating. We didn't sleep that night and he had to go into emergency brain surgery again to reduce the swelling.
"He had to have a shunt fitted in his head to control the pressure and he was back in intensive care on a life support machine again.
"Mentally this destroyed us. We were heartbroken and were back at step one."
The support for Charlie and his parents spurred them all on
Charlie with Rowland Thomas who donated this specialised exercise bike
After that major setback, Charlie began improving again and was moved into the rehabilitation centre of the hospital.
This meant he could see more of his friends and family and could focus on speech therapy, getting movement back into his right side of the body and walking.
Carole recalls the countless times when his mates - 12 of them at times - would come and see him and his face would light up.
"It was like a whole football team," she said.
And it was his friends and family who were supporting him and his parents during their ordeal.
Follow HertsLive on social media
Follow HertsLive on Facebook - Like our Facebook page to get the latest news in your feed and join in the lively discussions in the comments. Click here to like and follow it.
You can also join our local Facebook groups here.
Follow us on Twitter - For breaking news and the latest stories, click here to follow us on Twitter.
Follow us on Instagram - On the HertsLive Instagram page we share amazing pictures of our stunning county, and if you tag us in your posts, we could repost your picture on our page. Click here to follow Herts Live on Instagram.
Knowing he would need private therapy in the future, his friends set up a fundraising page to help pay for it.
People were running marathons and climbing mountains for him and to date, have raised nearly £20,000.
Something which the Crosby family will never forget.
They said: "We can't explain how grateful we are. The generosity shown has blown us away.
"But private therapy is so expensive. We have been unable to work as we could before and funding his rehab isn't realistic.
"So this goes a long way but it won't last - we will do everything we can to help him though, that's not even a question."
"He looks back at photos of himself and gets frustrated and upset - it is so cruel"
Charlie loved playing golf
Up until March this year, Charlie hadn't left the hospital, but he was advised to because of the coronavirus outbreak.
Charlie is clinically vulnerable so he moved back home with his mum and dad, who are now his main carers.
Although it is lovely having him home, there is only so much they can do and they do not hold professional health care knowledge.
Lee said: "We have tried to do what we can with the physio and the therapy and we've done a good job.
"But he still can't walk unaided, can't use his right arm and hand, and struggles with his speech.
"He has to wear a crash helmet because of the part of the skull they took out last year and he just gets so frustrated.
"He is the same person inside and is constantly on his phone looking at how he was before all of this. It is so cruel."
Next week is another hurdle which Charlie has to get over as he is going into hospital for another major surgery to fit a plate into the head where his skull was removed.
One thing that is for certain, which they admit, is the family would have "bitten someone's hand off if they said in a year he would be where he is now."
The family wanted to share their son's journey to raise awareness of the hidden dangers of AVM.
To help the family continue getting the support Charlie needs, you can click here.


You can save thousands of dollars on a swimming pool by opting for an above ground option rather than an in-ground swimming pool. But there are all sorts of questions that come with the decision to purchase an above ground swimming pool.
In-ground pools seem to have so many options, but what about above group pools? Rest assured, you have plenty of options to choose from… especially when it comes to shape and size.
These pools come in a ton of different sizes and three different shapes. Most above ground pools are round, rectangular, or oval.
Round pools are the most common. They have a standard size range from 10 to 33 feet across (with some really small easy set-up options as small as 6 feet in diameter) and the wall height can be anywhere between 30 and 54 inches high (with the smallest as low as 20 inches). Although 33 feet is a substantial amount of swimming area, round pools aren't usually useful for laps or sports swimming.
Can I Get an Oval or Rectangular Pool?
Oval and rectangular pools are available, and these shapes usually have sturdier designs. Oval and Rectangular pools taper at the corners to provide an adequate structure to the pool's sides.
Oval pools range in standard size of 12 or 16 feet wide by 16 to 32 feet long. Oval above ground pools require substantial side support.
Rectangular above ground pools are normally 12 or 16 feet wide and 20 to 32 feet long. Rectangular pools are best for lap pools or pools surrounded by a deck.
For either an oval or a rectangular above ground pool it's easy to set up a tether system for laps or sports swimming. However, there's no shape or size for above ground pools that would be quite suitable for diving.
The positives of above ground pools are straightforward. If you're looking for something to cool off with during the summer that is low maintenance, then an above ground pool is for you. You can put away the pools during the winter and store them in some cases.
One of the biggest pros of an above ground pool is that you save thousands of dollars. In-ground pools cost approximately $20,000, and that a no-frills version. In above-ground pools however, you can find a wide variety of shapes and sizes for less than $1,000.
So, what's the downside? Well, above ground pools have no impact on property value. If you're looking to increase the value of your home, an in-ground pool is your best option. In some cases, above ground pools may actually diminish the value of your property. Because they sit on one stretch of land for years at a time, the ground it's over is cut off from sunlight, and water sources. You may need to reseed or sod your lawn to fill in any gaps from your above ground pool. As previously mentioned, the other downside is that there is no above ground pool option available for diving.
What's the Deepest Above Ground Pool Option?
There are some variations of partially in-ground pools or the option of above-ground pools with a "deep end" (or "expandable") liner. However, when it comes to a standard above ground pool options, the deepest you can find is 54 inches or six feet of wall height.
The water line of most pools sits about 6 inches beneath the top of the wall when you account for some water loss from splashing. If you need a pool with water deeper than just under six feet, then you need either a deep-end liner or an in-ground pool.
What Are the Largest and Smallest Options?
Small pools are great for people who have limited space in their backyard; not to mention that it's a great way to save money. The smallest round pool you can find is about 12 feet in diameter. Oval pools run as small as 12 feet by 16 feet. Rectangle pools cater to the sporting swimmer, so they run a bit larger at 12 feet by 20 feet.
The largest sizes can benefit a large family or anyone who hosts many parties throughout the summer. Round pools run as large as 33 feet in diameter and 4 ½ feet deep. Oval and rectangular pools run deeper than the round options at 6 feet deep. Oval and rectangular pools are made as large as 16 feet wide by 32 feet long.
If you're looking for a pool but need something that can fit comfortably in your backyard as well as your budget, an above ground option is your best bet. If you're looking for the best above ground pool option for you, it's important to get the right size and shape.


text:
Solo Leveling Season 2: Delayed? Release Date & Everything We Know So Far!
"Solo Leveling" is an anime based on a Korean web publication written by Chu-Gong. It was printed under the Papyrus tag and announcement by the D&C Media. The Webnovel has published in English the series. In English, it goes by the title"Just I Level Up."
Solo Leveling Season 2
The narrative revolves in a world that's full of monsters. A group of people who go by the title"predators" combat these monsters. The lead of the story is Sung Jin-Woo, who unlike other predators, doesn't have any skill. But he is among the E seekers one of the predators.
Solo Leveling Season 2 Release Date
This series has a fan base all around the world. The show became successful after its first season finished 2020, in March and published in 2018. The fans have been asking since about the launch of this season.
No confirmation was received concerning the show's upcoming season. There were strategies of showing the release date for period 2, but so far there are no updates. The sequel season would probably be delayed until late 2021.
Updates on the season and its cast are awaited. But, according to news, the cast of the first season will be back for the next season. The cast of the first time comprised Hwang Chi-Yeul Park Hee-Jin, Jo Byung-Gyu, and Proceed Joon-Hee.
Confirmations on such members are to come from the group. Furthermore, there might be new characters who get added to the cast as the plot progresses. This means there may be additions to the cast.
There is very little insight into the plot of this next season. Therefore, the news says that the storyline of the season may follow from the very first year's scheme for the time being. The season showed two gates — one leading to worlds to the brighter along with other.
In the first season, Jin-Woo went to the search for his father and also beat up King Ant. The season may show whether he can locate his dad or not and what happens to his hunt. He may also face several new challenges and critters.
Tags Solo leveling, Solo leveling Season 2, Solo leveling Season 2 release date


Shrewsbury Town exited the FA Youth Cup at the Second Round Proper stage after defeat at Ilkeston on Wednesday night.
Despite taking the lead through a Chris Gallagher penalty early on, Town were beaten in heavy wind and rain at New Manor Ground courtesy of goals from Lewis Walker (2), Ryan Head and substitute Mason Barlow.
Ilkeston will host Newcastle United in the Third Round Proper as a result of their victory.
Head of Academy coaching Ian Dawes made five changes to the side which drew 1-1 with Wrexham at the weekend.
Ethan Jones, Callum Roberts, Callum Grogan, Chris Gallagher and John McAtee all returned to the starting line-up.
Town got the game underway in heavy wind and rain at New Manor Ground and took the lead in the 8th minute.
Joel Carta was tumbled in the box and Gallagher stepped up to slide the resulting spot-kick into the bottom left corner, sending the Ilkeston goalkeeper the wrong way.
McAtee then had a chance to double Town's lead a minute later when Mathew Jones pulled the ball back from the left, but the forward's shot from the edge of the box spun wide of the target.
And Ilkeston quickly found themselves an equaliser in the 13th minute.
The hosts threatened from a corner and when the ball came out to Rhys Dolan on the left, he crossed for Walker to plant a header into the bottom left corner from inside the box.
Shrewsbury almost retook the lead four minutes later, as Carta hit a first-time low shot from the edge of the box, but Jamie Hannis got down to his bottom right corner to tip the ball away.
Ethan Jones also looked to catch out the Ilkeston 'keeper at his near post with a fierce shot from a tight angle on the right but, at the other end, the hosts moved in front.
The young Shrews failed to clear Haydn Goddard's corner from the right and Head smashed the ball high into the roof of the net from 12 yards out.
Goalkeeper Leon Reece also had to remain alert to get his body behind a dipping shot from Ilkeston's Ben Morris two minutes later.
And the hosts held their advantage going into the break.
Ilkeston grabbed a third goal within a minute of the restart as Brandon Clarke crossed from the right and Walker acrobatically volleyed into the bottom left corner from the edge of the box.
Morris then also hit a half-volley over the Town crossbar from the edge of the box as Ilkeston continued to apply pressure.
At the other end, Callum Grogan saw a shot inside the box, following Callum Roberts' header back from Gallagher's corner, well blocked.
Reece was also called into action again as he had to pull off a smart low stop from Walker's close-range strike.
Ethan Jones also saw a fierce strike beaten away by the Ilkeston goalkeeper.
However, the hosts added a fourth in the 78th minute as substitute headed in from Barlow's corner.
There was one late chance for the Shrews to pull a goal back in injury-time as Ethan Jones beat two players on the inside left, but his shot at the end of it was straight at the goalkeeper.
And Ilkeston saw the game out to book their place in the Third Round Proper.
GOALS: Walker (13) (46), Head (23).
UNUSED SUBS: Billy Johnson, Max Thornberry, Charley Morton, Roy Lee.
UNUSED SUBS: Joel Bonner, Nathan Millis, Joe Ashley.


MotionUSA has expanded its capabilities to provide custom cables to fit your specific various requirements including custom cable designs and manufacturing for one-off or production quantities.
We can provide a wide variety of high quality cable types with any gauge wires needed, shielded or unshielded, straight or twisted pair, standard flex or high flex, and robotic.
We can also accommodate any variety of connector types including circular, D-Sub, power plugs, polarized, RCA, Molex, and Phoenix. Standard pin configurations, high density, crimp style, any soldering jobs. Whatever your cabling needs, we can help!
Call 800-463-5959 or send cable requirements to sales@motionusa.com to discuss your project and get a quote.
A world-leading manufacturer of circular connectors; suitable for nearly every industrial electronics application.
Cicoil invented flat cables in 1956, and are the world's leading manufacturer of flat cables and flat cable assemblies, complete with virtually any connector.
Manufacturer and supplier of a wide range of cables, wires & cables accessories with various characteristics.
Headquartered in Germany, icotek manufactures high quality products and aims to deliver complete customer and product support all over the world.
Igus uses its innovative polymer materials to develop functionally advanced, yet affordable polymer components and assemblies.
Contact MotionUSA: If you are looking for a quote or have any questions about Custom Cable Assemblies, please submit your request here! or call 1-800-463-5959 to speak with an engineer.


M also sells telescope eyepieces, as well as telescope accessories like filters, focusers, power supplies, and carrying cases.
Celestron expressly disclaims any lost profits, general, special, indirect or consequential damages which may result from breach of goku vs naruto who would win warranty, or arising out of the use or inability to use any Celestron product.
Previous Binocular/Spotting Scope Warranty Information, from January 1, 2005, through December 31, 2011, Celestron offered a No Fault Warranty on all binoculars and spotting scopes.
Free Gifts with delgrosso coupon code Purchase of select Telescopes.See All Member Benefits, helpCentre : Payment Security : Promotional CodesPromotional Codes, to Redeem a Promotional Code: Sign in to your account.Enter m Coupon in the Promotional Code Box and click Apply.Members have agents at their fingertips 24/7 for support with every device in their home, from Best Buy or not.Only one Promotional Code may be used per online order.Limitations of Normal Wear and Tear.Free Shipping on 250 order.For binoculars and spotting scopes, the customer must send a check or money order in the amount.95 made out to Celestron.Unused portions of a Promotional Code will become invalid after expiration.
Other Promotional Codes may be applied to any items (example: Save 10 on any purchase above 100).We make working with influencers easy, fast and cost-effective.Over time, optical coatings may develop small blemishes through normal wear and tear or abuse under various environmental conditions.Like LinkedIn, everyone has a social feed and profile, and can engage with the community.This warranty does not cover products that have been subject to abuse, misuse, physically damaged, altered, or had unauthorized repairs or modifications.This warranty is valid.S.A.Want to explore the cosmos without leaving your own backyard?Learn how some brands are seeing 500 ROI on influencer marketing using Dealspotr.As a brand, you can claim your store page to post exclusive discount codes, visible to our 1MM shoppers.
100 Success, gET deal 176 Used Today 6, oFF, dEAL 6 off to your 1st order.As olejo promo code seen on: Copyright Coupon Code, Inc.The Kensfish coupon discount will adjust your order total.Repashy Super Foods starting from.98 100 Success, gET deal 31 Used Today 7, oFF, dEAL, save 7 off..
Excludes clearance and Champion Life Collection items.45 off Sale Coupon Verified!Free Delivery.S, to discount classroom furniture get free shipping just apply this Hanes coupon code to your order!Some sellers also offer Thank.In store five golden rings gift ideas coupons can be printed at home or shown on your mobile device.Get..


text:
package com.simpligility.maven.plugins.androidndk.common;

import com.simpligility.maven.plugins.androidndk.AndroidNdk;
import org.apache.commons.io.FileUtils;
import org.apache.maven.artifact.Artifact;
import org.apache.maven.artifact.resolver.filter.AndArtifactFilter;
import org.apache.maven.artifact.resolver.filter.ArtifactFilter;
import org.apache.maven.artifact.resolver.filter.ExcludesArtifactFilter;
import org.apache.maven.artifact.resolver.filter.OrArtifactFilter;
import org.apache.maven.artifact.resolver.filter.ScopeArtifactFilter;
import org.apache.maven.model.Dependency;
import org.apache.maven.model.Exclusion;
import org.apache.maven.plugin.MojoExecutionException;
import org.apache.maven.plugin.logging.Log;
import org.apache.maven.project.MavenProject;
import org.apache.maven.shared.dependency.graph.DependencyGraphBuilder;
import org.apache.maven.shared.dependency.graph.DependencyNode;
import org.apache.maven.shared.dependency.graph.traversal.CollectingDependencyNodeVisitor;

import java.io.File;
import java.io.FileNotFoundException;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.LinkedHashSet;
import java.util.List;
import java.util.Scanner;
import java.util.Set;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

/**
 * @author Johan Lindquist
 */
public class NativeHelper
{

    public static final int NDK_REQUIRED_VERSION = 7;

    private MavenProject project;
    private DependencyGraphBuilder dependencyGraphBuilder;
    private Log log;

    public NativeHelper( MavenProject project, DependencyGraphBuilder dependencyGraphBuilder, Log log )
    {
        this.project = project;
        this.dependencyGraphBuilder = dependencyGraphBuilder;
        this.log = log;
    }

    public static boolean hasStaticNativeLibraryArtifact( Set<Artifact> resolveNativeLibraryArtifacts,
                                                          String ndkArchitecture )
    {
        for ( Artifact resolveNativeLibraryArtifact : resolveNativeLibraryArtifacts )
        {
            if ( Const.ArtifactType.NATIVE_IMPLEMENTATION_ARCHIVE.equals( resolveNativeLibraryArtifact.getType() ) )
            {
                return true;
            }
        }
        return false;
    }

    public static boolean hasSharedNativeLibraryArtifact( Set<Artifact> resolveNativeLibraryArtifacts,
                                                          String ndkArchitecture )
    {
        for ( Artifact resolveNativeLibraryArtifact : resolveNativeLibraryArtifacts )
        {
            if ( Const.ArtifactType.NATIVE_SYMBOL_OBJECT.equals( resolveNativeLibraryArtifact.getType() ) )
            {
                return true;
            }
        }
        return false;
    }

    public Set<Artifact> getNativeDependenciesArtifacts( boolean sharedLibraries )
            throws MojoExecutionException
    {
        log.debug( "Finding native dependencies. shared=" + sharedLibraries );
        final Set<Artifact> filteredArtifacts = new LinkedHashSet<Artifact>();
        final Set<Artifact> allArtifacts = new LinkedHashSet<Artifact>();

        // Add all dependent artifacts declared in the pom file
        // Note: The result of project.getDependencyArtifacts() can be an UnmodifiableSet so we
        //       have created our own above and add to that.
        allArtifacts.addAll( project.getDependencyArtifacts() );

        // Add all attached artifacts as well - this could come from the NDK mojo for example
        allArtifacts.addAll( project.getAttachedArtifacts() );

        // Add all transitive artifacts as well
        // this allows armeabi classifier -> apklib -> apklib -> apk chaining to only include armeabi in APK
        allArtifacts.addAll( project.getArtifacts() );

        for ( Artifact artifact : allArtifacts )
        {
            log.debug( "Checking artifact : " + artifact );
            // A null value in the scope indicates that the artifact has been attached
            // as part of a previous build step (NDK mojo)
            if ( isNativeLibrary( sharedLibraries, artifact.getType() ) && artifact.getScope() == null )
            {
                // Including attached artifact
                log.debug( "Including attached artifact: " + artifact + ". Artifact scope is not set." );
                filteredArtifacts.add( artifact );
            }
            else
            {
                if ( isNativeLibrary( sharedLibraries, artifact.getType() ) && (
                        Artifact.SCOPE_COMPILE.equals( artifact.getScope() ) || Artifact.SCOPE_RUNTIME
                                .equals( artifact.getScope() ) ) )
                {
                    log.debug( "Including attached artifact: " + artifact + ". Artifact scope is Compile or Runtime." );
                    filteredArtifacts.add( artifact );
                }
            }
        }

        Set<Artifact> transitiveArtifacts = processTransitiveDependencies( project.getDependencies(), sharedLibraries );

        filteredArtifacts.addAll( transitiveArtifacts );

        return filteredArtifacts;
    }

    private boolean isNativeLibrary( boolean sharedLibraries, String artifactType )
    {
        return ( sharedLibraries
                ? Const.ArtifactType.NATIVE_SYMBOL_OBJECT.equals( artifactType )
                : Const.ArtifactType.NATIVE_IMPLEMENTATION_ARCHIVE.equals( artifactType )
        );
    }

    private Set<Artifact> processTransitiveDependencies( List<Dependency> dependencies, boolean sharedLibraries )
            throws MojoExecutionException
    {
        final Set<Artifact> transitiveArtifacts = new LinkedHashSet<Artifact>();
        for ( Dependency dependency : dependencies )
        {
            if ( !Artifact.SCOPE_PROVIDED.equals( dependency.getScope() ) && !dependency.isOptional() )
            {
                final Set<Artifact> transArtifactsFor = processTransitiveDependencies( dependency, sharedLibraries );
                log.debug( "Found transitive dependencies for : " + dependency + " transDeps : " + transArtifactsFor );
                transitiveArtifacts.addAll( transArtifactsFor );
            }
        }

        return transitiveArtifacts;

    }

    private Set<Artifact> processTransitiveDependencies( Dependency dependency, boolean sharedLibraries )
            throws MojoExecutionException
    {
        log.debug( "Processing transitive dependencies for : " + dependency );

        try
        {
            final Set<Artifact> artifacts = new LinkedHashSet<Artifact>();

            final List<String> exclusionPatterns = new ArrayList<String>();
            if ( dependency.getExclusions() != null && !dependency.getExclusions().isEmpty() )
            {
                for ( final Exclusion exclusion : dependency.getExclusions() )
                {
                    exclusionPatterns.add( exclusion.getGroupId() + ":" + exclusion.getArtifactId() );
                }
            }
            final ArtifactFilter optionalFilter = new ArtifactFilter()
            {
                @Override
                public boolean include( Artifact artifact )
                {
                    return !artifact.isOptional();
                }
            };

            final AndArtifactFilter filter = new AndArtifactFilter();
            filter.add( new ExcludesArtifactFilter( exclusionPatterns ) );
            filter.add( new OrArtifactFilter( Arrays.<ArtifactFilter>asList( new ScopeArtifactFilter( "compile" ),
                    new ScopeArtifactFilter( "runtime" ),
                    new ScopeArtifactFilter( "test" ) ) ) );
            filter.add( optionalFilter );

            final DependencyNode node = dependencyGraphBuilder.buildDependencyGraph( project, filter );
            final CollectingDependencyNodeVisitor collectingVisitor = new CollectingDependencyNodeVisitor();
            node.accept( collectingVisitor );

            final List<DependencyNode> dependencies = collectingVisitor.getNodes();
            for ( final DependencyNode dep : dependencies )
            {
                final boolean isNativeLibrary = isNativeLibrary( sharedLibraries, dep.getArtifact().getType() );
                log.debug( "Processing library : " + dep.getArtifact() + " isNative=" + isNativeLibrary );
                if ( isNativeLibrary )
                {
                    artifacts.add( dep.getArtifact() );
                }
            }

            return artifacts;
        }
        catch ( Exception e )
        {
            throw new MojoExecutionException( "Error while processing transitive dependencies", e );
        }
    }

    public static void validateNDKVersion( File ndkHomeDir ) throws MojoExecutionException
    {
        final File ndkVersionFile = new File( ndkHomeDir, "RELEASE.TXT" );

        if ( !ndkVersionFile.exists() )
        {
            throw new MojoExecutionException(
                    "Could not locate RELEASE.TXT in the Android NDK base directory '" + ndkHomeDir.getAbsolutePath()
                            + "'.  Please verify your setup! " + AndroidNdk.PROPER_NDK_HOME_DIRECTORY_MESSAGE );
        }

        try
        {
            String versionStr = FileUtils.readFileToString( ndkVersionFile );
            validateNDKVersion( NDK_REQUIRED_VERSION, versionStr );
        }
        catch ( Exception e )
        {
            throw new MojoExecutionException( "Error while extracting NDK version from '"
                    + ndkVersionFile.getAbsolutePath() + "'. Please verify your setup! "
                    + AndroidNdk.PROPER_NDK_HOME_DIRECTORY_MESSAGE );
        }
    }

    public static void validateNDKVersion( int desiredVersion, String versionStr ) throws MojoExecutionException
    {

        int version = 0;

        if ( versionStr != null )
        {
            versionStr = versionStr.trim();
            Pattern pattern = Pattern.compile( "[r]([0-9]{1,3})([a-z]{0,1}).*" );
            Matcher m = pattern.matcher( versionStr );
            if ( m.matches() )
            {
                final String group = m.group( 1 );
                version = Integer.parseInt( group );
            }
        }

        if ( version < desiredVersion )
        {
            throw new MojoExecutionException( "You are running an old NDK (version " + versionStr + "), please update "
                    + "to at least r'" + desiredVersion + "' or later" );
        }
    }

    public static String[] getAppAbi( File applicationMakefile )
    {
        Scanner scanner = null;
        try
        {
            if ( applicationMakefile != null && applicationMakefile.exists() )
            {
                scanner = new Scanner( applicationMakefile );
                try
                {
                    while ( scanner.hasNextLine() )
                    {
                        String line = scanner.nextLine().trim();
                        if ( line.startsWith( "APP_ABI" ) )
                        {
                            return line.substring( line.indexOf( ":=" ) + 2 ).trim().split( " " );
                        }
                    }
                }
                finally
                {
                    scanner.close();
                }
            }
        }
        catch ( FileNotFoundException e )
        {
            // do nothing
        }
        return null;
    }

    /**
     * Extracts, if embedded correctly, the artifacts architecture from its classifier.  The format of the
     * classifier, if including the architecture is <architecture>-<classifier>.  If no
     * architecture is embedded in the classifier, 'armeabi' will be returned.
     *
     * @param artifact            The artifact to retrieve the classifier from.
     * @param defaultArchitecture The architecture to return if can't be resolved from the classifier
     * @return The retrieved architecture, or defaulArchitecture if not resolveable
     */
    public static String extractArchitectureFromArtifact( Artifact artifact, final String defaultArchitecture )
    {
        String classifier = artifact.getClassifier();
        if ( classifier != null )
        {
            //
            // We loop backwards to catch the case where the classifier is
            // potentially armeabi-v7a - this collides with armeabi if looping
            // through this loop in the other direction
            //

            for ( int i = AndroidNdk.NDK_ARCHITECTURES.length - 1; i >= 0; i-- )
            {
                String ndkArchitecture = AndroidNdk.NDK_ARCHITECTURES[ i ];
                if ( classifier.startsWith( ndkArchitecture ) )
                {
                    return ndkArchitecture;
                }
            }

        }
        // Default case is to return the default architecture
        return defaultArchitecture;
    }

    /**
     * Attempts to extract, from various sources, the applicable list of NDK architectures to support
     * as part of the build.
     *

     *

     * It retrieves the list from the following places:
     *

     *
ndkArchitecture parameter

     *
projects Application.mk - currently only a single architecture is supported by this method

     *

     *
     * @param ndkArchitectures    Space separated list of architectures.  This may be from configuration or otherwise
     * @param applicationMakefile The makefile (Application.mk) to retrieve the list from.
     * @param basedir             Directory the build is running from (to resolve files)
     * @return List of architectures to be supported by build.
     * @throws MojoExecutionException
     */
    public static String[] getNdkArchitectures( final String ndkArchitectures, final String applicationMakefile, final File basedir ) throws MojoExecutionException
    {
        // if there is a specified ndk architecture, return it
        if ( ndkArchitectures != null )
        {
            return ndkArchitectures.split( " " );
        }

        // if there is no application makefile specified, let's use the default one
        String applicationMakefileToUse = applicationMakefile;
        if ( applicationMakefileToUse == null )
        {
            applicationMakefileToUse = "jni/Application.mk";
        }

        // now let's see if the application file exists
        File appMK = new File( basedir, applicationMakefileToUse );
        if ( appMK.exists() )
        {
            String[] foundNdkArchitectures = getAppAbi( appMK );
            if ( foundNdkArchitectures != null )
            {
                return foundNdkArchitectures;
            }
        }

        // return a default ndk architecture
        return new String[]{ "armeabi" };
    }

    /**
     * Helper method for determining whether the specified architecture is a match for the
     * artifact using its classifier.  When used for architecture matching, the classifier must be
     * formed by <architecture>-<classifier>.
     * If the artifact is legacy and defines no valid architecture, the artifact architecture will
     * default to armeabi.
     *
     * @param ndkArchitecture Architecture to check for match
     * @param artifact        Artifact to check the classifier match for
     * @return True if the architecture matches, otherwise false
     */
    public static boolean artifactHasHardwareArchitecture( Artifact artifact, String ndkArchitecture,
                                                           String defaultArchitecture )
    {
        return Const.ArtifactType.NATIVE_SYMBOL_OBJECT.equals( artifact.getType() )
                && ndkArchitecture.equals( extractArchitectureFromArtifact( artifact, defaultArchitecture ) );
    }

}

redpajama_set_name: RedPajamaGithub