text:
\section{Details on bitext analysis}\label{sec:grammatical_gender_pronouns}
\paragraph{Complexity} We follow~\citet{chuntin} and compute the
corpus complexity as a measure of translation uncertainty. Concretely, having access to an alignment model (here, \texttt{fast-align}), the complexity of a corpus $d$ is computed by averaging the entropy of target words $y$ conditioned on the source $x$,
$L(d)=\frac{1}{|\mathcal{V}_x|}\sum_{x \in \mathcal{V}_x} H(y|x)$.
\paragraph{Coverage} We follow~\citet{tu-etal-2016-modeling} and measure the coverage of each source-target parallel pair as the ratio of source words being aligned to target words, having access to an alignment model (here, \texttt{fast-align}). We compute the coverage for source-target and target-source bitexts separately. Corpus-level statistics correspond to average sentence-level results.
\paragraph{Grammatical Gender Pronouns}
The complete lists of grammatic gender pronouns we use for \textsc{el} are:
[ο, του, τον, αυτός,  αυτού,  αυτόν, εκέινος, εκέινου, εκείνον, οποίος, οποίου, οποίον  ] and [η, της, την, αυτήν,  αυτής,  αυτήν, εκέινη, εκέινης, εκείνην, οποία, οποίας, οποίαν].
\paragraph{Lexical Differences (LeD)} We follow~\cite{Niu_Carpuat_2020} and compute the Lexical Differences score between two sentences $S_1$ and $S_2$ as the percentage of tokens that are not found in both, LeD$=\frac{
1}{2}(\frac{|S_1 /\ S_2|}{|S_1|}) + \frac{|S_2 /\ S_1|}{|S_2|}$.

\section{Result on development sets}\label{sec:dev_results}
Table~\ref{tab:dev_results} presents results on the main and secondary \textsc{nmt} tasks on \textsc{ted} developments sets. The refined bitext leads to consistent and significant improvements in \textsc{bleu}\xspace across language-pairs and translation directions.
\begin{table}[!t]
    \centering
    \scalebox{1.0}{
    \begin{tabular}{rccc}

    \rowcolor{gray!10}
    \multicolumn{4}{c}{\textit{Table}~\ref{tab:en2el_bleu}} \\
    \rowcolor{gray!5}
   \multicolumn{2}{c}{\textsc{en}$\rightarrow$\textsc{el}} & \multicolumn{2}{c}{\textsc{el}$\rightarrow$\textsc{en}}\\
    $1:$     & $25.50 \pm 0.15$ & $10:$ & $27.98 \pm 0.18$\\
    $2:$     & $25.52 \pm 0.07$ & $11:$ & $27.92 \pm 0.15$\\
    $3:$     & $24.55 \pm 0.25$ & $12:$ & $27.70 \pm 0.15$\\
    $4:$     & $25.35 \pm 0.14$ & $13:$ & $27.99 \pm 0.15$\\
    $5:$     &  $25.27 \pm 0.41$ & $14:$ & $28.36 \pm0.13$*\\
    $6:$     & $25.66 \pm 0.05$* & $15:$ & $28.34 \pm0.18$*\\
    $7:$     & $25.73 \pm 0.14$* &  $16:$ & $28.66 \pm0.14$* \\
    $8:$     & $25.71 \pm 0.19$* & $17:$ & $28.65 \pm0.27$*\\
    $9:$     & $\mathbf{25.91 \pm 0.09}$* & $18:$ & $\mathbf{29.00 \pm0.26}$*\\

    \addlinespace[1em]

    \rowcolor{gray!10}
    \multicolumn{4}{c}{\textit{Table}~\ref{tab:mt}} \\
    \rowcolor{gray!5}
    \multicolumn{2}{c}{\textsc{en}$\rightarrow$\textsc{ro}} & \multicolumn{2}{c}{\textsc{ro}$\rightarrow$\textsc{en}}\\

    $1:$ & $21.94\pm0.11$ & $3:$  & $24.98\pm0.16$ \\
    $2:$ & $\mathbf{22.05\pm0.03}$* & $4:$ & $\mathbf{26.11\pm0.20}$* \\

    \end{tabular}}
    \caption{\textsc{bleu}\xspace results on the \textsc{ted} developments sets for each of the results of Tables~\ref{tab:en2el_bleu} and \ref{tab:mt} (enumeration follows the main text Tables). * denotes one standard deviation improvements over the original bitexts.}
    \label{tab:dev_results}
\end{table}

\section{\texttt{Sockeye2} configuration details}\label{sec:sockeye}

We use the base Transformer architecture \cite{attention}.
with embedding size of $512$, transformer hidden size of $2{,}048$, $8$ attention heads, $6$ transformer layers, and dropout of $0.1$.
Target embeddings are tied with the output layer weights.
We train with label smoothing ($0.1$). We optimize with Adam~\cite{Kingma2015AdamAM} with a batch size of $4{,}096$ tokens and checkpoint models every $1{,}000$ updates. The initial learning rate is $0.0002$, and it is reduced by $30$\% after $4$ checkpoints without validation perplexity improvement. We stop training after $20$ checkpoints without improvement. We select the best checkpoint based on validation \textsc{bleu}\xspace~\cite{papineni-etal-2002-bleu}. All models are trained on a single GeForce \textsc{gtx} $1080$ \textsc{gpu}. Tables~\ref{tab:sockeye_main_config} presents details of \textsc{nmt}\xspace training with~\texttt{Sockeye2}.
\begin{table}[!t]
    \centering
    \scalebox{0.84}{
    \begin{tabular}{|l|}

    \hline
    \texttt{--weight-tying-type="trg\_softmax"} \#uni-\textsc{nmt}\\
    \texttt{--weight-tying-type="src\_trg\_softmax"} \#bi-\textsc{nmt} \\
    \texttt{--num-words $5000$:$5000$} \\
    \texttt{--label-smoothing $0.1$} \\
    \texttt{--encoder transformer} \\
    \texttt{--decoder transformer} \\
    \texttt{--num-layers $6$} \\
    \texttt{--transformer-attention-heads $84$} \\
    \texttt{--transformer-model-size $512$} \\
    \texttt{--num-embed $512$} \\
    \texttt{--transformer-feed-forward-num-hidden $2048$} \\
    \texttt{--transformer-preprocess n} \\
    \texttt{--transformer-postprocess dr} \\
    \texttt{--gradient-clipping-type none} \\
    \texttt{--transformer-dropout-attention $0.1$} \\
    \texttt{--transformer-dropout-act $0.1$} \\
    \texttt{--transformer-dropout-prepost $0.1$} \\
    \texttt{--max-seq-len $80$:$80$}\\
    \texttt{--batch-type word} \\
    \texttt{--batch-size $2048$} \\
    \texttt{--min-num-epochs $3$} \\
    \texttt{--initial-learning-rate $0.0002$} \\
    \texttt{--learning-rate-reduce-factor $0.7$} \\
    \texttt{--learning-rate-reduce-num-not-improved $4$} \\
    \texttt{--checkpoint-interval $1000$} \\
    \texttt{--keep-last-params $30$} \\
    \texttt{--max-num-checkpoint-not-improved $20$} \\
    \texttt{--decode-and-evaluate $1000$} \\
    \hline
    \end{tabular}}
    \caption{\textsc{nmt}\xspace configurations on \texttt{Sockeye2}}
    \label{tab:sockeye_main_config}
\end{table}
\newpage
\section{mt5 configuration details}\label{sec:mt5_settings}
Tables~\ref{tab:mT5_main_config} presents details of continued training of mT5 on \texttt{SimpleTransformers}.
\begin{table}[!t]
    \centering
    \scalebox{0.84}{
    \begin{tabular}{|l|}

    \hline
    \texttt{max-seq-length $100$} \\
    \texttt{train-batch-size $10$} \\
    \texttt{eval-batch-size $10$} \\
    \texttt{num-train-epochs $5$} \\
    \texttt{scheduler 'cosine schedule with warmup'} \\
    \texttt{evaluate-during-training True} \\
    \texttt{evaluate-during-training-steps $10000$} \\
    \texttt{learning-rate $0.0003$} \\
    \texttt{optimizer 'Adafactor'} \\
    \texttt{use-multiprocessing False} \\
    \texttt{save-model-every-epoch True} \\
    \texttt{use-early-stopping False} \\
    \texttt{do-lower-case True} \\
    \hline
    \end{tabular}}
    \caption{\textsc{nmt}\xspace configurations for continued training of mT5 on \texttt{SimpleTransformers}.}
    \label{tab:mT5_main_config}
\end{table}

\section{Data Statistics}\label{sec:data_statistics_appendix} Table~\ref{tab:data_stats} presents data statistics for WikiMatrix training data, and \textsc{ted} evaluation sets.
\begin{table}[!t]
    \centering
    \scalebox{0.9}{
    \begin{tabular}{lllr}
    \toprule
    \textbf{\textsc{language pair}} &  \textbf{\textsc{training}} & \textbf{\textsc{dev.}} & \textbf{\textsc{Test}} \\
    \bottomrule
    \addlinespace[0.2em]
    \textsc{el-en} & $750{,}585$  & $3{,}344$ & $4{,}431$\\
    \textsc{ro-en} & $582{,}134$  & $3{,}904$ & $4{,}631$ \\
    \bottomrule
    \end{tabular}}
    \caption{Data statistics after pre-processing.}
    \label{tab:data_stats}
\end{table}

\begin{table}[!t]
    \centering
    \scalebox{0.9}{
    \begin{tabular}{lll}
    \toprule
    \textbf{\textsc{language pair}} & \multicolumn{1}{c}{\textsc{uni-nmt}} & \multicolumn{1}{c}{\textsc{bi-nmt}}\\
    \bottomrule
    \addlinespace[0.2em]
    \textit{\textsc{en}\xspace} $\rightarrow$ \textit{\textsc{el}\xspace}     & $27.89\pm0.29$ & $27.92\pm0.06$ \\
     \textit{\textsc{el}\xspace} $\rightarrow$ \textit{\textsc{en}\xspace}        & $29.63\pm0.29$ & $29.57\pm0.36$\\
    \textsc{ro} $\rightarrow$ \textit{\textsc{en}\xspace}       &  $24.54\pm0.06$        & $24.69\pm0.11$\\
    \textsc{en} $\rightarrow$ \textsc{ro}       & $20.84\pm0.04$         & $20.73\pm0.12$\\
    \bottomrule
    \end{tabular}}
    \caption{\textsc{bleu}\xspace scores for \textsc{nmt} on equivalized bitexts using uni- (\textsc{uni-nmt})  vs. bi-directional \textsc{nmt}\xspace models (\textsc{bi-nmt}). Equivalizing the bitext with \textsc{bi-nmt} \textsc{nmt} yields comparable \textsc{bleu}\xspace  with \textsc{uni-nmt}
    }
    \label{tab:speeding_up}
\end{table}%

\section{Manual inspection of \textsc{bli}}\label{sec:bli_false_false_positives}
Table~\ref{tab:ground_truth_bli} presents manual analysis results on False Positives entries of the \textsc{muse} evaluation set for the \textsc{en-el} language-pair.
\begin{table}[!t]
    \centering
        \scalebox{0.5}{
    \begin{tabular}{lll@{\hskip 0.2in}|@{\hskip 0.2in}lll}
        \toprule
        \multicolumn{3}{c}{\textbf{Revised}} & \multicolumn{3}{c}{\textbf{Original}}  \\[1pt]
        \toprule
        αστεροειδές & star	              & 	?       & απόστολος & apostolos    & \xmark \\
        προσφέρεται & offers	          &  \cmark	    & βραχνό & raucous         &  \xmark \\
        κεραυνός & keravnos	              &  \xmark     & μπανζούλ & bangaon       & \xmark \\
        συμπυκνώνει & encapsulates	      & 	?       & βοηθητικές & auxiliary   & \cmark\\
        σεξτέτο & sexteto	              & \cmark 	    & ομιλήτρια & spokesperson & \cmark \\
        επιχειρηματολογία & argumentation & \cmark 	    & πρωτεργάτη & forerunner  &  \xmark \\
        επίπλωση & furniture	          & \cmark  	& αντιτρομοκρατική & anti-terrorist & \cmark \\
        μπουγκ & bug	                  & \xmark 	    & πλεκτά            & sweaters      & \cmark \\
        σχετικοί & related	              & \cmark 	    & εμβολιαστεί & vaccinated &    \cmark \\
        δορυφόρους & moons	              & \xmark 	    & αταξινόμητες & unclassified &  \cmark \\
        δειλή & timid	                  & \cmark	    & στέιν & steen                 & \xmark \\
        χάντινγκτον & huntingdon	      & \cmark 	    & χιλιοστό & millimeter & \cmark \\
        ποσότητες & amounts	              & \cmark	    & σελεστίν & célestine & \cmark \\
        πλακέ & squamous	              & \cmark 	    & κόβατς & kovács & \xmark\\
        αποποίηση & relinquishing	      & ?       	& σεμίνα & omni & \xmark \\
        ατμούς & vapors	                  & \cmark 	    & σπάιντερμαν & spider-man & \cmark \\
        τερματισμοί & endings	          & \cmark  	& πάνω & over & \cmark \\
        αλεξανδρινό & alexandrine	      & \cmark  	& ενδιαφέρων & love & \xmark \\
        σπασμοί & fits	                  & ? 	        & αγριόγατες & cats &      \xmark \\
        σίδερα & sidelines	              & \xmark	    & αγορα & trade & \cmark \\
        συνοδεύονται & are	              & \xmark 	    & επικεφαλίδα & header & \cmark \\
        διανέμονται & are	              & \xmark  	& μάσλοου & khan & \xmark \\
        θραύση & fracturing	              & \cmark  	& τεχνητά & artificially & \cmark \\
        κυβερνά & rule	                  & \cmark  	& πέτροβιτς & petrović & \cmark \\
        συνάξεις & meetings	              & \cmark  	& ανθίζει & flowers & \cmark \\
        χριστιανία & christianity	      & \cmark  	& ζήτω & vive & \xmark \\
        απειλούνται  & are	              & \xmark 	    & τυλίγει & picks & \xmark \\
        ποινικοποίηση & penalize	      & \cmark 	    & μπαέζ & ross & \xmark \\
        στερέωμα & stardom	              & \xmark 	    & φιλοδοξεί & is & \xmark \\
        τζαπ & elford	                  & \xmark 	    & τρυφερή & loving & ? \\
        ταυρομαχία & bullfighting	      & \cmark 	    & σωρός & remains & \xmark \\
        χειρός & handbags	              & ? 	        & χαλυβουργεία & works & \xmark \\
        κδ & cd	                          & ? 	        & μάιρα & chloe & \xmark \\
        τρομοκρατεί & terrorizes	      & \cmark 	    & συγκλόνισε & shocked & \cmark \\
        μακέι & mackey	                  & \cmark 	    & άτακτη & mischievous & \cmark \\
        ζάκυνθο & zakynthos	              & \cmark 	    & οταν & after & ?  \\
        συμπτωματολογία & symptomology	  & \cmark	    & εντομοφάγα & insectivores & \cmark \\
        πολυφυλετική & polyphyletic	      & \cmark 	    & κραδασμούς & vibrations & \cmark \\
        κούνια & cunha	                  & \xmark 	    & μπελάς & nuisance & \cmark \\
        καταβεβλημένος & overcome	      & \cmark 	    & πάστες & pastries & \cmark \\
        απάτες & scams	                  & \cmark 	    & διασπαστική & divisive & \cmark \\
        γιάννη & giannis	              & \cmark 	    & κατάληψη & capture & \xmark \\
        δηλητηριάσεις & poisonings	 & \cmark	   & παραδίδονται & surrender & \cmark \\
        φιλόξενοι & colorful	     & \xmark 	   & κλήρων & clergy & \cmark \\
        φημισμένος & renowned	     & \cmark      & σκεύη & vessels & \cmark \\
        φουσκωμένα & filled	         & ? 	       & λεπτονίων & leptons & \cmark \\
        υπονοούμενα & undertones	 & \cmark	   & εξάγονται & are & \xmark \\
        όριο & boundary	             & \cmark 	   & απότομο & abrupt & \cmark \\
        χαλάρωσε & relaxed	         & \cmark 	   & παρασυμπαθητικό & sympathetic & ? \\
        αισθητικός & aesthetic	     & \cmark 	   & ταρίχευση & embalming & \cmark \\
        ταμαντούα & tamanduas	     & \cmark 	   & κεκτημένο & precedent & \xmark \\
        εστίες & foci	             & ?	       & καλκούτα & kolkata & \cmark \\
        θεωρείται & is	             & \xmark 	   & σίρι & sirri & \cmark \\
        κορμό & trunk	             & \cmark 	   & ξεπερασμένο & obsolete & \cmark \\
        σπύρο & spyros	             & \cmark 	   & ανώμαλος & bumpy & \cmark \\
        αναισθητικά & anesthetics	 & \cmark	   & εξισορρόπησης & substance & \xmark \\
        στρατηγικές & strategic	     & \cmark 	   & πολυσακχαρίτης & polysaccharides & \cmark \\
        αναπνέει & breathe	         & \cmark 	   & επίπονες & persistent & \cmark \\
        εξουδετερώσει & neutralize	 & \cmark 	   & αμφιθέατρο & amphitheatre & \cmark \\
        μελαγχολική & melancholic	 & 	\cmark     & αναπληρωματικό & an & \xmark \\
        θυμήθηκε & recalled	         & \cmark 	   & εντελώς & entirely & \cmark\\
        πασχαλίτσα & ladybird	     & \cmark 	   & λιθόστρωτο & cobbled & \cmark \\
        πυροκροτητές & caps	         & ? 	       & διοικητικοί & administrative & \cmark \\
        κραυγαλέα & screaming	     & ? 	       & κομιστής & bearer & \cmark \\
        μολδαβίας & moldavia	     & \cmark 	   & συλλογικότητες & competitions & \xmark \\
        σαλιγκάρι & shilling	     & \xmark 	   & χουλιγκανισμού & micromanagement & \xmark \\
        ενισχυθεί & enhance	         & \cmark 	   & τσάρους & tsars & \cmark \\
        πρεσβυτέριο & presbytery	 & \cmark	   & ντόνελ & dorff & \xmark \\
        μάγιστρος & master	         & \cmark 	   & κίραν & kiran & \cmark \\
        αλτ & alt	                 & \cmark 	   & πρωτοποριακή & pioneering & \cmark \\
        χρονολογία & date	         & \cmark 	   & λένοξ & brookline & \xmark \\
        κανένα & any	             & \cmark 	   & λείπουν & are & \xmark \\
        κορμός & road	             & \xmark 	   & εξάντα & astronomy & \xmark \\
        καθαριστήριο & cleanup	     & \xmark 	   & πτωτική & downward & \cmark \\
        ανατεθεί & assigned	         & \cmark 	   & αρχιτεκτονικές & architectural & \cmark \\
        εξοικονόμηση & save	         & \cmark 	   & γαλλόφωνο & french-speaking & \cmark \\
        μπαρακούντα & barracudas	 & \cmark	   & μέντε & mede & \xmark \\
        ταυτοποίησης & identification	 & \cmark	   & εκθρονίζοντας & deposing & \cmark \\
        \toprule
    \end{tabular}}
    \caption{
    Manually labeled acceptability judgments for random $80$ error cases made by lexicons induced using the  original and revised bitexts. \cmark and {\xmark}
    denote acceptable and unacceptable translation, respectively. ? denotes word pairs that may be acceptable in rare or specific contexts.
    }
    \label{tab:ground_truth_bli}
\end{table}

\section{Streamlining equivalization}
Based on ablation analysis presented in Table~\ref{tab:en2el_bleu} the best equivalization strategies consider candidates from two \textsc{nmt}\xspace models trained independently to translate in opposite directions. In Table~\ref{tab:speeding_up} we show how our approach yields comparable results by replacing the two uni-directional models (\textsc{uni-nmt}) with a single bi-directional model (\textsc{bi-nmt})
while reducing training by $\sim30\%$.

\section{Introduction}

While human-written data remains the gold standard to train Neural Machine Translation (\textsc{nmt}\xspace) and Multilingual \textsc{nlp} models, there is growing evidence that synthetic bitext samples---sentence-pairs that are translated by \textsc{nmt}\xspace---benefit a wide range of tasks. They have been used to enable semi-supervised  \textsc{mt} training from monolingual data ~\cite{sennrich-etal-2016-improving, zhang-zong-2016-exploiting, hoang-etal-2018-iterative}, to induce bilingual lexicons~\cite{artetxe-etal-2019-bilingual, shi-etal-2021-bilingual}, and to port models trained on one language to another~\cite{conneau-etal-2018-xnli, yang-etal-2019-paws}.

While synthetic bitexts are useful additions to original training data for downstream tasks, it remains unclear how they differ from naturally occurring data. Some studies suggest that synthetic samples might be simpler and easier to learn~\citep{chuntin, xu-etal-2021-distilled}. Recognizing that naturally occurring bitext can be noisy,
for instance, when they are mined from comparable monolingual corpora~\cite{resnik-smith-2003-web,fung-yee-1998-ir,espla-etal-2019-paracrawl, schwenk-etal-2021-wikimatrix}, we hypothesize that synthetic bitext might also directly improve the equivalence of the two bitext sides. Thus synthetic samples might be useful not only for data augmentation but also to revise potentially noisy original bitext samples.

In this paper, we present a controlled empirical study comparing the quality of bitext mined from monolingual resources with a synthetic version generated via \textsc{mt}.
We focus on the widely used WikiMatrix bitexts for a distant (i.e, \textsc{en-el}) and a similar language-pair (i.e, \textsc{en-ro}), since it has been shown that this corpus contains a significant proportion of erroneous translations \cite{audit_multi}.
We generate synthetic bitext by translating the original training samples using \textsc{mt} systems trained on the bitext itself and therefore do not inject any additional supervision in the process. We also consider selectively replacing original samples with forward and backward synthetic translations based on a semantic equivalence classifier, which is also trained without additional supervision.

We show that the resulting synthetic bitext improves the quality of the original intrinsically using human assessments of equivalence and extrinsically on bilingual induction (\textsc{bli}) and \textsc{mt} tasks. We present an extensive analysis of synthetic data properties and of the impact of each step in its generation process. This study brings new insights into the use of synthetic samples in \textsc{nlp}. First, intrinsic evaluation shows that synthetic translations, in addition to ``normalizing'' the bitext~\citep{chuntin, xu-etal-2021-distilled},
could potentially provide reference translations that are more semantically equivalent to the source than the original ones.

Furthermore, the improved bitext provides more useful signals for \textsc{bli} tasks and \textsc{nmt}\xspace training in two settings (training from scratch; continued training), as confirmed by our extrinsic evaluations. Finally, ablation analyses that compare different ways to combine synthetic translations show that using \textit{both translation directions} and \textit{filtering using semantic equivalence} is key to improving bitext quality and calls for further exploration of best practices for using synthetic translations in \textsc{nlp} tasks.
\section{Background}
\paragraph{Synthetic Translations} Generating synthetic translations
has mainly been studied as a means of data augmentation for \textsc{nmt} through forward translation~\cite{zhang-zong-2016-exploiting} or back-translation~\cite{sennrich-etal-2016-improving, marie-etal-2020-tagged} of monolingual resources.
Moreover, recent lines of work use synthetic translations to augment the original parallel data: \citet{NEURIPS2020_7221e5c8} diversify the parallel data via translating both sides using multiple models and then merging them with the original to train a final \textsc{nmt} model;
\citet{jiao-etal-2020-data} employ a similar approach to rejuvenate inactive examples that contribute the least to the model performance. Sequence-level knowledge distillation \citep{kim-rush-2016-sequence}  can also be viewed as replacing original bitext with synthetic translations.  While its original goal was to guide the training of a student model of small capacity with the output of a teacher of high capacity, distillation is also necessary to effectively train some categories of \textsc{mt} architectures such as non-autoregressive models \citep{jiatao}. While it is not entirely clear why synthetic distilled samples are superior to original bitext in this case, recent studies suggest that the synthetic samples are simpler and thus easier to learn from \citep{chuntin, xu-etal-2021-distilled}.
\paragraph{Synthetic Data Selection} Prior work covers
a wide spectrum of different selection strategies on top of synthetic translations generated from monolingual samples. Each of them focuses on identifying samples with specific properties:
\citet{axelrod-etal-2011-domain} sample sentences that are most relevant to a target domain with the goal of creating pseudo in-domain bitext;
\citet{hoang-etal-2018-iterative} generate synthetic parallel data iteratively from increasingly better back-translation models for improving unsupervised \textsc{nmt};
\citet{fadaee-monz-2018-back} focus on the diversity of synthetic samples and
sample synthetic translations containing words that are difficult to predict using prediction losses and frequencies of words.
By contrast, our empirical study investigates whether synthetic translations can be used to \textit{\textbf{selectively replace}} original references to improve bitext quality rather than augmenting it.
\paragraph{Bitext Quality}
Mining bitext from the web results in large-scale corpora that are usually collected without guarantees about their quality. For instance, they contain noisy samples, ranging from untranslated sentences to sentences with no linguistic content~\cite{khayrallah-koehn-2018-impact,caswell-etal-2020-language}. Some of this noise is typically filtered out automatically using heuristics~\cite{ramirez-sanchez-etal-2020-bifixer} or \textsc{nmt} model scores~\cite{junczys-dowmunt-2018-dual, koehn-etal-2019-findings}. Yet, even after this noise filtering, a wide range of the remaining samples contains fine-grained semantic divergences~\cite{briakou-carpuat-2020-detecting}. Our past work explored strategies to mitigate the impact of these divergences on \textsc{mt} models by incorporating divergence tags as token-level factors \cite{briakou-carpuat-2021-beyond}, and designing an approach to automatically edit divergent samples with noisy supervision from monolingual resources \cite{bitextedit}.
By contrast, this work explores whether synthetic translations can be used to replace potentially fine-grained divergences using only the bitext we seek to revise.
\section{Approach}

\input{tables/algo}

This section describes the methods and data we use to produce revised bitexts for our empirical study.

\subsection{Methods for Revising Bitext}\label{sec:equivalize_NMT}
We rely on established techniques that can be applied using only the bitext that we seek to revise. First, we train \textsc{nmt}\xspace models on the original bitext to translate in both directions. For each original sentence-pair, we generate a pool of synthetic translations using \textsc{nmt}\xspace and apply a divergence ranking criterion to decide whether and how to replace the original references with a better translation. Algorithm~\ref{alg:algo} gives an overview of the process, and we describe each step below.
\paragraph{Generating synthetic translations} We train \textsc{nmt}\xspace models $M_{S \rightarrow T}$ and $M_{T \rightarrow S}$ on the original bitext to translate in each direction~(lines~$2$-$3$). For each sentence-pair,  they are used to generate two candidates for replacement
by forward and backward translation~(lines~$6$-$7$): ($S_i,M_{S \rightarrow T}(S_i)$) and ($M_{T \rightarrow S}(T_i),T_i$). As a result, \textsc{nmt}\xspace models translate the exact same data that they are trained on. We thus expect translation quality to be high
, and that local errors in the original bitext might be corrected by the translation patterns learned by \textsc{nmt}\xspace models on the entire corpus.

\paragraph{Selective Replacement} We propose to replace an original pair by a candidate \textit{only if}
 the candidate is predicted to better convey the meaning of the source than the original, which we refer to as the \textit{semantic equivalence condition}. We implement this by
 ranking the original sample ($S_i,T_i$), its revision by forward translation ($S_i,M_{S \rightarrow T}(S_i)$) and its revision by back-translation ($M_{T \rightarrow S}(T_i),T_i$), according to their degree of semantic equivalence. If none of the synthetic samples score higher than the original, it is not replaced~(line~$17$). Otherwise, the original is replaced by the highest scoring synthetic sample  (lines~$10$-$15$). As a result the cardinality of the bitext remains constant.
 The semantic equivalence condition ($d_{F}$ and $d_{B}$ (lines~$8$-$9$)) is implemented using divergentm\textsc{bert}, a divergent scorer introduced in our prior work \cite{briakou-carpuat-2020-detecting} that is trained on synthetic samples generated by perturbations of the original bitext (e.g., deletions, lexical or phrasal replacements) performed without any bilingual information.

\subsection{Experimental Set-Up}\label{sec:setup_human}

\paragraph{Bitext} We evaluate the use of synthetic translations for revising bitext on two language pairs of the WikiMatrix corpus~\cite{schwenk-etal-2021-wikimatrix}. WikiMatrix consists of sentence-pairs mined from Wikipedia pages using language agnostic sentence embeddings (\textsc{laser}\xspace)~\cite{laser}. Prior work indicates that, as expected, the corpus as a whole comprises many samples that are not exact translations: \citet{audit_multi} report that for more than half of the audited low-resource language-pairs, mined pairs are on average misaligned; \citet{briakou-carpuat-2020-detecting} find that $40\%$ of a random sample of the English-French bitext are not semantically equivalent, and include fine-grained meaning differences in addition to alignment noise.
We focus on bitexts with fewer than one million sentence pairs in Greek$\leftrightarrow$English (\textsc{el}$\leftrightarrow$\textsc{en}, with $750{,}585$ pairs) and
Romanian$\leftrightarrow$English (\textsc{ro}$\leftrightarrow$\textsc{en}, with $582{,}134$ pairs), because improving bitext is particularly needed in this data regime. In much higher resource settings, filtering strategies might be sufficient as there might be more high quality samples overall. In much lower resource settings, the data is likely too noisy or too small to effectively revise bitexts using \textsc{nmt}\xspace.
We filter out noisy pairs in the training data using
\texttt{bicleaner}~\cite{ramirez-sanchez-etal-2020-bifixer} so that our empirical study excludes the most obvious forms of noise, and focuses on the harder case of revising samples that standard preprocessing pipelines consider to be clean.\footnote{\url{https://github.com/bitextor/bicleaner}}

\paragraph{Preprocessing} We use Moses~\cite{koehn-etal-2007-moses} for punctuation normalization, true-casing, and tokenization. We learn $32$K \textsc{bpe}s~\cite{sennrich-etal-2016-neural} per language using \texttt{subword-nmt}~\footnote{\url{https://github.com/rsennrich/subword-nmt}}.

\paragraph{\textsc{nmt} Models} We use the base Transformer architecture \cite{attention} and include details on the exact architecture and training in Apendix~\ref{sec:sockeye}.

\paragraph{Selective Replacement} The divergence ranking models are trained using our public  implementation of divergentm\textsc{bert}~\cite{briakou-carpuat-2020-detecting}.\footnote{\url{https://github.com/Elbria/xling-SemDiv}}
Synthetic divergences are generated starting from the $5{,}000$ top scoring WikiMatrix sentences based on \textsc{laser} score (i.e., seed equivalents).
We fine-tune the "\textsc{bert}-Base Multilingual Cased" model~\cite{devlin-etal-2019-bert}
and set the margin equal to $5$ as per our original implementation. We use the same margin value for the margin score of Algorithm~\ref{alg:algo}.
\footnote{Our divergentm\textsc{bert} yields $84$ F$1$ on a set of English-French human-annotated fine-grained divergences in WikiMatrix collected in our prior work~\cite{briakou-carpuat-2020-detecting}.}

\section{Intrinsic Evaluation of Bitext Quality}

\subsection{Human evaluation}

We ask $3$ bilingual speakers to evaluate the quality of the \textsc{en}-\textsc{el} bitexts.
Given an original source sentence, they are asked to rank the original target and the candidate target in the order of their equivalence to the source. They are asked ``Which sentence conveys the meaning of the source better?'', and ties are allowed. A random sample of $100$ pairs from forward and backward \textsc{MT} is annotated.

As can be seen in Table~\ref{tab:intrinsic_evaluation_results}, $60\%$ of \textsc{all} synthetic candidates are better translations of the WikiMatrix reference, which confirms the potential of \textsc{nmt}\xspace for improving over original translations. Further ablations confirm the benefits of selecting these synthetic candidates with the semantic equivalence condition. When the divergent scorer ranks a candidate higher than the original
by a small margin (i.e., $0\leq d \leq 5$ given $d=R(S_i,M_{S\rightarrow T}(T_i))-R(S_i,T_i))$), human evaluation shows that the candidate is actually better than the original only $51\%$ of the times.
When using our exact semantic equivalence condition~($d>5$), candidates are judged as more equivalent than the original $87.5\%$ of the times,
and annotations within this set have a stronger agreement (i.e., $0.688$ Kendall's τ).
This indicates that
the condition
$d>5$ identifies more clear-cut examples of synthetic translations that fix semantic divergences
in the original data and can be thus used for selective replacement of imperfect references by better quality translations.

\begin{table}[!t]
    \centering
    \scalebox{1.00}{
    \begin{tabular}{lcc}
    \rowcolor{gray!10}
    Candidate set &  \% Equivalized  & Kendall's τ   \\
    \addlinespace[0.5em]
    \textsc{all}   & $60.0\%$ & $0.321$  \\
    $d<0$ & $26.4\%$ & $0.157$ \\
    $0\leq d \leq 5$ & $51.0\%$ & $0.234$  \\
    \rowcolor{gray!5}
    $d>5$     & $\mathbf{87.5\%}$ &  $\mathbf{0.688}$\\

    \end{tabular}}
    \caption{Human evaluation results for all evaluated pairs and ablation sets
    for different thresholds on divergent score differences between candidates and originals~(i.e., $d$).
    }\vspace{-0.5cm}
    \label{tab:intrinsic_evaluation_results}
\end{table}

\input{tables/analysis_tables}

Further inspection of the annotations reveals that most source-target WikiMatrix examples contain fine meaning differences ($56\%$). In those cases, we observe that most of the content between the sentences is shared, but either small segments are mistranslated (e.g., ``London'' instead of ``Athens'' in the first example of Table~\ref{tab:examples_led}), or some information is missing from either side of the pair (e.g., ``all six'' missing from the target side in the third example of Table~\ref{tab:examples_led}). Furthermore, more coarse-grained divergences are found less frequently ($12\%$)---in those cases, we notice that sentences are usually either topically related or structurally similar (e.g., length, syntax) with a few anchor words (e.g., last example in~Table~\ref{tab:examples_led}). Finally, $32\%$ of the times the original WikiMatrix pairs are perfect translations of each other.

\subsection{How do synthetic translations differ from originals?}

Figure~\ref{fig:LeD_variants} presents the distribution of lexical differences (i.e., computed using LeD---a score that captures lexical differences based on the percentages of tokens that are not found in two sentences~\cite{Niu_Carpuat_2020}) between original and synthetic translations (in \textsc{en}) for candidates that replace and do not replace the originals.~\footnote{LeD details are in~Appendix \ref{sec:grammatical_gender_pronouns}.}
First, we observe that a substantial amount of synthetic translations that do not replace original references ($40\%$) corresponds to small \textsc{LeD} scores ($<0.1$), suggesting that the equivalence criterion could fall back to the original sentence not because of the poor quality  of candidate references, but rather due to them being already close to the originals.
Furthermore, all synthetic translated instances are represented in almost all bins, with fewer instances found on the extreme bins of $>0.7$ \textsc{LeD} scores. Finally, synthetic translations that replace original references are mostly concentrated within the range $[0.2, 0.6]$ of LeD scores. This indicates that they share lexical content with the original, which further supports the hypothesis that synthetic translations revise fine-grained meaning differences in WikiMatrix in addition to alignment noise.

\subsection{How does the revised bitext differ from the original?}\label{sec:revised_properties}

\begin{figure}[!t]
    \centering
    {\includegraphics[width=0.43\textwidth]{analysis_figs/led_new_legends.pdf}}
    \caption{LeD differences of original vs. synthetic translations (\textsc{el}$\rightarrow$\textsc{en}).
    Replaced candidates share lexical content with the originals.
    }
    \label{fig:LeD_variants}
\end{figure}

Table~\ref{tab:biases} presents differences in statistics of the original vs. revised  WikiMatrix \textsc{en-el} bitexts to shed more light on the impact of selectively using synthetic translation for bitext quality improvement.~\footnote{Details on the metrics are in~Appendix \ref{sec:grammatical_gender_pronouns}.}  The refined bitext exhibits higher coverage (i.e., ratio of source words being aligned by any target words; rows $5$ and $13$) and smaller complexity (i.e., the diversity of target word choices given a source word~\cite{chuntin}) compared to the original bitext. Moreover, the use of synthetic translations introduces small decreases in the lexical types covered in the final corpus~(i.e., rows $3$ and $11$), which is expected as the additional coverage in the original corpus might be a result of divergent texts. Those observations are in line with prior work that seeks to characterize the nature of synthetic translations used in other settings, such as knowledge distillation~\cite{chuntin, xu-etal-2021-distilled}.

While fixing divergent references contributes to this simplification effect, \textsc{nmt}\xspace translations might also reinforce unwanted biases from the original bitext. For instance, the distribution of two grammatical gender pronouns on the English side is a little more imbalanced in the improved bitext than in the original (rows $6$-$7$ and $14$-$15$),
\footnote{We limit our analysis to \# occurrences for two grammatical gender pronouns. The complete list is in Appendix \ref{sec:grammatical_gender_pronouns}.}
likely due to gender bias in \textsc{nmt}\xspace~\cite{stanovsky-etal-2019-evaluating}. This calls for techniques to mitigate such biases \citep{saunders-byrne-2020-reducing,stafanovics-etal-2020-mitigating} for \textsc{nmt}\xspace and other downstream tasks.

\section{Extrinsic Evaluation of Bitext Quality}
Our previous analysis suggests that selective replacement of divergent references with synthetic translations results in bitext of \textit{improved quality}, with reduced level of noises and easier word-level mappings between the two languages, when compared to the original WikiMatrix corpus. To better understand how those differences impact downstream tasks, we contrast the improved bitext with the original through a series of extrinsic evaluations for \textsc{en-el} and \textsc{en-ro} languages that rely on parallel texts as training samples~(see \S\ref{sec:downstream_tasks}). First, we focus on the recent state-of-the-art unsupervised \textsc{bli} approach of \citet{shi-etal-2021-bilingual} that relies on word-alignments of extracted bitexts.
Second, we follow the recent bitext quality evaluation frameworks adopted by the ``Shared Task on Parallel Corpus Filtering and Alignment''  \cite{koehn-etal-2020-findings} and built neural machine translation systems from scratch and by continued training on a multilingual pre-trained transformer model.
Finally, we conduct extensive ablation experiments to test the impact of using synthetic translations without the semantic equivalence condition and contrast with familiar techniques used by prior work~(see \S\ref{sec:ablations}).

\begin{table*}[!t]
    \centering
    \scalebox{1.00}{
    \begin{tabular}{l@{\hskip 0.1in}l@{\hskip 0.4in}lllr@{\hskip 0.6in}omh}
    \toprule[2pt]
    &       &    \multicolumn{4}{c}{\textit{All}} & \textit{Low} & \textit{Medium} & \textit{High} \\

     \textbf{\textsc{pair}}   & \textbf{\textsc{bitext}} & \multicolumn{1}{c}{Precision} & Recall & F1 & \textsc{oov} rate  & \multicolumn{3}{c}{Precision}  \\\toprule
        \multirow{2}{*}{\rotatebox{0}{\textsc{el-en \Big\{}}} & Original     & $76.2$\phh       & $58.1$ & $65.9$ & $\mathbf{6.7\%}$ &  $59.4$ & $76.6$ & $81.4$\\
                                                        & Revised  & $\mathbf{77.6^*}$ & $\mathbf{58.6^*}$ & $\mathbf{66.8^*}$ & $7.5\%$ & $\mathbf{60.4^*}$ & $\mathbf{78.4^*}$ & $81.6$\\

       \multirow{2}{*}{\rotatebox{0}{\textsc{en-ro} \Big\{}}  & Original     & $89.2$ & $69.4$ & $78.1$ & $\mathbf{15.8\%}$ &  $78.6$ & $86.9$ & $87.1$\\
                                                        & Revised  & $\mathbf{90.8^*}$ &  $\mathbf{71.3^*}$ & $\mathbf{79.8^*}$ & $16.5\%$ & $\mathbf{80.0^*}$  & $\mathbf{87.5^*}$ & $86.9$\\
                                \toprule[2pt]
    \end{tabular}}
    \caption{
    Unsupervised \textsc{bli} extrinsic evaluation results on \textsc{muse} for the entire dataset (\textit{All}) and on subsets binned by frequency (i.e., right-most highlighted columns).
    Revised bitexts yield statistically significant~($*$) improvements over the original bitexts overall and for low-to-medium frequency dictionary entries.}
    \label{tab:BLI_main}
\end{table*}

\subsection{Experimental Set-Up}

\paragraph{\textsc{bli}}
The task of \textsc{bli} aims to induce a bilingual lexicon consisting of word translations in two languages.
We experiment with the recently proposed method of \citet{shi-etal-2021-bilingual} that combines extracted bitext and unsupervised word alignment to perform fully unsupervised induction based on extracted statistics of aligned word pairs.
The induced lexicons are evaluated based on \textsc{muse}~\cite{lample2018word} consisting of $45{,}515$ and $80{,}815$ dictionary entries for \textsc{el-en} and \textsc{en-ro}, respectively.\footnote{\url{https://github.com/facebookresearch/MUSE}}
We extract word alignments using m\textsc{bert}-based \textit{Simalign}\footnote{\url{https://github.com/cisnlp/simalign}} \cite{jalili-sabet-etal-2020-simalign} and statistics based on the implementation of \citet{shi-etal-2021-bilingual}.\footnote{\url{https://github.com/facebookresearch/bitext-lexind}}

\paragraph{\textsc{mt}}
We experiment with \textsc{mt} tasks following two approaches:
(1) training standard transformer seq2seq models from scratch;
(2) continued training for mT5~\cite{xue-etal-2021-mt5}, a multilingual pre-trained text-to-text transformer.
We evaluate translation quality with \textsc{bleu}\xspace~\cite{papineni-etal-2002-bleu}\footnote{\url{https://github.com/mjpost/sacrebleu}} on the official development and test splits of the \textsc{ted}\xspace corpus~\cite{qi-etal-2018-pre}. \footnote{Data statistics are found in Appendix~\ref{sec:data_statistics_appendix}.}
For (1) we follow the experimental settings described in~\S\ref{sec:setup_human}.
For (2) we initialize the weights of transformer with ``mT5-small'' which consists of $300$M parameters,\footnote{\url{https://github.com/google-research/multilingual-t5}}. We use the \texttt{simpletransformers} implementation.\footnote{\url{https://github.com/ThilinaRajapakse/simpletransformers}} We fine-tune for up to $5$ epochs and include the parameter settings in Appendix~\ref{sec:mt5_settings}.

\paragraph{Ablation Settings}
We compare the \textsc{nmt}\xspace models trained on the variants of the synthetic bitext to isolate the impact of replacement criteria and different candidates.\footnote{Results on development sets are in Appendix \ref{sec:dev_results}.}
For the former, we experiment with the \textbf{rejuvenation} approach of \citet{jiao-etal-2020-data} that replaces original references with forward translated candidates for the $10\%$ least active original samples measured by \textsc{nmt}\xspace probability scores.
Moreover, we experiment with \textbf{forward} and \textbf{backtranslation} baselines trained on bitexts that consist solely from target- or source-side candidate sentences (i.e., original references are entirely excluded) and with ablations that consider either forward or backward candidates for the proposed semantic equivalence condition.
Finally, we consider two alternatives to
the \textbf{semantic equivalence} condition based on divergent scores: the
\textbf{ranking} condition replaces a candidate if it scores higher than the original (i.e., margin with $d=0$) and the \textbf{thresholding} condition adds the additional constraint that candidates should rank higher than a threshold to replace the original pair.

\subsection{Extrinsic Evaluation Results}\label{sec:downstream_tasks}

\paragraph{\textsc{bli}}
Table~\ref{tab:BLI_main} presents results for unsupervised \textsc{bli} on the \textsc{muse} gold-standard dictionaries, for \textsc{el-en} and \textsc{en-ro}. Across languages, the revised bitexts induce better lexicons compared to the original WikiMatrix. Crucially, improvements are reported both in terms of Recall---which connects to the observation that the revised bitext exhibits higher coverage than the original and in terms of Precision---which connects to the noise reduction effect that impacts the extracted word alignments. Additionally, a break-down on the Precision of the induced lexicons binned by the frequency of \textsc{muse} source-side entries (i.e., last $3$ columns in Table~\ref{tab:BLI_main}) reveals that the improvements come from better induction of low- and medium-frequency words, which we expect are more sensitive to noisy misalignments that result from divergent bitext. Finally, those improvements are reported despite the small increase of the \textsc{oov} rate in the revised lexicons that results from the decrease in the lexical types covered in it, as mentioned in the analysis (i.e., ~\S\ref{sec:revised_properties}).

Furthermore, following the advice of \citet{kementchedjhieva-etal-2019-lost} who raise concerns on \textsc{bli} evaluations based on gold-standard pre-defined dictionaries, we accompany our evaluation with manual verification to confirm that our conclusions are consistent with those of the automatic evaluation. Concretely, we manually check the \textit{false positives} induced translation pairs from the original vs. the improved bitext. We found that $65/80$ are \textit{false false positives} (due to incompleteness of pre-defined dictionaries) for the improved bitext and $51/80$ for the original (see Appendix \ref{sec:bli_false_false_positives} for the complete list). This confirms that the metric improvements we observe are meaningful and suggests that the improved bitext help learn better mappings between source and target words.

\paragraph{\textsc{mt}}

\begin{table}[!t]
    \centering
     \scalebox{1.0}{
    \begin{tabular}{l@{\hskip 0.35in}c@{\hskip 0.35in}c}
    \toprule[2pt]
    \textsc{\textbf{pair}} & \textbf{\textsc{original}} & \textbf{\textsc{revised}} \\
    \toprule[0.5pt]
    \textsc{el}$\rightarrow$\textsc{en} & $28.15$ {\small$\pm 0.13$} & $\mathbf{29.63}$ {\small $\mathbf{\pm 0.29}$} \\
    \textsc{en}$\rightarrow$\textsc{el} & $27.08$ {\small$\pm 0.18$} & $\mathbf{27.89}$ {\small$\mathbf{\pm 0.05}$} \\
    \textsc{ro}$\rightarrow$\textsc{en} & $23.68$ {\small$\pm 0.12$} & $\mathbf{24.54}$ {\small$\mathbf{\pm 0.06}$} \\
    \textsc{en}$\rightarrow$\textsc{ro} & $20.65$ {\small$\pm 0.10$} & $\mathbf{20.84}$ {\small$\mathbf{\pm 0.04}$} \\
    \toprule[2pt]
    \end{tabular}}
    \caption{\textsc{bleu} on \textsc{nmt} training from scratch.
    }
    \label{tab:mt}
\end{table}
\begin{figure}[!t]
    \centering
    \includegraphics[scale=0.4]{analysis_figs/mt5_new_legends.png}
    \caption{\textsc{bleu} scores across epochs (x-axis) for continued training on mt5. The revised bitext improves translation quality compared to the original for all epochs and translation tasks.}\vspace{-0.3cm}
    \label{fig:mt5}
\end{figure}
\input{tables/English-Greek-main}

Table~\ref{tab:mt} presents translation quality (\textsc{bleu}\xspace) on \textsc{en}$\leftrightarrow$\textsc{ro} and \textsc{en}$\leftrightarrow$\textsc{el} tasks for \textsc{mt} training from scratch and Figure~\ref{fig:mt5} shows translation quality of mT5 continued training across epochs. Across tasks and settings, the revised bitext yields better translation quality than the original WikiMatrix data.
The consistent improvements we observe across the two settings suggest that the properties of the synthetic translations that replace original samples and bring those improvements are invariant to specific models. Moreover,
the magnitude of improvements is larger in the continued training setting compared to training from scratch (e.g., $\sim+0.8$ vs. $\sim+1.5$, for \textsc{en}$\rightarrow$\textsc{el}; $ \sim+0.2$ vs. $\sim+1.5$, for \textsc{ro}$\rightarrow$\textsc{en}). The latter suggests that improvements from using synthetic samples do not only come from the normalization effect (i.e., synthetic samples are easier to model by \textsc{nmt}) but also connect to the reduced noise in the training samples. This further complements our hypothesis that synthetic translations can improve the quality of imperfect references that should, in principle, yield noisy training signals---and thus impact the resulting quality---of different \textsc{mt} models.

\subsection{Ablation Study}\label{sec:ablations}
Table~\ref{tab:en2el_bleu} compares the translation quality (\textsc{bleu}) of \textsc{nmt} systems trained on different synthetic translations. By forcing the semantic equivalence condition when deciding whether a synthetic translation replaces an original, we revise $50\%$ of the latter
yielding the best results across directions with significant improvements~(i.e, increases do not lie within $1$ stdev of the original's bitext performance) of $+0.81$~(\textsc{en}$\rightarrow$\textsc{el}, row $9$) and $+1.49$~(\textsc{el}$\rightarrow$\textsc{en}, row $18$)
points over the original bitext.

\paragraph{Impact of semantic equivalence condition} Table~\ref{tab:en2el_bleu} shows that naively disregarding the original references and training only on synthetic translations gives mixed results: training on \textit{forward-translated} references only~(i.e., row $2$) gives small improvements ($+0.36$) over the model trained on WikiMatrix for \textsc{en}$\rightarrow$\textsc{el}, while it performs comparably to it for \textsc{el}$\rightarrow$\textsc{en}~(i.e.,~row~$11$). On the other hand, training on \textit{backward} data only~(i.e.,~row~$12$) improves \textsc{bleu} by a small margin ($+0.23$) for \textsc{mt} into \textsc{en} while it hurts \textsc{bleu} when translating into \textsc{el}~(i.e.,~row~$3$). This indicates that the good quality of the synthetic translations  cannot be taken for granted and motivates replacing original pairs under conditions that account for semantic controls.

The latter is further confirmed by results on the rejuvenation baseline: replacing candidates for the $10\%$ of the most inactive WikiMatrix samples results in small and insignificant
increases in \textsc{bleu} when compared to models trained on original WikiMatrix data~(i.e., rows $1$-$4$ and $10$-$13$).  This indicates that rejuvenation might not be well-suited to lower resource settings than the ones it was originally tested on \citep{jiao-etal-2020-data}.
The rejuvenation technique might be affected by the decreased \textsc{nmt}\xspace quality and calibration in lower resource settings. By contrast, using synthetic translations with semantic control mitigates their impact.

Finally, all three semantic control variants based on divergent scores
yield bitexts that improve \textsc{bleu} compared to the original WikiMatrix~(i.e.,~rows~$5$-$8$~and~$14$-$18$). Among them,
the \textit{margin} condition is the most successful, followed by the \textit{thresholding} variant. The breakdown of training statistics reveals the reason behind their differences:  the \textit{thresholding} condition is a more strict constraint as it only allows synthetic candidates to replace the original pairs if they are predicted as exact equivalents, allowing for fewer revisions of divergent pairs in WikiMatrix. By contrast, the condition based on \textit{margin} is a contrastive approach that allows for more revisions of the original data (i.e., a candidate might be a more fine-grained divergent of the source). The \textit{ranking} criterion is the least successful method---this is expected as the divergence ranker is not trained as a regression model.

\paragraph{Impact of bi-directional candidates}
Considering both forward (\dtf ) and backward (\dtb) translated candidates during selective replacement yields to further improvements ($0.22$-$0.44$ points) over bitext induced by the semantic equivalence condition with candidates from a single \textsc{nmt}\xspace model~(i.e.,~rows~$7$-$9$ and $16$-$18$). When forward and backward candidates are considered independently, they replace $34-37\%$ of the original pairs; in contrast, when considered together, they replace $50\%$ of original WikiMatrix pairs. As a result, there is no perfect overlap between the original pairs replaced by the forward vs. backward model, which  motivates the use of both to revise more divergences in WikiMatrix. This finding raises the question of whether using synthetic translations from both directions might benefit other scenarios, such as knowledge distillation.
\section{Conclusion}

This paper explored how synthetic translations can be used to revise bitext, using \textsc{nmt}\xspace models trained on the exact same data we seek to revise. Our extensive empirical study surprisingly shows that, even without access to further bilingual data or supervision, this approach improves the quality of the original bitext, especially when synthetic translations are generated in both translation directions and selectively replace the original using a semantic equivalence criterion. Specifically, our intrinsic evaluation showed that synthetic translations are of sufficient quality to improve over the original references, in addition to ``normalizing'' the bitext as suggested by prior work and corpus level statistics~\citep{chuntin, xu-etal-2021-distilled}.
Extrinsic evaluations further show that the replaced synthetic translations provide more useful signals for \textsc{bli} tasks and \textsc{nmt}\xspace training in two settings (i.e., training from scratch and continued training).

These findings provide a foundation for further exploration of the use of synthetic bitext. First, we focused our empirical study on language pairs and datasets where revising bitexts is the most needed and most likely to be useful: the resources available for these languages are not so large that mined bitext can simply be ignored or filtered with simple heuristics, yet there is enough data to build \textsc{nmt}\xspace systems of reasonable quality (i.e., $\sim600$K segments for \textsc{en-ro}, and $\sim750$K for \textsc{en-el}). While in principle, selective replacement of divergent references with synthetic translations should port to high-resource settings, where \textsc{nmt}\xspace is as good or better than for the languages considered in this work, other techniques are likely needed in low-resource settings where \textsc{nmt}\xspace quality is too low to provide reliable candidate translations. Second, having established that the revised bitext improves the quality of the original bitext in isolation, it remains to be seen how to best revise bitexts in more heterogeneous scenarios with diverse sources of parallel or monolingual corpora. Overall, as synthetic data generated by \textsc{nmt}\xspace is increasingly used to improve cross-lingual transfer in multilingual NLP, our study motivates taking a closer look at the properties of synthetic samples to better understand how they might impact downstream tasks beyond raw performance metrics. All bitexts are available at: \url{https://github.com/Elbria/xling-SemDiv-Equivalize}.
\section{Example Appendix}
\label{sec:appendix}

This is an appendix.

\end{document}
\section*{Acknowledgements}

We thank Marjan Ghazvininejad, Luke Zettlemoyer, Sida Wang,
Sweta Agrawal, Jordan Boyd-Graber, Pedro Rodriguez,
the anonymous reviewers and the \textsc{clip} lab at \textsc{umd} for helpful comments.
This material is based upon work supported by the National Science Foundation under Award No.\ $1750695$. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.

\subsection{MT evaluation of fine-tuning mT5}

\begin{figure}[!ht]
    \centering
    \includegraphics[scale=0.12]{analysis_figs/mt5.drawio.pdf}
    \caption{mT5 fine-tuned}
    \label{fig:mt5}
\end{figure}

\subsection{Evaluation on Bilingual Lexicon Induction}

\paragraph{Main Results} Table~\ref{tab:BLI_main}

\begin{table}[!ht]
    \centering
    \scalebox{0.85}{
    \begin{tabular}{llllll}
    \rowcolor{gray!10}
     Pair   & Bitext     & All & [1,5) & [5,100) & >100 \\
        \multirow{2}{*}{\rotatebox{0}{\textsc{el-en}}} & Original     & $76.2$ & $59.4$ & $76.6$ & $81.4$\\
                                                        & Equivalized  & $\mathbf{77.6^*}$ &  $\mathbf{60.4^*}$ & $\mathbf{78.4^*}$ & $81.6$\\
        \addlinespace[0.2cm]

       \multirow{2}{*}{\rotatebox{0}{\textsc{en-ro}}}  & Original     & $89.2$ & $78.6$ & $86.9$ & $87.1$\\
                                                        & Equivalized  & $\mathbf{90.8^*}$ & $\mathbf{80.0^*}$ & $\mathbf{87.5^*}$ & $86.9$\\
    \end{tabular}}
    \caption{Precision results on \textsc{muse} datasets for unsupervised \textsc{bli} tasks. Equivalized bitexts yield statistically significant ($*$) improvements over the original bitexts for low-to-medium frequency dictionary entries.}
    \label{tab:BLI_main}
\end{table}

\begin{table}[!ht]
    \centering
        \scalebox{0.5}{
    \begin{tabular}{llllll}
        αστεροειδές & star	              & 	?       & απόστολος & apostolos    & \xmark \\
        προσφέρεται & offers	          &  \cmark	    & βραχνό & raucous         &  \xmark \\
        κεραυνός & keravnos	              &  \xmark     & μπανζούλ & bangaon       & \xmark \\
        συμπυκνώνει & encapsulates	      & 	?       & βοηθητικές & auxiliary   & \cmark\\
        σεξτέτο & sexteto	              & \cmark 	    & ομιλήτρια & spokesperson & \cmark \\
        επιχειρηματολογία & argumentation & \cmark 	    & πρωτεργάτη & forerunner  &  \xmark \\
        επίπλωση & furniture	          & \cmark  	& αντιτρομοκρατική & anti-terrorist & \cmark \\
        μπουγκ & bug	                  & \xmark 	    & πλεκτά            & sweaters      & \cmark \\
        σχετικοί & related	              & \cmark 	    & εμβολιαστεί & vaccinated &    \cmark \\
        δορυφόρους & moons	              & \xmark 	    & αταξινόμητες & unclassified &  \cmark \\
        δειλή & timid	                  & \cmark	    & στέιν & steen                 & \xmark \\
        χάντινγκτον & huntingdon	      & \cmark 	    & χιλιοστό & millimeter & \cmark \\
        ποσότητες & amounts	              & \cmark	    & σελεστίν & célestine & \cmark \\
        πλακέ & squamous	              & \cmark 	    & κόβατς & kovács & \xmark\\
        αποποίηση & relinquishing	      & ?       	& σεμίνα & omni & \xmark \\
        ατμούς & vapors	                  & \cmark 	    & σπάιντερμαν & spider-man & \cmark \\
        τερματισμοί & endings	          & \cmark  	& πάνω & over & \cmark \\
        αλεξανδρινό & alexandrine	      & \cmark  	& ενδιαφέρων & love & \xmark \\
        σπασμοί & fits	                  & ? 	        & αγριόγατες & cats &      \xmark \\
        σίδερα & sidelines	              & \xmark	    & αγορα & trade & \cmark \\
        συνοδεύονται & are	              & \xmark 	    & επικεφαλίδα & header & \cmark \\
        διανέμονται & are	              & \xmark  	& μάσλοου & khan & \xmark \\
        θραύση & fracturing	              & \cmark  	& τεχνητά & artificially & \cmark \\
        κυβερνά & rule	                  & \cmark  	& πέτροβιτς & petrović & \cmark \\
        συνάξεις & meetings	              & \cmark  	& ανθίζει & flowers & \cmark \\
        χριστιανία & christianity	      & \cmark  	& ζήτω & vive & \xmark \\
        απειλούνται  & are	              & \xmark 	    & τυλίγει & picks & \xmark \\
        ποινικοποίηση & penalize	      & \cmark 	    & μπαέζ & ross & \xmark \\
        στερέωμα & stardom	              & \xmark 	    & φιλοδοξεί & is & \xmark \\
        τζαπ & elford	                  & \xmark 	    & τρυφερή & loving & ? \\
        ταυρομαχία & bullfighting	      & \cmark 	    & σωρός & remains & \xmark \\
        χειρός & handbags	              & ? 	        & χαλυβουργεία & works & \xmark \\
        κδ & cd	                          & ? 	        & μάιρα & chloe & \xmark \\
        τρομοκρατεί & terrorizes	      & \cmark 	    & συγκλόνισε & shocked & \cmark \\
        μακέι & mackey	                  & \cmark 	    & άτακτη & mischievous & \cmark \\
        ζάκυνθο & zakynthos	              & \cmark 	    & οταν & after & ?  \\
        συμπτωματολογία & symptomology	  & \cmark	    & εντομοφάγα & insectivores & \cmark \\
        πολυφυλετική & polyphyletic	      & \cmark 	    & κραδασμούς & vibrations & \cmark \\
        κούνια & cunha	                  & \xmark 	    & μπελάς & nuisance & \cmark \\
        καταβεβλημένος & overcome	      & \cmark 	    & πάστες & pastries & \cmark \\
        απάτες & scams	                  & \cmark 	    & διασπαστική & divisive & \cmark \\
        γιάννη & giannis	              & \cmark 	    & κατάληψη & capture & \xmark \\
        δηλητηριάσεις & poisonings	 & \cmark	   & παραδίδονται & surrender & \cmark \\
        φιλόξενοι & colorful	     & \xmark 	   & κλήρων & clergy & \cmark \\
        φημισμένος & renowned	     & \cmark      & σκεύη & vessels & \cmark \\
        φουσκωμένα & filled	         & ? 	       & λεπτονίων & leptons & \cmark \\
        υπονοούμενα & undertones	 & \cmark	   & εξάγονται & are & \xmark \\
        όριο & boundary	             & \cmark 	   & απότομο & abrupt & \cmark \\
        χαλάρωσε & relaxed	         & \cmark 	   & παρασυμπαθητικό & sympathetic & ? \\
        αισθητικός & aesthetic	     & \cmark 	   & ταρίχευση & embalming & \cmark \\
        ταμαντούα & tamanduas	     & \cmark 	   & κεκτημένο & precedent & \xmark \\
        εστίες & foci	             & ?	       & καλκούτα & kolkata & \cmark \\
        θεωρείται & is	             & \xmark 	   & σίρι & sirri & \cmark \\
        κορμό & trunk	             & \cmark 	   & ξεπερασμένο & obsolete & \cmark \\
        σπύρο & spyros	             & \cmark 	   & ανώμαλος & bumpy & \cmark \\
        αναισθητικά & anesthetics	 & \cmark	   & εξισορρόπησης & substance & \xmark \\
        στρατηγικές & strategic	     & \cmark 	   & πολυσακχαρίτης & polysaccharides & \cmark \\
        αναπνέει & breathe	         & \cmark 	   & επίπονες & persistent & \cmark \\
        εξουδετερώσει & neutralize	 & \cmark 	   & αμφιθέατρο & amphitheatre & \cmark \\
        μελαγχολική & melancholic	 & 	\cmark     & αναπληρωματικό & an & \xmark \\
        θυμήθηκε & recalled	         & \cmark 	   & εντελώς & entirely & \cmark\\
        πασχαλίτσα & ladybird	     & \cmark 	   & λιθόστρωτο & cobbled & \cmark \\
        πυροκροτητές & caps	         & ? 	       & διοικητικοί & administrative & \cmark \\
        κραυγαλέα & screaming	     & ? 	       & κομιστής & bearer & \cmark \\
        μολδαβίας & moldavia	     & \cmark 	   & συλλογικότητες & competitions & \xmark \\
        σαλιγκάρι & shilling	     & \xmark 	   & χουλιγκανισμού & micromanagement & \xmark \\
        ενισχυθεί & enhance	         & \cmark 	   & τσάρους & tsars & \cmark \\
        πρεσβυτέριο & presbytery	 & \cmark	   & ντόνελ & dorff & \xmark \\
        μάγιστρος & master	         & \cmark 	   & κίραν & kiran & \cmark \\
        αλτ & alt	                 & \cmark 	   & πρωτοποριακή & pioneering & \cmark \\
        χρονολογία & date	         & \cmark 	   & λένοξ & brookline & \xmark \\
        κανένα & any	             & \cmark 	   & λείπουν & are & \xmark \\
        κορμός & road	             & \xmark 	   & εξάντα & astronomy & \xmark \\
        καθαριστήριο & cleanup	     & \xmark 	   & πτωτική & downward & \cmark \\
        ανατεθεί & assigned	         & \cmark 	   & αρχιτεκτονικές & architectural & \cmark \\
        εξοικονόμηση & save	         & \cmark 	   & γαλλόφωνο & french-speaking & \cmark \\
        μπαρακούντα & barracudas	 & \cmark	   & μέντε & mede & \xmark \\
        ταυτοποίησης & identification	 & \cmark	   & εκθρονίζοντας & deposing & \cmark \\

    \end{tabular}}
    \caption{Ground-truth analysis on false positives. $65/80$ andf $51/80$ false positives are correct for the equivalized and original bitexts, respectively.}
    \label{tab:ground_truth_bli}
\end{table}

redpajama_set_name: RedPajamaArXiv


Show off your logo or marketing message while protecting your desk, counter top, or conference table by creating one of our custom logo coasters.
Custom full color cinch backpacks printed with your image, design or logo. These custom string backpacks have instant appeal for a variety of events and promotions.
Order custom jar openers printed with your image, design or logo. Our fabric with rubber jar opener is popular for everything from food promotions to trade shows.
Custom full color Beverage Coolers are printed with your image, design or logo. A large variety of personalized Cooler options to meet your budget and promotional needs.
Take advantage of a captive audience by creating unique luggage tags that are custom printed with your design. Your clients will have an easy time spotting their luggege or backpacks.
Custom Full color personalized iPad covers printed with your image, design or logo. Permanently custom printed with your design for a promotion that travels with your client.
Custom full color personalized Eye Glass Cases are a great way to keep your glasses protected from scratches. Send in your logo or design to create your order of Eye Glass Cases now.
Custom full color products printed with your image, design or logo. Unique promotional products imprinted in full color with your logo, image, or design. High quality imprint and full color range allow for photographic and complex design printing. A variety of custom printed products to fit your budget and promotional needs with products such as: a custom can koozie, cinch sack, jar opener, personalized iPad cover, luggage spotters and you can even make your own custom drink coasters (more about us).


text:
Fantasy 12, a tribute to the talents behind iconic record covers
The exhibition 'Fantasy 12' is a fantastical trip into iconic album artwork. It opens this week in Dublin's Copper House Gallery.
Luna Cohen-Solal October 12, 2016 Fantasy 12, Dublin, Record covers, The Designers Republic, Warp Records, Run The Jewels, Shamir, exhibitionComment
Solange Album Sneak Peak
Solange soon to release new album
Avenir Magazine September 28, 2016 Comment
Way Out West – Scandi mini-break for the musically minded
For its tenth-anniversary edition, Way Out West festival, which happened last weekend in Gothenburg, Sweden, welcomed legendary artists Massive Attack, PJ Harvey, Morrissey and Grace Jones as well as tons of rising acts.
Luna Cohen-Solal August 19, 2016 toxe, peaches, Grace jones, Way out West, jamie xx, the libertines, Joy, Anderson .Paak, Kaytranada, Chvrches, arca, Massive Attack, Morrissey, Stormzy, Festival, Sweden, GothenburgComment
Best of Way Out West Playlist
Tomorrow, Scandinavia's best summer festival will kick off in Gothenburg's luscious Slottsskogen park. For your listening pleasure, we have studied the Way Out West line-up carefully and hand-picked a playlist of the best tracks by artists performing over the weekend.
Luna Cohen-Solal August 10, 2016 Way out West, Peaches, Toxe, Spotify, Grace jones, Morrissey, festival, Gothenburg, playlistComment
Avenir's Guide to Way Out West Festival
As the long list of must-sees at Way Out West can potentially leave you curled up in a ball from the sheer terror triggered by indecisiveness, Avenir is here to help you with our top 5 acts to see in Gothenburg between the 11th and the 13th of August.
Avenir Magazine August 4, 2016 Way Out West, Festival, Juliana Barwick, Morrissey, Grace Jones, PJ Harvey, Sigur Rós, Nils Frahm, Ólafur Arnalds, Tim Hecker, Efterklang, Liima, José Gonzalez, Bob Moses, Radiohead, Kiasmos, Colder, 4ADComment
Goss - I Want To Know?????
"I Want To Know?????", the first track from Danish newcomer Goss, has gotten us hooked.
Luna Cohen-Solal July 21, 2016 goss, Danish, MØComment
Dakota Blue - The State Of Things
We stumbled upon this soothing gem of a song, and we just had to share it.
Luna Cohen-Solal July 12, 2016 Dakota Blue, New musicComment
"I survived the trenches of Worthy Farm"
A week after the end of Glastonbury, journalist Laura Steiner tells us the story of her first experience of the "trenches of Worthy Farm". Whether you've been to the festival or not, this is the only account of Glasto you'll ever need.
Avenir Magazine July 6, 2016 Glastonbury, glasto, kate tempest, coldplay, tame impala, beck, madness, sigur rós, shura, baaba mal, Laura Steiner, Andrew AllcockComment
AVENIR IN CONVERSATION WITH LCMDF
Helsinki/Berlin-based band LCMDF, formed of sisters Emma and Mia Kemmpainen, were one of our highlights from The Great Escape. We caught up with them about their beginnings, Beyonce, feminism and their conflicting views on Kanye West.
Luna Cohen-Solal June 30, 2016 LCMDF, The Great Escape, Finland, Berlin, Kanye West, Beyonce, Pitchfork, Heavenly, Icona Pop, feminism, sistersComment
"The new and the wonderful" Sónar 2016 Review
We flew to Barcelona over the weekend to check out Sónar, a festival that has been at the forefront of dance and electronic music since 1994.
Luna Cohen-Solal June 22, 2016 Music festival, Sónar, Barcelona, Lafawndah, Anohni, Oneothrix Point Never, Kaytranada, Danny L Harle, James Blake, Hudson Mohawke, Santigold, Tha Black Madonna, Sevdaliza, Kelela, Lady Leshurr, Gazelle Twin, Jean-Michel Jarre, New Order, Fatboy SlimComment
MABEL || TATE MODERN
Rising songwriter and artist Mabel has been commissioned by the Tate Modern to create a video project to commemorate the opening of the new Tate Modern building.
Avenir Magazine June 17, 2016 Comment
Sónar Festival at the forefront of digital culture
Happening this weekend in Barcelona, Sónar Festival will celebrate creativity, technology and experimentation with a rich programme of electronic music, installations, workshops, talks and more.
Luna Cohen-Solal June 14, 2016 Sónar, Barcelona, Festival, Music, technology, Digital cultureComment
Avenir's Field Day Pick: Molly Nilsson
Today's Field Day favourite is Sweden-born, Berlin-based Molly Nilsson.
Luna Cohen-Solal June 9, 2016 Molly Nilsson, Field Day, Dark Skies AssociationComment
Avenir's Field Day Pick: Formation
One of the acts we're most looking forward to seeing at Field Day is South London band formation, fronted by brothers Will and Matt Ritson.
Luna Cohen-Solal June 8, 2016 Formation, Will Ritson, Matt Ritson, Jackson Ducasse, Field DayComment
Avenir's Field Day Pick: Avalon Emerson
Avenir's first pick from the Field Day 2016 line-up is rising DJ-producer Avalon Emerson.
Luna Cohen-Solal June 7, 2016 Avalon Emerson, Field Day 2016Comment
It's almost Field Day ! We have gathered our favourites from the line-up in a Spotify playlist, enjoy!
Luna Cohen-Solal June 6, 2016 Field Day, Festival, Music festival, James Blake, PJ Harvey, SpotifyComment
Avenir in conversation with Ry X
A day after the completion of his tour, Avenir contributor Kasimiira Kontio sat down to talk to Ry about the journey behind Dawn, the creative community around him and being true to oneself.
Luna Cohen-Solal June 4, 2016 ry x, music, dawn, loma vistaComment
The Great Escape 2016 Photo Gallery
We had such a good time at The Great Escape this year! Here's a glimpse of what it looked like.
Luna Cohen-Solal June 2, 2016 Comment
Weekend Vibes
Sade's Babyfather is that cool start you need for the weekend..
Avenir Magazine May 27, 2016 Comment
It's that time of year again where you need to brush the cobwebs off your tent or research cheap places to stay in the centre of cities while you work out which festivals to attend this year. After The Great Escape opened the festival season with a bang this May, we decided to uncover some of the best festivals for summer 2016...
Avenir Magazine May 26, 2016 Music Festivals, music, Summer festivalComment

redpajama_set_name: RedPajamaCommonCrawl


Online roulette is so popular because it has basic rules but offers a great giochi slot gallina gratis 2014 variety of betting options.
Open an Account and Play for Real Money To get started at the real money online Roulette tables, click on Play for Real Money and hit Create Account.Of course, you don't need to bet on exact numbers. .Your 1400 Welcome Package is already waiting for you, don't miss your chance to hit the special jackpot!As we discussed, European versions of the game are always preferred and are widely available online in the USA at top online gambling sites in 2016. .When learning how to play roulette, the only distinction is that the American version adds a 00 slot to the wheel and table layout, which effectively reduces your odds and makes the European game a better bet. .
Although payouts are fairly standard, internet gambling sites always offer European roulette and occasionally host French games too. .
Find out for yourself why Roulette fans from around the world play their favorite casino game at our online Roulette tables.
With.65 chance of winning, it's one of your best bets.
Dozen or column bet - These bets pay 2:1 and give you.43 shot at winning if you play European roulette.
Online Roulette Your Favorite Game is Just A Click Away.If you're wondering what all the numbers on the table mean while trying to figure out how to play roulette, internet gambling fans should direct their attention to the wheel. .Whether you play the American, British or French version, you will have fun choosing your lucky numbers, colours and combinations!When it comes to casino online gambling, how to play roulette best is a topic worth discussing. .Learning the Basics of Roulette, whether you opt to play live or end up embracing online gambling roulette, learning how to play the wheel is simple. .Don't forget that you will always have the opportunity to play free roulette and gain more knowledge until you are ready for some real action!Choose Your Bonus Wisely: Casino bonuses are designed to boost your bankroll so you have extra funds for playing your favourite real money games.In addition to cameras and other technological measures, casinos also enforce security through russians roulette online game of conduct and behavior; for example, players at card games are required to keep the cards they are play online roulette in their hands visible at all times.Your message has been sent.Roulette Online Spinning the Lucky Wheel.


We are third generation engravers offering a full, professional engraving service including specialist glass engraving, as well as domestic key cutting, vehicle key cutting and programming and watch battery replacement.
We're located in High Street in the heart of West End with free parking available in the village. Pop in to see us, give us a call, or send your enquiry by email. We're here to help.
Need a new watch battery?
We can supply and replace most watch batteries, while you wait, with prices starting at £5. UK standard batteries with a three year guarantee.
Corporate Glass Awards with Specialist Etching / Glasses and Glassware / Cups and Trophies / Bespoke Awards / Rosettes, Medals and Badges / Ribbons, Pennants, Flags and Sashes / Wooden Plaques and Shields / Silverware and Pewter / Brass Plates and Laminate Signs / Commercial and Industrial Engraving / Chemical Etching / Jewellery Engraving (Wedding Rings, Bracelets, Cuff Links) / Memorials and Tributes / Gifts – Weddings, Christenings, Birthdays and more / Promotional Products including Pens and Mugs / Artwork and Design Service / Trophy Cleaning Service.
Smaller jobs can be engraved while you wait.


text:
S: Economist - https://econ.st/2MBmsgE (last access: 10 February 2017); The Guardian - https://bit.ly/2dgKvz5 (last access: 10 February 2017).
N: 1. Past participle, adjective and noun.
From the verb exile (c. 1300, from Old French essillier "exile, banish, expel, drive off" (12c.), from Late Latin exilare/exsilare, from Latin exilium/exsilium "banishment, exile; place of exile," from exul "banished person," from ex "away"; In ancient times folk etymology derived the second element from Latin solum "soil." Related: Exiled; exiling).
2. Leaving aside legal definitions mostly used by governments, issues of definition include individual motivation, self-concept, and some objective conditions of life. One's motives for leaving one's country are key to subsequently thinking of oneself as either an exile or an immigrant, and must be so understood. cursory review of the factors that define either exiles or immigrants suggests that it is difficult to disentangle the highly subjective elements that shape the decision to leave one's country. Departing for strange country is one element that exiles and immigrants have in common, but
qualitative differences define each role.
3. exile means also a person who is in exile.
4. Cultural Interrelation: We can mention, among others, the movie Fong juk (Exiled) directed by Johnnie To in 2006.
S: 1. OED - https://bit.ly/2RTPxtU (last access: 10 February 2017). 2. ScholRepos - https://bit.ly/2B1Eu7d (last access: 10 February 2017). 3. MW - https://bit.ly/2WmFkof (last access: 10 February 2017). 4. Filmaffinity - https://bit.ly/2CLADeO (last access: 10 February 2017).
SYN: exile (context)
S: MW - https://bit.ly/2WmFkof (lat access: 10 February 2017); ScholRepos - https://bit.ly/2B1Eu7d (last access: 10 February 2017).
CR: asylee, banished person, banishment, displaced person, emigrant, exile, exodus, expatriate, expatriation, immigration (EN), internally displaced people, refugee, stateless person.


Centroc member councils have just struck a deal for electricity with savings in excess of $350,000 in the first year. The new contract is for just under 1,000 small tariff sites across 12 participating member councils including Blayney, Cabonne, Central Tablelands Water, Cowra, Forbes, Lachlan, Lithgow, Oberon, Orange, Parkes, Upper Lachlan and Weddin.
The Orana Regional Organisation of Councils (OROC) also participated in the process, adding to the large buying power of the group.
"This is such a fantastic example of the efficiencies councils can achieve by working together. Our Council alone has saved $19,052 on electricity for small tariff sites," said Mayor Kathy Sajowitz.
"The procurement of electricity is such a complex area, so we engaged an external energy consultant, Energy and Management Services (EMS) to conduct the process. The effort by Council staff has been minimal, with the Consultant managing the project from start to finish," said Mayor Kathy Sajowitz.
"The savings of over $350,000 across the region equate to a percentage saving of around 13.46% for Oberon from the prices we are paying on the current contract," Clr Sajowitz.
"Centroc has been lobbying the State Government to change the current procurement legislation to enable ROCs to do more great work like this. The current legislation makes it quite hard for councils to get the best price for commodities like electricity, however we sought legal advice to ensure we were running a sound process," said Mayor Sajowitz.
"This really does demonstrate the great work being undertaken by ROCs, especially when councils all band together to achieve such great cost savings. Working with other ROCs, like OROC, is always something that we aim to do wherever possible," said Mayor Kathy Sajowitz.


Bureaucracy is a poor parent but thousands of Canadian children are raised by no other. They're lost in a forest of faulty agencies and government neglect, like modern versions of Hansel and Gretel, desperately needing a home and finding none. Also lost are thousands of couples eager to adopt. Many search the far side of the world, spending up to $35,000, while kids wanting parents wait in their own home town.
Torn from their original families by neglect, drug- or alcohol-fuelled violence, mental illness or sexual abuse, these youngsters badly need stability provided by new parents. Adoption would be "like a gift from heaven," says one social worker. But most never receive that gift. They remain almost invisible to policy makers and the public.
Eight-year-old Daniel has wanted a home since 1997, when his mother gave him up but kept his infant sister. Wounded by that abandonment, Daniel aches for a new family. "Are you here to adopt me?" he beseeches visitors. But Daniel is considered old for adoption. His mother has a history of mental problems, and he bears deep emotional scars. Daniel remains nobody's child. And he's not alone.
"People assume there are no kids needing adoption – in fact, there are thousands," says Judy Grove, executive director at the Adoption Council of Canada. "Nobody pays much attention to them.
It's estimated that more than 20,000 children across Canada are permanent wards of the state, meaning a bureaucracy is their legal guardian. But each year only about 1,200 become adopted, giving them homes. Meanwhile, Canadians adopt about 2,000 children yearly from outside the country. They go overseas for many reasons. Some crave babies, and infants are rare among wards of the state. Others assume Third World kids will have fewer problems than crown wards, since most children adopted through a CAS have ``special needs''.
But simply being 7 or 8 years old can be enough to warrant a special needs label. So can being of mixed race, or having a brother or sister also up for adoption, since a sibling group is harder to place than a single child. Instead of making that clear and aggressively recruiting families for Canadian kids, most governments and child protection agencies stand silently aside as people yearning to be parents leave the country.
``It's the failure of our child welfare system, more than any other, that drives them away,'' says Grove.
Canada's provinces and territories all run adoption differently. Some jurisdictions use a network of local children's aid societies while others rely on a central government department. Sometimes there's a confusing mix: Adoption in Halifax is the responsibility of a CAS but, in Truro, 90 kilometres away, it's done by the province.
No jurisdiction is more fragmented than Ontario, where adoptions are splintered between 52 children's aid societies and a host of private agencies. The result is a helter skelter system with huge differences in service, depending on where you live. Some cash-strapped CASs have literally closed their doors to prospective parents. Others warn of waits lasting three years, or more, before a social worker will check a family to see if it's even suitable to adopt.
No wonder prospective parents are discouraged, Grove says, adding that agency chaos is just one barrier to adoption. Its opposite – rigid regulation – poses another.
About 6,100 Ontario children are crown wards, meaning they've been permanently taken from their birth parents. But most of these families still have court-ordered access to their kids. And that access – by law – blocks any chance of a child's adoption.
In Ontario, adoption and access simply can't go together, even when families don't use their right to visit a youngster. According to provincial statistics, parents don't bother seeing their child in about a third of cases.
Rather than allowing adoption, plus contact with birth parents, Ontario's rigid law keeps thousands of crown wards off the waiting list for a new family. On arriving into care they're shunted into foster homes where most stay until they ``age out'' of the system on their 18th birthday. It can make for a rootless childhood, spent drifting through various foster placements, supervised by shifting social workers. And many continue drifting long after leaving the child protection system.
Covenant House staff in Toronto, who run a homeless shelter and drop-in centre for street kids, estimate that about half the 5,000 young people going through their doors every year have been in foster care.
It's part of a nationwide trend as agencies put new stress on child protection. There's less emphasis on keeping families together and more on whisking kids out of troubled homes. Costs have risen even faster, with Ontario CASs spending $772 million last year – a 75 per cent increase since 1997. But barely $7 million of that is earmarked for adoption. While the province spends freely on bringing kids into state care, it ignores the other side of the equation: getting them back out into adoptive homes, says Mark Ewer, executive director of the Catholic Children's Aid Society of Hamilton-Wentworth.
As co-ordinator, she tried lifting the barrier posed by access orders – and failed. ``I fought really hard to get that removed,'' she says, shaking her head. ``It's sad.'' Scarth can't suppress a peal of laughter when asked if she'd like to return to supervising Ontario's system.
With more children in care than ever before, adoption has never been so needed, but it doesn't happen as often as in earlier decades.
Thousands of Ontario kids were adopted through children's aid societies, each year in the 1960s and '70s. That dwindled to about 500 last year. And the kids, too, are different. Virtually all have been removed from troubled homes and have suffered family breakdown and other trauma. In contrast, decades ago, most adopted children were newborns given up by unwed mothers. That changed with widespread birth control, legalized abortion, and society's growing acceptance of young single moms. Now infant adoption is relatively unusual. And it's rarely done by a CAS. Private agencies entered the field in 1980, and the few infant adoptions that do take place are generally handled by these services, for a fee. Last year, they accounted for another 187 adoptions, almost all newborns.
Lots of children still need adoption, but they're older, and virtually all are labelled ``special needs''. To many adoptive parents going overseas, that designation sounds suspiciously like ``damaged goods,'' says Michael Grand, a University of Guelph psychologist studying adoption issues.
They don't understand that some reasons for a special needs designation are ``quite minor,'' he says, adding that each child is unique.
At 17 months, blue-eyed Brian is too young to worry about his future. But anyone planning to adopt him must consider that his mother was a teenage drug addict who fed her habit while pregnant. Raised in a Milton foster home since he was 3 weeks old, Brian has shown no sign of damage from drug exposure – so far. While playing with oversized building blocks, in the living room of his temporary home, he playfully clambers into a green storage tub. Foster parent Debbie Krantz chuckles, describing him as ``perfect'' and ``gorgeous.'' For parents adopting Brian ``it will be like winning the lottery,'' she says.
But until willing parents come forward, Brian remains nobody's child.
Many kids in CAS care do present a major challenge. Some have fetal alcohol syndrome, or a physical disability, or difficulty forming an attachment. Almost all crown wards have suffered the stress of seeing their family shatter, and many come from backgrounds involving drug or alcohol abuse by a parent. They can be difficult to raise. But so can children adopted from abroad.
A Mississauga pediatrician with a large practice seeing children from abroad estimates that 50 per cent, or more, arrive with an unsuspected medical diagnosis. ``We've seen kids who are truly depressed – grieving. They've lost their country and their culture,'' says Dr. Angelo Simone. Yet, there's a perception that overseas adoptees are less likely to have special needs than someone stuck in foster care at a local CAS, says Grand, at the University of Guelph.
The challenge is immense, and it's nation-wide. Every province and territory needs to do better on adoption, Grand says. But the system's troubles run deeper than policy gaps, a chaotic structure, poor planning and lack of money.
Attitudes on adoption need changing.
"We have belief systems that are completely out of date,'' says Robert Fulton, a Toronto social work consultant who advises children's aid societies and the province. ``Picture a kid in foster care,'' Fulton says, tracing a child's outline in the air. ``He has an access order with his mom and she uses it, maybe every six months, to see the kid. He's eight years old and has been in and out of five foster homes. The mother's a decent sort, she tries her best, but she's all screwed up. She's attached to the kid, he really likes her, but he couldn't live with her.
Traditionally, it was thought that only very young children could form a bond, or "`attach,'' to a new parent. In this view, older kids are considered virtually unadoptable, so their best option is to remain in foster care.
Good foster care can provide a firm base, he says, but a foster child remains much more at risk of being set adrift than the same kid adopted by the same family. Adoption is a lifelong commitment while foster parenting is like a job. People retire, or leave for other employment opportunities. Marital problems and divorce can end fostering. And people quit for health reasons. Then their foster kids are scattered.
More than a third of Ontario's crown wards have been through three or more foster homes, according to a 1999 review examining a large cross-section of these children. And 43 per cent have had three or more case workers. Often they were put in foster care and then sent back home, to their birth families, only to wind up in a different foster home a few months later.
Daniel's case is more difficult than most. His bitterness runs deep after being abandoned in favour of a younger sibling. And there's a risk he'll lash out at smaller kids. That's why he's at Woodview Children's Centre, in Burlington, rather than in foster care. But finding the right adoptive parent could be his salvation, says Doug Jackson, a youth worker at the centre where children learn anger management. Other youngsters here are older than Daniel and he hasn't had any incidents. "He's a delightful child. Charming. Very affectionate. He wants to please,'' Jackson says.
Ontario took a step toward quickening the adoption process in April, last year, when it imposed a detailed timetable on child protection cases. Meant to speed children's cases through the courts, it set a one-year limit for a child under 6 to be in temporary CAS care. It also set an even tougher deadline: once a kid is taken from home by an agency, the system has a maximum of 120 days to complete a child protection hearing. The deadlines are supposed to make kids, like Daniel, quickly available for adoption by cutting through legal tangles and forcing a rapid decision. But, even in this, the province has stumbled. Nothing happens when deadlines aren't met, and children remain in legal limbo.
Despite that, judges do take into account the rights of a birth parent, says Ontario Court Justice Lloyd Budgell. "I think they have to,'' says Budgell, designated spokesperson for Chief Justice Brian Lennox, of the Ontario Court of Justice. "You don't want to run roughshod over anyone. If you just took the case and followed it through – bang, bang, bang – I suppose you could force people to do certain things. I'm not sure it would be better.'' The deadline set out in the legislation "is just not realistic,'' he says.
The system doesn't have enough judges, and the government doesn't want to pay for more, he says. But that's just part of a bigger problem. "It's also a question of children's aid staff and other resources,'' Nevins says.


Chic Chapeau is the label of milliner and hat designer, Bobbi Heath. Bobbi uses her millinery skills to create headwear dreams. It could be a show-stopping "icing on the cake" creation for a day at Ascot, an elegant hat for the mother of the bride or groom, a fascinator to grace a prom night or a simple felt or straw hat, to wear everyday on the high street.
Sinamay with embroidered lace on a wired headband.
Sinamay covered with antique beaded lace with leather arrow on the elastic.
Black Sinamay with Satin covered button. Large White Flower on a headband.


Does Your College or Seminary Work for You?
Christian colleges and universities are hurting. A recent piece on CNNMoney reports these institutions are slashing tuition in an attempt to stay afloat.
The article cited declining church membership, especially among the young, as a nagging cause for the problems plaguing Christian schools.
But I wonder if the picture is a bit more complex. I wonder if the academic product itself is losing some luster.
Allow me to pose some questions. I'd love to hear your answers.
Do Bible colleges and seminaries have a clearly defined goal? And is that goal in concert with the students' goal for a higher education? As a former student, and as a parent and an employer, I've looked to academic institutions to equip students to succeed in the work world. That's the top priority, the first reason students and their parents would part with their tuition dollars. Do schools embrace this top priority?
Do Bible colleges and seminaries accept the responsibility to prepare students with practical skills, in addition to theory and theology? Some years ago we polled people in full-time ministry. We asked if they felt their colleges and seminaries adequately prepared them for real ministry. Only five percent said yes.
How are faculty members chosen? What is the top hiring priority? Is it the true demonstrated ability to effectively teach, inspire and equip students? Or, is it other stuff, such as academic pedigree, authored books, or name recognition?
Do schools pursue a customer focus? Are they passionately dedicated to customer (student) satisfaction? If so, you'd expect to see evidence such as regular follow-up with students and past students. Have you received follow-up questionnaires inquiring about how well your education prepared you for real-life ministry? Has your college or seminary asked how they might improve their service?
And a bonus question: What's with this thing called tenure? Is this really working to maintain a staff of consistently top performers who serve their students?
I have a number of friends who serve in Christian academia. They're good hard-working people. I just wonder if the system is serving them and their students as well as it might.
oh my~ this is timely for us!~ thankyou~ whew!
Since I do not work nor am I a student, I've re-worded your article to fit me. I'm continuing to ask (since reading your article-and before but probably not as intently)myself, what are my conversations doing to help the people I talk with generating? Are they getting anything out of them or am I just blowing wind? Do they leave the conversations we've shared thinking (as I do with your articles 🙂 ) or do they forget as soon as the conversation is over?
Thank you for your thoughts. Some Christians believe the network is an evil place and is no place for Christian articles; how I thank you that you don't.
I had a great experience at seminary and wouldn't trade it for anything. I even sit on an advisory board for the seminary–that's just how much I believe in what they do and how grateful I am to them for my education and formation. However, that is not to say that there isn't room for improvement and thankfully, they are looking to embrace some new things like online offerings and making coursework available to those who may not have undergraduate degrees.
2. As for practical skills, I'm part of a Facebook group called "Things They Didn't Teach us in Seminary". It's a fun group with people listing lighthearted things as well as some serious things that people deal with in ministry that no one taught them about. We're also able to provide each other with advice and support. To some degree, there are things that really, no one can prepare us for. Who could even begin to imagine some of the many and sundry things that one will deal with in ministry? The issues are as many as there are people on the face of the earth. Also, I wonder if seminaries don't teach some practical things because, particularly if they're not a strict denominational seminary, it would be too difficult to teach me about the intricacies of ministering in a Baptist, Methodist, Pentecostal, Quaker, etc. environment. Every denomination and/or church has its own polity and again, I think it would too difficult for seminaries to try to delve into all the complexities. Maybe seminaries assume (wrongly or rightly) that is training that our churches will provide.
4. Customer focus is a good point and I have not had any follow-up from my seminary. I wonder if seminaries are just short-staffed to even offer that kind of follow-up or maybe it's never occurred to them? It's a worthy point to pursue rather than relying on anecdotal stories from the occasional alum that might end up working with the seminary or staying in touch in one way or another. More than likely, most alums are never contacted beyond the requisite fundraising letter.
Very good thoughts. I attended seminary and many times I have felt that sem. could have done a better job for me. I think it would be very helpful if actual ministry practicum would be part of the course load. This would extend the time in school but I think it would be worth it. I really like the idea of schools following up on their students after graduation.
Not too long ago we had a Admissions Director of a Christian University visit our Church. He shared with the congregation the school's role in society, the Church and what it had to offer young people seeking a career serving the Lord. Outside there was a table with information and it all seemed like it was a dynamic opportunity, especially the opportunity for scholarship. After reading this, and based on experience, I wonder what the how it all is in reality, the day to day for the students and staff. I pray there's a turn around and just like churches and a nation need a revival, so do Christian Universities!


text:
Q: How to add running text between bibliography items? I'd like to create a bibliography (with the \thebibliography environment) where I need to add a few textual comments between some of the elements (for example, `See the following books for more information on foo:' and 'These resources help you better understand the bar:' etc.) I've seen solutions for adding a preamble to a bibliography ( Adding text between Bibliography header and the first bibitem ), however, I don't know how to use that solution for my needs. Is there any way to solve this problem, or would it be easier to manually create the whole bibliography?

A: Stealing and adapting part of an answer from this question,
here's what I came up with.  It gives you the bibcomment environment to introduce annotations that will take the form of paragraphs between your bibliography items.
\documentclass{article}

\newenvironment{bibcomment}{%
  \item[]%
  \begingroup%
  \par%
  \parshape0%
}{%
  \par%
  \endgroup%
}

\begin{document}

\begin{thebibliography}{1}

\begin{bibcomment}
  Here are two papers on denotational semantics; the first of these
  is the seminar paper on this topic.
\end{bibcomment}

\bibitem{scott-1971-tmscl}
D.~Scott and C.~Strachey.
\newblock Towards a mathematical semantics for computer languages.
\newblock In {\em Proceedings of the Symposium on Computers and Automata},
  pages 19--46, Brooklyn, NY, 1971. Polytechnic Press.

\bibitem{tennent-1976-dspl}
R.~D. Tennent.
\newblock The denotational semantics of programming languages.
\newblock {\em Communications of the ACM}, 19(8):437--453, August 1976.

\begin{bibcomment}
  They are followed by three excellent books on the subject.
\end{bibcomment}

\bibitem{tennent-1991-spl}
R.~D. Tennent.
\newblock {\em Semantics of Programming Languages}.
\newblock Englewood Cliffs, NJ, 1991.

\bibitem{gunter-1992-splst}
C.~A. Gunter.
\newblock {\em Semantics of Programming Languages: Structures and Techniques}.
\newblock Foundations of Computing Series. 1992.

\bibitem{mitchell-1996-fpl}
J.~C. Mitchell.
\newblock {\em Foundations for Programming Languages}.
\newblock 1996.

\end{thebibliography}
\end{document}

And this is what you get:

redpajama_set_name: RedPajamaStackExchange


Backflow preventers are installed to protect our drinking water from contamination due to improperly connected fixtures, irrigation systems, fire suppression systems, and more. Many of the checks and balances put into place include annually testing these backflow preventers.
Please be advised the Cross Connection Program addresses actual or potential backflow threats, including administering a program ensuring backflow preventers are tested annually. Bylaw 2010-64 states that failure to test a backflow preventer and provide test results can result in fines.
For more information, contact Environmental Coordinator on Thurs or Fri at 250-490-2562.
Check www.penticton.ca for a list of Certified Backflow Preventer Testers and more information on the importance of safe drinking water.