text:
info@genesiseducation.net
"For the body is not one member, but many."
Liz F. Bradley
Master's Degree, English Education
Master's Degree, Dramatic Arts
B.S. in Microbiology, Humanities
A two-time Harvard alum (M. Ed. and MFA), and a graduate of the University of Rochester, Liz is an educator with over a decade of experience both as a teacher and college admissions specialist. While in Boston, Liz served as a teaching fellow at the Harvard Graduate School of Education. After her first year of teaching English and writing classes at a college prep high school, she was awarded permanent teacher status, surpassing the five-year qualifying mark. Liz loves writing, acting, illustrated journaling, yoga, and spicy Thai food.
Lizzie Torres
Bachelor of Arts, English
After graduating from the University of California, Berkeley, Lizzie Torres received her Ed.M in Teaching and Curriculum from the Harvard Graduate School of Education. From Boston, Lizzie moved to New York City where she spent a decade teaching AP Literature and Language Arts in New York City. As a teacher, her classroom served as a lab site for city-wide inter-visitations focusing on reading and writing conferences. Throughout her tenure in New York, Lizzie served as Department Chair, Instructional Coach, College Counselor, Testing Coordinator and Coordinator of Student Affairs. More recently, Lizzie worked as the Senior Manager of Teaching and Learning for the LEARN Network in Chicago. In this role, she oversaw the professional development of teachers and school leaders across several K-8 schools. Lizzie worked with a broad range of training models from coaching individual teachers to presenting for large-scale conferences. Lizzie's training style is collaborative, supportive and interactive. Lizzie loves green tea, and because she cannot choose her favorite, her Kindle has at least 20 great books.
Eli Kramer
Carleton College, Sociology / Anthropology
Humboldt State University, Theatre / Political Science
New York University School of Continuing Education, Film Production
Eli is an instructor and executive functioning coach based in San Francisco whose relentless curiosity has guided him through a diverse range of study and work experience, including: writing, history, social science, acting, media production and legal investigation. The throughline has always been reading and writing to fit the context and audience at hand, which became the natural foundation of his teaching practice. With the guidance of his Harvard-educated mentor, Eli keeps sharpening his teaching skills. He approaches students with humor, enthusiasm and kindness. Eli recently completed a contract term at Convent and Stuart Hall High Schools in San Francisco, focusing on organization and mindset. His favorite things are reading, walking, meditating and making music.
Oxford University – M.St in English
University of Michigan – MFA in Fiction
Exeter University – BA in English
Kristin has spent her life ping-ponging back and forth over the Atlantic Ocean. Along the way, she's accumulated a pair of Master's degrees, seven years of teaching experience, and an abiding fondness for Keats. Her work as a high school teacher gave her particular zeal for teaching poetry and writing – subjects that can feel intimidating at first, but that will reward you more richly than you can imagine. Her hobbies include writing stories, reading novels, hanging around in art museums, and animatedly explaining why Dante and Virgil always turn left.
Rebecca Freedman
A.B, History
DePaul University. Chicago IL
Modern German Philosophy (Heidegger), Linguistics
Rebecca grew up in Paris, France, benefiting greatly from a bilingual upbringing and the strict French secondary school curriculum. While others have their sights set on the city of lights, she looked forward to the flexibility and intellectual openness of American academia. While a student at Princeton University, she was able to indulge in her academic curiosity to the fullest, culminating in a degree in the History of Science.
Rebecca has been a writer and teacher since moving to New York after graduation, and takes great joy in sharing her enthusiasm for the Humanities and Social Sciences with her students. She takes pride in tailoring her teaching style to fit the individual needs of her pupils, and in helping them realize their full intellectual potential through patient pedagogy, believing that learning is best achieved through support, collaboration, and constructive challenging.
When not found sorting through dusty archives, Rebecca is salsa dancing, reading French poetry, and inventing new vegetarian recipes.
James A. Dievler
Ph.D. in American Studies
B.A. in Philosophy
M.F.A. in Radio, Television, and Film
James has lectured in the university/college classroom for over twenty-five years. He has taught American History and Literature, Writing, World History, Western Literature, and Contemporary Literature at The Fashion Institute of Technology, Long Island University, Rutgers University, and Monmouth University. Today, he teaches World History and Philosophy in a virtual classroom and is ghostwriting a book on the sports industry. His passions are his creative writing, including short stories, long fiction, and screenplays, and New York/New Jersey history and culture. He resides in Jersey City where he celebrates the local history through his many community involvements. He is a member of the Bergen Square Historical Society and is the lead organizer for Bergen Square Day, an annual Jersey City festival of cultural heritage. He loves to travel and experience different cultures.
Jennifer Gersten
B.A., English
M.M.
Jennifer Gersten is an educator, writer, and violinist from New York. A Yale graduate (BA, English and creative writing), she has many years of experience teaching literature as well as ACT, SAT, ISEE/SSAT, and SHSAT and college essay and academic writing. She delights in helping students defuse academic challenges through conscientious, empathetic collaboration. For Jennifer, the excitement of teaching lies in helping her students realize new curiosities. Jennifer is the winner of 2018 Rubin Institute Prize in Music Criticism awarded by the leading writers in the field. Her reporting, profiles, essays, and classical music reviews have appeared in The Washington Post, NPR Music, Guernica, Harvard Magazine, and The Kenyon Review (online), among other publications. In February 2019, she was invited to appear on the PBS NewsHour's "IMHO" segment hosted by Judy Woodruff as an on-air essayist and violinist.
Maggie Rymsza
Major: History, Concentration in American Studies
University College of London, London, U.K
Maggie is a writer, researcher and educator based in Pennsylvania. She obtained her undergraduate degree in history, specializing in American studies. Since graduating from the University of Pennsylvania, she has spent the past several years traveling in America and Europe, continuing her work in academia through pursuing historical research, working as a mentor, and writing about her adventures along the way. She has worked for the Kislak Center for Special Collections and the American Philosophical Society and published her first book, entitled Images of America: Muncy, chronicling the history of her hometown. Finding happiness in learning, Maggie strives to bring her passion for writing and history to students to help them discover new ways of seeing the world. In her free time, Maggie loves cooking any meal with pasta, reading biographies, and amassing new vinyls for her collection.
Austin Gorman, PhD
Doctoral Degree in American Literature and Languages
Most recently, Austin was the Director of Writing Programs at Clemson University, where he specialized in teaching students the fundamentals of English grammar, rhetoric, and style. In this position, he worked in close collaboration with Clemson Honors College, where he advised dozens of applicants for the Rhodes Scholarship, the Truman Fellowship, and Fulbright Fellowship, respectively. For this reason, Austin is interested in finding writing contests, scholarships, and other awards that your elementary, middle school, and high school student can enter. A professional mentor in the areas of SAT/ACT test preparation, the academic essay, and a variety of personal/competition essays—from the common application essay for rising high school students to the John F. Kennedy Profile in Courage Essay competition and Ayn Rand Essay contest for grades 8-12—Austin expertly employs forms and templates (the instruction manual for American English) in his training of writers as young as seven. One of his points of pride comes from having served as the referee for the first successful Rhodes Scholar finalist at Clemson in 38 years. Austin has previously published two, peer-reviewed academic articles: "Stealth WAC," which is a case study derived from his experience working with international students on writing, published in The WAC (Writing Across the Curriculum) Journal in 2018, and "M*A*S*H, The Longest Yard, and the Integrationist Imagination in the Post-Segregation Era" in the Journal of American Studies. He is currently working on a novel, Onyx: American Celebrity, a roman clef about NBA superstar Dennis Rodman.
Lena Tsykynovska
University of Massachusetts, Amherst
MFA in writing
B.A. in Literature
Lena Tsykynovska is a teacher and poet from Odessa, Ukraine and Newton, Mass. She teaches literature and creative writing courses at the University of Massachusetts, Amherst, and is currently finishing up her Masters in Fine Arts. After earning her undergraduate degree in literature, she received a Fulbright grant to study Russian poetry at the University for the Humanities in Moscow, where she also taught literature courses at a school for adult learners. In addition to her work at the university, Lena is an instructor for the Juniper Young Writers Program, a week-long writing workshop for high school students. Her experience growing up between two languages and two cultures is what brought her to teaching literature, and is the reason she loves helping students find themselves in books.
Molly Horan
BA in Creative Writing and Globalization
After completing her undergraduate degree, Molly moved to New York City to pursue an MFA in writing for children in young adults. After graduating, she worked as an online journalist for many years, writing for The New York Times, Glamour, Conde Nast Traveler, and many other publications. For the last four years, she's worked as an adjunct professor at NYU teaching creative writing, and as a professor at The School of Visual Arts teaching literature and writing for the web. She has also taught technical writing, creative writing, and journalism at Fordham University, Baruch College, and The School of the New York Times. When not teaching, Molly enjoys musical theater (watching and writing it), swimming, and exploring every corner of New York.
Divya Sharma
Master's Degree, Teachers College, Post Secondary Higher Education
Bachelor of Arts, Global Studies
Divya Sharma is a California native that is passionate about education. He has had 9 years of experience with teaching students. Divya has taught and developed curriculum of English with context of history. During his undergraduate career he was elected as the Academic Affairs Commissioner — where he was charged with representing 30,000 undergraduate students while implementing programs to positively impact retention efforts on campus. Divya followed his passions and attended Columbia University to further his studies as a higher educational professional. In his spare time, Divya enjoys photography, running, and cooking!
Zulema Renee Summerfield
Master of Fine Arts, Creative Writing/Fiction
Bachelor of Arts, English/Creative Writing
• Books: Every Other Weekend, Little, Brown (2018); Everything Faces All Ways At Once, Fourteen Hills Press (2010)
• Short Stories: Guernica, The Threepenny Review, Heavy Feather Review, Transfer Magazine, Chaffey Review, Sand Canyon Review, 580 Split, Invisible City, We Still Like
• Semi-finalist, VCU Cabell First Novelist Award, 2019
• Barnes and Noble Discovery Award Selection, 2018
• MacDowell Colony Fellow, 2017
• Winner, Michael Rubin Book Award, 2010
• Winner, Clarke-Gross Novel-In Progress Award, 2008
Zulema Renee Summerfield is a writer, educator, and creative coach. She is the author of Every Other Weekend, out now in paperback from Little, Brown, and her short fiction has been published in a number of literary journals. She has taught writing in venues ranging from elementary to high schools and online classrooms. In addition to teaching elements of story and structure, Zulema specializes in goal management and creative problem-solving for students. A MacDowell Colony fellow, she is currently at work on a new novel and a collection of short stories. When she's not teaching or writing, Zulema enjoys swimming and spending time with her family.
Kelsea Jeon
Kelsea Jeon is a senior at Yale University majoring in History, with a concentration in Early America. At Yale, she is the Co-President of the Roosevelt Institute, Yale's only student-led public policy think tank. She is also a member of the Yale Mock Trial Association, one of the top college Mock Trial teams; and in 2018, she competed in the final round of the national championship, earning her title of an All-American Attorney. In her free time, Kelsea enjoys hanging out with her sisters in Yale's Zeta Xi chapter of Kappa Kappa Gamma, journaling, and doing yoga.
Lois Huang
Lois Huang is an entrepreneur and business executive. She studied at University of McGill in Canada, after which she joined the fashion industry, developing world famous brands for companies in North America (US and Canada). Lois Huang has had a strong passion for education since she began her university studies. Her inspiration and ambition are to deliver comprehensive learning solutions to help students become more productive throughout their learning journey. She wants to help students and families to pursue better educational opportunities and achieve innovative success.
Interested in our courses? Choose your grade level to learn more
See Courses
The Language Arts Mentorship Program (L.A.M.P) helps students in grades 4-12 build foundational learning skills to stimulate and communicate original thinking.
More Additional Links
Copyright Genesis Education. Powered by ZOOMPOINT.


text:
\section{Introduction}
The prevalence of pre-trained models has fueled exciting academic and industry progress in natural language processing. It has allowed practitioners to re-use computationally expensive training steps and bypass the most inaccessible portion of model training \cite{wolf2019huggingface}. In neural machine translation (NMT), these general pre-trained models still struggle with translating domain specific material and require further tuning to achieve desired performance. In this work, we focus on methods for adapting off-the-shelf, third party, pre-trained translation models in which no additional architectural changes or edits to the model's pre-training scheme are possible. Intuitively, domain adaptation using clean, in-domain parallel data should provide the best results. However, such data is often hard and expensive to obtain. Monolingual in-domain data is more abundant and, at the cost of translation quality, can be used to generate synthetic parallel data.

We aim to elucidate which domain adaptation approaches of off-the-shelf translation models best suit various low data resource scenarios to yield the highest in-domain translation quality. We explore the benefits and trade-offs of domain adaptation methods in combination and isolation. While setting up our experiments, we found English in-domain monolingual data to be much more readily available than in-domain data for other languages. Collecting high quality monolingual electronic, medical, and biomedical domain data for adapting out of English translation models (En$\rightarrow$*) proved to be difficult to the extent that we limit our study to models translating into English (*$\rightarrow$En). For all experiments, the source language is one of Russian, Chinese, Spanish, or Japanese and the target language is always English. Similarly, as the source language is always non-English, we limit the scope of our work to scenarios with differing access to in-domain parallel and target side monolingual data.

\begin{table*}
\centering
\begin{tabular}{|c|c|c|c|c|c|c|c|c|c|}
\hline
\multirow{2}{*}{\textbf{This Study}} & \multicolumn{3}{c|}{\textbf{In-Domain Data Scenario}} & \multicolumn{6}{c|}{\textbf{Adaptation Approaches}} \\
\cline{2-10}
& Parallel & Source Mono & Target Mono & FT & SF & BT & ST & TBT & TST \\
\hline
\cmark & \cmark & \xmark & \xmark & \cmark & \cmark & \xmark & \xmark & \xmark & \xmark \\
\hline
\xmark  & \cmark & \cmark & \xmark & \cmark & \cmark & \xmark & \cmark & \xmark & \cmark \\
\hline
\cmark & \cmark & \xmark & \cmark & \cmark & \cmark & \cmark & \xmark & \cmark & \xmark \\
\hline
\xmark & \cmark & \cmark & \cmark & \cmark & \cmark & \cmark & \cmark & \cmark & \cmark \\
\hline
\xmark & \xmark & \cmark & \cmark & \xmark & \cmark & \cmark & \cmark & \xmark & \xmark \\
\hline
\xmark & \xmark & \cmark & \xmark & \xmark & \xmark & \xmark & \cmark & \xmark & \xmark \\
\hline
\cmark & \xmark & \xmark & \cmark & \xmark & \cmark & \cmark & \xmark & \xmark & \xmark \\
\hline
\end{tabular}
\caption{\label{scenarios} Data Resource Scenarios and Corresponding Possible Adaptation Methods. Adaptation approaches include 1) FT - Finetuning, 2) SF - Shallow Fusion decoding with in-domain language models, 3) BT - Backtranslation, 4) ST - Self-training, 5) TBT - Tagged Backtranslation, 6) TST - Tagged Self-training}
\end{table*}

We examine domain adaptation approaches under three in-domain data availability scenarios: parallel data only, target side monolingual data only, and both parallel and target side monolingual data. We compare parallel in-domain fine-tuning, mixed-domain fine-tuning \citep{zhang-etal-2019-curriculum}, traditional back-translation \citep{sennrich-etal-2016-improving, edunov2018understanding}, tagged back-translation \citep{caswell-etal-2019-tagged}, and in-domain language model shallow fusion across scenarios where applicable. See Table \ref{scenarios} for a breakdown of data availability conditions and the fixed architecture adaptation methods that can be applied to each.

Further, we use of domain classifiers to mine additional in-domain parallel data - adding dimension to the quantity verses quality trade off encountered in back-translation discussions. Finally, we suggest an ensemble approach to mitigate degradation in original domain performance.

\section{Contributions}
Our main contributions include:
\begin{itemize}
    \item A systematic empirical comparison of domain adaptation approaches of third-party, fixed architecture transformer-based NMT models
    \item A simple ensemble method to preserve original domain performance while gaining translation ability across new domains
    \item An effective low resource parallel data augmentation approach to improve in-domain performance
    \item The release of consumer electronic and clinical domain datasets across Russian $\rightarrow$ English, Chinese $\rightarrow$ English, Spanish $\rightarrow$ English, and Japanese $\rightarrow$ English translation pairs and our code.
\end{itemize}

\section{Related Work}
\subsection{Low-Resource Machine Translation}
Solutions to low resource machine translation range from transfer learning approaches and data mining strategies like the ones we focus on in this paper, to meta learning approaches \citet{gu-etal-2018-meta}, pre-training methods \citet{song2019mass}, multilingual \citet{zoph-etal-2016-transfer} strategies, and methods making strategic use of monolingual source and target data \cite{zhang-zong-2016-exploiting}. \citet{NIPS2016_5b69b9cb} proposed a dual learning reinforcement learning solution to low-resource machine translation that takes advantage of large amounts of both source and target monolingual data. \citet{AhmadniaDorr+2019+268+278} combines dual learning with self-training and co-learning using the synthetic translation examples produced during the dual learning round trip translations to further train their translation models.

\subsection{Domain Adaptation Strategies}
Domain adaptation is a widely studied research area with strategies that vary by data access, training objective, and architectural changes. As we focus purely on data centric strategies for adapting off-the-shelf NMT models, so other adaptation approaches like adapter methods \citet{houlsby2019parameter}, differential adaptation, and deep fusion\citet{dou2019domain} that require access and/or edits to the NMT model's architecture fall out of the scope of this work. In section \ref{our-domain-adapt-methods} we give a detailed introduction to each domain adaption method we explore.

\subsection{Empirical Studies of Fixed-Architecture Domain Adaptation}
There are a couple of existing empirical comparisons of domain adaptation methods using LSTM neural machine translation models. \citet{chu-etal-2017-empirical} explores mixed domain fine-tuning and compares different in-domain up-sampling strategies to mitigate overfitting on generally low resource parallel domain data. Our work is most similar to that of \citet{Chenhui-Chu2018}. In their empirical study, \citet{Chenhui-Chu2018} compares fine-tuning NMT models on parallel mixed domain data with fine-tuning models on data that was synthetically generated via back-translation. Though they propose a single domain adaptation method for RNN based models in which they combine back-translation, mixed-domain fine-tuning, and shallow fusion strategies, they do not explore iterative combinations of these approaches and therefore do not give strong evidence for one method over another. They also don't consider tagged back-translation, multi-domain ensembling, or additional data mining strategies as we do in this work.

\citep{saunders2021domain} and \cite{chu-wang-2018-survey} perform literary surveys on domain adaptation approaches for neural machine translation. Other works have explored domain adaptation under one of the three situations we compare in our investigation. \citet{Sun2019AnES} studies training and adapting unsupervised translation models with exclusively monolingual data. They use cross-lingual language model pre-training \cite{NEURIPS2019_c04c19c2} to initialize their unsupervised neural machine translation (UNMT) models, then train and fine-tune their models according to different scenarios modulating the presence or absence of in-domain and out-of-domain source and target monolingual data.

\section{Methods} \label{our-domain-adapt-methods}
We focus on the efficacy of domain adaptation approaches for pre-trained models with access to different combinations of parallel and monolingual target language data. We assume access to out-of-domain NMT models in both language directions, but narrow our study to improving in-domain performance in the Other Language $\rightarrow$ English direction, using English $\rightarrow$ Other Language models solely for back-translation. We empirically compare domain adaptation methods separately and together. We only consider adaptation of a fixed-architecture base models.

\subsection{Fine-Tuning}
We characterize the compromise between minimizing general domain degradation and improving in-domain performance in our parallel data approaches. We also experiment with fine-tuning baseline models on solely parallel in-domain data and on a mix of original and in-domain data \citep{zhang-etal-2019-curriculum}.

\subsection{Back-Translation}
In back-translation \citep{sennrich-etal-2016-improving, edunov2018understanding, lample2018unsupervised}, target side monolingual data is used to generate synthetic parallel data. A reverse direction translation model translates the target language into the source language, often using sampling instead of greedy decoding to increase translation diversity. The forward direction translation model is fine-tuned on this generated parallel data. The reverse direction translation model can be used as is, or fine-tuned with available domain data before back-translation \citep{kumari-etal-2021-domain, artetxe-etal-2018-unsupervised}. In tagged back-translation \citep{caswell-etal-2019-tagged} a special token (e.g. <BT>) is prepended before the synthetically generated source sentence. This tag helps the model differentiates noisy synthetic translations from ground truth examples.

\subsection{Shallow Fusion Decoding}
Shallow fusion \citep{gulcehre2015using, lample2018unsupervised, DBLP:journals/corr/abs-1910-02555} combines the next token probability predicted by a pre-trained language model possessing parameters $\phi_{t}$ with the next token probability predicted by the NMT model's decoder $\theta_{t}$ at every time step $t$. The generated translation benefits from the fluidity and target language knowledge of the language model while relying on the NMT decoder for semantic content. The two probabilities are added with a language model coefficient $\lambda_{LM}$ scaling the language model's contribution.

\begin{equation}
\begin{split}
    P(y_t|y_{<t},x) = P_{NMT}(y_t|y_{<t},x;\theta_{t}) \\
                      + \lambda_{LM} * P_{LM}(y_t|y_{<t};\phi_{t})
\end{split}
\end{equation}

The language model is fine-tuned on target side monolingual data before shallow fusion decoding.

\subsection{Ensemble}
We propose using an ensemble of fine-tuned models with the base translation model to gain the benefits of adaptation across domains while maintaining high original domain performance. $k$ indicates the total number of models in the ensemble, we average their probability distributions over the next token at every decoding time step $t$.

$$P(y_t|y_{<t},x;\theta_{1} \ldots \theta_{k}) = \frac{1}{k} \sum_{i=1}^k P(y_t|y_{<t},x;\theta_{i})$$

Here $P(y_t|y_{<t},x;\theta_{i})$ is the probability of target token $y$ at time step $t$ for a single NMT model $i$ given the input tokens $x$ and previously generated tokens $y_{<t}$.

\section{Datasets}
We define low resources as falling between 5k and 9k in-domain parallel training examples and moderately low resource has an order of magnitude more data, in this case around ~47k examples. We create low resource consumer electronic and medical domain datasets for each language pair. We also gathered in-domain monolingual data for the medical and consumer electronic domains. Final data totals for each language, split, and domain are listed in Table \ref{data-totals}. We make the datasets and dataset creation code publicly available. \footnote{Anonymized}

\begin{table}
\centering
\resizebox{\columnwidth}{!}{
\begin{tabular}{|c|c|c|c|c|}
\hline
\textbf{Domain} & \textbf{Language Pair} & \textbf{Train} & \textbf{Val} & \textbf{Test} \\
\hline
\multirow{ 4}{*}{Electronic} & Zh $\rightarrow$ En & 7,041 & 475 & 479 \\
& Ja $\rightarrow$ En & 6,777 & 452 & 460 \\
& Es $\rightarrow$ En & 6,973 & 421 & 430 \\
& Ru $\rightarrow$ En & 7,276 & 478 & 522 \\
\hline
\multirow{ 4}{*}{Medical} & Zh $\rightarrow$ En & 8,760 & 448 & 446 \\
& Ja $\rightarrow$ En & 5,399 & 460 & 461 \\
& Es $\rightarrow$ En & 8,494 & 434 & 437 \\
& Ru $\rightarrow$ En & 5,401 & 507 & 493 \\
\hline
\multirow{ 1}{*}{Biomedical} & Ru $\rightarrow$ En & 46,782 & 279 & - \\
\hline
\end{tabular}}
\caption{\label{data-totals}
Total parallel examples for each split of each language pair.
}
\end{table}
\subsection{Parallel Consumer Electronic Dataset}

We used human generated translations from consumer electronic websites to construct the consumer electronic dataset. We crawled multilingual versions of XXXX\footnote{Website anonymized for review} website, matching translated versions of each page via their URLs.

To convert document level translations into aligned sentences, we separated sentences using NLTK's sentence splitter \footnote{https://www.nltk.org/api/nltk.tokenize.html} for English, Spanish, and Russian. We used the Spacy \footnote{https://spacy.io/models/zh} library's Chinese splitter to separate Mandarin sentences and the Konoha \footnote{https://github.com/himkt/konoha} library to split Japanese sentences. We then used the Vecalign library \footnote{https://github.com/thompsonb/vecalign} \citep{thompson-koehn-2019-vecalign} in conjunction with the Language-Agnostic SEntence Representations (LASER) multilingual embedding library \citep{artetxe-schwenk-2019-massively} to align translated document pairs on a sentence level. We selected sentence pairs within a set cosine distance range of 0.07 to 0.6 for the training split, where we define cosine distance as (1 - cosine similarity). For the validation and test splits, we used a narrower cosine distance range of 0.1 to 0.5 and removed overlapping validation and test examples from the train split. We manually cleaned the validation and test splits-- separating examples containing multiple sentences and removing sentence fragments lacking a clear meaning.

\subsection{Parallel Medical Dataset}
Parallel translations of medical domain data were gathered from translated pdfs publicly provided by the NIH U.S. National Library of Medicine \footnote{https://medlineplus.gov/languages/languages.html}. An identical process to the one used for the consumer electronic dataset was employed to create the parallel medical train, validation, and test splits.

\subsection{Parallel Biomedical Dataset}
We use the publicly available WMT'20 biomedical shared task train split for our Ru $\leftrightarrow$ En biomedical domain data. To explore the benefits of noisy parallel data, we also mine additional parallel in-domain data from the out-of-domain En $\leftrightarrow$ Ru WMT'21 News dataset. Here, noise comes from potential domain misclassification instead of from erroneous translation as with back-translation.

To collect this data, we trained English and Russian biomedical domain classifiers. Each classifier utilized a pre-trained BERT Base style encoder \cite{devlin2018bert} with added classification layers. Our Russian domain classifier used RuBERT Base \cite{kuratov2019adaptation}. An equal amount of 45K negative and positive classification examples were collected from the parallel En $\leftrightarrow$ Ru WMT'21 news task training data and the WMT'20 Biomedical Shared Task train set respectively.

We classified the English half of the entire ~26M parallel En $\leftrightarrow$ Ru WMT'21 news task training data, saving all sentences with predicted biomedical domain probabilities over 50\%. We then used our Russian classifier to predict biomedical domain probabilities for the Russian half of the parallel data. We averaged the classifier scores from the English and Russian domain classifiers and used this averaged score as our final selection criteria. See Table \ref{mind-parallel-data-results} for data totals corresponding to different probability score cutoffs.

\subsection{Monolingual Data} \label{mono-data}
We trained consumer electronic and medical domain binary classifiers to select in-domain monolingual data from the cc100 dataset \citep{conneau-etal-2020-unsupervised, wenzek-etal-2020-ccnet} \footnote{http://data.statmt.org/cc-100/}. When training the classifiers, target side in-domain data was used for the positive class and an equal amount of randomly sampled cc100 data was collected for the negative. After a total of 500k English sentences were classified as in-domain, the top 200k, 50k and \textit{n} (where \textit{n} is commensurate with parallel data totals for a given domain) examples with the highest in-domain probabilities were used in experiments.

\section{Experimental Setup}

\subsection{Base Models}
We start by training strong baseline models for all four language pairs: Spanish, Chinese, Russian and Japaneses to English. We train our models on WMT'21 news data. Table \ref{baselines} shows initial SacreBLEU \cite{post2018} results of our models on WMT'20 test sets as well as in-domain test sets. Our models are based on the transformer large architecture \cite{vaswani2017}. As suggested in \citet{Shoeybi2019MegatronLMTM}, we move the layer normalization step for every transformer block to before each multi-head attention and feed forward sub-layer. The NMT models have 240M parameters. They took between 22 and 24 hours to train on 64 Tesla-V100 32GB GPUs with a per GPU batch size of 16k tokens. We use an initial learning rate between 1e-4 and 5e-4 with between 8k and 30k warm-up steps and an Adam \cite{Kingma2015AdamAM} optimizer.

\begin{table}
\centering
\resizebox{\columnwidth}{!}{
\begin{tabular}{ccccc}
\hline
\textbf{Language pair} & \textbf{WMT} & \textbf{CE} & \textbf{Medical} & \textbf{Biomed}\\
\hline
Zh $\rightarrow$ En & 24.5  & 34.5 & 29.9 & - \\
Ja $\rightarrow$ En & 19.8  & 36.1 & 26.8 & - \\
Es $\rightarrow$ En & 39.9 & 46.1 & 50.1 & - \\
Ru $\rightarrow$ En & 36.2  & 25.6 & 27.7 & 38.5 \\
\hline\end{tabular}}
\caption{\label{baselines}
SacreBLEU scores of baseline models on WMT'20 for all language pairs except Es $\rightarrow$ En, and in-domain test sets for all languages. The Es $\rightarrow$ En scores are on WMT'12.}
\end{table}

We use byte-pair encoding (BPE) \cite{sennrich-etal-2016-neural} to create our NMT vocabularies. The BPE model was trained using the original domain data. The Zh $\rightarrow$ En, Ja $\rightarrow$ En, and Ru $\rightarrow$ En translation models have separate encoder and decoder vocabularies, while our Es $\rightarrow$ En model shares a single vocabulary between the encoder and decoder. Each vocabulary has 32k tokens. Our reverse direction base models (En $\rightarrow$ Other Language) used for back-translation experiments were trained in the same manner and with the same transformer architecture as our baseline forward direction models.

\subsection{Language Models}
Our language models use a similar 16-layer transformer decoder architecture to \citet{radford2019language} with the same pre-layer normalization edit recommended by \citet{Shoeybi2019MegatronLMTM} as in our base NMT models. Though all the language models are English, they are each distinctly trained for every language pair to ensure the decoder and language models have the same tokenizer vocabulary. They are all trained on News Crawl \footnote{http://data.statmt.org/news-crawl/} English data, then fine-tuned on the English half of the in-domain parallel datasets separately.

\subsection{Adaptation}
We fixed the fine-tuning learning rates to be between 1e-5 and 5e-6. Models were fine-tuned on 1 Tesla-V100 16GB GPU until in-domain validation BLEU scores plateaued. BLEU plateau occurred after only 1 epoch for Es-En fine-tuning experiments with a batch size of 1024 tokens. Zh-En, Ja-En, Ru-En models' validation BLEU stopped improving after 15-20 epochs, while the Ru-En models for the biomedical domain finished training after 1 epoch.

We back-translate our monolingual data described in section \ref{mono-data} with our reverse direction models generating synthetic parallel data from the top 200k, top 50k, and top $n$ (where $n$ equals the number parallel examples for that language pair and domain) monolingual examples. The top $n$ and top 50k parallel examples are a higher quality subset of the 200k examples, allowing us to characterize the impact of quantity verses quality of back-translated data in a low resource environment. We fine-tune our base models exclusively on back-translated data for our target side monolingual experiments and on a mix of human-translated and back-translated data for our combined parallel and target monolingual experiments.

\section{In-Domain Parallel Results}
The in-domain parallel results averaged across all language pairs and across the consumer electronic and medical domains are displayed in Table \ref{mono-only-and-parallel-only-results-summary}. Mixed-domain fine-tuning has slightly lower in-domain performance on average (a -0.6 difference) compared with fine-tuning on in-domain parallel data only. Mixed domain fine-tuning does help maintain original domain performance. We see an average original domain BLEU score of 27.4 for models tuned on mixed domain data and an average score of 23.8 for those fine-tuned on in-domain data only. Shallow fusion decoding with an in-domain language model boosts performance for all languages and domains.

\begin{table*}
    \centering
    \begin{tabular}{clcc}
    \hline
    \textbf{Scenario}  & \textbf{Adaptation Method} & \textbf{New-Domain Avg.} &  \textbf{Original Domain Avg.} \\
    \hline
    \multirow{5}{*}{Parallel Only} & Baseline & 34.6 & 30.1 \\
                                   & Ensemble & 39.3 & \bf{29.4} \\
                                   & Mixed-Domain & 42.1 & 27.4 \\
                                   & In-Domain & 42.7 & 23.8 \\
                                   & In-Domain + SF & \bf{43.0} & 25.8 \\
    \hline
    \multirow{5}{*}{Target Monolingual Only} & BT Top200k cc100 & 35.4 & 28.4 \\
                                             & BT Top50k cc100 & 35.9 & 27.3 \\
                                             & BT TopN cc100 & 35.8 & \bf{28.9} \\
                                             & BT Target Half & 39.0 & 27.3 \\
                                             & BT Target Half + SF & \bf{40.0} & 27.2 \\
    \hline
    \end{tabular}
    \caption{\label{mono-only-and-parallel-only-results-summary}
    A summary of results for parallel only and target monolingual only experiments. Each value is the BLEU score for the corresponding adaptation method averaged over every language pair and over the consumer electronic and medical domains. Ensemble denotes the scores for an ensemble of the baseline model and models fine-tuned on in-domain data. SF stands for shallow fusion. BT stands for back translation. TopN models were fine-tuned on an amount of back-translated cc100 data equal to the number of examples in their corresponding in-domain datasets.}
\end{table*}

\subsection{Mitigating Original Domain Degradation via Ensembling}
We ensemble our fine-tuned in-domain parallel and baseline models together. When ensembled, baseline performance remains within 0.7 BLEU of its original score across all languages. This is a huge improvement over the 10+ BLEU score drop seen when fine-tuning on the consumer electronic domain. No ensemble out performs their single fine-tuned model counterparts when evaluated on in-domain data. Nevertheless, the ensemble still achieves a several BLEU point improvement in each domain over the baseline and the average BLEU score across all domains is much higher when additionally comparing against any single model's out-of-domain performance. These results indicate that, when translating mixed domain or unknown domain data, ensembling in-domain models should lead to higher quality translations-- even when domains are drastically different (e.g. the consumer electronic and medical domains). Figure \ref{fig:parallel-method-results-and-tradeoffs} presents the original vs. new domain trade-off for the consumer electronic and medical domains averaged over all language pairs. Figure \ref{fig:in-and-out-of-domain-avg} highlights the advantage of ensembling. The x-axis values in \ref{fig:in-and-out-of-domain-avg} are the combined average consumer electronic and medical domain BLEU scores irrespective of the domain for which each model was fine-tuned.

\begin{figure*}[htb!]
    \centering
    \begin{subfigure}[b]{0.45\textwidth}
        \centering
        \includegraphics[width=\textwidth]{figures/org_vs_new_domain_tradoff.pdf}
        \caption{in-domain averages}
        \label{fig:in-domain-avg}
    \end{subfigure}
    \hfill
    \begin{subfigure}[b]{0.45\textwidth}
        \centering
        \includegraphics[width=\textwidth]{figures/parallel-all-avged.pdf}
        \caption{in-and-out of domain averages}
        \label{fig:in-and-out-of-domain-avg}
    \end{subfigure}
    \caption{Original vs. new domain performance trade-off across parallel adaptation methods. (a) shows the average original domain performance as a function of the average \textit{in-domain} BLEU score for each new domain across all languages, capturing this trade-off when translating one new domain at a time.  (b) displays the average \textit{in-and-out of domain} BLEU scores for each adaptation method over all language pairs, encapsulating trade off trends when translating text from multiple new domains simultaneously.}
    \label{fig:parallel-method-results-and-tradeoffs}
\end{figure*}

\subsection{Benefits of Mined In-Domain Parallel Data}
Fine-tuning the baseline Ru $\rightarrow$ En model with combined mined and original parallel data increased performance over fine-tuning with original data alone by 0.2 and 0.7 BLEU. A higher domain probability cutoff threshold, favoring reduced in-domain noise over larger data quantity, resulted in a 0.5 BLEU score difference between the two models trained with mined data. It should be noted that the additional parallel data was mined from the parallel Ru $\rightarrow$ En training set used to train the baseline model. Though the model saw all mined examples during initial baseline training, viewing these in-domain examples again during the fine-tuning stage still increased in-domain performance over fine-tuning on purely unseen data. See Table \ref{mind-parallel-data-results} for a result breakdown.

\begin{table}
\centering
\resizebox{\columnwidth}{!}{
\begin{tabular}{lccc}
\hline
\textbf{Model Description} & \textbf{Cutoff} & \textbf{Total} & \textbf{BLEU} \\
\hline
Baseline & - & - & 38.5 \\

Original Parallel & - & 46,782 & 41.3 \\

Original Parallel + Mined & .90 & 254,037 & 41.5 \\

\textbf{Original Parallel + Mined} & \textbf{.97} & \textbf{140,414} & \textbf{42.0} \\
\hline
\end{tabular}}
\caption{\label{mind-parallel-data-results}
The performance increase from adding mined parallel data to the biomedical Ru $\rightarrow$ En fine-tuning set. "cutoff" is the domain classifier probability threshold and "total" is the train set size with mined examples added.
}
\end{table}

\begin{figure*}
    \centering
    \begin{subfigure}[b]{0.32\textwidth}
        \centering
        \includegraphics[width=\textwidth]{figures/ru-backtrans-qaunt-qual.pdf}
        \caption{Zh $\rightarrow$ En}
        \label{fig:quant-qual-ru}
    \end{subfigure}
    \hfill
    \begin{subfigure}[b]{0.32\textwidth}
        \centering
        \includegraphics[width=\textwidth]{figures/zh-backtrans-qaunt-qual.pdf}
        \caption{Ru $\rightarrow$ En}
        \label{fig:quant-qual-zh}
    \end{subfigure}
    \hfill
    \begin{subfigure}[b]{0.32\textwidth}
        \centering
        \includegraphics[width=\textwidth]{figures/es-backtrans-qaunt-qual.pdf}
        \caption{Es $\rightarrow$ En}
        \label{fig:quant-qual-es}
    \end{subfigure}
    \caption{In-Domain BLEU scores after fine-tuning the baseline model on back-translated data. The green points correspond to scores from models fine-tuned on the back-translated target-half of the in-domain parallel datasets. The pink points are from models fine-tuned on back-translated cc100 data. Models with scores shown in green saw smaller volumes of high quality synthetic data compared to those in pink.}
    \label{fig:backtrans-quantity-vs-quality}
\end{figure*}

\section{Target Side Monolingual Results}
In the bottom half of Table \ref{mono-only-and-parallel-only-results-summary} we see that fine-tuning a base model on high quality back-translated data comes within approximately 3 BLEU points of fine-tuning on human translated in-domain parallel data on average. When analyzing individual models, the best Ja $\rightarrow$ En monolingual model matched the performance of the in-domain parallel model for the medical domain and surpassed it by 0.7 BLEU points in the consumer electronic domain.

\subsection{Back-Translated Quantity vs. Quality Trade-Off}
We compare fine-tuning on back-translated data mined from cc100 with fine-tuning on the back-translated English half of each in-domain parallel dataset. Across the language pairs, there seems to be no major difference in performance between models fine-tuned with 200k, 50k, or top \textit{n} totals of back-translated cc100 data. They each reach an average BLEU score between 35 and 36 as seen in Table \ref{mono-only-and-parallel-only-results-summary}. When base models are fine-tuned on the back-translated target half of the original in-domain parallel datasets, the model's performance increased by an average of 3.3 BLEU compared to the cc100 back-translation experiments. Even with over 20x less data, fine-tuning on clean (in terms of domain accuracy) back-translated examples out scores utilizing noisier data. This point is also illustrated in Figure \ref{fig:backtrans-quantity-vs-quality}.

\subsection{Shallow Fusion}
Across the board shallow fusion leads to within 1.0 BLEU score increase compared to the baseline scores in each domain. For Ru $\rightarrow$ En, Es $\rightarrow$ En, and Ja $\rightarrow$ En shallow fusion with in-domain language models also increases \textit{original domain} performance within 1.0 BLEU point of their original WMT'20 scores. This shows that even language models fine-tuned on out of domain data still have an advantageous impact when used for shallow fusion decoding.

\section{In-Domain Parallel + Target Side Monolingual Results}
We experimented with a number of approaches to combining back-translated data with in-domain parallel data. A summary of results for these experiments can be seen in Table \ref{mono-and-parallel-combined-results-summary}. We first used our baseline reverse direction model to back-translate the top 50k cc100 sentences from each domain. Baseline models fine-tuned on a mix of this data and in-domain parallel data improved an average of 6.2 BLEU points from the baseline. We then tried fine-tuning our \textit{reverse} direction model on our parallel domain data before back-translation. Combining this back-translated data with human translated parallel-data resulted in another +0.7 BLEU increase on average. Next we experimented with tagged back-translation. We prepended a special back-translation token ($<BT>$) to the beginning of every synthetic back-translated input from our previous iteration. Tagging back-translated examples actually slightly decreased the average BLEU score by -0.1 compared to not adding tags. Finally, we used in-domain shallow fusion decoding at inference time with our model fine-tuned via tagged back-translation for a +0.6 average performance boost. Despite our efforts, we found none to be as effective as fine-tuning on purely in-domain data or a mix of in-domain and out-of-domain parallel data.

\begin{table}
    \centering
    \resizebox{\columnwidth}{!}{
    \begin{tabular}{lc}
    \hline
    \textbf{Method} & \textbf{BLEU} \\
    \hline
    Baseline & 30.6\\
    In-Domain Parallel + BT w/ Base & 36.8 \\
    In-Domain Parallel + BT w/ Tuned & 37.5 \\
    In-Domain Parallel + Tagged BT w/ Tuned & 37.4 \\
    In-Domain Parallel + Tagged BT w/ Tuned + SF & 38.0 \\
    \hline
    Mixed-Domain Parallel & 38.7 \\
    In-Domain Parallel Only & 38.9 \\
    In-Domain Parallel Only + SF & \bf{39.0} \\
    \hline
    \end{tabular}}
    \caption{\label{mono-and-parallel-combined-results-summary}
    In-domain parallel + target side monolingual results for Ru$\rightarrow$En BLEU scores averaged across the consumer electronic, medical, and biomedical domains. Base models were fine-tuned on a mix of in-domain parallel data and back-translated Top50k cc100 data. SF stands for shallow fusion and BT stand for back-translation. "w/ Tuned" indicates where target side monolingual data was back-translated with a reverse direction model that has been fine-tuned on parallel in-domain data. Methods using human translated parallel data alone out preformed those combining back-translated and human translated parallel data.}
\end{table}

\section{Recommendations}
\begin{enumerate}
    \item In low resource situations, with access to both parallel and monolingual data (<200k monolingual examples, <10k parallel examples), don't spend time on back-translation. Instead focus on parallel in-domain and mixed domain fine-tuning.
    \item Ensemble in-domain and baseline models for more robust translations when translating mixed or unknown domains.
     \item Use an in-domain language model for shallow fusion decoding. It will most likely improve both your in-domain and original domain performance, especially when parallel domain data is not available. In-domain shallow fusion can be an effective adaptation approach even without fine-tuning the baseline translation model.
    \item If you only have monolingual data, back-translate the highest quality monolingual data possible, prioritize quality over data volume in low resource settings (<200k monolingual examples).
    \item It is better to mine a moderate amount of parallel data over a larger amount of in-domain monolingual data.
\end{enumerate}

\section{Conclusion}
We conduct an empirical study comparing parallel and monolingual data approaches to domain adaptation in NMT. We made recommendations on how to achieve the best in-domain translation performance with access to low resource parallel and/or monolingual domain data. Additionally, we explored model ensembling to reduce regression of original domain performance and the benefits of mined in-domain parallel data. We hope this work can guide others in their creation of high quality domain specific machine translation systems.

redpajama_set_name: RedPajamaArXiv


When you have a car accident issue, there are many benefits to choosing a local car accident lawyer. In addition to supporting a member of your Belvedere Tiburon community, you are also choosing a lawyer that is more convenient to travel to, being local to Belvedere Tiburon, than if you were to choose a lawyer outside of the Belvedere Tiburon area.
By hiring a reputable Belvedere Tiburon car accident lawyer, you can ensure that your case will be handled in the most professional manner possible. And finding a good, experienced local Belvedere Tiburon car accident lawyer will give you the comfort of knowing that if you should have another car accident issue in the future, your lawyer is right around the corner.