query: The query consists of users’ needs, leading to several research questions that span a paragraph. Each candidate passage is an abstract from a scientific paper. The objective of this information retrieval task is to identify the abstract that most effectively meets the user’s needs in the query : I aim to develop a neural machine translation (NMT) model capable of effectively translating between multiple languages, each with unique morpheme usage. I wish to address several limitations of traditional NMT models. Firstly, my model should not rely on a fixed vocabulary but rather be capable of processing, embedding, and understanding new, unseen words. To achieve this, I plan to implement an intelligent subword tokenization scheme, which should enable my model to surpass the performance of basic character-level NMT. Moreover, the model needs to be parameter-efficient to facilitate deployment on compact wearable devices. It should also be capable of managing long-distance context and grammar dependencies within paragraphs. My current approach involves using a form of hierarchical decoding. This method initially translates larger and longer word or phrase-level information swiftly, then fine-tunes subword or even character-level information with greater precision. My model needs to reach state-of-the-art performance on standard machine translation datasets.

positive:
In neural machine translation (NMT), researchers face the challenge of
un-seen (or out-of-vocabulary OOV) words translation. To solve this, some
researchers propose the splitting of western languages such as English and
German into sub-words or compounds. In this paper, we try to address this OOV
issue and improve the NMT adequacy with a harder language Chinese whose
characters are even more sophisticated in composition. We integrate the Chinese
radicals into the NMT model with different settings to address the unseen words
challenge in Chinese to English translation. On the other hand, this also can
be considered as semantic part of the MT system since the Chinese radicals
usually carry the essential meaning of the words they are constructed in.
Meaningful radicals and new characters can be integrated into the NMT systems
with our models. We use an attention-based NMT system as a strong baseline
system. The experiments on standard Chinese-to-English NIST translation shared
task data 2006 and 2008 show that our designed models outperform the baseline
model in a wide range of state-of-the-art evaluation metrics including LEPOR,
BEER, and CharacTER, in addition to BLEU and NIST scores, especially on the
adequacy-level translation. We also have some interesting findings from the
results of our various experiment settings about the performance of words and
characters in Chinese NMT, which is different with other languages. For
instance, the fully character level NMT may perform well or the state of the
art in some other languages as researchers demonstrated recently, however, in
the Chinese NMT model, word boundary knowledge is important for the model
learning.

positive:
Neural machine translation (NMT), a new approach to machine translation, has
been proved to outperform conventional statistical machine translation (SMT)
across a variety of language pairs. Translation is an open-vocabulary problem,
but most existing NMT systems operate with a fixed vocabulary, which causes the
incapability of translating rare words. This problem can be alleviated by using
different translation granularities, such as character, subword and hybrid
word-character. Translation involving Chinese is one of the most difficult
tasks in machine translation, however, to the best of our knowledge, there has
not been any other work exploring which translation granularity is most
suitable for Chinese in NMT. In this paper, we conduct an extensive comparison
using Chinese-English NMT as a case study. Furthermore, we discuss the
advantages and disadvantages of various translation granularities in detail.
Our experiments show that subword model performs best for Chinese-to-English
translation with the vocabulary which is not so big while hybrid word-character
model is most suitable for English-to-Chinese translation. Moreover,
experiments of different granularities show that Hybrid_BPE method can achieve
best result on Chinese-to-English translation task.

positive:
The existing machine translation systems, whether phrase-based or neural,
have relied almost exclusively on word-level modelling with explicit
segmentation. In this paper, we ask a fundamental question: can neural machine
translation generate a character sequence without any explicit segmentation? To
answer this question, we evaluate an attention-based encoder-decoder with a
subword-level encoder and a character-level decoder on four language
pairs--En-Cs, En-De, En-Ru and En-Fi-- using the parallel corpora from WMT'15.
Our experiments show that the models with a character-level decoder outperform
the ones with a subword-level decoder on all of the four language pairs.
Furthermore, the ensembles of neural models with a character-level decoder
outperform the state-of-the-art non-neural machine translation systems on
En-Cs, En-De and En-Fi and perform comparably on En-Ru.

negative:
Neural machine translation (NMT), a new approach to machine translation, has
achieved promising results comparable to those of traditional approaches such
as statistical machine translation (SMT). Despite its recent success, NMT
cannot handle a larger vocabulary because training complexity and decoding
complexity proportionally increase with the number of target words. This
problem becomes even more serious when translating patent documents, which
contain many technical terms that are observed infrequently. In NMTs, words
that are out of vocabulary are represented by a single unknown token. In this
paper, we propose a method that enables NMT to translate patent sentences
comprising a large vocabulary of technical terms. We train an NMT system on
bilingual data wherein technical terms are replaced with technical term tokens;
this allows it to translate most of the source sentences except technical
terms. Further, we use it as a decoder to translate source sentences with
technical term tokens and replace the tokens with technical term translations
using SMT. We also use it to rerank the 1,000-best SMT translations on the
basis of the average of the SMT score and that of the NMT rescoring of the
translated sentences with technical term tokens. Our experiments on
Japanese-Chinese patent sentences show that the proposed NMT system achieves a
substantial improvement of up to 3.1 BLEU points and 2.3 RIBES points over
traditional SMT systems and an improvement of approximately 0.6 BLEU points and
0.8 RIBES points over an equivalent NMT system without our proposed technique.

negative:
Neural machine translation (NMT) models typically operate with a fixed
vocabulary, but translation is an open-vocabulary problem. Previous work
addresses the translation of out-of-vocabulary words by backing off to a
dictionary. In this paper, we introduce a simpler and more effective approach,
making the NMT model capable of open-vocabulary translation by encoding rare
and unknown words as sequences of subword units. This is based on the intuition
that various word classes are translatable via smaller units than words, for
instance names (via character copying or transliteration), compounds (via
compositional translation), and cognates and loanwords (via phonological and
morphological transformations). We discuss the suitability of different word
segmentation techniques, including simple character n-gram models and a
segmentation based on the byte pair encoding compression algorithm, and
empirically show that subword models improve over a back-off dictionary
baseline for the WMT 15 translation tasks English-German and English-Russian by
1.1 and 1.3 BLEU, respectively.

negative:
The non-autoregressive models have boosted the efficiency of neural machine
translation through parallelized decoding at the cost of effectiveness when
comparing with the autoregressive counterparts. In this paper, we claim that
the syntactic and semantic structures among natural language are critical for
non-autoregressive machine translation and can further improve the performance.
However, these structures are rarely considered in the existing
non-autoregressive models. Inspired by this intuition, we propose to
incorporate the explicit syntactic and semantic structures of languages into a
non-autoregressive Transformer, for the task of neural machine translation.
Moreover, we also consider the intermediate latent alignment within target
sentences to better learn the long-term token dependencies. Experimental
results on two real-world datasets (i.e., WMT14 En-De and WMT16 En-Ro) show
that our model achieves a significantly faster speed, as well as keeps the
translation quality when compared with several state-of-the-art
non-autoregressive models.

negative:
Neural machine translation (NMT) is one of the best methods for understanding
the differences in semantic rules between two languages. Especially for
Indo-European languages, subword-level models have achieved impressive results.
However, when the translation task involves Chinese, semantic granularity
remains at the word and character level, so there is still need more
fine-grained translation model of Chinese. In this paper, we introduce a simple
and effective method for Chinese translation at the sub-character level. Our
approach uses the Wubi method to translate Chinese into English; byte-pair
encoding (BPE) is then applied. Our method for Chinese-English translation
eliminates the need for a complicated word segmentation algorithm during
preprocessing. Furthermore, our method allows for sub-character-level neural
translation based on recurrent neural network (RNN) architecture, without
preprocessing. The empirical results show that for Chinese-English translation
tasks, our sub-character-level model has a comparable BLEU score to the subword
model, despite having a much smaller vocabulary. Additionally, the small
vocabulary is highly advantageous for NMT model compression.

negative:
Translation into morphologically-rich languages challenges neural machine
translation (NMT) models with extremely sparse vocabularies where atomic
treatment of surface forms is unrealistic. This problem is typically addressed
by either pre-processing words into subword units or performing translation
directly at the level of characters. The former is based on word segmentation
algorithms optimized using corpus-level statistics with no regard to the
translation task. The latter learns directly from translation data but requires
rather deep architectures. In this paper, we propose to translate words by
modeling word formation through a hierarchical latent variable model which
mimics the process of morphological inflection. Our model generates words one
character at a time by composing two latent representations: a continuous one,
aimed at capturing the lexical semantics, and a set of (approximately) discrete
features, aimed at capturing the morphosyntactic function, which are shared
among different surface forms. Our model achieves better accuracy in
translation into three morphologically-rich languages than conventional
open-vocabulary NMT methods, while also demonstrating a better generalization
capacity under low to mid-resource settings.

negative:
Recent studies on direct speech translation show continuous improvements by
means of data augmentation techniques and bigger deep learning models. While
these methods are helping to close the gap between this new approach and the
more traditional cascaded one, there are many incongruities among different
studies that make it difficult to assess the state of the art. Surprisingly,
one point of discussion is the segmentation of the target text. Character-level
segmentation has been initially proposed to obtain an open vocabulary, but it
results on long sequences and long training time. Then, subword-level
segmentation became the state of the art in neural machine translation as it
produces shorter sequences that reduce the training time, while being superior
to word-level models. As such, recent works on speech translation started using
target subwords despite the initial use of characters and some recent claims of
better results at the character level. In this work, we perform an extensive
comparison of the two methods on three benchmarks covering 8 language
directions and multilingual training. Subword-level segmentation compares
favorably in all settings, outperforming its character-level counterpart in a
range of 1 to 3 BLEU points.

negative:
Dealing with the complex word forms in morphologically rich languages is an
open problem in language processing, and is particularly important in
translation. In contrast to most modern neural systems of translation, which
discard the identity for rare words, in this paper we propose several
architectures for learning word representations from character and morpheme
level word decompositions. We incorporate these representations in a novel
machine translation model which jointly learns word alignments and translations
via a hard attention mechanism. Evaluating on translating from several
morphologically rich languages into English, we show consistent improvements
over strong baseline methods, of between 1 and 1.5 BLEU points.

negative:
A great proportion of sequence-to-sequence (Seq2Seq) models for Neural
Machine Translation (NMT) adopt Recurrent Neural Network (RNN) to generate
translation word by word following a sequential order. As the studies of
linguistics have proved that language is not linear word sequence but sequence
of complex structure, translation at each step should be conditioned on the
whole target-side context. To tackle the problem, we propose a new NMT model
that decodes the sequence with the guidance of its structural prediction of the
context of the target sequence. Our model generates translation based on the
structural prediction of the target-side context so that the translation can be
freed from the bind of sequential order. Experimental results demonstrate that
our model is more competitive compared with the state-of-the-art methods, and
the analysis reflects that our model is also robust to translating sentences of
different lengths and it also reduces repetition with the instruction from the
target-side context for decoding.

negative:
Neural machine translation (NMT) takes deterministic sequences for source
representations. However, either word-level or subword-level segmentations have
multiple choices to split a source sequence with different word segmentors or
different subword vocabulary sizes. We hypothesize that the diversity in
segmentations may affect the NMT performance. To integrate different
segmentations with the state-of-the-art NMT model, Transformer, we propose
lattice-based encoders to explore effective word or subword representation in
an automatic way during training. We propose two methods: 1) lattice positional
encoding and 2) lattice-aware self-attention. These two methods can be used
together and show complementary to each other to further improve translation
performance. Experiment results show superiorities of lattice-based encoders in
word-level and subword-level representations over conventional Transformer
encoder.

negative:
Neural Machine Translation (NMT) is an end-to-end learning approach for
automated translation, with the potential to overcome many of the weaknesses of
conventional phrase-based translation systems. Unfortunately, NMT systems are
known to be computationally expensive both in training and in translation
inference. Also, most NMT systems have difficulty with rare words. These issues
have hindered NMT's use in practical deployments and services, where both
accuracy and speed are essential. In this work, we present GNMT, Google's
Neural Machine Translation system, which attempts to address many of these
issues. Our model consists of a deep LSTM network with 8 encoder and 8 decoder
layers using attention and residual connections. To improve parallelism and
therefore decrease training time, our attention mechanism connects the bottom
layer of the decoder to the top layer of the encoder. To accelerate the final
translation speed, we employ low-precision arithmetic during inference
computations. To improve handling of rare words, we divide words into a limited
set of common sub-word units ("wordpieces") for both input and output. This
method provides a good balance between the flexibility of "character"-delimited
models and the efficiency of "word"-delimited models, naturally handles
translation of rare words, and ultimately improves the overall accuracy of the
system. Our beam search technique employs a length-normalization procedure and
uses a coverage penalty, which encourages generation of an output sentence that
is most likely to cover all the words in the source sentence. On the WMT'14
English-to-French and English-to-German benchmarks, GNMT achieves competitive
results to state-of-the-art. Using a human side-by-side evaluation on a set of
isolated simple sentences, it reduces translation errors by an average of 60%
compared to Google's phrase-based production system.

negative:
Recently, neural machine translation (NMT) has emerged as a powerful
alternative to conventional statistical approaches. However, its performance
drops considerably in the presence of morphologically rich languages (MRLs).
Neural engines usually fail to tackle the large vocabulary and high
out-of-vocabulary (OOV) word rate of MRLs. Therefore, it is not suitable to
exploit existing word-based models to translate this set of languages. In this
paper, we propose an extension to the state-of-the-art model of Chung et al.
(2016), which works at the character level and boosts the decoder with
target-side morphological information. In our architecture, an additional
morphology table is plugged into the model. Each time the decoder samples from
a target vocabulary, the table sends auxiliary signals from the most relevant
affixes in order to enrich the decoder's current state and constrain it to
provide better predictions. We evaluated our model to translate English into
German, Russian, and Turkish as three MRLs and observed significant
improvements.

negative:
Neural machine translation (NMT) is nowadays commonly applied at the subword
level, using byte-pair encoding. A promising alternative approach focuses on
character-level translation, which simplifies processing pipelines in NMT
considerably. This approach, however, must consider relatively longer
sequences, rendering the training process prohibitively expensive. In this
paper, we discuss a novel, Transformer-based approach, that we compare, both in
speed and in quality to the Transformer at subword and character levels, as
well as previously developed character-level models. We evaluate our models on
4 language pairs from WMT'15: DE-EN, CS-EN, FI-EN and RU-EN. The proposed novel
architecture can be trained on a single GPU and is 34% percent faster than the
character-level Transformer; still, the obtained results are at least on par
with it. In addition, our proposed model outperforms the subword-level model in
FI-EN and shows close results in CS-EN. To stimulate further research in this
area and close the gap with subword-level NMT, we make all our code and models
publicly available.

negative:
In recent years, Neural Machine Translation (NMT) has been shown to be more
effective than phrase-based statistical methods, thus quickly becoming the
state of the art in machine translation (MT). However, NMT systems are limited
in translating low-resourced languages, due to the significant amount of
parallel data that is required to learn useful mappings between languages. In
this work, we show how the so-called multilingual NMT can help to tackle the
challenges associated with low-resourced language translation. The underlying
principle of multilingual NMT is to force the creation of hidden
representations of words in a shared semantic space across multiple languages,
thus enabling a positive parameter transfer across languages. Along this
direction, we present multilingual translation experiments with three languages
(English, Italian, Romanian) covering six translation directions, utilizing
both recurrent neural networks and transformer (or self-attentive) neural
networks. We then focus on the zero-shot translation problem, that is how to
leverage multi-lingual data in order to learn translation directions that are
not covered by the available training material. To this aim, we introduce our
recently proposed iterative self-training method, which incrementally improves
a multilingual NMT on a zero-shot direction by just relying on monolingual
data. Our results on TED talks data show that multilingual NMT outperforms
conventional bilingual NMT, that the transformer NMT outperforms recurrent NMT,
and that zero-shot NMT outperforms conventional pivoting methods and even
matches the performance of a fully-trained bilingual system.

negative:
Subword units are an effective way to alleviate the open vocabulary problems
in neural machine translation (NMT). While sentences are usually converted into
unique subword sequences, subword segmentation is potentially ambiguous and
multiple segmentations are possible even with the same vocabulary. The question
addressed in this paper is whether it is possible to harness the segmentation
ambiguity as a noise to improve the robustness of NMT. We present a simple
regularization method, subword regularization, which trains the model with
multiple subword segmentations probabilistically sampled during training. In
addition, for better subword sampling, we propose a new subword segmentation
algorithm based on a unigram language model. We experiment with multiple
corpora and report consistent improvements especially on low resource and
out-of-domain settings.

negative:
Applying the Transformer architecture on the character level usually requires
very deep architectures that are difficult and slow to train. These problems
can be partially overcome by incorporating a segmentation into tokens in the
model. We show that by initially training a subword model and then finetuning
it on characters, we can obtain a neural machine translation model that works
at the character level without requiring token segmentation. We use only the
vanilla 6-layer Transformer Base architecture. Our character-level models
better capture morphological phenomena and show more robustness to noise at the
expense of somewhat worse overall translation quality. Our study is a
significant step towards high-performance and easy to train character-based
models that are not extremely large.

negative:
Neural machine translation (NMT) has achieved notable success in recent
times, however it is also widely recognized that this approach has limitations
with handling infrequent words and word pairs. This paper presents a novel
memory-augmented NMT (M-NMT) architecture, which stores knowledge about how
words (usually infrequently encountered ones) should be translated in a memory
and then utilizes them to assist the neural model. We use this memory mechanism
to combine the knowledge learned from a conventional statistical machine
translation system and the rules learned by an NMT system, and also propose a
solution for out-of-vocabulary (OOV) words based on this framework. Our
experiments on two Chinese-English translation tasks demonstrated that the
M-NMT architecture outperformed the NMT baseline by $9.0$ and $2.7$ BLEU points
on the two tasks, respectively. Additionally, we found this architecture
resulted in a much more effective OOV treatment compared to competitive
methods.

negative:
Phrases play an important role in natural language understanding and machine
translation (Sag et al., 2002; Villavicencio et al., 2005). However, it is
difficult to integrate them into current neural machine translation (NMT) which
reads and generates sentences word by word. In this work, we propose a method
to translate phrases in NMT by integrating a phrase memory storing target
phrases from a phrase-based statistical machine translation (SMT) system into
the encoder-decoder architecture of NMT. At each decoding step, the phrase
memory is first re-written by the SMT model, which dynamically generates
relevant target phrases with contextual information provided by the NMT model.
Then the proposed model reads the phrase memory to make probability estimations
for all phrases in the phrase memory. If phrase generation is carried on, the
NMT decoder selects an appropriate phrase from the memory to perform phrase
translation and updates its decoding state by consuming the words in the
selected phrase. Otherwise, the NMT decoder generates a word from the
vocabulary as the general NMT decoder does. Experiment results on the Chinese
to English translation show that the proposed model achieves significant
improvements over the baseline on various test sets.

negative:
Document-level machine translation incorporates inter-sentential dependencies
into the translation of a source sentence. In this paper, we propose a new
framework to model cross-sentence dependencies by training neural machine
translation (NMT) to predict both the target translation and surrounding
sentences of a source sentence. By enforcing the NMT model to predict source
context, we want the model to learn "contextualized" source sentence
representations that capture document-level dependencies on the source side. We
further propose two different methods to learn and integrate such
contextualized sentence embeddings into NMT: a joint training method that
jointly trains an NMT model with the source context prediction model and a
pre-training & fine-tuning method that pretrains the source context prediction
model on a large-scale monolingual document corpus and then fine-tunes it with
the NMT model. Experiments on Chinese-English and English-German translation
show that both methods can substantially improve the translation quality over a
strong document-level Transformer baseline.

negative:
The necessity of using a fixed-size word vocabulary in order to control the
model complexity in state-of-the-art neural machine translation (NMT) systems
is an important bottleneck on performance, especially for morphologically rich
languages. Conventional methods that aim to overcome this problem by using
sub-word or character-level representations solely rely on statistics and
disregard the linguistic properties of words, which leads to interruptions in
the word structure and causes semantic and syntactic losses. In this paper, we
propose a new vocabulary reduction method for NMT, which can reduce the
vocabulary of a given input corpus at any rate while also considering the
morphological properties of the language. Our method is based on unsupervised
morphology learning and can be, in principle, used for pre-processing any
language pair. We also present an alternative word segmentation method based on
supervised morphological analysis, which aids us in measuring the accuracy of
our model. We evaluate our method in Turkish-to-English NMT task where the
input language is morphologically rich and agglutinative. We analyze different
representation methods in terms of translation accuracy as well as the semantic
and syntactic properties of the generated output. Our method obtains a
significant improvement of 2.3 BLEU points over the conventional vocabulary
reduction technique, showing that it can provide better accuracy in open
vocabulary translation of morphologically rich languages.

negative:
Deep Learning techniques are powerful in mimicking humans in a particular set
of problems. They have achieved a remarkable performance in complex learning
tasks. Deep learning inspired Neural Machine Translation (NMT) is a proficient
technique that outperforms traditional machine translation. Performing
machine-aided translation on Indic languages has always been a challenging task
considering their rich and diverse grammar. The neural machine translation has
shown quality results compared to the traditional machine translation
approaches. The fully automatic machine translation becomes problematic when it
comes to low-resourced languages, especially with Sanskrit. This paper presents
attention mechanism based neural machine translation by selectively focusing on
a particular part of language sentences during translation. The work shows the
construction of Sanskrit to Hindi bilingual parallel corpus with nearly 10K
samples and having 178,000 tokens. The neural translation model equipped with
an attention mechanism has been trained on Sanskrit to Hindi parallel corpus.
The approach has shown the significance of attention mechanisms to overcome
long-term dependencies, primarily associated with low resources Indic
languages. The paper shows the attention plots on testing data to demonstrate
the alignment between source and translated words. For the evaluation of the
translated sentences, manual score based human evaluation and automatic
evaluation metric based techniques have been adopted. The attention mechanism
based neural translation has achieved 88% accuracy in human evaluation and a
BLEU score of 0.92 on Sanskrit to Hindi translation.

negative:
Most existing machine translation systems operate at the level of words,
relying on explicit segmentation to extract tokens. We introduce a neural
machine translation (NMT) model that maps a source character sequence to a
target character sequence without any segmentation. We employ a character-level
convolutional network with max-pooling at the encoder to reduce the length of
source representation, allowing the model to be trained at a speed comparable
to subword-level models while capturing local regularities. Our
character-to-character model outperforms a recently proposed baseline with a
subword-level encoder on WMT'15 DE-EN and CS-EN, and gives comparable
performance on FI-EN and RU-EN. We then demonstrate that it is possible to
share a single character-level encoder across multiple languages by training a
model on a many-to-one translation task. In this multilingual setting, the
character-level encoder significantly outperforms the subword-level encoder on
all the language pairs. We observe that on CS-EN, FI-EN and RU-EN, the quality
of the multilingual character-level translation even surpasses the models
specifically trained on that language pair alone, both in terms of BLEU score
and human judgment.

negative:
The authors of (Cho et al., 2014a) have shown that the recently introduced
neural network translation systems suffer from a significant drop in
translation quality when translating long sentences, unlike existing
phrase-based translation systems. In this paper, we propose a way to address
this issue by automatically segmenting an input sentence into phrases that can
be easily translated by the neural network translation model. Once each segment
has been independently translated by the neural machine translation model, the
translated clauses are concatenated to form a final translation. Empirical
results show a significant improvement in translation quality for long
sentences.

negative:
We propose a language-independent approach for improving statistical machine
translation for morphologically rich languages using a hybrid morpheme-word
representation where the basic unit of translation is the morpheme, but word
boundaries are respected at all stages of the translation process. Our model
extends the classic phrase-based model by means of (1) word boundary-aware
morpheme-level phrase extraction, (2) minimum error-rate training for a
morpheme-level translation model using word-level BLEU, and (3) joint scoring
with morpheme- and word-level language models. Further improvements are
achieved by combining our model with the classic one. The evaluation on English
to Finnish using Europarl (714K sentence pairs; 15.5M English words) shows
statistically significant improvements over the classic model based on BLEU and
human judgments.

negative:
In neural machine translation (NMT), generation of a target word depends on
both source and target contexts. We find that source contexts have a direct
impact on the adequacy of a translation while target contexts affect the
fluency. Intuitively, generation of a content word should rely more on the
source context and generation of a functional word should rely more on the
target context. Due to the lack of effective control over the influence from
source and target contexts, conventional NMT tends to yield fluent but
inadequate translations. To address this problem, we propose context gates
which dynamically control the ratios at which source and target contexts
contribute to the generation of target words. In this way, we can enhance both
the adequacy and fluency of NMT with more careful control of the information
flow from contexts. Experiments show that our approach significantly improves
upon a standard attention-based NMT system by +2.3 BLEU points.

negative:
Nearly all previous work on neural machine translation (NMT) has used quite
restricted vocabularies, perhaps with a subsequent method to patch in unknown
words. This paper presents a novel word-character solution to achieving open
vocabulary NMT. We build hybrid systems that translate mostly at the word level
and consult the character components for rare words. Our character-level
recurrent neural networks compute source word representations and recover
unknown target words when needed. The twofold advantage of such a hybrid
approach is that it is much faster and easier to train than character-based
ones; at the same time, it never produces unknown words as in the case of
word-based models. On the WMT'15 English to Czech translation task, this hybrid
approach offers an addition boost of +2.1-11.4 BLEU points over models that
already handle unknown words. Our best system achieves a new state-of-the-art
result with 20.7 BLEU score. We demonstrate that our character models can
successfully learn to not only generate well-formed words for Czech, a
highly-inflected language with a very complex vocabulary, but also build
correct representations for English source words.

negative:
It has been shown that increasing model depth improves the quality of neural
machine translation. However, different architectural variants to increase
model depth have been proposed, and so far, there has been no thorough
comparative study.
  In this work, we describe and evaluate several existing approaches to
introduce depth in neural machine translation. Additionally, we explore novel
architectural variants, including deep transition RNNs, and we vary how
attention is used in the deep decoder. We introduce a novel "BiDeep" RNN
architecture that combines deep transition RNNs and stacked RNNs.
  Our evaluation is carried out on the English to German WMT news translation
dataset, using a single-GPU machine for both training and inference. We find
that several of our proposed architectures improve upon existing approaches in
terms of speed and translation quality. We obtain best improvements with a
BiDeep RNN of combined depth 8, obtaining an average improvement of 1.5 BLEU
over a strong shallow baseline.
  We release our code for ease of adoption.

negative:
Most of the existing Neural Machine Translation (NMT) models focus on the
conversion of sequential data and do not directly use syntactic information. We
propose a novel end-to-end syntactic NMT model, extending a
sequence-to-sequence model with the source-side phrase structure. Our model has
an attention mechanism that enables the decoder to generate a translated word
while softly aligning it with phrases as well as words of the source sentence.
Experimental results on the WAT'15 English-to-Japanese dataset demonstrate that
our proposed model considerably outperforms sequence-to-sequence attentional
NMT models and compares favorably with the state-of-the-art tree-to-string SMT
system.

negative:
Exposing diverse subword segmentations to neural machine translation (NMT)
models often improves the robustness of machine translation as NMT models can
experience various subword candidates. However, the diversification of subword
segmentations mostly relies on the pre-trained subword language models from
which erroneous segmentations of unseen words are less likely to be sampled. In
this paper, we present adversarial subword regularization (ADVSR) to study
whether gradient signals during training can be a substitute criterion for
exposing diverse subword segmentations. We experimentally show that our
model-based adversarial samples effectively encourage NMT models to be less
sensitive to segmentation errors and improve the performance of NMT models in
low-resource and out-domain datasets.

negative:
We present IntelliCAT, an interactive translation interface with neural
models that streamline the post-editing process on machine translation output.
We leverage two quality estimation (QE) models at different granularities:
sentence-level QE, to predict the quality of each machine-translated sentence,
and word-level QE, to locate the parts of the machine-translated sentence that
need correction. Additionally, we introduce a novel translation suggestion
model conditioned on both the left and right contexts, providing alternatives
for specific words or phrases for correction. Finally, with word alignments,
IntelliCAT automatically preserves the original document's styles in the
translated document. The experimental results show that post-editing based on
the proposed QE and translation suggestions can significantly improve
translation quality. Furthermore, a user study reveals that three features
provided in IntelliCAT significantly accelerate the post-editing task,
achieving a 52.9\% speedup in translation time compared to translating from
scratch. The interface is publicly available at
https://intellicat.beringlab.com/.

negative:
Recurrent sequence generators conditioned on input data through an attention
mechanism have recently shown very good performance on a range of tasks in-
cluding machine translation, handwriting synthesis and image caption gen-
eration. We extend the attention-mechanism with features needed for speech
recognition. We show that while an adaptation of the model used for machine
translation in reaches a competitive 18.7% phoneme error rate (PER) on the
TIMIT phoneme recognition task, it can only be applied to utterances which are
roughly as long as the ones it was trained on. We offer a qualitative
explanation of this failure and propose a novel and generic method of adding
location-awareness to the attention mechanism to alleviate this issue. The new
method yields a model that is robust to long inputs and achieves 18% PER in
single utterances and 20% in 10-times longer (repeated) utterances. Finally, we
propose a change to the at- tention mechanism that prevents it from
concentrating too much on single frames, which further reduces PER to 17.6%
level.

negative:
Most of the Neural Machine Translation (NMT) models are based on the
sequence-to-sequence (Seq2Seq) model with an encoder-decoder framework equipped
with the attention mechanism. However, the conventional attention mechanism
treats the decoding at each time step equally with the same matrix, which is
problematic since the softness of the attention for different types of words
(e.g. content words and function words) should differ. Therefore, we propose a
new model with a mechanism called Self-Adaptive Control of Temperature (SACT)
to control the softness of attention by means of an attention temperature.
Experimental results on the Chinese-English translation and English-Vietnamese
translation demonstrate that our model outperforms the baseline models, and the
analysis and the case study show that our model can attend to the most relevant
elements in the source-side contexts and generate the translation of high
quality.

negative:
Neural machine translation (NMT) has achieved impressive performance on
machine translation task in recent years. However, in consideration of
efficiency, a limited-size vocabulary that only contains the top-N highest
frequency words are employed for model training, which leads to many rare and
unknown words. It is rather difficult when translating from the low-resource
and morphologically-rich agglutinative languages, which have complex morphology
and large vocabulary. In this paper, we propose a morphological word
segmentation method on the source-side for NMT that incorporates morphology
knowledge to preserve the linguistic and semantic information in the word
structure while reducing the vocabulary size at training time. It can be
utilized as a preprocessing tool to segment the words in agglutinative
languages for other natural language processing (NLP) tasks. Experimental
results show that our morphologically motivated word segmentation method is
better suitable for the NMT model, which achieves significant improvements on
Turkish-English and Uyghur-Chinese machine translation tasks on account of
reducing data sparseness and language complexity.

negative:
Neural machine translation (NMT) systems aim to map text from one language
into another. While there are a wide variety of applications of NMT, one of the
most important is translation of natural language. A distinguishing factor of
natural language is that words are typically ordered according to the rules of
the grammar of a given language. Although many advances have been made in
developing NMT systems for translating natural language, little research has
been done on understanding how the word ordering of and lexical similarity
between the source and target language affect translation performance. Here, we
investigate these relationships on a variety of low-resource language pairs
from the OpenSubtitles2016 database, where the source language is English, and
find that the more similar the target language is to English, the greater the
translation performance. In addition, we study the impact of providing NMT
models with part of speech of words (POS) in the English sequence and find
that, for Transformer-based models, the more dissimilar the target language is
from English, the greater the benefit provided by POS.

negative:
Neural machine translation (MT) models obtain state-of-the-art performance
while maintaining a simple, end-to-end architecture. However, little is known
about what these models learn about source and target languages during the
training process. In this work, we analyze the representations learned by
neural MT models at various levels of granularity and empirically evaluate the
quality of the representations for learning morphology through extrinsic
part-of-speech and morphological tagging tasks. We conduct a thorough
investigation along several parameters: word-based vs. character-based
representations, depth of the encoding layer, the identity of the target
language, and encoder vs. decoder representations. Our data-driven,
quantitative evaluation sheds light on important aspects in the neural MT
system and its ability to capture word structure.

negative:
Neural Machine Translation (NMT) can be used to generate fluent output. As
such, language models have been investigated for incorporation with NMT. In
prior investigations, two models have been used: a translation model and a
language model. The translation model's predictions are weighted by the
language model with a hand-crafted ratio in advance. However, these approaches
fail to adopt the language model weighting with regard to the translation
history. In another line of approach, language model prediction is incorporated
into the translation model by jointly considering source and target
information. However, this line of approach is limited because it largely
ignores the adequacy of the translation output.
  Accordingly, this work employs two mechanisms, the translation model and the
language model, with an attentive architecture to the language model as an
auxiliary element of the translation model. Compared with previous work in
English--Japanese machine translation using a language model, the experimental
results obtained with the proposed Dynamic Fusion mechanism improve BLEU and
Rank-based Intuitive Bilingual Evaluation Scores (RIBES) scores. Additionally,
in the analyses of the attention and predictivity of the language model, the
Dynamic Fusion mechanism allows predictive language modeling that conforms to
the appropriate grammatical structure.

negative:
Standard neural machine translation (NMT) is on the assumption of
document-level context independent. Most existing document-level NMT methods
are satisfied with a smattering sense of brief document-level information,
while this work focuses on exploiting detailed document-level context in terms
of multiple forms of document embeddings, which is capable of sufficiently
modeling deeper and richer document-level context. The proposed document-aware
NMT is implemented to enhance the Transformer baseline by introducing both
global and local document-level clues on the source end. Experiments show that
the proposed method significantly improves the translation performance over
strong baselines and other related studies.

negative:
Neural machine translation (NMT) aims at solving machine translation (MT)
problems using neural networks and has exhibited promising results in recent
years. However, most of the existing NMT models are shallow and there is still
a performance gap between a single NMT model and the best conventional MT
system. In this work, we introduce a new type of linear connections, named
fast-forward connections, based on deep Long Short-Term Memory (LSTM) networks,
and an interleaved bi-directional architecture for stacking the LSTM layers.
Fast-forward connections play an essential role in propagating the gradients
and building a deep topology of depth 16. On the WMT'14 English-to-French task,
we achieve BLEU=37.7 with a single attention model, which outperforms the
corresponding single shallow model by 6.2 BLEU points. This is the first time
that a single NMT model achieves state-of-the-art performance and outperforms
the best conventional model by 0.7 BLEU points. We can still achieve BLEU=36.3
even without using an attention mechanism. After special handling of unknown
words and model ensembling, we obtain the best score reported to date on this
task with BLEU=40.4. Our models are also validated on the more difficult WMT'14
English-to-German task.

negative:
Translating characters instead of words or word-fragments has the potential
to simplify the processing pipeline for neural machine translation (NMT), and
improve results by eliminating hyper-parameters and manual feature engineering.
However, it results in longer sequences in which each symbol contains less
information, creating both modeling and computational challenges. In this
paper, we show that the modeling problem can be solved by standard
sequence-to-sequence architectures of sufficient depth, and that deep models
operating at the character level outperform identical models operating over
word fragments. This result implies that alternative architectures for handling
character input are better viewed as methods for reducing computation time than
as improved ways of modeling longer sequences. From this perspective, we
evaluate several techniques for character-level NMT, verify that they do not
match the performance of our deep character baseline model, and evaluate the
performance versus computation time tradeoffs they offer. Within this
framework, we also perform the first evaluation for NMT of conditional
computation over time, in which the model learns which timesteps can be
skipped, rather than having them be dictated by a fixed schedule specified
before training begins.

negative:
Neural Machine Translation (NMT) for low-resource languages suffers from low
performance because of the lack of large amounts of parallel data and language
diversity. To contribute to ameliorating this problem, we built a baseline
model for English-Hausa machine translation, which is considered a task for
low-resource language. The Hausa language is the second largest Afro-Asiatic
language in the world after Arabic and it is the third largest language for
trading across a larger swath of West Africa countries, after English and
French. In this paper, we curated different datasets containing Hausa-English
parallel corpus for our translation. We trained baseline models and evaluated
the performance of our models using the Recurrent and Transformer
encoder-decoder architecture with two tokenization approaches: standard
word-level tokenization and Byte Pair Encoding (BPE) subword tokenization.

negative:
In this paper we demonstrate that $\textit{context free grammar (CFG) based
methods for grammar induction benefit from modeling lexical dependencies}$.
This contrasts to the most popular current methods for grammar induction, which
focus on discovering $\textit{either}$ constituents $\textit{or}$ dependencies.
Previous approaches to marry these two disparate syntactic formalisms (e.g.
lexicalized PCFGs) have been plagued by sparsity, making them unsuitable for
unsupervised grammar induction. However, in this work, we present novel neural
models of lexicalized PCFGs which allow us to overcome sparsity problems and
effectively induce both constituents and dependencies within a single model.
Experiments demonstrate that this unified framework results in stronger results
on both representations than achieved when modeling either formalism alone.
Code is available at https://github.com/neulab/neural-lpcfg.

negative:
Neural machine translation is a recently proposed approach to machine
translation. Unlike the traditional statistical machine translation, the neural
machine translation aims at building a single neural network that can be
jointly tuned to maximize the translation performance. The models proposed
recently for neural machine translation often belong to a family of
encoder-decoders and consists of an encoder that encodes a source sentence into
a fixed-length vector from which a decoder generates a translation. In this
paper, we conjecture that the use of a fixed-length vector is a bottleneck in
improving the performance of this basic encoder-decoder architecture, and
propose to extend this by allowing a model to automatically (soft-)search for
parts of a source sentence that are relevant to predicting a target word,
without having to form these parts as a hard segment explicitly. With this new
approach, we achieve a translation performance comparable to the existing
state-of-the-art phrase-based system on the task of English-to-French
translation. Furthermore, qualitative analysis reveals that the
(soft-)alignments found by the model agree well with our intuition.

negative:
Most modern neural machine translation (NMT) systems rely on presegmented
inputs. Segmentation granularity importantly determines the input and output
sequence lengths, hence the modeling depth, and source and target vocabularies,
which in turn determine model size, computational costs of softmax
normalization, and handling of out-of-vocabulary words. However, the current
practice is to use static, heuristic-based segmentations that are fixed before
NMT training. This begs the question whether the chosen segmentation is optimal
for the translation task. To overcome suboptimal segmentation choices, we
present an algorithm for dynamic segmentation based on the Adaptative
Computation Time algorithm (Graves 2016), that is trainable end-to-end and
driven by the NMT objective. In an evaluation on four translation tasks we
found that, given the freedom to navigate between different segmentation
levels, the model prefers to operate on (almost) character level, providing
support for purely character-level NMT models from a novel angle.

negative:
Multilingual neural machine translation (NMT), which translates multiple
languages using a single model, is of great practical importance due to its
advantages in simplifying the training process, reducing online maintenance
costs, and enhancing low-resource and zero-shot translation. Given there are
thousands of languages in the world and some of them are very different, it is
extremely burdensome to handle them all in a single model or use a separate
model for each language pair. Therefore, given a fixed resource budget, e.g.,
the number of models, how to determine which languages should be supported by
one model is critical to multilingual NMT, which, unfortunately, has been
ignored by previous work. In this work, we develop a framework that clusters
languages into different groups and trains one multilingual model for each
cluster. We study two methods for language clustering: (1) using prior
knowledge, where we cluster languages according to language family, and (2)
using language embedding, in which we represent each language by an embedding
vector and cluster them in the embedding space. In particular, we obtain the
embedding vectors of all the languages by training a universal neural machine
translation model. Our experiments on 23 languages show that the first
clustering method is simple and easy to understand but leading to suboptimal
translation accuracy, while the second method sufficiently captures the
relationship among languages well and improves the translation accuracy for
almost all the languages over baseline methods

negative:
Neural Machine Translation (NMT) is a new approach for Machine Translation
(MT), and due to its success, it has absorbed the attention of many researchers
in the field. In this paper, we study NMT model on Persian-English language
pairs, to analyze the model and investigate the appropriateness of the model
for scarce-resourced scenarios, the situation that exists for Persian-centered
translation systems. We adjust the model for the Persian language and find the
best parameters and hyper parameters for two tasks: translation and
transliteration. We also apply some preprocessing task on the Persian dataset
which yields to increase for about one point in terms of BLEU score. Also, we
have modified the loss function to enhance the word alignment of the model.
This new loss function yields a total of 1.87 point improvements in terms of
BLEU score in the translation quality.

negative:
Neural machine translation (NMT) has arguably achieved human level parity
when trained and evaluated at the sentence-level. Document-level neural machine
translation has received less attention and lags behind its sentence-level
counterpart. The majority of the proposed document-level approaches investigate
ways of conditioning the model on several source or target sentences to capture
document context. These approaches require training a specialized NMT model
from scratch on parallel document-level corpora. We propose an approach that
doesn't require training a specialized model on parallel document-level corpora
and is applied to a trained sentence-level NMT model at decoding time. We
process the document from left to right multiple times and self-train the
sentence-level model on pairs of source sentences and generated translations.
Our approach reinforces the choices made by the model, thus making it more
likely that the same choices will be made in other sentences in the document.
We evaluate our approach on three document-level datasets: NIST
Chinese-English, WMT'19 Chinese-English and OpenSubtitles English-Russian. We
demonstrate that our approach has higher BLEU score and higher human preference
than the baseline. Qualitative analysis of our approach shows that choices made
by model are consistent across the document.

negative:
Neural machine translation (NMT), a new approach to machine translation, has
achieved promising results comparable to those of traditional approaches such
as statistical machine translation (SMT). Despite its recent success, NMT
cannot handle a larger vocabulary because the training complexity and decoding
complexity proportionally increase with the number of target words. This
problem becomes even more serious when translating patent documents, which
contain many technical terms that are observed infrequently. In this paper, we
propose to select phrases that contain out-of-vocabulary words using the
statistical approach of branching entropy. This allows the proposed NMT system
to be applied to a translation task of any language pair without any
language-specific knowledge about technical term identification. The selected
phrases are then replaced with tokens during training and post-translated by
the phrase translation table of SMT. Evaluation on Japanese-to-Chinese,
Chinese-to-Japanese, Japanese-to-English and English-to-Japanese patent
sentence translation proved the effectiveness of phrases selected with
branching entropy, where the proposed NMT model achieves a substantial
improvement over a baseline NMT model without our proposed technique. Moreover,
the number of translation errors of under-translation by the baseline NMT model
without our proposed technique reduces to around half by the proposed NMT
model.

negative:
The quality of machine translation is rapidly evolving. Today one can find
several machine translation systems on the web that provide reasonable
translations, although the systems are not perfect. In some specific domains,
the quality may decrease. A recently proposed approach to this domain is neural
machine translation. It aims at building a jointly-tuned single neural network
that maximizes translation performance, a very different approach from
traditional statistical machine translation. Recently proposed neural machine
translation models often belong to the encoder-decoder family in which a source
sentence is encoded into a fixed length vector that is, in turn, decoded to
generate a translation. The present research examines the effects of different
training methods on a Polish-English Machine Translation system used for
medical data. The European Medicines Agency parallel text corpus was used as
the basis for training of neural and statistical network-based translation
systems. The main machine translation evaluation metrics have also been used in
analysis of the systems. A comparison and implementation of a real-time medical
translator is the main focus of our experiments.

negative:
We introduce a neural machine translation model that views the input and
output sentences as sequences of characters rather than words. Since word-level
information provides a crucial source of bias, our input model composes
representations of character sequences into representations of words (as
determined by whitespace boundaries), and then these are translated using a
joint attention/translation model. In the target language, the translation is
modeled as a sequence of word vectors, but each word is generated one character
at a time, conditional on the previous character generations in each word. As
the representation and generation of words is performed at the character level,
our model is capable of interpreting and generating unseen word forms. A
secondary benefit of this approach is that it alleviates much of the challenges
associated with preprocessing/tokenization of the source and target languages.
We show that our model can achieve translation results that are on par with
conventional word-based models.

negative:
Zero pronouns (ZPs) are frequently omitted in pro-drop languages, but should
be recalled in non-pro-drop languages. This discourse phenomenon poses a
significant challenge for machine translation (MT) when translating texts from
pro-drop to non-pro-drop languages. In this paper, we propose a unified and
discourse-aware ZP translation approach for neural MT models. Specifically, we
jointly learn to predict and translate ZPs in an end-to-end manner, allowing
both components to interact with each other. In addition, we employ
hierarchical neural networks to exploit discourse-level context, which is
beneficial for ZP prediction and thus translation. Experimental results on both
Chinese-English and Japanese-English data show that our approach significantly
and accumulatively improves both translation performance and ZP prediction
accuracy over not only baseline but also previous works using external ZP
prediction models. Extensive analyses confirm that the performance improvement
comes from the alleviation of different kinds of errors especially caused by
subjective ZPs.

negative:
In this paper, we introduce a hybrid search for attention-based neural
machine translation (NMT). A target phrase learned with statistical MT models
extends a hypothesis in the NMT beam search when the attention of the NMT model
focuses on the source words translated by this phrase. Phrases added in this
way are scored with the NMT model, but also with SMT features including
phrase-level translation probabilities and a target language model.
Experimental results on German->English news domain and English->Russian
e-commerce domain translation tasks show that using phrase-based models in NMT
search improves MT quality by up to 2.3% BLEU absolute as compared to a strong
NMT baseline.

negative:
Neural machine translation, a recently proposed approach to machine
translation based purely on neural networks, has shown promising results
compared to the existing approaches such as phrase-based statistical machine
translation. Despite its recent success, neural machine translation has its
limitation in handling a larger vocabulary, as training complexity as well as
decoding complexity increase proportionally to the number of target words. In
this paper, we propose a method that allows us to use a very large target
vocabulary without increasing training complexity, based on importance
sampling. We show that decoding can be efficiently done even with the model
having a very large target vocabulary by selecting only a small subset of the
whole target vocabulary. The models trained by the proposed approach are
empirically found to outperform the baseline models with a small vocabulary as
well as the LSTM-based neural machine translation models. Furthermore, when we
use the ensemble of a few models with very large target vocabularies, we
achieve the state-of-the-art translation performance (measured by BLEU) on the
English->German translation and almost as high performance as state-of-the-art
English->French translation system.

negative:
Interest in larger-context neural machine translation, including
document-level and multi-modal translation, has been growing. Multiple works
have proposed new network architectures or evaluation schemes, but potentially
helpful context is still sometimes ignored by larger-context translation
models. In this paper, we propose a novel learning algorithm that explicitly
encourages a neural translation model to take into account additional context
using a multilevel pair-wise ranking loss. We evaluate the proposed learning
algorithm with a transformer-based larger-context translation system on
document-level translation. By comparing performance using actual and random
contexts, we show that a model trained with the proposed algorithm is more
sensitive to the additional context.

negative:
Machine translation is one of the applications of natural language processing
which has been explored in different languages. Recently researchers started
paying attention towards machine translation for resource-poor languages and
closely related languages. A widespread and underlying problem for these
machine translation systems is the variation in orthographic conventions which
causes many issues to traditional approaches. Two languages written in two
different orthographies are not easily comparable, but orthographic information
can also be used to improve the machine translation system. This article offers
a survey of research regarding orthography's influence on machine translation
of under-resourced languages. It introduces under-resourced languages in terms
of machine translation and how orthographic information can be utilised to
improve machine translation. We describe previous work in this area, discussing
what underlying assumptions were made, and showing how orthographic knowledge
improves the performance of machine translation of under-resourced languages.
We discuss different types of machine translation and demonstrate a recent
trend that seeks to link orthographic information with well-established machine
translation methods. Considerable attention is given to current efforts of
cognates information at different levels of machine translation and the lessons
that can be drawn from this. Additionally, multilingual neural machine
translation of closely related languages is given a particular focus in this
survey. This article ends with a discussion of the way forward in machine
translation with orthographic information, focusing on multilingual settings
and bilingual lexicon induction.

negative:
To simultaneously capture syntax and global semantics from a text corpus, we
propose a new larger-context recurrent neural network (RNN) based language
model, which extracts recurrent hierarchical semantic structure via a dynamic
deep topic model to guide natural language generation. Moving beyond a
conventional RNN-based language model that ignores long-range word dependencies
and sentence order, the proposed model captures not only intra-sentence word
dependencies, but also temporal transitions between sentences and
inter-sentence topic dependencies. For inference, we develop a hybrid of
stochastic-gradient Markov chain Monte Carlo and recurrent autoencoding
variational Bayes. Experimental results on a variety of real-world text corpora
demonstrate that the proposed model not only outperforms larger-context
RNN-based language models, but also learns interpretable recurrent multilayer
topics and generates diverse sentences and paragraphs that are syntactically
correct and semantically coherent.

negative:
Sentences in a well-formed text are connected to each other via various links
to form the cohesive structure of the text. Current neural machine translation
(NMT) systems translate a text in a conventional sentence-by-sentence fashion,
ignoring such cross-sentence links and dependencies. This may lead to generate
an incoherent target text for a coherent source text. In order to handle this
issue, we propose a cache-based approach to modeling coherence for neural
machine translation by capturing contextual information either from recently
translated sentences or the entire document. Particularly, we explore two types
of caches: a dynamic cache, which stores words from the best translation
hypotheses of preceding sentences, and a topic cache, which maintains a set of
target-side topical words that are semantically related to the document to be
translated. On this basis, we build a new layer to score target words in these
two caches with a cache-based neural model. Here the estimated probabilities
from the cache-based neural model are combined with NMT probabilities into the
final word prediction probabilities via a gating mechanism. Finally, the
proposed cache-based neural model is trained jointly with NMT system in an
end-to-end manner. Experiments and analysis presented in this paper demonstrate
that the proposed cache-based model achieves substantial improvements over
several state-of-the-art SMT and NMT baselines.

negative:
The basic concept in Neural Machine Translation (NMT) is to train a large
Neural Network that maximizes the translation performance on a given parallel
corpus. NMT is then using a simple left-to-right beam-search decoder to
generate new translations that approximately maximize the trained conditional
probability. The current beam search strategy generates the target sentence
word by word from left-to- right while keeping a fixed amount of active
candidates at each time step. First, this simple search is less adaptive as it
also expands candidates whose scores are much worse than the current best.
Secondly, it does not expand hypotheses if they are not within the best scoring
candidates, even if their scores are close to the best one. The latter one can
be avoided by increasing the beam size until no performance improvement can be
observed. While you can reach better performance, this has the draw- back of a
slower decoding speed. In this paper, we concentrate on speeding up the decoder
by applying a more flexible beam search strategy whose candidate size may vary
at each time step depending on the candidate scores. We speed up the original
decoder by up to 43% for the two language pairs German-English and
Chinese-English without losing any translation quality.

negative:
How to effectively adapt neural machine translation (NMT) models according to
emerging cases without retraining? Despite the great success of neural machine
translation, updating the deployed models online remains a challenge. Existing
non-parametric approaches that retrieve similar examples from a database to
guide the translation process are promising but are prone to overfit the
retrieved examples. In this work, we propose to learn Kernel-Smoothed
Translation with Example Retrieval (KSTER), an effective approach to adapt
neural machine translation models online. Experiments on domain adaptation and
multi-domain machine translation datasets show that even without expensive
retraining, KSTER is able to achieve improvement of 1.1 to 1.5 BLEU scores over
the best existing online adaptation methods. The code and trained models are
released at https://github.com/jiangqn/KSTER.

negative:
In this paper, we study a new learning paradigm for Neural Machine
Translation (NMT). Instead of maximizing the likelihood of the human
translation as in previous works, we minimize the distinction between human
translation and the translation given by an NMT model. To achieve this goal,
inspired by the recent success of generative adversarial networks (GANs), we
employ an adversarial training architecture and name it as Adversarial-NMT. In
Adversarial-NMT, the training of the NMT model is assisted by an adversary,
which is an elaborately designed Convolutional Neural Network (CNN). The goal
of the adversary is to differentiate the translation result generated by the
NMT model from that by human. The goal of the NMT model is to produce high
quality translations so as to cheat the adversary. A policy gradient method is
leveraged to co-train the NMT model and the adversary. Experimental results on
English$\rightarrow$French and German$\rightarrow$English translation tasks
show that Adversarial-NMT can achieve significantly better translation quality
than several strong baselines.

negative:
In the past few years, attention mechanisms have become an indispensable
component of end-to-end neural machine translation models. However, previous
attention models always refer to some source words when predicting a target
word, which contradicts with the fact that some target words have no
corresponding source words. Motivated by this observation, we propose a novel
attention model that has the capability of determining when a decoder should
attend to source words and when it should not. Experimental results on NIST
Chinese-English translation tasks show that the new model achieves an
improvement of 0.8 BLEU score over a state-of-the-art baseline.

negative:
This paper describes the acquisition, preprocessing, segmentation, and
alignment of an Amharic-English parallel corpus. It will be helpful for machine
translation of a low-resource language, Amharic. We freely released the corpus
for research purposes. Furthermore, we developed baseline statistical and
neural machine translation systems; we trained statistical and neural machine
translation models using the corpus. In the experiments, we also used a large
monolingual corpus for the language model of statistical machine translation
and back-translation of neural machine translation. In the automatic
evaluation, neural machine translation models outperform statistical machine
translation models by approximately six to seven Bilingual Evaluation
Understudy (BLEU) points. Besides, among the neural machine translation models,
the subword models outperform the word-based models by three to four BLEU
points. Moreover, two other relevant automatic evaluation metrics, Translation
Edit Rate on Character Level and Better Evaluation as Ranking, reflect
corresponding differences among the trained models.

negative:
Neural Machine Translation (NMT) is a new approach to machine translation
that has made great progress in recent years. However, recent studies show that
NMT generally produces fluent but inadequate translations (Tu et al. 2016b; Tu
et al. 2016a; He et al. 2016; Tu et al. 2017). This is in contrast to
conventional Statistical Machine Translation (SMT), which usually yields
adequate but non-fluent translations. It is natural, therefore, to leverage the
advantages of both models for better translations, and in this work we propose
to incorporate SMT model into NMT framework. More specifically, at each
decoding step, SMT offers additional recommendations of generated words based
on the decoding information from NMT (e.g., the generated partial translation
and attention history). Then we employ an auxiliary classifier to score the SMT
recommendations and a gating function to combine the SMT recommendations with
NMT generations, both of which are jointly trained within the NMT architecture
in an end-to-end manner. Experimental results on Chinese-English translation
show that the proposed approach achieves significant and consistent
improvements over state-of-the-art NMT and SMT systems on multiple NIST test
sets.

negative:
Neural Machine Translation (NMT) can be improved by including document-level
contextual information. For this purpose, we propose a hierarchical attention
model to capture the context in a structured and dynamic manner. The model is
integrated in the original NMT architecture as another level of abstraction,
conditioning on the NMT model's own previous hidden states. Experiments show
that hierarchical attention significantly improves the BLEU score over a strong
NMT baseline with the state-of-the-art in context-aware methods, and that both
the encoder and decoder benefit from context in complementary ways.

negative:
A large number of significant assets are available online in English, which
is frequently translated into native languages to ease the information sharing
among local people who are not much familiar with English. However, manual
translation is a very tedious, costly, and time-taking process. To this end,
machine translation is an effective approach to convert text to a different
language without any human involvement. Neural machine translation (NMT) is one
of the most proficient translation techniques amongst all existing machine
translation systems. In this paper, we have applied NMT on two of the most
morphological rich Indian languages, i.e. English-Tamil and English-Malayalam.
We proposed a novel NMT model using Multihead self-attention along with
pre-trained Byte-Pair-Encoded (BPE) and MultiBPE embeddings to develop an
efficient translation system that overcomes the OOV (Out Of Vocabulary) problem
for low resourced morphological rich Indian languages which do not have much
translation available online. We also collected corpus from different sources,
addressed the issues with these publicly available data and refined them for
further uses. We used the BLEU score for evaluating our system performance.
Experimental results and survey confirmed that our proposed translator (24.34
and 9.78 BLEU score) outperforms Google translator (9.40 and 5.94 BLEU score)
respectively.

negative:
Standard neural machine translation (NMT) is on the assumption that the
document-level context is independent. Most existing document-level NMT
approaches are satisfied with a smattering sense of global document-level
information, while this work focuses on exploiting detailed document-level
context in terms of a memory network. The capacity of the memory network that
detecting the most relevant part of the current sentence from memory renders a
natural solution to model the rich document-level context. In this work, the
proposed document-aware memory network is implemented to enhance the
Transformer NMT baseline. Experiments on several tasks show that the proposed
method significantly improves the NMT performance over strong Transformer
baselines and other related studies.

negative:
In machine translation (MT) that involves translating between two languages
with significant differences in word order, determining the correct word order
of translated words is a major challenge. The dependency parse tree of a source
sentence can help to determine the correct word order of the translated words.
In this paper, we present a novel reordering approach utilizing a neural
network and dependency-based embeddings to predict whether the translations of
two source words linked by a dependency relation should remain in the same
order or should be swapped in the translated sentence. Experiments on
Chinese-to-English translation show that our approach yields a statistically
significant improvement of 0.57 BLEU point on benchmark NIST test sets,
compared to our prior state-of-the-art statistical MT system that uses sparse
dependency-based reordering features.

negative:
Compared to traditional statistical machine translation (SMT), neural machine
translation (NMT) often sacrifices adequacy for the sake of fluency. We propose
a method to combine the advantages of traditional SMT and NMT by exploiting an
existing phrase-based SMT model to compute the phrase-based decoding cost for
an NMT output and then using this cost to rerank the n-best NMT outputs. The
main challenge in implementing this approach is that NMT outputs may not be in
the search space of the standard phrase-based decoding algorithm, because the
search space of phrase-based SMT is limited by the phrase-based translation
rule table. We propose a soft forced decoding algorithm, which can always
successfully find a decoding path for any NMT output. We show that using the
forced decoding cost to rerank the NMT outputs can successfully improve
translation quality on four different language pairs.

negative:
Modern statistical machine translation (SMT) systems usually use a linear
combination of features to model the quality of each translation hypothesis.
The linear combination assumes that all the features are in a linear
relationship and constrains that each feature interacts with the rest features
in an linear manner, which might limit the expressive power of the model and
lead to a under-fit model on the current data. In this paper, we propose a
non-linear modeling for the quality of translation hypotheses based on neural
networks, which allows more complex interaction between features. A learning
framework is presented for training the non-linear models. We also discuss
possible heuristics in designing the network structure which may improve the
non-linear learning performance. Experimental results show that with the basic
features of a hierarchical phrase-based machine translation system, our method
produce translations that are better than a linear model.

negative:
This paper presents an empirical study of two machine translation-based
approaches for Vietnamese diacritic restoration problem, including phrase-based
and neural-based machine translation models. This is the first work that
applies neural-based machine translation method to this problem and gives a
thorough comparison to the phrase-based machine translation method which is the
current state-of-the-art method for this problem. On a large dataset, the
phrase-based approach has an accuracy of 97.32% while that of the neural-based
approach is 96.15%. While the neural-based method has a slightly lower
accuracy, it is about twice faster than the phrase-based method in terms of
inference speed. Moreover, neural-based machine translation method has much
room for future improvement such as incorporating pre-trained word embeddings
and collecting more training data.

negative:
We first observe a potential weakness of continuous vector representations of
symbols in neural machine translation. That is, the continuous vector
representation, or a word embedding vector, of a symbol encodes multiple
dimensions of similarity, equivalent to encoding more than one meaning of the
word. This has the consequence that the encoder and decoder recurrent networks
in neural machine translation need to spend substantial amount of their
capacity in disambiguating source and target words based on the context which
is defined by a source sentence. Based on this observation, in this paper we
propose to contextualize the word embedding vectors using a nonlinear
bag-of-words representation of the source sentence. Additionally, we propose to
represent special tokens (such as numbers, proper nouns and acronyms) with
typed symbols to facilitate translating those words that are not well-suited to
be translated via continuous vectors. The experiments on En-Fr and En-De reveal
that the proposed approaches of contextualization and symbolization improves
the translation quality of neural machine translation systems significantly.

negative:
Recently, the development of neural machine translation (NMT) has
significantly improved the translation quality of automatic machine
translation. While most sentences are more accurate and fluent than
translations by statistical machine translation (SMT)-based systems, in some
cases, the NMT system produces translations that have a completely different
meaning. This is especially the case when rare words occur.
  When using statistical machine translation, it has already been shown that
significant gains can be achieved by simplifying the input in a preprocessing
step. A commonly used example is the pre-reordering approach.
  In this work, we used phrase-based machine translation to pre-translate the
input into the target language. Then a neural machine translation system
generates the final hypothesis using the pre-translation. Thereby, we use
either only the output of the phrase-based machine translation (PBMT) system or
a combination of the PBMT output and the source sentence.
  We evaluate the technique on the English to German translation task. Using
this approach we are able to outperform the PBMT system as well as the baseline
neural MT system by up to 2 BLEU points. We analyzed the influence of the
quality of the initial system on the final result.

negative:
What are the units of text that we want to model? From bytes to multi-word
expressions, text can be analyzed and generated at many granularities. Until
recently, most natural language processing (NLP) models operated over words,
treating those as discrete and atomic tokens, but starting with byte-pair
encoding (BPE), subword-based approaches have become dominant in many areas,
enabling small vocabularies while still allowing for fast inference. Is the end
of the road character-level model or byte-level processing? In this survey, we
connect several lines of work from the pre-neural and neural era, by showing
how hybrid approaches of words and characters as well as subword-based
approaches based on learned segmentation have been proposed and evaluated. We
conclude that there is and likely will never be a silver bullet singular
solution for all applications and that thinking seriously about tokenization
remains important for many applications.

negative:
Neural machine translation (NMT) becomes a new state-of-the-art and achieves
promising translation results using a simple encoder-decoder neural network.
This neural network is trained once on the parallel corpus and the fixed
network is used to translate all the test sentences. We argue that the general
fixed network cannot best fit the specific test sentences. In this paper, we
propose the dynamic NMT which learns a general network as usual, and then
fine-tunes the network for each test sentence. The fine-tune work is done on a
small set of the bilingual training data that is obtained through similarity
search according to the test sentence. Extensive experiments demonstrate that
this method can significantly improve the translation performance, especially
when highly similar sentences are available.

negative:
We propose a simple solution to use a single Neural Machine Translation (NMT)
model to translate between multiple languages. Our solution requires no change
in the model architecture from our base system but instead introduces an
artificial token at the beginning of the input sentence to specify the required
target language. The rest of the model, which includes encoder, decoder and
attention, remains unchanged and is shared across all languages. Using a shared
wordpiece vocabulary, our approach enables Multilingual NMT using a single
model without any increase in parameters, which is significantly simpler than
previous proposals for Multilingual NMT. Our method often improves the
translation quality of all involved language pairs, even while keeping the
total number of model parameters constant. On the WMT'14 benchmarks, a single
multilingual model achieves comparable performance for
English$\rightarrow$French and surpasses state-of-the-art results for
English$\rightarrow$German. Similarly, a single multilingual model surpasses
state-of-the-art results for French$\rightarrow$English and
German$\rightarrow$English on WMT'14 and WMT'15 benchmarks respectively. On
production corpora, multilingual models of up to twelve language pairs allow
for better translation of many individual pairs. In addition to improving the
translation quality of language pairs that the model was trained with, our
models can also learn to perform implicit bridging between language pairs never
seen explicitly during training, showing that transfer learning and zero-shot
translation is possible for neural translation. Finally, we show analyses that
hints at a universal interlingua representation in our models and show some
interesting examples when mixing languages.

negative:
Differently from the traditional statistical MT that decomposes the
translation task into distinct separately learned components, neural machine
translation uses a single neural network to model the entire translation
process. Despite neural machine translation being de-facto standard, it is
still not clear how NMT models acquire different competences over the course of
training, and how this mirrors the different models in traditional SMT. In this
work, we look at the competences related to three core SMT components and find
that during training, NMT first focuses on learning target-side language
modeling, then improves translation quality approaching word-by-word
translation, and finally learns more complicated reordering patterns. We show
that this behavior holds for several models and language pairs. Additionally,
we explain how such an understanding of the training process can be useful in
practice and, as an example, show how it can be used to improve vanilla
non-autoregressive neural machine translation by guiding teacher model
selection.

negative:
Neural machine translation (NMT) models generally adopt an encoder-decoder
architecture for modeling the entire translation process. The encoder
summarizes the representation of input sentence from scratch, which is
potentially a problem if the sentence is ambiguous. When translating a text,
humans often create an initial understanding of the source sentence and then
incrementally refine it along the translation on the target side. Starting from
this intuition, we propose a novel encoder-refiner-decoder framework, which
dynamically refines the source representations based on the generated
target-side information at each decoding step. Since the refining operations
are time-consuming, we propose a strategy, leveraging the power of
reinforcement learning models, to decide when to refine at specific decoding
steps. Experimental results on both Chinese-English and English-German
translation tasks show that the proposed approach significantly and
consistently improves translation performance over the standard encoder-decoder
framework. Furthermore, when refining strategy is applied, results still show
reasonable improvement over the baseline without much decrease in decoding
speed.

negative:
Neural language models learn word representations, or embeddings, that
capture rich linguistic and conceptual information. Here we investigate the
embeddings learned by neural machine translation models, a recently-developed
class of neural language model. We show that embeddings from translation models
outperform those learned by monolingual models at tasks that require knowledge
of both conceptual similarity and lexical-syntactic role. We further show that
these effects hold when translating from both English to French and English to
German, and argue that the desirable properties of translation embeddings
should emerge largely independently of the source and target languages.
Finally, we apply a new method for training neural translation models with very
large vocabularies, and show that this vocabulary expansion algorithm results
in minimal degradation of embedding quality. Our embedding spaces can be
queried in an online demo and downloaded from our web page. Overall, our
analyses indicate that translation-based embeddings should be used in
applications that require concepts to be organised according to similarity
and/or lexical function, while monolingual embeddings are better suited to
modelling (nonspecific) inter-word relatedness.

negative:
In a pipeline speech translation system, automatic speech recognition (ASR)
system will transmit errors in recognition to the downstream machine
translation (MT) system. A standard machine translation system is usually
trained on parallel corpus composed of clean text and will perform poorly on
text with recognition noise, a gap well known in speech translation community.
In this paper, we propose a training architecture which aims at making a neural
machine translation model more robust against speech recognition errors. Our
approach addresses the encoder and the decoder simultaneously using adversarial
learning and data augmentation, respectively. Experimental results on IWSLT2018
speech translation task show that our approach can bridge the gap between the
ASR output and the MT input, outperforms the baseline by up to 2.83 BLEU on
noisy ASR output, while maintaining close performance on clean text.

negative:
Neural machine translation (NMT) often makes mistakes in translating
low-frequency content words that are essential to understanding the meaning of
the sentence. We propose a method to alleviate this problem by augmenting NMT
systems with discrete translation lexicons that efficiently encode translations
of these low-frequency words. We describe a method to calculate the lexicon
probability of the next word in the translation candidate by using the
attention vector of the NMT model to select which source word lexical
probabilities the model should focus on. We test two methods to combine this
probability with the standard NMT probability: (1) using it as a bias, and (2)
linear interpolation. Experiments on two corpora show an improvement of 2.0-2.3
BLEU and 0.13-0.44 NIST score, and faster convergence time.

negative:
Every person speaks or writes their own flavor of their native language,
influenced by a number of factors: the content they tend to talk about, their
gender, their social status, or their geographical origin.
  When attempting to perform Machine Translation (MT), these variations have a
significant effect on how the system should perform translation, but this is
not captured well by standard one-size-fits-all models.
  In this paper, we propose a simple and parameter-efficient adaptation
technique that only requires adapting the bias of the output softmax to each
particular user of the MT system, either directly or through a factored
approximation.
  Experiments on TED talks in three languages demonstrate improvements in
translation accuracy, and better reflection of speaker traits in the target
text.

negative:
Neural machine translation has become a major alternative to widely used
phrase-based statistical machine translation. We notice however that much of
research on neural machine translation has focused on European languages
despite its language agnostic nature. In this paper, we apply neural machine
translation to the task of Arabic translation (Ar<->En) and compare it against
a standard phrase-based translation system. We run extensive comparison using
various configurations in preprocessing Arabic script and show that the
phrase-based and neural translation systems perform comparably to each other
and that proper preprocessing of Arabic script has a similar effect on both of
the systems. We however observe that the neural machine translation
significantly outperform the phrase-based system on an out-of-domain test set,
making it attractive for real-world deployment.

negative:
System combination is an important technique for combining the hypotheses of
different machine translation systems to improve translation performance.
Although early statistical approaches to system combination have been proven
effective in analyzing the consensus between hypotheses, they suffer from the
error propagation problem due to the use of pipelines. While this problem has
been alleviated by end-to-end training of multi-source sequence-to-sequence
models recently, these neural models do not explicitly analyze the relations
between hypotheses and fail to capture their agreement because the attention to
a word in a hypothesis is calculated independently, ignoring the fact that the
word might occur in multiple hypotheses. In this work, we propose an approach
to modeling voting for system combination in machine translation. The basic
idea is to enable words in hypotheses from different systems to vote on words
that are representative and should get involved in the generation process. This
can be done by quantifying the influence of each voter and its preference for
each candidate. Our approach combines the advantages of statistical and neural
methods since it can not only analyze the relations between hypotheses but also
allow for end-to-end training. Experiments show that our approach is capable of
better taking advantage of the consensus between hypotheses and achieves
significant improvements over state-of-the-art baselines on Chinese-English and
English-German machine translation tasks.

negative:
For different language pairs, word-level neural machine translation (NMT)
models with a fixed-size vocabulary suffer from the same problem of
representing out-of-vocabulary (OOV) words. The common practice usually
replaces all these rare or unknown words with a <UNK> token, which limits the
translation performance to some extent. Most of recent work handled such a
problem by splitting words into characters or other specially extracted subword
units to enable open-vocabulary translation. Byte pair encoding (BPE) is one of
the successful attempts that has been shown extremely competitive by providing
effective subword segmentation for NMT systems. In this paper, we extend the
BPE style segmentation to a general unsupervised framework with three
statistical measures: frequency (FRQ), accessor variety (AV) and description
length gain (DLG). We test our approach on two translation tasks: German to
English and Chinese to English. The experimental results show that AV and DLG
enhanced systems outperform the FRQ baseline in the frequency weighted schemes
at different significant levels.

negative:
Multilingual machine translation, which translates multiple languages with a
single model, has attracted much attention due to its efficiency of offline
training and online serving. However, traditional multilingual translation
usually yields inferior accuracy compared with the counterpart using individual
models for each language pair, due to language diversity and model capacity
limitations. In this paper, we propose a distillation-based approach to boost
the accuracy of multilingual machine translation. Specifically, individual
models are first trained and regarded as teachers, and then the multilingual
model is trained to fit the training data and match the outputs of individual
models simultaneously through knowledge distillation. Experiments on IWSLT, WMT
and Ted talk translation datasets demonstrate the effectiveness of our method.
Particularly, we show that one model is enough to handle multiple languages (up
to 44 languages in our experiment), with comparable or even better accuracy
than individual models.

negative:
We study the application of active learning techniques to the translation of
unbounded data streams via interactive neural machine translation. The main
idea is to select, from an unbounded stream of source sentences, those worth to
be supervised by a human agent. The user will interactively translate those
samples. Once validated, these data is useful for adapting the neural machine
translation model.
  We propose two novel methods for selecting the samples to be validated. We
exploit the information from the attention mechanism of a neural machine
translation system. Our experiments show that the inclusion of active learning
techniques into this pipeline allows to reduce the effort required during the
process, while increasing the quality of the translation system. Moreover, it
enables to balance the human effort required for achieving a certain
translation quality. Moreover, our neural system outperforms classical
approaches by a large margin.

negative:
Neural machine translation (NMT) has recently become popular in the field of
machine translation. However, NMT suffers from the problem of repeating or
missing words in the translation. To address this problem, Tu et al. (2017)
proposed an encoder-decoder-reconstructor framework for NMT using
back-translation. In this method, they selected the best forward translation
model in the same manner as Bahdanau et al. (2015), and then trained a
bi-directional translation model as fine-tuning. Their experiments show that it
offers significant improvement in BLEU scores in Chinese-English translation
task. We confirm that our re-implementation also shows the same tendency and
alleviates the problem of repeating and missing words in the translation on a
English-Japanese task too. In addition, we evaluate the effectiveness of
pre-training by comparing it with a jointly-trained model of forward
translation and back-translation.

negative:
The manuscript presents an experiment at implementation of a Machine
Translation system in a MapReduce model. The empirical evaluation was done
using fully implemented translation systems embedded into the MapReduce
programming model. Two machine translation paradigms were studied: shallow
transfer Rule Based Machine Translation and Statistical Machine Translation.
  The results show that the MapReduce model can be successfully used to
increase the throughput of a machine translation system. Furthermore this
method enhances the throughput of a machine translation system without
decreasing the quality of the translation output.
  Thus, the present manuscript also represents a contribution to the seminal
work in natural language processing, specifically Machine Translation. It first
points toward the importance of the definition of the metric of throughput of
translation system and, second, the applicability of the machine translation
task to the MapReduce paradigm.

negative:
Recently, neural machine translation (NMT) has been extended to
multilinguality, that is to handle more than one translation direction with a
single system. Multilingual NMT showed competitive performance against pure
bilingual systems. Notably, in low-resource settings, it proved to work
effectively and efficiently, thanks to shared representation space that is
forced across languages and induces a sort of transfer-learning. Furthermore,
multilingual NMT enables so-called zero-shot inference across language pairs
never seen at training time. Despite the increasing interest in this framework,
an in-depth analysis of what a multilingual NMT model is capable of and what it
is not is still missing. Motivated by this, our work (i) provides a
quantitative and comparative analysis of the translations produced by
bilingual, multilingual and zero-shot systems; (ii) investigates the
translation quality of two of the currently dominant neural architectures in
MT, which are the Recurrent and the Transformer ones; and (iii) quantitatively
explores how the closeness between languages influences the zero-shot
translation. Our analysis leverages multiple professional post-edits of
automatic translations by several different systems and focuses both on
automatic standard metrics (BLEU and TER) and on widely used error categories,
which are lexical, morphology, and word order errors.

negative:
Neural Machine Translation (NMT) leverages one or more trained neural
networks for the translation of phrases. Sutskever introduced a sequence to
sequence based encoder-decoder model which became the standard for NMT based
systems. Attention mechanisms were later introduced to address the issues with
the translation of long sentences and improving overall accuracy. In this
paper, we propose a singular model for Neural Machine Translation based on
encoder-decoder models. Most translation models are trained as one model for
one translation. We introduce a neutral/universal model representation that can
be used to predict more than one language depending on the source and a
provided target. Secondly, we introduce an attention model by adding an overall
learning vector to the multiplicative model. With these two changes, by using
the novel universal model the number of models needed for multiple language
translation applications are reduced.

negative:
The interest in statistical machine translation systems increases currently
due to political and social events in the world. A proposed Statistical Machine
Translation (SMT) based model that can be used to translate a sentence from the
source Language (English) to the target language (Arabic) automatically through
efficiently incorporating different statistical and Natural Language Processing
(NLP) models such as language model, alignment model, phrase based model,
reordering model, and translation model. These models are combined to enhance
the performance of statistical machine translation (SMT). Many implementation
tools have been used in this work such as Moses, Gizaa++, IRSTLM, KenLM, and
BLEU. Based on the implementation, evaluation of this model, and comparing the
generated translation with other implemented machine translation systems like
Google Translate, it was proved that this proposed model has enhanced the
results of the statistical machine translation, and forms a reliable and
efficient model in this field of research.

negative:
An effective method to generate a large number of parallel sentences for
training improved neural machine translation (NMT) systems is the use of the
back-translations of the target-side monolingual data. The standard
back-translation method has been shown to be unable to efficiently utilize the
available huge amount of existing monolingual data because of the inability of
translation models to differentiate between the authentic and synthetic
parallel data during training. Tagging, or using gates, has been used to enable
translation models to distinguish between synthetic and authentic data,
improving standard back-translation and also enabling the use of iterative
back-translation on language pairs that underperformed using standard
back-translation. In this work, we approach back-translation as a domain
adaptation problem, eliminating the need for explicit tagging. In the approach
-- \emph{tag-less back-translation} -- the synthetic and authentic parallel
data are treated as out-of-domain and in-domain data respectively and, through
pre-training and fine-tuning, the translation model is shown to be able to
learn more efficiently from them during training. Experimental results have
shown that the approach outperforms the standard and tagged back-translation
approaches on low resource English-Vietnamese and English-German neural machine
translation.

negative:
We introduce our efforts towards building a universal neural machine
translation (NMT) system capable of translating between any language pair. We
set a milestone towards this goal by building a single massively multilingual
NMT model handling 103 languages trained on over 25 billion examples. Our
system demonstrates effective transfer learning ability, significantly
improving translation quality of low-resource languages, while keeping
high-resource language translation quality on-par with competitive bilingual
baselines. We provide in-depth analysis of various aspects of model building
that are crucial to achieving quality and practicality in universal NMT. While
we prototype a high-quality universal translation system, our extensive
empirical analysis exposes issues that need to be further addressed, and we
suggest directions for future research.

negative:
We investigate the potential of attention-based neural machine translation in
simultaneous translation. We introduce a novel decoding algorithm, called
simultaneous greedy decoding, that allows an existing neural machine
translation model to begin translating before a full source sentence is
received. This approach is unique from previous works on simultaneous
translation in that segmentation and translation are done jointly to maximize
the translation quality and that translating each segment is strongly
conditioned on all the previous segments. This paper presents a first step
toward building a full simultaneous translation system based on neural machine
translation.

negative:
Retrieving information from correlative paragraphs or documents to answer
open-domain multi-hop questions is very challenging. To deal with this
challenge, most of the existing works consider paragraphs as nodes in a graph
and propose graph-based methods to retrieve them. However, in this paper, we
point out the intrinsic defect of such methods. Instead, we propose a new
architecture that models paragraphs as sequential data and considers multi-hop
information retrieval as a kind of sequence labeling task. Specifically, we
design a rewritable external memory to model the dependency among paragraphs.
Moreover, a threshold gate mechanism is proposed to eliminate the distraction
of noise paragraphs. We evaluate our method on both full wiki and distractor
subtask of HotpotQA, a public textual multi-hop QA dataset requiring multi-hop
information retrieval. Experiments show that our method achieves significant
improvement over the published state-of-the-art method in retrieval and
downstream QA task performance.

negative:
Neural machine translation is a relatively new approach to statistical
machine translation based purely on neural networks. The neural machine
translation models often consist of an encoder and a decoder. The encoder
extracts a fixed-length representation from a variable-length input sentence,
and the decoder generates a correct translation from this representation. In
this paper, we focus on analyzing the properties of the neural machine
translation using two models; RNN Encoder--Decoder and a newly proposed gated
recursive convolutional neural network. We show that the neural machine
translation performs relatively well on short sentences without unknown words,
but its performance degrades rapidly as the length of the sentence and the
number of unknown words increase. Furthermore, we find that the proposed gated
recursive convolutional network learns a grammatical structure of a sentence
automatically.

negative:
Neural machine translation (NMT) has significantly improved the quality of
automatic translation models. One of the main challenges in current systems is
the translation of rare words. We present a generic approach to address this
weakness by having external models annotate the training data as Experts, and
control the model-expert interaction with a pointer network and reinforcement
learning. Our experiments using phrase-based models to simulate Experts to
complement neural machine translation models show that the model can be trained
to copy the annotations into the output consistently. We demonstrate the
benefit of our proposed framework in outof-domain translation scenarios with
only lexical resources, improving more than 1.0 BLEU point in both translation
directions English to Spanish and German to English

negative:
Neural Machine Translation (NMT) is a new approach to machine translation
that has shown promising results that are comparable to traditional approaches.
A significant weakness in conventional NMT systems is their inability to
correctly translate very rare words: end-to-end NMTs tend to have relatively
small vocabularies with a single unk symbol that represents every possible
out-of-vocabulary (OOV) word. In this paper, we propose and implement an
effective technique to address this problem. We train an NMT system on data
that is augmented by the output of a word alignment algorithm, allowing the NMT
system to emit, for each OOV word in the target sentence, the position of its
corresponding word in the source sentence. This information is later utilized
in a post-processing step that translates every OOV word using a dictionary.
Our experiments on the WMT14 English to French translation task show that this
method provides a substantial improvement of up to 2.8 BLEU points over an
equivalent NMT system that does not use this technique. With 37.5 BLEU points,
our NMT system is the first to surpass the best result achieved on a WMT14
contest task.

negative:
As the quality of machine translation rises and neural machine translation
(NMT) is moving from sentence to document level translations, it is becoming
increasingly difficult to evaluate the output of translation systems.
  We provide a test suite for WMT19 aimed at assessing discourse phenomena of
MT systems participating in the News Translation Task. We have manually checked
the outputs and identified types of translation errors that are relevant to
document-level translation.

negative:
By using a trigram model and fine-tuning a pretrained BERT model for sequence
classification, we show that machine translation and human translation can be
classified with an accuracy above chance level, which suggests that machine
translation and human translation are different in a systematic way. The
classification accuracy of machine translation is much higher than of human
translation. We show that this may be explained by the difference in lexical
diversity between machine translation and human translation. If machine
translation has independent patterns from human translation, automatic metrics
which measure the deviation of machine translation from human translation may
conflate difference with quality. Our experiment with two different types of
automatic metrics shows correlation with the result of the classification task.
Therefore, we suggest the difference in lexical diversity between machine
translation and human translation be given more attention in machine
translation evaluation.

negative:
In this paper, we evaluate the translation of negation both automatically and
manually, in English--German (EN--DE) and English--Chinese (EN--ZH). We show
that the ability of neural machine translation (NMT) models to translate
negation has improved with deeper and more advanced networks, although the
performance varies between language pairs and translation directions. The
accuracy of manual evaluation in EN-DE, DE-EN, EN-ZH, and ZH-EN is 95.7%,
94.8%, 93.4%, and 91.7%, respectively. In addition, we show that
under-translation is the most significant error type in NMT, which contrasts
with the more diverse error profile previously observed for statistical machine
translation. To better understand the root of the under-translation of
negation, we study the model's information flow and training data. While our
information flow analysis does not reveal any deficiencies that could be used
to detect or fix the under-translation of negation, we find that negation is
often rephrased during training, which could make it more difficult for the
model to learn a reliable link between source and target negation. We finally
conduct intrinsic analysis and extrinsic probing tasks on negation, showing
that NMT models can distinguish negation and non-negation tokens very well and
encode a lot of information about negation in hidden states but nevertheless
leave room for improvement.

negative:
We study a formalization of the grammar induction problem that models
sentences as being generated by a compound probabilistic context-free grammar.
In contrast to traditional formulations which learn a single stochastic
grammar, our grammar's rule probabilities are modulated by a per-sentence
continuous latent variable, which induces marginal dependencies beyond the
traditional context-free assumptions. Inference in this grammar is performed by
collapsed variational inference, in which an amortized variational posterior is
placed on the continuous variable, and the latent trees are marginalized out
with dynamic programming. Experiments on English and Chinese show the
effectiveness of our approach compared to recent state-of-the-art methods when
evaluated on unsupervised parsing.

negative:
We present a literature and empirical survey that critically assesses the
state of the art in character-level modeling for machine translation (MT).
Despite evidence in the literature that character-level systems are comparable
with subword systems, they are virtually never used in competitive setups in
WMT competitions. We empirically show that even with recent modeling
innovations in character-level natural language processing, character-level MT
systems still struggle to match their subword-based counterparts.
Character-level MT systems show neither better domain robustness, nor better
morphological generalization, despite being often so motivated. However, we are
able to show robustness towards source side noise and that translation quality
does not degrade with increasing beam size at decoding time.

negative:
Evaluating translation models is a trade-off between effort and detail. On
the one end of the spectrum there are automatic count-based methods such as
BLEU, on the other end linguistic evaluations by humans, which arguably are
more informative but also require a disproportionately high effort. To narrow
the spectrum, we propose a general approach on how to automatically expose
systematic differences between human and machine translations to human experts.
Inspired by adversarial settings, we train a neural text classifier to
distinguish human from machine translations. A classifier that performs and
generalizes well after training should recognize systematic differences between
the two classes, which we uncover with neural explainability methods. Our
proof-of-concept implementation, DiaMaT, is open source. Applied to a dataset
translated by a state-of-the-art neural Transformer model, DiaMaT achieves a
classification accuracy of 75% and exposes meaningful differences between
humans and the Transformer, amidst the current discussion about human parity.

negative:
Media coverage has a substantial effect on the public perception of events.
Nevertheless, media outlets are often biased. One way to bias news articles is
by altering the word choice. The automatic identification of bias by word
choice is challenging, primarily due to the lack of gold-standard data sets and
high context dependencies. In this research project, I aim to devise data sets
and methods to identify media bias. To achieve this, I plan to research methods
using natural language processing and deep learning while employing models and
using analysis concepts from psychology and linguistics. The first results
indicate the effectiveness of an interdisciplinary research approach. My vision
is to devise a system that helps news readers become aware of media coverage
differences caused by bias. So far, my best performing BERT-based model is
pre-trained on a larger corpus consisting of distant labels, indicating that
distant supervision has the potential to become a solution for the difficult
task of bias detection.

negative:
We introduce and describe the results of a novel shared task on bandit
learning for machine translation. The task was organized jointly by Amazon and
Heidelberg University for the first time at the Second Conference on Machine
Translation (WMT 2017). The goal of the task is to encourage research on
learning machine translation from weak user feedback instead of human
references or post-edits. On each of a sequence of rounds, a machine
translation system is required to propose a translation for an input, and
receives a real-valued estimate of the quality of the proposed translation for
learning. This paper describes the shared task's learning and evaluation setup,
using services hosted on Amazon Web Services (AWS), the data and evaluation
metrics, and the results of various machine translation architectures and
learning protocols.