  Most neural machine translation systems still translate sentences in
isolation. To make further progress, a promising line of research additionally
considers the surrounding context in order to provide the model potentially
missing source-side information, as well as to maintain a coherent output. One
difficulty in training such larger-context (i.e. document-level) machine
translation systems is that context may be missing from many parallel examples.
To circumvent this issue, two-stage approaches, in which sentence-level
translations are post-edited in context, have recently been proposed. In this
paper, we instead consider the viability of filling in the missing context. In
particular, we consider three distinct approaches to generate the missing
context: using random contexts, applying a copy heuristic or generating it with
a language model. In particular, the copy heuristic significantly helps with
lexical coherence, while using completely random contexts hurts performance on
many long-distance linguistic phenomena. We also validate the usefulness of
tagged back-translation. In addition to improving BLEU scores as expected,
using back-translated data helps larger-context machine translation systems to
better capture long-range phenomena.

Computation and Language