  Text revision refers to a family of natural language generation tasks, where
the source and target sequences share moderate resemblance in surface form but
differentiate in attributes, such as text formality and simplicity. Current
state-of-the-art methods formulate these tasks as sequence-to-sequence learning
problems, which rely on large-scale parallel training corpus. In this paper, we
present an iterative in-place editing approach for text revision, which
requires no parallel data. In this approach, we simply fine-tune a pre-trained
Transformer with masked language modeling and attribute classification. During
inference, the editing at each iteration is realized by two-step span
replacement. At the first step, the distributed representation of the text
optimizes on the fly towards an attribute function. At the second step, a text
span is masked and another new one is proposed conditioned on the optimized
representation. The empirical experiments on two typical and important text
revision tasks, text formalization and text simplification, show the
effectiveness of our approach. It achieves competitive and even better
performance than state-of-the-art supervised methods on text simplification,
and gains better performance than strong unsupervised methods on text
formalization \footnote{Code and model are available at
\url{https://github.com/jingjingli01/OREO}}.

Computation and Language