There are concerns that neural language models may preserve some of the stereotypes of the underlying societies that generate the large corpora needed to train these models. For example, gender bias is a significant problem when generating text, and its unintended memorization could impact the user experience of many applications (e.g., the smart-compose feature in Gmail). In this paper, we introduce a novel architecture that decouples the representation learning of a neural model from its memory management role. This architecture allows us to update a memory module with an equal ratio across gender types addressing biased correlations directly in the latent space. We experimentally show that our approach can mitigate the gender bias amplification in the automatic generation of articles news while providing similar perplexity values when extending the Sequence2Sequence architecture.
This paper focuses on the timely and important topic of unintended social bias in language models and proposes an approach to mitigate gender bias in the automatic generation of news articles. The automatic text generation is done by a sequence-to-sequence-model using word embeddings. The pattern of these embeddings in an multi-dimensional space can be associated with the stereotypes found in the corpora the models are trained on. To mitigate this bias the authors combine the network with a memory module storing additional gender information. This approach is inspired by the memory module used by Kaiser et al. (2017) to remember rare events, replacing the vector to track the age of items with a key representing the gender. 
The proposed network architecture ("Seq2Seq+FairRegion") is evaluated using the bias score presented by Zhao et al. (2017) and compared to the sequence-to-sequende-models "Seq2Seq2" by Sutskever, Vinyals and Lee (2014) and "Seq2Seq+Attention" by Bahdanau, Cho and Bengio (2015).
Besides some typographical errors ("work embedding") the paper is overall well written but unfortunately the flow of the text could be improved to make it easier to follow. The authors should perhaps consider to explain the structure of the paper in the introduction. There are some errors according the brackets of the in-text citations which should be corrected. 
The two figures both illustrate the memory module and could be easily combined to one because the second figure just adds the encoding stage of the architecture. The caption of figure 1 could be used to further explain the "Fair Region" in the text (section 2). In section 4 the authors explain how to compute the bias score. Because these calculations are used to evaluate their methods, this section should perhaps be a subsection to section 5 ("experiments").
To evaluate the proposed method the authors use newspaper articles from Chile, Peru and Mexiko and compare the bias amplification metric for the trained models. The results are shown for the datasets from Peru and Mexico (Chile is missing) with a larger bias amplification for the Peru dataset. It could be justified more clearly why the experiments compare the bias amplification scores of different countries while the main focus of the paper is to mitigate gender bias in general.
While this is an important topic, the contribution is rather minor and not well presented. Specifically the experiements should be extended.