Though visual information has been introduced for enhancing neural machine translation (NMT), its effectiveness strongly relies on the availability of large amounts of bilingual parallel sentence pairs with manual image annotations. In this paper, we present a universal visual representation learned over the monolingual corpora with image annotations, which overcomes the lack of largescale bilingual sentence-image pairs, thereby extending image applicability in NMT. In detail, a group of images with similar topics to the source sentence will be retrieved from a light topic-image lookup table learned over the existing sentence-image pairs, and then is encoded as image representations by a pretrained ResNet. An attention layer with a gated weighting is to fuse the visual information and text information as input to the decoder for predicting target translations. In particular, the proposed method enables the visual information to be integrated into large-scale text-only NMT in addition to the multimodel NMT. Experiments on four widely used translation datasets, including the WMT'16 English-to-Romanian, WMT'14 English-to-German, WMT'14 Englishto-French, and Multi30K, show that the proposed approach achieves significant improvements over strong baselines.
The authors propose to augment NMT with a grounded inventory of images.  The intuition is clear and the premise is very tempting.  The key architectural choice is to allow the transformer to use language embeddings to attend into a topic-image lookup table.  The proportion is learned to balance how much signal comes from each source.    Figure 4, attempts to investigate the importance of this sharing and its effects on performance.
While reviewing this paper I went back and read the EN-DE evaluation data for the last few years trying to see how often I could reason that images would help and I came up severely lacking.  For example, "The old system of private arbitration courts is off the table" from DE-EN 2016 Dev doesn't seem like it should benefit from this architecture.  It's then hard for me to square that with the +VR gains seen throughout this work on non-grounded datasets.  I trust that the authors did in fact achieve these results but I cannot figure out how or why.  This is all further confused by the semantic topics used for clustering the images which ignores stop words and therefore spatial relations or any grammatical nuances.  
In contrast, it does make sense that Multi30K would benefit from this architecture.  As a minor note, were different feature extractors compared? The recent flurry of papers on multimodal transformers indicate that deeper resnet stacks correspond to improved downstream performance.  Is that also true in this domain?