Although attention-based Neural Machine Translation have achieved great success, attention-mechanism cannot capture the entire meaning of the source sentence because the attention mechanism generates a target word depending heavily on the relevant parts of the source sentence. The report of earlier studies has introduced a latent variable to capture the entire meaning of sentence and achieved improvement on attention-based Neural Machine Translation. We follow this approach and we believe that the capturing meaning of sentence benefits from image information because human beings understand the meaning of language not only from textual information but also from perceptual information such as that gained from vision. As described herein, we propose a neural machine translation model that introduces a continuous latent variable containing an underlying semantic extracted from texts and images. Our model, which can be trained endto-end, requires image information only when training. Experiments conducted with an English–German translation task show that our model outperforms over the baseline.
1 INTRODUCTION
Neural machine translation (NMT) has achieved great success in recent years (Sutskever et al., 2014; Bahdanau et al., 2015). In contrast to statistical machine translation, which requires huge phrase and rule tables, NMT requires much less memory. However, the most standard model, NMT with attention (Bahdanau et al., 2015) entails the shortcoming that the attention mechanism cannot capture the entire meaning of a sentence because it generates a target word while depending heavily on the relevant parts of the source sentence (Tu et al., 2016). To overcome this problem, Variational Neural Machine Translation (VNMT), which outperforms NMT with attention introduces a latent variable to capture the underlying semantic from source and target (Zhang et al., 2016). We follow the motivation of VNMT, which is to capture underlying semantic of a source.
Image information is related to language. For example, we human beings understand the meaning of language by linking perceptual information given by the surrounding environment and language (Barsalou, 1999). Although it is natural and easy for humans, it is difficult for computers to understand different domain’s information integrally. Solving this difficult task might, however, bring great improvements in natural language processing. Several researchers have attempted to link language and images such as image captioning by Xu et al. (2015) or image generation from sentences by Reed et al. (2016). They described the possibility of integral understanding of images and text. In machine translation, we can expect an improvement using not only text information but also image information because image information can bridge two languages.
As described herein, we propose the neural machine translation model which introduces a latent variable containing an underlying semantic extracted from texts and images. Our model includes an explicit latent variable z, which has underlying semantics extracted from text and images by introducing a Variational Autoencoder (VAE) (Kingma et al., 2014; Rezende et al., 2014). Our model,
∗First two authors contributed equally.
Green dotted lines denote that π and encoded y are used only when training.
which can be trained end-to-end, requires image information only when training. As described herein, we tackle the task with which one uses a parallel corpus and images in training, while using a source corpus in translating. It is important to define the task in this manner because we rarely have a corresponding image when we want to translate a sentence. During translation, our model generates a semantic variable z from a source, integrates variable z into a decoder of neural machine translation system, and then finally generates the translation. The difference between our model and VNMT is that we use image information in addition to text information.
For experiments, we used Multi30k (Elliott et al., 2016), which includes images and the corresponding parallel corpora of English and German. Our model outperforms the baseline with two evaluation metrics: METEOR (Denkowski & Lavie, 2014) and BLEU (Papineni et al., 2002). Moreover, we obtain some knowledge related to our model and Multi30k. Finally, we present some examples in which our model either improved, or worsened, the result.
Our paper contributes to the neural machine translation research community in three ways.
• We present the first neural machine translation model to introduce a latent variable inferred from image and text information. We also present the first translation task with which one uses a parallel corpus and images in training, while using a source corpus in translating.
• Our translation model can generate more accurate translation by training with images, especially for short sentences.
• We present how the translation of source is changed by adding image information compared to VNMT which does not use image information.
2 BACKGROUND
Our model is the extension of Variational Neural Machine Translation (VNMT) (Zhang et al., 2016). Our model is also viewed as one of the multimodal translation models. In our model, VAE is used to introduce a latent variable. We describe the background of our model in this section.
2.1 VARIATIONAL NEURAL MACHINE TRANSLATION
The VNMT translation model introduces a latent variable. This model’s architecture shown in Figure 1 excludes the arrow from π. This model involves three parts: encoder, inferrer, and decoder. In the encoder, both the source and target are encoded by bidirectional-Recurrent Neural Networks (bidirectional-RNN) and a semantic representation is generated. In the inferrer, a latent variable z is
modeled from a semantic representation by introducing VAE. In the decoder, a latent variable z is integrated in the Gated Recurrent Unit (GRU) decoder; also, a translation is generated.
Our model is followed by architecture, except that the image is also encoded to obtain a latent variable z.
2.2 MULTIMODAL TRANSLATION
Multimodal Translation is the task with which one might one can use a parallel corpus and images. The first papers to study multimodal translation are Elliott et al. (2015) and Hitschler & Riezler (2016). It was selected as a shared task in Workshop of Machine Translation 2016 (WMT161). Although several studies have been conducted (Caglayan et al., 2016; Huang et al., 2016; Calixto et al., 2016; Libovický et al., 2016; Rodrı́guez Guasch & Costa-jussà, 2016; Shah et al., 2016), they do not show great improvement, especially in neural machine translation (Specia et al., 2016). Here, we introduce end-to-end neural network translation models like our model.
Caglayan et al. (2016) integrate an image into an NMT decoder. They simply put source context vectors and image feature vectors extracted from ResNet-50’s ‘res4f relu’ layer (He et al., 2016) into the decoder called multimodal conditional GRU. They demonstrate that their method does not surpass the text-only baseline: NMT with attention.
Huang et al. (2016) integrate an image into a head of source words sequence. They extract prominent objects from the image by Region-based Convolutional Neural Networks (R-CNN) (Girshick, 2015). Objects are then converted to feature vectors by VGG-19 (Simonyan & Zisserman, 2014) and are put into a head of source words sequence. They demonstrate that object extraction by R-CNN contributes greatly to the improvement. This model achieved the highest METEOR score in NMTbased models in WMT16, which we compare to our model in the experiment. We designate this model as CMU.
Caglayan et al. (2016) argue that their proposed model did not achieve improvement because they failed to benefit from both text and images. We assume that they failed to integrate text and images because they simply put images and text into neural machine translation despite huge gap exists between image information and text information. Our model, however, presents the possibility of benefitting from images and text because text and images are projected to their common semantic space so that the gap of images and text would be filled.
2.3 VARIATIONAL AUTO ENCODER
VAE was proposed in an earlier report of the literature Kingma et al. (2014); Rezende et al. (2014). Given an observed variable x, VAE introduces a continuous latent variable z, with the assumption that x is generated from z. VAE incorporates pθ(x|z) and qϕ(z|x) into an end-to-end neural network. The lower bound is shown below.
LVAE = −DKL [qϕ(z|x)||pθ(z)] + Eqϕ(z|x) [log pθ(x|z)] ≤ log pθ(x) (1)
3 NEURAL MACHINE TRANSLATION WITH LATENT SEMANTIC OF IMAGE AND TEXT
We propose a neural machine translation model which explicitly has a latent variable containing an underlying semantic extracted from both text and image. This model can be seen as an extension of VNMT by adding image information.
Our model can be drawn as a graphical model in Figure 3. Its lower bound is
L = −DKL [qϕ(z|x,y,π)||pθ(z|x)] + Eqϕ(z|x,y,π) [log pθ(y|z,x)] , (2)
where x,y,π, z respectively denote the source, target, image and latent variable, and pθ and qϕ respectively denote the prior distribution and the approximate posterior distribution. It is noteworthy in Eq. (2) that we want to model p(z|x,y,π), which is intractable. Therefore we model qϕ(z|x,y,π)
1http://www.statmt.org/wmt16/
z
x y
z
x y
Figure 2: VNMT
z
x y
z
x y
π
Figure 3: Our model
instead, and also model prior pθ(z|x) so that we can generate a translation from the source in testing. Derivation of the formula is presented in the appendix.
We model all distributions in Eq. (2) by neural networks. Our model architecture is divisible into three parts: 1) encoder, 2) inferrer, and 3) decoder.
3.1 ENCODER
In the encoder, the semantic representation he is obtained from the image, source, and target. We propose several methods to encode an image. We show how these methods affect the translation result in the Experiment section. This representation is used in the inferrer. This section links to the green part of Figure 1.
3.1.1 TEXT ENCODING
The source and target are encoded in the same way as Bahdanau et al. (2015). The source is converted to a sequence of 1-of-k vector and is embedded to demb dimensions. We designate it as the source sequence. Then, a source sequence is put into bidirectional RNN. Representation hi is obtained by concatenating h⃗i and ⃗hi : h⃗i = RNN(h⃗i−1, Ewi), ⃗hi = RNN( ⃗hi+1, Ewi),hi = [h⃗i; ⃗hi], where Ewi is the embedded word in a source sentence, hi ∈ Rdh , and h⃗i, ⃗hi ∈ R dh 2 . It is conducted through i = 0 to i = Tf , where Tf is the sequence length. GRU is implemented in bidirectional RNN so that it can attain long-term dependence. Finally, we conduct mean-pooling to hi and obtain the source representation vector as hf = 1Tf ∑Tf i hi. The exact same process is applied to target to obtain target representation hg .
3.1.2 IMAGE ENCODING AND SEMANTIC REPRESENTATION
We use Convolutional Neural Networks (CNN) to extract feature vectors from images. We propose several ways of extracting image features.
Global (G) The image feature vector is extracted from the image using a CNN. With this method, we use a feature vector in the certain layer as π. Then π is encoded to the image representation vector hπ simply by affine transformation as
hπ = Wππ + bπ where Wπ ∈ Rdπ×dfc7 , bπ ∈ Rdπ . (3)
Global and Objects (G+O) First we extract some prominent objects from images in some way. Then, we obtain fc7 image feature vectors π from the original image and extracted objects using a CNN. Therefore π takes a variable length. We handle π in two ways: average and RNN encoder. In average (G+O-AVG), we first obtain intermediate image representation vector h′π by affine transformation in Eq. (3). Then, the average of h′π becomes the image representation
vector: hπ = ∑l i h ′ πi l , where l is the length of h ′ π .
In RNN encoder (G+O-RNN), we first obtain h′π by affine transformation in Eq. (3). Then, we encode h′π in the same way as we encode text in Section 3.1.1 to obtain hπ .
Global and Objects into source and target (G+O-TXT) Thereby, we first obtain h′π by affine transformation in Eq. (3). Then, we put sequential vector h′π into the head of the source sequence and target sequence. In this case, we set dπ to be the same dimension as demb. In fact, the source sequence including h′π is only used to model qϕ(z|x,y,π). Context vector c (Eq. (15)) and pθ(z|x) are computed by a source sequence that does not include h′π . We encode the source sequence including h′π as Section 3.1.1 to obtain hf and hg . In this case, hπ is not obtained. Image information is contained in hf and hg .
All representation vectors hf , hg and hπ are concatenated to obtain a semantic representation vector as he = [hf ;hg;hπ], where he ∈ Rde=2×dh+dπ (in G+O-TXT: he = [hf ;hg], where he ∈ Rde=2×dh ). It is an input of the multimodal variational neural inferrer.
3.2 INFERRER
We model the posterior qϕ(z|x,y,π) using a neural network and also the prior pθ(z|x) by neural network. This section links to the black and grey part of Figure 1.
3.2.1 NEURAL POSTERIOR APPROXIMATOR
Modeling the true posterior pθ(z|x,y,π) is usually intractable. Therefore, we consider modeling of an approximate posterior qϕ(z|x,y,π) by introducing VAE. We assume that the posterior qϕ(z|x,y,π) has the following form:
qϕ(z|x,y,π) = N (z;µ(x,y,π),σ(x,y,π)2I). (4)
The mean µ and standard deviation σ of the approximate posterior are the outputs of neural networks.
Starting from the variational neural encoder, a semantic representation vector he is projected to latent semantic space as
hz = g(W (1) z he + b (1) z ), (5)
where W (1)z ∈ Rdz×(de) b(1)z ∈ Rdz . g(·) is an element-wise activation function, which we set as tanh(·). Gaussian parameters of Eq. (4) are obtained through linear regression as
µ = Wµhz + bµ, logσ 2 = Wσhz + bσ, (6)
where µ, logσ2 ∈ Rdz .
3.2.2 NEURAL PRIOR MODEL
We model the prior distribution pθ(z|x) as follows:
pθ(z|x) = N (z;µ′(x),σ′(x) 2 I).
(7)
µ′ and σ′ are generated in the same way as that presented in Section 3.2.1, except for the absence of y and π as inputs. Because of the absence of representation vectors, the dimensions of weight in equation (5) for prior model are W ′(1) z ∈ Rdz×dh , b ′(1) z ∈ Rdz . We use a reparameterization trick to obtain a representation of latent variable z: h′z = µ+σϵ, ϵ ∼ N (0, I). During translation, h′z is set as the mean of pθ(z|x). Then, h′z is projected onto the target space as
h′e = g(W (2) z h ′ z + b (2) z ) where h ′ e ∈ Rde . (8)
h′e is then integrated into the neural machine translation’s decoder.
3.3 DECODER
This section links to the orange part of Figure 1. Given the source sentence x and the latent variable z, decoder defines the probability over translation y as
p(y|z,x) = T∏
j=1
p(yj |y<j , z,x). (9)
How we define the probability over translation y is fundamentally the same as VNMT, except for using conditional GRU instead of GRU. Conditional GRU involves two GRUs and an attention mechanism. We integrate a latent variable z into the second GRU. We describe it in the appendix.
3.4 MODEL TRAINING
Monte Carlo sampling method is used to approximate the expectation over the posterior Eq. (2), Eqϕ(z|x,y,π) ≈ 1L ∑L l=1 log pθ(y|x,h (l) z ), where L is the number of samplings. The training objective is defined as
L(θ, ϕ) = −DKL [qϕ(z|x,y,π)||pθ(z|x)] + 1
L L∑ l=1 T∑ j=1 log pθ(yj |y<j ,x,h(l)z ), (10)
where hz = µ+ σ · ϵ, ϵ ∼ N (0, I). The first term, KL divergence, can be computed analytically and is differentiable because both distributions are Gaussian. The second term is also differentiable. We set L as 1. Overall, the objective L is differentiable. Therefore, we can optimize the parameter θ and variational parameter ϕ using gradient ascent techniques.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
We used Multi30k (Elliott et al., 2016) as the dataset. Multi30k have an English description and a German description for each corresponding image. We handle 29,000 pairs as training data, 1,014 pairs as validation data, and 1,000 pairs as test data.
Before training, punctuation normalization and lowercase are applied to both English and German sentences by Moses (Koehn et al., 2007) scripts2. Compound-word splitting is conducted only to German sentences using Sennrich et al. (2016)3. Then we tokenize sentences2 and use them as training data. We produce vocabulary dictionaries from training data. The vocabulary becomes 10,211 words for English and 13,180 words for German after compound-word splitting.
Image features are extracted using VGG-19 CNN (Simonyan & Zisserman, 2014). We use 4096- dimensional fc7 features. To extract the object’s region, we use Fast R-CNN (Girshick, 2015). Fast R-CNN is trained on ImageNet and MSCOCO dataset 4.
All weights are initialized byN (0, 0.01I). We use the adadelta algorithm as an optimization method. The hyperparameters used in the experiment are presented in the Appendix. All models are trained with early stopping. When training, VNMT is fine-tuned by NMT model and our models are finetuned using VNMT. When translating, we use beam-search. The beam-size is set as 12. Before evaluation, we restore split words to the original state and de-tokenize2 generated sentences.
We implemented proposed models based on dl4mt5. Actually, dl4mt is fundamentally the same model as Bahdanau et al. (2015), except that its decoder employs conditional GRU6. We implemented VNMT also with conditional GRU so small difference exists between our implementation
2https://github.com/moses-smt/mosesdecoder/blob/master/scripts/tokenizer/{normalize-punctuation, lowercase, tokenizer, detokenizer}.perl
3https://github.com/rsennrich/subword-nmt 4https://github.com/rbgirshick/fast-rcnn/tree/coco 5https://github.com/nyu-dl/dl4mt-tutorial 6The architecture is described at https://github.com/nyu-dl/dl4mt-tutorial/blob/master/docs/cgru.pdf
and originally proposed VNMT which employs normal GRU as a decoder. We evaluated results based on METEOR and BLUE using MultEval 7.
4.2 RESULT
Table 1 presents experiment results. It shows that our models outperforms the baseline in both METEOR and BLEU. Figure 4 shows the plot of METEOR score of baselines and our models models in validation. Figure 5 shows the plot of METEOR score and the source sentence length.
4.3 QUANTITATIVE ANALYSIS
Table 1 shows that G scores the best in proposed models. In G, we simply put the feature of the original image. Actually, proposed model does not benefit from R-CNN, presumably because we can not handle sequences of image features very well. For example, G+O-AVG uses the average of multiple image features, but it only makes the original image information unnecessarily confusing.
Figure 4 shows that G and G+O-AVG outperforms VNMT almost every time, but all model scores increase suddenly in the 17,000 iteration validation. We have no explanation for this behavior. Figure 4 also shows that G and G+O-AVG scores fluctuate more moderately than others. We state that G and G+O-AVG gain stability by adding image information. When one observes the difference between the test score and the validation score for each model, baseline scores decrease more than proposed model scores. Especially, the G score increases in the test, simply because proposed models produce a better METEOR score on average, as shown in Figure 4.
Figure 5 shows that G and G+O-AVG make more improvements on baselines in short sentences than in long sentences, presumably because qϕ(z|x,y,π) can model z well when a sentence is short. Image features always have the same dimension, but underlying semantics of the image and text differ. We infer that when the sentence is short, image feature representation can afford to approximate the underlying semantic, but when a sentence is long, image feature representation can not approximate the underlying semantic.
Multi30k easily becomes overfitted, as shown in Figure 8 and 9 in the appendix. This is presumably because 1) Multi30k is the descriptions of image, making the sentences short and simple, and 2) Multi30k has 29,000 sentences, which could be insufficient. In the appendix, we show how the parameter setting affects the score. One can see that decay-c has a strong effect. Huang et al. (2016) states that their proposed model outperforms the baseline (NMT), but we do not have that observation. It can be assumed that their baseline parameters are not well tuned.
4.4 QUALITATIVE ANALYSIS
We presented the top 30 sentences, which make the largest METEOR score difference between G and VNMT, to native speakers of German and get the overall comments. They were not informed of
7https://github.com/jhclark/multeval, we use meteor1.5 instead of meteor1.4, which is the default of MultEval.
our model training with image in addition to text. These comments are summarized into two general remarks. One is that G translates the meaning of the source material more accurately than VNMT. The other is that our model has more grammatical errors as prepositions’ mistakes or missing verbs compared to VNMT. We assume these two remarks are reasonable because G is trained with images which mainly have a representation of noun rather than verb, therefore can capture the meaning of materials in sentence.
Figure 6 presents the translation results and the corresponding image which G translates more accurately than VNMT in METEOR. Figure 7 presents the translation results and the corresponding image which G translates less accurately than VNMT in METEOR. Again, we note that our model does not use image during translating. In Figure 6, G translates ”a white and black dog” correctly while VNMT translates it incorrectly implying ”a white dog and a black dog”. We assume that G correctly translates the source because G captures the meaning of material in the source. In Figure 7, G incorrectly translates the source. Its translation result is missing the preposition meaning ”at”, which is hardly represented in image.We present more translation examples in appendix.
5 CONCLUSION
As described herein, we proposed the neural machine translation model that explicitly has a latent variable that includes underlying semantics extracted from both text and images. Our model outperforms the baseline in both METEOR and BLEU scores. Experiments and analysis present that our model can generate more accurate translation for short sentences. In qualitative analysis, we present that our model can translate nouns accurately while our model make grammatical errors.
A DERIVATION OF LOWER BOUNDS
The lower bound of our model can be derived as follows: p(y|x) = ∫ p(y, z|x)dz
= ∫ p(z|x)p(y|z,x)dz
log p(y|x) = log ∫
q(z|x,y,π)p(z|x)p(y|z,x) q(z|x,y,π) dz
≥ ∫
q(z|x,y,π) log p(z|x)p(y|z,x) q(z|x,y,π) dz
= ∫ q(z|x,y,π) ( log
p(z|x) q(z|x,y)
+ log p(y|z,x) ) dz
= −DKL [q(z|x,y,π)||p(z|x)] + Eq(z|x,y,π) [log p(y|z,x)] = L
B CONDITIONAL GRU
Conditional GRU is implemented in dl4mt. Caglayan et al. (2016) extends Conditional GRU to make it capable of receiving image information as input. The first GRU computes intermediate representation s′j as
s′j = (1− o′j)⊙ s′j + o′j ⊙ sj−1 (11) s′j = tanh(W ′E [yj−1] + r ′ j ⊙ (U ′sj−1)) (12)
r′j = σ(W ′ rE [yj−1] + U ′ rsj−1) (13) o′j = σ(W ′ oE [yj−1] + U ′ osj−1) (14)
where E ∈ Rdemb×dt signifies the target word embedding, s′j ∈ Rdh denotes the hidden state, r′j ∈ Rdh and o′j ∈ Rdh respectively represent the reset and update gate activations. dt stands for the dimension of target; the unique number of target words. [W ′,W ′r,W ′ o] ∈ Rdh×demb , [U ′, U ′r, U ′o] ∈ Rdh×dh are the parameters to be learned.
Context vector cj is obtained as
cj = tanh  Tf∑ i=1 αijhi  (15) αij =
exp(eij)∑Tf k=1 exp(ekj)
(16)
eij = Uatttanh(Wcatthi +Watts ′ j) (17)
where [Uatt,Wcatt,Watt] ∈ Rdh×dh are the parameters to be learned. The second GRU computes sj from s′j , cj and h ′ e as
sj = (1− o′j)⊙ sj + oj ⊙ s′j (18) sj = tanh(Wcj + rj ⊙ (Us′j) + V h′e) (19)
rj = σ(Wrcj + Urs ′ j + Vrh ′ e) (20) oj = σ(Wocj + Uos ′ j + Voh ′ e) (21)
where sj ∈ Rdh stands for the hidden state, rj ∈ Rdh and oj ∈ Rdh are the reset and update gate activations. [W,Wr,Wo] ∈ Rdh×dh , [U,Ur, Uo] ∈ Rdh×dh , [V, Vr, Vo] ∈ Rdh×dz are the
parameters to be learned. We introduce h′e obtained from a latent variable here so that a latent variable can affect the representation sj through GRU units.
Finally, the probability of y is computed as
uj = Lutanh(E [yj−1] + Lssj + Lxcj) (22) P (yj |yj−1, sj , cj) = Softmax(uj) (23)
where Lu ∈ Rdt×demb , Ls ∈ Rdemb×dh and Lc ∈ Rdemb×dh are the parameters to be learned.
C TRAINING DETAIL
C.1 HYPERPARAMETERS
Table 2 presents parameters that we use in the experiments.
We found that Multi30k dataset is easy to overfit. Figure 8 and Figure 9 present training cost and validation METEOR score graph of the two experimental settings of the NMT model. Table 3 presents the hyperparameters which were used in the experiments. Large decay-c ans small batchsize give the better METEOR scores in the end. Training is stopped if there is no validation cost improvements over the last 10 validations.
0 10000 20000 30000 40000 50000 60000 70000 iteration
0
20
40
60
80
100
co st
Training cost
1 2
Figure 8: NMT Training Cost 0 10 20 30 40 50 60 70 iteration (x 1000)
0
10
20
30
40
50
60
M E T E O R
Validation METEOR
1 2
Figure 9: NMT Validation METEOR score
Figure 10 presents the English word length histogram of the Multi30k test dataset. Most sentences in the Multi30k are less than 20 words. We assume that this is one of the reasons why Multi30k is easy to overfit.
C.2 COST GRAPH
Figure 11 and 12 present the training cost and validation cost graph of each models. Please note that VNMT fine-tuned NMT, and other models fine-tuned VNMT.
C.3 TRANSLATION EXAMPLES
We present some selected translations from VNMT and our proposed model (G). As of translation 3 to 5 our model give the better METEOR scores than VNMT and as of translation 6 to 8 VNMT give the better METEOR scores than our models.
I have problems understanding the motivation of this paper. The authors claimed to have captured a latent representation of text and image during training and can translate better without images at test time, but didn't demonstrate convincingly that images help (not to mention the setup is a bit strange when there are no images at test time). What I see are only speculative comments: "we observed some gains, so these should come from our image models". The qualitative analysis doesn't convince me that the models have learned latent representations; I am guessing the gains are due to less overfitting because of the participation of images during training. 
The dataset is too small to experiment with NMT. I'm not sure if it's fair to compare their models with NMT and VNMT given the following description in Section 4.1 "VNMT is fine-tuned by NMT and our models are fine-tuned with VNMT". There should be more explanation on this.
Besides, I have problems with the presentation of this paper.
(a) There are many symbols being used unnecessary. For example: f & g are used for x (source) and y (target) in Section 3.1. 
(b) The ' symbol is not being used in a consistent manner, making it sometimes hard to follow the paper. For example, in section 3.1.2, there are references about h'_\pi obtained from Eq. (3) which is about h_\pi (yes, I understand what the authors mean, but there can be better ways to present that).
(c) I'm not sure if it's correct in Section 3.2.2 h'_z is computed from \mu and \sigma. So how \mu' and \sigma' are being used ?
(d) G+O-AVG should be something like G+O_{AVG}. The minus sign makes it looks like there's an ablation test there. Similarly for other symbols.
Other things: no explanations for Figure 2 & 3. There's a missing \pi symbol in Appendix A before the KL derivation.