Maximilian Nickel Facebook AI Research maxn@fb.com We investigate grounded sentence representations, where we train a sentence encoder to predict the image features of a given caption i.e., we try to imagine how a sentence would be depicted visuallyand use the resultant features as sentence representations. We examine the quality of the learned representations on a variety of standard sentence representation quality benchmarks, showing improved performance for grounded models over non-grounded ones. In addition, we thoroughly analyze the extent to which grounding contributes to improved performance, and show that the system also learns improved word embeddings. Following the word embedding upheaval of the past few years, one of NLP's next big challenges has become the hunt for universal sentence representations: generic representations of sentence meaning that can be plugged into any kind of system or pipeline. Examples include Paragraph2Vec (Le and Mikolov, 2014), C-Phrase (Pham et al., 2015), SkipThought (Kiros et al., 2015) and Fast-Sent (Hill et al., 2016a). These representations tend to be learned from large corpora in an unsupervised setting, much like word embeddings, and eectively transferred to the task at hand. Purely text-based semantic models, which represent word meaning as a distribution over other words (Harris, 1954; Turney and Pantel, 2010; Clark, 2015), suer from the grounding problem (Harnad, 1990). It has been shown that grounding leads to improved performance on a variety of word-level tasks (Baroni, 2016; Kiela, 2017). Unsupervised sentence representation models are often doubly exposed to the grounding problem, especially if they represent sentence mean-1 Work done while at Facebook AI Research. ings as a distribution over other sentences, as in SkipThought (Kiros et al., 2015). Here, we examine whether grounding also leads to improved sentence representations. In short, the grounding problem is characterized by the lack of an association between symbols and external information. We address this problem by aligning text with paired visual data and hypothesize that sentence representations can be enriched with external informationi.e., groundedby forcing them to capture visual semantics. We investigate the performance of these representations and the eect of grounding on a variety of semantic benchmarks. There has been much recent interest in generating actual images from text (Goodfellow et al., 2014; van den Oord et al., 2016; Mansimov et al., 2016). Our method takes a slightly dierent approach: instead of predicting actual images, we train a deep recurrent neural network to predict the latent feature representation of images. That is, we are specifically interested in the semantic content of visual representations and how useful that information is for learning sentence representations. One can think of this as trying to imagine, or form a mental picture, of a sentence's meaning (Chrupaa et al., 2015). Much like a sentence's meaning in classical semantics is given by its model-theoretic ground truth (Tarski, 1944), our ground truth is provided by images. Grounding is likely to be more useful for concrete words and sentences: a sentence such as democracy is a political system does not yield any coherent mental picture. In order to accommodate the fact that much of language is abstract, we take sentence representations obtained using text-only data (which are better for representing abstract meaning) and combine them with the grounded representations that our system learns (which are good for representing concrete meaning), leading to multi-modal sentence representations. In what follows, we introduce a system for grounding sentence representations by learning to predict visual content. Although it is not the primary aim of this work, it is important to first examine how well this system achieves what it is trained to do, by evaluating on the COCO5K image and caption retrieval task. We then analyze the performance of grounded representations on a variety of sentence-level semantic transfer tasks, showing that grounding increases performance over text-only representations. We then investigate an important open question in multi-modal semantics: to what extent are improvements in semantic performance due to grounding, rather than to having more data or data from a dierent distribution? In the remainder, we analyze the role that concreteness plays in representation quality and show that our system learns grounded word embedding projections that outperform non-grounded ones. To the best of our knowledge, this is the first work to comprehensively study grounding for distributed sentence representations on such a wide set of semantic benchmark tasks. Sentence representations Although there appears to be a consensus with regard to the methodology for learning word representations, this is much more of an open problem for sentence representations. Recent work has ranged from trying to learn to compose word embeddings (Le and Mikolov, 2014; Pham et al., 2015; Wieting et al., 2016; Arora et al., 2017), to neural architectures for predicting the previous and next sentences (Kiros et al., 2015) or learning representations via large-scale supervised tasks (Conneau et al., 2017). In particular, SkipThought (Kiros et al., 2015) led to an increased interest in learning sentence representations. Hill et al. (2016a) compare a wide selection of unsupervised and supervised methods, including a basic caption prediction system that is similar to ours. That study finds that dier-ent learning methods are preferable for dierent intended applications, i.e., that the matter of optimal universal sentence representations is as of yet far from decided. InferSent (Conneau et al., 2017) recently showed that supervised sentence representations can be of very high quality. Here, we learn grounded sentence representations in a supervised setting, combine them with standard unsupervised sentence representations, and show how grounding can help for a variety of sentence-level tasks. Multi-modal semantics Language grounding in semantics has been motivated by evidence that human meaning representations are grounded in perceptual experience (Jones et al., 1991; Perfetti, 1998; Andrews et al., 2009; Riordan and Jones, 2011). That is, despite ample evidence of humans representing meaning with respect to an external environment and sensorimotor experience (Barsalou, 2008; Louwerse, 2008), standard semantic models rely solely on textual data. This gives rise to an infinite regress in text-only semantic representations, i.e., words are defined in terms of other words, ad infinitum . The field of multi-modal semantics, which aims to address this issue by enriching textual representations with information from other modalities, has mostly been concerned with word representations (Bruni et al., 2014; Baroni, 2016; Kiela, 2017, and references therein). Learning multi-modal representations that ground text-only representations has been shown to improve performance on a variety of core NLP tasks. This work is most closely related to that of Chrupaa et al. (2015), who also aim to ground language by relating images to captions: here, we additionally address abstract sentence meaning; have a dierent architecture, loss function and fusion strategy; and explicitly focus on grounded universal sentence representations. Bridging vision and language There is a large body of work that involves jointly embedding images and text, at the word level (Frome et al., 2013; Joulin et al., 2016), the phrase level (Karpathy et al., 2014; Li et al., 2016), and the sentence level (Karpathy and Fei-Fei, 2015; Klein et al., 2015; Kiros et al., 2015; Chen and Zitnick, 2015; Reed et al., 2016). Our model similarly learns to map sentence representations to be consistent with a visual semantic space, and we focus on studying how these grounded text representations transfer to NLP tasks. Moreover, there has been a lot of work in recent years on the task of image caption generation (Bernardi et al., 2016; Vinyals et al., 2015; Mao et al., 2015; Fang et al., 2015). Here, we do the opposite: we predict the correct image (features) from the caption, rather than the caption from the image (features). Similar ideas were recently successfully applied to multi-modal machine translation 409 (Elliott and Kdr, 2017; Gella et al., 2017; Lee et al., 2017). Recently, Das et al. (2017) trained dialogue agents to communicate about images, trying to predict image features as well. In the following, let D = {( I k , C k )} Nk = 1 be a dataset where each image I k is associated with one or more captions C k = { C 1 , . . ., C |C| k } . A prominent example of such a dataset is COCO (Lin et al., 2014), which consists of images with up to 5 corresponding captions for each image. The objective of our approach is to encode a given sentence, i.e., a caption C , and learn to ground it in the corresponding image I . To encode the sentence, we train a bidirectional LSTM (BiLSTM) on the caption, where the input is a sequence of projected word embeddings. We combine the final left-to-right and right-to-left hidden states of the LSTM and take the element-wise maximum to obtain a sentence encoding. We then examine three distinct methods for grounding the sentence encoding. In the first method, we try to predict the image features (Cap2Img). That is, we learn to map the caption to the same space as the image features that represent the correct image. We call this strong perceptual grounding, where we take the visual input directly into account. An alternative method is to exploit the fact that one image in COCO has multiple captions (Cap2Cap), and to learn to predict which other captions are valid descriptions of the same image. This approach is strictly speaking not perceptually grounded, but exploits the fact that there is an implicit association between the captions and the shared underlying image, and so could be considered a weaker version of grounding. Finally, we experiment with a model that optimizes both these objectives jointly: that is, we predict both images and alternative captions for the same image (Cap2Both). Thus, Cap2Both incorporates both strong perceptual and weak implicit grounding. Please see Figure 1 for an illustration of the various models. In what follows, we discuss them in more technical detail. To learn sentence representations, we employ a bidirectional LSTM architecture. In particular, let x = ( x 1 , . . ., x T ) be an input sequence where each word is represented via an embedding x t  R n . Using a standard LSTM (Hochreiter and Schmid-huber, 1997), the hidden state at time t , denoted h t  R m , is computed via h t + 1 , c t + 1 = LSTM ( x t , h t , c t |  ) where c t denotes the cell state of the LSTM and where  denotes its parameters. To exploit contextual information in both input directions, we process input sentences using a bidirectional LSTM, that reads an input sequence in both normal and reverse order. In particular, for an input sequence x of length T , we compute the hidden state at time t , h t  R 2 m via h ft + 1 = LSTM ( x t , h ft , c ft |  f ) h bt + 1 = LSTM ( x T  t , h bt , c bt |  b ) Here, the two LSTMs process x in a forward and a backward order, respectively. We subsequently use max : R d  R d  R d to combine them into their element-wise maximum, yielding the representation of a caption after it has been processed with the BiLSTM: h T = max ( h ft , h bt ) We use GloVe vectors (Pennington et al., 2014) for our word embeddings. The embeddings are kept fixed during training, which allows a trained sentence encoder to transfer to tasks (and a vocabulary) that it has not yet seen, provided GloVe embeddings are available. Since GloVe representations are not tuned to represent grounded information, we learn a global transformation of GloVe space to grounded word space. Specifically, let x  R n be the original GloVe embeddings. We then learn a linear map U  R n  n such that x = U x and use x as input to the BiLSTM. The linear map U and the BiLSTM are trained jointly. Let v  RI be the latent representation of an image (e.g.the final layer of a ResNet). To ground captions in the images that they describe, we map h T into the latent space of image representations such that their similarity is maximized. In other words, we aim to predict the latent features of an image from its caption. The mapping of caption to image space is performed via a series of projections p 0 = h T p  + 1 =  ( P  p  ) 410 Funny cat sitting on laptop max( , ) Cute kitten laying on keyboard C a p 2 I m g C a p 2 C a p Figure 1: Model architecture: predicting either an image (Cap2Img), an alternative caption (Cap2Cap), or both at the same time (Cap2Both). where  denotes a non-linearity such as ReLUs or tanh. By jointly training the BiLSTM with these latent projections, we can then ground the language model in its visual counterpart. In particular, let  =  BiLSTM  { P  } L  = 1 be the parameters of the BiLSTM as well as the projection layers. We then minimize the following ranking loss: L C2I (  ) = (cid:213) ( I , C )D f rank ( I , C ) + f rank ( C , I ) (1) where f rank ( a , b ) = (cid:213) b 0 N a [   sim ( a , b ) + sim ( a , b 0 )] + where [ x ] + = max ( 0 , x ) denotes the threshold function at zero and  defines the margin. Furthermore, N a denotes the set of negative samples for an image or caption and sim ( , ) denotes a similarity measure between vectors. In the following, we employ the cosine similarity, i.e., sim ( a , b ) = h a , b i k a kk b k . Although this loss is not smooth at zero, it can be trained end-to-end using subgradient methods. Compared to e.g. an l 2 regression loss, Equation (1) is less susceptible to error incurred by subspaces of the visual representation that are irrelevant to the high level visual semantics. Empirically, we found it to be more robust to overfitting. 3.3 Cap2Cap Let x = ( x 1 , . . ., x T ) , y = ( y 1 , . . ., y S ) be a caption pair that describes the same image. To learn weakly grounded representations, we employ a standard sequence-to-sequence model (Sutskever et al., 2014), whose task is to predict y from x . As in the Cap2Cap model, let h T be the representation of the input sentence after it has been processed with a BiLSTM. We then model the joint probability of y given x as p ( y | x ) = S (cid:214) s = 1 p ( y s | h T , y 1 , . . ., y s  1 ,  ) . To model the conditional probability of y s we use the usual multiclass classification approach over the vocabulary of the corpus V such that p ( y s = k | h T , y 1 , . . ., y s  1 ,  ) = e h v k , y s i  |V| j = 1 e h v j , y s i . Here, y s =  ( WV g s + b ) and g s is hidden state of the decoder LSTM at time s . To learn the model parameters, we minimize the negative log-likelihood over all caption pairs, i.e., L C2C (  ) =  (cid:213) x , y D | y | (cid:213) s = 1 log p ( y s | h T , y 1 , . . ., y s  1 ,  ) . 3.4 Cap2Both Finally, we also integrate both concepts of grounding into a joint model, where we optimize the following loss function: L C2B (  ) = LC 2 I (  ) + LC 2 C (  ) . 411 3.5 Grounded universal representations On their own, features from this system are likely to suer from the fact that training on COCO introduces biases: aside from the inherent dataset bias in COCO itself, the system will only have coverage for concrete concepts. COCO is also a much smaller dataset than e.g. the Toronto Books Corpus often used in purely text-based methods (Kiros et al., 2015). As such, grounded representations are potentially less universal than text-based alternatives, which also cover abstract concepts. There is evidence that meaning is dually coded in the human brain: while abstract concepts are processed in linguistic areas, concrete concepts are processed in both linguistic and visual areas (Paivio, 1990). Anderson et al. (2017) recently corroborated this hypothesis using semantic representations and fMRI studies. In our case, we want to be able to accommodate concrete sentence meanings, for which our vision-centric system is likely to help; as well as abstract sentence meanings, where trying to imagine what democracy is a political system might look like will probably only introduce noise. Hence, we optionally complement our systems' representations with more abstract universal sentence representations trained on language-only data (specifically, the Toronto Books Corpus). Although it would be interesting to examine multitask scenarios where these representations are jointly learned, we leave this for future work. Here, instead, we combine grounded and language-only representations using simple concatenation, i.e., r gs = r grounded || r ling  only . Concatenation has been proven to be a strong and straightforward mid-level multi-modal fusion method, previously explored in multi-modal semantics for word representations (Bruni et al., 2014; Kiela and Bot-tou, 2014). We call the combined system GroundSent (GS), and distinguish between sentences perceptually grounded in images (GroundSent-Img), weakly grounded in captions (GroundSent-Cap) or grounded in both (GroundSent-Both). We use 300-dimensional GloVe (Pennington et al., 2014) embeddings, trained on WebCrawl, for the initial word representations and optimize using Adam (Kingma and Ba, 2015). We use ELU (Clev-ert et al., 2016) for the non-linearity in projection layers, set dropout to 0.5 and use a dimensionality of 1024 for the LSTM. The network was initialized with orthogonal matrices for the recurrent layers (Saxe et al., 2014) and He initialization (He et al., 2015) for all other layers. The learning rate and margin were tuned on the validation set using grid search. We use the same COCO splits as Karpathy and Fei-Fei (2015) for training (113,287 images), validation (5000 images) and testing (5000 images). Image features for COCO were obtained by transferring the final layer from a ResNet-101 (He et al., 2016) trained on ImageNet (ILSVRC 2015). We are specifically interested in how well (grounded) universal sentence representations transfer to dierent tasks. To evaluate this, we perform experiments for a variety of tasks. In all cases, we compare against layer-normalized SkipThought vectors, a well-known high-performing sentence encoding method (Ba et al., 2016). To ensure that we use the exact same evaluations, with identical hyperparameters and settings, we evaluate all systems with the same evaluation pipeline, namely SentEval (Conneau and Kiela, 2018) 2 . Following previous work in the field, the idea is to take universal sentence representations and to learn a simple classifier on top for each of the transfer tasksthe higher the quality of the sentence representation, the better the performance on these transfer tasks should be. We evaluate on the following well-known and widely used evaluations: movie review sentiment (MR) (Pang and Lee, 2005), product reviews (CR) (Hu and Liu, 2004), subjectivity classification (SUBJ) (Pang and Lee, 2004), opinion polarity (MPQA) (Wiebe et al., 2005), paraphrase identifi-cation (MSRP) (Dolan et al., 2004) and sentiment classification (SST, binary version) (Socher et al., 2013). Accuracy is measured in all cases, except for MRPC, which measures accuracy and the F1-score. 2 See https://github.com/facebookresearch/SentEval. The aim of SentEval is to encompass a comprehensive set of benchmarks that has been loosely established in the research community as the standard for evaluating sentence representations. Recent years have seen an increased interest in entailment classification as an appropriate evaluation of sentence representation quality. We evaluate representations on two well-known entailment, or natural language inference, datasets: the large-scale SNLI dataset (Bowman et al., 2015) and the SICK dataset (Marelli et al., 2014). We implement a simple logistic regression on top of the sentence representation. In the cases of SNLI and SICK, as is the standard for these datasets, the representations for the individual sentences u and v are combined by using h u , v , u  v , | u  v |i as the input features. We tune the seed and an l 2 penalty on the validation sets for each, and train using Adam (Kingma and Ba, 2015), with a learning rate of 0.001 and a batch size of 32. Although it is not the primary aim of this work to learn a state-of-the-art image and caption retrieval system, it is important to first establish the capability of our system to do what it is trained to do. Table 1 shows the results on the COCO5K caption and image retrieval tasks for the two models that predict image features. We compare our system against several wellknown approaches, namely Deep Visual-Semantic Alignments (DVSA) (Karpathy and Fei-Fei, 2015), Fisher Vectors (FV) (Klein et al., 2015) and Order Embeddings (OE) (Vendrov et al., 2015). As the results show, Cap2Img performs very well on this task, outperforming the compared models on caption retrieval and being very close to order embeddings on image retrieval 3 . The fact that the system outperforms Order Embeddings on caption retrieval suggests that it has a better sentence encoder. Cap2Both does not work as well on this task as the image-only case, probably because interference from the language signal makes the problem harder to optimize. The results indicate that the system has learned to predict image features from captions, and captions from images, at a level exceeding or close to the state-of-the-art on this task. Having established that we can learn high-quality grounded sentence encodings, the core question we now wish to examine is how well grounded sentence representations transfer. In this section, we combine our grounded features with the 3 In fact, we found that we can achieve better performance on this task by reducing the dimensionality of the encoder. A lower dimensionality in the encoder also reduces the transferability of the features, unfortunately, so we leave a more thorough investigation of this phenomenon for future work. high-quality layer-normalized SkipThought representations of Ba et al. (2016), leading to multimodal sentence representations as described in Section 3.5. That is, we concatenate Cap2Cap, Cap2Img or Cap2Both and Skip-Thought with Layer Normalization (ST-LN) representations, yielding GroundSent-Cap, GroundSent-Img and GroundSent-Both representations, respectively. We report performance of ST-LN using SentEval, which led to slightly dierent numbers than what is reported in their paper 4 . Table 2 shows the results for the semantic classification and entailment tasks. Note that all systems use the exact same evaluation pipeline, which makes them directly comparable. We can see that in all cases, grounding increases the performance. The question of which type of grounding works best is more dicult: generally, grounding with Cap2Cap and Cap2Both appears to do slightly better on most tasks, but on e.g. SST, Cap2Img works better. The entailment task results (SNLI and SICK in Table 2) show a similar picture: in all cases grounding improves performance. It is important to note that, in this work, we are not necessarily concerned with replacing the state-of-the-art on these tasks: there are systems that perform better. We are primarily interested in whether grounding helps relative to text-only baselines. We find that it does. 4 This is probably due to dierent seeds, optimization methods and other minor implementational details that differ between the original work and SentEval. An important open question is whether the increase in performance in multi-modal semantic models is due to qualitatively dierent information from grounding , or simply due to the fact that we have more parameters or data from a dierent distribution . In order to examine this, we implement a SkipThought-like model that also uses a bidirectional LSTM with element-wise max on the final hidden layer (henceforth referred to as STb). This model is architecturally identical to the sentence encoder used before: it can be thought of as Cap2Cap, but where the objective is not to predict an alternative caption, but to predict the previous and next sentence in the Toronto Books Corpus, just like SkipThought (Kiros et al., 2015). We train a 1024-dimensional and 2048-dimensional STb model (for one full iteration, with all other hyperparameters identical to Cap2Cap) to compare against: if grounding improves results because it introduces qualitatively dierent information, rather than just from having more parameters (i.e., a higher embedding dimensionality), we should expect the multi-modal GroundSent models to perform better not only than STb-1024, but also than STb-2048, which has the same number of parameters (recall that GroundSent models are combinations of grounded and linguistic-only representations). In addition, we compare against an ensemble of two dierent STb-1024 models (i.e., a concatenation of two separately trained STb-1024), to check that we are not (just) observing an ensemble eect. As Table 3 shows, a more nuanced picture emerges in this comparison: grounding helps more for some datasets than for others. Grounded models outperform the STb-1024 model (which uses much more datathe Toronto Books Corpus is much larger than COCO) in all cases, often already without concatenating the textual modality. The ensemble of two STb-1024 models performs better than the individual one, and so does the higher-dimensional one. In the cases of CR and MRPC (F1), it appears that improved performance is due to having more data or ensemble eects. For the other datasets, grounding clearly yields better results. These results indicate that grounding does indeed capture qualitatively dierent information, yielding better universal sentence representations. There are a few other important questions to investigate. The average abstractness or concreteness of the evaluation datasets may have a large impact on performance. In addition, word embeddings from the learned projection from GloVe input embeddings, which now provides a generic word-embedding grounding method even for words that are not present in the image-caption training data, can be examined. As we have seen, performance across datasets and models can vary substantially. A dataset's concreteness plays an important role in the relative merit of applying grounding: a dataset consisting mostly of abstract words is less likely to benefit from grounding than one that uses mostly concrete words. In order to examine this eect, we calculate the average concreteness of the evalua-Model MEN SimLex RW W353 GloVe 0.805 0.408 0.451 0.738 Cap2Both 0.819 0.467 0.487 0.712 Cap2Img 0.845 0.515 0.523 0.753 Table 5: Spearman  s correlation on four standard semantic similarity evaluation benchmarks. tion datasets used in this study. Table 4 shows the average human-annotated concreteness ratings for all words (where available) in each dataset. The ratings were obtained by Brysbaert et al. (2014) in a large-scale study, yielding scores for 40,000 English words. We observe that the two entailment datasets are more concrete, which is due to the fact that the premises are derived from caption datasets (Flickr30K in the case of SNLI; Flickr8K and video captions in the case of SICK). This explains why grounding can clearly be seen to help in these cases. For the semantic classification tasks, the more concrete datasets are MRPC and SST. The picture is less clear for the first, but in SST we see that the grounded representations definitely do work better. Concreteness values make it easier to analyze performance, but are apparently not always direct indicators of improvements with grounding. Our models contain a projection layer that maps the GloVe word embeddings that they receive as inputs to a dierent embedding space. There has been a lot of interest in grounded word representations in recent years, so it is interesting to examine what kind of word representations our models learn. We omit Cap2Cap for reasons of space (it performs similarly to Cap2Both). As shown in Table 5, the grounded word projections that our network learns yield higher-quality word embeddings on four standard lexical semantic similarity benchmarks: MEN (Bruni et al., 2014), SimLex-999 (Hill et al., 2016b), Rare Words (Luong et al., 2013) and WordSim-353 (Finkelstein et al., 2001). We have investigated grounding for universal sentence representations. We achieved good performance on caption and image retrieval tasks on the large-scale COCO dataset. We subsequently showed how the sentence encodings that the sys-415 tem learns can be transferred to various NLP tasks, and that grounded universal sentence representations lead to improved performance. We analyzed the source of improvements from grounding, and showed that the increased performance appears to be due to the introduction of qualitatively dierent information (i.e., grounding), rather than simply having more parameters or applying ensemble methods. Lastly, we showed that our systems learned high-quality grounded word embeddings that outperform non-grounded ones on standard semantic similarity benchmarks. It could well be that our methods are even more suited for more concrete tasks, such as visual question answering, visual storytelling, or image-grounded dialogue an avenue worth exploring in future work. In addition, it would be interesting to explore multi-task learning for sentence representations where one of the tasks involves grounding. We thank the anonymous reviewers for their helpful comments and suggestions. Part of Fig. 1 is licensed from dougwoods/CC-BY-2.0/flickr.com/photos/deerwooduk/682390157.
abstain objective result objective abstain abstain abstain abstain abstain abstain abstain result abstain objective objective abstain method abstain abstain method abstain result method objective result objective result objective other other other other other other abstain other other other other other abstain other method other method other other abstain abstain abstain abstain method method method method method method method abstain abstain method abstain abstain method method abstain abstain method method method abstain method abstain method abstain abstain method method method method method abstain method method method abstain abstain method abstain abstain method method method abstain method abstain other abstain abstain method method abstain method abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain result abstain abstain abstain abstain result result abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain method abstain abstain result abstain abstain abstain abstain method abstain method result objective result result result result objective abstain other other