Functional Distributional Semantics is a recently proposed framework for learning distributional semantics that provides linguistic interpretability. It models the meaning of a word as a binary classifier rather than a numerical vector. In this work, we propose a method to train a Functional Distributional Semantics model with grounded visual data. We train it on the Visual Genome dataset, which is closer to the kind of data encountered in human language acquisition than a large text corpus. On four external evaluation datasets, our model outperforms previous work on learning semantics from Visual Genome. 1 1 Introduction The target of distributional semantics models is to understand and represent the meanings of words from their distributions in large corpus. Many approaches learn a numerical vector for each word, which encodes its distributional information. They can be roughly divided into two categories: frequency-based methods such as co-occurrence matrix (Sahlgren, 2006), and prediction-based methods such as Word2vec (Mikolov et al., 2013). More recently, progress has been made in learning word representations in a specific context, which are also called contextualized embeddings. Examples include ELMo (Peters et al., 2018) and BERT (Devlin et al., 2019). Functional Distributional Semantics is a framework that not only provides contextualized semantic representations, but also provides more interpretability. It was first proposed by Emerson and Copestake (2016), and it explicitly separates the modeling of words and the modeling of objects and events. This is a fundamental distinction in predicate logic. While logic is not necessary for all NLP 1 Our code and models are publicly available at: https: //github.com/williamLyh/PixieVGModel tasks, it is an essential tool for modeling many semantic phenomena (for example, see: Cann, 1993; Allan, 2001; Kamp and Reyle, 2013). For semantic research questions, having a logical interpretation is a clear advantage over vector-based models. We will explain the framework in Section 2.2. Another issue with distributional semantic models, as discussed by Emerson (2020c), is the symbol grounding problem  if meanings of words are defined in terms of other words, the definitions are circular. During human language acquisition, words are learned while interacting with the physical world, rather than from text or speech alone. An important goal for a semantic theory is to explain how language relates to the world, and how this relationship is learned. We focus on the Visual Genome dataset, not only because it provides relatively fine-grained annotations, but also it is similar to realistic circumstance encountered during language acquisition, as we will explain in Section 2.3. Our main theoretical contribution is to adapt the Functional Distributional Semantics framework to better suit visual data. This is a step approaching the completion of long-term goal: leveraging previous work (Emerson, 2020a), we could joint train the Functional Distributional Semantics model with both textual and visual data. In order to make it compatible with modern techniques for machine vision, while retaining its logical interpretability, we replace the RBM of previous work with a Gaussian MRF, as explained in Section 3. Our main empirical contribution is to demonstrate the effectiveness of the resulting model. In Section 4.1, we intrinsically evaluate the major components of our model, to see how well they fit the training data. In Section 4.2, we evaluate our model on four external evaluation datasets, comparing against previous approaches to learning from Visual Genome, as well as strong text-based baselines. Not only do we confirm Herbelot (2020)'s finding that learning from grounded data is more 3976 data-efficient than learning from text alone, but our model outperforms the previous approaches, demonstrating the value of our functional approach. There is extensive research on learning language semantics from grounded visual data. Visual-Semantic Embedding and Visual Concept Learning in Visual Question Answering are two representative frameworks. Some works under these frameworks share the idea with our Functional Distributional Semantics model that textual labels are modeled as classifiers over the semantic space. Visual-Semantic Embedding (Frome et al., 2013) learns joint representations of vision and language in a common visual-semantic space. Kiros et al. (2014) proposed to unify the textual and visual embeddings via multimodal neural-based language models. Ren et al. (2016) models images as points in the Visual-Semantic space, while text are Gaussian distributions over them. Visual Concept Learning contributes to various visual linguistic applications, such as image captioning (Karpathy and Fei-Fei, 2015) and Visual Question Answering (Antol et al., 2015). Some works in applying neural symbolic approach to VQA share similar ideas of learning visual concepts with our model. For example, Mao et al. (2018) learn neural operators to capture attributes (concepts) of objects and map them into attribute-specific space. Then questions are parsed into executable programs. Han et al. (2019) further learn the relations between objects as metaconcepts. Our work differs from them in two main aspects. Firstly, our framework supports truth-conditional semantics, as explained in Section 2.2, and therefore provides more logical interpretability. Unlike the above works which always assume images are given, we use a generative model which allows us to perform inference on textual labels alone, as illustrated in Fig. 3 and explained in Section 3.4. Secondly, we learn semantics from the Visual Genome dataset, which is considered more similar to the data encountered during language acquisition, as explained in Section 2.3. Functional Distributional Semantics was first proposed by Emerson and Copestake (2016). The framework takes model-theoretic semantics as a starting point, defining meaning in terms of truth . Given an individual (also called an entity ), and given a predicate (the meaning of a content word), we can ask whether the predicate is true of that individual. Note that an individual could be a person, an object, or an event , following neo-Davidsonian event semantics (Davidson, 1967; Parsons, 1990). Functional Distributional Semantics therefore separates the modeling of words and individuals. An individual is represented in a high-dimensional feature space. The term pixie refers to the representation of an individual (Emerson and Copestake, 2017). A predicate is formalized as a binary classifier over pixies. It assigns the value true if an individual with those features could be described by the predicate, and it assigns false otherwise. Such a classifier is called a semantic function . The model is separated into a world model and a lexicon model . The lexicon model consists of semantic functions. Following situation semantics (Barwise and Perry, 1983), the world model defines a distribution over situations . Each situation consists of a set of individuals, connected by semantic roles. In our work, we only consider two types of semantic roles: ARG1 and ARG2. For example, the sentence a computer is on a desk' describes a situation with three individuals: the computer, the desk, and the event of the computer being on the desk. The computer is the ARG1 of the event, and the desk is the ARG2, as shown in Figs. 1 and 2. Unlike other distributional models, Functional Distributional Semantics is interpretable in formal semantic terms, and supports first-order logic (Emerson, 2020b). Emerson (2020a) proposed an autoencoder-like structure which can be trained efficiently from semantic dependency graphs. Because individuals are explicitly modeled, grounding the pixies is more theoretically sound than grounding word vectors. The framework has clear potential for learning grounded semantics, which we explore in this paper. The Visual Genome dataset contains over 108,000 images and five different formats of annotations, including regions, attributes, relations, object instances and question answering. In this work, we only consider the relations, which are formulated as predicate triples. Each triple contains two objects in the image and one relation between them. The objects are identified with bounding boxes, as illus-3977 Figure 1: An example image in Visual Genome, annotated with the relation [Computer', ON', Desk'] trated in Fig. 1. The object predicates are nouns or noun phrases, and the relation predicates are verbs, prepositions or prepositional phrases. Many works use Visual Genome as a grounded data source. For example, Fukui et al. (2016) use it to ground its visual question answering system. Furthermore, the fine-grained annotations make Visual Genome a compelling dataset for studying lexical semantics. As discussed by Herbelot (2020), Visual Genome is similar in size to what a young child is exposed to, and the annotations are similar to simple utterances encountered during early language acquisition. Kuzmenko and Herbelot (2019) and Herbelot (2020) learn semantics from the annotations, while discarding the images themselves. They trained word embeddings with a count-based method and a Skip-gram-based method, respectively. This methodology, of extracting word relations from an annotated image dataset, was also analyzed and justified by Schlangen (2019). In fact, Vero and Copestake (2021) analyze the different modalities in Visual Genome in terms of information gain, and conclude that, for enriching a textual model, the relational information provides more potential than the visual information. To our knowledge, there has been no previous attempt to use grounded visual data to train a Functional Distributional Semantics model, nor to utilize the visual information of Visual Genome to learn natural language semantics. We will explain the probabilistic structure of our model in Section 3.1, and how we train the components in Sections 3.2 and 3.3. In Section 3.4, we present an inference model to infer latent pixies from words and the context. We define a graphical model which jointly generates pixies and predicates, as shown in Fig. 2. It has two parts. The world model is shown in the top blue box, which models the distribution of situations, or in other words, the joint distribution of pixies. It is an undirected graphical model, with probabilistic dependence according to the ARG1 and ARG2 roles, as further explained in Section 3.2. The lexicon model is shown in the bottom red box, which models each predicate as a semantic function. It is a directed graphical model. For each pixie, it produces a probability of truth for each predicate (which are not observed), as well as generating a single predicate (which is observed), as further explained in Section 3.3. Our framework can perform contextualized inference of predicate triples, where the world model provides contextual dependency while the lexicon model focuses on individual predicate, as further expalined in Section 3.4. Given a labeled image triple, the model can be trained by maximizing the likelihood of generating the data, including both observed predicates and observed pixies. The likelihood can be split into two parts, as shown in Eq. 1, where s is a situation (a pixie for each individual), and g is a semantic dependency graph (a predicate for each individual). The first term is the likelihood of generating the observed situation, modeled by the world model. The second term is the likelihood of generating the dependency graph given an observed situation, modeled by the lexicon model. Therefore, we can optimize parameters of the two parts separately. log P ( s, g ) = log P ( s ) + log P ( g | s ) (1) 3.2 World Model The world model learns the joint distribution of pixies, as shown in the top half of Fig. 2. The individuals are grounded by images, so we can obtain the pixie vectors by extracting visual features for individuals from their corresponding images. For object pixies, they are grounded by their corresponding bounding boxes. For event pixies, Visual Genome does not have labeled bounding boxes for them and their meaning tends to be more abstract, so we use the whole image to ground them. As a feature extractor, we use ResNet101, a Convolutional Neural Network (CNN) pre-trained on ImageNet. To further reduce redundant dimensions, we perform PCA on the last layer of the CNN. We take the output of PCA as the pixie space X . A situation s is a collection of pixies within a semantic graph. In this work, we only consider graphs with three nodes, connected by the roles ARG1 and ARG2, to match the structure of Visual Genome relations. In previous work, the world model was implemented as a Restricted Boltzmann Machine (RBM). However, an RBM uses binary-valued vectors, which is not compatible with the real-valued vectors produced by a CNN. Furthermore, an RBM does not give normalized probabilities, which means that computationally expensive techniques are required, such as MCMC, used by (Emerson and Copestake, 2016), or Belief Propagation, used by (Emerson, 2020a). We model situations with a Gaussian Markov Random Field (MRF). For an n -dimensional pixie space, this gives a 3 n -dimensional Gaussian distribution, with parameters  and  for the mean and covariance. As shown in the first term of Eq. 1, we would like to maximize P ( s ) . For a Gaussian distribution, the maximum likelihood estimate (MLE) has a closed-form solution, which is simply the sample mean and sample covariance. However, because we assume the left and right pixies in Fig. 2 are conditionally independent given the event pixie, we force the top right and bottom left pixie blocks of the precision matrix   1 to be zero. We raise this assumption for the consideration of applying the Functional Distributional Semantics model to larger graphs with more individuals in the future. The assumption does not affect performance on word similarity datasets, but it slightly damages performance on contextual inference datasets. Detailed results and discussion are given in Appendix A.4. The lexicon model learns a list of semantic functions, each corresponds to a word in predicate vocabulary V . The semantic function t r ( x ) for a given predicate r is a logistic regression classifier over the pixie space, with a weight vector v r . From the perspective of deep learning, this is a single neural net layer with a sigmoid activation function. As shown in Eq. 3, the output is a probabilistic truth value ranging between (0 , 1) . As shown in the second row of Fig. 2, all semantic functions are applied to each pixie. Based on the probabilities of truth, a single predicate is generated. The probability of generating a specific predicate r for a given pixie x is computed as shown in Eq. 4. The more likely a predicate is to be true, the more likely it is to be generated. The lexicon model is optimized to maximize log P ( g | s ) , the log-likelihood of generating the predicates given the pixies. This can be done by gradient descent. When learning from Visual Genome, pixies are grounded by images. However, when applying the model to text, the pixies are latent. We provide an inference model to infer latent pixie distributions given observed predicates. This inference model is used in Section 4.2 on textual evaluation datasets. Exact inference of the posterior P ( s | g ) is intractable, because this requires integrating over the high-dimensional latent space of s . This is a common problem when working with probabilistic models. Therefore we use a variational inference algorithm to approximate the posterior distribution 3979 P ( s | g ) with a Gaussian distribution Q ( s ) . For sim-plicity,we assume that each dimension of Q ( s ) is independent, so its covariance matrix is diagonal. In Fig. 3, the graphical model illustrates this assumption, as there is no connection among the pixie nodes in the middle row. Following the procedure of variational inference, the approximate distribution Q ( s ) is optimized to maximize the Evidence Lower Bound (ELBO), given in Eq. 5. This can be done by gradient descent. L = EQ (cid:104) log P ( g | s ) (cid:105)  D KL (cid:0) Q ( s ) || P ( s ) (cid:1) (5) The first term measures how well Q ( s ) matches the observed predicates, according to the lexicon model P ( g | s ) . The second term measures how well Q ( s ) matches the world model P ( s ) . We would like to emphasize the likelihood of generating the observed predicates, so we down-weight the second term with a hyper-parameter  , similarly to a  VAE (Higgins et al., 2017). Detailed analysis on the effects of  is discussed in Appendix A.7. Exactly computing the first term is intractable. Emerson (2020a) used a probit approximation, but we instead follow Daunizeau (2017), who derived the more accurate approximations given in Eqs. 6 and 7, where x has mean  and variance  . The second approximation is particularly important, as we aim to maximize the log-likelihood. The second term of Eq. 5 is the Kullback-Leibler (KL) divergence between two Gaussians, which has the closed-form formula given in Eq. 8, where k is the total dimensionality. DKL ( Q || P ) = 1 2 (cid:104) log |  P | |  Q |  k + tr (  1 P  Q ) +(  Q   P ) T   1 P (  Q   P ) (cid:105) (8) As illustrated in Fig. 3, variational inference allows us to calculate quantities such as the probability that an animal which has a tail is a horse. To obtain the inferred distribution for a single pixie, we need to marginalize the situation distribution Q ( s ) . From the independence assumption, this simply means taking the parameters for the desired pixie. Then we can apply the semantic function for r on P Q R Y Z X "animal" "has" "tail" [ "animal": 0.58 , horse": 0.43, "bear": 0.35, "dog": 0.29, "cat": 0.22, ] "animal" "has" "paw" [ "animal": 0.56 , bear": 0.42, "horse": 0.36, "cat": 0.35, "dog": 0.31, ] Figure 3: Graphical inference model: The pixies X , Y and Z in the middle row are jointly inferred from the observed predicates P , Q and R in the bottom row, using variational inference. Although Q ( s ) assumes independence, its parameters are jointly inferred based on all predicates. This is because the KL-divergence in Eq. 8 depends on  P , which is nonzero between each pair of pixies linked by a semantic role. For example, in Fig. 3, the truth of horse' for X depends on the observed predicate tail' or paw'. This is not a direct dependence between words, but rather relies on three intermediate representations (the three pixies), all of which are expressed in terms of visual features. The first term of the ELBO connects the semantic function for tail' or paw' to the variational parameters for Z . The second term of the ELBO connects the variational parameters for Z and Y (based on the world model covariance for ARG2) as well as Y and X (based on the world model covariance for ARG1). Finally the semantic function for horse' is applied to the variational distribution for X . and an animal with paws is more likely to be a bear. We notice that the truth values are generally low for all semantic functions. Even the highest truth is only around 0.58. This illustrates that the model is not very certain, which might be expected since the model is performing inference on visual features, but the training image data is noisy. For some evaluation datasets, we need to perform inference given a single predicate. This can be done by marginalizing the joint distribution. Which pixie variable to choose, out of the three, should depend on the Part-Of-Speech (POS) of the word. For nouns, the pixie node X or Z should be used, as a noun should play the role of ARG1 or ARG2. For verbs and prepositions, the node Y should be used, as they usually describe the relation. To train our model, we follow the same preprocessing and filtering of Visual Genome as Herbelot (2020). Details of pre-processing and hyperparameters are given in the appendix. In this section, we examine whether a Gaussian MRF is a suitable choice for the world model, and whether the pixies in the pixie space are linearly separable such that the logistic semantic functions can successfully classify them. The world model learns a Gaussian distribution for the observed situations. In this section, we justify this choice by evaluating the fitting errors. Fig. 4 shows density histograms for two example pixie dimensions and their corresponding best-fit (MLE) Gaussian curves. The left histogram is an example for a majority of the pixie dimensions, which is tightly matched by the best-fit Gaussian. In other cases, as shown on the right, there are im-balanced tails and asymmetry. Despite their skewness and kurtosis, which make them look more like a Gamma distribution, they are still generally bell-shaped and the departure is not so heavy. To quantify the errors, we measure the Wasser-stein distance, the area of the histogram missing from the best-fit Gaussian. Across all 100 pixie dimensions, the mean percentage missing is 7% with a variance of 1% . A more flexible model might give better modeling performance, which could be a future improvement direction. Nonetheless, we consider this level of error to be acceptably low. Value Figure 4: Density histograms for two selected pixie dimensions, across the 2.8M training instances. Best-fit Gaussian curves of the histograms are shown in red. In this experiment, we investigate if our approach to model the semantic functions as logistic regression classifiers is suitable. In particular, a logistic regression classifier is a linear classifier, which means if the data is not linearly separable, it would have inferior performance. We computed the Area Under Curve for the Receiver Operating Characteristic (AUC-ROC), for all predicates in the vocabulary. For each predicate we randomly select equal amount of negative example pixies with its positive examples. The average score is 0.79 for object predicates, and 0.58 for event predicates. We also present the ROC for a few example predicates in Fig. 5. We can see that object classifiers have generally better performance. The classifier for racket' shows slightly worse performance than the others, whose reason might be its lower frequency. Compared to object predicates, the semantic func-3981 tions for event predicates generally perform worse. There are two potential reasons which could be improved in future work. Firstly, we used visual features generated from the whole image to represent the event pixie, which is often not specific enough to identify the event. Secondly, a logistic regression classifier might not be sophisticated enough for this classification problem. In this section, we use external semantic evaluation datasets, to give a direct comparison against previous work, and to test whether our model can generalize beyond the training data. We evaluate on two lexical similarity datasets in Section 4.2.2, and two contextual datasets in Section 4.2.3. We compare against two types of baseline: models trained on a large corpus and models trained on Visual Genome. For these datasets, our model must assign similarity scores for predicate or triple pairs, which we compute as follows. The pixie values are inferred from the first predicate or triple in the pair. Then all semantic functions from the predicate vocabulary are applied to that pixie. Then the ranking of the second predicate in the pair over all potential predicates in the evaluation dataset is taken as the similarity score. Therefore, smaller ranking means higher similarity between predicates. Finally, because there are discrepancies between vocabularies used in Visual Genome and the evaluation datasets, we follow Herbelot (2020) in filtering the evaluation datasets according to the Visual Genome vocabulary, and use the filtered datasets to evaluate all models. For the two lexical datasets, we exactly follow Herbelot's filtering conditions to give a direct comparison. For the contextual datasets, this filtering is too strict, resulting in zero vocabulary coverage. For these datasets, we apply looser filtering, with details given in the appendix. This also requires retraining our model and the Visual Genome baselines on a more loosely filtered training set. Visual Genome Baselines : We re-implement two previously proposed models learning distributional semantics from Visual Genome, described in Section 2.3. A simple count-based model was proposed by Kuzmenko and Herbelot (2019), which we refer to as VG-count. Herbelot (2020) improved on this and proposed EVA, a Skip-gram model trained on the same kind of co-occurrence data. We also implement an image-retrieval baseline which we refer to as VG-retrieval. This baseline simply retrieves all image boxes whose annotations match the indexing predicate. Visual features are extracted in the same method as our model, as described in 3.2, and then averaged across all retrieved images to obtain the representation for a given predicate. This baseline illustrates the performance can be achieved when only using the visual information of Visual Genome. Large Corpus Baselines : We trained two Skip-gram Word2vec models (Mikolov et al., 2013) using 1 billion and 6 billion tokens from Wikipedia, using Gensim ( Rehurek and Sojka, 2010). We will refer to them as Word2vec-1B and Word2vec-6B. The window sizes are set to be 10 in two directions, so they contextualize with far more words than our model. We also use Glove (Pennington et al., 2014) trained on 6 billion Wikipedia tokens as another strong baseline, which we refer to as Glove-6B. For all three baselines, the dimensionality is set to 300. Compared to the large corpus baselines, our model has fewer parameters per word (100 vs. 300), and is trained on far fewer data points (2.8M relation triples vs. 1B or 6B tokens). We use two lexical similarity/relatedness datasets, MEN (Bruni et al., 2014) and Simlex-999 (Hill et al., 2015), both of which give scores for pairs of words. MEN contains 3000 word pairs, and SimLex-999 contains 999 pairs. After filtering for the Visual Genome vocabulary, we have 584 pairs for MEN and 169 pairs for SimLex-999. MEN evaluates relatedness, while SimLex-999 evaluates similarity. For example, coffee' and cup' are related, but not similar. Capturing similarity rather than relatedness is hard for most text-based distributional semantics models because they build concept representations based on their co-occurrence in corpora, which generally reflects relatedness but not similarity. However, similarity might be more directly reflected in terms of visual features which can be captured by our model. The results are shown in Tab. 1. Our model outperforms the two baselines trained on Visual Genome, and matched the performance of Word2vec-1B (the difference is statistically insignificant, p> 0 . 5 ). median similarity score to the out-of-vocabulary pairs), it still achieves 0.304. Using the loosely filtered training set, our model can achieve the even higher score of 0.670 (on the same strictly filtered subset of MEN). This illustrates that one limit of our model's performance is the size of the Visual Genome dataset. In contrast, the performance of Word2vec does not improve much as the training data increases from 1B to 6B, which suggests there is a limit on how much can be learnt from local textual co-occurrence information alone. On SimLex-999, our model achieves 0.431, which outperforms all baselines. Compared to Glove-6B, the strongest baseline, it is weakly significant ( p< 0 . 15 ). This might justify our point that there is advantage of learning similarity from visual features. Additionally, our model can use parameters and data more effectively and efficiently than Word2vec and Glove, achieving better performance with less training data and fewer parameters. Compared with VG-count and EVA, our model can understand more semantics because it learns from the visual information. While compared with VG-retrieval, our model can leverage textual co-occurrence. As far as we know, we have achieved a new state of the art on learning lexical semantics from Visual Genome. Combining results across all four datasets (including the contextual datasets below), the difference between our model and EVA is highly significant ( p< 0 . 001 ). We consider two contextual evaluation datasets. GS2011 (Grefenstette and Sadrzadeh, 2011) gives similarities of verbs in a given context. Each data point is a pair of subject-verb-object triples, where only the verbs are different. For example, [ta-ble',show', result'] and [table', express', re-sult'] are judged highly similar. The dataset has 199 distinct triple pairs and 2500 judgment records from different annotators. The evaluation metric is Spearman correlation across all judgments. As Van de Cruys et al. (2013) point out, the second verb in each pair is often nonsensical when combined with the corresponding subject and object. Therefore, we only compare the triple pairs in a single direction, inferring pixies from the first triple and applying the second verb's semantic function. RELPRON (Rimell et al., 2016) evaluates compositional semantics. It contains a list of terms, each associated with around 10 properties. Each property is a noun modified by a subject or object relative clause. For example, the term theater' has the subject property [building', show', film'] and object property [audience', exit', building']. The task is to find the correct properties for each term, evaluated as Mean Average Precision (MAP). The development set contains 65 terms and 518 properties; the test set, 73 terms and 569 properties. Under the loosely filtered condition, our subset of GS2011 contains 252 similarity judgments; RELPRON, 57 terms and 150 properties. Rimell et al. (2016) find that vector addition performs surprisingly well at combining contextual information. Therefore, for all baselines, we represent a triple by taking the addition of the three word representations. As aforementioned, we retrain our model and the VG baselines with loosely filtered data. The results are shown in Tab. 1. The corpus models outperform the VG models. However, this is perhaps expected given that the vocabulary in GS2011 and RELPRON is more formal, and even when they are covered in Visual Genome, their frequencies are low: for RELPRON, 54% of the covered vocabulary has frequency below 100, com-3983 0.0 0.2 0.4 0.6 0.8 1.0 False positive rate 0.0 0.2 0.4 0.6 0.8 1.0 T r u e p o s i t i v e r a t e cat blanket racket computer box flag in behind Figure 6: ROC curves of the semantic functions for selected predicates, for the truth-regularized model. pared to only 6% for MEN. Furthermore, GS2011 evaluates similarity of verbs, but we saw in Section 4.1.2 that our model is less accurate for verbs. However, our model outperforms all VG baselines on both datasets. This suggests that our model is less affected by data sparsity. For the baselines, if a training triple contains multiple rare predicates, the sparsity problem is compounded. However, our model relies on the images, whose visual features are shared across the whole training set. To make the probabilistic truth values more interpretable, Emerson (2020a) proposes a regularization term which penalizes the model if all truth values stay close to 0. This would modify the loss function in Eq. 4, to give Eq. 10, with a hyperparameter  that we set to 0.5. We find that adding the log-truth term improves performance on intrinsic evaluation, but decreases performance on extrinsic evaluation. Applying the analysis in Section 4.1.2, the average AUC-ROC is 0.86 for object predicates and 0.60 for event predicates. This is illustrated in Fig. 6 for the same example predicates as Fig. 5. In contrast, when evaluating on MEN and SimLex-999, this model achieves only 0.602 and 0.381 respectively. On GS2011 and RELPRON, the model achieves lower performance of 0.112 and 0.056. The log-truth term makes predicates true over larger regions of pixie space. As shown by the intrinsic evaluation, this is helpful when considering each classifier individually. However, the regions of different predicates also overlap more, which seems to hurt their overall performance on the external datasets. To quantify this, we calculate the total truth of all predicates, for 1000 randomly selected images. For the original version of our model, on average 0.83 predicates are true for an image. This is slightly below 1, illustrating the problem Emerson aimed to avoid. However, with the log-truth term, it becomes 25.5, which may have over-corrected the problem. In this paper, we proposed a method to train a Functional Distributional Semantics model with visual data. Our model outperformed the previous works and achieved a new state of the art on learning natural language semantics from Visual Genome. Further to this, our model achieved better performance than Word2vec and Glove on Simlex-999 and matched Word2vec-1B on MEN. This shows that our model can use parameters and data more efficiently than Word2vec and Glove. Additionally, we also showed that our model can successfully be used to make contextual inferences. As future work, we could leverage previous work to jointly train the Functional Distributional Semantics model with both visual and textual data, such that we could improve the vocabulary coverage and have better understanding of abstract words.
abstain abstain objective method result abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain method objective abstain method objective result method abstain other other abstain other other other other method other other other objective method method method other other abstain other other other other other other other other other other other method other other other other other other objective other method other other other other other other other other other other other method method method method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain method abstain method method method method abstain method abstain abstain abstain method abstain abstain method abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method method abstain abstain abstain abstain abstain method abstain abstain method abstain method abstain abstain abstain method method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain method abstain method method method method abstain abstain abstain abstain method method abstain method method abstain abstain abstain method abstain abstain abstain method abstain abstain method abstain method method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method method abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective objective result result result result