In modern recommender systems, there are usually comments or reviews from users that justify their ratings for different items. Trained on such textual corpus, explainable recommendation models learn to discover user interests and generate personalized explanations. Though able to provide plausible explanations, existing models tend to generate repeated sentences for different items or empty sentences with insufficient details. This begs an interesting question: can we immerse the models in a multimodal environment to gain proper awareness of real-world concepts and alleviate above shortcomings? To this end, we propose a visually-enhanced approach named METER with the help of visualization generation and textimage matching discrimination: the explainable recommendation model is encouraged to visualize what it refers to while incurring a penalty if the visualization is incongruent with the textual explanation. Experimental results and a manual assessment demonstrate that our approach can improve not only the text quality but also the diversity and explainability of the generated explanations. Explainable recommender systems have recently attracted increasing attention both in industry and in the academic community. Such systems aim to provide high-quality recommendations and simultaneously generate explanations for the recommendations (Zhang et al., 2014; Zhang and Chen, 2020). The explanations not only can bridge the gap between how systems and users perceive the relevance of the recommended items, but also can serve to shed light on the recommendation decision process so as to avoid a black box. To provide appropriate explanations, feature-based (Zhang et al., 2014), graph-based (Xian et al., 2019, 2020; Geng et al., 2022; Fu et al., 2020), sentence-based (Chen et al., 2019a; Li et al., 2020, 2021a, 2022), causality-based (Tan et al., 2021, 2022; Xu et al., 2021a,b) Inputs : User A, Item 1, Feat. and neural-symbolic (Shi et al., 2020; Chen et al., 2021, 2022) approaches have been explored in recent years. Among them, PETER (Li et al., 2021a) is a representative sentence-based method that directly generates explanation sentences for given useritem pairs based on Personalized Transformer. While PETER outperforms previous methods in terms of both explainability and text quality metrics, it also suffers from several shortcomings: PETER tends to repeat certain universally applicable safe sentences as explanations (e.g., the hotel is very nice). For the 32,003 records in the test split of the TripAdvisor dataset by Li et al. (2020), PETER only generates around 8,100 unique sentences. The duplicate rate is close to 75%, while in reality, the duplicate rate of the TripAdvisor ground truth explanations is only 5.4%. In addition, such models are trained solely on a textual corpus, lacking real-world experiences to generate more authentic explanations, which may lead to empty sentences with insufficient details. Recently, Vokenization (Tan and Bansal, 2020) demonstrates that language understanding can be improved with token-level visual supervisions. This motivates us to consider enhancing text explanation generation with the aid of real-world images. of explanation generation model that is immersed in a multimodal environment. The goal is to encourage it to perceive real-world signals and generate visually-enhanced explanations to better assist a user's decision. Specifically, we propose the Multimodally-Enhanced Transformer for Explainable Recommendation ( METER ) approach for improved text explanations based on conditional image generation and textimage matching. Unlike traditional caption-to-image generation, our training sentences are explanations that are more comprehensive reviews based on user experiences rather than simple abstract descriptions of the image content. We adopt the generation order rating  text  image based on the consideration that the generation difficulty should gradually increase. With this approach, we seek to guide the model to understand real-world concepts regarding both item attributes and user interests (e.g., a spacious room or modern decoration). Furthermore, METER is encouraged to visualize what it is talking about for the given useritem pair and is penalized in case of a mismatch between the generated visualization and the textual explanation. This is in line with the spirit of the context token prediction module in Li et al. (2021a). While PETER only predicts text tokens as contextual information, our METER additionally generates visual tokens as a supplement. We claim that if a sentence contains more real-world concepts, it is easier to visualize it as an image with higher fidelity. To this end, we introduce a textimage matching discriminator based on contrastive learning which helps to improve both the diversity and faithfulness of the textual explanations. Beyond an auxiliary task for text generation, another advantage of METER is that the generated image visualizations may provide intuitive visual explanations in addition to rating scores and textual explanations. To empirically evaluate our framework, we conduct experiments and user studies on two real-world datasets in terms of diversity and faithfulness of text explanations, as well as consistency and quality of image visualizations. Our results reveal that using the proposed METER leads to improvements on text diversity and faithfulness, and that the generated image visualizations show high fidelity and good consistency. Overall, we make the following key contributions:  To the best of our knowledge, this is the first exploration of a multimodal explainable recommender system that jointly generates rating scores, textual explanations, and images. The system will also be promising in creative advertising applications. By immersing the model into a multimodal environment, we help it explore the real-world concepts mentioned in the text explanations and in turn enable it to generate more diverse and faithful natural language rationales that are consistent with visual grounding. Experiments and a user study on real-world datasets demonstrate the superiority of our approach over several strong baselines. Visually-Guided Language Learning There have been numerous efforts on utilizing visual information to facilitate language tasks. The general strategy they typically pursue is to obtain cross-modally aligned semantics through visual grounding. Gella et al. (2017); Zhang et al. (2020); Sigurdsson et al. (2020) draw on the visual modality to bridge the gap between languages and conduct visual grounding to improve unsupervised cross-lingual word mapping or machine translation. Vokenization (Tan and Bansal, 2020) assigns each text token with a corresponding voken and improves text-based pretraining with contextualized, visual-grounded supervisions. VidLanKD (Tang et al., 2021) further solves the shortcomings of Tan and Bansal (2020) by first learning a multimodal teacher model on video-language dataset and then transferring knowledge to the student language model through distillation. Shen et al. (2021) discovers visual impressions from text-only corpus to improve open-domain dialog generation. Li et al. (2021b) learns visionlanguage representations with cross-modal contrastive learning on a combination of pure text corpus and imagetext pairs to advance both single modal and multi-modal downstream tasks. Recently, DALL-E (Ramesh et al., 2021) merges text and visual tokens as a single stream of data and employs a universal Transformer to autoregressively model the multimodal stream. The astonishing success of these methods inspires us to guide personalized explanation generation with visual signals. Generate Explanations for Recommendation Explainable recommendation has been an important task in both research and industry (Zhang and Chen, 2020). Early approaches mainly attempt to 245 ImageDecoder The dcor is very nice and the room is very comfortable Gen. visualization Gen. explanation MatchingDiscriminator Matchscore !" (  !" & % ( % , make latent factor models interpretable by aligning each latent dimension with the explicit meaning (Zhang et al., 2014; Chen et al., 2016). In recent years, numerous neural models have been proposed to explain recommendations based on user reviews (Chen et al., 2019c,a). There have also been attempts to generate purely visual explanations (Chen et al., 2019b; Tangseng and Okatani, 2020). Compared with other explanation styles for recommendation, sentence-based methods are more straightforward and have been at the center of attention in recent times. Explanation sentences can either be generated by filling predefined templates (Zhang et al., 2014; Wang et al., 2018) or through flexible natural language approaches such as Attn2Seq (Dong et al., 2017), based on recurrent neural networks, and PETER (Li et al., 2021a), which is powered by a personalized Transformer. NETE (Li et al., 2020) combines the advantage of the two styles and produces template-controlled explanations by learning from sentence templates, which is an early form of prompt-based generation. However, none of the previous work has integrated textual and visual features and provided multimodal explanations. To the best of our knowledge, METER is also the first approach to draw on vision for improved textual explanation generation. The goal of our METER framework is to give an estimated rating score  r u,i that reflects a user u 's preference towards item i and generate a multimodal explanation to justify the estimated rating. The generated multi-modal explanation consists of a text sentence  E u,i and an image visualization  V u,i . The latter may serve as a supplement to the textual explanation for better explainability when text alone provides insufficient information. Moreover, the METER recommendation explanation model is encouraged to visualize what it is talking about for the useritem pairs and will be punished if the generated visualization does not match its textual explanation. By doing so, we aim to improve the quality, diversity, as well as faithfulness of the generated text explanations through visual grounding. In the following, we shall first elaborate how to represent visual information into visual tokens and how to encode the positional embeddings for different types of tokens used in METER. Subsequently, we describe the Multimodal Enhanced Transformer for autoregressive multimodal explanation generation. Moreover, we will introduce the textimage matching discriminator, which guides the multimodal Transformer to generate better and more diversified text explanations. Finally, we summarize the training objectives of our framework for rating prediction and explanation generation. To introduce visual signals into the Transformer structure, we follow the idea of VQ-VAEs (van den Oord et al., 2017) to encode an image I  RH  W  3 into a sequence of discrete patch-level visual tokens z q  R h  w  d , where H and W is the original size of the input image, h  w is the number of visual patches, and d is the patch-level feature dimensionality. The visual tokens are constructed by vector-quantization through a learned discrete codebook Z = { z k } Kk =1  R d of visual representations. To balance efficiency and perceptual quality, we adopt VQ-GAN (Esser et al., 2021) as the visual encoder and decoder in our framework. We first pre-train the vector-quantized visual patch encoder E , decoder G , and the discrete codebook Z on our collected images. With these pretrained components, we can encode an input image I with the encoder E as  z = E ( I )  R h  w  d . Next, we serialize  z and conduct element-wise quantization for individual encoding  z j of  z onto its closest codebook entry z k : z q = (cid:32) arg min z k Z   z j  z k  (cid:33)  R h  w  d The resulting z q are served as the encoded visual tokens { v j } mj =1 of the input image. As for the sequence of visual tokens  z q = {  v j } mj =1 produced by METER autoregressively, we can utilize the decoder G to transform it back to a generated original size image  I :  I = G ( z q )  RH  W  3 . Five distinct types of input tokens can be distinguished: user ID, item ID, feature word, text tokens for explanation, and visual tokens. With the aforementioned vector-quantized visual patch encoder, we obtain a visual token representation for a given image. For text explanations, we directly tokenize them into text token sequences. Intuitively, the generated explanation should reflect both the user's interest preferences and the item attributes. Hence, we have user IDs and item IDs as two special types of tokens to guide the model to talk about the correct topics. Finally, the feature words can serve as conditional inputs to specialize the topic of explanation. To represent tokens as embeddings, we prepare four embedding codebooks: U for user IDs, I for item IDs, V for text tokens and feature words, and Z for visual tokens. We set a fixed length m for visual tokens and a maximum length n for text tokens. Thus, the input sequence S 0 can be represented as S 0 = [ u, i, f, e 1 ,    , e n , v 1 ,    , v m ] . Before feeding the token sequence into METER, we provide positional embeddings for non-visual tokens and visual tokens separately. As the visual information has a spatial prior and is organized in a 2-D grid, we adopt an axial positional embedding (Ho et al., 2019) for visual tokens. In addition, we prepare an embedding codebook P for non-visual tokens. The final input sequence representation is the addition of token embeddings and the corresponding positional embeddings. Given a input sequence, we use a Multimodally-Enhanced Transformer to encode it and predict the next token, which can be either a text or visual token. When the input sequence starts with the special token [ BOS ] alone, the model also predicts the rating score for the candidate useritem pair and contextual words that could reflect the user's preference and the item's attributes. Suppose our multimodal Transformer has L layers, each with h -head multi-head self-attention, and d is the input embedding dimensionality. Then, for input sequence S l at layer l  [0 , L  1] , the encoded sequence S l +1 can be computed as follows (specif-ically SL denotes the final-layer output): S l +1 = FFN l (Attention ( S l WQ , S l WK , S l WV )) Here, WQ , WK , WV  R d  d h are weight matrices for projecting query, key, and value respectively (Vaswani et al., 2017), d h = d/h is the dimensionality for each head. FFN l is a feed-forward module consisting of two fully-connected layers with ReLU in between for the l -th Transformer layer. The Attention function is defined as Attention( Q , K , V ) = Softmax (cid:18) QK   d h (cid:19) V with a scaling factor  d h that maintains the order of magnitude in features. We adopt a similar masking strategy as Li et al. (2021a): the user & item IDs both can attend to all tokens in the sequence, while other non-ID tokens (including feature words, text tokens, and visual tokens) all retain the traditional causal attention masking in order to avoid any leakage of future information. Figure 2 (a) provides an illustration of our masking strategy. Assuming the final-layer output from the Transformer is SL = [ s u , s i , s f , { s e } , { s v } ] , this also serves as a representation of the input sequence for next generation iteration. We can use these vector representations to enable the following four tasks: 247 Rating prediction The first representation s u is used to conduct rating score prediction. We regard the score prediction as a regression problem and the goal is to predict the score  r u,i for the given pair of user/item IDs. Due to the adopted masking strategy, u and i can both attend to each other and capture the correlation between them. Here we make use of a two-layer fully-connected network with sigmoid activation  to map s u to a scalar score value:  r u,i =  ( s u W 1 + b 1 ) W 2 + b 2 , where the dimensionality of input, hidden layer, and output are d , d , and 1 respectively. Mean Squared Error loss (MSE) is used for rating score regression: L r = E ( u,i ) T ( r u,i   r u,i ) 2 where r u,i is the ground-truth rating score and T represents the training corpus. Context token prediction The second representation s i is designed to predict the context words for a given useritem pair. Similar to s u , s i also absorbs the words that are related to a certain user's preference and an item's attributes. Thus, this auxiliary task is able to force the Transformer to exploit the information hidden in the user ID and item ID. Such design can mitigate the problem of identical explanations being generated. By passing s i into a single fully-connected layer with Softmax activation, we can obtain a probability distribution over the vocabulary V for the context word: P c = Softmax ( s i W c + b c ) , where the dimensionality of input and output are d and |V| , respectively. The predicted context tokens are the topn words with the highest probability. If we represent the probabilities of these context words C as { p tc } nt =1 , then the negative log likelihood (NLL) loss can be computed as: L c = E (cid:34) 1 n n (cid:88) t =1  log p tc (cid:35) Explanation/visualization generation The generation of explanation words and visual codes follows the autoregressive style, i.e., decoding one token at a time from left to right. Text generation is triggered by the special [ BOS ] token, upon which we repeatedly decode words until [ EOS ] is sampled. If the number of generated text tokens before [ EOS ] is less than n , we pad the sequence with [ PAD ]. If the text sequence length is greater than n , we cut it off at length n . To obtain the visual code sequence  V , we iterate METER for a fixed number of m steps conditioned on the text explanation  E and the previously generated visual code sequence. Similar to context word prediction, we adopt a single fully-connected layer for text representations { s e } to produce probability distributions over the text vocabulary V . As for visual representations { s v } , we employ another fully-connected layer to produce probability distributions over the discrete visual codebook Z . We can then sample words and visual codes from the obtained probability distributions. For simplicity, we employ greedy decoding as the sampling method to select the word/code with the highest probability. If we denote the probabilities of the sampled words and visual codes as { p te } nt =1 and { p tv } mt =1 , respectively, then the token-level language modeling loss for text and visual code generation can be expressed as: L e = E (cid:34) 1 n n (cid:88) t =1  log p te (cid:35) +   E (cid:34) 1 m m (cid:88) t =1  log p tv (cid:35) where  is a hyperparameter used to balance the training of textual and visual token generation. Textimage matching METER is capable of generating textimage explanation pairs. However, we still need to know whether and to what degree the generated image visualization matches the text explanation from a global perspective. Hence we adopt a textimage matching discriminator D to measure the degree of congruency. From another aspect, if a generated sentence contains more real-world concepts, it is easier to ground the sentence to corresponding visual tokens and obtain an image visualization with higher fidelity. With contrastive training, we in turn push METER to generate text explanations with more grounded details. Our discriminator is equipped with two separate encoders for the visual token sequence and the text sequence. Assuming the outputs of the two encoders to be  E and  V , we can construct positive training text image pairs from the ground truth, as well as negative ones through alternate pairings. Thus, the discriminator loss can be written as: L d = E [log ( D ( E , V ))] + E (cid:104) log (cid:16) 1  D (cid:16) E , V (cid:17)(cid:105) + E (cid:104) log (cid:16) 1  D (cid:16) E , V (cid:17)(cid:105) In summary, the overall training objective function J consists of the aforementioned four losses: J = min  (  e L e +  d L d +  r L r +  c L c ) Here,  denotes all trainable parameters, while  e ,  d ,  r ,  c are regularization weights to help 248 Figure 3: t-SNE visualization for the top 88 clusters of sentence semantics when threshold is 0.95. balance the learning of different tasks. METER is then trained on J in an end-to-end manner. To conduct experiments, we adopt two publicly available explainable recommendation datasets proposed in Li et al. (2020). For each dataset, train-ing/validation/testing splits are created following the ratio of 8 : 1 : 1 . To enable the visually-enhanced model proposed in this paper, we compile a collection of images portraying real-world concepts. The real-world concepts are obtained by clustering sentence semantics with the help of Sentence-BERT (Reimers and Gurevych, 2019). At first, we use Sentence-BERT to compute the embeddings of all text explanation sentences. Since many ground-truth explanations have similar semantic meanings, we conduct fast clustering to aggregate these explanation sentences into different groups representing similar concepts and topics. Figure 3 gives a t-SNE visualization (Van der Maaten and Hinton, 2008) of the top 88 clusters if setting the similarity threshold to 0.95. From the figure, we can have a glimpse of what kinds of topics these explanation typically show. To ensure a proper amount of clusters, we set the threshold to 0.85. Thus we obtain 16,577 clusters consists of the most common 99,066 explanations for TripAdvisor and 64,937 clusters which cover 283,895 explanations for Yelp. The explanation sentences at cluster centers are then used as query input to search relevant images through Google Images API. For TripAdvisor and Yelp, we retrieve the top 20 and top 10 images for each centric explanation sentence. As a result, we have a visual concept pool of 331,540 and 649,370 After collecting enough images about dataset-aware topics, we assign each text explanation the most suitable image visualization by calculating the similarity between the two modalities with CLIP model (Radford et al., 2021). In this way, we build the textual recommendation explanationimage visualization pairs for both datasets and then train METER on the constructed multimodal pairs. In Figure 4, we provide several text explanations with their corresponding assigned image visualizations. Table 1 shows the statistics of the established multimodal explainable recommendation datasets. Note that the TripAdvisor dataset mainly focuses on the hotel and travel domain, while the majority of the Yelp data is about restaurants. Records in the two datasets consist of: user ID, item ID, rating score (from 1 to 5 ), feature word, text explanation, and image visualization aligned with the text explanation. To ensure better representative ability of the visual encoder used in METER, the three components (i.e., encoder, decoder, and visual codebook) of VQ-GAN are first pre-trained on the collected images of the two datasets. For image visualization generation, we first sample 32 candidate images conditioned on the corresponding explanations, and then use the trained textimage discriminator to produce match scores. The image with the highest match score is finally selected as output. The embedding size d of METER is set to 256 , the dimensionality of the feed-forward network's hidden layer is 1 , 024 . The maximum text length n of the explanation sequence is set to 15 , while the length of the visual token sequence m is set to 256 , and the standard image size for VQ-GAN is set to 256  256 . We keep the most frequent 20 , 000 words as the text vocabulary, while the size of the discrete visual codebook is 1 , 024 . The Multimodally-Enhanced Transformer uses L = 8 layers, each endowed with a multi-head attention with h = 8 heads. We set the regularization weights  e ,  d ,  r , and  c to 1 . 0 , 1 . 0 , 0 . 1 , and 1 . 0 , respectively. And we choose 7 . 0 as the value of the balancing hyperparameter  . The METER model is trained with Adam optimization (Kingma and Ba, 2015) under a batch size of 32 , and the learning rate is set to 5  10  4 . We conduct all experiments on NVIDIA Quadro RTX 6000 GPUs. We conduct our evaluation from three perspectives  explanation generation performance, textimage matching performance, and rating prediction performance. For each of the three aspects, we adopt both automatic and manual forms of evaluation (see Sec. 4.6). For explanation performance, we measure the text quality, diversity, and explainability of the generated explanations. For the text quality, we adopt BLEU-1 and BLEU-4, as well as ROUGE-1 and ROUGE-2. To overcome the drawbacks of the two traditional metrics, we also employ Unique Sentence Ratio (USR) proposed by Li et al. (2020) to quantify the diversity of the generated sentences. For the diversity in feature word level, we adopt Feature Diversity (DIV) proposed in Li et al. (2020), which measures the intersection of features between any two generated explanations. In explainable recommendation, an explanation will normally be valued more by users if it justifies a recommendation's advantage using certain feature words as specified in the datasets. Thus, we adopt two more metrics tailored for explainability evaluation proposed by Li et al. (2020)  Feature Matching Ratio (FMR) and Feature Coverage Ratio (FCR). Specifically, FMR measures whether a generated explanation contains the feature in the ground-truth, while FCR is computed as the number of distinct features contained in the generated explanations, divided by the total number of features in the whole dataset. To assess the textimage matching, we adopt CLIPScore (CS) proposed by Hessel et al. (2021) as an objective metric to measure the degree of correspondence for cross-modality pairs. For the rating prediction performance, we rely on two standard metrics  Root Mean Square Error (RMSE) and Mean Absolute Error (MAE). By including the recommendation experiment, we merely seek to prove that the rating scores predicted by our method are sufficiently 250 Figure 5: Qualitative results generated by METER with a conditional feature word as input: strong to merit explanation generation, because if a rating prediction is inaccurate, the generated explanation will be less meaningful. For the performance comparison, we consider several baselines with regard to the task of explanation generation: Attn2Seq (Dong et al., 2017) learns to encode attributes into vectors, and then invokes an attention mechanism to generate reviews conditioned on the attribute vector. Transformer (Vaswani et al., 2017) treats user and item IDs as words and trains on the explanation generation task with a vanilla Transformer structure through language modeling. NETE (Li et al., 2020) designed a tailored GRU module to incorporate the given feature into the decoding stage. The system can generate template-like explanations while also making recommendations. PETER (Li et al., 2021a) is a simple and effective framework that attempts to use the IDs to predict the words in the target explanation. It is built upon a modified attention mask of the Transformer model. With regard to mere recommendation, we compare with two traditional methods in addition to NETE and PETER: PMF (Salakhutdinov and Mnih, 2007) conducts probabilistic matrix factorization in latent space. SVD++ (Koren, 2008) combines factor and neighborhood models to enhance the accuracy. In this section, we evaluate the performance of the proposed METER approach on two real-world datasets and compare with several representative explanation generation methods in Table 2 and recommendation models in Table 3. From Table 2, we can see that METER achieves the best FMR and DIV against all other methods, showing that METER can cover more diverse feature words during generation while maintaining good explainability. METER notably improves the USR over PETER but is slightly lower than NETE. Note that NETE is a template-based approach so it naturally achieves high USR scores. Among all methods, METER exhibits the best balance between text quality and text diversity, while being the only method that can produce both text and images, with reasonably high Image Consistency. Since automatic metrics cannot completely reflect the quality and faithfulness of generated text explanations, we also conduct a user study in the next subsection for further verification. Moreover, Table 3 indicates that METER can achieve comparable rating performance to other approaches. In Figure 5, we present several real examples illustrating how METER is able to jointly generate not only high-quality rating scores and text explanations but also image visualizations. Taking the first case in (b) as an example, we observe how METER creates coherent explanations 251 Methods Yelp TripAdvisor RMSE  MAE  RMSE  MAE  PMF 1.09 0.88 0.87 0.70 SVD++ 1.01 0.78 0.80 0.61 NETE 1.01 0.79 0.79 0.60 PETER 1.01 0.78 0.81 0.63 METER 1.01 0.79 0.80 0.61 Table 3: Recommendation performance comparison in terms of RMSE and MAE among several methods. rather than directly copying the feature word into the generated sentence, leading to greater diversity. 4.6 User Study To genuinely assess the quality of text explanations generated by METER and whether the image visualization matches the text explanation, we conduct a user study on the faithfulness of the generated text explanations with associated visual grounding. We randomly sampled 500 generated explainable contextual sentences as well as corresponding image visualizations. For comparison, we also randomly pick 500 samples from the baselines and randomly mixed them with the samples from our method. We asked 30 human subjects to provide a rating range from 1  5 , where larger scores represent better faithfulness and diversity. For better evaluation, we also provide the original user/item information and ground-truth explanation sentence for their reference. We consider Faithfulness as a criterion to assess the degree of explainability of the text, which encompasses both its readability and its cogency to the human participants. A higher Diversity represents more lexically varied generated context. We further consider Consistency representing to what extent the generated images match the associated generated sentence, while higher Quality scores indicate the generated image contains clearer details and better fidelity. We then calculate the overall scores by averaging the ratings given by each human participant across 500 samples each from both the baseline and from our method. The results are reported in Table 4 and show that our method can generate diverse and faithful explanation sentences of a higher quality than PETER, while also attaining a high image quality and good cross-modal consistency. We also provide an ablation study of the training tasks on TripAdvisor dataset. According to Table 5, the context prediction task has a big influence on Sentence Image Faithfulness Diversity Consistency Quality Baselines 3.41 2.96 2.54 3.04 Ours 4.57 3.70 3.06 4.19 Table 4: Manual evaluation performance between METER and baselines. the explainability and diversity of the generated explanations. The feature word has a vital role in deciding the topic for the model to consider. Obviously the rating prediction task is important for recommendation performance, while the visual generation task is decisive for the image consistency score. As we expected, the discriminator loss can assist the model to generate both diverse explanations and better image visualizations. In this paper, we propose METER, the first attempt to jointly generate rating scores, text explanations, and corresponding image visualizations. We immerse our model in a multimodal environment by putting all modalities to one shared Transformer decoder structure. A textimage matching discriminator is further introduced to encourage sentences with more groundable and fine-grained concepts. Experimental results demonstrate that our framework can provide diverse and faithful text explanations, together with image visualizations as additional intuitive explanations. This proves that visual information offers auxiliary knowledge for the explanation generation model to gain awareness of real-world semantics. Our dataset and code are available at https://github.com/ jeykigung/METER . In the future, we plan to investigate generating visually-enhanced explanations for more domains such as fashion and movie. We appreciate the valuable feedback and suggestions of the reviewers. This work was supported in part by NSF IIS 1910154, 2007907, and 2046457. Any opinions, findings, conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect those of the sponsors.
abstain abstain abstain abstain objective objective abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain objective abstain abstain objective abstain method abstain abstain abstain method abstain result abstain method objective objective objective objective objective other other other other other other other other abstain other other other other other other other other other objective abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain method abstain method other method abstain abstain abstain abstain abstain abstain method abstain abstain method method method abstain method abstain method other abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method method abstain abstain method method abstain method method abstain method method abstain method abstain method other abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain result abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain objective abstain abstain objective abstain other objective other other other