As a more natural and intelligent interaction manner, multimodal task-oriented dialog system recently has received great attention and many remarkable progresses have been achieved. Nevertheless, almost all existing studies follow the pipeline to first learn intra-modal features separately and then conduct simple feature concatenation or attention-based feature fusion to generate responses, which hampers them from learning inter-modal interactions and conducting cross-modal feature alignment for generating more intention-aware responses. To address these issues, we propose UniTranSeR, a Unified Transformer Semantic Representation framework with feature alignment and intention reasoning for multimodal dialog systems. Specifically, we first embed the multimodal features into a unified Transformer semantic space to prompt inter-modal interactions, and then devise a feature alignment and intention reasoning (FAIR) layer to perform cross-modal entity alignment and fine-grained key-value reasoning, so as to effectively identify user's intention for generating more accurate responses. Experimental results verify the effectiveness of UniTranSeR, showing that it significantly outperforms state-of-the-art approaches on the representative MMD dataset. The multimodal task-oriented dialog systems are designed to help users achieve specific goals such as clothing recommendation or restaurant reservation, which is in growing demand in the current business environment. As a leading study, Saha et al. (2018) released a multimodal dialog dataset (MMD) in the online retail domain. Based on such a benchmark dataset, many multimodal dialog models incorporating domain knowledge have recently been proposed (Chauhan et al., 2019; Zhang et al.,  Corresponding author. 2019, 2021), which basically exploit taxonomy-based method (Liao et al., 2018; Cui et al., 2019) or attention-based method (Nie et al., 2019; He et al., 2020) to incorporate knowledge base (KB) information for better performance. Though achieving remarkable progress, existing multimodal task-oriented dialog systems still suffer from the following three limitations. Firstly , prior models only learn the intra-modal features (including textual features, visual features and domain knowledge) separately before fusing them. Since these multimodal cues in general can enhance and complement each other, projecting them into a unified semantic space to learn the inter-modal features, with no doubt, can help improve the abilities of natural language understanding, which in turn will benefit the response generation. Secondly , prior models only conduct simple feature concatenation (Saha et al., 2018; Nie et al., 2019) or attention-based feature fusion (Cui et al., 2019) af-103 ter acquiring intra-modal representations, but without learning fine-grained alignment between different modalities before fusion, which is not favorable to query knowledge for accurate multimodal response generation. Take the dialog in Figure 1 as an example, when answering the user's query on similar style of jackets, the model is expected to align the word jackets with the corresponding visual features for proper semantic complement and entity enhancement. Thirdly , prior models basically lack the capability of entity-level reasoning, which prevents them from performing reasoning over crucial entities to guide intention-aware response generation. For example, in Figure 1, when the user asks show some similar jackets in black color , the chatbot is expected to properly explore the pivot attribute black that connects the start query cue jackets with the target recommended product images. Specifically, the model needs to perform a 2 -hop reasoning over triples (jacket_q, attribute, black_v) and (black_q, image, jacket_v) and obtain the intended 4 images. To address the aforementioned limitations, we propose a Unified Transformer Semantic Representation framework with feature alignment and intention reasoning, UniTranSeR for short. Specifically, to address the first limitation, we stand on the shoulder of Vision-and-Language Pre-training (VLP) methods (Lu et al., 2019; Li et al., 2019; Chen et al., 2020; Li et al., 2021) to propose a unified-modal Transformer encoder, which is used to project all the multimodal features into a unified semantic space to prompt inter-modality interactions, with the objective of learning better representations. Based on the unified encoder, we further address the second limitation by designing a feature alignment module to perform cross-modal feature alignment. Finally, to address the third limitation, we devise a fine-grained intention reasoning module for capturing users' real intentions, by leveraging a key-value attention based memory mechanism to perform multi-hop knowledge query for generating text or image responses. We conduct experiments on MMD, one of the most influential benchmark datasets for multimodal dialog generation. We follow the mainstream evaluation script of dialog generation and demonstrate that UniTranSeR significantly outperforms the current state-of-the-art baselines. Ablation study also shows the efficacy of each component in improving the performance of dialog generation, and a further case study reveals that our model can effectively perform fine-grained token-level feature alignment for multimodal dialog generation. 2.1 Unimodal Dialog Systems Recent years has witnessed the remarkable success in textual dialog systems, which can be roughly divided into two categories: open-domain conversations with casual chi-chat (Song et al., 2020; Gangal et al., 2021; Chan et al., 2021; Yang et al., 2021) and task-oriented dialog systems (Pei et al., 2021; Santra et al., 2021; Wang et al., 2021; Mi et al., 2021; Madotto et al., 2021; Gou et al., 2021; Raghu et al., 2021), which are designed to help users achieve specific goals. Early efforts mainly adopt a sequence-to-sequence (Seq2Seq) architecture, but cannot work well in KB retrieval and reasoning. To alleviate this problem, copy mechanism (Eric and Manning, 2017) have been adopted and many memory augmented Seq2Seq models have been proposed (Bordes et al., 2017; Wen et al., 2018; Madotto et al., 2018; Wu et al., 2019; Reddy et al., 2019; Qin et al., 2019; Wang et al., 2020; Qin et al., 2020), which achieve promising results. With the flourishing of social media platforms, massive amounts of multimedia data are generated daily, which poses great demand for multimodal dialog systems. However, due to the lack of large-scale multimodal dialog datasets, researches in this domain have been limited. To this end, Saha et al. (2018) provided a vertical retail domain dataset MMD to promote the research and proposed a multimodal hierarchical encoder-decoder model (MHRED) as a baseline. Based on MHRED, Liao et al. (2018) incorporated the style tips into a knowledge-aware multimodal dialog model (KMD). Cui et al. (2019) designed a user attention-guided multimodal dialog system (UMD) by additionally considering the hierarchical product taxonomy and user's attention to products. Chauhan et al. (2019) introduced an ordinal and attribute aware multimodal dialog system (OAM) by employing a novel position and attribute aware attention mechanism. Later, Nie et al. (2019) proposed a multimodal dialog system with adaptive decoders (MAGIC), which can incorporate different forms of domain knowledge to generate different kinds of responses. Recently, combining with 104 show some similar T-shirts but in a lighter color T-shirt_231 color nattier_blue nattier_blue_style img img_url Unified-modal Transformer Semantic Encoder [CLS] 0/1 K VTVI T-shirt_232 color dark-grey dark_grey_style img img_url Q Align Reason general response intention-aware response image output Hierarchical Transformer Response Decoder HTR Decoder UTS Encoder show FAIR Layer Text Image Knowledge Figure 2: The Proposed Framework. Transformer (Vaswani et al., 2017), He et al. (2020) advanced a multimodal dialog system via capturing context-aware dependencies of semantic elements (MATE) for textual response generation. Most existing multimodal dialog systems learn intra-modal features separately for later feature concatenation or fusion. Different from them, our proposed UniTranSeR can project all the multimodal features into a unified semantic space to perform fine-grained feature alignment and intention reasoning, which can lead to more accurate responses. Vision-and-Language Pre-training (VLP) (Lu et al., 2019; Li et al., 2021) is another line of research relevant to our work, but different from ours in that it focuses more on boosting the performance of representation learning, while the multimodal dialog systems focus more on multi-turn multimodal interaction between users and agents. The proposed UniTranSeR mainly comprises three parts: Unified-modal Transformer Semantic (UTS) encoder (Sec. 3.1), Feature Alignment and Intention Reasoning (FAIR) layer (Sec. 3.2), and Hierarchical Transformer Response (HTR) decoder (Sec. 3.3), as shown in Figure 2. We define the multimodal dialog generation task as generating the most likely response sequence Y = { y 1 , y 2 ,    , y n } and selecting topk most matched images, giving multimodal context utterances U = { u 1 , u 2 , . . . , u | U | } and multimodal knowledge base B as inputs. The probability of a textual response can be formally defined as, P ( Y | U, B ) = n (cid:89) t =1 P ( y t | y 1 , . . . , y t  1 , U, B ) (1) where y t represents the current token decoded by the HTR decoder. The UTS encoder is used to project all the multimodal features into a unified vector space for inter-modal interactions, while the FAIR layer is designed to align cross-modal hidden features, with textual features and visual features from previous UTS encoder as inputs. Similar to MAGIC (Nie et al., 2019), our HTR decoder is designed to decode three types of responses: general responses that refer to the highly frequent responses (e.g., courtesy greetings) in the conversation, such as How can I help you? ; intention-aware responses that refer to the task-oriented utterances, such as Found some similar black leather-jackets for you ; and multimodal responses that refer to the intention-aware responses with image output. The response type is determined by a query vector Q from the FAIR layer, in which an intention classifier is trained to decide which kind of response should be given out. We first use a text embedder and an image embedder to extract textual features and visual features, respectively, and extract informative features from external knowledge by utilizing both text and image embedders. Afterwards, we feed these three kinds of features into a unified Transformer encoder for unified-modal semantic representation learning. Text Embedder. To learn textual intra-modal features, we use a BERT tokenizer to split the input sentence into words and exploit a single transformer layer to obtain these words' initial embed-dings. Note the self-attention mechanism in Transformer is order-less. So, it is necessary to encode the words' position as additional inputs. The final 105 representation for each word is derived via summing up its word embedding and position embedding, followed by a layer normalization (LN) layer. Image Embedder. To learn visual intra-modal features, we use a contour slicer to cut the input images into patches and exploit ResNet-50 (He et al., 2016) to extract these patches' visual features. We notice that people usually focus on four parts of a clothing image: head, upper body, lower body, and feet, so we intuitively use an equal-height mode to slice an image into four patches, which efficiently solves the problem of region feature extraction, without using complex target detection networks such as Faster R-CNN (Ren et al., 2015). Then, we feed the patches into ResNet-50 to get the patches' initial embed-dings. Similarly, we also encode the position features for each patch via a 4 -dimensional vector [ image _ index, patch _ index, width, height ] . Both visual and position features are then fed through a fully-connected (FC) layer, to be projected into the same embedding space. The final visual embedding for each patch is obtained by first summing up the two FC outputs, and then passing them through an LN layer. Knowledge Embedder. To integrate informative features from external knowledge 1 into the task-oriented dialog, we equip the product knowledge base for each utterance through searching a fashion item table provided by MMD. We then treat these searched knowledge entries into the same triplet format, i.e., (product, match, product) , (product, attribute, value) , (product, celebrity, pas-sion_score) . Next, for the text and image elements of these triples, we use the text and image embed-ders to obtain their respective representations. Unified Transformer Encoder. After obtaining the multimodal initial embeddings, denoted as h t , h v and h k respectively, we project them into a unified semantic space to obtain interactive representations by using a unified Transformer encoder. Specifically, in each utterance, the textual features, visual features and informative features correspond to l tokens with [TXT], 4 tokens 2 with [IMG] and 4 tokens 3 with [KNG]. In order to integrate 1 External knowledge of MMD includes: style tips graph, attributes table and celebrities histogram, as shown in Figure 1. 2 Note when an utterance contains multiple images, it can be unrolled into a sequence of utterances, each containing a single image, the same as previous work. Unified-modal MLM show some similar [MASK] ... T-shirts Unified-modal MPM show some similar ... T-shirts Figure 3: Illustration of MLM and MPM. dialog history of previous rounds, we initialize the current [CLS] p by using the representation of the previous round [CLS] p  1 . The output hidden state representations can then be phrased as: H p = f (cid:0) [CLS] p  1 h pt [TXT] h pv [IMG] h pk [KNG] (cid:1) (2) where f (  ) denotes the Transformer encoder, H p 0 denotes the hidden state representation of the current round [CLS] p , which is regarded as the contextual semantic vector of the entire utterance in this round, H p 1: l denotes the representations for the text sequence, H pl +1: l +4 denotes the representations for the patch sequence, and H pl +5: l +8 denotes the representations for knowledge entries. Note the superscript p is omitted for simplicity if no confusion occurs in the following discussion. To obtain better representations, we introduce the Masked Language Modeling (MLM) loss and Masked Patch Modeling (MPM) loss to train them. We denote the input words as w = { w 1 , . . . , w l } , the image patches as v = { v 1 , . . . , v 4 } , the knowledge elements as k = { k 1 , . . . , k 4 } , and the mask indices as m  NL , where N is the natural numbers and L is the length of masked tokens. In MLM, we randomly mask out the input words with a probability of 15% , and replace the masked ones w m with a special token [MASK], as illustrated in Figure 3. The goal is to predict these masked words by attentively integrating the information of their surrounding words w \ m , image patches v and knowledge elements k , by minimizing the following loss: LMLM (  ) =  E ( w,v,k )  U log P  (cid:0) w m | w \ m , v, k (cid:1) (3) Similar to MLM, in MPM, we also randomly mask out the image patches and use zeros tensor to replace them, as shown in Figure 3. Unlike textual words that can be categorized as discrete labels, visual features are high-dimensional and continuous tensors, thus cannot be supervised via a negative log-likelihood loss. Following UNITER (Chen et al., 2020), we built the MPM loss as: LMPM (  ) = E ( w,v,k )  U g  (cid:0) v m | v \ m , w, k (cid:1) (4) where v m are masked image patches and v \ m are remaining patches. g  (cid:0) v m | v \ m , w, k (cid:1) = L (cid:88) i =1 (cid:13)(cid:13)(cid:13) f  (cid:16) v ( i ) m (cid:17)  h v ( i ) m (cid:13)(cid:13)(cid:13) 2 2 (5) To align the cross-modal features for accurate intention classification and knowledge query, we devise a feature alignment and intention reasoning (FAIR) layer. In feature alignment, we use Image-Text Matching (ITM) and Word-Patch Alignment 4 (WPA) to conduct a two-level alignment. That is, ITM is used to align text and image in sentence-level, while WPA is used to align each split word and each sliced patch in token-level. In intention reasoning, we fuse f ( [CLS] ) and aligned entities' hidden state representations to obtain a query vector Q , which is then used for intention classification and knowledge query. Image-Text Matching (ITM). In ITM, we use the output f ( [CLS] ) of the unified Transformer encoder to compute the match probability of the sampled pair. Specifically, we feed f ( [CLS] ) into an FC layer and a sigmoid function to predict a probability score P  ( w, v ) , which is between 0 and 1 . During training, we sample a positive or negative pair ( w, v ) from the dataset D at each step. The negative pair is created by randomly replacing the image or text in the same batch. We employ a binary cross-entropy loss for optimization: LITM (  ) =  E ( w,v )  D [ y log P  ( w, v )+ (1  y ) log (1  P  ( w, v ))] (6) where y is a binary truth label. Note here we only use ITM to train image-text pairs but without considering the knowledge vector, because it has already matched the textual sequence when being searched out. Word-Patch Alignment (WPA). For more fine-grained alignment between each word and image patch, we introduce a WPA technology, which is used to train the consistency and exclusiveness between these cross-modal features to prompt alignment. We use a WPA loss to supervise the process, 4 A modified version of the previous Word-Region Alignment (WRA), which can be adapted to the alignment between textual words and visual patches. LWPA (  ) =  (cid:88) l i =1 (cid:88) 4 j =1 T ij   ( w i , v j ) (7) where  denotes the cos(  ) similarity function, T  R l  4 is a ground truth table and each T ij  T is a binary label 0 or 1. During training, we sample positive or negative pairs ( w i , v j ) from each multimodal utterance to construct a probability table, as shown in Figure 2. The above loss function LWPA is then used to update the parameters  . During inference, we continue to fuse aligned entities' hidden state representation and f ( [CLS] ) to obtain a unified query vector Q , which contains multimodal query information with entity enhancement, and will be used for subsequent intention reasoning. Intention Classify (IC). Given the query vector Q , this component aims to understand the users' intention and thereafter determine which type of response should be generated. To be clear, there are a total of 17 types labeled in the MMD dataset, and each user's utterance is labeled with a specific intention type. Following MAGIC, we customize the type of response specifically for each intention, as shown in Table 1. Subsequently, we leverage an MLP layer to predict Q 's probability distribution and select the highest probability to generate a response. Besides, a cross-entropy loss is applied to optimizing the intention classifier: LIC (  ) = (cid:88) | U | i =1 (cid:88) 17 j =1 I  ij log P  ( I ij | Q ) (8) where P  ( I ij | Q ) denotes the probability of being predicted as intention I ij , and I  ij is a ground truth label. The intention classifier is trained by the loss function LIC (  ) to update parameter  , and finally outputs a reliable intention prediction result I in the inference phase. Knowledge Query (KQ). Given the predicted intention result I , this component first determines whether knowledge query is required based on Table 1. If required, we adopt a key-value memory mechanism to query all embedded knowledge triples 5 . Specifically, these embedded knowledge triples are divided into key parts and value parts, which are respectively denoted as vector K and vector V . Note here K is obtained through a linear 5 The triple is in the form of ( head, relation, tail ) 107 Id Intention categories Response type Component Id Intention categories Response type Component 1 greeting general IC 10 ask-attribute intention-aware IC+KQ 2 self-info general IC 11 suited-for intention-aware IC+KQ 3 give-criteria multimodal IC+KQ+MR 12 celebrity intention-aware IC+KQ 4 show-image multimodal IC+KQ+MR 13 filter-results multimodal IC+KQ+MR 5 give-description multimodal IC+KQ+MR 14 sort-results multimodal IC+KQ+MR 6 show-more multimodal IC+KQ+MR 15 switch-synset general IC 7 show-orientation multimodal IC+KQ+MR 16 buy general IC 8 show-similar multimodal IC+KQ+MR 17 exit general IC 9 goes-with intention-aware IC+KQ Table 1: The categories of user's intentions, their corresponding response types and required components. fusion of the embedded head-entities and relations. The knowledge query process is as follows:  i = Softmax (cid:0) QT  K i (cid:1) (9) VT = (cid:88) | M | i =1  i V i (10) where  i denotes the attentive probability score for K i , | M | is the number of knowledge triples, and VT is a weighted sum of V i , which will be used for textual decoding in an intention-aware response. Multi-hop Recommend (MR). Given the predicted intention result I and one-hop query result VT , this component first needs to determine whether an image recommendation is required based on Table 1. If required, we continue to use VT as a query vector to perform another hop query over the entire knowledge base, which implies that the product images will be recommended, if the key parts of their corresponding triples have high similarity to VT . Specifically,  i = Softmax (cid:0) VT T  K i (cid:1) (11) After deriving  i , we use VI = { q i } , an image pointer vector, to select images with top  i for recommendation, where q i = (cid:26) 1 , if V i = 1 1  512 0 , otherwise (12) and 1 1  512 is a column vector with each element equal to 1 , which denotes for the special token [URL] of the image's link. Note here 512 is the embedding size in our unified Transformer encoder. It is not difficult to see that UniTranSeR can extend the above one-hop knowledge query to multi-hop by iteratively performing attention-based key-value reasoning and ultimately achieve multi-hop image recommendation. As mentioned earlier, we used a hierarchy mechanism to decode different types of response sequences, including general responses, intention-aware responses and multimodal responses. They Dataset Statistics Train Valid Test Dialogs 105,439 22,595 22,595 Proportion 70% 15% 15% Table 2: Statistics of the MMD dataset. share the same uni-directional Transformer layer, but the semantic representations fed to this decoder are different. Specifically, for general responses, we just take the sentence-level representations f ( [CLS] ) as input. For intention-aware responses, we take the concatenation of f ( [CLS] ) and attentive vector VT followed by an FC layer as input. For multimodal responses, we take the input for the intention-aware responses, as well as VI , the image pointer vector, as input. To evaluate the performance of UniTranSeR, we conduct experiments on the widely-used benchmark dataset MMD contributed by Saha et al. (2018). The MMD dataset consists of over 150k conversations between users and chatbots in the retail domain, and each conversation describes a complete online shopping process. During the conversations, the user proposes his/her requirements in multimodal utterances and the chatbot introduces different products step by step until they make a deal. In our experiments, we follow Nie et al. (2019) to partition MMD. The statistics the dataset after partition are presented in Table 2, and more detailed statistics can be found in Appendix A.4. Following several previous work (Nie et al., 2019; He et al., 2020; Zhang et al., 2021), we use Bleun , Nist and Recall@ k to evaluate our model over two basic tasks separately, i.e., text task and image task. For the text task, we employ the proposed HTR decoder to produce all general responses and intention-aware responses. As the length of 20.07% target responses in MMD is less than 4 , such as Hello! and Thanks a lot! , we follow Nie et al. (2019) to calculate Bleun by 108 Methods Text Task Image Task Bleu-1 Bleu-2 Bleu-3 Bleu-4 Nist Recall@1 Recall@2 Recall@3 PreviousMethods MHRED (Saha et al., 2018) 32.60 25.14 23.21 20.52 3.0901 0.7980 0.8859 0.9345 KMD (Liao et al., 2018) ---0.9198 0.9552 0.9755 UMD (Cui et al., 2019) 42.78 33.69 28.06 23.73 -0.9796 0.9980 0.9990 OAM (Chauhan et al., 2019) 48.30 38.24 32.03 27.42 4.3236 -MAGIC (Nie et al., 2019) 50.71 39.57 33.15 28.57 4.2135 0.9813 0.9927 0.9965 MATE (He et al., 2020) 56.55 47.89 42.48 38.06 --Ours UniTranSeR 63.27 55.93 51.31 48.07 4.9774 0.9983 0.9995 0.9998 Table 3: Main results. varying n from 1 to 4. Note higher Bleu and Nist scores indicate that more n -gram overlaps exist between the predicted and target responses, and hence are more favorable. For the image task, we adopt Recall@ k to evaluate the efficacy of image response, where k is varied from 1 to 3. Note the image response is correct only if the positive image is recommended in the topk product images. We compare our model with the following state-of-the-art baselines. MHRED (Saha et al., 2018) 6 is the first baseline work to integrate the visual features into a hierarchical encoder-decoder model for their constructed MMD dataset. KMD (Liao et al., 2018) incorporates the style tips into the memory augmented neural model and adopts deep reinforcement learning to boost the performance. UMD (Cui et al., 2019) 7 proposes a user attention-guided multimodal dialog system by considerring the hierarchical product taxonomy and the user's attention to products. OAM (Chauhan et al., 2019) proposes a novel ordinal and attribute aware attention mechanism for multimodal dialog generation. MAGIC (Nie et al., 2019) 8 adopts the adaptive decoders with intention understanding to explicitly generate three types of responses. MATE (He et al., 2020) 9 utilizes a multimodal element-level encoder to integrate dialog context and leverages a knowledge-aware two-stage decoder for response generation, and achieves state-of-the-art performance. Following Saha et al. (2018) and Nie et al. (2019), we utilize two-turn utterances prior to the target response as the context, and set the vocabulary size to 26 , 422 . In our trainings, the batch size is set to 64, learning rate is set to 1 e  4 and the max number of training epoches is set to 1 e 4 . Adam optimizer is used to optimize all models. All experiments are conducted with PyTorch. More details about hyperparameter settings can be found in Appendix A.1. Automatic Evaluation Following KMD, UMD and MAGIC, we evaluate model performance automatically from two aspects: text response and image response. From the results in Table 3, we can observe that our model UniTranSeR achieves the state-of-the-art performance on both tasks. Specifically, in text task, UniTranSeR exhibits the highest Bleun with varying n from 1 to 4 compared with other baselines, indicating that our model can generate responses closer to the golden ones. Moreover, our model outperforms MATE, a recent model that can capture context-aware dependencies of semantic elements, by 26 . 3% in Bleu-4 score, which verifies the effectiveness of our model in learning cross-modal feature alignment and conduct intention reasoning to generate more accurate and informative responses. In image task, an extremely difficult performance improvement can be observed, which further verifies the superiority of our model. Human Evaluation The human evaluation mainly focuses on four aspects: fluency, relevance, correctness, and informativeness, which are all important for task-oriented dialogue systems (Cui et al., 2019; Nie et al., 2019; He et al., 2020). We first randomly selected 200 dialogs from the MMD datasets, and used different models to generate responses, including UMD, OAM, MAGIC, MATE 109 Model Flue. and UniTranSeR. Then, we hired human experts to score the responses and golden responses in blind review on a scale from 1 to 5 , which simulated a real-life multimodal task-oriented conversation scenario. By calculating the average score of the above metrics, we obtained the final manual evaluation results, as shown in Table 4. It can be observed that UniTranSeR consistently outperforms the other four models on all metrics, which is in line with the results of automatic evaluation. In this part, we perform ablation experiments to evaluate the effectiveness of each component. We focus on five crucial components and set them accordingly: 1) w/o UTS Encoder denotes that we use a BiGRU to replace the unified-modal Transformer encoder for multimodal encoding; 2) w/o HTR Decoder denotes that we use a Uni-directional GRU to replace the hierarchical Transformer decoder for response generation; 3) w/o ITM denotes that we remove the LITM loss to make the parameters not updated; 4) w/o WPA denotes that we remove the LWPA loss and just regard the sentence-level representation f ( [CLS] ) as query vector Q to query knowledge; 5) w/o IR Module denotes that we remove the IC and KQ components and just adopt the context vector f ( [CLS] ) to generate responses 10 ; From Table 5, we can observe that removing each component will result in a performance degradation. Specifically, w/o IR Module causes 54.96% drops in Bleu-4 score and 54.18% drops in Nist 10 Equivalent to generating general responses, since there is no knowledge query. score, which verifies the great efficacy of intention classify and knowledge query components. Moreover, w/o WPA, w/o ITM and w/o UTS Encoder respectively cause 28.54%, 20.48% and 14.37% drops in Nist score, which further demonstrates the effectiveness of cross-modal feature alignment and unified-modal semantic encoding. To better illustrate the advantage of our model and understand what the feature alignment module has learned, we visualize several examples of text-to-image attention, as shown in Figure 4. It can be observed that our model is able to capture fine-grained entity alignment between different modalities. The reason may be that: 1) We adopt a unified-modal Transformer semantic encoder, which enables to map different modalities of semantic cues into a same vector space to prompt inter-modality interactions for better representations; 2) Based on the obtained representations, the WPA technology can help supervise fine-grained word-patch alignment, which is beneficial to identifying user's real intention and generate more intention-aware responses. In this paper, we propose a Unified Transformer Semantic Representation framework with feature alignment and intention reasoning, referred to UniTranSeR. Specifically, we project the multimodal features into a unified semantic space by utilizing a Transformer encoder to prompt inter-modal interactions. We further design a feature alignment and intention reasoning layer to conduct cross-modal feature alignment and fine-grained intention rea-110 soning, with the objective of generating more accurate and intention-aware responses. Experiments on the representative MMD dataset demonstrate the effectiveness and superior performance of our UniTranSeR model in both automatic and human evaluation.
abstain abstain objective objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective other method method method objective result other other other other other other other other other other other other other objective method abstain method abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method other abstain abstain other method other abstain method abstain abstain method abstain abstain abstain method method method abstain method method abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain method other abstain abstain abstain method abstain other abstain other abstain abstain method method method abstain method abstain abstain method method method method abstain abstain method abstain method method method abstain abstain method abstain method abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain result abstain result result result abstain abstain abstain abstain result abstain abstain result abstain abstain abstain result result result objective method objective objective