The rapid development of conversational assistants accelerates the study on conversational question answering (QA). However, the existing conversational QA systems usually answer users' questions with a single knowledge source, e.g., paragraphs or a knowledge graph, but overlook the important visual cues, let alone multiple knowledge sources of different modalities. In this paper, we hence define a novel research task, i.e., multimodal conversational question answering (MMCoQA), aiming to answer users' questions with multimodal knowledge sources via multi-turn conversations. This new task brings a series of research challenges, including but not limited to priority, consistency, and complementarity of multimodal knowledge. To facilitate the data-driven approaches in this area, we construct the first multimodal conversational QA dataset, named MMConvQA. Questions are fully annotated with not only natural language answers but also the corresponding evidence and valuable decontextualized self-contained questions. Meanwhile, we introduce an end-to-end baseline model, which divides this complex research task into question understanding, multi-modal evidence retrieval, and answer extraction. Moreover, we report a set of benchmarking results, and the results indicate that there is ample room for improvement. The ever-increasing variety of information leads to the current information explosion. Question answering (QA) systems play an important role in alleviating information overload by providing users brief and accurate answers. Towards this end, a great many QA systems have been developed by utilizing external knowledge sources to obtain the correct answer, including knowledge-based QA (Deng et al., 2019), document-based QA (Wang et al., 2018), and community-based QA (Fang et al., 2016). Recently, as the rapid Which city features a green copper statue of a woman holding a torch? development of conversational assistants, there is growing interest in all matters conversational. Conversational QA, aiming to satisfy users' complex information needs via multi-turn conversations, attracts a lot of attention. The existing conversational QA systems usually rely on a single knowledge source, e.g., paragraphs or a knowledge graph, and assume it contains enough evidence to extract answers to users' questions. However, these conversational QA systems are limited in real-world QA scenarios due to the following reasons. On the one hand, the important visual cues are overlooked in the existing conversational QA systems. As an old saying goes, a picture is worth a thousand words, namely a picture can often vividly express a lot of information. For example, as shown in Figure 1, the question Which city features a green copper statue of a woman holding a torch? can be naturally answered by looking at the related picture. On the other hand, the series of questions in a conversation may dynamically require multiple knowledge sources that encompass different modalities rather than only one constant knowledge source. As shown in Figure 1, three questions in the conversation involve images, passages, and structured tables respectively to extract the correct answers. In fact, although QA systems have been well studied thus far, conversational question answering with multiple knowledge sources of multi-modalities is still untapped. In this paper, we hence define this novel research task, i.e., multimodal conversational question answering (MMCoQA), aiming to answer users' questions with multimodal knowledge sources via multiturn conversations. MMCoQA is indeed non-trivial due to the following research challenges. 1) Priority of multimodal knowledge. For a specific question, one modality may be more suitable for locating its corresponding answer than the others. For example, questions about numerical inquiries like date or statistics are better answered by utilizing tables. Different from the previous conversational QA tasks, the most appropriate modality that can be used to answer the current question is not given in MMCoQA. Given the conversation context, how to correctly determine the appropriate modality for the current question is a challenge. 2) Consistency of multimodal knowledge. Different modalities may provide consistent evidence to extract the correct answer for a question. For example, for the first question in Figure 1, the visual modality provides intuitional and direct information, while the related paragraph The Statue of Liberty ..., off the coast of New York City. She holds a torch in her raised right hand ... also reveals a certain of cues to indicate the correct answer. How to utilize the consistency among different modalities to verify the answer is another challenge. 3) Complementarity of multimodal knowledge. Some questions may require evidences of different modalities to reason the final answer. For example, the question Billy Slater played for the NRL team in 2006 with a character holding what on the logo? must be answered based on both the table about Billy Slater's career and the image of his team logo. Therefore, to answer these questions, the system is required to have the ability of reasoning across multiple modalities. More importantly, the aforementioned three issues are not standalone but interweaved as conversation goes. Thus, MMCoQA is not the simple combination of multimodal QA and conversational QA but requires deep multimodal understanding and reasoning abilities across multi-turn conversations, which leaves ample room to study. To advance the progress of building MMCoQA systems using data-driven approaches, we construct the MMConvQA dataset, the first dataset for MMCoQA (see Table 1).Each question is fully annotated with not only the natural language answer but also the related evidence. Besides, the valuable decontextualized self-contained questions are also annotated for all questions. Hence, MMConvQA can be used to develop individual system modules for multimodal conversational search, conversational question rewrite, and multimodal QA systems. Accordingly, we introduce an end-to-end baseline model and provide a set of bench-marking results, which may facilitate a lot of exciting ongoing researches in the area. The contributions of this work are threefold:  To the best of our knowledge, this is the first work towards the multimodal conversational question answering problem. We clearly define the research scope of this task and identify its potential research challenges. We construct the first dataset, MMConvQA, for the multimodal conversational QA task. MMConvQA contains multiple supervised labels, including related evidence, answers, and decontextualized questions, which facilitates the data-driven approaches in this community. We introduce an end-to-end model as the baseline and report a set of results. Experiment results indicate the significant room for future improvement. Besides, the data and codes of this work are released 1 . Conversational QA is a relatively new topic in the QA community. Benefiting from the released dataset (Reddy et al., 2019; Choi et al., 2018), text-based conversational QA has been greatly developed. Researchers proposed to model and filter the conversation context via binary term classification (Voskarides et al., 2020) and question rewriting (Elgohary et al., 2019; Vakulenko et al., 2021; Yu et al., 2020). Recently, some efforts (Qu et al., 2020; Li et al., 2022; Anantha et al., 2021b) 1 https://github.com/liyongqi67/MMCoQA . expanded conversational QA to the open-domain setting, where the related passages must be retrieved rather than given directly. In addition to text-based conversational QA, knowledge-based conversational QA (Christmann et al., 2019) was also developed to answer conversational questions based on a knowledge base. Saha et al. (2018a) created a large-scale dataset and Shen et al. (2019) proposed a multi-task learning framework to resolve coreference in conversations and detect entities simultaneously. However, these existing methods only involved one knowledge source and the important visual cues were overlooked. Our work is also closely related to multimodal QA. Essentially, the task of VQA (Jing et al., 2020; Shah et al., 2019) is multimodal and involves images and textual questions. However, in this work, we are more interested in question answering with multimodal knowledge sources. In fact, QA with multiple mediums has been studied for a long time. For example, in the year of 2011, Nie et al. proposed to enrich textual question answering with image and video data. Besides, Textbook QA (Kemb-havi et al., 2017) and TVQA (Lei et al., 2018) were also explored under specific scenes. Recently, Hannan et al. introduced the ManymodalQA challenge, where the questions are ambiguous and the modality is not easily determined based solely upon the question. Talmor et al. innovatively introduced the complex question scenario to multimodal QA, where a complex question requires several modalities to answer. In this work, we believe that conversational QA is a natural scenario for combining multimdoal knowledge sources, where different modalities are dynamically required as the conversation moves on. In fact, there are few websites or applications where we can directly obtain a huge amount of questions that are answered with multimodal knowledge sources, let alone in the conversational form. Fortunately, we notice that the MMQA (Talmor et al., 2021) dataset contains a number of complex questions answered with multiple modalities of knowledge. Considering that an important intention of developing conversational QA systems is to gradually satisfy users' complex information needs via multi-turn conversations (Dalton et al., 2020), we intuitively propose to decompose these complex questions into conversational questions (Saha et al., 2018c). For example, as shown in Figure 2, the complex question The player not wearing a helmet in 2019-20 Buffalo Sabres season free agents was on what team? can be better presented in a conversation Q1: Which player not wear a helmet in 2019-20 Buffalo Sabres season free agents and Q2: He was on what team in that season?. However, if we obtain conversational questions only by decomposing complex questions, the number of questions in a conversation is rather limited since a complex question can only be decomposed into two or three questions. Therefore, we automatically generate potential conversations as references for annotators to refine. Generate potential conversations . Observing that the follow-up questions in a conversation are usually related to the topics that have occurred in the previous questions or answers, we thus add the questions that contain the same entities into one po-4222 Q1: The player not wearing a helmet in 19-20 Buffalo Sabres season free agents was on what team? tential conversation. Specifically, for all questions in the question pool, i.e., the MMQA dataset, we identify the entities of the question text and answer text. We then randomly select a question from the question pool as the seed of a conversation. We argue that a user may be interested in the entities that he/she have asked and the new entities occurred in the system's responses. We hence randomly select one from the identified entities in previous questions and answers as the user's next point of interest. Then we randomly select a question from the question pool that contains the selected entity as the follow-up question in the conversation. Once a question is selected to conduct a conversation, it will be removed from the question pool to keep the diversity of conducted conversations. Continually add follow-up questions until the conversation turn exceeds a certain number or there is no corresponding question in the question pool. Repeat the above process, and finally we obtain a number of artificial conversations. The automatically generated conversations are unnatural: 1) there are a lot of complex questions requiring multi-hop logical reasoning, which are not common in daily conversations (Reddy et al., 2019). 2) The sequential questions in the potential conversations lack coherence of dialogues such as coreference and ellipsis. And 3) some questions in the conversation may be not consistent with the whole conversation. Therefore, annotators are involved to manually decompose complex questions and refine (including rewrite, delete and rearrange) the conversational questions towards the real conversation scenario. Decompose complex questions . To facilitate the decomposition of complex questions, the types and intermediate answers of complex questions provided in the MMQA dataset, are also shown to the annotators. The types of questions indicate the logic and the target number of decomposed questions for a complex question. For example, Q1 of the Potential Conversation in Figure 2 is a complex question and its type is Compose(TableQ, Im-ageQ). Compose(A,B) means question A containing an entity is the answer of question B, while TableQ and ImageQ indicate that questions A and B can be answered with tables and images, respectively. Therefore, the annotator can easily decompose this complex question into two sequential questions according to its type. Notably, each annotator is required to decompose a complex question into self-contained questions that can be answered without the conversation context. Refine conversational questions . After the decomposition, the same annotator refines conversational questions for an artificial conversation. Each annotator is showed some typical examples in the existing conversational QA datasets before the annotation. After fully understanding the linguistic phenomena in conversational QA, such as coreference and ellipsis, annotators write conversational questions for artificial conversations. It is worth mentioning that they have the right to delete and rearrange questions in a conversation to guarantee the smooth conversation flow. They can also report to delete a whole conversation if they think it is poor-quality. Data quality . Four students that majored in computer science and have NLP research experience are invited to make annotations. To ensure the quality of collected conversation data, we apply the 5-step scheme of training, annotation, checking, modification, and re-checking. Before the collection of data, we carry out training for all participants to 4223 Table 2: Statistics for MMConvQA dataset. explain the annotation guidelines (see Appendix A) for about two hours. Each conversation will be checked by another annotator and the unqualified ones will be returned to modify. It is worth mentioning that since we only write conversational questions rather than give answers to questions, we do not need to calculate the annotation agreement of answers. MMConvQA contains 1,179 conversations and 5,753 QA pairs. There are 4.88 QA pairs on average for each conversation, as summarized in Table 2. The multimodal knowledge collection consists of 218,285 passages, 10,042 tables, and 57,058 images. Each question is annotated with the related evidence (a table, an image or a passage in the knowledge collection), and a natural language answer. Besides, each question is also accompanied with a corresponding self-contained question. Question Analysis . Figure 3 shows sunburst plots of question types in MMConvQA. We can see that most of the first words are similar to those questions in other conversational QA datasets (Choi et al., 2018; Reddy et al., 2019; Saha et al., 2018b). The and In are frequently used because they usually relate to the coherence of conversations, such as The actor. There are also some special patterns in MMConvQA featuring multi-modalities. For example, What Color pattern is related to the visual modality and How Many may refer to the tables. On average, each question contains 14.4 words, while this number in the MMQA dataset is 19.2. This illustrates that we well decompose the W h a t w a s i s c o l o r y ea r I n p a r t r o l e t h e w h a t T h e ac t o r W ho W h i c h W h e n A m ong I s H o w W h e r e i s m a ny t h e t h e d i d l ook i ng w a s w a s p l a y e d i s p l a y s Figure 3: Distribution of the digram prefixes of questions in the MMConvQA dataset. complex questions. The average number of words for gold questions is 15.5, which is slightly bigger than that of the conversational questions. This is because conversational questions embody the linguistic phenomena of dialogues, such as coreference and ellipsis, thus have less words than gold questions. It is worth mentioning that two different complex questions may produce the same single question. Therefore, there are some duplicated questions in a conversation. Answer Analysis . The types of answers in MMConvQA are diverse. Most of answers are text spans of passages, cells of tables, and titles of images, whereas some answers do not exactly overlap with the evidence. For example, the answer to the question The singer of Take Me As I Am' is shown wearing what item on her neck? is scarf, which needs to be detected based on a related image rather than the title of the image. Apart from single answers, 9.9% questions require a list of answers. For example, the answer to the question who is the owner of cape town knight riders? is Shah Rukh Khan and Juhi Chawla. On average, each answer contains 2.11 words. Modality Analysis . As summarized in Table 2, there are 45.6% questions can be answered with textual passages. Besides, 29.8% and 24.6% questions must be answered based on images and tables, respectively. Among the 1,179 conversations in this dataset, 57.7% conversations involve two different modalities of knowledge and 24.4% conversations involve three modalities. This indicates that as conversations proceed, questions dynamically require different modalities of knowledge to answer. To better illustrate the conversation flow, 4224 T e x tQ Im a g e Q T a b le Q T e x tQ Im a g e Q T a b le Q T e x tQ Im a g e Q T a b le Q T e x tQ Im a g e Q T a b le Q T e x tQ Im a g e Q T a b le Q T e x tQ Im a g e Q T a b le Q T e x tQ Im a g e Q T a b le Q T e x tQ Im a g e Q T a b le Q 1 2 3 4 5 6 7 8 Figure 4: Transitions of modalities as conversation goes. we visualize the transition of modalities as conversation progresses in Figure 4. It is observed that the transitions of modalities are frequent. For example, about 70% table questions at the first turn transform to the text and the image questions at the second turn. And as the turn number increases, the bonds become cluttered, which indicates that more conversations involve multiple modalities. Linguistic Phenomena . To measure the quality of the conversational questions and analyze their linguistic phenomena, we sample 100 follow-up questions in the development set and annotate various phenomena. Our analysis shows that around 33% questions do not rely on coreference with the conversational history and are answerable on their own. Around 57% questions contain explicit coreference markers such as he, she, it. The remaining 10% do not have explicit coreference markers but refer to an entity or event implicitly. Another feature of open-retrieval conversational QA is the topic switch. Among the questions, 24% change the conversation topic (WikiEntity). We introduce a M ultimodal Conversational QA system with A daptive E xtractors, MAE for short, as a baseline model. As Figure 5 illustrates, MAE divides the MMCoQA task into three steps: conversational question understanding, multimodal evidence retrieval, and adaptive answer extraction. Assume that the current turn in a conversation is k and the current question is q k . The conversation context for the current question q k is denoted as H k = { q 1 , a 1 , ..., q k  1 , a k  1 } . A multimodal knowledge collection that contains different modalities of items is given, denoted as C = {C p C t C i } , where C p , C t , and C i are the sets of passages, tables, and images, respectively. The system is required to retrieve the related evidence from the knowledge collection C and extract a natural language span  a k to answer the question q k . To understand the current question with the conversation context H k , we apply the sliding window mechanism (Qu et al., 2020) to filter the previous questions. We feed the reformatted question q  k into the BERT network (Devlin et al., 2019) to obtain the question representation, which is formulated as, v q = W q F q ( q  k ) , (1) where F q is the BERT based question encoder, W q is the question projection matrix, and v q  R d q . The table's representation is computed via, v j t = W t F t ( t  j ) , (3) where F t is the BERT based encoder and v jt  R d t . For different modalities of items in C , we pass them to different knowledge encoders. For each passage p j in C p , we obtain its representation v ip as follows, v jp = W p F p ( p j ) , (2) where F p is the BERT based passage encoder, W p is the passage projection matrix, v jp  R d p . Following the prior work (Herzig et al., 2020; Talmor et al., 2021), we linearize tables by rows as t  j to obtain their representations. v ji = W i F i ( i j ) , (4) where F i is a pretrained Resnet (He et al., 2016) network on ImageNet, and v ji  R d i . Noticed that d q , d p , d t , d i have the same dimension. To facilitate large-scale retrieval, we apply the dense retriever mechanism inspired from open-domain QA (Karpukhin et al., 2020). Differently, 4225 Images Documents Tables Multimodal Knowledge Collection Question Encoder MultimodalKnowledge Encoders Multimodal Knowledge Space Text Extractor Table Extractor Image Extractor Modality Detection Answer Ranking Question Representation Passage Representation Table Representation Image Representation On what dates was the Better with U Tour in the city? we have three knowledge encoders F p , F t , F i , and they are independent from questions in order to enable strong precomputed multimodal encodings and execute the efficient maximum inner product search (Lee et al., 2019). We first pretrain the question encoder and the knowledge encoders and then input all items in C into knowledge encoders to obtain their representations. The parameters of the knowledge encoders are frozen in the following training phases. Benefiting from this, we can efficiently calculate the similarity s a between a given question embedding v q and all knowledge item embeddings via the inner product, and select the topN r items I r as evidence, where N r is the number of the retrieved items. The retrieved evidence list I r contains items of different modalities, and different modalities need different answer extractors. We hence first detect the most appropriate modality for the question. We regard the modality detection as a multi-class classification task where the network takes a question as input to predict the probabilities of three modalities. The classifier is formulated as, s b = f ( W c F c ( q  k )) , (5) where f () denotes the softmax function, F c is the question encoder and s b  R 3 . TextExtractor . It is basically a machine read-ing comprehension model. Given the reformulated question and a passage in I r as input, TextExtractor predicts an answer span by computing two scores for each token in a passage in P r to be the start token and the end token, respectively. TableExtractor . Following the previous work (Herzig et al., 2020), we concatenate the question text to the linearized table sequence, and encode them using BERT. Two linear classifiers are then followed to compute the probability of the token being the start token and the end token of the answer span, respectively. ImageExtractor . We collect the answers in the training set as the answer set for testing (Talmor et al., 2021). We extract the visual feature v i for an image with the ResNet, and append the question text with all the answers in the answer set as a text sequence. And then we input the text sequence into the BERT to obtain the representations for all tokens, which are then simply combined with the visual feature v i . Similarly, two linear classifiers are then followed to compute the probability of the token in the text sequence being the start token and the end token. The answer extraction score s c for a candidate answer predicted by the above three extractors is defined as the average of the probabilities of the start and the end token. For each candidate answer, we compute its final score as the sum of the retrieval score s a , the modality score s b , and the answer extraction score s c . The training details are illustrated in Appendix B. 6 Experiments 6.1 Evaluation Protocols We comprehensively evaluated the baseline models based on their performance in evidence retrieval and answer extraction. We adopted Recall and NDCG to evaluate the coverage and the rank position of the retrieval list. Following previous conversational QA tasks (Reddy et al., 2019), we reported macro-average F1 in the word level and Exact Match (EM) to estimate the performance of answer extraction. We evaluated the open-retrieval conversational QA system ORConvQA and a multimodalQA model ManyModalQA on our MMCoQA dataset. And 4226 Table 3: Performance of various methods on the test set. to better illustrate the characteristics of the dataset, we developed several variants of the MAE: including w/o conversation context , gold question , gold answer , evidence given , QR(question rewrite) , and pretrain . Please see Appendix C for the implement details. The results are summarized in Table 3. By analyzing the results, we gained the following insights. (1) The existing open-retrieval conversational QA and multimodal QA methods cannot handle the MMCoQA problem well, since they are either single-modal or single-turn. (2) The results of the MAE variants partly evaluate the quality of the MMConvQA dataset. When the conversation context is removed, the performance drops, which verifies the dependency on the conversation context. Appending the previous gold answers or directly using the gold questions improve the performance, which is consistent with the dataset construction strategy. (3) Using extra data benefits the model, which illustrates that the size of the dataset is kind of small and pretraining can alleviate this problem. (4) When we manually complemented the relevant evidence into the retrieval list, it outperforms the normal MAE model a lot. It seems that the evidence retrieval is a bottleneck for the current model because the relation among multimodal knowledge is complex as claimed before. Modality analysis . We summarized the performance of MAE-EG on the three different modal questions in Figure 6(a). It can be seen that the performance on ImageQ is the worst. It may be because that our ImageExtractor is a little coarse and 1 2 3 4 5 6 0 1 0 2 0 3 0 4 0 5 0 6 0 F 1 T u rn N u m b er T ex tQ T ab leQ Im ag eQ more fine-grained interactions are expected. Besides, we selected some items that associated with same entities and visualized their embeddings in Figure 6(b). It is observed that the images' embeddings are isolated, which illustrates that the visual and semantic meanings are not well-aligned. Some text's and tables' embeddings are partly syncretic, but it is still far away from an ideal common space where the embeddings of different modal items are evenly distributed according to their meanings. It seems that the successful dense retrieval scheme for document retrieval needs to be further modified for the multi-modal retrieval. We define a novel and practical task, i.e., MMCoQA, and identify its research challenges, including priority, consistency, and complementarity of multimodal knowledge. We construct the MMConvQA dataset, containing multiple supervised labels to facilitate related researches in this community. We also report a set of results and analyze the current bottleneck. The work described in this paper was supported by Research Grants Council of Hong Kong(PolyU/5210919, PolyU/15207821), National Natural Science Foundation of China (62076212) and PolyU internal grants (ZVQ0).
abstain abstain objective abstain objective abstain method result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain abstain result objective method objective objective result objective abstain other other other other other other other other abstain other abstain other other other other other method result abstain abstain abstain result method abstain result abstain method abstain objective method method abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain method abstain abstain method method abstain abstain abstain method method method method method abstain abstain abstain abstain result abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective method result other