Why Does the VQA Model Answer No?: Improving Reasoning through Visual and Linguistic Inference
In order to make Visual Question Answering (VQA) explainable, previous studies not only visualize the attended region of a VQA model but also generate textual explanations for its answers. However, when the model’s answer is ‘no,’ existing methods have difficulty in revealing detailed arguments that lead to that answer. In addition, previous methods are insufficient to provide logical bases, when the question requires common sense to answer. In this paper, we propose a novel textual explanation method to overcome the aforementioned limitations. First, we extract keywords that are essential to infer an answer from a question. Second, for a pre-trained explanation generator, we utilize a novel Variable-Constrained Beam Search (VCBS) algorithm to generate phrases that best describes the relationship between keywords in images. Then, we complete an explanation by feeding the phrase to the generator. Furthermore, if the answer to the question is “yes” or “no,” we apply Natural Langauge Inference (NLI) to identify whether contents of the question can be inferred from the explanation using common sense. Our user study, conducted in Amazon Mechanical Turk (MTurk), shows that our proposed method generates more reliable explanations compared to the previous methods. Moreover, by modifying the VQA model’s answer through the output of the NLI model, we show that VQA performance increases by 1.1% from the original model.
The paper proposes a novel method for explaining VQA systems. Different from most previous work, the proposed approach generates textual explanations based on three steps. First, it extracts keywords from the question. Then an explanation sentence is decoded (based on an RNN image captioner) through the proposed Variable-Constrained Beam Search (VCBS) algorithm to satisfy the keyword constraints. Finally, 3) checking through linguistic inference whether the explanation sentence can be used as a premise to infer the question and answer. I would recommend for acceptance. The paper proposes an alternative approach to VQA explanations, together with a few supporting algorithms such as VCBS. It is potentially helpful to future work on textual explanations and explainable AI in general. At a high level, it is ambiguous to decide what is a reasonable explanation for many “no” answers. For example, one usually cannot provide stronger justification than “there is indeed no one” or “I don’t see anyone” to the question “Is there anyone in the room” with an answer “no.” The paper frames this explanation generation task as a linguistic inference task and checks entailment between the explanation and the question-answer pair. While it is debatable whether this is optimal, the proposed approach provides valuable insights on what constitutes a good explanation. However, the proposed approach also has noticeable weaknesses. It relies on external models or tools for natural language inference, and such inference does not take into account the visual context of the image. Also, the explanations generated from the proposed model only justify the answer but are not introspective, and they do not reflect the decision process of the target VQA model.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Why Does the VQA Model Answer No?: Improving Reasoning through Visual and Linguistic Inference
In order to make Visual Question Answering (VQA) explainable, previous studies not only visualize the attended region of a VQA model but also generate textual explanations for its answers. However, when the model’s answer is ‘no,’ existing methods have difficulty in revealing detailed arguments that lead to that answer. In addition, previous methods are insufficient to provide logical bases, when the question requires common sense to answer. In this paper, we propose a novel textual explanation method to overcome the aforementioned limitations. First, we extract keywords that are essential to infer an answer from a question. Second, for a pre-trained explanation generator, we utilize a novel Variable-Constrained Beam Search (VCBS) algorithm to generate phrases that best describes the relationship between keywords in images. Then, we complete an explanation by feeding the phrase to the generator. Furthermore, if the answer to the question is “yes” or “no,” we apply Natural Langauge Inference (NLI) to identify whether contents of the question can be inferred from the explanation using common sense. Our user study, conducted in Amazon Mechanical Turk (MTurk), shows that our proposed method generates more reliable explanations compared to the previous methods. Moreover, by modifying the VQA model’s answer through the output of the NLI model, we show that VQA performance increases by 1.1% from the original model.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



The paper proposes a novel method for explaining VQA systems. Different from most previous work, the proposed approach generates textual explanations based on three steps. First, it extracts keywords from the question. Then an explanation sentence is decoded (based on an RNN image captioner) through the proposed Variable-Constrained Beam Search (VCBS) algorithm to satisfy the keyword constraints. Finally, 3) checking through linguistic inference whether the explanation sentence can be used as a premise to infer the question and answer. I would recommend for acceptance. The paper proposes an alternative approach to VQA explanations, together with a few supporting algorithms such as VCBS. It is potentially helpful to future work on textual explanations and explainable AI in general. At a high level, it is ambiguous to decide what is a reasonable explanation for many “no” answers. For example, one usually cannot provide stronger justification than “there is indeed no one” or “I don’t see anyone” to the question “Is there anyone in the room” with an answer “no.” The paper frames this explanation generation task as a linguistic inference task and checks entailment between the explanation and the question-answer pair. While it is debatable whether this is optimal, the proposed approach provides valuable insights on what constitutes a good explanation. However, the proposed approach also has noticeable weaknesses. It relies on external models or tools for natural language inference, and such inference does not take into account the visual context of the image. Also, the explanations generated from the proposed model only justify the answer but are not introspective, and they do not reflect the decision process of the target VQA model.
assistant
The paper proposes a novel method for explaining VQA systems. Different from most previous work, the proposed approach generates textual explanations based on three steps. First, it extracts keywords from the question. Then an explanation sentence is decoded (based on an RNN image captioner) through the proposed Variable-Constrained Beam Search (VCBS) algorithm to satisfy the keyword constraints. Finally, 3) checking through linguistic inference whether the explanation sentence can be used as a premise to infer the question and answer. I would recommend for acceptance. The paper proposes an alternative approach to VQA explanations, together with a few supporting algorithms such as VCBS. It is potentially helpful to future work on textual explanations and explainable AI in general. At a high level, it is ambiguous to decide what is a reasonable explanation for many “no” answers. For example, one usually cannot provide stronger justification than “there is indeed no one” or “I don’t see anyone” to the question “Is there anyone in the room” with an answer “no.” The paper frames this explanation generation task as a linguistic inference task and checks entailment between the explanation and the question-answer pair. While it is debatable whether this is optimal, the proposed approach provides valuable insights on what constitutes a good explanation. However, the proposed approach also has noticeable weaknesses. It relies on external models or tools for natural language inference, and such inference does not take into account the visual context of the image. Also, the explanations generated from the proposed model only justify the answer but are not introspective, and they do not reflect the decision process of the target VQA model.